feat!: Expose consensus debug port, restart EN on config file change (#42)

## What ❔

Subj

## Why ❔

QoL

## Checklist

<!-- Check your PR fulfills the following items. -->
<!-- For draft PRs check the boxes as you complete them. -->

- [x] PR title corresponds to the body of PR (we generate changelog
entries from PRs).
- [ ] Documentation comments have been added / updated.

.ansible-lint

@@ -2,3 +2,6 @@ skip_list:
   - 'yaml'
   - 'risky-shell-pipe'
   - 'role-name'
+
+exclude_paths:
+  - example_playbooks

.github/ISSUE_TEMPLATE/bug_report.md

@@ -26,10 +26,11 @@ Describe what actually happened.
 
 #### 🖥️ Environment
 
-Any relevant environment details like: 
+Any relevant environment details like:
+
 * Ansible version
 * Operating system
-* External node version
+* External Node version
 
 #### 📋 Additional Context
 

CONTRIBUTING.md

@@ -5,7 +5,7 @@ sovereignty! We welcome contributions from anyone on the internet, and are grate
 
 ## Ways to contribute
 
-There are many ways to contribute to the external node role:
+There are many ways to contribute to the External Node role:
 
 1. Open issues: if you find a bug, have something you believe needs to be fixed, or have an idea for a feature, please
    open an issue.

README.md

@@ -1,6 +1,6 @@
 # ansible-en-role
 
-Ansible role to deploy and configure zkSync Era External Node, including DB isntance setup on the same machine, Traefik as reverse proxy, and Prometheus monitoring (PostgreSQL exporter, Node exporter, cAdvisor, Traefik, External Node native metrics, and VictoriaMetrics vmagent to scrape all of them).
+Ansible role to deploy and configure zkSync Era External Node, including DB instance setup on the same machine, Traefik as reverse proxy, and Prometheus monitoring (PostgreSQL exporter, cAdvisor, Traefik, External Node native metrics, and VictoriaMetrics vmagent to scrape all of them).
 
 Make sure to configure Prometheus remote write endpoint to send metrics to centralized metrics storage.
 
@@ -14,7 +14,9 @@ This role has been tested on:
 
 ## Usage
 
-This role contains variables which has to be set:
+For a very simple minimal working example, see example_playbooks directory
+
+Minimal required variables that have to be set:
 
 ```yaml
 database_name: ""
@@ -26,6 +28,17 @@ l1_chain_id: ""
 l2_chain_id: ""
 ```
 
+Additional arbitrary environment variables can be passed to External Node container:
+
+```yaml
+additional_env_vars:
+  - { name: "EN_ADDITIONAL_VAR1", value: "Value1" }
+  - { name: "EN_ADDITIONAL_VAR2", value: "Value2" }
+  - { name: "EN_ADDITIONAL_VAR3", value: "Value3" }
+```
+
+Please refer to [External Node docs](https://github.com/matter-labs/zksync-era/tree/main/docs/guides/external-node/prepared_configs) to find values for different zkSync Era chains.
+
 If you want to use monitoring (which we highly recommend), you have to change these variables:
 
 ```yaml
@@ -40,7 +53,7 @@ prometheus_remote_write_auth_password: "password"
 prometheus_remote_write_common_label: "matterlabs"
 ```
 
-This role also has option to secure your server and allow traffic only from specified IP address in case if you want
+This role also has the option to secure your server and allow traffic only from specified IP address in case if you want
 to use some load balancer in front of your node, while not having fancy cloud security groups at your disposal:
 
 ```yaml
@@ -54,7 +67,7 @@ iptables_packages:
 loadbalancer_ip: "1.2.3.4"
 ```
 
-In most of cases, you'd want to change PostgreSQL parameters (we recommend to use <https://pgtune.leopard.in.ua/> with "Online transaction processing system" preset as sane defaults), so you can do it using `postgres_arguments` variable, eg:
+In most cases, you'd want to change PostgreSQL parameters, so you can do it using `postgres_arguments` variable, eg:
 
 ```yaml
 postgres_arguments:
@@ -66,23 +79,39 @@ postgres_arguments:
   - -c
 ```
 
-We recommend using pgtune [online]<https://pgtune.leopard.in.ua/> or [self-hosted](https://github.com/le0pard/pgtune) version with with "Online transaction processing system" preset as a good starting point for generating optimal config for your hardware.
+We recommend using pgtune [online](https://pgtune.leopard.in.ua/) or [self-hosted](https://github.com/le0pard/pgtune) version with "Online transaction processing system" preset as a good starting point for generating optimal config for your hardware.
+
+If you want to use basic auth for inbound requests, you have to change next variables:
+
+```yaml
+enable_basic_auth: true
+basic_auth_secret: "htpasswd-generated-secret"
+```
+
+Basic auth secret can be generated by `htpasswd` and `sed` for interpolation:
+```echo $(htpasswd -nb <username> <password>) | sed -e s/\\$/\\$\\$/g```
 
 ## Step-by-step guide
 
-1. Install ansible collection on your machine from where you will run ansible:
+1. Install the ansible collection on your machine from where you will run ansible:
 `ansible-galaxy collection install community.general`
 
-2. Prepare latest database backup on your host. you can download it from our [public GCS bucket](https://storage.googleapis.com/zksync-era-mainnet-external-node-backups/external_node_latest.pgdump).
-you should place it to `{{ storage_directory }}/pg_backups` directory. By default, `{{ storage_directory }}` is `/usr/src/en`
+2. Prepare the latest database backup on your host. you can download it from our public GCS buckets:
+Skip this step if you are recovering from a snapshot!
 
-3. **OPTIONAL**: If you already have external-node, you can copy tree directory to new host. Copy external-node database tree to `{{ storage_directory }}/db`.
+* [Era Mainnet latest dump](https://en-backups.matterlabs.dev/)
+* [Era Sepolia Testnet latest dump](https://storage.googleapis.com/zksync-era-testnet-sepolia-external-node-backups/external_node_latest.pgdump)
 
-**Keep in mind, tree should be older than PostgreSQL database backup.**
+Downloaded dump, if needed, should be unarchived and named `external_node_latest.pgdump`. File should be placed into `{{ storage_directory }}/pg_backups` directory (`/usr/src/en/pg_backups` by default).
 
-4. Run ansible-playbook using this role. We recommend to encrypt next variables with ansible-vault or some another way:
+3. **OPTIONAL**: If you already have running node, you can copy its tree and state directory to a new host's `{{ storage_directory }}/db`. (`/usr/src/en/db` by default)
+Skip this step if you are recovering from a snapshot!
 
-```
+**Keep in mind that tree and state should be older than PostgreSQL database backup.**
+
+4. Run ansible-playbook using this role. We recommend encrypting next variables with ansible-vault or some another way:
+
+```yaml
 database_username
 database_password
 eth_l1_url
@@ -92,6 +121,20 @@ vm_auth_password
 
 5. Connect to your host, and see status of `postgres` container. It can take a lot of time before PostgreSQL database backup will be restored (hours to days, depending on your disk throughput and IOPS), after which PostgreSQL server will be ready for use. Once `postgres` becomes "healthy", `external_node` runs automatically.
 
+## Snapshots Recovery
+
+Example config enabling recovery from a snapshot:
+
+```yaml
+- enable_snapshots_recovery: true
+- snapshots_bucket_base_url: "snapshots-bucket-name"
+```
+
+Snapshot buckets:
+
+* Era Mainnet: `zksync-era-mainnet-external-node-snapshots`
+* Era Sepolia Testnet: `zksync-era-boojnet-external-node-snapshots`
+
 ## Example Playbook
 
 ```yaml
@@ -107,7 +150,6 @@ vm_auth_password
     l2_chain_id: "324"
     l1_chain_id: "1"
     enable_tls: false
-    partner_id: matterlabs
   vars_files:
     - secrets/mainnet_secrets.yml
   roles:

defaults/main.yml

@@ -8,12 +8,12 @@ docker_install_compose: true
 docker_version: "25.0.3"
 docker_compose_version: "v2.23.0"
 
-# Versions of external node and 3rd party components
+# Versions of External Node and 3rd party components
 traefik_version: 2.11
 postgres_version: 14
-external_node_version: 21.0.2
-vmagent_version: 1.95.1
-node_exporter_version: 1.7.0
+external_node_version: 27.2.0
+external_node_raw_docker_tag: ""
+vmagent_version: 1.100.1
 cadvisor_version: 0.47.2
 postgres_exporter_version: 0.15.0
 
@@ -56,16 +56,45 @@ postgres_arguments:
   - max_parallel_maintenance_workers=4
   - -c
   - checkpoint_timeout=1800
+enable_postgres_replication: false
+# IP address of the interface replication
+postgres_replications_arguments: []
+postgres_replica_user_name: ""
+postgres_replica_user_password: ""
+postgres_replica_auth_method: "scram-sha-256"
+postgres_replication_bind_address: ""
+postgres_replica_address: ""
+backup_db_user: ""
+backup_db_password: ""
+backup_db_name: ""
 
 # Enable TLS for traefik
 enable_tls: false
 acme_email: ""
 domain_name: ""
 
+# Enable basic auth for External Node
+enable_basic_auth: false
+basic_auth_secret: ""
+
 # Force restore pg database
 force_pg_restore: false
 
-# External node and database options
+# Use a snapshot to recover
+enable_snapshots_recovery: false
+snapshots_bucket_base_url: ""
+
+# https://github.com/matter-labs/zksync-era/blob/main/docs/guides/external-node/09_decentralization.md
+enable_consensus: false
+consensus_secrets_file: ""
+consensus_port: 3054
+consensus_outbound: []
+consensus_debug_port: 5000
+enable_consensus_debug_port: false
+expose_consensus_debug_port: false
+consensus_debug_port_path_prefix: "/consensus_debug"
+
+# External Node and database options
 database_name: ""
 database_username: ""
 database_password: ""
@@ -77,6 +106,10 @@ rpc_http_port: 3060
 rpc_ws_port: 3061
 healthcheck_port: 3081
 metrics_port: 3082
+rust_log: zksync_external_node=info,zksync_core=info,zksync_core::sync_layer=info,zksync_server=info,zksync_prover=info,zksync_contract_verifier=info,zksync_dal=info,zksync_eth_client=info,zksync_storage=info,zksync_db_manager=info,zksync_merkle_tree=info,zksync_state=info,zksync_utils=info,zksync_types=info,loadnext=info,dev_ticker=info,vm=info,block_sizes_test=info,zksync_verification_key_generator_and_server=info,zksync_object_store=info,setup_key_generator_and_server=info,zksync_circuit_synthesizer=info,zksync_queued_job_processor=info,zksync_health_check=info
+
+# Additional env vars passed to External Node
+additional_env_vars: []
 
 # Monitoring options section
 enable_monitoring: false

example_playbooks/mainnet_with_snapshots_recovery/README.md

@@ -0,0 +1,23 @@
+# Mainnet Snapshots Recovery playbook
+
+This directory is simple example how to set up EN using this role. It comes with snapshots recovery enabled by default.\
+**Note that for simplicity it's using postgres database
+with a very unsecure password and the EN is just started on the same machine**
+
+To run this playbook, first install dependencies
+
+```shell
+ ansible-galaxy install -r requirements.yml
+ ```
+
+and then you can run the playbook using
+
+```shell
+ ansible-playbook playbook.yml -i hosts.ini -K
+ ```
+
+To see logs you can use
+
+```shell
+ docker logs en-external_node-1
+ ```

example_playbooks/mainnet_with_snapshots_recovery/hosts.ini

@@ -0,0 +1,2 @@
+[local]
+localhost ansible_connection=local

example_playbooks/mainnet_with_snapshots_recovery/playbook.yml

@@ -0,0 +1,16 @@
+---
+- hosts: all
+  become: true
+  vars:
+    database_name: "zksync_ext_node_mainnet"
+    database_username: "postgres"
+    database_password: "notsecurepassword"
+    eth_l1_url: "https://ethereum-rpc.publicnode.com"
+    main_node_url: "https://zksync2-mainnet.zksync.io"
+    l1_chain_id: "1"
+    l2_chain_id: "324"
+    enable_snapshots_recovery: true
+    snapshots_bucket_base_url: "zksync-era-mainnet-external-node-snapshots"
+
+  roles:
+    - external_node

example_playbooks/mainnet_with_snapshots_recovery/requirements.yml

@@ -0,0 +1,17 @@
+---
+roles:
+  - name: geerlingguy.docker
+    src: https://github.com/geerlingguy/ansible-role-docker
+    version: "7.1.0"
+  - name: external_node
+    src: https://github.com/matter-labs/ansible-en-role
+    version: "v3.3.0"
+
+collections:
+  - name: community.docker
+      version: 4.5.2
+  - name: community.general
+    version: 8.4.0
+  # Collection for the replication only.
+  - name: community.postgresql
+    version: 3.7.0

handlers/main.yml

@@ -0,0 +1,8 @@
+---
+- name: Restart external-node service
+  community.docker.docker_compose_v2:
+    project_src: "{{ configuration_directory }}"
+    files: "{{ docker_compose_files }}"
+    state: restarted
+    services:
+      - external_node

meta/main.yml

@@ -2,12 +2,12 @@
 dependencies:
   - src: geerlingguy.docker
     version: "7.1.0"
-    when: docker_install
+    when: docker_install_compose
 
 galaxy_info:
   role_name: external_node
   author: matter-labs
-  description: External node setup
+  description: External Node setup
   license: "license (MIT, APACHE)"
   min_ansible_version: "2.13.9"
   platforms:

tasks/firewall.yml

@@ -49,6 +49,23 @@
     source: "{{ loadbalancer_ip | mandatory }}"
     jump: ACCEPT
 
+- name: Allow consensus port traffic from any IP
+  when: enable_consensus
+  ansible.builtin.iptables:
+    chain: INPUT
+    protocol: tcp
+    destination_port: "{{ consensus_port }}"
+    jump: ACCEPT
+
+- name: Allow postgres replication traffic from replica only
+  when: enable_postgres_replication
+  ansible.builtin.iptables:
+    chain: INPUT
+    protocol: tcp
+    destination_port: 5432
+    source: "{{ postgres_replica_address }}"
+    jump: ACCEPT
+
 - name: Set default policy to DROP
   ansible.builtin.iptables:
     chain: INPUT

tasks/main.yml

@@ -9,3 +9,7 @@
 
 - name: Prepare configs
   ansible.builtin.include_tasks: provision.yml
+
+- name: Configure replication on main instance
+  ansible.builtin.include_tasks: replication.yml
+  when: enable_postgres_replication

tasks/provision.yml

@@ -32,13 +32,37 @@
     - l2_chain_id
     - l1_chain_id
 
+- name: "Verify that required variables for replication is set"
+  when: enable_postgres_replication
+  ansible.builtin.assert:
+    that:
+      - postgress_replication_required_var != ""
+    fail_msg: "{{ postgress_replication_required_var }} needs to be set for the role for postgres replication to work"
+    success_msg: "Required variable for postgres replication {{ postgress_replication_required_var }} isn't empty"
+  loop_control:
+    loop_var: postgress_replication_required_var
+  with_items:
+    - enable_postgres_replication
+    - postgres_replication_bind_address
+    - postgres_replica_address
+    - postgres_replications_arguments
+    - postgres_replica_user_name
+    - postgres_replica_user_password
+
 - name: Check required en vars empty
   ansible.builtin.fail:
     msg: "Variable '{{ item }}' is empty"
   when: vars[item] == ""
   with_items: "{{ en_required_variables }}"
+- name: "Verify consensus debug port configuration"
+  ansible.builtin.fail:
+    msg: "Cannot expose consensus debug port (expose_consensus_debug_port=true) if it is not enabled (enable_consensus_debug_port=false)."
+  when:
+    - enable_consensus
+    - expose_consensus_debug_port
+    - not enable_consensus_debug_port
 
-- name: Copy main configs
+- name: Create main configs
   ansible.builtin.template:
     src: '{{ item.src }}'
     dest: '{{ item.dest }}'
@@ -50,8 +74,10 @@
       dest: "{{ configuration_directory }}/external_node.env"
     - src: "templates/postgres.env.j2"
       dest: "{{ configuration_directory }}/postgres.env"
+  loop_control:
+    label: "{{ item.dest }}"
 
-- name: Copy restore script
+- name: Create restore script
   register: restore_dump_script
   ansible.builtin.template:
     src: 'templates/restore_dump.sh.j2'
@@ -64,7 +90,7 @@
   when: enable_monitoring and ( vars[item] == "" )
   with_items: "{{ monitoring_required_variables }}"
 
-- name: Copy monitoring configs
+- name: Create monitoring configs
   when: enable_monitoring
   ansible.builtin.template:
     src: '{{ item.src }}'
@@ -76,23 +102,32 @@
     - src: "templates/vmagent-config.yml.j2"
       dest: "{{ configuration_directory }}/vmagent-config.yml"
 
-- name: Run docker-compose without monitoring
-  when: not enable_monitoring
-  ansible.builtin.shell:
-    cmd: nohup docker compose -f docker-compose.yaml up -d &
-    chdir: "{{ configuration_directory }}"
-  changed_when: false
+- name: Create consensus config
+  when: enable_consensus
+  ansible.builtin.template:
+    src: "templates/consensus_config.yaml.j2"
+    dest: "{{ configuration_directory }}/consensus_config.yaml"
+    mode: '0644'
+  notify: Restart external-node service
 
-- name: Run docker-compose with monitoring
-  when: enable_monitoring and (not restore_dump_script.changed)
-  ansible.builtin.shell:
-    cmd: nohup docker compose -f monitoring.yaml -f docker-compose.yaml up -d &
-    chdir: "{{ configuration_directory }}"
-  changed_when: false
+- name: Decrypt consensus_secrets
+  when: enable_consensus
+  ansible.builtin.copy:
+    src: "{{ consensus_secrets_file }}"
+    dest: "{{ configuration_directory }}/consensus_secrets.yaml"
+    decrypt: true
+    mode: '0600'
+  notify: Restart external-node service
 
-- name: Run docker-compose with monitoring with recreation
-  when: enable_monitoring and restore_dump_script.changed
-  ansible.builtin.shell:
-    cmd: nohup docker compose -f monitoring.yaml -f docker-compose.yaml up -d --force-recreate &
-    chdir: "{{ configuration_directory }}"
-  changed_when: false
+- name: Set docker compose files list
+  ansible.builtin.set_fact:
+    docker_compose_files: "{{ ['docker-compose.yaml'] + (['monitoring.yaml'] if enable_monitoring else []) }}"
+
+- name: Run docker compose services (non-blocking)
+  community.docker.docker_compose_v2:
+    project_src: "{{ configuration_directory }}"
+    files: "{{ docker_compose_files }}"
+    state: present
+    pull: "{{ docker_pull_policy | default('missing') }}"
+    recreate: "{{ 'always' if restore_dump_script.changed else 'auto' }}"
+    wait: false

tasks/replication.yml

@@ -0,0 +1,60 @@
+---
+
+- name: Install libpq-dev packages
+  ansible.builtin.apt:
+    update_cache: true
+    name: libpq-dev
+
+- name: Install psycopg2 python package
+  ansible.builtin.pip:
+    name: psycopg2
+
+- name: Grant user replication access for replication.
+  community.postgresql.postgresql_pg_hba:
+    dest: "{{ storage_directory }}/postgres/pg_hba.conf"
+    contype: host
+    users: "{{ postgres_replica_user_name }}"
+    source: "{{ postgres_replica_address }}/32"
+    databases: replication
+    method: "{{ postgres_replica_auth_method }}"
+
+- name: Create postgres replication user
+  community.postgresql.postgresql_user:
+    login_host: "{{ postgres_replication_bind_address }}"
+    login_user: "{{ database_username }}"
+    login_password: "{{ database_password }}"
+    name: "{{ postgres_replica_user_name }}"
+    password: "{{ postgres_replica_user_password }}"
+    role_attr_flags: "REPLICATION"
+
+- name: Create replication slot if doesn't exist
+  community.postgresql.postgresql_slot:
+    login_host: "{{ postgres_replication_bind_address }}"
+    login_user: "{{ database_username }}"
+    login_password: "{{ database_password }}"
+    slot_name: replica
+
+- name: Reload postgres configuration
+  community.postgresql.postgresql_query:
+    login_host: "{{ postgres_replication_bind_address }}"
+    login_user: "{{ database_username }}"
+    login_password: "{{ database_password }}"
+    query: "SELECT pg_reload_conf()"
+
+- name: Create postgres backup user
+  community.postgresql.postgresql_user:
+    login_host: "{{ postgres_replication_bind_address }}"
+    login_user: "{{ database_username }}"
+    login_password: "{{ database_password }}"
+    name: "{{ backup_db_user }}"
+    password: "{{ backup_db_password }}"
+
+- name: Grant role pg_read_all_data to backup user
+  community.postgresql.postgresql_membership:
+    login_host: "{{ postgres_replication_bind_address }}"
+    login_user: "{{ database_username }}"
+    login_password: "{{ database_password }}"
+    group: pg_read_all_data
+    target_roles:
+      - "{{ backup_db_user }}"
+    state: present

templates/consensus_config.yaml.j2

@@ -0,0 +1,13 @@
+server_addr: '0.0.0.0:3054'
+public_addr: '{{ ansible_default_ipv4.address }}:{{ consensus_port }}'
+max_payload_size: 5000000
+gossip_dynamic_inbound_limit: 200
+{% if enable_consensus_debug_port %}
+debug_page_addr: "0.0.0.0:{{ consensus_debug_port }}"
+{% endif %}
+rpc_config:
+  get_block_rate:
+    burst: 5
+    refresh: # 0.2s
+      seconds: 0
+      nanos: 200000000

templates/docker-compose.yaml.j2

@@ -8,10 +8,13 @@ services:
       - "--log.level=INFO"
       - "--providers.docker=true"
       - "--providers.docker.exposedbydefault=false"
-      - "--entrypoints.web.address=:80"
-      - "--entrypoints.external_node_health.address=:3080"
+      - "--entryPoints.web.address=:80"
+      - "--entryPoints.external_node_health.address=:3080"
+{% if enable_consensus %}
+      - "--entryPoints.external_node_consensus.address=:{{ consensus_port }}"
+{% endif %}
 {% if enable_tls %}
-      - "--entrypoints.websecure.address=:443"
+      - "--entryPoints.websecure.address=:443"
       - "--certificatesresolvers.en_resolver.acme.tlschallenge=true"
       - "--certificatesresolvers.en_resolver.acme.storage=/letsencrypt/acme.json"
       - "--certificatesresolvers.myresolver.acme.email={{ acme_email }}"
@@ -40,14 +43,27 @@ services:
       - ./restore_dump.sh:/docker-entrypoint-initdb.d/restore_dump.sh
     env_file:
       - postgres.env
+{% if enable_postgres_replication %}
+    ports:
+      - "{{ postgres_replication_bind_address }}:5432:5432"
+{% endif %}
     command:
       - postgres
       - -c
 {% for argument in postgres_arguments %}
       - {{ argument }}
 {% endfor %}
+{% if enable_postgres_replication %}
+{% for repl_argument in postgres_replications_arguments %}
+      - {{ repl_argument }}
+{% endfor %}
+{% endif %}
   external_node:
+{% if not external_node_raw_docker_tag %}
     image: "matterlabs/external-node:v{{ external_node_version }}"
+{% else %}
+    image: "matterlabs/external-node:{{ external_node_raw_docker_tag }}"
+{% endif %}
     restart: unless-stopped
     depends_on:
       postgres:
@@ -64,16 +80,45 @@ services:
       - "traefik.http.routers.external_node_main.entrypoints=web"
 {% endif %}
       - "traefik.http.routers.external_node_main.service=external_node_main"
-
       - "traefik.http.services.external_node_health.loadbalancer.server.port={{ healthcheck_port }}"
       - "traefik.http.routers.external_node_health.rule=PathPrefix(`/`)"
       - "traefik.http.routers.external_node_health.entrypoints=external_node_health"
       - "traefik.http.routers.external_node_health.service=external_node_health"
+{% if enable_basic_auth %}
+      - "traefik.http.routers.external_node_main.middlewares=external_node_auth"
+      - "traefik.http.middlewares.external_node_auth.basicauth.users={{ basic_auth_secret }}"
+{% endif %}
+{% if enable_consensus %}
+      - "traefik.tcp.services.external_node_consensus.loadbalancer.server.port={{ consensus_port }}"
+      - "traefik.tcp.routers.external_node_consensus.rule=HostSNI(`*`)"
+      - "traefik.tcp.routers.external_node_consensus.entrypoints=external_node_consensus"
+      - "traefik.tcp.routers.external_node_consensus.service=external_node_consensus"
+{% endif %}
+{% if enable_consensus and expose_consensus_debug_port %}
+      - "traefik.http.services.external_node_consensus_debug.loadbalancer.server.port={{ consensus_debug_port }}"
+      - "traefik.http.routers.external_node_consensus_debug.rule=PathPrefix(`{{ consensus_debug_port_path_prefix }}`)"
+{% if enable_tls %}
+      - "traefik.http.routers.external_node_consensus_debug.entrypoints=websecure"
+      - "traefik.http.routers.external_node_consensus_debug.tls.certresolver=myresolver"
+{% else %}
+      - "traefik.http.routers.external_node_consensus_debug.entrypoints=web"
+{% endif %}
+      - "traefik.http.routers.external_node_consensus_debug.service=external_node_consensus_debug"
+{% if enable_basic_auth %}
+      - "traefik.http.routers.external_node_consensus_debug.middlewares=external_node_auth"
+{% endif %}
+{% endif %}
     expose:
       - {{ rpc_http_port }}
       - {{ rpc_ws_port }}
       - {{ healthcheck_port }}
       - {{ metrics_port }}
+{% if enable_consensus %}
+      - {{ consensus_port }}
+{% if expose_consensus_debug_port %}
+      - {{ consensus_debug_port }}
+{% endif %}
+{% endif %}
     environment:
       ZKSYNC_HOME: "/"
       EN_STATE_CACHE_PATH: /db/state_keeper
@@ -86,7 +131,12 @@ services:
       CHAIN_STATE_KEEPER_VALIDATION_COMPUTATIONAL_GAS_LIMIT: 2000000
       DATABASE_POOL_SIZE: 200
       EN_MAX_BLOCKS_PER_TREE_BATCH: 200
-      RUST_LOG: zksync_core=debug,zksync_dal=info,zksync_eth_client=info,zksync_merkle_tree=info,zksync_storage=info,zksync_state=debug,zksync_types=info,vm=info,zksync_external_node=info,zksync_utils=debug
+      MISC_LOG_FORMAT: json
+      RUST_LOG: {{ rust_log }}
+{% if enable_consensus %}
+      EN_CONSENSUS_CONFIG_PATH: /etc/consensus_config.yaml
+      EN_CONSENSUS_SECRETS_PATH: /run/secrets/consensus_secrets
+{% endif %}
     healthcheck:
       test: [ "CMD", "curl", "-f", "http://localhost:{{ healthcheck_port }}/health" ]
       interval: 1m
@@ -95,6 +145,21 @@ services:
       start_period: 1m
     volumes:
       - "{{ storage_directory }}/db:/db"
+{% if enable_consensus %}
+      - "{{ configuration_directory }}/consensus_config.yaml:/etc/consensus_config.yaml"
+{% endif %}
     env_file:
       - "external_node.env"
       - "postgres.env"
+    command:
+{% if enable_consensus %}
+      - --enable-consensus
+    secrets:
+      - consensus_secrets
+{% endif %}
+
+{% if enable_consensus %}
+secrets:
+  consensus_secrets:
+    file: consensus_secrets.yaml
+{% endif %}

templates/external_node.env.j2

@@ -2,4 +2,16 @@ EN_ETH_CLIENT_URL="{{ eth_l1_url | mandatory }}"
 EN_MAIN_NODE_URL="{{ main_node_url | mandatory }}"
 EN_L2_CHAIN_ID="{{ l2_chain_id | mandatory }}"
 EN_L1_CHAIN_ID="{{ l1_chain_id | mandatory }}"
+{% if enable_snapshots_recovery %}
+EN_SNAPSHOTS_RECOVERY_ENABLED="true"
+EN_SNAPSHOTS_OBJECT_STORE_MODE="GCSAnonymousReadOnly"
+EN_SNAPSHOTS_OBJECT_STORE_BUCKET_BASE_URL="{{ snapshots_bucket_base_url | mandatory }}"
+{% endif %}
+
 DATABASE_URL="postgres://{{ database_username | mandatory }}:{{ database_password | mandatory }}@postgres/{{ database_name | mandatory }}"
+
+{% if additional_env_vars is defined and additional_env_vars|length > 0 %}
+{% for env_var in additional_env_vars %}
+{{ env_var.name }}="{{ env_var.value }}"
+{% endfor %}
+{% endif %}

templates/monitoring.yaml.j2

@@ -21,18 +21,6 @@ services:
       - "--remoteWrite.vmProtoCompressLevel=2"
     restart: always
 
-  node-exporter:
-    image: "prom/node-exporter:v{{ node_exporter_version }}"
-    volumes:
-      - /proc:/host/proc:ro
-      - /sys:/host/sys:ro
-      - /:/rootfs:ro
-    restart: unless-stopped
-    command:
-      - '--path.procfs=/host/proc'
-      - '--path.sysfs=/host/sys'
-      - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)'
-
   cadvisor:
     image: "gcr.io/cadvisor/cadvisor:v{{ cadvisor_version }}"
     volumes:

templates/restore_dump.sh.j2

@@ -3,7 +3,7 @@ set -e
 
 {% if force_pg_restore %}
 pg_restore --clean --exit-on-error -j $(nproc --all) -d postgres -U $POSTGRES_USER --no-owner --no-privileges --disable-triggers --create /pg_backups/external_node_latest.pgdump
-{% else %}
+{% elif not enable_snapshots_recovery %}
 if psql -U $POSTGRES_USER -d postgres -lqt | cut -d \| -f 1 | grep -qw "{{ database_name }}"; then
     echo "Database already exists"
 else

templates/vmagent-config.yml.j2

@@ -14,14 +14,6 @@ scrape_configs:
       - source_labels: [instance]
         target_label: instance
         replacement: '{{ node_name | mandatory }}'
-  - job_name: node-exporter
-    static_configs:
-      - targets:
-          - "node-exporter:9100"
-    relabel_configs:
-      - source_labels: [instance]
-        target_label: instance
-        replacement: '{{ node_name | mandatory }}'
   - job_name: cadvisor
     static_configs:
       - targets:
@@ -41,7 +33,8 @@ scrape_configs:
   - job_name: traefik
     static_configs:
       - targets:
-          - "traefik:8080"
+          # traefik uses network host, so docker DNS wouldn't work.
+          - "127.0.0.1:8080"
     relabel_configs:
       - source_labels: [instance]
         target_label: instance