Merge pull request #32 from matter-labs/fix-typo

fix: Fixed typo in validation step for replication

.ansible-lint

@@ -2,3 +2,6 @@ skip_list:
   - 'yaml'
   - 'risky-shell-pipe'
   - 'role-name'
+
+exclude_paths:
+  - example_playbooks

.github/ISSUE_TEMPLATE/bug_report.md

@@ -26,10 +26,11 @@ Describe what actually happened.
 
 #### 🖥️ Environment
 
-Any relevant environment details like: 
+Any relevant environment details like:
+
 * Ansible version
 * Operating system
-* External node version
+* External Node version
 
 #### 📋 Additional Context
 

CONTRIBUTING.md

@@ -5,7 +5,7 @@ sovereignty! We welcome contributions from anyone on the internet, and are grate
 
 ## Ways to contribute
 
-There are many ways to contribute to the external node role:
+There are many ways to contribute to the External Node role:
 
 1. Open issues: if you find a bug, have something you believe needs to be fixed, or have an idea for a feature, please
    open an issue.

README.md

@@ -1,12 +1,23 @@
 # ansible-en-role
-Ansible role for setup external node.
+
+Ansible role to deploy and configure zkSync Era External Node, including DB instance setup on the same machine, Traefik as reverse proxy, and Prometheus monitoring (PostgreSQL exporter, cAdvisor, Traefik, External Node native metrics, and VictoriaMetrics vmagent to scrape all of them).
+
+Make sure to configure Prometheus remote write endpoint to send metrics to centralized metrics storage.
+
+Role has been tested and used internally on bare metal Hetzner instances.
 
 ## Requirements
+
 This role has been tested on:
+
 * Ubuntu 22.04, Jammy Jellyfish; Ansible 2.13.9
 
 ## Usage
-This role contains variables which has to be set:
+
+For a very simple minimal working example, see example_playbooks directory
+
+Minimal required variables that have to be set:
+
 ```yaml
 database_name: ""
 database_username: ""
@@ -17,36 +28,48 @@ l1_chain_id: ""
 l2_chain_id: ""
 ```
 
-If you want to use monitoring, you can use next variables:
+Additional arbitrary environment variables can be passed to External Node container:
+
 ```yaml
-# Monitoring options section
-enable_monitoring: false
-node_name: ""
-prometheus_remote_write: false
-prometheus_remote_write_url: ""
-prometheus_remote_write_auth: false
-prometheus_remote_write_auth_username: ""
-prometheus_remote_write_auth_password: ""
-prometheus_remote_write_label: ""
+additional_env_vars:
+  - { name: "EN_ADDITIONAL_VAR1", value: "Value1" }
+  - { name: "EN_ADDITIONAL_VAR2", value: "Value2" }
+  - { name: "EN_ADDITIONAL_VAR3", value: "Value3" }
 ```
 
-This role also has option to secure your server and allow traffic only from specified ip in case if you want
-to use some load balancer in front of your node:
+Please refer to [External Node docs](https://github.com/matter-labs/zksync-era/tree/main/docs/guides/external-node/prepared_configs) to find values for different zkSync Era chains.
+
+If you want to use monitoring (which we highly recommend), you have to change these variables:
+
+```yaml
+# Monitoring options section
+enable_monitoring: true
+node_name: "some-unique-node-identifier"
+prometheus_remote_write: true
+prometheus_remote_write_url: "https://metrics.example.org"
+prometheus_remote_write_auth: true
+prometheus_remote_write_auth_username: "admin"
+prometheus_remote_write_auth_password: "password"
+prometheus_remote_write_common_label: "matterlabs"
+```
+
+This role also has the option to secure your server and allow traffic only from specified IP address in case if you want
+to use some load balancer in front of your node, while not having fancy cloud security groups at your disposal:
 
 ```yaml
 # Security options
-use_predefined_iptables: false
-disable_ssh_password_auth: false
+use_predefined_iptables: true
+disable_ssh_password_auth: true
 iptables_packages:
   - iptables
   - iptables-persistent
-# Variable can be used in case with accept external traffic only from one ip
-loadbalancer_ip: ""
+# Variable to be used to accept external traffic only from single specified IP
+loadbalancer_ip: "1.2.3.4"
 ```
 
-In some cases, you may need to change postgres parameters, so you can do it using `postgres_arguments` variable:
-```yaml
+In most cases, you'd want to change PostgreSQL parameters, so you can do it using `postgres_arguments` variable, eg:
 
+```yaml
 postgres_arguments:
   - log_error_verbosity=terse
   - -c
@@ -54,58 +77,63 @@ postgres_arguments:
   - -c
   - shared_buffers=47616MB
   - -c
-  - effective_cache_size=142848MB
-  - -c
-  - maintenance_work_mem=2GB
-  - -c
-  - checkpoint_completion_target=0.9
-  - -c
-  - wal_buffers=16MB
-  - -c
-  - default_statistics_target=500
-  - -c
-  - random_page_cost=1.1
-  - -c
-  - effective_io_concurrency=200
-  - -c
-  - work_mem=2573kB
-  - -c
-  - huge_pages=try
-  - -c
-  - min_wal_size=4GB
-  - -c
-  - max_wal_size=16GB
-  - -c
-  - max_worker_processes=74
-  - -c
-  - max_parallel_workers_per_gather=37
-  - -c
-  - max_parallel_workers=74
-  - -c
-  - max_parallel_maintenance_workers=4
-  - -c
-  - checkpoint_timeout=1800
 ```
-We recommend to use [pgtune](https://github.com/le0pard/pgtune) to choose optimal config for your hardware. 
+
+We recommend using pgtune [online](https://pgtune.leopard.in.ua/) or [self-hosted](https://github.com/le0pard/pgtune) version with "Online transaction processing system" preset as a good starting point for generating optimal config for your hardware.
+
+If you want to use basic auth for inbound requests, you have to change next variables:
+
+```yaml
+enable_basic_auth: true
+basic_auth_secret: "htpasswd-generated-secret"
+```
+
+Basic auth secret can be generated by `htpasswd` and `sed` for interpolation:
+```echo $(htpasswd -nb <username> <password>) | sed -e s/\\$/\\$\\$/g```
 
 ## Step-by-step guide
 
-1. Install ansible collection on your machine from where you will run ansible:
+1. Install the ansible collection on your machine from where you will run ansible:
 `ansible-galaxy collection install community.general`
-2. Prepare latest database backup on your host. you can download it from our [public GCS bucket](https://storage.googleapis.com/zksync-era-mainnet-external-node-backups/external_node_latest.pgdump).
-you should place it to `{{ storage_directory }}/pg_backups` directory. By default, `{{ storage_directory }}` is `/usr/src/en` 
-3. **OPTIONAL**: If you already have external-node, you can copy tree directory to new host. Copy external-node database tree to `{{ storage_directory }}/db`. 
-**Keep in mind, tree should be older than postgres database backup.**
-4. Run ansible-playbook using this role. We recommend to encrypt next variables with ansible-vault or some another way:
-```
+
+2. Prepare the latest database backup on your host. you can download it from our public GCS buckets:
+Skip this step if you are recovering from a snapshot!
+
+* [Era Mainnet latest dump](https://storage.googleapis.com/zksync-era-mainnet-external-node-backups/external_node_latest.pgdump)
+* [Era Sepolia Testnet latest dump](https://storage.googleapis.com/zksync-era-testnet-sepolia-external-node-backups/external_node_latest.pgdump)
+
+Downloaded dump file should be placed into `{{ storage_directory }}/pg_backups` directory (`/usr/src/en/pg_backups` by default)
+
+3. **OPTIONAL**: If you already have running node, you can copy its tree and state directory to a new host's `{{ storage_directory }}/db`. (`/usr/src/en/db` by default)
+Skip this step if you are recovering from a snapshot!
+
+**Keep in mind that tree and state should be older than PostgreSQL database backup.**
+
+4. Run ansible-playbook using this role. We recommend encrypting next variables with ansible-vault or some another way:
+
+```yaml
 database_username
 database_password
 eth_l1_url
 vm_auth_username
 vm_auth_password
 ```
-5. Connect to your host, and see status of postgres container. It can take a lot of time before postgres database backup will be restored 
-and postgres server will be ready for use. After postgres goes healty status, external-node runs automatically.
+
+5. Connect to your host, and see status of `postgres` container. It can take a lot of time before PostgreSQL database backup will be restored (hours to days, depending on your disk throughput and IOPS), after which PostgreSQL server will be ready for use. Once `postgres` becomes "healthy", `external_node` runs automatically.
+
+## Snapshots Recovery
+
+Example config enabling recovery from a snapshot:
+
+```yaml
+- enable_snapshots_recovery: true
+- snapshots_bucket_base_url: "snapshots-bucket-name"
+```
+
+Snapshot buckets:
+
+* Era Mainnet: `zksync-era-mainnet-external-node-snapshots`
+* Era Sepolia Testnet: `zksync-era-boojnet-external-node-snapshots`
 
 ## Example Playbook
 
@@ -122,7 +150,6 @@ and postgres server will be ready for use. After postgres goes healty status, ex
     l2_chain_id: "324"
     l1_chain_id: "1"
     enable_tls: false
-    partner_id: matterlabs
   vars_files:
     - secrets/mainnet_secrets.yml
   roles:
@@ -131,9 +158,9 @@ and postgres server will be ready for use. After postgres goes healty status, ex
 
 ## License
 
-Ansible role for external node is distributed under the terms of either
+Ansible role for zkSync Era External Node is distributed under the terms of either
 
-- Apache License, Version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or <http://www.apache.org/licenses/LICENSE-2.0>)
-- MIT license ([LICENSE-MIT](LICENSE-MIT) or <https://opensource.org/blog/license/mit/>)
+* Apache License, Version 2.0, ([LICENSE-APACHE](LICENSE-APACHE) or <http://www.apache.org/licenses/LICENSE-2.0>)
+* MIT license ([LICENSE-MIT](LICENSE-MIT) or <https://opensource.org/blog/license/mit/>)
 
 at your option.

defaults/main.yml

@@ -8,12 +8,12 @@ docker_install_compose: true
 docker_version: "25.0.3"
 docker_compose_version: "v2.23.0"
 
-# Versions of external node and 3rd party components
+# Versions of External Node and 3rd party components
 traefik_version: 2.11
 postgres_version: 14
-external_node_version: 21.0.2
-vmagent_version: 1.95.1
-node_exporter_version: 1.7.0
+external_node_version: 24.26.0
+external_node_raw_docker_tag: ""
+vmagent_version: 1.100.1
 cadvisor_version: 0.47.2
 postgres_exporter_version: 0.15.0
 
@@ -56,17 +56,35 @@ postgres_arguments:
   - max_parallel_maintenance_workers=4
   - -c
   - checkpoint_timeout=1800
-
+enable_postgres_replication: false
+# IP address of the interface replication
+postgres_replications_arguments: []
+postgres_replication_bind_address: ""
+postgres_replica_address: ""
 
 # Enable TLS for traefik
 enable_tls: false
 acme_email: ""
 domain_name: ""
 
+# Enable basic auth for External Node
+enable_basic_auth: false
+basic_auth_secret: ""
+
 # Force restore pg database
 force_pg_restore: false
 
-# External node and database options
+# Use a snapshot to recover
+enable_snapshots_recovery: false
+snapshots_bucket_base_url: ""
+
+# https://github.com/matter-labs/zksync-era/blob/main/docs/guides/external-node/09_decentralization.md
+enable_consensus: false
+consensus_secrets_file: ""
+consensus_port: 3054
+consensus_outbound: []
+
+# External Node and database options
 database_name: ""
 database_username: ""
 database_password: ""
@@ -78,6 +96,10 @@ rpc_http_port: 3060
 rpc_ws_port: 3061
 healthcheck_port: 3081
 metrics_port: 3082
+rust_log: zksync_external_node=info,zksync_core=info,zksync_core::sync_layer=info,zksync_server=info,zksync_prover=info,zksync_contract_verifier=info,zksync_dal=info,zksync_eth_client=info,zksync_storage=info,zksync_db_manager=info,zksync_merkle_tree=info,zksync_state=info,zksync_utils=info,zksync_types=info,loadnext=info,dev_ticker=info,vm=info,block_sizes_test=info,zksync_verification_key_generator_and_server=info,zksync_object_store=info,setup_key_generator_and_server=info,zksync_circuit_synthesizer=info,zksync_queued_job_processor=info,zksync_health_check=info
+
+# Additional env vars passed to External Node
+additional_env_vars: []
 
 # Monitoring options section
 enable_monitoring: false
@@ -87,7 +109,7 @@ prometheus_remote_write_url: ""
 prometheus_remote_write_auth: false
 prometheus_remote_write_auth_username: ""
 prometheus_remote_write_auth_password: ""
-prometheus_remote_write_label: ""
+prometheus_remote_write_common_label: ""
 
 # Security options
 use_predefined_iptables: false

example_playbooks/mainnet_with_snapshots_recovery/README.md

@@ -0,0 +1,23 @@
+# Mainnet Snapshots Recovery playbook
+
+This directory is simple example how to set up EN using this role. It comes with snapshots recovery enabled by default.\
+**Note that for simplicity it's using postgres database
+with a very unsecure password and the EN is just started on the same machine**
+
+To run this playbook, first install dependencies
+
+```shell
+ ansible-galaxy install -r requirements.yml
+ ``` 
+
+and then you can run the playbook using
+
+```shell
+ ansible-playbook playbook.yml -i hosts.ini -K
+ ``` 
+
+To see logs you can use
+
+```shell
+ docker logs en-external_node-1 
+ ``` 

example_playbooks/mainnet_with_snapshots_recovery/hosts.ini

@@ -0,0 +1,2 @@
+[local]
+localhost ansible_connection=local

example_playbooks/mainnet_with_snapshots_recovery/playbook.yml

@@ -0,0 +1,16 @@
+---
+- hosts: all
+  become: true
+  vars:
+    database_name: "zksync_ext_node_mainnet"
+    database_username: "postgres"
+    database_password: "notsecurepassword"
+    eth_l1_url: "https://ethereum-rpc.publicnode.com"
+    main_node_url: "https://zksync2-mainnet.zksync.io"
+    l1_chain_id: "1"
+    l2_chain_id: "324"
+    enable_snapshots_recovery: true
+    snapshots_bucket_base_url: "zksync-era-mainnet-external-node-snapshots"
+
+  roles:
+    - external_node

example_playbooks/mainnet_with_snapshots_recovery/requirements.yml

@@ -0,0 +1,12 @@
+---
+roles:
+  - name: geerlingguy.docker
+    src: https://github.com/geerlingguy/ansible-role-docker
+    version: "7.1.0"
+  - name: external_node
+    src: https://github.com/matter-labs/ansible-en-role
+    version: "v3.3.0"
+
+collections:
+  - name: community.general
+    version: 8.4.0

meta/main.yml

@@ -2,12 +2,12 @@
 dependencies:
   - src: geerlingguy.docker
     version: "7.1.0"
-    when: docker_install
+    when: docker_install_compose
 
 galaxy_info:
   role_name: external_node
   author: matter-labs
-  description: External node setup
+  description: External Node setup
   license: "license (MIT, APACHE)"
   min_ansible_version: "2.13.9"
   platforms:

tasks/firewall.yml

@@ -49,6 +49,23 @@
     source: "{{ loadbalancer_ip | mandatory }}"
     jump: ACCEPT
 
+- name: Allow consensus port traffic from any IP
+  when: enable_consensus
+  ansible.builtin.iptables:
+    chain: INPUT
+    protocol: tcp
+    destination_port: "{{ consensus_port }}"
+    jump: ACCEPT
+
+- name: Allow postgres replication traffic from replica only
+  when: enable_postgres_replication
+  ansible.builtin.iptables:
+    chain: INPUT
+    protocol: tcp
+    destination_port: 5432
+    source: "{{ postgres_replica_address }}"
+    jump: ACCEPT
+
 - name: Set default policy to DROP
   ansible.builtin.iptables:
     chain: INPUT

tasks/provision.yml

@@ -32,13 +32,28 @@
     - l2_chain_id
     - l1_chain_id
 
+- name: "Verify that required variables for replication is set"
+  when: enable_postgres_replication
+  ansible.builtin.assert:
+    that:
+      - postgress_replication_required_var != ""
+    fail_msg: "{{ postgress_replication_required_var }} needs to be set for the role for postgres replication to work"
+    success_msg: "Required variable for postgres replication {{ postgress_replication_required_var }} isn't empty"
+  loop_control:
+    loop_var: postgress_replication_required_var
+  with_items:
+    - enable_postgres_replication
+    - postgres_replication_bind_address
+    - postgres_replica_address
+    - postgres_replications_arguments
+
 - name: Check required en vars empty
   ansible.builtin.fail:
     msg: "Variable '{{ item }}' is empty"
   when: vars[item] == ""
   with_items: "{{ en_required_variables }}"
 
-- name: Copy main configs
+- name: Create main configs
   ansible.builtin.template:
     src: '{{ item.src }}'
     dest: '{{ item.dest }}'
@@ -51,7 +66,7 @@
     - src: "templates/postgres.env.j2"
       dest: "{{ configuration_directory }}/postgres.env"
 
-- name: Copy restore script
+- name: Create restore script
   register: restore_dump_script
   ansible.builtin.template:
     src: 'templates/restore_dump.sh.j2'
@@ -64,7 +79,7 @@
   when: enable_monitoring and ( vars[item] == "" )
   with_items: "{{ monitoring_required_variables }}"
 
-- name: Copy monitoring configs
+- name: Create monitoring configs
   when: enable_monitoring
   ansible.builtin.template:
     src: '{{ item.src }}'
@@ -76,23 +91,38 @@
     - src: "templates/vmagent-config.yml.j2"
       dest: "{{ configuration_directory }}/vmagent-config.yml"
 
+- name: Create consensus config
+  when: enable_consensus
+  ansible.builtin.template:
+    src: "templates/consensus_config.yaml.j2"
+    dest: "{{ configuration_directory }}/consensus_config.yaml"
+    mode: '0644'
+
+- name: Decrypt consensus_secrets
+  when: enable_consensus
+  ansible.builtin.copy:
+    src: "{{ consensus_secrets_file }}"
+    dest: "{{ configuration_directory }}/consensus_secrets.yaml"
+    decrypt: true
+    mode: '0600'
+
 - name: Run docker-compose without monitoring
   when: not enable_monitoring
   ansible.builtin.shell:
-    cmd: nohup docker compose -f docker-compose.yaml up -d &
+    cmd: nohup docker compose -f docker-compose.yaml up -d </dev/null >/dev/null 2>&1 &
     chdir: "{{ configuration_directory }}"
   changed_when: false
 
 - name: Run docker-compose with monitoring
   when: enable_monitoring and (not restore_dump_script.changed)
   ansible.builtin.shell:
-    cmd: nohup docker compose -f monitoring.yaml -f docker-compose.yaml up -d &
+    cmd: nohup docker compose -f monitoring.yaml -f docker-compose.yaml up -d </dev/null >/dev/null 2>&1 &
     chdir: "{{ configuration_directory }}"
   changed_when: false
 
 - name: Run docker-compose with monitoring with recreation
   when: enable_monitoring and restore_dump_script.changed
   ansible.builtin.shell:
-    cmd: nohup docker compose -f monitoring.yaml -f docker-compose.yaml up -d --force-recreate &
+    cmd: nohup docker compose -f monitoring.yaml -f docker-compose.yaml up -d --force-recreate </dev/null >/dev/null 2>&1 &
     chdir: "{{ configuration_directory }}"
   changed_when: false

templates/consensus_config.yaml.j2

@@ -0,0 +1,9 @@
+server_addr: '0.0.0.0:3054'
+public_addr: '{{ ansible_default_ipv4.address }}:{{ consensus_port }}'
+max_payload_size: 5000000
+gossip_dynamic_inbound_limit: 100
+gossip_static_outbound:
+{% for item in consensus_outbound %}
+  - key: {{ item.key }}
+    addr: {{ item.addr }}
+{% endfor %}

templates/docker-compose.yaml.j2

@@ -8,14 +8,20 @@ services:
       - "--log.level=INFO"
       - "--providers.docker=true"
       - "--providers.docker.exposedbydefault=false"
-      - "--entrypoints.web.address=:80"
-      - "--entrypoints.external_node_health.address=:3080"
+      - "--entryPoints.web.address=:80"
+      - "--entryPoints.external_node_health.address=:3080"
+{% if enable_consensus %}
+      - "--entryPoints.external_node_consensus.address=:{{ consensus_port }}"
+{% endif %}
 {% if enable_tls %}
-      - "--entrypoints.websecure.address=:443"
+      - "--entryPoints.websecure.address=:443"
       - "--certificatesresolvers.en_resolver.acme.tlschallenge=true"
       - "--certificatesresolvers.en_resolver.acme.storage=/letsencrypt/acme.json"
       - "--certificatesresolvers.myresolver.acme.email={{ acme_email }}"
 {% endif %}
+      - "--metrics.prometheus=true"
+      - "--metrics.prometheus.entryPoint=metrics"
+      - "--entryPoints.metrics.address=:8080"
     volumes:
       - "/var/run/docker.sock:/var/run/docker.sock"
 {% if enable_tls %}
@@ -37,14 +43,29 @@ services:
       - ./restore_dump.sh:/docker-entrypoint-initdb.d/restore_dump.sh
     env_file:
       - postgres.env
+{% if enable_postgres_replication %}
+    environment:
+      POSTGRES_HOST_AUTH_METHOD: "host replication replicator {{ postgres_replica_address }}/32 md5"
+    ports:
+      - "{{ postgres_replication_interface }}:5432:5432"
+{% endif %}
     command:
       - postgres
       - -c
 {% for argument in postgres_arguments %}
       - {{ argument }}
 {% endfor %}
+{% if enable_postgres_replication %}
+{% for repl_argument in postgres_replications_arguments %}
+      - {{ repl_argument }}
+{% endfor %}
+{% endif %}
   external_node:
+{% if not external_node_raw_docker_tag %}
     image: "matterlabs/external-node:v{{ external_node_version }}"
+{% else %}
+    image: "matterlabs/external-node:{{ external_node_raw_docker_tag }}"
+{% endif %}
     restart: unless-stopped
     depends_on:
       postgres:
@@ -66,11 +87,24 @@ services:
       - "traefik.http.routers.external_node_health.rule=PathPrefix(`/`)"
       - "traefik.http.routers.external_node_health.entrypoints=external_node_health"
       - "traefik.http.routers.external_node_health.service=external_node_health"
+{% if enable_basic_auth %}
+      - "traefik.http.routers.external_node_main.middlewares=external_node_auth"
+      - "traefik.http.middlewares.external_node_auth.basicauth.users={{ basic_auth_secret }}"
+{% endif %}
+{% if enable_consensus %}
+      - "traefik.tcp.services.external_node_consensus.loadbalancer.server.port={{ consensus_port }}"
+      - "traefik.tcp.routers.external_node_consensus.rule=HostSNI(`*`)"
+      - "traefik.tcp.routers.external_node_consensus.entrypoints=external_node_consensus"
+      - "traefik.tcp.routers.external_node_consensus.service=external_node_consensus"
+{% endif %}
     expose:
       - {{ rpc_http_port }}
       - {{ rpc_ws_port }}
       - {{ healthcheck_port }}
       - {{ metrics_port }}
+{% if enable_consensus %}
+      - {{ consensus_port }}
+{% endif %}
     environment:
       ZKSYNC_HOME: "/"
       EN_STATE_CACHE_PATH: /db/state_keeper
@@ -83,7 +117,12 @@ services:
       CHAIN_STATE_KEEPER_VALIDATION_COMPUTATIONAL_GAS_LIMIT: 2000000
       DATABASE_POOL_SIZE: 200
       EN_MAX_BLOCKS_PER_TREE_BATCH: 200
-      RUST_LOG: zksync_core=debug,zksync_dal=info,zksync_eth_client=info,zksync_merkle_tree=info,zksync_storage=info,zksync_state=debug,zksync_types=info,vm=info,zksync_external_node=info,zksync_utils=debug
+      MISC_LOG_FORMAT: json
+      RUST_LOG: {{ rust_log }}
+{% if enable_consensus %}
+      EN_CONSENSUS_CONFIG_PATH: /etc/consensus_config.yaml
+      EN_CONSENSUS_SECRETS_PATH: /run/secrets/consensus_secrets
+{% endif %}
     healthcheck:
       test: [ "CMD", "curl", "-f", "http://localhost:{{ healthcheck_port }}/health" ]
       interval: 1m
@@ -92,6 +131,21 @@ services:
       start_period: 1m
     volumes:
       - "{{ storage_directory }}/db:/db"
+{% if enable_consensus %}
+      - "{{ configuration_directory }}/consensus_config.yaml:/etc/consensus_config.yaml"
+{% endif %}
     env_file:
       - "external_node.env"
       - "postgres.env"
+    command:
+{% if enable_consensus %}
+      - --enable-consensus
+    secrets:
+      - consensus_secrets
+{% endif %}
+
+{% if enable_consensus %}
+secrets:
+  consensus_secrets:
+    file: consensus_secrets.yaml
+{% endif %}

templates/external_node.env.j2

@@ -2,4 +2,16 @@ EN_ETH_CLIENT_URL="{{ eth_l1_url | mandatory }}"
 EN_MAIN_NODE_URL="{{ main_node_url | mandatory }}"
 EN_L2_CHAIN_ID="{{ l2_chain_id | mandatory }}"
 EN_L1_CHAIN_ID="{{ l1_chain_id | mandatory }}"
+{% if enable_snapshots_recovery %}
+EN_SNAPSHOTS_RECOVERY_ENABLED="true"
+EN_SNAPSHOTS_OBJECT_STORE_MODE="GCSAnonymousReadOnly"
+EN_SNAPSHOTS_OBJECT_STORE_BUCKET_BASE_URL="{{ snapshots_bucket_base_url | mandatory }}"
+{% endif %}
+
 DATABASE_URL="postgres://{{ database_username | mandatory }}:{{ database_password | mandatory }}@postgres/{{ database_name | mandatory }}"
+
+{% if additional_env_vars is defined and additional_env_vars|length > 0 %}
+{% for env_var in additional_env_vars %}
+{{ env_var.name }}="{{ env_var.value }}"
+{% endfor %}
+{% endif %}

templates/monitoring.yaml.j2

@@ -9,7 +9,7 @@ services:
     command:
 {% if prometheus_remote_write %}
       - "--remoteWrite.url={{ prometheus_remote_write_url }}"
-      - "--remoteWrite.label={{ prometheus_remote_write_label }}"
+      - "--remoteWrite.label={{ prometheus_remote_write_common_label }}"
 {% if prometheus_remote_write_auth %}
       - "--remoteWrite.basicAuth.username={{ prometheus_remote_write_auth_username }}"
       - "--remoteWrite.basicAuth.password={{ prometheus_remote_write_auth_password }}"
@@ -21,18 +21,6 @@ services:
       - "--remoteWrite.vmProtoCompressLevel=2"
     restart: always
 
-  node-exporter:
-    image: "prom/node-exporter:v{{ node_exporter_version }}"
-    volumes:
-      - /proc:/host/proc:ro
-      - /sys:/host/sys:ro
-      - /:/rootfs:ro
-    restart: unless-stopped
-    command:
-      - '--path.procfs=/host/proc'
-      - '--path.sysfs=/host/sys'
-      - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)'
-
   cadvisor:
     image: "gcr.io/cadvisor/cadvisor:v{{ cadvisor_version }}"
     volumes:

templates/restore_dump.sh.j2

@@ -3,7 +3,7 @@ set -e
 
 {% if force_pg_restore %}
 pg_restore --clean --exit-on-error -j $(nproc --all) -d postgres -U $POSTGRES_USER --no-owner --no-privileges --disable-triggers --create /pg_backups/external_node_latest.pgdump
-{% else %}
+{% elif not enable_snapshots_recovery %}
 if psql -U $POSTGRES_USER -d postgres -lqt | cut -d \| -f 1 | grep -qw "{{ database_name }}"; then
     echo "Database already exists"
 else

templates/vmagent-config.yml.j2

@@ -14,14 +14,6 @@ scrape_configs:
       - source_labels: [instance]
         target_label: instance
         replacement: '{{ node_name | mandatory }}'
-  - job_name: node-exporter
-    static_configs:
-      - targets:
-          - "node-exporter:9100"
-    relabel_configs:
-      - source_labels: [instance]
-        target_label: instance
-        replacement: '{{ node_name | mandatory }}'
   - job_name: cadvisor
     static_configs:
       - targets:
@@ -38,3 +30,12 @@ scrape_configs:
       - source_labels: [instance]
         target_label: instance
         replacement: '{{ node_name | mandatory }}'
+  - job_name: traefik
+    static_configs:
+      - targets:
+          # traefik uses network host, so docker DNS wouldn't work.
+          - "127.0.0.1:8080"
+    relabel_configs:
+      - source_labels: [instance]
+        target_label: instance
+        replacement: '{{ node_name | mandatory }}'