diff --git a/crates/engine/tree/src/backfill.rs b/crates/engine/tree/src/backfill.rs index aa075b6ad..5c4e56d29 100644 --- a/crates/engine/tree/src/backfill.rs +++ b/crates/engine/tree/src/backfill.rs @@ -2,7 +2,7 @@ //! //! - Backfill sync: Sync to a certain block height in stages, e.g. download data from p2p then //! execute that range. -//! - Live sync: In this mode the nodes is keeping up with the latest tip and listens for new +//! - Live sync: In this mode the node is keeping up with the latest tip and listens for new //! requests from the consensus client. //! //! These modes are mutually exclusive and the node can only be in one mode at a time. diff --git a/crates/engine/tree/src/lib.rs b/crates/engine/tree/src/lib.rs index 109cf7693..7a1e0b4a7 100644 --- a/crates/engine/tree/src/lib.rs +++ b/crates/engine/tree/src/lib.rs @@ -1,5 +1,78 @@ //! This crate includes the core components for advancing a reth chain. //! +//! ## Functionality +//! +//! The components in this crate are involved in: +//! * Handling and reacting to incoming consensus events ([`EngineHandler`](engine::EngineHandler)) +//! * Advancing the chain ([`ChainOrchestrator`](chain::ChainOrchestrator)) +//! * Keeping track of the chain structure in-memory ([`TreeState`](tree::TreeState)) +//! * Performing backfill sync and handling its progress ([`BackfillSync`](backfill::BackfillSync)) +//! * Downloading blocks ([`BlockDownloader`](download::BlockDownloader)), and +//! * Persisting blocks and performing pruning +//! ([`PersistenceService`](persistence::PersistenceService)) +//! +//! ## Design and motivation +//! +//! The node must keep up with the state of the chain and validate new updates to the chain state. +//! +//! In order to respond to consensus messages and advance the chain quickly, validation code must +//! avoid database write operations and perform as much work as possible in-memory. This requirement +//! is what informs the architecture of the components this crate. +//! +//! ## Chain synchronization +//! +//! When the node receives a block with an unknown parent, or cannot find a certain block hash, it +//! needs to download and validate the part of the chain that is missing. +//! +//! There are two mutually-exclusive ways the node can bring itself in sync with the chain: +//! * Backfill sync: downloading and validating large ranges of blocks in a structured manner, +//! performing different parts of the validation process in sequence. +//! * Live sync: By responding to new blocks from a connected consensus layer and downloading any +//! missing blocks on the fly. +//! +//! To determine which sync type to use, the node checks how many blocks it needs to execute to +//! catch up to the tip of the chain. If this range is large, backfill sync will be used. Otherwise, +//! life sync will be used. +//! +//! The backfill sync is driven by components which implement the +//! [`BackfillSync`](backfill::BackfillSync) trait, like [`PipelineSync`](backfill::PipelineSync). +//! +//! ## Handling consensus messages +//! +//! Consensus message handling is performed by three main components: +//! 1. The [`EngineHandler`](engine::EngineHandler), which takes incoming consensus mesesages and +//! manages any requested backfill or download work. +//! 2. The [`EngineApiRequestHandler`](engine::EngineApiRequestHandler), which processes messages +//! from the [`EngineHandler`](engine::EngineHandler) and delegates them to the +//! [`EngineApiTreeHandler`](tree::EngineApiTreeHandler). +//! 3. The [`EngineApiTreeHandler`](tree::EngineApiTreeHandler), which processes incoming tree +//! events, such as new payload events, sending back requests for any needed backfill or download +//! work. +//! +//! ## Chain representation +//! +//! The chain is represented by the [`TreeState`](tree::TreeState) data structure, which keeps +//! tracks of blocks by hash and number, as well as keeping track of parent-child relationships +//! between blocks. The hash and number of the current head of the canonical chain is also tracked +//! in the [`TreeState`](tree::TreeState). +//! +//! ## Persistence model +//! +//! Because the node minimizes database writes in the critical path for handling consensus messages, +//! it must perform in the background. +//! +//! Performing writes in the background has two advantages: +//! 1. As mentioned, writes are not in the critical path of request processing. +//! 2. Writes can be larger and done at a lower frequency, allowing for more efficient writes. +//! +//! This is achieved by spawning a separate thread which is sent different commands corresponding to +//! different types of writes, for example a command to write a list of transaction, or remove a +//! specific range of blocks. +//! +//! The persistence service must also respond to these commands, to ensure that any in-memory state +//! that is on-disk can be cleaned up, conserving memory and allowing us to add new blocks +//! indefinitely. +//! //! ## Feature Flags //! //! - `test-utils`: Export utilities for testing