diff --git a/docs/crates/db.md b/docs/crates/db.md index b08383b7a..2c2977b06 100644 --- a/docs/crates/db.md +++ b/docs/crates/db.md @@ -8,7 +8,7 @@ The database is a central component to Reth, enabling persistent storage for dat Within Reth, the database is organized via "tables". A table is any struct that implements the `Table` trait. -[File: crates/storage/db/src/abstraction/table.rs](https://github.com/paradigmxyz/reth/blob/1563506aea09049a85e5cc72c2894f3f7a371581/crates/storage/db/src/abstraction/table.rs#L55-L82) +[File: crates/storage/db-api/src/table.rs](https://github.com/paradigmxyz/reth/blob/bf9cac7571f018fec581fe3647862dab527aeafb/crates/storage/db-api/src/table.rs#L64-L93) ```rust ignore pub trait Table: Send + Sync + Debug + 'static { @@ -32,7 +32,7 @@ pub trait Value: Compress + Decompress + Serialize {} The `Table` trait has two generic values, `Key` and `Value`, which need to implement the `Key` and `Value` traits, respectively. The `Encode` trait is responsible for transforming data into bytes so it can be stored in the database, while the `Decode` trait transforms the bytes back into its original form. Similarly, the `Compress` and `Decompress` traits transform the data to and from a compressed format when storing or reading data from the database. -There are many tables within the node, all used to store different types of data from `Headers` to `Transactions` and more. Below is a list of all of the tables. You can follow [this link](https://github.com/paradigmxyz/reth/blob/1563506aea09049a85e5cc72c2894f3f7a371581/crates/storage/db/src/tables/mod.rs#L161-L188) if you would like to see the table definitions for any of the tables below. +There are many tables within the node, all used to store different types of data from `Headers` to `Transactions` and more. Below is a list of all of the tables. You can follow [this link](https://github.com/paradigmxyz/reth/blob/bf9cac7571f018fec581fe3647862dab527aeafb/crates/storage/db/src/tables/mod.rs#L274-L414) if you would like to see the table definitions for any of the tables below. - CanonicalHeaders - HeaderTerminalDifficulties @@ -41,18 +41,18 @@ There are many tables within the node, all used to store different types of data - BlockBodyIndices - BlockOmmers - BlockWithdrawals -- TransactionBlocks - Transactions - TransactionHashNumbers +- TransactionBlocks - Receipts +- Bytecodes - PlainAccountState - PlainStorageState -- Bytecodes - AccountsHistory - StoragesHistory - AccountChangeSets - StorageChangeSets -- HashedAccount +- HashedAccounts - HashedStorages - AccountsTrie - StoragesTrie @@ -60,28 +60,41 @@ There are many tables within the node, all used to store different types of data - StageCheckpoints - StageCheckpointProgresses - PruneCheckpoints +- VersionHistory +- BlockRequests +- ChainState
## Database -Reth's database design revolves around it's main [Database trait](https://github.com/paradigmxyz/reth/blob/eaca2a4a7fbbdc2f5cd15eab9a8a18ede1891bda/crates/storage/db/src/abstraction/database.rs#L21), which implements the database's functionality across many types. Let's take a quick look at the `Database` trait and how it works. +Reth's database design revolves around it's main [Database trait](https://github.com/paradigmxyz/reth/blob/bf9cac7571f018fec581fe3647862dab527aeafb/crates/storage/db-api/src/database.rs#L8-L52), which implements the database's functionality across many types. Let's take a quick look at the `Database` trait and how it works. -[File: crates/storage/db/src/abstraction/database.rs](https://github.com/paradigmxyz/reth/blob/eaca2a4a7fbbdc2f5cd15eab9a8a18ede1891bda/crates/storage/db/src/abstraction/database.rs#L21) +[File: crates/storage/db-api/src/database.rs](https://github.com/paradigmxyz/reth/blob/bf9cac7571f018fec581fe3647862dab527aeafb/crates/storage/db-api/src/database.rs#L8-L52) ```rust ignore -/// Main Database trait that spawns transactions to be executed. -pub trait Database { - /// RO database transaction - type TX: DbTx + Send + Sync + Debug; - /// RW database transaction - type TXMut: DbTxMut + DbTx + TableImporter + Send + Sync + Debug; +/// Main Database trait that can open read-only and read-write transactions. +/// +/// Sealed trait which cannot be implemented by 3rd parties, exposed only for consumption. +pub trait Database: Send + Sync { + /// Read-Only database transaction + type TX: DbTx + Send + Sync + Debug + 'static; + /// Read-Write database transaction + type TXMut: DbTxMut + DbTx + TableImporter + Send + Sync + Debug + 'static; + + /// Create read only transaction. + #[track_caller] + fn tx(&self) -> Result; + + /// Create read write transaction only possible if database is open with write access. + #[track_caller] + fn tx_mut(&self) -> Result; /// Takes a function and passes a read-only transaction into it, making sure it's closed in the /// end of the execution. - fn view(&self, f: F) -> Result + fn view(&self, f: F) -> Result where - F: Fn(&::TX) -> T, + F: FnOnce(&Self::TX) -> T, { let tx = self.tx()?; @@ -93,9 +106,9 @@ pub trait Database { /// Takes a function and passes a write-read transaction into it, making sure it's committed in /// the end of the execution. - fn update(&self, f: F) -> Result + fn update(&self, f: F) -> Result where - F: Fn(&::TXMut) -> T, + F: FnOnce(&Self::TXMut) -> T, { let tx = self.tx_mut()?; @@ -135,183 +148,183 @@ where The `Database` defines two associated types `TX` and `TXMut`. -[File: crates/storage/db/src/abstraction/database.rs](https://github.com/paradigmxyz/reth/blob/main/crates/storage/db/src/abstraction/database.rs#L11) +[File: crates/storage/db-api/src/database.rs](https://github.com/paradigmxyz/reth/blob/bf9cac7571f018fec581fe3647862dab527aeafb/crates/storage/db-api/src/database.rs#L54-L78) The `TX` type can be any type that implements the `DbTx` trait, which provides a set of functions to interact with read only transactions. -[File: crates/storage/db/src/abstraction/transaction.rs](https://github.com/paradigmxyz/reth/blob/main/crates/storage/db/src/abstraction/transaction.rs#L36) +[File: crates/storage/db-api/src/transaction.rs](https://github.com/paradigmxyz/reth/blob/bf9cac7571f018fec581fe3647862dab527aeafb/crates/storage/db-api/src/transaction.rs#L7-L29) ```rust ignore /// Read only transaction pub trait DbTx: Send + Sync { /// Cursor type for this read-only transaction type Cursor: DbCursorRO + Send + Sync; - /// DupCursor type for this read-only transaction + /// `DupCursor` type for this read-only transaction type DupCursor: DbDupCursorRO + DbCursorRO + Send + Sync; /// Get value - fn get(&self, key: T::Key) -> Result, Error>; + fn get(&self, key: T::Key) -> Result, DatabaseError>; /// Commit for read only transaction will consume and free transaction and allows /// freeing of memory pages - fn commit(self) -> Result; + fn commit(self) -> Result; + /// Aborts transaction + fn abort(self); /// Iterate over read only values in table. - fn cursor(&self) -> Result, Error>; + fn cursor_read(&self) -> Result, DatabaseError>; /// Iterate over read only values in dup sorted table. - fn cursor_dup(&self) -> Result, Error>; + fn cursor_dup_read(&self) -> Result, DatabaseError>; + /// Returns number of entries in the table. + fn entries(&self) -> Result; + /// Disables long-lived read transaction safety guarantees. + fn disable_long_read_transaction_safety(&mut self); } ``` The `TXMut` type can be any type that implements the `DbTxMut` trait, which provides a set of functions to interact with read/write transactions and the associated cursor types. -[File: crates/storage/db/src/abstraction/transaction.rs](https://github.com/paradigmxyz/reth/blob/main/crates/storage/db/src/abstraction/transaction.rs#L49) +[File: crates/storage/db-api/src/transaction.rs](https://github.com/paradigmxyz/reth/blob/bf9cac7571f018fec581fe3647862dab527aeafb/crates/storage/db-api/src/transaction.rs#L31-L54) ```rust ignore /// Read write transaction that allows writing to database pub trait DbTxMut: Send + Sync { /// Read-Write Cursor type type CursorMut: DbCursorRW + DbCursorRO + Send + Sync; - /// Read-Write DupCursor type + /// Read-Write `DupCursor` type type DupCursorMut: DbDupCursorRW + DbCursorRW + DbDupCursorRO + DbCursorRO + Send + Sync; + /// Put value to database - fn put(&self, key: T::Key, value: T::Value) -> Result<(), Error>; + fn put(&self, key: T::Key, value: T::Value) -> Result<(), DatabaseError>; /// Delete value from database - fn delete(&self, key: T::Key, value: Option) -> Result; + fn delete(&self, key: T::Key, value: Option) + -> Result; /// Clears database. - fn clear(&self) -> Result<(), Error>; - /// Cursor for writing - fn cursor_write(&self) -> Result, Error>; - /// DupCursor for writing - fn cursor_dup_write( - &self, - ) -> Result, Error>; + fn clear(&self) -> Result<(), DatabaseError>; + /// Cursor mut + fn cursor_write(&self) -> Result, DatabaseError>; + /// `DupCursor` mut. + fn cursor_dup_write(&self) -> Result, DatabaseError>; } ``` -Lets take a look at the `DbTx` and `DbTxMut` traits in action. Revisiting the `Transaction` struct as an example, the `Transaction::get_block_hash()` method uses the `DbTx::get()` function to get a block header hash in the form of `self.get::(number)`. +Let's take a look at the `DbTx` and `DbTxMut` traits in action. -[File: crates/storage/provider/src/transaction.rs](https://github.com/paradigmxyz/reth/blob/main/crates/storage/provider/src/transaction.rs#L106) +Revisiting the `DatabaseProvider` struct as an exampl, the `DatabaseProvider::header_by_number()` function uses the `DbTx::get()` function to get a header from the `Headers` table. + +[File: crates/storage/provider/src/providers/database/provider.rs](https://github.com/paradigmxyz/reth/blob/bf9cac7571f018fec581fe3647862dab527aeafb/crates/storage/provider/src/providers/database/provider.rs#L1319-L1336) ```rust ignore - -impl<'this, DB> Transaction<'this, DB> -where - DB: Database, -{ +impl HeaderProvider for DatabaseProvider { //--snip-- - /// Query [tables::CanonicalHeaders] table for block hash by block number - pub(crate) fn get_block_hash(&self, number: BlockNumber) -> Result { - let hash = self - .get::(number)? - .ok_or(ProviderError::CanonicalHash { number })?; - Ok(hash) + fn header_by_number(&self, num: BlockNumber) -> ProviderResult> { + self.static_file_provider.get_with_static_file_or_database( + StaticFileSegment::Headers, + num, + |static_file| static_file.header_by_number(num), + || Ok(self.tx.get::(num)?), + ) } - //--snip-- -} -//--snip-- -impl<'a, DB: Database> Deref for Transaction<'a, DB> { - type Target = ::TXMut; - fn deref(&self) -> &Self::Target { - self.tx.as_ref().expect("Tried getting a reference to a non-existent transaction") - } + //--snip-- } ``` -The `Transaction` struct implements the `Deref` trait, which returns a reference to its `tx` field, which is a `TxMut`. Recall that `TxMut` is a generic type on the `Database` trait, which is defined as `type TXMut: DbTxMut + DbTx + Send + Sync;`, giving it access to all of the functions available to `DbTx`, including the `DbTx::get()` function. - Notice that the function uses a [turbofish](https://techblog.tonsser.com/posts/what-is-rusts-turbofish) to define which table to use when passing in the `key` to the `DbTx::get()` function. Taking a quick look at the function definition, a generic `T` is defined that implements the `Table` trait mentioned at the beginning of this chapter. -[File: crates/storage/db/src/abstraction/transaction.rs](https://github.com/paradigmxyz/reth/blob/main/crates/storage/db/src/abstraction/transaction.rs#L38) +[File: crates/storage/db-api/src/transaction.rs](https://github.com/paradigmxyz/reth/blob/bf9cac7571f018fec581fe3647862dab527aeafb/crates/storage/db-api/src/transaction.rs#L15) ```rust ignore -fn get(&self, key: T::Key) -> Result, Error>; +fn get(&self, key: T::Key) -> Result, DatabaseError>; ``` This design pattern is very powerful and allows Reth to use the methods available to the `DbTx` and `DbTxMut` traits without having to define implementation blocks for each table within the database. -Lets take a look at a couple examples before moving on. In the snippet below, the `DbTxMut::put()` method is used to insert values into the `CanonicalHeaders`, `Headers` and `HeaderNumbers` tables. +Let's take a look at a couple examples before moving on. In the snippet below, the `DbTxMut::put()` method is used to insert values into the `CanonicalHeaders`, `Headers` and `HeaderNumbers` tables. -[File: crates/storage/provider/src/block.rs](https://github.com/paradigmxyz/reth/blob/main/crates/storage/provider/src/block.rs#L121-L125) +[File: crates/storage/provider/src/providers/database/provider.rs](https://github.com/paradigmxyz/reth/blob/bf9cac7571f018fec581fe3647862dab527aeafb/crates/storage/provider/src/providers/database/provider.rs#L2606-L2745) ```rust ignore - tx.put::(block.number, block.hash())?; - // Put header with canonical hashes. - tx.put::(block.number, block.header.as_ref().clone())?; - tx.put::(block.hash(), block.number)?; +self.tx.put::(block_number, block.hash())?; +self.tx.put::(block_number, block.header.as_ref().clone())?; +self.tx.put::(block.hash(), block_number)?; ``` +Let's take a look at the `DatabaseProviderRW` struct, which is used to create a mutable transaction to interact with the database. +The `DatabaseProviderRW` struct implements the `Deref` and `DerefMut` trait, which returns a reference to its first field, which is a `TxMut`. Recall that `TxMut` is a generic type on the `Database` trait, which is defined as `type TXMut: DbTxMut + DbTx + Send + Sync;`, giving it access to all of the functions available to `DbTx`, including the `DbTx::get()` function. + This next example uses the `DbTx::cursor()` method to get a `Cursor`. The `Cursor` type provides a way to traverse through rows in a database table, one row at a time. A cursor enables the program to perform an operation (updating, deleting, etc) on each row in the table individually. The following code snippet gets a cursor for a few different tables in the database. -[File: crates/stages/src/stages/execution.rs](https://github.com/paradigmxyz/reth/blob/main/crates/stages/src/stages/execution.rs#L93-L101) +[File: crates/static-file/static-file/src/segments/headers.rs](https://github.com/paradigmxyz/reth/blob/bf9cac7571f018fec581fe3647862dab527aeafb/crates/static-file/static-file/src/segments/headers.rs#L22-L58) ```rust ignore -// Get next canonical block hashes to execute. - let mut canonicals = db_tx.cursor_read::()?; - // Get header with canonical hashes. - let mut headers = db_tx.cursor_read::()?; - // Get bodies (to get tx index) with canonical hashes. - let mut cumulative_tx_count = db_tx.cursor_read::()?; - // Get transaction of the block that we are executing. - let mut tx = db_tx.cursor_read::()?; - // Skip sender recovery and load signer from database. - let mut tx_sender = db_tx.cursor_read::()?; - +# Get a cursor for the Headers table +let mut headers_cursor = provider.tx_ref().cursor_read::()?; +# Then we can walk the cursor to get the headers for a specific block range +let headers_walker = headers_cursor.walk_range(block_range.clone())?; ``` Lets look at an examples of how cursors are used. The code snippet below contains the `unwind` method from the `BodyStage` defined in the `stages` crate. This function is responsible for unwinding any changes to the database if there is an error when executing the body stage within the Reth pipeline. -[File: crates/stages/src/stages/bodies.rs](https://github.com/paradigmxyz/reth/blob/main/crates/stages/src/stages/bodies.rs#L205-L238) +[File: crates/stages/stages/src/stages/bodies.rs](https://github.com/paradigmxyz/reth/blob/bf9cac7571f018fec581fe3647862dab527aeafb/crates/stages/stages/src/stages/bodies.rs#L267-L345) ```rust ignore - /// Unwind the stage. - async fn unwind( - &mut self, - db: &mut Transaction<'_, DB>, - input: UnwindInput, - ) -> Result> { - let mut tx_count_cursor = db.cursor_write::()?; - let mut block_ommers_cursor = db.cursor_write::()?; - let mut transaction_cursor = db.cursor_write::()?; +/// Unwind the stage. +fn unwind(&mut self, provider: &DatabaseProviderRW, input: UnwindInput) { + self.buffer.take(); - let mut entry = tx_count_cursor.last()?; - while let Some((key, count)) = entry { - if key.number() <= input.unwind_to { - break - } + let static_file_provider = provider.static_file_provider(); + let tx = provider.tx_ref(); + // Cursors to unwind bodies, ommers + let mut body_cursor = tx.cursor_write::()?; + let mut ommers_cursor = tx.cursor_write::()?; + let mut withdrawals_cursor = tx.cursor_write::()?; + let mut requests_cursor = tx.cursor_write::()?; + // Cursors to unwind transitions + let mut tx_block_cursor = tx.cursor_write::()?; - tx_count_cursor.delete_current()?; - entry = tx_count_cursor.prev()?; - - if block_ommers_cursor.seek_exact(key)?.is_some() { - block_ommers_cursor.delete_current()?; - } - - let prev_count = entry.map(|(_, v)| v).unwrap_or_default(); - for tx_id in prev_count..count { - if transaction_cursor.seek_exact(tx_id)?.is_some() { - transaction_cursor.delete_current()?; - } - } + let mut rev_walker = body_cursor.walk_back(None)?; + while let Some((number, block_meta)) = rev_walker.next().transpose()? { + if number <= input.unwind_to { + break } - //--snip-- - } + // Delete the ommers entry if any + if ommers_cursor.seek_exact(number)?.is_some() { + ommers_cursor.delete_current()?; + } + // Delete the withdrawals entry if any + if withdrawals_cursor.seek_exact(number)?.is_some() { + withdrawals_cursor.delete_current()?; + } + + // Delete the requests entry if any + if requests_cursor.seek_exact(number)?.is_some() { + requests_cursor.delete_current()?; + } + + // Delete all transaction to block values. + if !block_meta.is_empty() && + tx_block_cursor.seek_exact(block_meta.last_tx_num())?.is_some() + { + tx_block_cursor.delete_current()?; + } + + // Delete the current body value + rev_walker.delete_current()?; + } + //--snip-- +} ``` -This function first grabs a mutable cursor for the `CumulativeTxCount`, `BlockOmmers` and `Transactions` tables. +This function first grabs a mutable cursor for the `BlockBodyIndices`, `BlockOmmers`, `BlockWithdrawals`, `BlockRequests`, `TransactionBlocks` tables. -The `tx_count_cursor` is used to get the last key value pair written to the `CumulativeTxCount` table and delete key value pair where the cursor is currently pointing. - -The `block_ommers_cursor` is used to get the block ommers from the `BlockOmmers` table at the specified key, and delete the entry where the cursor is currently pointing. - -Finally, the `transaction_cursor` is used to get delete each transaction from the last `TXNumber` written to the database, to the current tx count. +Then it gets a walker of the block body cursor, and then walk backwards through the cursor to delete the block body entries from the last block number to the block number specified in the `UnwindInput` struct. While this is a brief look at how cursors work in the context of database tables, the chapter on the `libmdbx` crate will go into further detail on how cursors communicate with the database and what is actually happening under the hood.