feat(docs): MDBX freelist troubleshooting (#5358)

book/installation/build-for-arm-devices.md

@@ -62,12 +62,12 @@ A simple and safe approach to achieve this is to use the Armbian build framework
 
 To be able to build an Armbian image and set those values, you will need to:
 - Clone the Armbian build framework repository
-```shell
+```bash
 git clone https://github.com/armbian/build
 cd build
 ```
 - Run the compile script with the following parameters:
-```shell
+```bash
 ./compile.sh \
 BUILD_MINIMAL=yes \
 BUILD_DESKTOP=no \

book/run/troubleshooting.md

@@ -6,6 +6,64 @@ This page tries to answer how to deal with the most popular issues.
 
 ## Database
 
+### Slow database inserts and updates
+
+If you're:
+1. Running behind the tip
+2. Have slow canonical commit time according to the `Canonical Commit Latency time` chart on [Grafana dashboard](./observability.md#prometheus--grafana) (more than 2-3 seconds)
+3. Seeing warnings in your logs such as 
+   ```console
+   2023-11-08T15:17:24.789731Z  WARN providers::db: Transaction insertion took too long block_number=18528075 tx_num=2150227643 hash=0xb7de1d6620efbdd3aa8547c47a0ff09a7fd3e48ba3fd2c53ce94c6683ed66e7c elapsed=6.793759034s
+   ```
+
+then most likely you're experiencing issues with the [database freelist](https://github.com/paradigmxyz/reth/issues/5228).
+To confirm it, check if the values on the `Freelist` chart on [Grafana dashboard](./observability.md#prometheus--grafana)
+is greater than 10M.
+
+Currently, there are two main ways to fix this issue.
+
+
+#### Compact the database
+It will take around 5-6 hours and require **additional** disk space located on the same or different drive
+equal to the [freshly synced node](../installation/installation.md#hardware-requirements).
+
+1. Clone Reth
+   ```bash
+   git clone https://github.com/paradigmxyz/reth
+   cd reth
+   ```
+2. Build database debug tools
+   ```bash
+   make db-tools
+   ```
+3. Run compaction (this step will take 5-6 hours, depending on the I/O speed)
+   ```bash
+   ./db-tools/mdbx_copy -c $(reth db path) reth_compact.dat
+   ```
+4. Stop Reth
+5. Backup original database
+   ```bash
+   mv $(reth db path)/mdbx.dat reth_old.dat
+   ```
+6. Move compacted database in place of the original database
+   ```bash
+   mv reth_compact.dat $(reth db path)/mdbx.dat
+   ```
+7. Start Reth
+8. Confirm that the values on the `Freelist` chart is near zero and the values on the `Canonical Commit Latency time` chart
+is less than 1 second.
+9. Delete original database
+   ```bash
+   rm reth_old.dat
+   ```
+
+#### Re-sync from scratch
+It will take the same time as initial sync.
+
+1. Stop Reth
+2. Drop the database using [`reth db drop`](../cli/db.md#reth-db-drop)
+3. Start reth
+
 ### Database write error
 
 If you encounter an irrecoverable database-related errors, in most of the cases it's related to the RAM/NVMe/SSD you use. For example:
@@ -33,6 +91,6 @@ Caused by:
     git clone https://github.com/paradigmxyz/reth
     cd reth
     make db-tools
-    db-tools/mdbx_chk $(reth db path)/mdbx.dat | tee mdbx_chk.log
+    ./db-tools/mdbx_chk $(reth db path)/mdbx.dat | tee mdbx_chk.log
     ```
     If `mdbx_chk` has detected any errors, please [open an issue](https://github.com/paradigmxyz/reth/issues) and post the output from the `mdbx_chk.log` file.