Commit Graph

81 Commits

Author SHA1 Message Date
6c91322251 feat(eth-wire): derive serde traits for eth-wire types (#348)
* wip: start adding serde to stuff

* add serde traits to NewBlockHashes

* derive serde traits for the rest of the types

* derive serde traits for p2p subprotocol messages

* cargo fmt
2022-12-06 18:05:22 -08:00
4643f840d7 chore(deps): get rid of unused deps (#339) 2022-12-06 18:02:04 -08:00
5e37ef8226 style(net): use pin_project on eceis stream (#338) 2022-12-06 18:02:04 -08:00
a68cb3f357 feat(stage): Add execution stage (#287)
* Execution stage

* StorageProviderRef. cleanup, fmt/clippy

* block inclusion

* insert blocks for execution stage

* Sanity test for execution stage

* nits

* Update crates/stages/src/stages/execution.rs

Co-authored-by: Georgios Konstantopoulos <me@gakonst.com>

* Update crates/stages/Cargo.toml

Co-authored-by: Georgios Konstantopoulos <me@gakonst.com>

* Update crates/stages/src/stages/execution.rs

Co-authored-by: Georgios Konstantopoulos <me@gakonst.com>

* Update crates/stages/src/stages/execution.rs

Co-authored-by: Georgios Konstantopoulos <me@gakonst.com>

* feat(execution) Execution change diff. (#323)

* Insert to execution changediffs

* progress on exec stage unwind

* wip

* Remove tables:BlockBody from execution stage

* unwind passing

* Nits cleanup and refactoring

* DatabaseIntegrity errors

* KECCAK_EMPTY acc to None

* Update crates/interfaces/src/db/tables.rs

Co-authored-by: Georgios Konstantopoulos <me@gakonst.com>

* Update crates/interfaces/src/provider/block.rs

Co-authored-by: Georgios Konstantopoulos <me@gakonst.com>

* Update crates/stages/src/stages/execution.rs

Co-authored-by: Georgios Konstantopoulos <me@gakonst.com>

* nits

Co-authored-by: Georgios Konstantopoulos <me@gakonst.com>

Co-authored-by: Georgios Konstantopoulos <me@gakonst.com>
2022-12-06 18:02:04 -08:00
dda8df7341 refactor: headers client (#249)
* refactor: headers client

* chore: rustfmt

* chore(clippy): make clippy happy

* address comments

* use Error instead

* fix(sync): headers test client & stage tests (#255)

* headers test client & stage tests

* fix timeout test

* fix import

Co-authored-by: Roman Krasiuk <rokrassyuk@gmail.com>
2022-11-25 14:00:06 +01:00
2e3c220104 fix: fix flaky test (#250) 2022-11-24 13:26:22 -08:00
d6a11c7f48 chore(deps): bump ethers + ethereum types + revm (#228)
* chore(deps): bump ethers + ethereum types

* bump revm

* chore(clippy): make clippy happy
2022-11-23 11:54:54 +01:00
82b37b9bfb feat(sync): sender recovery stage (#181)
* feat(sync): sender recovery stage

* execute tests

* more tests & cleanup

* senders cont

* clean up & comments

* clippy

* cleanup

* comments

* put back arbitrary dep
2022-11-23 00:22:17 -08:00
027fc2bbf2 feat(db): derive Compact codec (#177) 2022-11-22 20:46:45 -08:00
7e693046c6 feat(executor): Execute block of transactions and return tx patches (#238)
* wip executor commit

* finish commit, make execution as functions

* rlp block encoding, fixing proof

* Block execution and test example
2022-11-22 22:57:22 +01:00
a523cb7024 feat: finish concurrent body downloader (#220)
* refactor: remove timeout config from downloader

The timeout should be controlled by the client
implementation.

* feat: downloader request retries

* test: add concurrent body downloader tests

* chore: fmt

Co-authored-by: Georgios Konstantopoulos <me@gakonst.com>
2022-11-22 21:06:49 +01:00
89ffaf541b feat: add tracing crate (#236) 2022-11-22 20:48:32 +01:00
09c5c3449e feat(executor): transaction and receipts verification (#195)
* transaction verification

* Add providers to interfaces

* add receipt check in executor verify

* Removed receipts, added bloom

* post merge header checks

* fmt clippy

* gas price type removal

* rm leftover

* move consts to contstant.rs
2022-11-16 11:10:36 -08:00
f8fddcdfa4 feat(p2p): integrate txpool in p2p (#208) 2022-11-15 22:27:41 +01:00
b60ced1de1 feat: add missing message handling (#200)
* feat: add missing message handling

* refactor: new block message handling

* feat: add events and commands for transaction handling

* more work in transactions

* chore: silence warnings
2022-11-15 10:34:28 +01:00
92a7818512 feat: bodies stage (#190)
* chore: clean up `.gitignore`

* fix: make RO cursors `Send + Sync`

* feat(wip): bodies stage

* driveby: improve docs

* chore: don't panic if we're the first stage

* chore: use `Vec` for ommers

* feat: error handling in bodies downloader

* chore: remove stale comment

* chore: pascal-case stage id

* refactor: remove unused new fns

* refactor: distinguish downloaders with prefix

* refactor: move downloader errs to own module

* refactor: `stream_bodies` -> `bodies_stream`

* test: fix borked imports in header stage

* test: clean up header tests

* test: add basic body stage tests

* test: add 2 more body stage test skeletons

* test: move generator test utils to own module

* refactor: move proof functions to primitives crate

* feat: add block generator test utils

* test: more body stage tests

* chore: fix typo (`Cannonical*` -> `Canonical`)

* docs: document `bodies_to_download`

* test: more body stage tests

* test: more body stage tests

* refactor: clean up body stage tests a bit

* test: fix broken tests

* refactor: clean up body stage tests

* test: more body stage tests
2022-11-15 08:49:11 +01:00
a77018fe03 chore: bump git deps in Cargo.lock (#187) 2022-11-10 19:57:11 +01:00
fd6a7cb628 WIP: feat: ipc support (#144)
* feat: ipc support

* copy futures

* feat: port ipc implementation

* cleanup

* add test

* fix clippy

* add request test
2022-11-10 19:47:00 +01:00
9e35d58b05 feat(primitive): Signer recovery (#179)
* feat(consensus): Signer recovery and tx validation

* Signature hash and use seckp256k1 over k256

* use deref_more for transactions

* cleanup and fix for eip1559 hash

* fix hash calculation on decoding
2022-11-09 18:11:32 +01:00
86ffb4756d feat(bin): simple reth node command (#158)
* feat(bin): simple reth node command

* use silent flag

* small chore
2022-11-07 12:10:09 -08:00
5721262740 refactor(net): unify capability types (#168)
* refactor(net): unify capability types

* refactor(net): unify capability types
2022-11-07 18:04:17 +01:00
b7cdfbf4f9 feat(net): network scaffolding (#110)
* feat(net): network scaffolding
2022-11-07 09:27:08 +01:00
f6d3a49d28 feat(sync): stage tx index (#149)
* init

* feat(sync): tx index stage

* create test runner

* finish tests

* clippy

* cleanup & docs

* add more comments

* revert headers test refactor

* rm unused function
2022-11-04 03:28:18 -07:00
699db03187 feat(db): db error handling and Provider scaffolding (#154)
* feat(db): db error handling and Provider scaffolding
* remove one map_err
* fix after merge
2022-11-03 17:00:58 +01:00
bff27a4154 fix(rpc): type encoding (#155)
* fix(rpc): type encoding

* copy bytes from ethers
2022-11-03 03:04:09 -07:00
3c72a12aff feat(eth-wire): Implement p2p stream (#114)
* wip: p2pstream

* add comment for handshake timeout

* temp allow some lint violations

* ignore unused_variables

* start of ping task

 * TODO: make it compile
 * TODO: test ping/pong/disconnect state machine
 * TODO: send subprotocol messages to stream
 * TODO: encode non-hello p2p messages as snappy encoding without using
   an encoder
 * TODO: create test comparing encoder to hand-written snappy encoding
   for ping, pong, disconnect messages

* implement message handling in stream poll method

 * restricts S to be Stream+Sink for P2PStream to implement Stream
 * start of a poll-based refactor

* impl Stream and Sink for P2PStream

 * add tests
 * TODO: make stream/sink types compatible
 * TODO: handshake message ids
 * TODO: inner poll fn
 * TODO: pinger interval
 * TODO: ethstream test
 * TODO: passthrough test

* create pingers and test

* impl working timeout interval pinger

 * it should be much easier to poll for pings and detect timeouts now

* use pinger in p2p stream

 * change item produced by stream so it's compatible with EthStream
   * add note on pros/cons
 * shorten message sends in stream

* improve errors and remove redundant methods

* fix handshake

* debugging printlns

* fix encoding and decoding

 * switch to snappy formatting for non-hello p2p messages

* cargo fmt

* perform handshake in ethstream over p2pstream test

 * remove check for `Hello` messages outside of the handshake because
   `P2PStream`s should assume messages sent in the sink are subprotocol
   messages, not `p2p` messages.

* impl From<EthVersion> for CapabilityMessage

* remove printlns

* add total_message method to EthVersion

* decode Hello in handshake

 * disallow protocol versions other than v5

* Integrate snappy and implement message size limits

* document constants, move stream definition

* fix missing hello message id

* implement shared capabilities

 * todo: test shared capabilities
 * todo: determine how / when / why to support multiple capabilities
 * removes obsolete authed and offset fields

* add sink api TODOs

* remove les

 * should add protocols when necessary rather than name unsupported
   protocols

* fix snappy compression length

 * add test for p2pstream over a passthrough codec which tests that
   peers agree on a single shared capability

* fix some clippy lints
2022-11-03 00:01:54 -07:00
c232a72338 test(stage): headers linear downloader (#143)
* headers stage linear test

* cleanup unwind
2022-11-02 17:28:44 -07:00
ac2f3fcd8a feat: Consensus crate and verification functions. (#152)
* wip executor

* wip

* Cleanup added some checks and structure to executor

* adding additional block/header checks

* add basefee calculation and check

* some cleanup

* Sanity check test

* Test for sanity check

* move verification to consensus crate

* cleanup

* Better Error handling
2022-11-02 12:59:51 +01:00
30ef6bff21 feat(rlp): add smol-str feature (#146) 2022-10-28 11:54:01 +02:00
61b8829bdf feat(sync): headers stage (#126)
* port headers stage

* clean up, utils & tests

* more tests

* unwind cleanup & tests

* remove unused rand crate
2022-10-25 16:50:01 -07:00
ce64fefd78 feat(net): add discv4 crate (#113)
* port kad

* feat: port kad bucket

* feat: add discv4

* chore: rustfmt

* cargo update

* just reuse discv5 table

* test: add rlp tests

* message encoding

* feat: impl codec roundtrip testing

* more work in message handling

* implement ping

* feat: impl commands

* cleanup

* more cleanup

* trim config

* more docs

* feat: implement recursive lookup

* docs

* cleanup config

* feat: implement update stream

* chore: config cleanup

* docs: add crate docs

* feat: more testing

* fix deny

* clarify ring

* docs: more docs

* use discv5 master

* docs: address review and add comments

* update readme

* rustmft

* chore(clippy): make clippy happy
2022-10-25 14:23:24 +02:00
ee41dfac35 feat(db): set-upChangeSet tables (#135)
* add AccountChangeSet

* add StorageChangeSet

* add tests to TxNumberAddress

* minor fixes
2022-10-25 03:47:45 -07:00
5c55c57c5e feat(db): switch to reth-libmdbx-rs, bump reth-mdbx-sys to 0.12.1 (#133)
* feat(db): Add mdbx-rs apache licenced code 55e234

* feat(db): replace mdbx with reth-mdbx, metadata changes

* chore(db): bump mdbx-sys to 0.12.1
2022-10-25 02:50:02 -07:00
2a8189d7c9 chore(db): Add tests and fuzz target to BlockNumHash (#130)
* add tests and fuzz target for BlockNumHash

* use block 1 for test_block_num_hash

* new clippy stds

* fmt
2022-10-25 01:23:10 -07:00
0998fc3eb7 chore: bump tiny-keccak (#127) 2022-10-24 17:06:25 -07:00
77ea53bb6e chore: reexport hex (#128) 2022-10-24 17:06:12 -07:00
65247ef203 feat: metrics (#92)
* feat(stages): `stage.progress` metric

* docs: metrics
2022-10-24 15:16:45 +02:00
8bc8bd6820 feat: add IntegerList type Elias-Fano (#111)
* add elias

* docs: should not be empty

* change models doc

* add arbitrary feature

* add simple test

* add test-utils to primitives

* cfg arbitrary
2022-10-24 06:14:59 -07:00
15bd88e637 headers(part2) - feat: add Downloader trait and test utils (#118)
* feat(interfaces): implement header client traits

* feat: add downloader trait implementer

* feat: use explicit error type instead of ok(false)

* feat: add constructor to HeaderLocked

* test: scaffold mock consensus, downloader and headersclient helpers

* test: implement test consensus

* test: implement test headers client

* refactor: cleanup download headers

* chore: fix lint

* s/test_utils/test_helpers

* headers(part 3) feat: implement Linear downloader (#119)

* feat: add headers downloaders crate

* feat: more scaffolding

* interfaces: generalize retryable erros

* feat: implement linear downloader

* fix linear downloader tests & add builder

* extend & reverse

* feat: linear downloader generics behind arc and reversed return order (#120)

* put client & consensus behind arc and return headers in rev

* cleanup

Co-authored-by: Roman Krasiuk <rokrassyuk@gmail.com>

* extract test_utils

* cargo fmt

Co-authored-by: Roman Krasiuk <rokrassyuk@gmail.com>
2022-10-24 04:56:43 -07:00
a4e505132c headers(part1) feat(interfaces): introduce implicit trait bounds (#117)
* feat(ifaces): modify DB GAT to have implicit bounds

https://sabrinajewson.org/blog/the-better-alternative-to-lifetime-gats\#the-better-gats

* test(ifaces): impl new trait for mock

* feat(ifaces): impl db container

we use that downstream to avoid consuming the db transactions
when committing a stage

* fix(test): explicitly define lifetime

* test: ensure can spawn

* test: pls review

* chore: lints/warnings

* feat(db): impl the new gats

* test(db): try to make real db work with gats like in stages

* test(db): make Stage trait take DBContainer

otherwise we cannot call db.commit()
gst

* feat(stages): impl new traits

* chore(db): cleanup tests

* chore: remove unused imports
2022-10-23 23:10:59 -07:00
8af8c89018 feat: support finalized and safe variants (#115) 2022-10-21 20:02:06 +02:00
483bcdf9ab feat(db): Database Abstraction (#101)
* database tx traits

* wip build passes

* Db and tx results

* nightly and db GAT

* Impl tx, wip cursor

* Move Decode to Table Key, working cursor trait

* wip dupsort

* build all Cursor abstraction

* cleanup

* wip cleanup

* old stages

* codecs moved o interface,stages wip

* resolve db ref, it builds

* Add tx commit after execution

* fmt

* Remove sync send restriction

* Add missing rw cursor functions

* Cleanup, added missing cursor fn. rust toolchain

* fmt

* add nightly to ci

* deny dead_code, remove unwrap

* rm printfn, stages fix, bench fix
2022-10-20 08:32:13 -07:00
e9945b565a chore: reexport sha3 from primitives (#109)
* chore: reexport sha3 from primitives

* feat: add keccak256

* replace sha3 with tinykeccak
2022-10-20 16:43:29 +02:00
4298e3aa87 feat(rpc): more rpc scaffolding (#99)
* feat(rpc): more rpc scaffolding

* use Into<String>

* rm unused error
2022-10-19 22:12:53 +02:00
630baf5d70 feat(db): fuzzing & benchmark (#86)
* fuzz with test-fuzz

* move fuzzing to db/codecs

* add criterion & iai

* print encoded size sum on criterion benchmark

* fix BlockNumHash encode

* add gh action for benchmarks

* don't sum results

* test ci values

* Revert "test ci values"

This reverts commit cc47fd80538b2c0073592f824c2693c927021f8f.

* specify criterion version

* add docs

* remove benchmark job
2022-10-19 13:08:07 -07:00
2e19f94048 feat: impl rlp for receipt (#83)
* feat: impl rlp for receipt

* fix: change to bloom

* chore: rustfmt
2022-10-17 07:13:02 -07:00
063b444792 feat(db): codec encoding/decoding (#51)
* wip

* add table macro

* add simple put get test with Address

* add Env.view and Env.update

* docs

* slightly change the test

* add initial table initialization and placeholders

* lint & some

* replace String with str

* add error.rs

* add docs to encode

* add docs

* clamp

* add source on libmdbx_max_page_size

* add BlockNumer_BlockHash

* add scale

* set header filed to bytes Bytes

* remove unwrap

* restrict scale to chosen types

* into bytes

* add postcard

* changed to BlockNumHash

* add proc_macro_attribute codecs

* fix feature flagging

* set a version for postcard

* cleanup

* seal ScaleOnly

* remove unnecessary dependencies

* properly encode/decode blocknumhash

* change Account codec to scale

* add missing feature to scale

* add codec to a couple more types

* silence clippy

* add docs about table encoding

* move and add reth-codecs

* clippy

* make proc-macro visible

* add README.md
2022-10-17 01:04:57 -07:00
8009d997c0 Implement ETH P2P (#81)
* refactor: move things to types

* feat(ethwire): bring in message type from ethp2p

30c11138d5/src/message.rs

* feat(ethwire): add eth-stream with Stream/Sink impls

* feat(ethwire): make Sink error an EthStreamError

* feat(ecies): expose util module

* add more deps

* feat: add broadcast newblockhashes

* fix: rlp serialize with message-id first

* chore: cleanup doc

* wip: test eth connection

* bump cargo lock

* feat: add rlp codec and get stream tests working

* fix: convert RlpCodec to PassthroughCodec

we were rlp encoding twice

* Revert "fix: convert RlpCodec to PassthroughCodec"

This reverts commit 6e6e0a58112c14d7ffba62dc83f9747ddc280641.

This does not handle framing, which would fail decoding if a partial
write happened to the socket.

* add tracing

* refactor: add handshake error

* feat(ethwire): add status handshake

* test(ethwire): handshake works

* refactor: expose EthStream::new

* chore: clippy lints

* fix: get rid of rlp codec

we can instead use LengthLimitedCodec from Tokio IO, which we re-export
as PassthroughCodec

* fix(eth): add handshake + msg error checks

1. 10MB message lengths
2. Same Genesis, Version, Chain on Status Handshake

* chore: ignore result_large_err

Co-authored-by: Matthias Seitz <matthias.seitz@outlook.de>
2022-10-16 20:10:25 -07:00
5dfe5ac29b feat(net): add Framed ECIES implementation (#80)
* feat(net): add ecies crate

* cargo fmt

* add hex-literal and proptest to dev-dependencies

 * adds std feature to reth-rlp

* document util

* document more

* chore: allow missing docs in ecies/algorith.rs

* feat(ecies): improve error handling

remove anyhow and strictly type all errors

* refactor(ecies): movem ingress/egress to lib.rs

* chore(ecies): allow missing docs in mac

* chore(ecies): cleanup utils

* refactor(ecies): move ECIES Codec to separate file

* refactor(ecies): rename proto to stream

* add test scaffold

* implement server/client read/write test

* chore: clippy / fmt

Co-authored-by: Georgios Konstantopoulos <me@gakonst.com>
2022-10-15 18:20:17 -07:00
07a83b163f feat(primitives): add H128 (#79)
* feat(primitives): add H128

 * uses patch for now

* cargo fmt
2022-10-14 20:06:26 -07:00