Skip to main content
This page follows a single block through the whole path. Each stage is annotated with the corresponding code location so you can find the implementation.

​
End-to-end sequence

​
Stage 1: Execution and tracing

The write node syncs and executes blocks normally over P2P. The pipeline tracer hooks into the EVM as tracing.Hooks and collects data during execution. OnCommit and OnBlockDBStart are Leafage’s extensions to upstream Geth. They are defined in core/tracing/hooks.go and dispatched by core/blockchain.go in ProcessBlock.
There are two ways to obtain the StateDiff: by default it is converted directly from the StateDB commit set received in OnCommit; with enable_prestate_tracer configured, it is collected at the instruction level by the prestate tracer instead. The former is more accurate and cheaper; the latter is for clients without a commit hook.

​
Stage 2: Serialization and distribution

OnCommit runs four uploads concurrently (tracer/pipeline_tracer.go): After the uploads complete, the write node publishes a notification to Kafka when it sets the new canonical head (writeBlockAndSetHead in core/blockchain.go). Before publishing, it uses getCommonAncestor to compare the last published block with the current head:
  • Only new blocks → changeType: 1, newBlocks in ascending height order
  • A dropped branch exists → changeType: 2, carrying both dropBlocks and newBlocks
Only the Leader instance publishes Kafka messages; standby nodes still upload to S3. The Leader is decided by pipeline’s etcd election.
S3 upload and Kafka publish are not one atomic operation. The corresponding S3 objects are usually ready by the time a consumer receives the notification, but implementations must tolerate brief 404s and retry — both leafage-evm and consistency-checker retry with backoff.

​
Stage 3: State ingestion

leafage-evm’s KafkaUpdater (bin/leafage-evm/src/updater/kafka_updater.rs) consumes the notification:
1

Parse the notification

Take the block hash and parent hash from newBlocks.
2

Fetch from S3 in parallel

Fetch the Header by hash and the StateDiff by state root. When the parent and current block share the same state root, the diff fetch is skipped and treated as an empty diff.
3

Update the StateTree

Call tree.update_block(block_info, block_diff) to attach the new diff layer at the head of the linked list.
4

Commit the offset

After a successful flush to disk, write it to offset_dir for crash recovery.
The StateTree is a linked list of diff layers: each layer stores only the changes relative to its parent layer. A query walks down from the head and returns on the first hit; if no layer hits, it falls through to RocksDB.

​
Stage 4: Finalization

When the block depth exceeds --diff-depth-limit (default 64), the oldest layer is flushed to RocksDB and removed from memory.
  • State node: overwrites address / address || slot directly, keeping only the latest value.
  • Archive node: dual write. address || block_num keeps historical versions, and address || u64::MAX serves as the fast path for the latest value. Historical queries use RocksDB’s seek_for_prev to locate the nearest version not greater than the target height.

​
Stage 5: Query serving

nodex-proxy parses the block parameter in the request, combines it with lastBlockNumber from etcd to decide the node pool, then picks a specific node by weight or round-robin. leafage-evm executes with revm on its local state and returns the result. Two automatic failover paths:

​
Parallel path: consistency checking

consistency-checker consumes the same Kafka notification but is not on leafage-evm’s data path. Its processing order (Process in check/check.go):
1

Message validation and deduplication

Duplicate or already-processed messages are aligned and then advanced directly, avoiding retry deadlocks.
2

Prefetch from S3

While waiting for replicas, prefetch the BlockValidation and the key list for the same height in parallel.
3

Wait for replicas to converge

Poll eth_blockNumber on all replicas every check_interval_ms (default 20ms) until a ready_ratio (default 0.8) share of replicas has caught up to that height. If the threshold is still not met after check_timeout_ms (default 2000ms), the check fails.
4

Write to the local Pebble DB

Two indexes: h{hash} → BlockInfo, n{number} → hash, for JSON-RPC queries.
5

Mark forks

List all BlockValidation objects at the same height in the S3 external bucket and rewrite the non-canonical ones as is_fork: true.
6

Publish external notifications

Send the drop notifications first, then the new-block notifications, to the external Kafka topic.
7

Post-publish recheck

Re-check the fork marks at the same height with a fresh LIST to catch objects uploaded while waiting for replicas.
Meanwhile, the result of every polling round is written back to etcd: each node’s stateType (1 caught up / 2 lagging / 3 offline) and the chain’s lastBlockNumber. This is what nodex-proxy bases its routing on.

​
Reorg handling at three layers

Leafage handles chain reorgs at three separate layers. Understanding how they divide the work helps you avoid misjudging which layer a problem is in.
writeBlockAndSetHead uses getCommonAncestor to compare the last published block with the new head, takes the path from the common ancestor to the old head as dropBlocks and the path to the new head as newBlocks, and sends them together in one message with changeType: 2.
The StateTree indexes diff layers with two maps: hash_diff_map holds all blocks (including fork blocks), while num_diff_map tracks only the canonical chain. As a result, queries by hash can still reach fork state, while queries by height always follow the canonical chain.During S3 catch-up, blocks near the chain head may land on the wrong branch. --catchup-safe-depth specifies how many blocks near the chain head are backfilled along the exact parent-hash chain from Kafka notifications instead of being looked up by height index. The value should be larger than the target chain’s maximum reorg depth (for example, Moonriver uses 64); 0 disables it.
  • On changeType: 2, it first rewrites the BlockValidation objects for dropBlocks as is_fork: true, then scans the other objects at the same heights.
  • Periodic scan as a fallback: every fork_scan_interval_sec (default 60 seconds) it looks back fork_scan_lookback heights.
  • External consumers detect forks through the is_fork field of OuterBlockChangeNotification.

​
Cold start and catch-up

On startup, leafage-evm chooses its path based on the persisted Kafka offset:
S3 catch-up backfills block by block by height; the batch size is controlled by --init-task-queue-size (default 256). The processing order for each height:
  1. List the objects at that height in the external bucket with the prefix {chainID}[/{version}]/{height}/.
  2. Take the hash whose is_fork is false.
  3. Fetch the Header and StateDiff from the internal bucket by that hash.
This is an implicit contract between consistency-checker and leafage-evm: fork marks are written on the BlockValidation objects in the external bucket, and leafage-evm relies on them during catch-up to pick the canonical branch. If the checker stalls, catch-up degrades to reading and judging objects one by one when a height has multiple objects.
The usual way to bring up a brand-new node is to download a RocksDB snapshot and then catch up from Kafka, which typically completes within minutes, rather than replaying from genesis.

​
Observability points

When troubleshooting “data didn’t arrive”, check these metrics in data-path order: