> ## Documentation Index
> Fetch the complete documentation index at: https://docs.leafage.chaintable.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Data flow

> The complete path of a block from execution on the write node to being queryable via eth_call, including reorgs, catch-up, and cold start

This page follows a single block through the whole path. Each stage is annotated with the corresponding code location so you can find the implementation.

## End-to-end sequence

```mermaid theme={null}
sequenceDiagram
    participant N as P2P network
    participant G as Write node
    participant S3 as S3 (internal/external buckets)
    participant K as Kafka internal topic
    participant L as leafage-evm
    participant C as consistency-checker
    participant E as etcd
    participant P as nodex-proxy

    N->>G: New block
    G->>G: Execute block, tracer collects<br/>traces / events / stateDiff
    G->>S3: OnCommit uploads concurrently<br/>Header, StateDiff, BlockFile, Validation
    G->>K: writeBlockAndSetHead publishes<br/>BlockChangeNotification (Leader only)

    par State ingestion
        K-->>L: Notification
        L->>S3: Fetch Header + StateDiff
        L->>L: Push onto StateTree head
    and Consistency checking
        K-->>C: Same notification
        C->>L: Poll eth_blockNumber
        C->>S3: Read / rewrite BlockValidation
        C->>E: Update node state + lastBlockNumber
        C->>C: Publish external notification
    end

    E-->>P: watch event
    P->>L: Route eth_call
```

## Stage 1: Execution and tracing

The write node syncs and executes blocks normally over P2P. The pipeline tracer hooks into the EVM as `tracing.Hooks` and collects data during execution.

| Hook                    | Timing                 | What it collects                                |
| ----------------------- | ---------------------- | ----------------------------------------------- |
| `OnBlockStart`          | Block execution starts | Initializes `BlockCtx`, creates `BlockFile`     |
| `OnTxStart` / `OnTxEnd` | Every transaction      | Transaction metadata, flattens the call tree    |
| `OnEnter` / `OnExit`    | Every call frame       | Builds the call tree, marks failure propagation |
| `OnOpcode`              | Every instruction      | Detects `SSTORE`, preloads prestate             |
| `OnLog`                 | Event emitted          | Records the event and the trace it belongs to   |
| `OnCommit`              | After state commit     | Generates `BlockStorageDiff`, triggers upload   |

`OnCommit` and `OnBlockDBStart` are Leafage's extensions to upstream Geth. They are defined in `core/tracing/hooks.go` and dispatched by `core/blockchain.go` in `ProcessBlock`.

<Note>
  There are two ways to obtain the StateDiff: by default it is converted directly from the StateDB commit set received in `OnCommit`; with `enable_prestate_tracer` configured, it is collected at the instruction level by the prestate tracer instead. The former is more accurate and cheaper; the latter is for clients without a commit hook.
</Note>

## Stage 2: Serialization and distribution

`OnCommit` runs four uploads concurrently (`tracer/pipeline_tracer.go`):

| Data            | Bucket          | Key                                           | Format      | Consumer                                  |
| --------------- | --------------- | --------------------------------------------- | ----------- | ----------------------------------------- |
| Header          | Internal bucket | `{chainID}[/{version}]/{blockHash}/block`     | JSON + gzip | leafage-evm                               |
| StateDiff       | Internal bucket | `{chainID}[/{version}]/{stateRoot}/stateDiff` | RLP         | leafage-evm                               |
| BlockFile       | External bucket | `{chainID}[/{version}]/{blockHash}`           | JSON + gzip | External consumers                        |
| BlockValidation | External bucket | `{chainID}[/{version}]/{height}/{blockHash}`  | JSON + gzip | consistency-checker, leafage-evm catch-up |

After the uploads complete, the write node publishes a notification to Kafka when it sets the new canonical head (`writeBlockAndSetHead` in `core/blockchain.go`). Before publishing, it uses `getCommonAncestor` to compare the last published block with the current head:

* Only new blocks → `changeType: 1`, `newBlocks` in ascending height order
* A dropped branch exists → `changeType: 2`, carrying both `dropBlocks` and `newBlocks`

Only the Leader instance publishes Kafka messages; standby nodes still upload to S3. The Leader is decided by pipeline's etcd election.

<Warning>
  S3 upload and Kafka publish are not one atomic operation. The corresponding S3 objects are usually ready by the time a consumer receives the notification, but implementations must tolerate brief 404s and retry — both leafage-evm and consistency-checker retry with backoff.
</Warning>

## Stage 3: State ingestion

leafage-evm's `KafkaUpdater` (`bin/leafage-evm/src/updater/kafka_updater.rs`) consumes the notification:

<Steps>
  <Step title="Parse the notification">
    Take the block hash and parent hash from `newBlocks`.
  </Step>

  <Step title="Fetch from S3 in parallel">
    Fetch the Header by hash and the StateDiff by state root. When the parent and current block share the same state root, the diff fetch is skipped and treated as an empty diff.
  </Step>

  <Step title="Update the StateTree">
    Call `tree.update_block(block_info, block_diff)` to attach the new diff layer at the head of the linked list.
  </Step>

  <Step title="Commit the offset">
    After a successful flush to disk, write it to `offset_dir` for crash recovery.
  </Step>
</Steps>

The StateTree is a linked list of diff layers: each layer stores only the changes relative to its parent layer. A query walks down from the head and returns on the first hit; if no layer hits, it falls through to RocksDB.

## Stage 4: Finalization

When the block depth exceeds `--diff-depth-limit` (default 64), the oldest layer is flushed to RocksDB and removed from memory.

* **State node**: overwrites `address` / `address || slot` directly, keeping only the latest value.
* **Archive node**: dual write. `address || block_num` keeps historical versions, and `address || u64::MAX` serves as the fast path for the latest value. Historical queries use RocksDB's `seek_for_prev` to locate the nearest version not greater than the target height.

## Stage 5: Query serving

```mermaid theme={null}
flowchart LR
    C["Client"] -->|"POST /:chainId"| P["nodex-proxy"]
    P -->|"latest / ≤64 from chain head"| S["State node"]
    P -->|">64 from chain head"| A["Archive node"]
    P -->|"error code -39008"| N["Native node"]
    S -->|"-39006 retry"| A
```

nodex-proxy parses the block parameter in the request, combines it with `lastBlockNumber` from etcd to decide the node pool, then picks a specific node by weight or round-robin. leafage-evm executes with revm on its local state and returns the result.

Two automatic failover paths:

| Error code | Meaning                                                           | Retry target                        |
| ---------- | ----------------------------------------------------------------- | ----------------------------------- |
| `-39006`   | `StateBlockNotFound`, the State node has no state for that height | Archive node                        |
| `-39008`   | `CosmosPrecompile`, requires native chain capabilities            | Native node (path rewritten to `/`) |

## Parallel path: consistency checking

consistency-checker consumes the same Kafka notification but is not on leafage-evm's data path. Its processing order (`Process` in `check/check.go`):

<Steps>
  <Step title="Message validation and deduplication">
    Duplicate or already-processed messages are aligned and then advanced directly, avoiding retry deadlocks.
  </Step>

  <Step title="Prefetch from S3">
    While waiting for replicas, prefetch the BlockValidation and the key list for the same height in parallel.
  </Step>

  <Step title="Wait for replicas to converge">
    Poll `eth_blockNumber` on all replicas every `check_interval_ms` (default 20ms) until a `ready_ratio` (default 0.8) share of replicas has caught up to that height. If the threshold is still not met after `check_timeout_ms` (default 2000ms), the check fails.
  </Step>

  <Step title="Write to the local Pebble DB">
    Two indexes: `h{hash}` → BlockInfo, `n{number}` → hash, for JSON-RPC queries.
  </Step>

  <Step title="Mark forks">
    List all BlockValidation objects at the same height in the S3 external bucket and rewrite the non-canonical ones as `is_fork: true`.
  </Step>

  <Step title="Publish external notifications">
    Send the drop notifications first, then the new-block notifications, to the external Kafka topic.
  </Step>

  <Step title="Post-publish recheck">
    Re-check the fork marks at the same height with a fresh LIST to catch objects uploaded while waiting for replicas.
  </Step>
</Steps>

Meanwhile, the result of every polling round is written back to etcd: each node's `stateType` (1 caught up / 2 lagging / 3 offline) and the chain's `lastBlockNumber`. This is what nodex-proxy bases its routing on.

## Reorg handling at three layers

Leafage handles chain reorgs at three separate layers. Understanding how they divide the work helps you avoid misjudging which layer a problem is in.

<AccordionGroup>
  <Accordion title="Write node: produces dropBlocks">
    `writeBlockAndSetHead` uses `getCommonAncestor` to compare the last published block with the new head, takes the path from the common ancestor to the old head as `dropBlocks` and the path to the new head as `newBlocks`, and sends them together in one message with `changeType: 2`.
  </Accordion>

  <Accordion title="leafage-evm: fork layers and safe backfill">
    The StateTree indexes diff layers with two maps: `hash_diff_map` holds all blocks (including fork blocks), while `num_diff_map` tracks only the canonical chain. As a result, queries by hash can still reach fork state, while queries by height always follow the canonical chain.

    During S3 catch-up, blocks near the chain head may land on the wrong branch. `--catchup-safe-depth` specifies how many blocks near the chain head are backfilled along the exact parent-hash chain from Kafka notifications instead of being looked up by height index. The value should be larger than the target chain's maximum reorg depth (for example, Moonriver uses 64); `0` disables it.
  </Accordion>

  <Accordion title="consistency-checker: marking and external notifications">
    * On `changeType: 2`, it first rewrites the BlockValidation objects for `dropBlocks` as `is_fork: true`, then scans the other objects at the same heights.
    * Periodic scan as a fallback: every `fork_scan_interval_sec` (default 60 seconds) it looks back `fork_scan_lookback` heights.
    * External consumers detect forks through the `is_fork` field of `OuterBlockChangeNotification`.
  </Accordion>
</AccordionGroup>

## Cold start and catch-up

On startup, leafage-evm chooses its path based on the persisted Kafka offset:

```text theme={null}
Read the offset from offset_dir
  ├─ offset >= Kafka low watermark → resume consuming from the offset
  └─ offset missing or expired     → catch up from S3 first, then consume from the latest position
```

S3 catch-up backfills block by block by height; the batch size is controlled by `--init-task-queue-size` (default 256). The processing order for each height:

1. List the objects at that height in the external bucket with the prefix `{chainID}[/{version}]/{height}/`.
2. Take the hash whose `is_fork` is false.
3. Fetch the Header and StateDiff from the internal bucket by that hash.

<Tip>
  This is an implicit contract between consistency-checker and leafage-evm: fork marks are written on the BlockValidation objects in the external bucket, and leafage-evm relies on them during catch-up to pick the canonical branch. If the checker stalls, catch-up degrades to reading and judging objects one by one when a height has multiple objects.
</Tip>

The usual way to bring up a brand-new node is to download a RocksDB snapshot and then catch up from Kafka, which typically completes within minutes, rather than replaying from genesis.

## Observability points

When troubleshooting "data didn't arrive", check these metrics in data-path order:

| Step                                                 | Metric                                                                                  | Component           |
| ---------------------------------------------------- | --------------------------------------------------------------------------------------- | ------------------- |
| Block execution complete                             | `LatestBlockNumber`                                                                     | pipeline            |
| S3 upload complete                                   | `LatestUploadedBlockNumber`; the gap from the previous item reflects the upload backlog | pipeline            |
| Kafka publish time                                   | `BlockPushTimer`                                                                        | pipeline            |
| Replica wait time                                    | `pipeline_replica_ready_wait_seconds`, `pipeline_replica_ready_timeouts_total`          | consistency-checker |
| Confirmed height                                     | `pipeline_block_num`                                                                    | consistency-checker |
| End-to-end latency from write node to external Kafka | `pipeline_block_ingress_to_outer_kafka_seconds`                                         | consistency-checker |
| Fork mark anomalies                                  | `pipeline_fork_scan_rewrites_total`, `pipeline_drop_block_rewrite_failures_total`       | consistency-checker |
| Query failure rate                                   | `jrpcx_rpc_calls_failed`                                                                | nodex-proxy         |