> ## Documentation Index
> Fetch the complete documentation index at: https://docs.leafage.chaintable.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Deployment

> Deployment topologies from single-machine validation to production clusters, resource planning, and common operations

One chain is one independent deployment unit. This page describes three topologies: single-machine validation, single-chain production, and a multi-chain cluster.

## Infrastructure

A production deployment needs three pieces of shared infrastructure:

| Component | Requirements                                                                                                                                     |
| --------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| Kafka     | The internal topic must be a **single partition** (it carries the totally ordered block stream); the external topic can be partitioned as needed |
| S3        | Two buckets: the internal bucket (Header + StateDiff) and the external bucket (BlockFile + Validation), which can be shared across chains        |
| etcd v3   | 3 nodes recommended; holds node registration, Leader election, chain height, and routing configuration                                           |

Multiple chains share the same infrastructure: all Kafka topics, S3 keys, and etcd keys start with the `chainID`.

## Setup 1: single-machine validation

No Kafka or S3 needed. The write node exposes the `trace` namespace and leafage-evm polls it in HTTP mode. Suitable for local development and for validation when adapting a new chain.

```bash theme={null}
geth --http --http.api=eth,debug,trace --http.addr=0.0.0.0 --http.port=8545

leafage-evm standalone \
  --db-path /tmp/leafage \
  --listen-addr 0.0.0.0:8659 \
  --chain-cfg 1 \
  --rpc-addr http://127.0.0.1:8545
```

## Setup 2: single-chain production

```yaml theme={null}
services:
  beacon:
    image: sigp/lighthouse:v8.0.1
    command:
      - lighthouse
      - beacon_node
      - --network=mainnet
      - --datadir=/var/data/lighthouse
      - --execution-endpoint=http://geth:8551
      - --execution-jwt=/var/data/geth/jwtsecret
      - --checkpoint-sync-url=https://mainnet.checkpoint.sigp.io

  geth:
    image: public.ecr.aws/b2h7a5c4/chaintable/ethereum-writer:v1.17.3-debank-3
    ports: ["8666:8545"]
    entrypoint:
      - /app/geth
      - --datadir=/var/data/geth
      - --syncmode=full
      - --gcmode=archive
      - --authrpc.addr=0.0.0.0
      - --authrpc.jwtsecret=/var/data/geth/jwtsecret
      - --http
      - --http.addr=0.0.0.0
      - --http.api=net,web3,eth,admin,debug,txpool,engine,trace
      - --vmtrace=pipeline
      - '--vmtrace.jsonconfig={"region":"ap-northeast-1","node_x_bucket":"nodex-internal","chain_table_bucket":"chaintable-pipeline","brokers":["kafka-1:9092"],"etcd_endpoints":["http://etcd:2379"]}'

  leafage-evm:
    image: public.ecr.aws/b2h7a5c4/chaintable/leafage-evm-x:v1.2.31
    ports: ["8659:8659"]
    environment: ["RUST_LOG=info"]
    command:
      - standalone
      - --db-path=/nodex-eth
      - --listen-addr=0.0.0.0:8659
      - --chain-cfg=1
      - --kafka-s3-config=/etc/leafage/kafka_s3.json
      - --prometheus-addr=0.0.0.0:9100

  consistency-checker:
    image: consistency-checker:latest
    command: ["-config", "/config/config.yml"]
```

nodex-proxy is usually deployed separately and covers all chains:

```bash theme={null}
./node-proxy -config config.yaml -listen 8663
```

### Resource planning (ETH mainnet)

| Service                    | CPU     | Memory |
| -------------------------- | ------- | ------ |
| beacon (Lighthouse)        | 4 cores | 12 GB  |
| geth (full sync + archive) | 4 cores | 24 GB  |
| leafage-evm                | 4 cores | 16 GB  |

leafage-evm's QPS scales roughly linearly with the number of CPU cores. To increase query throughput, scale only this one service's CPU and memory.

| Disk                    | Capacity |
| ----------------------- | -------- |
| beacon + geth (archive) | \~850 GB |
| leafage-evm (archive)   | \~450 GB |
| leafage-evm (state)     | \~150 GB |

Disks need at least 3000 IOPS; SSD is recommended, and reserve 20–30% capacity headroom.

On AWS, gp3's default 3000 IOPS / 125 MB/s is enough for steady-state operation (one block every 12 seconds), but two phases of the first startup put higher pressure on the disk:

* Snapshot extraction is limited by sequential write throughput.
* Catch-up after snapshot restore is limited by random read/write IOPS.

<Note>
  Enabling `--ancient.prune` on the write node reduces disk usage significantly: historical block bodies and receipts are pruned once they leave the most recent 90000-block window. Downstream consumers read from S3 and do not depend on the write node keeping historical block bodies. Requires `--syncmode full`.
</Note>

## Setup 3: multi-chain cluster

```text theme={null}
Per chain:
  1 × consensus client (PoS chains)
  1–2 × write nodes (pipeline's etcd election decides which one publishes to Kafka)
  1 × consistency-checker
  N × leafage-evm (mainly State, add Archive as needed)

Shared globally:
  1 group of nodex-proxy (multiple instances, managing multiple chains by chainId)
  Kafka / S3 / etcd
  Prometheus + Grafana
```

You can deploy two write nodes: both upload to S3, and only the Leader publishes to Kafka. The standby takes over when the primary fails, avoiding a data path outage.

## Common operations

<AccordionGroup>
  <Accordion title="Add a query node">
    1. Download the RocksDB snapshot from S3 and extract it to `--db-path`
    2. Start leafage-evm; it automatically catches up to the chain head from the Kafka offset or S3
    3. It registers itself in etcd on startup; once consistency-checker confirms it has caught up, the checker sets `stateType` to `1`
    4. nodex-proxy picks up the change through its watch and, once the health check passes, adds the node to the load balancing pool

    The whole process usually completes within minutes, with no manual intervention on the proxy.
  </Accordion>

  <Accordion title="Take a query node offline">
    Just stop the process. leafage-evm deletes its own etcd key on exit; even if it exits abnormally, consistency-checker marks it offline and deletes it after its polling fails.
  </Accordion>

  <Accordion title="Switch the write node Leader">
    Through nodex-proxy's admin API:

    ```bash theme={null}
    curl -X POST http://proxy:8663/1/writers/switchLeader -d '{...}'
    curl http://proxy:8663/1/writers/leader
    ```

    You can also modify `{chainID}/writers/leader` in etcd directly. After the switch, the new Leader waits for `grace_period` before it starts publishing.
  </Accordion>

  <Accordion title="Version switching">
    In version mode, modify `{chainID}/version` in etcd:

    * After consistency-checker detects the version change, it releases or acquires the singleton topic's publish lock and aligns the topic to the new version's progress
    * nodex-proxy rewrites requests for the base chain ID to the `{chainId}-{version}` node pool

    Clients need no changes.
  </Accordion>

  <Accordion title="Rewind and resync">
    If leafage-evm's state becomes abnormal, you can rewind the chain head to an earlier block and resync from S3:

    ```bash theme={null}
    leafage-evm rewind --db-path /nodex-eth --to-block 12345
    ```
  </Accordion>
</AccordionGroup>

## Troubleshooting

| Symptom                                     | Check order                                                                                                                                                                                                    |
| ------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Query node not catching up with blocks      | Is Kafka consumption normal → Are S3 fetches reporting errors (check the `updater` target with `RUST_LOG=info`) → Has the offset expired and triggered a full catch-up                                         |
| External consumers receive no notifications | Is the checker's `pipeline_replica_ready_timeouts_total` growing → Does the replica have a node that has lagged for a long time and lowered the ready ratio → In version mode, does it hold the singleton lock |
| proxy reports no available nodes            | Does the node key exist in etcd → Is `stateType` 3 → Does the health check pass (inspect with `/{chainId}/getAllNodes`)                                                                                        |
| Historical queries fail                     | Is there an Archive node in the pool → Does the requested height fall below `--historical-height` without `--historical-rpc` configured                                                                        |
| Duplicate-height objects on S3              | Normal; reorgs leave fork blocks behind. Confirm that the checker marked the non-canonical ones with `is_fork: true` (see `pipeline_fork_scan_rewrites_total`)                                                 |
| Write node disk grows too fast              | Is `--ancient.prune` enabled; does pipeline's `s3_temp_dir` have leftover files that were not uploaded                                                                                                         |

## References

* [leafage-evm deployment guide](https://github.com/Chaintable/leafage-evm/blob/main/docs/deploy/Deploy.md) — includes one-click scripts, snapshot download, and directory layout
* [nodex-proxy deployment guide](https://github.com/Chaintable/nodex-proxy/blob/main/docs/deployment_cn.md) (Chinese) — Docker Compose, Kubernetes, systemd, and production tuning
* [consistency-checker deployment notes](https://github.com/Chaintable/consistency-checker/tree/main/deploy)