Deployment

One chain is one independent deployment unit. This page describes three topologies: single-machine validation, single-chain production, and a multi-chain cluster.

Infrastructure

A production deployment needs three pieces of shared infrastructure:

Component	Requirements
Kafka	The internal topic must be a single partition (it carries the totally ordered block stream); the external topic can be partitioned as needed
S3	Two buckets: the internal bucket (Header + StateDiff) and the external bucket (BlockFile + Validation), which can be shared across chains
etcd v3	3 nodes recommended; holds node registration, Leader election, chain height, and routing configuration

Multiple chains share the same infrastructure: all Kafka topics, S3 keys, and etcd keys start with the chainID.

Setup 1: single-machine validation

No Kafka or S3 needed. The write node exposes the trace namespace and leafage-evm polls it in HTTP mode. Suitable for local development and for validation when adapting a new chain.

geth --http --http.api=eth,debug,trace --http.addr=0.0.0.0 --http.port=8545

leafage-evm standalone \
  --db-path /tmp/leafage \
  --listen-addr 0.0.0.0:8659 \
  --chain-cfg 1 \
  --rpc-addr http://127.0.0.1:8545

Setup 2: single-chain production

services:
  beacon:
    image: sigp/lighthouse:v8.0.1
    command:
      - lighthouse
      - beacon_node
      - --network=mainnet
      - --datadir=/var/data/lighthouse
      - --execution-endpoint=http://geth:8551
      - --execution-jwt=/var/data/geth/jwtsecret
      - --checkpoint-sync-url=https://mainnet.checkpoint.sigp.io

  geth:
    image: public.ecr.aws/b2h7a5c4/chaintable/ethereum-writer:v1.17.3-debank-3
    ports: ["8666:8545"]
    entrypoint:
      - /app/geth
      - --datadir=/var/data/geth
      - --syncmode=full
      - --gcmode=archive
      - --authrpc.addr=0.0.0.0
      - --authrpc.jwtsecret=/var/data/geth/jwtsecret
      - --http
      - --http.addr=0.0.0.0
      - --http.api=net,web3,eth,admin,debug,txpool,engine,trace
      - --vmtrace=pipeline
      - '--vmtrace.jsonconfig={"region":"ap-northeast-1","node_x_bucket":"nodex-internal","chain_table_bucket":"chaintable-pipeline","brokers":["kafka-1:9092"],"etcd_endpoints":["http://etcd:2379"]}'

  leafage-evm:
    image: public.ecr.aws/b2h7a5c4/chaintable/leafage-evm-x:v1.2.31
    ports: ["8659:8659"]
    environment: ["RUST_LOG=info"]
    command:
      - standalone
      - --db-path=/nodex-eth
      - --listen-addr=0.0.0.0:8659
      - --chain-cfg=1
      - --kafka-s3-config=/etc/leafage/kafka_s3.json
      - --prometheus-addr=0.0.0.0:9100

  consistency-checker:
    image: consistency-checker:latest
    command: ["-config", "/config/config.yml"]

nodex-proxy is usually deployed separately and covers all chains:

./node-proxy -config config.yaml -listen 8663

Resource planning (ETH mainnet)

Service	CPU	Memory
beacon (Lighthouse)	4 cores	12 GB
geth (full sync + archive)	4 cores	24 GB
leafage-evm	4 cores	16 GB

leafage-evm’s QPS scales roughly linearly with the number of CPU cores. To increase query throughput, scale only this one service’s CPU and memory.

Disk	Capacity
beacon + geth (archive)	~850 GB
leafage-evm (archive)	~450 GB
leafage-evm (state)	~150 GB

Disks need at least 3000 IOPS; SSD is recommended, and reserve 20–30% capacity headroom. On AWS, gp3’s default 3000 IOPS / 125 MB/s is enough for steady-state operation (one block every 12 seconds), but two phases of the first startup put higher pressure on the disk:

Snapshot extraction is limited by sequential write throughput.
Catch-up after snapshot restore is limited by random read/write IOPS.

Enabling --ancient.prune on the write node reduces disk usage significantly: historical block bodies and receipts are pruned once they leave the most recent 90000-block window. Downstream consumers read from S3 and do not depend on the write node keeping historical block bodies. Requires --syncmode full.

Setup 3: multi-chain cluster

Per chain:
  1 × consensus client (PoS chains)
  1–2 × write nodes (pipeline's etcd election decides which one publishes to Kafka)
  1 × consistency-checker
  N × leafage-evm (mainly State, add Archive as needed)

Shared globally:
  1 group of nodex-proxy (multiple instances, managing multiple chains by chainId)
  Kafka / S3 / etcd
  Prometheus + Grafana

You can deploy two write nodes: both upload to S3, and only the Leader publishes to Kafka. The standby takes over when the primary fails, avoiding a data path outage.

Common operations

Add a query node

Download the RocksDB snapshot from S3 and extract it to --db-path
Start leafage-evm; it automatically catches up to the chain head from the Kafka offset or S3
It registers itself in etcd on startup; once consistency-checker confirms it has caught up, the checker sets stateType to 1
nodex-proxy picks up the change through its watch and, once the health check passes, adds the node to the load balancing pool

The whole process usually completes within minutes, with no manual intervention on the proxy.

Take a query node offline

Just stop the process. leafage-evm deletes its own etcd key on exit; even if it exits abnormally, consistency-checker marks it offline and deletes it after its polling fails.

Switch the write node Leader

Through nodex-proxy’s admin API:

curl -X POST http://proxy:8663/1/writers/switchLeader -d '{...}'
curl http://proxy:8663/1/writers/leader

You can also modify {chainID}/writers/leader in etcd directly. After the switch, the new Leader waits for grace_period before it starts publishing.

Version switching

In version mode, modify {chainID}/version in etcd:

After consistency-checker detects the version change, it releases or acquires the singleton topic’s publish lock and aligns the topic to the new version’s progress
nodex-proxy rewrites requests for the base chain ID to the {chainId}-{version} node pool

Clients need no changes.

Rewind and resync

If leafage-evm’s state becomes abnormal, you can rewind the chain head to an earlier block and resync from S3:

leafage-evm rewind --db-path /nodex-eth --to-block 12345

Troubleshooting

Symptom	Check order
Query node not catching up with blocks	Is Kafka consumption normal → Are S3 fetches reporting errors (check the `updater` target with `RUST_LOG=info`) → Has the offset expired and triggered a full catch-up
External consumers receive no notifications	Is the checker’s `pipeline_replica_ready_timeouts_total` growing → Does the replica have a node that has lagged for a long time and lowered the ready ratio → In version mode, does it hold the singleton lock
proxy reports no available nodes	Does the node key exist in etcd → Is `stateType` 3 → Does the health check pass (inspect with `/{chainId}/getAllNodes`)
Historical queries fail	Is there an Archive node in the pool → Does the requested height fall below `--historical-height` without `--historical-rpc` configured
Duplicate-height objects on S3	Normal; reorgs leave fork blocks behind. Confirm that the checker marked the non-canonical ones with `is_fork: true` (see `pipeline_fork_scan_rewrites_total`)
Write node disk grows too fast	Is `--ancient.prune` enabled; does pipeline’s `s3_temp_dir` have leftover files that were not uploaded

References

leafage-evm deployment guide — includes one-click scripts, snapshot download, and directory layout
nodex-proxy deployment guide (Chinese) — Docker Compose, Kubernetes, systemd, and production tuning
consistency-checker deployment notes

Getting started

Core concepts

Architecture

Components

Guides

Reference

Infrastructure

Setup 1: single-machine validation

Setup 2: single-chain production

Resource planning (ETH mainnet)

Setup 3: multi-chain cluster

Common operations

Troubleshooting

References

​Infrastructure

​Setup 1: single-machine validation

​Setup 2: single-chain production

​Resource planning (ETH mainnet)

​Setup 3: multi-chain cluster

​Common operations

​Troubleshooting

​References

Infrastructure

Setup 1: single-machine validation

Setup 2: single-chain production

Resource planning (ETH mainnet)

Setup 3: multi-chain cluster

Common operations

Troubleshooting

References