> ## Documentation Index
> Fetch the complete documentation index at: https://docs.leafage.chaintable.com/llms.txt
> Use this file to discover all available pages before exploring further.

# nodex-proxy

> JSON-RPC gateway: service discovery, block-context routing, load balancing, and traffic governance

nodex-proxy converges a leafage-evm cluster into a single RPC endpoint. It decides by the request's block context whether to send it to a State, Archive, or Native node, and takes on traffic governance duties such as rate limiting, mirroring, and observability.

| Item              | Value                                                                                  |
| ----------------- | -------------------------------------------------------------------------------------- |
| Repository        | [Chaintable/nodex-proxy](https://github.com/Chaintable/nodex-proxy)                    |
| Language          | Go 1.22+                                                                               |
| License           | Apache-2.0                                                                             |
| Core dependencies | [Hertz](https://github.com/cloudwego/hertz), etcd client v3, OpenTelemetry, zap, sonic |

## Ports and paths

| Port   | Purpose                                                                  |
| ------ | ------------------------------------------------------------------------ |
| `8663` | JSON-RPC (`POST /:chainId`) and the admin API, sharing the same listener |
| `8664` | Prometheus metrics                                                       |

One process serves multiple chains at the same time; chains are distinguished by the `chainId` in the path. Hexadecimal IDs are normalized to decimal strings.

<Warning>
  The admin API shares the port with JSON-RPC and can change routing state. This port should not be exposed outside the internal network, or it needs to be placed behind an external authentication layer.
</Warning>

## Request lifecycle

```mermaid theme={null}
flowchart LR
    C["Client"] --> H["Hertz :8663"]
    H --> PRE["Pre-processing"]
    PRE --> SEL["Node selection"]
    SEL --> UP["Reverse proxy to upstream"]
    UP --> RETRY{"Error code<br/>needs retry?"}
    RETRY -->|"Yes"| SEL
    RETRY -->|"No"| POST["Post-processing"]
    POST --> C
```

Pre-processing runs in a fixed order:

```text theme={null}
Record request → method deny list → method name validation → metrics → per-method rate limiting → method-specific rewrites → request mirroring
```

Post-processing:

```text theme={null}
Parse response → record response → method-specific post-processing → slow request / error log → metrics
```

## Node selection

First decide the node pool by block context, then pick a specific node within the pool by strategy.

| Request context                                            | Node pool |
| ---------------------------------------------------------- | --------- |
| `latest` / `pending`                                       | State     |
| Explicit height, within 64 blocks of the known chain head  | State     |
| Explicit height, more than 64 blocks behind the chain head | Archive   |
| `Contains` type context                                    | State     |
| Explicit archive flag (request header)                     | Archive   |
| Native retry                                               | Native    |

The chain head height comes from `{chainId}/lastBlockNumber` in etcd, maintained by consistency-checker. When the chain height is unknown, it conservatively selects an Archive node.

The two pools fall back to each other: when the State pool is empty, the Archive pool is used, and vice versa. Within each pool, `Available` nodes are used first.

### Load balancing strategies

<Tabs>
  <Tab title="random (default)">
    Selects by weighted probability. Non-batch requests first filter candidate nodes by method routing rules, then select by weight. Nodes without an explicit weight default to `100`.
  </Tab>

  <Tab title="round_robin">
    Round-robin over the candidate pool. Gateway weights and method routing are not applied.
  </Tab>
</Tabs>

### Automatic retries

| Error code                    | Condition                                    | Retry target                                |
| ----------------------------- | -------------------------------------------- | ------------------------------------------- |
| `-39006` `StateBlockNotFound` | The first request was not an archive request | Archive node                                |
| `-39008` `CosmosPrecompile`   | —                                            | Native node, upstream path rewritten to `/` |

## Service discovery

Watches an etcd prefix (`etcd_prefix` is configurable); the following key suffixes are interpreted as runtime data:

| Key suffix                            | Purpose                                                          |
| ------------------------------------- | ---------------------------------------------------------------- |
| `{chainId}/nodes/{nodeKey}`           | State / Archive nodes                                            |
| `{chainId}/nativeNodes/{nodeKey}`     | Native fallback nodes                                            |
| `{chainId}/lastBlockNumber`           | Current chain head height                                        |
| `{chainId}/gateway`                   | Node weights and method routing                                  |
| `{chainId}/mirror/{addrKey}`          | Mirror targets and optional rate limits                          |
| `{chainId}/version`                   | Overrides base chain routing to a versioned chain ID             |
| `{chainId}/{version}/nodes/{nodeKey}` | Versioned nodes, internal ID normalized to `{chainId}-{version}` |

The proxy watches `PUT` / `DELETE` events to add and remove nodes and update configuration in real time, with no restart needed.

### Health check

A node that newly appears in etcd does not join the node pool immediately:

<Steps>
  <Step title="Probe">
    Send an RPC call to the node to verify reachability. State / Archive nodes use `getLatestBlock`, Native nodes use `eth_blockNumber`.
  </Step>

  <Step title="Retry">
    On failure, retry every 5 seconds until `node_health_check_max_wait` (default 300 seconds).
  </Step>

  <Step title="Join">
    Only after passing the check does the node enter the load balancing pool.
  </Step>
</Steps>

### Version routing

When a request uses the base chain ID with no explicit version suffix, the proxy reads `{chainId}/version` from etcd and rewrites the request to the `{chainId}-{version}` node pool. This makes version switching transparent to clients.

## Traffic governance

| Capability               | Config option                      | Description                                                                  |
| ------------------------ | ---------------------------------- | ---------------------------------------------------------------------------- |
| Method deny list         | `method_denied`                    | List of methods rejected outright; includes `txpool_*` and others by default |
| Method name validation   | `method_name_checker`              | Validates the method name format with a regex                                |
| Per-method rate limiting | `rate_limiter.rpc_methods`         | Token bucket, RPS configured per method                                      |
| Block range limit        | `block_range_query_limit`          | Limits block range queries; can rewrite to `latest`                          |
| Request mirroring        | `request_mirror`                   | Asynchronously copies to a shadow backend without affecting the main request |
| Slow request log         | `observability_log.slow_threshold` | Threshold configured per method                                              |

## Usage reporting

Once `usage` is configured, RPC time is aggregated locally by the `client-id` request header and written to Kafka when 10000 aggregation keys or `report_interval` (default 5s) is reached.

```json theme={null}
{
  "id": "3c9d1b7e-52aa-4f0e-8d21-77b4e0c9a1f2",
  "client_id": "instance:019f45e26c307c86bd45ab350bb52ca8",
  "service": "leafage",
  "resource_type": "read",
  "usage": 123,
  "timestamp": 1783568373000
}
```

* `usage` is the aggregated time in milliseconds, with a minimum of 1.
* A missing `client-id` is recorded as `unknown`.
* Sending is best-effort: the last batch is sent on graceful shutdown; loss is acceptable when the process crashes or Kafka fails.
* The current number of aggregation keys is reflected by `jrpcx_usage_aggregation_keys`.

## Admin API

| Endpoint group                                                                 | Purpose                                               |
| ------------------------------------------------------------------------------ | ----------------------------------------------------- |
| `/getChains`, `/:chainId/getAllNodes`, `/:chainId/debug_chooseOneNode`         | View chains, nodes, and selection behavior            |
| `/:chainId/addNode`, `/updateNode/:nodeKey`, `/deleteNode/:nodeKey`            | Node changes persisted to etcd                        |
| `/:chainId/addLocalNode`, `/deleteLocalNode/:nodeKey`                          | In-memory only, not written to etcd                   |
| `/:chainId/setWeight`, `/getWeight`, `/deleteWeight`                           | Weight management                                     |
| `/:chainId/addMethodRoute`, `/removeMethodRoute`, `/deleteMethodRoute/:method` | Per-method include / exclude routing                  |
| `/:chainId/addMirror`, `/deleteMirror`, `/deleteAllMirrors`                    | Mirror targets (persisted to etcd)                    |
| `/:chainId/writers`, `/writers/leader`, `/writers/switchLeader`                | View write nodes, view and switch the pipeline Leader |

`debug_chooseOneNode` is useful when troubleshooting routing problems: it returns the node that would be selected under the current request conditions, without actually forwarding.

## Metrics

Exposed at `:8664/metrics`, with common labels `host`, `target`, `chain_id`, `chain_version`.

| Metric                                                       | Type                |
| ------------------------------------------------------------ | ------------------- |
| `jrpcx_rpc_calls_started` / `finished` / `failed`            | Counter             |
| `jrpcx_rpc_calls_time`                                       | Histogram           |
| `jrpcx_rpc_batch_calls_finished` / `_time`                   | Counter / Histogram |
| `jrpcx_rpc_request_payload_sizes` / `response_payload_sizes` | Histogram           |
| `jrpcx_rpc_http_status_code`                                 | Counter             |
| `jrpcx_rpc_calls_cache_hits`                                 | Counter             |

Some metrics carry extra labels:

| Metric                    | Extra labels                                |
| ------------------------- | ------------------------------------------- |
| Per-method metrics        | `method`                                    |
| `jrpcx_rpc_calls_started` | `sourcedapp`                                |
| Failure metrics           | `status_code`, `upstream_related`, `reason` |

## Running

```bash theme={null}
go build -o node-proxy cmd/proxy/main.go
./node-proxy -config config/config.example.yaml -listen 8663
```

Multiple instances can connect to the same etcd cluster to scale horizontally, each maintaining its own local selector state.

## Development

```bash theme={null}
make build
make test
make lint
make ci
```

## Related documents

* [`docs/architecture_cn.md`](https://github.com/Chaintable/nodex-proxy/blob/main/docs/architecture_cn.md) (Chinese) — Component boundaries and runtime architecture
* [`docs/deployment_cn.md`](https://github.com/Chaintable/nodex-proxy/blob/main/docs/deployment_cn.md) (Chinese) — Docker Compose, Kubernetes, systemd, and production tuning