---
title: "Observability"
description: "Lifecycle stages, the metrics that cover each, healthy ranges, and what a breach means."
audience: both
type: reference
status: live
lang: en
updated: "2026-09-04"
publish: true
---
# Observability

This page is the metric contract for a BTR deployment. For every stage between a price arriving at
NX Rates and an LP's realised return it names the few metrics that decide whether that stage is
working, where each is emitted, and what a breach means operationally. Read it to build alerting, to
triage an incident, or to check that a dashboard number means what its label claims. Units and
scales are stated with each metric, because most recurring mistakes here are unit errors.

## Scope

Four rules govern every entry below.

1. **A refusal is an event, on every surface that can emit one.** If an off-chain component declines to do something, that increments a counter; silence means "nothing was asked", never "everything was refused". The on-chain contracts are the exception and the reason stage 6 exists: a revert emits nothing, so the refusal has to be mirrored off chain to be visible at all.
2. **Units belong in the definition.** Every rate below states its scale. PBPS is $10^6$, bps is $10^4$, and the fee is half the round-trip spread.
3. **Additive series carry an integrity signal.** Any metric that is a sum over blocks publishes whether its input range was ingested once.
4. **Price-source pipelines are scoped.** Stages marked **(NX Rates)** describe the NX Rates aggregation-and-push pipeline. They apply only to deployments sourcing marks from NX Rates; a deployment using on-chain or INTERNAL pricing has stages 6 onward and none of the pipeline stages.

## Lifecycle stages

| # | Stage | Decisive metrics | Emitted at | Healthy | Breach means |
|---|---|---|---|---|---|
| 1 | Price ingestion (NX Rates) | `nxr_provider_ticks_total{exchange}`, `nxr_provider_last_ticker_age_ms{exchange}`, `nxr_provider_ticks_rejected_total{exchange,reason}` | aggregator health surface | age < 30 s per live venue | A venue is dark; the composite silently reweights onto the survivors |
| 2 | Composite index (TDWAP) | `nxr_tickers_provider_stale`, `nxr_tickers_provider_dead`, `nxr_emit_rejected_total{reason}` | aggregator health surface | stale 0, dead 0 | The index is quoting off decayed legs |
| 3 | Bar continuity | `nxr_s10_flat_fill_total{producer}`, `nxr_s10_hole_total{producer}`, `nxr_bars_written_total{producer}` | aggregator health surface | flat_fill / bars_written near 0 | Sustained above ~0.5 the feed is not arriving and the chart is fabricated continuity |
| 4 | Quote signing and cosign quorum | `nxr_signed_sigma_cache_ready` | aggregator health surface | sigma cache ready 1 | Quorum cannot be met; the keeper has nothing valid to push |
| 5 | Oracle relay push | `/latency` p99 by leg, `/triggers` records | keeper health surface | detect-to-broadcast p99 < 200 ms | Pushes are late, so the mark is stale before it lands |
| 6 | Pool mark state | `feeds.tradableRatio`, `feeds.byGate{}`, `feeds.sigmaZero`, `feeds.worstAgeRatio` | indexer health surface | ratio 1.0, sigmaZero 0, worstAgeRatio < 0.5 | The fraction of the book that will revert on the next swap |
| 7 | Swap execution | `bot_tradable_ratio`, `bot_blocked_legs_total{reason}`, `bot_gate_legs_blocked` | bot health surface | ratio 1.0, blocked flat | Legs are being refused |
| 8 | Swap pricing | `mm.spread.avg` (bps), `feeAvgBps`, `oev_bps` | Collector timeseries and summary | realized fee bps at or above `minFeePbps / 100` (there is no max) | Realized fee below the floor, i.e. mispriced risk |
| 9 | Fee capture and LP return | `fee.lp.usd`, `fee.proto.usd`, `lp_return`, `apr.fee` | internal health surface | fee APR positive and stable | LPs are not being paid for the flow they absorbed |
| 10 | Liquidity provision and withdrawal | `liq.dep.usd`, `liq.wd.usd`, `liq.net.usd` | indexer health surface | net flow explicable by APR | Silent exit ahead of a problem the other metrics have not surfaced |
| 11 | Coverage | `cov.c`, `util.liq` | Collector timeseries | $c$ near 1.0 | $c < 1$ means withdrawals are haircut; the toll is being charged |
| 12 | Risk parameter retuning | Guardian breach predicates | Keeper alert channel | no standing Critical | A deployed parameter contradicts the live tape |
| 13 | Keeper fleet health | `/gas` `spend_eth`, `deferred_pushes`, `gate`, `ratio` | keeper health surface | gate open, deferrals bounded | The fleet is priced out of the chain, or is burning budget |
| 14 | Indexer integrity | `ingest.cursorLead`, `ingest.cursorBehindTip`, `ingest.lastEventAgeSec` | indexer health surface | cursorLead $\ge 0$, lastEventAgeSec bounded | Negative lead means the next scan double counts every additive series |

## Stage detail

### 6: pool mark state, and the gate mirror

A swap's first gate is `FeedMathLib.gate()`. It reverts on four conditions, checked in this order: feed paused, stale, mark zero, confidence over the halt threshold ([Oracles §8.2](/docs/3-4-oracles#82-staleness-protection)). A revert emits nothing, and a leg refused at quote time never reaches a chain, so there is no on-chain trace whatsoever, which is the whole reason for the mirror below.

The collector evaluates the same predicate off chain against the feed state it multicalls, and publishes the ratio at `GET /protocol/health`:

```json
{
  "feeds": {
    "total": 0, "tradable": 0, "tradableRatio": 1.0,
    "byGate": { "ok": 0, "stale": 0, "paused": 0, "dead": 0, "uncertain": 0, "unreadable": 0 },
    "sigmaZero": 0, "aging": 0, "worstAgeRatio": 0.0, "worstFeed": "",
    "blocked": [
      { "name": "", "symbol": "", "gate": "", "ageSec": 0, "ttl": 0,
        "ageRatio": 0.0, "sigma": 0, "confidence": 0, "maxDeviationBps": 0 }
    ]
  }
}
```

- `tradableRatio` is the one number that answers "is the protocol open for business". Healthy is 1.0. Alert below 1.0 sustained, page below 0.5.
- `byGate` gives the reason without reading the roster. `paused` is an operator action, `stale` is a keeper failure, `unreadable` is a missing feed. Each demands a different response.
- `sigmaZero` counts feeds that are otherwise healthy but publish $\sigma = 0$. The dynamic fee then collapses to `minFee` while the mark keeps moving, which is free adverse selection. It is invisible in any freshness view, so it gets its own counter. Healthy is 0.
- `worstAgeRatio` is $a/\tau$. Above 0.5 is one missed heartbeat, since the ops rule is $\tau \approx 2 \times$ heartbeat. Above 1.0 the leg is dead.
- `blocked` is sorted worst first, so an operator reads the top of the list rather than all of it.

Two correctness notes on the mirror:

- Age is computed on the **observed-at clock** the contract gates on, $t_{obs} = \min(t_{src}, t_{upd})$, not on $t_{upd}$ alone. $t_{upd}$ is the relay landing time, so using it understates age by the relay lag and can report a feed fresh that the pool reverts on.
- `confidence`, `flags`, `maxDeviation` and `sourceTs` are snapshotted and persisted. Without them the mirror sees only staleness, which is two of the four gate conditions.

`oracle_feed_state` carries `confidence`, `flags`, `max_deviation_bps`, `source_ts` and `gate`. Persisting the verdict means an outage keeps its reason instead of surviving only as a period when volume was zero.

The mirror is feed state, not pool state, so it does not see `BaseDepegged`, `PriceOutsideRefBand` or a halted asset flag.

### 7: swap execution

The flow bot computes the blocked set on a preflight sweep and publishes it:

- `bot_gate_legs` and `bot_gate_legs_blocked`, gauges from the sweep.
- `bot_tradable_ratio`, the fraction of legs that currently quote. `-1` before the first sweep, never a flattering 1.
- `bot_gate_sweep_timestamp_seconds`, so a stalled sweep is distinguishable from a healthy book.
- `bot_blocked_legs_total{reason}`, a counter keyed by gate reason class.

Two design points:

- The sweep is recorded **before** the early return that fires when fewer than two legs are executable. Recording after it would skip exactly the fully-gated case, which is the one that matters.
- Reasons are collapsed to a bounded label class. The raw reason embeds live numbers (`stale(age=421s>ttl=600s)`), which is unbounded label cardinality. The class is what an operator alerts on; the numbers stay in the JSONL and the pod log.

### 8: swap pricing and units

`Swapped.spreadPbps` is PBPS, base $10^6$, despite being stored in a column named `spread_bps`. Two different divisors come off that one word. With $S$ the spread in PBPS:

$$f_{bps} = \frac{S}{200}, \qquad S_{bps} = \frac{S}{100}, \qquad \texttt{minFee}_{bps} = \frac{\texttt{minFeePbps}}{100}$$

| Quantity | From | Divisor | Scale |
|---|---|---|---|
| Realized fee | `spread_pbps` | 200 | bps |
| Spread | `spread_pbps` | 100 | bps |
| `minFeePbps` | `getAsset` | 100 | bps |

The 200 is two conversions at once: 100 for PBPS to bps, and 2 because the fee is half the round-trip spread. A fee rate that is already a rate, such as `minFeePbps`, uses 100 and not 200, so a comparison of a realized fee against the floor must divide the floor by 100 first. The constants are named `SPREAD_PBPS_PER_FEE_BP` and `SPREAD_PBPS_PER_BP` in one place, so no reader can silently re-derive the wrong one.

`minFeePbps` is the **only** fee rate `getAsset` returns, and it is a two-sided-bounded per-leg **floor** on the spread, not a target and not a ceiling. Everything else `getAsset` returns is inventory (`reserves`, `liabilities`, `anchor`, `liquidityIndexWad`, `minLiquidity`), shape (`presetId`, the dispersion band), or exit/decimal metadata (`vegaBps`, `haircutSuppressorBps`, `decimals`, `deadSeedPow10`). A dashboard therefore cannot plot a per-leg fee band from config and must not present one: the quoted spread has no upper bound by design, because a $\sigma$, confidence or staleness term that drives it wide is pricing real risk.

Snapshot writers resolve ABI fields by name, never by position: `minFeeBps` and `minFeePbps` differ by one letter and by a factor of 100.

`fee_bps_realized` and `oev_bps` in `dex_swap_fees` are computed from amounts rather than from the spread word.

### 9 and 10: fees, LP return, liquidity flow

`dex_swap_fees` stores both legs' fees, realized fee bps, execution price, LP return and the liquidity it was earned against, one row per `Swapped` log, deduplicated on `swap_id`.

The asset is the primitive: asset APR is absolute fees over that asset's liquidity. Pool APR is the mean of its assets'.

Two code paths compute it and only one carries the elapsed-span guard, so scope matters.

- The **ingest loop** annualizes over the real elapsed span $t_{now} - t_{then}$ and returns null below `MIN_APR_WINDOW_MS` = 1 h, so the loop-published metrics never annualize noise.
- The **served `/apr` endpoint** calls `buildFeeApr(qdb, registry, windowMs)`, which annualizes over the *requested* window (default 7 days) with no elapsed-span check and no null gate.

During cold start or after an ingest gap, `/apr` returns a small non-null number that reads as real yield and is not.

Volume counts hop 0 only, so a multi-pool route is one user swap. Fees count every hop, because each transited pool really charged and really earned. Same event stream, deliberately different reduction, and `hop_index` makes both provable.

`Swapped` carries `markPrice`, `midPrice` and `covToll`, so total LP revenue is reconstructible from logs alone. All three are `tokenOut` per `tokenIn`, chained over every leg, so they are directly comparable to the realized $\texttt{amountOut}/\texttt{amountIn}$. OEV decomposes exactly on chain: with $p_x$ the execution price, $p_m$ the mid and $p_o$ the oracle mark,

$$\text{OEV} = \frac{p_x - p_m}{p_m}, \qquad \text{skew} = \frac{p_m - p_o}{p_o}$$

The two prices are exact WAD, never a packed float: a value that goes straight into a log is never packed, because ABI padding takes every non-indexed arg to a full word and the encode would save no bytes while costing gas and mantissa precision. The reconstruction therefore inherits no encoding error.

`covToll` is withheld from the output before the fee and retained in the reserve, so a reader that ignores it books it as coverage drift rather than as revenue.

### 11: coverage and haircuts

`cov.c` and `util.liq` are served per asset.

`applyHaircut` fires on every withdrawal path when reserves < liabilities, but only `LiabilitySwapped` carries a `haircut` field. A same-asset withdrawal emits `Withdrawn`, which has no haircut field, so that haircut is not visible on chain. `previewWithdraw` is the off-chain read that recovers it.

### 12: risk parameter retuning

The guardian evaluates the predicates below with severities and pages the on-call operator, with a six-hour re-page cooldown per breach and key. The transport is an operational detail and is not published.

Predicates:

- `min_fee_below_two_theta` (Critical)
- `feed_sigma_zero` (Critical)
- `mark_expired` (Critical)
- `dev_cap_below_theta` (Critical)
- `band_below_floor` (High)
- `feed_unreadable` (High)
- `dev_cap_below_drift` (High)
- `mark_aging` (Medium)
- `measurement_missing` (Medium)
- `measurement_stale` (Medium)

The collector subscribes to swap, liquidity and oracle-signer-governance topics. The parameter and halt events (`AssetParamsUpdated`, `RiskFencesUpdated`, `BoundedAssetParamsUpdated`, `CurveUpdated`, `FeeParamsUpdated`, `AssetHalted`, `AssetUnhalted`, and the oracle's `FeedPaused` / `FeedUnpaused` / `FeedUpdated` / `FeedRegistered` / `FeedExpBiasUpdated` set) are snapshot triggers rather than indexed topics, so retuning and halting are not queryable from the indexer. The next release adds `FeedWidenRequested` / `FeedWidenExecuted` / `FeedWidenCancelled` to that set; `FeedWidenExecuted` carries both before/after pairs, and `oldBand == newBand` marks a pure wedge release rather than a config loosening ([Oracles §8.3](/docs/3-4-oracles#83-deviation-bounds)).

The V4 **push** path is indexable and is not on that list: `SlotsPushed(seq, sourceTsDs, acceptedMask, blobHash)` on both entry points, plus `LanesSkipped(slotId, laneMask)` whenever a lane fail-softs. One V4 outcome has no event at all — a whole slot stepped over for failing the source-second monotonicity check — so slot age has to be polled, not subscribed.

### 14: indexer integrity

Every additive series is a sum over blocks, so a replayed range doubles it. The ingest loop guards this at startup by bumping the cursor past the indexed tip, and publishes the guard in `/protocol/health`. With $c$ the cursor and $T$ the indexed tip:

$$\ell = c - 1 - T \ge 0$$

- `ingest.cursorLead` is $\ell$. **Negative means the next scan replays indexed blocks and double counts every additive series.**
- `ingest.cursorBehindTip`, the boolean form, for alerting.
- `ingest.lastEventAgeSec`, the age of the newest indexed event. `/health` on the collector is static and returns ok even with a dead indexer, which is also what the container health check probes. This is the number that detects it.

Reorg handling is a five-block confirmation depth and nothing more: no block-hash tracking and no rollback path. A reorg deeper than five blocks corrupts additive series with no signal. The depth is a single fleet-wide constant, not a per-chain setting, so a chain with weaker finality than the current target needs it re-derived before that chain is added.

## Where each surface lives

The health and metrics surfaces described on this page are **operator-internal**. Their addresses,
routes and transports are not published, are not reachable from the public internet, and are not
part of any integration contract: nothing an integrator builds should depend on them, and nothing
here is a supported API.

What is public is the on-chain state itself and the read-only gateway documented in
[API & SDK Reference](/docs/5-2-1-api-sdk-reference). Every claim on this page can be verified from
those two, which is the level at which the protocol intends to be checkable.

**Who runs the nodes.** The mark pipeline and the relays that push to it are a distributed network,
and the first nodes are operated by the BTR team, with independent operators onboarded as the
signer and relay sets widen. That is the same shape the comparable systems document: Hyperliquid's
oracle prices are published by its validator set as a stake-weighted aggregate of major-venue
prices, and GMX's price updates are submitted by authorized keeper accounts against a decentralized
oracle network. In all three cases the security argument rests on what the chain verifies, not on
who happens to be running a node: for BTR that is the k-of-n signature over each mark
([3.6](/docs/3-6-oracle-price-push-security)), the staleness and deviation gates that reject a bad
push regardless of its source ([3.4 §8](/docs/3-4-oracles)), and a relay path that carries no
signing key and therefore cannot forge, only delay.

## NX Rates pipeline observability

> Applies ONLY to deployments that source marks from the NX Rates aggregator and relay them through the oracle keeper. Deployments using INTERNAL or on-chain price sources stop at the pool-mark stage above and have no pipeline surfaces.

### 1 to 3: price ingestion and bars

NX Rates exports a dense provider surface. Stage 3 covers continuity rather than coverage.

The s10 producer carries the last close forward when no tick arrives in a 10 s bucket, so at bar level an outage and a quiet market look alike. `mitch::Bar` carries `tick_count` and `FLAG_S10_FLAT_FILL`; two counters lift that into metric space.

- `nxr_s10_flat_fill_total{producer}`, incremented per emitted bar with `tick_count == 0`.
- `nxr_s10_hole_total{producer}`, for a bucket with neither a tick nor a last close. That case emits no bar at all, so it is invisible in the bar stream.

Read them as a ratio against `nxr_bars_written_total{producer}`. A ticker whose flat-fill share is sustained above ~0.5 is not quiet, it is dark.

`tick_count` on a real bar counts changed composites, not raw provider ticks, because of the delta gate in the producer. It is therefore a lower bound. Use it to prove liveness, never to measure venue tick rate: that is stage 1's job.

### 4: quote signing and cosign quorum

Signature gathering (NXR-internal) enforces k-of-n under a 40 ms deadline and fails closed. `nxr_signed_sigma_cache_ready` is the readiness signal; a quorum that starts failing otherwise presents downstream as stage 6 staleness.

### 5: relay push

`ExternalOracle` emits no event on push: observability is `getFeed()` state polling. That is a gas decision with a consequence. A 66-feed relay batch produces zero log entries, so acceptance and rejection are knowable only from the keeper's own view or by diffing polled state.

The keeper publishes `/triggers`, `/thresholds`, `/latency` and `/gas`. Latency is well covered. Outcomes are coarser: `TriggerRecord.pushed` is a boolean with no reason, and the failure paths (send error, submit timeout, revert, dropped past TTL, unstick exhausted) are log-only.

`/gas` `spend_eth` is credited only from confirmed successful receipts, so it is confirmed successful spend, not total spend: reverted pushes and unstick cancel transactions are not in it.

### 13: keeper fleet health

`/gas` publishes:

- The base-fee regime.
- The gate position with its hysteresis bounds.
- How long the gate has been shut.
- The longest deferral episode.
- Discretionary pushes withheld.
- Liveness pushes forced through a shut gate.
- Confirmed spend.

Healthy is `gate: open` with `deferred_pushes` growing only during genuine fee spikes. `forced_pushes` non-zero is the design working, not a fault: liveness pushes relay at any price.

## Related

- [Oracle Price-Push Security](/docs/3-6-oracle-price-push-security)
- [Oracles](/docs/3-4-oracles)
- [Access Control, Roles & Emergency Powers](/docs/3-1-access-control-roles-emergency-powers)
- [Spread and Fees](/docs/1-1-4-spread-fees)