---
title: "Observability"
description: "What BTR emits, which surface carries it, and the conditions that page an operator."
audience: both
type: reference
status: live
lang: en
updated: "2026-09-17"
publish: true
aliases: [3-8-observability]
---
# Observability

The metric contract for a BTR deployment: what is emitted, where to read it, and what pages someone.

Four rules govern every entry below.

1. **A refusal is an event** on every surface that can emit one. An off-chain component that declines to do something increments a counter; silence means "nothing was asked", never "everything was refused". The contracts are the exception: a revert emits nothing, so on-chain refusals have to be mirrored off chain to be visible at all.
2. **Units belong in the definition.** PBPS is $10^6$, bps is $10^4$, and the fee is half the round-trip spread. Most recurring mistakes here are unit errors.
3. **Additive series carry an integrity signal.** Any metric that is a sum over blocks publishes whether its input range was ingested once.
4. **Price-source pipelines are scoped.** Rows marked **(NX Rates)** apply only to deployments sourcing marks from the NX Rates aggregator and relaying them through the [oracle keeper](/docs/3-1-5-oracle-keeper). A deployment using INTERNAL or on-chain pricing starts at pool mark state and has no pipeline surfaces.

## 1. What is emitted

### 1.1 Off-chain metrics

| Stage | Decisive metrics | Healthy |
|---|---|---|
| Price ingestion (NX Rates) | `nxr_provider_ticks_total{exchange}`, `nxr_provider_last_ticker_age_ms{exchange}`, `nxr_provider_ticks_rejected_total{exchange,reason}` | age < 30 s per live venue |
| Composite index, TDWAP (NX Rates) | `nxr_tickers_provider_stale`, `nxr_tickers_provider_dead`, `nxr_emit_rejected_total{reason}` | stale 0, dead 0 |
| Bar continuity (NX Rates) | `nxr_s10_flat_fill_total{producer}`, `nxr_s10_hole_total{producer}`, `nxr_bars_written_total{producer}` | flat_fill / bars_written near 0 |
| Quote signing and cosign quorum (NX Rates) | `nxr_signed_sigma_cache_ready` | ready 1 |
| Oracle relay push (NX Rates) | `/latency` p99 by leg, `/triggers` records | detect-to-broadcast p99 < 200 ms |
| Pool mark state | `feeds.tradableRatio`, `feeds.byGate{}`, `feeds.sigmaZero`, `feeds.worstAgeRatio` | ratio 1.0, sigmaZero 0, worstAgeRatio < 0.5 |
| Swap execution | `bot_tradable_ratio`, `bot_blocked_legs_total{reason}`, `bot_gate_legs`, `bot_gate_legs_blocked`, `bot_gate_sweep_timestamp_seconds` | ratio 1.0, blocked flat |
| Swap pricing | `mm.spread.avg` (bps), `feeAvgBps`, `oev_bps` | realized fee bps at or above `minFeePbps / 100`; there is no max |
| Fee capture and LP return | `fee.lp.usd`, `fee.proto.usd`, `lp_return`, `apr.fee` | fee APR positive and stable |
| Liquidity provision and withdrawal | `liq.dep.usd`, `liq.wd.usd`, `liq.net.usd` | net flow explicable by APR |
| Coverage | `cov.c`, `util.liq` | $c$ near 1.0 |
| Risk parameter retuning | guardian breach predicates ([§3.1](#31-guardian-predicates)) | no standing Critical |
| Keeper fleet health | `/gas` `spend_native`, `deferred_pushes`, `gate`, `ratio` | gate open, deferrals bounded |
| Indexer integrity | `ingest.cursorLead`, `ingest.cursorBehindTip`, `ingest.lastEventAgeSec` | cursorLead $\ge 0$, lastEventAgeSec bounded |

Notes that change how these read:

- **Bar continuity.** The s10 producer carries the last close forward when no tick arrives in a 10 s bucket, so at bar level an outage and a quiet market look alike. `mitch::Bar` carries `tick_count` and `FLAG_S10_FLAT_FILL`; `nxr_s10_flat_fill_total` increments per emitted bar with `tick_count == 0`, and `nxr_s10_hole_total` counts a bucket with neither a tick nor a last close (no bar is emitted at all, so it is invisible in the bar stream). Read both as a ratio against `nxr_bars_written_total`. `tick_count` counts changed composites, not raw provider ticks, because of the producer's delta gate, so it is a lower bound: use it to prove liveness, never to measure venue tick rate.
- **Cosign quorum.** Signature gathering (NXR-internal) enforces k-of-n under a 40 ms deadline and fails closed. A failing quorum presents downstream as pool-mark staleness, not as its own signal.
- **Swap execution.** `bot_tradable_ratio` is `-1` before the first sweep, never a flattering 1. The sweep is recorded **before** the early return that fires below two executable legs, because that is exactly the fully-gated case that matters. `bot_blocked_legs_total{reason}` collapses reasons to a bounded label class: the raw reason embeds live numbers (`stale(age=421s>ttl=600s)`), i.e. unbounded label cardinality. The class is what an operator alerts on; the numbers stay in the JSONL and the pod log.
- **Keeper fleet.** `/gas` publishes the base-fee regime, the gate position with its hysteresis bounds, how long the gate has been shut, the longest deferral episode, discretionary pushes withheld, liveness pushes forced through a shut gate, and confirmed spend. `spend_native` is credited only from confirmed successful receipts: reverted pushes and unstick cancel transactions are not in it. `forced_pushes` non-zero is the design working, not a fault: liveness pushes relay at any price.

### 1.2 On-chain events

| Class | Events | Indexable |
|---|---|---|
| Swap | `Swapped` (`spreadPbps`, `markPrice`, `midPrice`, `covToll`) | yes |
| Liquidity and solvency | `SolvencyUpdated(cWad, navBase, claimBase)`, `LiabilitySwapped`, `Withdrawn` | yes |
| Oracle push (V4) | `SlotsPushed(seq, sourceTsDs, acceptedMask, blobHash)` on both entry points, `LanesSkipped(slotId, laneMask)` | yes |
| Oracle signer governance | signer-governance topics | yes |
| Params and halts | `AssetParamsUpdated`, `RiskFencesUpdated`, `BoundedAssetParamsUpdated`, `CurveUpdated`, `FeeParamsUpdated`, `AssetHalted`, `AssetUnhalted` | no, snapshot trigger only |
| Feed lifecycle | `FeedPaused`, `FeedUnpaused`, `FeedUpdated`, `FeedRegistered`, `FeedExpBiasUpdated` | no, snapshot trigger only |
| Deviation-band widening | `FeedWidenRequested`, `FeedWidenExecuted`, `FeedWidenCancelled` | no, snapshot trigger only |

- Retuning and halting are **not queryable from the indexer**: they are snapshot triggers, not indexed topics.
- `FeedWidenExecuted` carries both before/after pairs; `oldBand == newBand` marks a pure wedge release rather than a config loosening ([oracle keeper](/docs/3-1-5-oracle-keeper)).
- Two outcomes emit nothing. `ExternalOracle` emits no event on push (a gas decision: a 66-feed relay batch produces zero log entries, so acceptance and rejection are knowable only from the keeper's own view or by diffing polled `getFeed()` state), and a V4 slot stepped over for failing the source-second monotonicity check has no event, so slot age must be polled, not subscribed.
- `Swapped` carries `markPrice`, `midPrice` and `covToll`, so total LP revenue is reconstructible from logs alone. All three are `tokenOut` per `tokenIn`, chained over every leg, hence directly comparable to the realized $\texttt{amountOut}/\texttt{amountIn}$. OEV decomposes exactly on chain: with $p_x$ the execution price, $p_m$ the mid and $p_o$ the oracle mark,

$$\text{OEV} = \frac{p_x - p_m}{p_m}, \qquad \text{skew} = \frac{p_m - p_o}{p_o}$$

  The two prices are exact WAD, never a packed float: ABI padding takes every non-indexed arg to a full word, so packing would save no bytes while costing gas and mantissa precision. The reconstruction therefore inherits no encoding error. `covToll` is withheld from the output before the fee and retained in the reserve, so a reader that ignores it books it as coverage drift rather than revenue.
- Only `LiabilitySwapped` carries a `haircut` field (`face - face · C` when positive). A same-asset `Withdrawn` carries none; its shortfall below face is `face · (1 - min(c_leg, C))`, recoverable from `previewWithdraw` or from the paired `SolvencyUpdated` and the leg's books. `SolvencyUpdated` fires on every LP entrypoint of an armed pool; `Pool.poolSolvencyWad()` reads the pool rate live. `cov.c` and `util.liq` are served per asset.

### 1.3 Units and scales

`Swapped.spreadPbps` is PBPS, base $10^6$, despite being stored in a column named `spread_bps`. Two different divisors come off that one word. With $S$ the spread in PBPS:

$$f_{bps} = \frac{S}{200}, \qquad S_{bps} = \frac{S}{100}, \qquad \texttt{minFee}_{bps} = \frac{\texttt{minFeePbps}}{100}$$

| Quantity | From | Divisor | Scale |
|---|---|---|---|
| Realized fee | `spread_pbps` | 200 | bps |
| Spread | `spread_pbps` | 100 | bps |
| `minFeePbps` | `getAsset` | 100 | bps |

The 200 is two conversions at once: 100 for PBPS to bps, and 2 because the fee is half the round-trip spread. A rate that is already a rate, such as `minFeePbps`, uses 100 and not 200, so comparing a realized fee against the floor means dividing the floor by 100 first. The constants are named `SPREAD_PBPS_PER_FEE_BP` and `SPREAD_PBPS_PER_BP` in one place, so no reader can silently re-derive the wrong one. Snapshot writers resolve ABI fields by name, never by position: `minFeeBps` and `minFeePbps` differ by one letter and by a factor of 100. `fee_bps_realized` and `oev_bps` in `dex_swap_fees` are computed from amounts, not from the spread word.

`minFeePbps` is the **only** fee rate `getAsset` returns, and it is a two-sided-bounded per-leg **floor** on the spread, not a target and not a ceiling. Everything else `getAsset` returns is inventory (`reserves`, `liabilities`, `anchor`, `liquidityIndexWad`, `minLiquidity`), shape (`presetId`, the dispersion band), or credit-cap and decimal metadata (`vegaBps`, `depositCapCode`, `maxLiabWeightBps`, `decimals`, `deadSeedPow10`). A dashboard therefore cannot plot a per-leg fee band from config and must not present one: the quoted spread has no upper bound by design, because a $\sigma$, confidence or staleness term that drives it wide is pricing real risk. See [Spread and Fees](/docs/1-1-4-spread-fees).

## 2. Where to read it

| Surface | Carries | Public |
|---|---|---|
| Contract events | swap, solvency, oracle push, signer governance ([§1.2](#12-on-chain-events)) | yes, on chain |
| Read-only gateway | [API & SDK Reference](/docs/5-2-1-api-sdk-reference) | yes |
| Aggregator health surface (NX Rates) | ingestion, composite index, bar continuity, cosign readiness | no |
| Keeper health surface | `/triggers`, `/thresholds`, `/latency`, `/gas`; relay push and fleet health | no |
| Indexer health surface | `GET /protocol/health`: pool mark state, liquidity flow, ingest integrity | no |
| Collector timeseries and summary | swap pricing, coverage, `dex_swap_fees`, `oracle_feed_state` | no |
| Bot health surface | preflight gate sweep, blocked legs | no |
| Internal health surface | fee capture, LP return, fee APR | no |
| Keeper alert channel | guardian breach predicates ([§3.1](#31-guardian-predicates)) | no |

Every health and metrics surface above is **operator-internal**. Their addresses, routes and transports are not published, are not reachable from the public internet, and are not part of any integration contract: nothing an integrator builds should depend on them, and nothing here is a supported API. Public instead is the on-chain state itself plus the read-only gateway, and every claim on this page is verifiable from those two.

### 2.1 The gate mirror

A swap's first gate is `FeedMathLib.gate()`. It reverts on four conditions, checked in this order: feed paused, stale, mark zero, confidence over the halt threshold ([oracle keeper](/docs/3-1-5-oracle-keeper)). A revert emits nothing, and a leg refused at quote time never reaches a chain, so the off-chain mirror is the only trace. The collector evaluates the same predicate off chain against the feed state it multicalls, and publishes it at `GET /protocol/health`:

```json
{
  "feeds": {
    "total": 0, "tradable": 0, "tradableRatio": 1.0,
    "byGate": { "ok": 0, "stale": 0, "paused": 0, "dead": 0, "uncertain": 0, "unreadable": 0 },
    "sigmaZero": 0, "aging": 0, "worstAgeRatio": 0.0, "worstFeed": "",
    "blocked": [
      { "name": "", "symbol": "", "gate": "", "ageSec": 0, "ttl": 0,
        "ageRatio": 0.0, "sigma": 0, "confidence": 0, "maxDeviationBps": 0 }
    ]
  }
}
```

- `tradableRatio` answers "is the protocol open for business". Healthy is 1.0.
- `byGate` gives the reason without reading the roster. `paused` is an operator action, `stale` is a keeper failure, `unreadable` is a missing feed. Each demands a different response.
- `sigmaZero` counts feeds that are otherwise healthy but publish $\sigma = 0$. The dynamic fee then collapses to `minFee` while the mark keeps moving, which is free adverse selection. It is invisible in any freshness view, so it gets its own counter. Healthy is 0.
- `worstAgeRatio` is $a/\tau$. Above 0.5 is one missed heartbeat, since the ops rule is $\tau \approx 2 \times$ heartbeat. Above 1.0 the leg is dead.
- `blocked` is sorted worst first, so an operator reads the top of the list rather than all of it.

Two correctness notes. Age is computed on the **observed-at clock** the contract gates on, $t_{obs} = \min(t_{src}, t_{upd})$, not on $t_{upd}$ alone: $t_{upd}$ is the relay landing time, so using it understates age by the relay lag and can report a feed fresh that the pool reverts on. And `confidence`, `flags`, `maxDeviation` and `sourceTs` are snapshotted and persisted; without them the mirror sees only staleness, which is two of the four gate conditions. `oracle_feed_state` carries `confidence`, `flags`, `max_deviation_bps`, `source_ts` and `gate`, so an outage keeps its reason instead of surviving only as a period when volume was zero.

The mirror is feed state, not pool state, so it does not see `BaseDepegged`, `PriceOutsideRefBand` or a halted asset flag ([flow guards](/docs/3-2-1-flow-guards)).

### 2.2 Fees, LP return and liquidity flow

`dex_swap_fees` stores both legs' fees, realized fee bps, execution price, LP return and the liquidity it was earned against, one row per `Swapped` log, deduplicated on `swap_id`. The asset is the primitive: asset APR is absolute fees over that asset's liquidity, and pool APR is the mean of its assets'.

Two code paths compute it and only one carries the elapsed-span guard, so scope matters.

| Path | Window | Guard |
|---|---|---|
| Ingest loop | real elapsed span $t_{now} - t_{then}$ | returns null below `MIN_APR_WINDOW_MS` = 1 h, so loop-published metrics never annualize noise |
| Served `/apr`, `buildFeeApr(qdb, registry, windowMs)` | the *requested* window, default 7 days | none: no elapsed-span check, no null gate |

During cold start or after an ingest gap, `/apr` returns a small non-null number that reads as real yield and is not.

Volume counts hop 0 only, so a multi-pool route is one user swap. Fees count every hop, because each transited pool really charged and really earned. Same event stream, deliberately different reduction, and `hop_index` makes both provable.

### 2.3 Indexer integrity

Every additive series is a sum over blocks, so a replayed range doubles it. The ingest loop guards this at startup by bumping the cursor past the indexed tip and publishes the guard in `/protocol/health`. With $c$ the cursor and $T$ the indexed tip:

$$\ell = c - 1 - T \ge 0$$

- `ingest.cursorLead` is $\ell$. **Negative means the next scan replays indexed blocks and double counts every additive series.**
- `ingest.cursorBehindTip`, the boolean form, for alerting.
- `ingest.lastEventAgeSec`, the age of the newest indexed event. `/health` on the collector is static and returns ok even with a dead indexer, which is also what the container health check probes; this is the number that detects it.

Reorg handling is a five-block confirmation depth and nothing more: no block-hash tracking and no rollback path. A reorg deeper than five blocks corrupts additive series with no signal. The depth is a single fleet-wide constant, not a per-chain setting, so a chain with weaker finality than the current target needs it re-derived before that chain is added.

### 2.4 Who runs the nodes

The mark pipeline and the relays that push to it are a distributed network. The first nodes are run by the BTR team, with independent operators onboarded as the signer and relay sets widen. Comparable systems have the same shape: Hyperliquid's oracle prices are published by its validator set as a stake-weighted aggregate of major-venue prices, and GMX's price updates are submitted by authorized keeper accounts against a decentralized oracle network.

In all three cases the security argument rests on what the chain verifies, not on who runs a node. For BTR that is the k-of-n signature over each mark, the staleness and deviation gates that reject a bad push regardless of source ([oracle keeper](/docs/3-1-5-oracle-keeper)), and a relay path that carries no signing key and therefore cannot forge, only delay.

## 3. What alerts

### 3.1 Guardian predicates

The [guardian](/docs/3-1-4-guardian) evaluates these predicates and pages the on-call operator, with a six-hour re-page cooldown per breach and key. The transport is an operational detail and is not published.

| Severity | Predicates |
|---|---|
| Critical | `min_fee_below_two_theta`, `feed_sigma_zero`, `mark_expired`, `dev_cap_below_theta` |
| High | `band_below_floor`, `feed_unreadable`, `dev_cap_below_drift` |
| Medium | `mark_aging`, `measurement_missing`, `measurement_stale` |

A standing Critical means a deployed parameter contradicts the live tape; retune via the [risk steward](/docs/3-1-3-risk-steward).

### 3.2 Metric conditions

| Condition | Surface | What a breach means |
|---|---|---|
| `tradableRatio` below 1.0 sustained: alert. Below 0.5: page | indexer | the fraction of the book that will revert on the next swap |
| `feeds.sigmaZero` above 0 | indexer | free adverse selection ([§2.1](#21-the-gate-mirror)) |
| `feeds.worstAgeRatio` above 0.5 | indexer | one missed heartbeat; above 1.0 the leg is dead |
| `ingest.cursorBehindTip` true | indexer | the next scan double counts every additive series |
| `ingest.lastEventAgeSec` unbounded | indexer | dead indexer behind a `/health` that still returns ok |
| `bot_tradable_ratio` below 1.0, `bot_blocked_legs_total{reason}` rising, stalled `bot_gate_sweep_timestamp_seconds` | bot | legs are being refused, or the sweep itself stopped |
| realized fee bps below `minFeePbps / 100` | collector | mispriced risk |
| fee APR falling or unstable | internal | LPs are not being paid for the flow they absorbed |
| `liq.net.usd` outflow not explicable by APR | indexer | silent exit ahead of a problem the other metrics have not surfaced |
| `cov.c` below 1.0 | collector | same-asset exits in kind are capped and the toll is being charged ([flow guards](/docs/3-2-1-flow-guards)) |
| `/gas` `gate` shut, `deferred_pushes` unbounded | keeper | the fleet is priced out of the chain, or is burning budget |
| detect-to-broadcast p99 above 200 ms | keeper | pushes are late, so the mark is stale before it lands |
| `nxr_signed_sigma_cache_ready` 0 | aggregator | quorum cannot be met; the keeper has nothing valid to push |
| `nxr_tickers_provider_stale` or `_dead` above 0 | aggregator | the index is quoting off decayed legs |
| flat_fill / bars_written sustained above ~0.5 | aggregator | the feed is not arriving and the chart is fabricated continuity |
| `nxr_provider_last_ticker_age_ms` above 30 s | aggregator | a venue is dark; the composite silently reweights onto the survivors |

Keeper push outcomes are coarser than latency and do not alert cleanly: `TriggerRecord.pushed` is a boolean with no reason, and the failure paths (send error, submit timeout, revert, dropped past TTL, unstick exhausted) are log-only.

## Related

- [Oracle keeper](/docs/3-1-5-oracle-keeper)
- [Guardian](/docs/3-1-4-guardian)
- [Guardian routines](/docs/3-2-2-guardian-routines)
- [Flow guards](/docs/3-2-1-flow-guards)
- [Access control](/docs/3-1-overview)
- [Spread and Fees](/docs/1-1-4-spread-fees)