Observability

The metric contract for a BTR deployment: what is emitted, where to read it, and what pages someone.

Four rules govern every entry below.

1. A refusal is an event on every surface that can emit one. An off-chain component that declines to do something increments a counter; silence means “nothing was asked”, never “everything was refused”. The contracts are the exception: a revert emits nothing, so on-chain refusals have to be mirrored off chain to be visible at all.
2. Units belong in the definition. PBPS is ${10}^6$, bps is ${10}^4$, and the fee is half the round-trip spread. Most recurring mistakes here are unit errors.
3. Additive series carry an integrity signal. Any metric that is a sum over blocks publishes whether its input range was ingested once.
4. Price-source pipelines are scoped. Rows marked (NX Rates) apply only to deployments sourcing marks from the NX Rates aggregator and relaying them through the oracle keeper. A deployment using INTERNAL or on-chain pricing starts at pool mark state and has no pipeline surfaces.

1. What is emitted

1.1 Off-chain metrics

Notes that change how these read:

• Bar continuity. The s10 producer carries the last close forward when no tick arrives in a 10 s bucket, so at bar level an outage and a quiet market look alike. mitch::Bar carries tick_count and FLAG_S10_FLAT_FILL; nxr_s10_flat_fill_total increments per emitted bar with tick_count == 0, and nxr_s10_hole_total counts a bucket with neither a tick nor a last close (no bar is emitted at all, so it is invisible in the bar stream). Read both as a ratio against nxr_bars_written_total. tick_count counts changed composites, not raw provider ticks, because of the producer’s delta gate, so it is a lower bound: use it to prove liveness, never to measure venue tick rate.
• Cosign quorum. Signature gathering (NXR-internal) enforces k-of-n under a 40 ms deadline and fails closed. A failing quorum presents downstream as pool-mark staleness, not as its own signal.
• Swap execution. bot_tradable_ratio is -1 before the first sweep, never a flattering 1. The sweep is recorded before the early return that fires below two executable legs, because that is exactly the fully-gated case that matters. bot_blocked_legs_total{reason} collapses reasons to a bounded label class: the raw reason embeds live numbers (stale(age=421s>ttl=600s)), i.e. unbounded label cardinality. The class is what an operator alerts on; the numbers stay in the JSONL and the pod log.
• Keeper fleet. /gas publishes the base-fee regime, the gate position with its hysteresis bounds, how long the gate has been shut, the longest deferral episode, discretionary pushes withheld, liveness pushes forced through a shut gate, and confirmed spend. spend_native is credited only from confirmed successful receipts: reverted pushes and unstick cancel transactions are not in it. forced_pushes non-zero is the design working, not a fault: liveness pushes relay at any price.

1.2 On-chain events

• Retuning and halting are not queryable from the indexer: they are snapshot triggers, not indexed topics.
• FeedWidenExecuted carries both before/after pairs; oldBand == newBand marks a pure wedge release rather than a config loosening (oracle keeper).
• Two outcomes emit nothing. ExternalOracle emits no event on push (a gas decision: a 66-feed relay batch produces zero log entries, so acceptance and rejection are knowable only from the keeper’s own view or by diffing polled getFeed() state), and a V4 slot stepped over for failing the source-second monotonicity check has no event, so slot age must be polled, not subscribed.
• Swapped carries markPrice, midPrice and covToll, so total LP revenue is reconstructible from logs alone. All three are tokenOut per tokenIn, chained over every leg, hence directly comparable to the realized $\mathtt{a}\mathtt{m}\mathtt{o}\mathtt{u}\mathtt{n}\mathtt{t}\mathtt{O}\mathtt{u}\mathtt{t}/\mathtt{a}\mathtt{m}\mathtt{o}\mathtt{u}\mathtt{n}\mathtt{t}\mathtt{I}\mathtt{n}$. OEV decomposes exactly on chain: with $p_x$ the execution price, $p_m$ the mid and $p_o$ the oracle mark,

The two prices are exact WAD, never a packed float: ABI padding takes every non-indexed arg to a full word, so packing would save no bytes while costing gas and mantissa precision. The reconstruction therefore inherits no encoding error. covToll is withheld from the output before the fee and retained in the reserve, so a reader that ignores it books it as coverage drift rather than revenue.

• Only LiabilitySwapped carries a haircut field (face - face · C when positive). A same-asset Withdrawn carries none; its shortfall below face is face · (1 - min(c_leg, C)), recoverable from previewWithdraw or from the paired SolvencyUpdated and the leg’s books. SolvencyUpdated fires on every LP entrypoint of an armed pool; Pool.poolSolvencyWad() reads the pool rate live. cov.c and util.liq are served per asset.

1.3 Units and scales

Swapped.spreadPbps is PBPS, base ${10}^6$, despite being stored in a column named spread_bps. Two different divisors come off that one word. With $S$ the spread in PBPS:

The 200 is two conversions at once: 100 for PBPS to bps, and 2 because the fee is half the round-trip spread. A rate that is already a rate, such as minFeePbps, uses 100 and not 200, so comparing a realized fee against the floor means dividing the floor by 100 first. The constants are named SPREAD_PBPS_PER_FEE_BP and SPREAD_PBPS_PER_BP in one place, so no reader can silently re-derive the wrong one. Snapshot writers resolve ABI fields by name, never by position: minFeeBps and minFeePbps differ by one letter and by a factor of 100. fee_bps_realized and oev_bps in dex_swap_fees are computed from amounts, not from the spread word.

minFeePbps is the only fee rate getAsset returns, and it is a two-sided-bounded per-leg floor on the spread, not a target and not a ceiling. Everything else getAsset returns is inventory (reserves, liabilities, anchor, liquidityIndexWad, minLiquidity), shape (presetId, the dispersion band), or credit-cap and decimal metadata (vegaBps, depositCapCode, maxLiabWeightBps, decimals, deadSeedPow10). A dashboard therefore cannot plot a per-leg fee band from config and must not present one: the quoted spread has no upper bound by design, because a $\sigma$, confidence or staleness term that drives it wide is pricing real risk. See Spread and Fees.

2. Where to read it

Every health and metrics surface above is operator-internal. Their addresses, routes and transports are not published, are not reachable from the public internet, and are not part of any integration contract: nothing an integrator builds should depend on them, and nothing here is a supported API. Public instead is the on-chain state itself plus the read-only gateway, and every claim on this page is verifiable from those two.

2.1 The gate mirror

A swap’s first gate is FeedMathLib.gate(). It reverts on four conditions, checked in this order: feed paused, stale, mark zero, confidence over the halt threshold (oracle keeper). A revert emits nothing, and a leg refused at quote time never reaches a chain, so the off-chain mirror is the only trace. The collector evaluates the same predicate off chain against the feed state it multicalls, and publishes it at GET /protocol/health:

{
  "feeds": {
    "total": 0, "tradable": 0, "tradableRatio": 1.0,
    "byGate": { "ok": 0, "stale": 0, "paused": 0, "dead": 0, "uncertain": 0, "unreadable": 0 },
    "sigmaZero": 0, "aging": 0, "worstAgeRatio": 0.0, "worstFeed": "",
    "blocked": [
      { "name": "", "symbol": "", "gate": "", "ageSec": 0, "ttl": 0,
        "ageRatio": 0.0, "sigma": 0, "confidence": 0, "maxDeviationBps": 0 }
    ]
  }
}

• tradableRatio answers “is the protocol open for business”. Healthy is 1.0.
• byGate gives the reason without reading the roster. paused is an operator action, stale is a keeper failure, unreadable is a missing feed. Each demands a different response.
• sigmaZero counts feeds that are otherwise healthy but publish $\sigma =0$. The dynamic fee then collapses to minFee while the mark keeps moving, which is free adverse selection. It is invisible in any freshness view, so it gets its own counter. Healthy is 0.
• worstAgeRatio is $a/\tau$. Above 0.5 is one missed heartbeat, since the ops rule is $\tau \approx 2\times$ heartbeat. Above 1.0 the leg is dead.
• blocked is sorted worst first, so an operator reads the top of the list rather than all of it.

Two correctness notes. Age is computed on the observed-at clock the contract gates on, $t_{obs}=\min (t_{src},t_{upd})$, not on $t_{upd}$ alone: $t_{upd}$ is the relay landing time, so using it understates age by the relay lag and can report a feed fresh that the pool reverts on. And confidence, flags, maxDeviation and sourceTs are snapshotted and persisted; without them the mirror sees only staleness, which is two of the four gate conditions. oracle_feed_state carries confidence, flags, max_deviation_bps, source_ts and gate, so an outage keeps its reason instead of surviving only as a period when volume was zero.

The mirror is feed state, not pool state, so it does not see BaseDepegged, PriceOutsideRefBand or a halted asset flag (flow guards).

2.2 Fees, LP return and liquidity flow

dex_swap_fees stores both legs’ fees, realized fee bps, execution price, LP return and the liquidity it was earned against, one row per Swapped log, deduplicated on swap_id. The asset is the primitive: asset APR is absolute fees over that asset’s liquidity, and pool APR is the mean of its assets’.

Two code paths compute it and only one carries the elapsed-span guard, so scope matters.

During cold start or after an ingest gap, /apr returns a small non-null number that reads as real yield and is not.

Volume counts hop 0 only, so a multi-pool route is one user swap. Fees count every hop, because each transited pool really charged and really earned. Same event stream, deliberately different reduction, and hop_index makes both provable.

2.3 Indexer integrity

Every additive series is a sum over blocks, so a replayed range doubles it. The ingest loop guards this at startup by bumping the cursor past the indexed tip and publishes the guard in /protocol/health. With $c$ the cursor and $T$ the indexed tip:

• ingest.cursorLead is $\ell$. Negative means the next scan replays indexed blocks and double counts every additive series.
• ingest.cursorBehindTip, the boolean form, for alerting.
• ingest.lastEventAgeSec, the age of the newest indexed event. /health on the collector is static and returns ok even with a dead indexer, which is also what the container health check probes; this is the number that detects it.

Reorg handling is a five-block confirmation depth and nothing more: no block-hash tracking and no rollback path. A reorg deeper than five blocks corrupts additive series with no signal. The depth is a single fleet-wide constant, not a per-chain setting, so a chain with weaker finality than the current target needs it re-derived before that chain is added.

2.4 Who runs the nodes

The mark pipeline and the relays that push to it are a distributed network. The first nodes are run by the BTR team, with independent operators onboarded as the signer and relay sets widen. Comparable systems have the same shape: Hyperliquid’s oracle prices are published by its validator set as a stake-weighted aggregate of major-venue prices, and GMX’s price updates are submitted by authorized keeper accounts against a decentralized oracle network.

In all three cases the security argument rests on what the chain verifies, not on who runs a node. For BTR that is the k-of-n signature over each mark, the staleness and deviation gates that reject a bad push regardless of source (oracle keeper), and a relay path that carries no signing key and therefore cannot forge, only delay.

3. What alerts

3.1 Guardian predicates

The guardian evaluates these predicates and pages the on-call operator, with a six-hour re-page cooldown per breach and key. The transport is an operational detail and is not published.

A standing Critical means a deployed parameter contradicts the live tape; retune via the risk steward.

3.2 Metric conditions

Keeper push outcomes are coarser than latency and do not alert cleanly: TriggerRecord.pushed is a boolean with no reason, and the failure paths (send error, submit timeout, revert, dropped past TTL, unstick exhausted) are log-only.

Related

• Oracle keeper
• Guardian
• Guardian routines
• Flow guards
• Access control
• Spread and Fees