Observability

This page is the metric contract for a BTR deployment. For every stage between a price arriving at NX Rates and an LP’s realised return it names the few metrics that decide whether that stage is working, where each is emitted, and what a breach means operationally. Read it to build alerting, to triage an incident, or to check that a dashboard number means what its label claims. Units and scales are stated with each metric, because most recurring mistakes here are unit errors.

Scope

Four rules govern every entry below.

1. A refusal is an event, on every surface that can emit one. If an off-chain component declines to do something, that increments a counter; silence means “nothing was asked”, never “everything was refused”. The on-chain contracts are the exception and the reason stage 6 exists: a revert emits nothing, so the refusal has to be mirrored off chain to be visible at all.
2. Units belong in the definition. Every rate below states its scale. PBPS is ${10}^6$, bps is ${10}^4$, and the fee is half the round-trip spread.
3. Additive series carry an integrity signal. Any metric that is a sum over blocks publishes whether its input range was ingested once.
4. Price-source pipelines are scoped. Stages marked (NX Rates) describe the NX Rates aggregation-and-push pipeline. They apply only to deployments sourcing marks from NX Rates; a deployment using on-chain or INTERNAL pricing has stages 6 onward and none of the pipeline stages.

Lifecycle stages

Stage detail

6: pool mark state, and the gate mirror

A swap’s first gate is FeedMathLib.gate(). It reverts on four conditions, checked in this order: feed paused, stale, mark zero, confidence over the halt threshold (Oracles §8.2). A revert emits nothing, and a leg refused at quote time never reaches a chain, so there is no on-chain trace whatsoever, which is the whole reason for the mirror below.

The collector evaluates the same predicate off chain against the feed state it multicalls, and publishes the ratio at GET /protocol/health:

{
  "feeds": {
    "total": 0, "tradable": 0, "tradableRatio": 1.0,
    "byGate": { "ok": 0, "stale": 0, "paused": 0, "dead": 0, "uncertain": 0, "unreadable": 0 },
    "sigmaZero": 0, "aging": 0, "worstAgeRatio": 0.0, "worstFeed": "",
    "blocked": [
      { "name": "", "symbol": "", "gate": "", "ageSec": 0, "ttl": 0,
        "ageRatio": 0.0, "sigma": 0, "confidence": 0, "maxDeviationBps": 0 }
    ]
  }
}

• tradableRatio is the one number that answers “is the protocol open for business”. Healthy is 1.0. Alert below 1.0 sustained, page below 0.5.
• byGate gives the reason without reading the roster. paused is an operator action, stale is a keeper failure, unreadable is a missing feed. Each demands a different response.
• sigmaZero counts feeds that are otherwise healthy but publish $\sigma =0$. The dynamic fee then collapses to minFee while the mark keeps moving, which is free adverse selection. It is invisible in any freshness view, so it gets its own counter. Healthy is 0.
• worstAgeRatio is $a/\tau$. Above 0.5 is one missed heartbeat, since the ops rule is $\tau \approx 2\times$ heartbeat. Above 1.0 the leg is dead.
• blocked is sorted worst first, so an operator reads the top of the list rather than all of it.

Two correctness notes on the mirror:

• Age is computed on the observed-at clock the contract gates on, $t_{obs}=\min (t_{src},t_{upd})$, not on $t_{upd}$ alone. $t_{upd}$ is the relay landing time, so using it understates age by the relay lag and can report a feed fresh that the pool reverts on.
• confidence, flags, maxDeviation and sourceTs are snapshotted and persisted. Without them the mirror sees only staleness, which is two of the four gate conditions.

oracle_feed_state carries confidence, flags, max_deviation_bps, source_ts and gate. Persisting the verdict means an outage keeps its reason instead of surviving only as a period when volume was zero.

The mirror is feed state, not pool state, so it does not see BaseDepegged, PriceOutsideRefBand or a halted asset flag.

7: swap execution

The flow bot computes the blocked set on a preflight sweep and publishes it:

• bot_gate_legs and bot_gate_legs_blocked, gauges from the sweep.
• bot_tradable_ratio, the fraction of legs that currently quote. -1 before the first sweep, never a flattering 1.
• bot_gate_sweep_timestamp_seconds, so a stalled sweep is distinguishable from a healthy book.
• bot_blocked_legs_total{reason}, a counter keyed by gate reason class.

Two design points:

• The sweep is recorded before the early return that fires when fewer than two legs are executable. Recording after it would skip exactly the fully-gated case, which is the one that matters.
• Reasons are collapsed to a bounded label class. The raw reason embeds live numbers (stale(age=421s>ttl=600s)), which is unbounded label cardinality. The class is what an operator alerts on; the numbers stay in the JSONL and the pod log.

8: swap pricing and units

Swapped.spreadPbps is PBPS, base ${10}^6$, despite being stored in a column named spread_bps. Two different divisors come off that one word. With $S$ the spread in PBPS:

The 200 is two conversions at once: 100 for PBPS to bps, and 2 because the fee is half the round-trip spread. A fee rate that is already a rate, such as minFeePbps, uses 100 and not 200, so a comparison of a realized fee against the floor must divide the floor by 100 first. The constants are named SPREAD_PBPS_PER_FEE_BP and SPREAD_PBPS_PER_BP in one place, so no reader can silently re-derive the wrong one.

minFeePbps is the only fee rate getAsset returns, and it is a two-sided-bounded per-leg floor on the spread, not a target and not a ceiling. Everything else getAsset returns is inventory (reserves, liabilities, anchor, liquidityIndexWad, minLiquidity), shape (presetId, the dispersion band), or exit/decimal metadata (vegaBps, haircutSuppressorBps, decimals, deadSeedPow10). A dashboard therefore cannot plot a per-leg fee band from config and must not present one: the quoted spread has no upper bound by design, because a $\sigma$, confidence or staleness term that drives it wide is pricing real risk.

Snapshot writers resolve ABI fields by name, never by position: minFeeBps and minFeePbps differ by one letter and by a factor of 100.

fee_bps_realized and oev_bps in dex_swap_fees are computed from amounts rather than from the spread word.

9 and 10: fees, LP return, liquidity flow

dex_swap_fees stores both legs’ fees, realized fee bps, execution price, LP return and the liquidity it was earned against, one row per Swapped log, deduplicated on swap_id.

The asset is the primitive: asset APR is absolute fees over that asset’s liquidity. Pool APR is the mean of its assets’.

Two code paths compute it and only one carries the elapsed-span guard, so scope matters.

• The ingest loop annualizes over the real elapsed span $t_{now}-t_{then}$ and returns null below MIN_APR_WINDOW_MS = 1 h, so the loop-published metrics never annualize noise.
• The served /apr endpoint calls buildFeeApr(qdb, registry, windowMs), which annualizes over the requested window (default 7 days) with no elapsed-span check and no null gate.

During cold start or after an ingest gap, /apr returns a small non-null number that reads as real yield and is not.

Volume counts hop 0 only, so a multi-pool route is one user swap. Fees count every hop, because each transited pool really charged and really earned. Same event stream, deliberately different reduction, and hop_index makes both provable.

Swapped carries markPrice, midPrice and covToll, so total LP revenue is reconstructible from logs alone. All three are tokenOut per tokenIn, chained over every leg, so they are directly comparable to the realized $\mathtt{a}\mathtt{m}\mathtt{o}\mathtt{u}\mathtt{n}\mathtt{t}\mathtt{O}\mathtt{u}\mathtt{t}/\mathtt{a}\mathtt{m}\mathtt{o}\mathtt{u}\mathtt{n}\mathtt{t}\mathtt{I}\mathtt{n}$. OEV decomposes exactly on chain: with $p_x$ the execution price, $p_m$ the mid and $p_o$ the oracle mark,

The two prices are exact WAD, never a packed float: a value that goes straight into a log is never packed, because ABI padding takes every non-indexed arg to a full word and the encode would save no bytes while costing gas and mantissa precision. The reconstruction therefore inherits no encoding error.

covToll is withheld from the output before the fee and retained in the reserve, so a reader that ignores it books it as coverage drift rather than as revenue.

11: coverage and haircuts

cov.c and util.liq are served per asset.

applyHaircut fires on every withdrawal path when reserves < liabilities, but only LiabilitySwapped carries a haircut field. A same-asset withdrawal emits Withdrawn, which has no haircut field, so that haircut is not visible on chain. previewWithdraw is the off-chain read that recovers it.

12: risk parameter retuning

The guardian evaluates the predicates below with severities and pages the on-call operator, with a six-hour re-page cooldown per breach and key. The transport is an operational detail and is not published.

Predicates:

• min_fee_below_two_theta (Critical)
• feed_sigma_zero (Critical)
• mark_expired (Critical)
• dev_cap_below_theta (Critical)
• band_below_floor (High)
• feed_unreadable (High)
• dev_cap_below_drift (High)
• mark_aging (Medium)
• measurement_missing (Medium)
• measurement_stale (Medium)

The collector subscribes to swap, liquidity and oracle-signer-governance topics. The parameter and halt events (AssetParamsUpdated, RiskFencesUpdated, BoundedAssetParamsUpdated, CurveUpdated, FeeParamsUpdated, AssetHalted, AssetUnhalted, and the oracle’s FeedPaused / FeedUnpaused / FeedUpdated / FeedRegistered / FeedExpBiasUpdated set) are snapshot triggers rather than indexed topics, so retuning and halting are not queryable from the indexer. The next release adds FeedWidenRequested / FeedWidenExecuted / FeedWidenCancelled to that set; FeedWidenExecuted carries both before/after pairs, and oldBand == newBand marks a pure wedge release rather than a config loosening (Oracles §8.3).

The V4 push path is indexable and is not on that list: SlotsPushed(seq, sourceTsDs, acceptedMask, blobHash) on both entry points, plus LanesSkipped(slotId, laneMask) whenever a lane fail-softs. One V4 outcome has no event at all — a whole slot stepped over for failing the source-second monotonicity check — so slot age has to be polled, not subscribed.

14: indexer integrity

Every additive series is a sum over blocks, so a replayed range doubles it. The ingest loop guards this at startup by bumping the cursor past the indexed tip, and publishes the guard in /protocol/health. With $c$ the cursor and $T$ the indexed tip:

• ingest.cursorLead is $\ell$. Negative means the next scan replays indexed blocks and double counts every additive series.
• ingest.cursorBehindTip, the boolean form, for alerting.
• ingest.lastEventAgeSec, the age of the newest indexed event. /health on the collector is static and returns ok even with a dead indexer, which is also what the container health check probes. This is the number that detects it.

Reorg handling is a five-block confirmation depth and nothing more: no block-hash tracking and no rollback path. A reorg deeper than five blocks corrupts additive series with no signal. The depth is a single fleet-wide constant, not a per-chain setting, so a chain with weaker finality than the current target needs it re-derived before that chain is added.

Where each surface lives

The health and metrics surfaces described on this page are operator-internal. Their addresses, routes and transports are not published, are not reachable from the public internet, and are not part of any integration contract: nothing an integrator builds should depend on them, and nothing here is a supported API.

What is public is the on-chain state itself and the read-only gateway documented in API & SDK Reference. Every claim on this page can be verified from those two, which is the level at which the protocol intends to be checkable.

Who runs the nodes. The mark pipeline and the relays that push to it are a distributed network, and the first nodes are operated by the BTR team, with independent operators onboarded as the signer and relay sets widen. That is the same shape the comparable systems document: Hyperliquid’s oracle prices are published by its validator set as a stake-weighted aggregate of major-venue prices, and GMX’s price updates are submitted by authorized keeper accounts against a decentralized oracle network. In all three cases the security argument rests on what the chain verifies, not on who happens to be running a node: for BTR that is the k-of-n signature over each mark (3.6), the staleness and deviation gates that reject a bad push regardless of its source (3.4 §8), and a relay path that carries no signing key and therefore cannot forge, only delay.

NX Rates pipeline observability

Applies ONLY to deployments that source marks from the NX Rates aggregator and relay them through the oracle keeper. Deployments using INTERNAL or on-chain price sources stop at the pool-mark stage above and have no pipeline surfaces.

1 to 3: price ingestion and bars

NX Rates exports a dense provider surface. Stage 3 covers continuity rather than coverage.

The s10 producer carries the last close forward when no tick arrives in a 10 s bucket, so at bar level an outage and a quiet market look alike. mitch::Bar carries tick_count and FLAG_S10_FLAT_FILL; two counters lift that into metric space.

• nxr_s10_flat_fill_total{producer}, incremented per emitted bar with tick_count == 0.
• nxr_s10_hole_total{producer}, for a bucket with neither a tick nor a last close. That case emits no bar at all, so it is invisible in the bar stream.

Read them as a ratio against nxr_bars_written_total{producer}. A ticker whose flat-fill share is sustained above ~0.5 is not quiet, it is dark.

tick_count on a real bar counts changed composites, not raw provider ticks, because of the delta gate in the producer. It is therefore a lower bound. Use it to prove liveness, never to measure venue tick rate: that is stage 1’s job.

4: quote signing and cosign quorum

Signature gathering (NXR-internal) enforces k-of-n under a 40 ms deadline and fails closed. nxr_signed_sigma_cache_ready is the readiness signal; a quorum that starts failing otherwise presents downstream as stage 6 staleness.

5: relay push

ExternalOracle emits no event on push: observability is getFeed() state polling. That is a gas decision with a consequence. A 66-feed relay batch produces zero log entries, so acceptance and rejection are knowable only from the keeper’s own view or by diffing polled state.

The keeper publishes /triggers, /thresholds, /latency and /gas. Latency is well covered. Outcomes are coarser: TriggerRecord.pushed is a boolean with no reason, and the failure paths (send error, submit timeout, revert, dropped past TTL, unstick exhausted) are log-only.

/gas spend_eth is credited only from confirmed successful receipts, so it is confirmed successful spend, not total spend: reverted pushes and unstick cancel transactions are not in it.

13: keeper fleet health

/gas publishes:

• The base-fee regime.
• The gate position with its hysteresis bounds.
• How long the gate has been shut.
• The longest deferral episode.
• Discretionary pushes withheld.
• Liveness pushes forced through a shut gate.
• Confirmed spend.

Healthy is gate: open with deferred_pushes growing only during genuine fee spikes. forced_pushes non-zero is the design working, not a fault: liveness pushes relay at any price.

Related

• Oracle Price-Push Security
• Oracles
• Access Control, Roles & Emergency Powers
• Spread and Fees