Observability
The metric contract for a BTR deployment: what is emitted, where to read it, and what pages an operator.
Four rules govern every entry below.
- A refusal is an event on every surface that can emit one. An off-chain component that declines to do something increments a counter; silence means “nothing was asked”, never “everything was refused”. The contracts are the exception: a revert emits nothing, so on-chain refusals have to be mirrored off chain to be visible.
- Units belong in the definition. PBPS is , bps is , and the fee is half the round-trip spread. Most recurring mistakes here are unit errors.
- Additive series carry an integrity signal. Any metric that is a sum over blocks publishes whether its input range was ingested once.
- Price-source pipelines are scoped. Rows marked (mark pipeline) apply only to deployments sourcing marks from the off-chain mark pipeline and relaying them through the oracle keeper. A deployment using INTERNAL or on-chain pricing starts at pool mark state and has no pipeline surfaces.
1. What is emitted
1.1 Off-chain metrics
| Stage | Decisive metrics | Healthy |
|---|---|---|
| Price ingestion, composite index, bar continuity, quote signing (mark pipeline) | per-venue tick age and reject counts, composite staleness, bar-continuity gap counters, cosign-quorum readiness; internal to the mark pipeline, not part of this repo’s surface | all fresh, quorum ready |
| Oracle relay push (mark pipeline) | per-leg push latency, per-trigger push records | detect-to-broadcast p99 < 200 ms |
| Pool mark state | feeds.tradable_ratio, feeds.by_gate{}, feeds.sigma_zero, feeds.worst_age_ratio | ratio 1.0, sigma_zero 0, worst_age_ratio < 0.5 |
| Swap execution | bot_tradable_ratio, bot_blocked_legs_total{reason}, bot_gate_legs, bot_gate_legs_blocked, bot_gate_sweep_timestamp_seconds | ratio 1.0, blocked flat |
| Swap pricing | mm.spread.avg (bps), feeAvgBps, realized OEV in bps | realized fee bps at or above minFeePbps / 100; there is no max |
| Fee capture and LP return | fee.lp.usd, fee.proto.usd, lp_return, apr.fee | fee APR positive and stable |
| Liquidity provision and withdrawal | liq.dep.usd, liq.wd.usd, liq.net.usd | net flow explicable by APR |
| Coverage | cov.c, util.liq | near 1.0 |
| Risk parameter retuning | guardian breach predicates (§3.1) | no standing Critical |
| Keeper fleet health | spend_native, deferred_pushes, gas-gate position and ratio | gate open, deferrals bounded |
| Indexer integrity | ingest.cursor_lead, ingest.cursor_behind_tip, ingest.last_event_age_sec | cursor_lead , last_event_age_sec bounded |
Notes that change how these read:
- Bar continuity. The mark pipeline’s bar producer carries the last close forward when no tick arrives in a bucket, so at bar level an outage and a quiet market look alike; its flat-fill and gap counters are internal to that pipeline, not this repo. Use them to prove liveness, never to measure venue tick rate.
- Cosign quorum. Signature gathering inside the mark pipeline enforces k-of-n under a bounded deadline and fails closed. A failing quorum presents downstream as pool-mark staleness, not as its own signal.
- Swap execution.
bot_tradable_ratiois-1before the first sweep, not 1. The sweep is recorded before the early return that fires below two executable legs, because the fully-gated case is the one that matters.bot_blocked_legs_total{reason}collapses reasons to a bounded label class: the raw reason embeds live numbers (stale(age=421s>ttl=600s)), i.e. unbounded label cardinality. The class is what an operator alerts on; the numbers stay in the structured log. - Keeper fleet. The fleet surface publishes the base-fee regime, the gate position with its hysteresis bounds, how long the gate has been shut, the longest deferral episode, discretionary pushes withheld, liveness pushes forced through a shut gate, and confirmed spend.
spend_nativeis credited only from confirmed successful receipts: reverted pushes and unstick cancel transactions are not in it.forced_pushesnon-zero is not a fault: liveness pushes relay at any price.
1.2 On-chain events
| Class | Events | Indexable |
|---|---|---|
| Swap | Swapped (packed: prices = mark and mid, outState = covToll, index and spreadPbps) | yes |
| Liquidity and solvency | SolvencyUpdated(cWad, navBase, claimBase), LiabilitySwapped, Withdrawn | yes |
| Oracle push | none: cadence from push tx calldata and receipts, lane state from storage | no, lane state is read |
| Mark-store governance | AuthUpdated(tier, signers, k, relayers), OpQueued, OpCancelled, Migrated | yes |
| Params and halts | AssetParamsUpdated, RiskFencesUpdated, BoundedAssetParamsUpdated, CurveUpdated, FeeParamsUpdated, AssetHalted, AssetUnhalted | no, snapshot trigger only |
| Lane lifecycle | LaneListed, LaneHalted(lane, tierMask, halted), FeedLaneUpdated (factory) | no, snapshot trigger only |
| Balance and index movers | IndexUpdated(reason), Deposited, Donated, FlashLoanExecuted, TreasuryClaimed, Swept, UntrackedSynced, HookDustWrittenOff | yes |
- Retuning and halting are not queryable from the indexer: they are snapshot triggers, not indexed topics.
- A
reanchoremits no event of its own beyond the store’sOpQueued; the lane word shows the anchor dropped (dark, markless) until the next push (oracle keeper). - A push names no lane. A monotonic skip (
srcSecs <= obsat the lane) and a band refusal (the lane goes dark) are both read from the lane word, not subscribed. Swappedpacks its payload into five words (layout):amounts,fees,prices,outBook,outState. It carriesmarkPrice,midPriceandcovToll, so total LP revenue is reconstructible from logs alone. All three aretokenOutpertokenIn, chained over every leg, hence directly comparable to the realized . OEV decomposes exactly on chain: with the execution price, the mid and the oracle mark,
The two prices are exact WAD, one 128-bit field each in prices and saturated at 2^128 - 1, not a packed float, so the reconstruction inherits no encoding error. covToll is withheld from the output before the fee and retained in the reserve, so a reader that ignores it books it as coverage drift rather than revenue.
- Only
LiabilitySwappedcarries ahaircutfield (face - face · Cwhen positive). A same-assetWithdrawncarries none; its shortfall below face is the coverage haircutface · (1 - min(c_leg, C))plus the exit toll;previewWithdrawreturns the net payout and the haircut, and the pairedSolvencyUpdatedand the leg’s books give the haircut too.SolvencyUpdatedfires on every LP entrypoint of an armed pool;Pool.poolSolvencyWad()reads the pool rate live.cov.candutil.liqare served per asset.
1.3 Units and scales
spreadPbps (bits 224..239 of Swapped.outState) is PBPS, base , not bps. A column name does not indicate its scale. Two different divisors come off that one word. With the spread in PBPS:
| Quantity | From | Divisor | Scale |
|---|---|---|---|
| Realized fee | Swapped.outState >> 224 | 200 | bps |
| Spread | Swapped.outState >> 224 | 100 | bps |
minFeePbps | getAsset | 100 | bps |
The 200 is two conversions at once: 100 for PBPS to bps, and 2 because the fee is half the round-trip spread. A rate that is already a rate, such as minFeePbps, uses 100 and not 200, so comparing a realized fee against the floor means dividing the floor by 100 first. The constants are named SPREAD_PBPS_PER_FEE_BP and SPREAD_PBPS_PER_BP in one place. Snapshot writers resolve ABI fields by name, never by position: minFeeBps and minFeePbps differ by one letter and by a factor of 100. Realized fee bps and OEV bps are computed from the swap amounts, not from the spread word.
minFeePbps is the only fee rate getAsset returns, and it is a two-sided-bounded per-leg floor on the spread, not a target and not a ceiling. Everything else getAsset returns is inventory (reserves, liabilities, anchor, liquidityIndexWad, minLiquidity), shape (curveId, the dispersion band), or credit-cap and decimal metadata (vegaBps, depositCapCode, maxLiabWeightBps, decimals, deadSeedPow10). A dashboard therefore cannot plot a per-leg fee band from config and must not present one: the quoted spread has no upper bound, because a , confidence or staleness term that widens it is pricing real risk. See Spread and Fees.
2. Where to read it
| Surface | Carries | Public |
|---|---|---|
| Contract events | swap, solvency, oracle push, signer governance (§1.2) | yes, on chain |
| Read-only gateway | API & SDK Reference | yes |
| Operator-internal telemetry | ingestion, keeper fleet, indexer integrity, bot preflight | no |
| Operator alert channel | guardian breach predicates (§3.1) | no |
Every health and metrics surface above is operator-internal. Their addresses, routes and transports are not published, are not reachable from the public internet, and are not part of any integration contract. Integrations must not depend on them. The public surfaces are on-chain state and the read-only gateway; every claim on this page is verifiable from those two.
2.1 The gate mirror
A swap’s first gate is FeedMathLib.gate(). It reverts on four conditions, checked in this order: feed halted, stale, mark zero, confidence over the halt threshold (oracle keeper). A revert emits nothing, and a leg refused at quote time never reaches a chain, so the off-chain mirror is the only trace. The indexer evaluates the same predicate off chain against the feed state it multicalls, and publishes the result on an operator-internal surface: how much of the book is tradable, which of the four gate conditions each blocked leg failed, and how far past its TTL the worst leg has drifted. on an otherwise-healthy feed is treated as a defect and counted separately, because it is invisible in any freshness view.
Two correctness notes. Age is computed on the observed-at clock the contract gates on, , not on alone: is the relay landing time, so using it understates age by the relay lag and can report a feed fresh that the pool reverts on. And confidence, flags, maximum deviation and source timestamp are snapshotted alongside the gate result; without them the mirror sees only staleness, which is two of the four gate conditions, and an outage would be recorded only as a period of zero volume, without its cause.
The mirror is feed state, not pool state, so it does not see BaseDepegged, PriceOutsideRefBand or a halted asset flag (flow guards).
2.2 Fees, LP return and liquidity flow
Fee telemetry stores both legs’ fees, realized fee bps, execution price, LP return and the liquidity it was earned against, one row per Swapped log, deduplicated on the swap’s log identity. The asset is the primitive: asset APR is absolute fees over that asset’s liquidity, and pool APR is the mean of its assets’.
Fee APR is an annualization over an elapsed span, so a span shorter than one hour is not annualized at all: below that window the figure is null rather than a small number that looks like yield. The window is stored with the figure.
Volume counts hop 0 only, so a multi-pool route is one user swap. Fees count every hop, because each transited pool charged a fee. Both reductions run over the same event stream; the per-row hop index distinguishes them.
2.3 Indexer integrity
Every additive series is a sum over blocks, so a replayed range doubles it. The ingest loop guards this at startup by bumping the cursor past the indexed tip and publishes the guard on the internal health surface. With the cursor and the indexed tip:
ingest.cursor_leadis . Negative means the next scan replays indexed blocks and double counts every additive series.ingest.cursor_behind_tip, the boolean form, for alerting.ingest.last_event_age_sec, the age of the newest indexed event. This is the number that proves the indexer is still advancing.
Indexed series are confirmed to a per-chain depth chosen against that chain’s finality.
2.4 Who runs the nodes
The mark pipeline and the relays that push to it are a distributed network. The first nodes are run by the BTR team, with independent operators onboarded as the signer and relay sets widen.
The security argument rests on what the chain verifies, not on who runs a node. For BTR that is the k-of-n signature over each mark, the staleness and deviation gates that reject a bad push regardless of source (oracle keeper), and a relay path that carries no signing key and therefore cannot forge, only delay.
3. What alerts
3.1 Guardian predicates
The risk keeper’s guardian module (observe and alert only; the on-chain guardian is a separate role) evaluates these predicates and pages the on-call operator. Re-pages are cooled down per breach and key. The cooldown and the transport are operational details and are not published.
| Severity | Predicates |
|---|---|
| Critical | min_fee_below_two_theta, feed_sigma_zero, mark_expired, dev_cap_below_theta |
| High | band_below_floor, feed_unreadable, dev_cap_below_drift |
| Medium | mark_aging, measurement_missing, measurement_stale |
A standing Critical means a deployed parameter contradicts the live tape; retune via the risk steward.
3.2 Metric conditions
| Condition | Surface | What a breach means |
|---|---|---|
tradable_ratio below 1.0 sustained: alert. Below 0.5: page | indexer | the fraction of the book that will revert on the next swap |
feeds.sigma_zero above 0 | indexer | on an otherwise-healthy feed, treated as a defect (§2.1) |
feeds.worst_age_ratio above 0.5 | indexer | one missed heartbeat; above 1.0 the leg is dead |
ingest.cursor_behind_tip true | indexer | the next scan double counts every additive series |
ingest.last_event_age_sec unbounded | indexer | the indexer has stopped advancing |
bot_tradable_ratio below 1.0, bot_blocked_legs_total{reason} rising, stalled bot_gate_sweep_timestamp_seconds | bot | legs are being refused, or the sweep itself stopped |
realized fee bps below minFeePbps / 100 | indexer | mispriced risk |
| fee APR falling or unstable | internal | LPs are not being paid for the flow they absorbed |
liq.net.usd outflow not explicable by APR | indexer | silent exit ahead of a problem the other metrics have not surfaced |
cov.c below 1.0 | indexer | same-asset exits in kind are capped and the toll is being charged (flow guards) |
gas gate shut, deferred_pushes unbounded | keeper | the fleet is priced out of the chain, or is burning budget |
| detect-to-broadcast p99 above 200 ms | keeper | pushes are late, so the mark is stale before it lands |
| mark-pipeline cosign readiness false | aggregator | quorum cannot be met; the keeper has nothing valid to push |
| mark-pipeline composite staleness above 0 | aggregator | the index is quoting off decayed legs |
| mark-pipeline bar flat-fill ratio sustained above ~0.5 | aggregator | the feed is not arriving and the chart is flat-filled |
| mark-pipeline per-venue tick age above 30 s | aggregator | a venue is dark; the composite reweights onto the remaining venues |