Observability

The metric contract for a BTR deployment: what is emitted, where to read it, and what pages an operator.

Four rules govern every entry below.

  1. A refusal is an event on every surface that can emit one. An off-chain component that declines to do something increments a counter; silence means “nothing was asked”, never “everything was refused”. The contracts are the exception: a revert emits nothing, so on-chain refusals have to be mirrored off chain to be visible.
  2. Units belong in the definition. PBPS is 106, bps is 104, and the fee is half the round-trip spread. Most recurring mistakes here are unit errors.
  3. Additive series carry an integrity signal. Any metric that is a sum over blocks publishes whether its input range was ingested once.
  4. Price-source pipelines are scoped. Rows marked (mark pipeline) apply only to deployments sourcing marks from the off-chain mark pipeline and relaying them through the oracle keeper. A deployment using INTERNAL or on-chain pricing starts at pool mark state and has no pipeline surfaces.

1. What is emitted

1.1 Off-chain metrics

StageDecisive metricsHealthy
Price ingestion, composite index, bar continuity, quote signing (mark pipeline)per-venue tick age and reject counts, composite staleness, bar-continuity gap counters, cosign-quorum readiness; internal to the mark pipeline, not part of this repo’s surfaceall fresh, quorum ready
Oracle relay push (mark pipeline)per-leg push latency, per-trigger push recordsdetect-to-broadcast p99 < 200 ms
Pool mark statefeeds.tradable_ratio, feeds.by_gate{}, feeds.sigma_zero, feeds.worst_age_ratioratio 1.0, sigma_zero 0, worst_age_ratio < 0.5
Swap executionbot_tradable_ratio, bot_blocked_legs_total{reason}, bot_gate_legs, bot_gate_legs_blocked, bot_gate_sweep_timestamp_secondsratio 1.0, blocked flat
Swap pricingmm.spread.avg (bps), feeAvgBps, realized OEV in bpsrealized fee bps at or above minFeePbps / 100; there is no max
Fee capture and LP returnfee.lp.usd, fee.proto.usd, lp_return, apr.feefee APR positive and stable
Liquidity provision and withdrawalliq.dep.usd, liq.wd.usd, liq.net.usdnet flow explicable by APR
Coveragecov.c, util.liqc near 1.0
Risk parameter retuningguardian breach predicates (§3.1)no standing Critical
Keeper fleet healthspend_native, deferred_pushes, gas-gate position and ratiogate open, deferrals bounded
Indexer integrityingest.cursor_lead, ingest.cursor_behind_tip, ingest.last_event_age_seccursor_lead ≥0, last_event_age_sec bounded

Notes that change how these read:

  • Bar continuity. The mark pipeline’s bar producer carries the last close forward when no tick arrives in a bucket, so at bar level an outage and a quiet market look alike; its flat-fill and gap counters are internal to that pipeline, not this repo. Use them to prove liveness, never to measure venue tick rate.
  • Cosign quorum. Signature gathering inside the mark pipeline enforces k-of-n under a bounded deadline and fails closed. A failing quorum presents downstream as pool-mark staleness, not as its own signal.
  • Swap execution. bot_tradable_ratio is -1 before the first sweep, not 1. The sweep is recorded before the early return that fires below two executable legs, because the fully-gated case is the one that matters. bot_blocked_legs_total{reason} collapses reasons to a bounded label class: the raw reason embeds live numbers (stale(age=421s>ttl=600s)), i.e. unbounded label cardinality. The class is what an operator alerts on; the numbers stay in the structured log.
  • Keeper fleet. The fleet surface publishes the base-fee regime, the gate position with its hysteresis bounds, how long the gate has been shut, the longest deferral episode, discretionary pushes withheld, liveness pushes forced through a shut gate, and confirmed spend. spend_native is credited only from confirmed successful receipts: reverted pushes and unstick cancel transactions are not in it. forced_pushes non-zero is not a fault: liveness pushes relay at any price.

1.2 On-chain events

ClassEventsIndexable
SwapSwapped (packed: prices = mark and mid, outState = covToll, index and spreadPbps)yes
Liquidity and solvencySolvencyUpdated(cWad, navBase, claimBase), LiabilitySwapped, Withdrawnyes
Oracle pushnone: cadence from push tx calldata and receipts, lane state from storageno, lane state is read
Mark-store governanceAuthUpdated(tier, signers, k, relayers), OpQueued, OpCancelled, Migratedyes
Params and haltsAssetParamsUpdated, RiskFencesUpdated, BoundedAssetParamsUpdated, CurveUpdated, FeeParamsUpdated, AssetHalted, AssetUnhaltedno, snapshot trigger only
Lane lifecycleLaneListed, LaneHalted(lane, tierMask, halted), FeedLaneUpdated (factory)no, snapshot trigger only
Balance and index moversIndexUpdated(reason), Deposited, Donated, FlashLoanExecuted, TreasuryClaimed, Swept, UntrackedSynced, HookDustWrittenOffyes
  • Retuning and halting are not queryable from the indexer: they are snapshot triggers, not indexed topics.
  • A reanchor emits no event of its own beyond the store’s OpQueued; the lane word shows the anchor dropped (dark, markless) until the next push (oracle keeper).
  • A push names no lane. A monotonic skip (srcSecs <= obs at the lane) and a band refusal (the lane goes dark) are both read from the lane word, not subscribed.
  • Swapped packs its payload into five words (layout): amounts, fees, prices, outBook, outState. It carries markPrice, midPrice and covToll, so total LP revenue is reconstructible from logs alone. All three are tokenOut per tokenIn, chained over every leg, hence directly comparable to the realized amountOut/amountIn. OEV decomposes exactly on chain: with px the execution price, pm the mid and po the oracle mark,

OEV=px-pmpm,skew=pm-popo

The two prices are exact WAD, one 128-bit field each in prices and saturated at 2^128 - 1, not a packed float, so the reconstruction inherits no encoding error. covToll is withheld from the output before the fee and retained in the reserve, so a reader that ignores it books it as coverage drift rather than revenue.

  • Only LiabilitySwapped carries a haircut field (face - face · C when positive). A same-asset Withdrawn carries none; its shortfall below face is the coverage haircut face · (1 - min(c_leg, C)) plus the exit toll; previewWithdraw returns the net payout and the haircut, and the paired SolvencyUpdated and the leg’s books give the haircut too. SolvencyUpdated fires on every LP entrypoint of an armed pool; Pool.poolSolvencyWad() reads the pool rate live. cov.c and util.liq are served per asset.

1.3 Units and scales

spreadPbps (bits 224..239 of Swapped.outState) is PBPS, base 106, not bps. A column name does not indicate its scale. Two different divisors come off that one word. With S the spread in PBPS:

fbps=S200,Sbps=S100,minFeebps=minFeePbps100

QuantityFromDivisorScale
Realized feeSwapped.outState >> 224200bps
SpreadSwapped.outState >> 224100bps
minFeePbpsgetAsset100bps

The 200 is two conversions at once: 100 for PBPS to bps, and 2 because the fee is half the round-trip spread. A rate that is already a rate, such as minFeePbps, uses 100 and not 200, so comparing a realized fee against the floor means dividing the floor by 100 first. The constants are named SPREAD_PBPS_PER_FEE_BP and SPREAD_PBPS_PER_BP in one place. Snapshot writers resolve ABI fields by name, never by position: minFeeBps and minFeePbps differ by one letter and by a factor of 100. Realized fee bps and OEV bps are computed from the swap amounts, not from the spread word.

minFeePbps is the only fee rate getAsset returns, and it is a two-sided-bounded per-leg floor on the spread, not a target and not a ceiling. Everything else getAsset returns is inventory (reserves, liabilities, anchor, liquidityIndexWad, minLiquidity), shape (curveId, the dispersion band), or credit-cap and decimal metadata (vegaBps, depositCapCode, maxLiabWeightBps, decimals, deadSeedPow10). A dashboard therefore cannot plot a per-leg fee band from config and must not present one: the quoted spread has no upper bound, because a σ, confidence or staleness term that widens it is pricing real risk. See Spread and Fees.

2. Where to read it

SurfaceCarriesPublic
Contract eventsswap, solvency, oracle push, signer governance (§1.2)yes, on chain
Read-only gatewayAPI & SDK Referenceyes
Operator-internal telemetryingestion, keeper fleet, indexer integrity, bot preflightno
Operator alert channelguardian breach predicates (§3.1)no

Every health and metrics surface above is operator-internal. Their addresses, routes and transports are not published, are not reachable from the public internet, and are not part of any integration contract. Integrations must not depend on them. The public surfaces are on-chain state and the read-only gateway; every claim on this page is verifiable from those two.

2.1 The gate mirror

A swap’s first gate is FeedMathLib.gate(). It reverts on four conditions, checked in this order: feed halted, stale, mark zero, confidence over the halt threshold (oracle keeper). A revert emits nothing, and a leg refused at quote time never reaches a chain, so the off-chain mirror is the only trace. The indexer evaluates the same predicate off chain against the feed state it multicalls, and publishes the result on an operator-internal surface: how much of the book is tradable, which of the four gate conditions each blocked leg failed, and how far past its TTL the worst leg has drifted. σ=0 on an otherwise-healthy feed is treated as a defect and counted separately, because it is invisible in any freshness view.

Two correctness notes. Age is computed on the observed-at clock the contract gates on, tobs=min(tsrc,tupd), not on tupd alone: tupd is the relay landing time, so using it understates age by the relay lag and can report a feed fresh that the pool reverts on. And confidence, flags, maximum deviation and source timestamp are snapshotted alongside the gate result; without them the mirror sees only staleness, which is two of the four gate conditions, and an outage would be recorded only as a period of zero volume, without its cause.

The mirror is feed state, not pool state, so it does not see BaseDepegged, PriceOutsideRefBand or a halted asset flag (flow guards).

2.2 Fees, LP return and liquidity flow

Fee telemetry stores both legs’ fees, realized fee bps, execution price, LP return and the liquidity it was earned against, one row per Swapped log, deduplicated on the swap’s log identity. The asset is the primitive: asset APR is absolute fees over that asset’s liquidity, and pool APR is the mean of its assets’.

Fee APR is an annualization over an elapsed span, so a span shorter than one hour is not annualized at all: below that window the figure is null rather than a small number that looks like yield. The window is stored with the figure.

Volume counts hop 0 only, so a multi-pool route is one user swap. Fees count every hop, because each transited pool charged a fee. Both reductions run over the same event stream; the per-row hop index distinguishes them.

2.3 Indexer integrity

Every additive series is a sum over blocks, so a replayed range doubles it. The ingest loop guards this at startup by bumping the cursor past the indexed tip and publishes the guard on the internal health surface. With c the cursor and T the indexed tip:

ℓ=c-1-T≥0

  • ingest.cursor_lead is ℓ. Negative means the next scan replays indexed blocks and double counts every additive series.
  • ingest.cursor_behind_tip, the boolean form, for alerting.
  • ingest.last_event_age_sec, the age of the newest indexed event. This is the number that proves the indexer is still advancing.

Indexed series are confirmed to a per-chain depth chosen against that chain’s finality.

2.4 Who runs the nodes

The mark pipeline and the relays that push to it are a distributed network. The first nodes are run by the BTR team, with independent operators onboarded as the signer and relay sets widen.

The security argument rests on what the chain verifies, not on who runs a node. For BTR that is the k-of-n signature over each mark, the staleness and deviation gates that reject a bad push regardless of source (oracle keeper), and a relay path that carries no signing key and therefore cannot forge, only delay.

3. What alerts

3.1 Guardian predicates

The risk keeper’s guardian module (observe and alert only; the on-chain guardian is a separate role) evaluates these predicates and pages the on-call operator. Re-pages are cooled down per breach and key. The cooldown and the transport are operational details and are not published.

SeverityPredicates
Criticalmin_fee_below_two_theta, feed_sigma_zero, mark_expired, dev_cap_below_theta
Highband_below_floor, feed_unreadable, dev_cap_below_drift
Mediummark_aging, measurement_missing, measurement_stale

A standing Critical means a deployed parameter contradicts the live tape; retune via the risk steward.

3.2 Metric conditions

ConditionSurfaceWhat a breach means
tradable_ratio below 1.0 sustained: alert. Below 0.5: pageindexerthe fraction of the book that will revert on the next swap
feeds.sigma_zero above 0indexerσ=0 on an otherwise-healthy feed, treated as a defect (§2.1)
feeds.worst_age_ratio above 0.5indexerone missed heartbeat; above 1.0 the leg is dead
ingest.cursor_behind_tip trueindexerthe next scan double counts every additive series
ingest.last_event_age_sec unboundedindexerthe indexer has stopped advancing
bot_tradable_ratio below 1.0, bot_blocked_legs_total{reason} rising, stalled bot_gate_sweep_timestamp_secondsbotlegs are being refused, or the sweep itself stopped
realized fee bps below minFeePbps / 100indexermispriced risk
fee APR falling or unstableinternalLPs are not being paid for the flow they absorbed
liq.net.usd outflow not explicable by APRindexersilent exit ahead of a problem the other metrics have not surfaced
cov.c below 1.0indexersame-asset exits in kind are capped and the toll is being charged (flow guards)
gas gate shut, deferred_pushes unboundedkeeperthe fleet is priced out of the chain, or is burning budget
detect-to-broadcast p99 above 200 mskeeperpushes are late, so the mark is stale before it lands
mark-pipeline cosign readiness falseaggregatorquorum cannot be met; the keeper has nothing valid to push
mark-pipeline composite staleness above 0aggregatorthe index is quoting off decayed legs
mark-pipeline bar flat-fill ratio sustained above ~0.5aggregatorthe feed is not arriving and the chart is flat-filled
mark-pipeline per-venue tick age above 30 saggregatora venue is dark; the composite reweights onto the remaining venues