Oracle Keeper Operations

For the operator running btr-keeper oracle, the process that relays NX Rates-signed marks into ExternalOracle. The push mechanism, the quorum design and the guard chain belong to Oracle Price-Push Security and Oracles §4.2; the metrics belong to Observability. This page is what the person holding the relay key does with it.

1. What the key is, and what it is not

The signed push path — pushSignedV4(blob, sigs) on the live V4, batchPushSigned on V1 — has no sender check. Authority is the k-of-n signature set recovered from sigs against the granted signers mapping; msg.sender is unpermissioned and pays gas. V4’s session path pushV4(blob) does pin msg.sender, to the relay named in a quorum-signed SessionGrant, but that grant is issued by the quorum and revocable by any one signer, so it still confers no price authority. The keeper’s key is therefore a funding key, not a price authority, and it must not be the deployer, the owner, a guardian, or an NX Rates attester.

Nothing enforces that separation. There is no startup gate comparing the relay address against AccessControl.owner(), the guardian set or the signer set — it is an operator obligation, and on Arc today it is violated: the owner EOA is also the live push sender. A keeper-host compromise there is an owner-key compromise. Rotate the relay key to an address that holds no role.

The consequence that matters operationally: losing the relay key is a liveness incident, not a price incident. Anybody can relay in its place. Conversely, no amount of relay-side care compensates for a compromised attester quorum: that bound is the reference feed and the guardian, not you.

2. When the keeper pushes

Three triggers, evaluated per feed against the last landed on-chain push:

The deviation trigger is edge-relative, not a static θ. The full rationale, the formula for $E_i\left(t\right)$ and the measured cadence are in Oracles §10.9; that section is authoritative, this one is the operator summary. theta_bps remains in the config as the fallback path and is what edge_kappa_pct = 0 selects.

The CI-spike trigger exists because a pinned mark does not move: a depeg onset that holds the price flat while dispersion explodes trips neither the boundary nor the heartbeat.

A feed carrying no trigger of its own rides along on an already-due blob when its own deviation already sits within rider_boundary_pct percent of its own push boundary (60 on Arc). The legacy rider_band_pct test — proximity to the feed’s on-chain maxDeviation — is the edge_kappa_pct = 0 branch only, and maxDeviation is roughly 200× the boundary on a stable, so it carried no information about whether a feed was about to fire.

heartbeat_s is a staleness bound the keeper enforces on itself, not a liveness watchdog. The ops rule coupling it to the contract is ttl ≈ 2·heartbeat, and the keeper hard-fails at startup on any feed with heartbeat_s > ttl/2 (override KEEPER_ALLOW_LONG_HEARTBEAT=1, which is a bring-up tool, not a fix).

Push rate is bounded by three separate objects, and only the first is a spacing rule:

• burst_gap_s (1 s on Arc, valid range 1–36): the refractory a feed still holding hourly budget observes. The floor is 1 s because V4 accepts one write per slot per source second.
• feeds[].min_push_gap_s, at least MIN_BATCH_PUSH_GAP_S = 36 s and strictly below heartbeat_s: the refractory a feed falls back to once its hourly tokens are spent.
• CADENCE_CAP_PER_H = 100, a trailing-hour token bucket per asset, and manifest_cap_per_h = 360 on Arc, a trailing-hour average over whole blobs. Neither is a minimum spacing; manifest_cap_per_h must be ≥ CADENCE_CAP_PER_H.

Worst-case hourly spend is set by the buckets, not by the gaps.

θ, heartbeat, min_push_gap_s, ttl and maxDeviationBps are per-deployment and per-feed. There is no protocol-wide number to quote: the first three live in the keeper’s oracle.<chain>.toml (or the corresponding tier of fleet.<chainId>.yaml), the last two are written on chain by registerFeed and can only ever be tightened afterwards by updateFeed.

3. Both oracles, or neither

A deployment running the per-asset reference band carries two ExternalOracle instances at two addresses: the primary that prices, and the independent reference that bounds cumulative drift (Oracle Price-Push Security §5.1). Every non-base spoke bands against the reference and fail-closes when the reference goes stale, so refreshing only the primary bricks every spoke swap once the reference passes its TTL.

The two cannot share a blob. The signed-push EIP-712 domain binds verifyingContract, so a blob signed for the primary can never verify on the reference: each needs its own NX Rates-signed blob bound to its own address, which in practice means a second keeper instance with its own config (oracle-reference.<chain>.toml) and its own manifest. Fill and arm both in the same session. A parked reference oracle strands every spoke once the reference passes its ttl — the gate reverts on age above ttlSecs, not above ttl/2; min(ttl/2, 30 s) is only the premium-free grace before the staleness surcharge starts widening the quote. It fails silently: the 2026-07 incident ran 4.5 days with 13 dead feeds before anyone noticed, which is why the feed-liveness guard in §6 exists.

4. Startup gates

The keeper refuses to arm rather than pushing into a misconfiguration. Knowing what each gate asserts turns a boot failure into a one-line diagnosis.

Config load (deny_unknown_fields throughout): zero or duplicate feed_id, a zero oracle address, an unset chain_id, a missing [gas] policy or a missing [alert] block are all rejected. ci_spike_bps (1–100), rider_band_pct (1–100) and max_age_ms (1–60000) are bounds-checked. The mainnet profile additionally requires chain_id == 1.

Chain: eth_chainId is fetched and hard-compared against the configured chain_id. Both are explicitly required; neither has a silent default.

Feeds: every configured feed_id is getFeed()-ed before the loop starts. A missing feed aborts. On the V1 wire, feedIdOf(tickerId) is cross-checked against the configured feed_id so a config typo cannot relay a mark into the wrong feed. On the V2 wire, EPOCH() must match the resolved config epoch and each feed’s on-chain globalIndex and expBias must match its lane record, or marks would decode at the wrong scale.

On the live wire v5 the arm is a lane reconcile only: feedIdAt(globalIndex) must equal the configured feed_id for every bound feed. There is no EPOCH gate, deliberately — V4 has no EPOCH immutable, the clock is cyclic. There is also no on-chain expBias cross-check, because V4 exposes no per-feed config getter; a producer running a bias the chain has since changed is caught by the deviation band, and only where the lane already holds a mark to band against. Reconcile the declared bias against the FeedRegistered / FeedExpBiasUpdated logs, or against expHeadroom, after any rebias.

Quorum: on-chain signerThreshold must equal the configured signer_threshold, on-chain signerCount must equal the number of pinned signers, and every pinned signer must read granted. The keeper pins the expected attester set explicitly; a silently added or removed key is a boot failure, not a runtime surprise.

Upstream: the NX Rates signed catalog must cover every subscribed feed (a superset is fine), must be bound to the configured oracle address and chain id, and must serve a quorum at least as large as signer_threshold.

Relay set: when keeper_set holds more than one address, a deterministic soft leader relays each push and the rest arm a jittered fallback (relay_fallback_ms + relay_jitter_ms × index), so a stalled leader is covered without an O(N) reverting-transaction storm. The running keeper’s own address must appear in keeper_set; startup fails loudly otherwise.

Anti-pick-off: pools listed in pools have every asset’s deployed minFeePbps checked against 2θ. Leaving pools empty turns the gate off, which is why the same invariant is also evaluated by the risk keeper (Risk Steward Operations §5).

Live pushing needs both --execute and KEEPER_EXECUTE=1. Without both, the keeper runs the full loop and broadcasts nothing, which is the correct way to validate a config change.

5. Revert taxonomy

V4 is not all-or-nothing. Only framing, quorum, session and acceptance-window failures revert the whole push; everything price-shaped fails soft, per lane or per slot, and the rest of the blob lands. Read the revert and the accepted mask, not the gas.

Reverts — the whole blob is discarded:

Fail-soft — the push lands, the affected lane or slot does not:

The band case is the one that self-heals, up to a ceiling. The allowance is maxDeviationBps + min(6·σ·√(dt/1800), 9·maxDeviationBps), so it widens with the attested source-time gap: a feed that fell behind during an outage clears itself once the true move fits inside the widened band. Past 10·maxDeviationBps it does not, and nothing the relay key holds moves it. The release is the owner’s, and shipping in the next release it is an in-contract one: requestFeedWiden(feedId, maxDeviationBps, ttlSecs) → BASE delay → executeFeedWiden(feedId), guardian-or-owner cancelFeedWiden veto (§9). Until it lands, recovery is an oracle redeploy plus a BASE-tier UPDATE_ORACLE repoint per affected leg. Formula and terms: Oracles §8.3.

V1’s CooldownActive() (one mark per feed per block) and its StaleData(age, bound) against an immutable maxRelayLagSecs do not exist on V4.

V4 pushes are indexable. Both entry points emit SlotsPushed(seq, sourceTsDs, acceptedMask, blobHash), and a record that fail-softed any lane also emits LanesSkipped(slotId, laneMask). Alert on a rising skip count and on a flat acceptedMask bit; the silent whole-slot skip above is the one outcome no event covers, so watch slot age directly.

6. The feed-liveness guard

btr-keeper guards is a separate role with a separate key: it sweeps the full feed catalog of both ExternalOracle instances, pages a human, then pauses what stays dead. It is the only component that checks the reference oracle, which carries no pools and is therefore invisible to every pool-scoped check.

Its heartbeat is not configured in its own file; it is read from the pusher’s oracle.*.toml via --oracle-config, so the heartbeat a guard checks is by construction the heartbeat the pusher promises. Four gates bound a pause, and each reports distinctly, because “would have paused” and “paused” must never read the same:

• The rolling limits.max_auto_actions_per_hour budget, consumed only by a landed broadcast.
• --execute and GUARDS_EXECUTE=1.
• An AccessControl.isGuardian(signer) read at startup.
• The broadcast itself.

Alerts are never bounded by the budget.

Load-time rules refuse a config that could go dark:

• At least one alert stage must exist.
• The earliest alert stage must fire strictly before the earliest pause stage.
• An armed load rejects zero addresses and rejects reference_oracle == oracle.
• --run requires one --oracle-config per address in [contracts].

Because the pause leg signs as a guardian, the guard’s key is a guardian key and inherits Guardian Operations, including the fact that unpauseFeed is owner-only, so a guard-driven pause always ends with a human.

7. Before a change

• Never on a laptop. A long-lived live process belongs to the cluster Deployment that owns pushes for that chain; a second live relay signing from a workstation is a duplicate-push source.
• Dry-run first. --once without --execute runs the full tick, exercises every startup gate, and broadcasts nothing. A failed or timed-out tick exits non-zero, so it is usable as a gate.
• Reconcile ordinals against chain, not against the repo. Offline tests recompute feed ordinals from the same deploy scripts the config was generated from, so they agree with the config by construction and stay silent when the contract disagrees. Only a chain read catches that.
• Change both instances together when a change touches the wire format, the epoch, the signer set or the feed roster (§3).
• Check the fences you are about to load against: getFeed(feedId) for each feed’s live ttlSecs and maxDeviationBps. There is no per-feed lag bound to reconcile on V4 — the past bound is the contract constant MAX_RECON_AGE = 6 h, above every deployed ttl (600 / 3,600 / 7,200 s), and nothing validates ttl against it. Both live fields are tighten-only from every instant lever; the only inverse is the owner’s timelocked wedge release (§9), so treat a tighten as a decision, not a setting.

8. After a change

Watch the push path end to end rather than the process:

• Liveness: the keeper marks a heartbeat after every successful tick and its probe restarts the process when that heartbeat goes stale. A total upstream outage returns an error rather than a silent green tick, precisely so the probe fires before the on-chain TTL halts pools.
• Latency: the health surface serves p50/p99 per leg. Quote a number only from there; the budgets are split per leg because the legs have different physics (quote fetch, pre-submission, submit).
• Triggers: it records one line per feed per relay decision, reports the effective per-feed θ and heartbeat actually in force, and reports gas spend and the low-balance gate.
• On chain: getFeed(feedId).updatedAtSecs advancing on every feed, on both oracles.
• Downstream: feeds.tradableRatio and feeds.worstAgeRatio, which are the fraction of the book that will revert on the next swap (Observability §6).

A newly registered feed deserves one extra look, and on V4 for the opposite reason. registerFeed seeds no mark and no σ: it writes the config lane, stamps the slot clock and leaves the price lane at the STALE sentinel, so the feed reads stale and its first push carries no deviation band at all — there is no previous mark to band it against. Get that first push in under supervision and verify the landed mark against the source before the leg is listed. Every later push is banded normally, and one that drifts past 10·maxDeviationBps is back to the same unbanded first push — reachable only through the owner’s wedge release (§9), never from this key.

9. Escalate

Nothing on this list is reachable from the relay key. Recognising that early is most of the job.

When you hand over a wedge, hand it over grouped by slot. executeFeedWiden stamps the one clock the slot’s eight lanes share, so every wedged lane of a stalled slot must be released in the same batch; a lane left out reads age ~0 again on the next accepted push and is fail-open on its own leg until its own push or release lands. The slot then skips until a blob sourced strictly after the execute arrives — observe-sign-relay lag plus one cadence, about 30 s at the live rate, for all eight lanes, so expect one dark cycle across the slot and do not read it as a relay fault. The off-chain ceremony is script/OracleV4Unwedge.s.sol; its preview() and verify() take no key, broadcast nothing, and print UNRELEASED MATE for any wedged lane the selection leaves behind. After the feed quotes again, re-tighten with updateFeed if the ceremony widened: the release leaves the band where the request set it.

10. Checklist

Before arming an instance

• chain_id in the config matches the RPC and the target deployment.
• oracle address matches the instance this tier is meant to feed, and the NX Rates catalog is bound to that same address.
• signers and signer_threshold match signers() / signerThreshold() on chain.
• Every feed_id resolves via getFeed(), and the lane reconcile agrees with the config: V1 feedIdOf(tickerId), V2/V4-wire the lane record, wire v5 feedIdAt(globalIndex).
• On wire v5, the declared expBias per feed matches the chain — the arm does not check it, so read it off FeedRegistered / FeedExpBiasUpdated logs or expHeadroom.
• heartbeat_s ≤ ttl/2 on every feed, with no KEEPER_ALLOW_LONG_HEARTBEAT override in the manifest.
• min_push_gap_s ≥ 36 and < heartbeat_s on every feed.
• [gas] and [alert] are present, and the pager has been tested to a real inbox.
• The signing address appears in keeper_set, and is not the deployer, owner, guardian, or an attester. Nothing on the keeper checks this — read AccessControl.owner(), isGuardian and signers() against it by hand.
• The reference instance is filled and armed in the same session as the primary.
• --once dry-run exits zero.

Before switching to live

• --execute and KEEPER_EXECUTE=1 are both set, and set nowhere else.
• No second live relay for the same tier is running anywhere, workstation included.
• The liveness probe and the heartbeat the keeper writes agree on the same path.

After the first live tick

• getFeed().updatedAtSecs advancing on every feed, on both oracles.
• Per-leg p99 within budget, and the trigger mix on the health surface as expected.
• Gas spend and the low-balance gate sane for the cadence.
• feeds.tradableRatio at 1.0 and feeds.sigmaZero at 0.
• Any feed not advancing checked for flags bit 0 (paused) before the relay is blamed.

11. Related