---
title: "Oracle Keeper Operations"
description: "Operator runbook for the mark-pushing relay: what the key can and cannot do, the startup gates, the revert taxonomy, and the checks before and after a deploy."
audience: tech
type: guide
status: live
lang: en
updated: "2026-09-04"
publish: true
---
# Oracle keeper operations

For the operator running `btr-keeper oracle`, the process that relays NX Rates-signed marks into
`ExternalOracle`. The push mechanism, the quorum design and the guard chain belong to
[Oracle Price-Push Security](/docs/3-6-oracle-price-push-security) and
[Oracles §4.2](/docs/3-4-oracles#42-signed-push-path-batchpushsigned-k-of-n-quorum); the metrics
belong to [Observability](/docs/3-8-observability). This page is what the person holding the relay
key does with it.

---

## 1. What the key is, and what it is not

The signed push path — `pushSignedV4(blob, sigs)` on the live V4, `batchPushSigned` on V1 — has **no
sender check**. Authority is the k-of-n signature set recovered from `sigs` against the granted
`signers` mapping; `msg.sender` is unpermissioned and pays gas. V4's session path `pushV4(blob)` does
pin `msg.sender`, to the relay named in a quorum-signed `SessionGrant`, but that grant is issued by
the quorum and revocable by any one signer, so it still confers no price authority. The keeper's key
is therefore a **funding key, not a price authority**, and it must not be the deployer, the owner, a
guardian, or an NX Rates attester.

**Nothing enforces that separation.** There is no startup gate comparing the relay address against
`AccessControl.owner()`, the guardian set or the signer set — it is an operator obligation, and on
Arc today it is violated: the owner EOA is also the live push sender. A keeper-host compromise there
is an owner-key compromise. Rotate the relay key to an address that holds no role.

| The relay key can | The relay key cannot |
|---|---|
| Land a validly co-signed blob, paying gas | Author a mark. It holds no attester key and the contract verifies signatures, not the sender |
| Choose *when* to push and *which* feeds ride along | Land a lane outside the deviation band, or a slot at a source second the slot has already written. Both fail per lane or per slot rather than reverting the blob |
| Stop pushing (which halts nothing, but ages every feed toward its TTL) | Pause, halt, widen, or un-halt anything. Those are guardian and owner levers ([Guardian Operations](/docs/3-9-guardian-operations)) |

The consequence that matters operationally: **losing the relay key is a liveness incident, not a
price incident.** Anybody can relay in its place. Conversely, no amount of relay-side care
compensates for a compromised attester quorum: that bound is the reference feed and the guardian,
not you.

---

## 2. When the keeper pushes

Three triggers, evaluated per feed against the last landed on-chain push:

| Trigger | Rule | Configured at |
|---|---|---|
| Deviation | $\lvert m - p_{\text{last}}\rvert / p_{\text{last}} > \kappa \cdot E_i(t)$, a fixed share of the edge the pool is currently quoting for that leg | `edge_kappa_pct` (50 on Arc); `feeds[].theta_bps` is the fallback when the pool's `minFeePbps` is unknown |
| Heartbeat | the per-feed maximum time between on-chain pushes has elapsed | `feeds[].heartbeat_s` |
| CI spike | the NX confidence interval has widened by at least `ci_spike_bps` since the last push | `nxr.ci_spike_bps` (required, 1–100) |

The deviation trigger is **edge-relative, not a static θ**. The full rationale, the formula for
$E_i(t)$ and the measured cadence are in
[Oracles §10.9](/docs/3-4-oracles#109-push-triggers-deviation-measured-against-the-spread-it-defends);
that section is authoritative, this one is the operator summary. `theta_bps` remains in the config as
the fallback path and is what `edge_kappa_pct = 0` selects.

The CI-spike trigger exists because a **pinned** mark does not move: a depeg onset that holds the
price flat while dispersion explodes trips neither the boundary nor the heartbeat.

A feed carrying no trigger of its own **rides along** on an already-due blob when its own deviation
already sits within `rider_boundary_pct` percent of its own push boundary (60 on Arc). The legacy
`rider_band_pct` test — proximity to the feed's on-chain `maxDeviation` — is the
`edge_kappa_pct = 0` branch only, and `maxDeviation` is roughly 200× the boundary on a stable, so it
carried no information about whether a feed was about to fire.

`heartbeat_s` is a **staleness bound the keeper enforces on itself**, not a liveness watchdog. The
ops rule coupling it to the contract is `ttl ≈ 2·heartbeat`, and the keeper hard-fails at startup on
any feed with `heartbeat_s > ttl/2` (override `KEEPER_ALLOW_LONG_HEARTBEAT=1`, which is a bring-up
tool, not a fix).

Push rate is bounded by three separate objects, and only the first is a spacing rule:

- **`burst_gap_s`** (1 s on Arc, valid range 1–36): the refractory a feed still holding hourly budget
  observes. The floor is 1 s because V4 accepts one write per slot per **source second**.
- **`feeds[].min_push_gap_s`**, at least `MIN_BATCH_PUSH_GAP_S` = 36 s and strictly below
  `heartbeat_s`: the refractory a feed falls back to once its hourly tokens are spent.
- **`CADENCE_CAP_PER_H` = 100**, a trailing-hour token bucket **per asset**, and
  `manifest_cap_per_h` = 360 on Arc, a trailing-hour average over whole **blobs**. Neither is a
  minimum spacing; `manifest_cap_per_h` must be ≥ `CADENCE_CAP_PER_H`.

Worst-case hourly spend is set by the buckets, not by the gaps.

θ, heartbeat, `min_push_gap_s`, `ttl` and `maxDeviationBps` are per-deployment and per-feed. There
is no protocol-wide number to quote: the first three live in the keeper's `oracle.<chain>.toml` (or
the corresponding tier of `fleet.<chainId>.yaml`), the last two are written on chain by
`registerFeed` and can only ever be tightened afterwards by `updateFeed`.

---

## 3. Both oracles, or neither

A deployment running the per-asset reference band carries **two** `ExternalOracle` instances at two
addresses: the primary that prices, and the independent reference that bounds cumulative drift
([Oracle Price-Push Security §5.1](/docs/3-6-oracle-price-push-security#51-per-push-band-vs-cumulative-band)).
Every non-base spoke bands against the reference and **fail-closes when the reference goes stale**,
so refreshing only the primary bricks every spoke swap once the reference passes its TTL.

The two cannot share a blob. The signed-push EIP-712 domain binds `verifyingContract`, so a blob
signed for the primary can never verify on the reference: each needs its own NX Rates-signed blob
bound to its own address, which in practice means a **second keeper instance** with its own config
(`oracle-reference.<chain>.toml`) and its own manifest. Fill and arm both in the same session. A
parked reference oracle strands every spoke once the reference passes its `ttl` — the gate reverts on
age above `ttlSecs`, not above `ttl/2`; `min(ttl/2, 30 s)` is only the premium-free grace before the
staleness surcharge starts widening the quote. It fails silently: the 2026-07
incident ran 4.5 days with 13 dead feeds before anyone noticed, which is why the feed-liveness guard
in §6 exists.

---

## 4. Startup gates

The keeper refuses to arm rather than pushing into a misconfiguration. Knowing what each gate
asserts turns a boot failure into a one-line diagnosis.

**Config load** (`deny_unknown_fields` throughout): zero or duplicate `feed_id`, a zero `oracle`
address, an unset `chain_id`, a missing `[gas]` policy or a missing `[alert]` block are all rejected.
`ci_spike_bps` (1–100), `rider_band_pct` (1–100) and `max_age_ms` (1–60000) are bounds-checked. The
`mainnet` profile additionally requires `chain_id == 1`.

**Chain**: `eth_chainId` is fetched and hard-compared against the configured `chain_id`. Both are
explicitly required; neither has a silent default.

**Feeds**: every configured `feed_id` is `getFeed()`-ed before the loop starts. A missing feed
aborts. On the V1 wire, `feedIdOf(tickerId)` is cross-checked against the configured `feed_id` so a
config typo cannot relay a mark into the wrong feed. On the V2 wire, `EPOCH()` must match the
resolved config epoch and each feed's on-chain `globalIndex` and `expBias` must match its lane
record, or marks would decode at the wrong scale.

**On the live wire v5 the arm is a lane reconcile only**: `feedIdAt(globalIndex)` must equal the
configured `feed_id` for every bound feed. There is no `EPOCH` gate, deliberately — V4 has no `EPOCH`
immutable, the clock is cyclic. There is also **no on-chain `expBias` cross-check**, because V4
exposes no per-feed config getter; a producer running a bias the chain has since changed is caught by
the deviation band, and only where the lane already holds a mark to band against. Reconcile the
declared bias against the `FeedRegistered` / `FeedExpBiasUpdated` logs, or against `expHeadroom`,
after any rebias.

**Quorum**: on-chain `signerThreshold` must equal the configured `signer_threshold`, on-chain
`signerCount` must equal the number of pinned `signers`, and every pinned signer must read granted.
The keeper pins the expected attester set explicitly; a silently added or removed key is a boot
failure, not a runtime surprise.

**Upstream**: the NX Rates signed catalog must cover every subscribed feed (a superset is fine), must
be bound to the configured oracle address and chain id, and must serve a quorum at least as large as
`signer_threshold`.

**Relay set**: when `keeper_set` holds more than one address, a deterministic soft leader relays each
push and the rest arm a jittered fallback (`relay_fallback_ms` + `relay_jitter_ms` × index), so a
stalled leader is covered without an O(N) reverting-transaction storm. The running keeper's own
address **must** appear in `keeper_set`; startup fails loudly otherwise.

**Anti-pick-off**: pools listed in `pools` have every asset's deployed `minFeePbps` checked against
2θ. Leaving `pools` empty turns the gate off, which is why the same invariant is also evaluated by
the risk keeper ([Risk Steward Operations §5](/docs/3-11-risk-steward-operations#5-the-standing-invariants)).

Live pushing needs **both** `--execute` and `KEEPER_EXECUTE=1`. Without both, the keeper runs the
full loop and broadcasts nothing, which is the correct way to validate a config change.

---

## 5. Revert taxonomy

**V4 is not all-or-nothing.** Only framing, quorum, session and acceptance-window failures revert the
whole push; everything price-shaped fails soft, per lane or per slot, and the rest of the blob lands.
Read the revert **and the accepted mask**, not the gas.

Reverts — the whole blob is discarded:

| Revert | Cause | Operator action |
|---|---|---|
| `BadBlobHeader()` | Wrong version byte, a length that does not match `11 + 5·nP + 5·nS + 3·nC`, a `gi` not strictly ascending inside a section, or a lane with a non-zero reserved top bit | Wire-format mismatch between NX Rates and the deployed contract. Stop and reconcile `wire_version` |
| `NotAuth()` | A recovered signer is not granted, the signatures are unsorted / duplicated, or a session push came from an address that is not the granted relay (or carried `seq > maxSeq`) | Compare `signers()` and `session()` on chain against the pinned set. Signatures must be sorted by recovered address |
| `SessionExpired()` | The session's `expiresAt` has passed | Open a new session; grants cap at 1 h |
| `StaleTimestamp()` | The header's reconstructed source time is older than `MAX_RECON_AGE` = 6 h | The blob was withheld or the fetch path is badly behind. Check `/latency`, not the chain |
| `FutureTimestamp()` | The header's source time leads `block.timestamp` by more than 5 s | Clock skew on the producer or the sequencer |
| `InvalidInput()` | `sigs.length % 65 != 0`, or fewer signatures than `signerThreshold` | A relay or quorum-assembly bug |
| `FeedNotFound(feedId)` | An admin call named a feed the instance does not carry | Config points at an instance that does not carry it |

Fail-soft — the push lands, the affected lane or slot does not:

| Outcome | Cause | How you see it |
|---|---|---|
| Lane skipped | Unregistered lane, a write of the STALE sentinel, a deviation-band breach, or σ above `MAX_SIGMA_PBPS` | Bit clear in `acceptedMask`, bit set in `LanesSkipped(slotId, laneMask)` |
| Lane accepted but not written | The feed is paused | Counts as accepted in `acceptedMask` while the lane is never written. Check `getFeed(feedId)` flags before suspecting the relay. Next release: `pauseFeed` clears the lane, so the feed reads DEAD for the pause and until a push lands after the unpause, and that push is banded over the whole pause rather than over one cadence |
| **Whole slot skipped, silently** | The blob's source second is not strictly newer than the slot's stored one | **No event at all.** The only signal is that the slot's eight lanes did not advance. This is what a re-broadcast, a reordered relay, or two pushes inside one source second look like |

**The band case is the one that self-heals, up to a ceiling.** The allowance is
`maxDeviationBps + min(6·σ·√(dt/1800), 9·maxDeviationBps)`, so it widens with the attested source-time
gap: a feed that fell behind during an outage clears itself once the true move fits inside the
widened band. Past `10·maxDeviationBps` it does not, and nothing the relay key holds moves it. The
release is the owner's, and **shipping in the next release** it is an in-contract one:
`requestFeedWiden(feedId, maxDeviationBps, ttlSecs)` → `BASE` delay → `executeFeedWiden(feedId)`,
guardian-or-owner `cancelFeedWiden` veto (§9). Until it lands, recovery is an oracle redeploy plus a
`BASE`-tier `UPDATE_ORACLE` repoint per affected leg. Formula and terms:
[Oracles §8.3](/docs/3-4-oracles#83-deviation-bounds).

V1's `CooldownActive()` (one mark per feed per block) and its `StaleData(age, bound)` against an
immutable `maxRelayLagSecs` do not exist on V4.

**V4 pushes are indexable.** Both entry points emit
`SlotsPushed(seq, sourceTsDs, acceptedMask, blobHash)`, and a record that fail-softed any lane also
emits `LanesSkipped(slotId, laneMask)`. Alert on a rising skip count and on a flat `acceptedMask`
bit; the silent whole-slot skip above is the one outcome no event covers, so watch slot age directly.

---

## 6. The feed-liveness guard

`btr-keeper guards` is a separate role with a separate key: it sweeps the **full feed catalog of
both** `ExternalOracle` instances, pages a human, then pauses what stays dead. It is the only
component that checks the reference oracle, which carries no pools and is therefore invisible to
every pool-scoped check.

Its heartbeat is not configured in its own file; it is read from the pusher's `oracle.*.toml` via
`--oracle-config`, so the heartbeat a guard checks is by construction the heartbeat the pusher
promises. Four gates bound a pause, and each reports distinctly, because "would have paused" and
"paused" must never read the same:

- The rolling `limits.max_auto_actions_per_hour` budget, consumed only by a landed broadcast.
- `--execute` **and** `GUARDS_EXECUTE=1`.
- An `AccessControl.isGuardian(signer)` read at startup.
- The broadcast itself.

Alerts are never bounded by the budget.

Load-time rules refuse a config that could go dark:

- At least one `alert` stage must exist.
- The earliest alert stage must fire strictly before the earliest pause stage.
- An armed load rejects zero addresses and rejects `reference_oracle == oracle`.
- `--run` requires one `--oracle-config` per address in `[contracts]`.

Because the pause leg signs as a guardian, the guard's key is a guardian key and inherits
[Guardian Operations](/docs/3-9-guardian-operations), including the fact that `unpauseFeed` is
owner-only, so a guard-driven pause always ends with a human.

---

## 7. Before a change

- **Never on a laptop.** A long-lived live process belongs to the cluster Deployment that owns
  pushes for that chain; a second live relay signing from a workstation is a duplicate-push source.
- **Dry-run first.** `--once` without `--execute` runs the full tick, exercises every startup gate,
  and broadcasts nothing. A failed or timed-out tick exits non-zero, so it is usable as a gate.
- **Reconcile ordinals against chain, not against the repo.** Offline tests recompute feed ordinals
  from the same deploy scripts the config was generated from, so they agree with the config by
  construction and stay silent when the *contract* disagrees. Only a chain read catches that.
- **Change both instances together** when a change touches the wire format, the epoch, the signer
  set or the feed roster (§3).
- **Check the fences you are about to load against**: `getFeed(feedId)` for each feed's live
  `ttlSecs` and `maxDeviationBps`. There is no per-feed lag bound to reconcile on V4 — the past bound
  is the contract constant `MAX_RECON_AGE` = 6 h, above every deployed ttl (600 / 3,600 / 7,200 s),
  and nothing validates ttl against it. Both live fields are tighten-only from every instant lever;
  the only inverse is the owner's timelocked wedge release (§9), so treat a tighten as a decision,
  not a setting.

---

## 8. After a change

Watch the push path end to end rather than the process:

- **Liveness**: the keeper marks a heartbeat after every successful tick and its probe restarts the
  process when that heartbeat goes stale. A total upstream outage returns an error rather than a
  silent green tick, precisely so the probe fires **before** the on-chain TTL halts pools.
- **Latency**: the health surface serves p50/p99 per leg. Quote a number only from there; the
  budgets are split per leg because the legs have different physics (quote fetch, pre-submission,
  submit).
- **Triggers**: it records one line per feed per relay decision, reports the effective per-feed θ
  and heartbeat actually in force, and reports gas spend and the low-balance gate.
- **On chain**: `getFeed(feedId).updatedAtSecs` advancing on every feed, on **both** oracles.
- **Downstream**: `feeds.tradableRatio` and `feeds.worstAgeRatio`, which are the fraction of the book
  that will revert on the next swap ([Observability §6](/docs/3-8-observability#6-pool-mark-state-and-the-gate-mirror)).

A newly registered feed deserves one extra look, and on V4 for the opposite reason. `registerFeed`
seeds **no mark and no σ**: it writes the config lane, stamps the slot clock and leaves the price lane
at the STALE sentinel, so the feed reads stale and its **first push carries no deviation band at all**
— there is no previous mark to band it against. Get that first push in under supervision and verify
the landed mark against the source before the leg is listed. Every later push is banded normally, and
one that drifts past `10·maxDeviationBps` is back to the same unbanded first push — reachable only
through the owner's wedge release (§9), never from this key.

---

## 9. Escalate

| Symptom | Escalate to |
|---|---|
| Signature verification failing against a set that should be granted, or an attester key believed leaked | Guardian: `revokeSigner` is immediate and halts pushing, which is the fail-safe ([Guardian Operations §4](/docs/3-9-guardian-operations#4-what-you-are-the-last-line-of-defence-against)) |
| A feed pushing authentic marks you believe are wrong | Guardian: `pauseFeed` |
| Band permanently too tight after an outage (past `10·maxDeviationBps`) | Owner. Shipping in the next release: `requestFeedWiden` → `BASE` delay → `executeFeedWiden`, which clears the lane and the band anchor so the next push lands unbanded; the magnitude is optional, so a pure release passes the live band and ttl back verbatim. Until it ships, a fresh oracle deployment plus a `BASE`-tier `UPDATE_ORACLE` repoint per affected leg — 37 legs on Arc. Plan hours, not minutes |
| `ttlSecs` or `maxDeviationBps` structurally wrong for the deployment | Owner: `updateFeed` tightens either instantly; loosening either is the timelocked widen above, which carries both fields on one op. Too-loose is a live-with-it until the `BASE` delay clears |
| Both oracles stale simultaneously | Treat as an incident: every spoke is fail-closed. `security@btr.markets` |

Nothing on this list is reachable from the relay key. Recognising that early is most of the job.

**When you hand over a wedge, hand it over grouped by slot.** `executeFeedWiden` stamps the one clock
the slot's eight lanes share, so every wedged lane of a stalled slot must be released in the same
batch; a lane left out reads age ~0 again on the next accepted push and is fail-*open* on its own leg
until its own push or release lands. The slot then skips until a blob sourced strictly after the
execute arrives — observe-sign-relay lag plus one cadence, about 30 s at the live rate, for all eight
lanes, so expect one dark cycle across the slot and do not read it as a relay fault. The off-chain
ceremony is `script/OracleV4Unwedge.s.sol`; its `preview()` and `verify()` take no key, broadcast
nothing, and print `UNRELEASED MATE` for any wedged lane the selection leaves behind. After the feed
quotes again, re-tighten with `updateFeed` if the ceremony widened: the release leaves the band where
the request set it.

---

## 10. Checklist

**Before arming an instance**

- [ ] `chain_id` in the config matches the RPC and the target deployment.
- [ ] `oracle` address matches the instance this tier is meant to feed, and the NX Rates catalog is
      bound to that same address.
- [ ] `signers` and `signer_threshold` match `signers()` / `signerThreshold()` on chain.
- [ ] Every `feed_id` resolves via `getFeed()`, and the lane reconcile agrees with the config: V1
      `feedIdOf(tickerId)`, V2/V4-wire the lane record, wire v5 `feedIdAt(globalIndex)`.
- [ ] On wire v5, the declared `expBias` per feed matches the chain — the arm does **not** check it,
      so read it off `FeedRegistered` / `FeedExpBiasUpdated` logs or `expHeadroom`.
- [ ] `heartbeat_s ≤ ttl/2` on every feed, with no `KEEPER_ALLOW_LONG_HEARTBEAT` override in the
      manifest.
- [ ] `min_push_gap_s ≥ 36` and `< heartbeat_s` on every feed.
- [ ] `[gas]` and `[alert]` are present, and the pager has been tested to a real inbox.
- [ ] The signing address appears in `keeper_set`, and is not the deployer, owner, guardian, or an
      attester. Nothing on the keeper checks this — read `AccessControl.owner()`, `isGuardian` and
      `signers()` against it by hand.
- [ ] The **reference** instance is filled and armed in the same session as the primary.
- [ ] `--once` dry-run exits zero.

**Before switching to live**

- [ ] `--execute` and `KEEPER_EXECUTE=1` are both set, and set nowhere else.
- [ ] No second live relay for the same tier is running anywhere, workstation included.
- [ ] The liveness probe and the heartbeat the keeper writes agree on the same path.

**After the first live tick**

- [ ] `getFeed().updatedAtSecs` advancing on every feed, on both oracles.
- [ ] Per-leg p99 within budget, and the trigger mix on the health surface as expected.
- [ ] Gas spend and the low-balance gate sane for the cadence.
- [ ] `feeds.tradableRatio` at 1.0 and `feeds.sigmaZero` at 0.
- [ ] Any feed not advancing checked for flags bit 0 (paused) before the relay is blamed.

---

## 11. Related

| Page | Content |
|---|---|
| [Oracle Price-Push Security](/docs/3-6-oracle-price-push-security) | Quorum design, the guard chain, signer ceremony |
| [Oracles](/docs/3-4-oracles) | FeedData, band formula, staleness, V2 |
| [Observability](/docs/3-8-observability) | Every metric named above, with healthy ranges |
| [Guardian Operations](/docs/3-9-guardian-operations) | The levers this role escalates to |
| [Risk Steward Operations](/docs/3-11-risk-steward-operations) | The other bounded key, and the shared 2θ invariant |
| [Depeg Halt](/docs/3-5-depeg-halt) | What a stale reference feed actually blocks |