Budget governor¶
Implemented in src/pr_review_agent/budget.py, over the ledger table
declared in STORAGE.md. The design decisions behind it,
including what was cut and why, are recorded in
the design note.
Because billing is subscription mode, the governor is not optional. All Claude surfaces share one usage pool, so an unbounded reviewer does not merely overspend โ it locks the host operator out of their own interactive Claude Code sessions until the window resets. That is why it lands before the review worker: the spending rails exist before anything can spend.
๐ช Five layers, cheapest first¶
| Layer | Mechanism | State |
|---|---|---|
| 1 | Allowlist, bot filter, draft skip, cold-start watermark | done โ TRIGGERS.md |
| 2 | Path exclusions, diff-size caps, pre-flight token estimate | done โ WORKSPACE.md and below |
| 3 | Per-run ceiling: max tokens, turn cap, wall-clock timeout | done, with one gap โ see below |
| 4 | Rolling windows and pacing, by reserve-then-settle | done |
| 5 | A degradation ladder rather than a hard stop | done |
| 6 | A circuit breaker, converging onto the limit nobody publishes | done โ below |
Three more bound spend without being layers, because they are properties of how the agent runs rather than decisions about one claim. They are named here so that "every control on spend" is one list rather than four pages:
| Control | What it bounds | Where |
|---|---|---|
worker.count |
How many reviews can be in flight at once, and so how fast the windows can be drawn down | WORKER.md |
engine.timeout_seconds |
The wall clock one run cannot outlive, which is the only bound when an engine reports no usage | ENGINE.md |
queue.max_attempts |
How often one trigger may be retried, and so how many times a single failure can be paid for | QUEUE.md |
And one control is a layer decision about a claim, but about a claim's pull request rather than its windows, so it has a section of its own:
| Control | What it bounds | Where |
|---|---|---|
budget.min_review_interval_seconds |
How often one pull request may be reviewed | below |
budget.mention_min_review_interval_seconds |
The same, when a person asked directly | below |
budget.max_reviews_per_pull_request |
How many reviews one pull request may have in a day | below |
Layer 1 is the classifier โ it is the first budget layer, which is why its rejections are logged at a level an operator actually sees.
Where layer 3's three ceilings ended up¶
This layer was specified against an SDK, and the move to
a subprocess seam scattered it. None of the three is a
budget key, and one of them is not enforced at all.
Max tokens is max_run_tokens, reserved up front by admit. Done.
The turn cap is ours, not a flag. Inside one run the CLI ends an
over-long conversation itself, with error_max_turns, which the adapter maps
to TRUNCATED. Across runs, queue.DEFAULT_MAX_ATTEMPTS bounds how often a
single trigger can reach an engine at all, and each attempt reserves again
through admit. A budget.max_turns key would be a second bound on an
already-bounded quantity.
The wall clock is engine.timeout_seconds, enforced by the subprocess
boundary and โ since layer 3 โ validated at startup as strictly below
queue.DEFAULT_LEASE. That relationship used to be a comment. It matters
because the lease carries an expiry rather than a heartbeat precisely
because a run cannot outlive its clock: break it and the lease lapses under a
live worker, a second worker reserves against the same windows, and the first
one's settle discards a review that was paid for.
The gap: a reservation is still a forecast, not a ceiling. Nothing stops a
run spending more than it reserved, which is why
the exhausted rung is reachable only by an
overrun. Every way to close it assumes the engine reports tokens โ
--max-budget-usd is denominated in dollars, a wall-clock-to-token conversion
is a fabricated rate, and watching usage stream by needs an engine that
streams usage โ while Capabilities.usage_reporting
exists because some engine will not report any. Building the strong control
first and the universal one never is the wrong order, so the clock ships and
the token ceiling goes to
#20, whose
breaker is the right shape for a bound nobody can enforce mid-run. The
argument is recorded in
the layer 3 design note.
Layer 2 needs only a diff, which is why it lands before the engine rather than with it.
The diff-size caps arrived first, with the workspace.
max_changed_files and max_changed_lines both exist because neither bounds
the other โ two thousand one-line files pass a line cap and still bury the
engine, and one fifty-thousand-line generated file passes a file cap.
They are in this section, rather than beside the code that reads them,
because every spending cap belongs in one place. Being here also makes them
reloadable on SIGHUP, which is why the workspace is handed them per
checkout rather than holding a snapshot.
Be clear about what they bound: the engine's input, not the disk. A fetch pulls every object reachable from the head, so a commit that adds a large blob and a later one that removes it still downloads it while reporting no changed lines at all.
๐ซ Path exclusions¶
1 2 3 4 | |
Lockfiles, vendored trees, generated code and minified bundles. Reviewing
them is close to worthless and they dominate diff size, which makes this the
largest saving available for zero tokens. The built-in list lives in the
package, config/excluded_paths.py, so a config file does not repeat it; its
generated-code globs โ Protocol Buffers, OpenAPI/Swagger stubs, GraphQL
codegen, gRPC stubs, Go generators โ are copied from pr-agent's
settings/generated_code_ignore.toml at commit 10bbd9a, under the MIT
licence, with attribution in the module.
Exclusions apply to the size caps and to the diff the engine is shown, and
they are the same list applied by the same mechanism. Each pattern becomes
a git pathspec โ :(exclude,glob)<pattern> โ passed to both the
git diff --numstat the caps are measured on and the git diff that
produces Checkout.diff. They cannot drift apart, because there is only one
of them.
That second half holds by construction rather than by care:
ENGINE.md's ReviewRequest carries the checkout and
deliberately does not repeat the diff beside it, so Checkout.diff is the
only diff in the system.
Both halves are needed. Applying exclusions to the engine alone would leave a vendored-dependency bump refused on a size cap for lines the engine was never going to see โ a pull request rejected for content that does not exist.
excluded_paths adds to the built-in list rather than replacing it, so
a config file names only what is special about its repository.
default_exclusions: false drops the built-ins and leaves excluded_paths
alone in force: a repository that genuinely reviews its lockfiles is a real
repository. A pattern may not begin with :, because the
pathspec magic is the agent's to supply and a pattern that rewrites its own
meaning is not something an operator can predict from reading their own
configuration file.
The size gate moved to make this possible¶
It now runs after the fetch, on --numstat, rather than before it on the
API's totals. That is not a preference; GET /pulls/{n} reports three
aggregate integers with no per-path breakdown, and a lockfile cannot be
subtracted from an integer.
So "refused before anything is written to disk" is now "refused before a worktree exists and before any diff can reach an engine". What was given up is affordable: a fetch costs bandwidth and disk, and these caps are a spending control over tokens. The section above already conceded that they bound what the engine reads rather than what the fetch downloads โ the property surrendered was never the one doing the work.
The alternative was GET /pulls/{n}/files, which paginates to thirty
requests per claimed trigger, truncates at three thousand files, and is the
endpoint WORKSPACE.md
already rejected for the diff.
On an incremental round the caps measure the narrower range. A pull
request whose whole diff is over max_changed_lines is reviewed when the
commits since the last completed round fit under it, and a fixup that is
itself over the cap is still refused. This widens what may be reviewed, and
tests/test_workspace_since.py pins the bound: the caps apply to exactly the
range the engine is shown, because one variable feeds both git diff calls.
A refusal logs both figures โ what survived exclusion and what the API reported. "Refused at 3 files" is baffling beside a pull request GitHub says has 900; the gap between the two numbers is the explanation.
๐ฎ The pre-flight token estimate¶
The last refusal that costs nothing. Governor.preflight predicts a run's
cost and refuses it when the prediction exceeds max_run_tokens โ or when
nothing is left to review after exclusions, which a lockfile-only pull
request now is.
The worker calls it once, after the checkout and immediately before the engine: the tree has to be on disk for the reviewable line count to exist, and nothing may be spent after it says no. A refused row is abandoned rather than retried, since the same head predicts the same cost.
1 | |
No fitted intercept. A review has a fixed overhead โ the prompt, the instructions, the first file read โ but the only question asked here is whether a pull request is large enough to refuse, and under-predicting a fifty-line change is harmless because a fifty-line change is nowhere near the cap. The fixed cost is amortised into the rate, where it makes large diffs predict slightly high: the safe direction.
rate is fitted against the ledger โ ฮฃ used_tokens รท ฮฃ reviewed_lines โ
over settled rows whose usage_confidence is exact. Rows from an engine
that reported no usage are excluded, because fitting a rate to a run nobody
measured would turn the known-weaker guarantee into a
confidently wrong number.
reviewed_lines is the column migration 5 added, and it is written by
settle from Checkout.reviewed โ never by the engine.
ReviewResult.usage is budget.Usage, so putting the field there would
make an adapter responsible for reporting the size it was handed, and an
adapter that under-reported would bias the rate downward. A spending control
must not take its input from the thing it controls.
Only a completed review writes it. A truncated run was cut off with work
outstanding, a failed one produced nothing, and a usage-limited one settles
at exact zero โ each spent less than reviewing those lines actually costs,
so each would fit a rate below the truth and the estimate would refuse less
than it should. Every other run leaves the column NULL, which is how a row
says nothing about tokens per line rather than saying something wrong.
Only a full round is a sample. An incremental
round pays the same fixed
overhead โ the instructions, the earlier findings, the reviewer reading the
tree โ over fewer lines, so its tokens per line run high, and a fit mixing
the two would drift with the ratio of pushes to first reviews rather than
with what a review costs. settle writes reviewed_since, the head an
incremental round diffed from, and the fit reads only rows where it is NULL.
Every row written before the column existed was a full round, which is what
NULL says. The rate predicts a whole pull request, which is what the cap
refuses; an incremental round is still checked against it, on its own
narrower line count.
An incremental round with nothing reviewable in it โ usually a force-push that changed no content โ is refused for free like any empty pull request, but its notice says nothing has changed since the previous head rather than that every path is excluded, which would be false.
The cold start errs high¶
A fresh database has no rows to fit against, and the first runs are exactly when an over-estimate is cheapest to get wrong. So 40 tokens per line until ten fittable rows exist, after which the fit takes over outright.
Forty is above what a review is expected to cost, which over-refuses rather
than overspending; against the shipped max_run_tokens it puts the threshold
at 1,500 reviewable lines. Neither number is configurable: an operator's
escape hatch is max_run_tokens, which they already have to choose, and a
second knob multiplying into it is CONFIG.md's failure mode.
Declining to estimate until data existed was the alternative, and it leaves
the least-calibrated moment unguarded โ a run that truly costs 200,000
tokens against a 60,000 reservation completes, and the overrun surfaces at
settle, after the tokens are gone. That overrun is precisely what reaches
exhausted.
A permanent floor โ max(fitted, 40) โ was also rejected. After five hundred
runs proving reviews cost twelve tokens a line it would still refuse pull
requests the agent has direct evidence it can afford. A rate that can only be
revised upward is not a fit.
One run is not one turn¶
Worth knowing before the fit is trusted too far. The claude adapter asks
for schema-constrained output, and the CLI re-prompts by itself when the
model's answer does not fit โ see
ENGINE.md. That
retry is not something the adapter can disable or bound.
The ledger stays honest, because the envelope's usage covers every attempt. The fit stays honest too, for the same reason: it is fitted against what runs actually cost, retries included, so a corpus with a normal retry rate predicts a normal retry rate.
What this does affect is variance. A run that hits the retry path costs a
multiple of one that does not, over the same reviewable lines, so the residual
around the fitted rate is wider than a line-count model suggests โ and the
estimate is a refusal threshold rather than a reservation, which is the
reading that survives that variance. The reservation is still
max_run_tokens, and it is what the retry path is actually bounded by.
Refusing releases the reservation¶
The reservation is taken by admit, inside the claim, before the pull
request's size is knowable โ the facts read happens once per claimed
trigger, because reading it per open pull request per cycle is the design
POLLER.md exists to refuse. So a pre-flight refusal cannot
happen before a reservation exists; what it must do instead is hand the
reservation straight back.
preflight settles the row at zero tokens in the same call as the
refusal. A caller that refused and forgot to settle would leave a full
reservation charged against every window until it aged out, because
nothing releases a reservation early
by design. Making the decision and the release one call means that failure
cannot be introduced by a caller.
A refused row records used_tokens = 0 at confidence exact โ the cost is
not unknown, it is known to be nothing โ and leaves reviewed_lines NULL, so
a refusal never contributes to the fit.
A refusal says so on the pull request¶
preflight returns the sentence to post, or None to go ahead, rather
than a bool. A bool was all the worker needed in order to stop; it was
not enough to say why it stopped, and a trigger the agent acknowledged and
then answered with silence is the defect issue #78 names. The worker cannot
re-derive the sentence without duplicating the branch above โ two copies of
one rule, which would drift the first time a third refusal is added โ so the
component that refuses is the one that writes it, beside the numbers it
reports to the journal.
Each notice names the setting that would turn the refusal into a review:
budget.excluded_paths when nothing survived them, budget.max_run_tokens
when the prediction did not fit. The worker hands it to
Publisher.notify, which is also what
posts the size gate's own PullRequestTooLarge.notice.
The free refusal before the checkout¶
preflight is the last free refusal; it is not the first. One question is
asked earlier still, the moment GET /pulls/{n} returns: is this pull
request still open? A trigger enqueued while it was open can be claimed
after it has been merged, and reviewing it then costs exactly as much as
reviewing a live one and produces something nobody will read.
That row settles the same way a refusal does โ used_tokens = 0 at exact,
reviewed_lines NULL, then abandon โ but records stop_reason = closed
rather than refused. Both mean no engine ran; they differ in what an
operator should do about a lot of them. refused says a limit may be too
tight. closed says the queue is draining more slowly than the repository
moves, and the answer is workers or pacing, not tokens. Collapsing them
would make GROUP BY stop_reason unable to tell those apart, which is the
thing the column exists for. See
WORKER.md.
๐ง Human headroom¶
The agent is capped at a configurable share of each plan window
(reviewer_share_pct, default 40 %), never the whole allowance. A runaway
agent can degrade interactive Claude Code but cannot lock a maintainer out of
it.
This is the direct answer to "weekly threshold, no overspending": the agent's ceiling is deliberately below the plan's.
๐ Windows, one shape¶
Every limit is tokens recorded in the ledger within a trailing duration. The effective ceiling for a run is the tightest of them; the ladder rung comes from the worst utilisation among them.
| Window | Duration | Limit |
|---|---|---|
| session | 5 h | session_tokens ร share |
| weekly | 7 d | weekly_tokens ร share |
| daily | 24 h | weekly_tokens ร share รท 7 |
| contributor | 7 d | weekly_tokens ร share ร per_contributor_pct โ only when configured |
A weekly cap alone would permit burning the allowance on Monday, which is what the daily window prevents: at most a seventh of the week in any day.
That is a deliberate reading of a gap in the original specification, which
paced the day as weekly_remaining รท days_remaining. A rolling weekly window
never resets, so days_remaining has no value to take. A flat seventh needs no
week anchor โ one the plan does not publish and we would have had to invent โ
and it is stricter: an agent idle since Monday cannot burn four days'
allowance on Friday. Unspent allowance is not a loss here. It is headroom for
humans, which is the point.
The contributor window is the only one scoped to a person. It measures
what the claim's own actor_id has spent over the same rolling week, so it is
built per claim rather than per configuration, and it joins the list only when
per_contributor_pct is set โ unset, the other three behave exactly as they
did before it existed. Being in the list means it degrades and refuses on the
ordinary ladder below, but only for the contributor being admitted: everyone
else's headroom is measured separately, which is the whole purpose. headroom(),
the operator readout, has no contributor to scope to and therefore reports the
three shared windows only.
The cap is meaningless on a one-person allowlist โ the one account able to
trigger anything would simply meet its own cap. It earns its keep once several
people can trigger reviews and one monopolising the week is a real outcome.
actor_id has been on every ledger row since the governor shipped, precisely
so this stayed possible: the ledger is append-only, and attribution is the one
field that cannot be backfilled.
โฑ The pacer: one pull request's rate¶
Implemented in src/pr_review_agent/pacing.py, and called from admit
after the windows and before the reservation is written.
The windows above bound a total. They cannot tell one pull request consuming the day from thirty sharing it, and that distinction is the whole reason this exists: a branch under active development produces triggers at the rate somebody pushes to it, once a maintainer is mentioning the agent alongside those pushes. Each of those reviews is paid for in full, and each is liable to be superseded by the next push before it can be posted.
So a second question is asked of the same ledger:
| Key | Default | Refuses when |
|---|---|---|
min_review_interval_seconds |
900 | this pull request was reviewed less than 15 minutes ago |
mention_min_review_interval_seconds |
300 | the same, for a @claude โ a person is waiting, so it is shorter |
max_reviews_per_pull_request |
unset | this pull request has had that many reviews in a trailing 24 hours |
A refusal here defers; it does not drop¶
It is an ordinary admission refusal, which in this queue means the row stays
pending, costs no attempt, and is offered again on the next claim. Nothing
is lost, and nothing is silently answered by somebody else's run.
That is also where the saving comes from. A deferred trigger is reviewed once the interval has passed, against whatever the head is then โ so a burst of pushes and mentions inside one interval collapses into one review of the final head instead of one review each. The fold closes the rest of the burst when that review posts, so the queue does not then drain them one interval apart.
The cost of a deferral is one indexed SELECT over a table the governor
already writes. No GitHub call, no engine, no reservation.
The cap's trailing day is the governor's own DAILY, imported rather than
restated. Both now live in config/budget.py, because the governor imports
the pacer and a duration owned by either would have to be copied into the
other to reach it โ and copied is how two definitions of "a day" come to
disagree.
Why the ledger, and not runs¶
runs holds reviews that produced findings. A run that reached an engine and
then failed spent tokens just the same, and a pacer blind to it would let a
pull request that crash-loops the engine be re-reviewed without limit โ which
is the case that empties an allowance fastest. Every admitted run writes a
ledger row, so the ledger is the honest record of this pull request reached
an engine. Migration 11 puts repo and pr_number on it for exactly this
query.
Rows written before that migration carry NULL in both and read as no history, which is the same answer a pull request nobody has reviewed gets. Reading a missing column as recent would defer every review on an upgraded database until a backfill nobody can do had happened.
This is not the contributor window again¶
The two look alike โ a rolling duration, a refusal on the same ladder โ and they are deliberately different axes:
per_contributor_pct |
the pacer | |
|---|---|---|
| Scopes to | one person | one pull request |
| Measured in | tokens | time since the last review, and a count |
| Answers | one contributor monopolising the week | one branch monopolising the day |
A repository can hit either without the other: one maintainer opening thirty
pull requests trips the contributor window and no pacer; one contributor's
single branch under a rebase-heavy week trips the pacer and no window. The
cap is off by default for the reason per_contributor_pct is โ a repository
with one active pull request at a time gains nothing from it, and a default
that refuses work nobody asked it to refuse is the wrong direction for a
knob.
What it does not do¶
It posts nothing. A deferral and a cap refusal are WARNING log lines;
no comment is written to say the agent is waiting. The contributor sees the
๐ reaction only once the trigger is actually claimed, which for a deferred
mention is one interval later than they typed it. That is the deliberate
trade: a comment saying "I am waiting" costs a GitHub write on every paced
trigger and would itself need rate limiting.
๐ค Several repositories, one allowance¶
A daemon serves one repository, with its own GitHub token and its own
allowlist. Several of them share one budget by sharing one store.path: the
ledger carries no repository column and the circuit breaker is global to the
file, so every window already sums across whoever wrote it, and a trip in one
repository refuses claims in all of them. The pool is shared by
construction. Reserve-then-settle below is what makes that safe across
processes and not merely across threads โ SQLite's single write lock is
per-file.
Whose numbers govern that pool is a separate question, and the one thing
that does not fall out for free. Each process would otherwise police the shared
allowance using its own config.yaml, and two files that disagree do not split
the budget between them: there is one usage total, so the most permissive file
keeps admitting runs after every other has correctly stopped.
So exactly one configuration is the authority. It publishes five keys โ
session_tokens, weekly_tokens, max_run_tokens, reviewer_share_pct,
per_contributor_pct โ to a one-row budget_policy table, and every other
daemon adopts them. Both budget.authority and budget.comply default to
true, and comply is read only when authority is false, so a lone daemon
governs its own store with nothing added to its file.
Three properties are worth stating, because each is a failure mode that was available and is now closed:
- A second authority refuses to start, naming the first. Getting the fleet wrong is loud rather than a silent race for who writes the row last.
- Start order does not matter. A daemon with
authority: falsethat finds no policy exits 3, and the shipped unit restarts it, so it waits for its authority rather than requiring a sequence. - An authority's
SIGHUPreaches a running complier with no signal to it. The policy is read inside the very transaction the reservation is written in, so the next claim simply measures against the new numbers.
enabled is deliberately not published: a brake that could only be pulled
fleet-wide could not stop one misbehaving repository. Neither are the
size caps or excluded_paths, which describe a repository
rather than an allowance. comply: false is the escape hatch, and the one
remaining way to overspend a shared pool, so it is logged at WARNING when
another configuration is the authority.
The per-contributor window becomes cross-repository under a shared store,
because it keys on the global GitHub actor_id. That is tighter than before,
not looser: one person's weekly share is now measured across every repository
they can trigger.
๐ Reserve-then-settle¶
Checking the remaining allowance is not sufficient under concurrency. Three workers can each observe sufficient budget, each start a run, and collectively breach the cap โ while every one of them checked correctly.
So the governor:
- reserves the per-run maximum inside the same
BEGIN IMMEDIATEtransaction as the queue claim, throughclaim()'sadmithook; - runs;
- settles against reported usage and releases the remainder.
This invariant is the main reason to prefer SQLite. Its single write lock makes the reservation atomic with the dequeue for free, where PostgreSQL would need explicit row locking and a broker-plus-store split would need a distributed transaction. See STORAGE.md.
1 2 3 4 5 6 | |
One query measures a window, and it is the whole guarantee:
1 2 | |
A settled row counts what it actually spent; an unsettled one counts its full reservation. The second worker therefore sees the first's reservation as already spent, before the first has finished.
tests/test_budget_concurrency.py pins this with eight real threads over one
database file.
A crashed worker's reservation stays charged¶
An unsettled reservation keeps counting until it ages out of its rolling window. It is never released early, and there is no expiry sweep.
The original specification said a reservation expires with the lease, "so a
crashed worker does not leak allowance". This honours the intent rather than
the letter. A crashed run has probably already spent tokens, and
queue.DEFAULT_MAX_ATTEMPTS is 3, so releasing on expiry would let one trigger
spend invisibly three times over. Leaking means held forever, which a rolling
window already rules out; it does not mean granting an amnesty. For a spending
control the pessimistic direction is the safe one.
The happy consequence: the ledger needs no expires_at, no state column and
no sweeper.
A lost reservation is closed by the next admission¶
What identifies a reservation, while it is open, is its dedupe_key. The
ledger says so with a unique partial index โ
ledger_open โ rather than only by convention, and the
convention alone was not enough.
settle matches on the key and the owner, and both clauses stay, because
they answer different questions. Uniqueness โ is there exactly one open
reservation for this trigger? โ is the index's job. Entitlement โ is it
ours? โ is the owner's, and it is what makes a worker whose lease lapsed
get False from settle and discard a result it is no longer allowed to
post.
Without the index they collided. daemon.supervise restarts a crashed worker
as the same worker: owner is minted once at startup, so an attempt that
died holding a reservation and then re-claimed the row once its lease lapsed
reserved again under the identical (dedupe_key, owner). One settle then
matched two open rows, updated both, reported no single reservation, and the
worker logged "no reservation to settle" and threw the review away โ
unpublished, unfinished, and already paid for. Since every review
posts its own comment, that is a lost comment rather than a
skipped edit.
So admit closes the predecessor before it reserves, inside the claim's own
transaction, at the full reserved_tokens it had taken:
| stop_reason | settles at | why |
|---|---|---|
refused |
zero | no engine ran, and that is provable |
engine_unavailable |
zero | the subprocess never started |
lost |
the whole reservation | nobody knows what it spent |
lost is pessimistic on purpose, for the reason the rest of this section is:
a crashed worker may have spent anything, and for a spending control the
pessimistic direction is the safe one. It is the same rule the worker already
applies to a failure at or after the engine call.
It hands back no allowance. The row was counting its whole reservation
while it sat open and counts exactly the same once closed; the change is that
the charge is now explicit. To an operator, a lost row reads as this was
paid for and nobody knows what it bought โ one is a crash, a run of them is
a worker dying mid-review โ and what they charge is a ceiling rather than a
measurement, so the windows they fill are tighter than the real spend, never
looser. They are countable:
1 | |
A caught failure settles at what is knowable¶
A lost worker's reservation stays charged, as above, and is closed as
lost by whatever admits that trigger next. A worker that caught
an exception is not lost, and settles โ but at what? The spend is unknowable
for an engine killed mid-run, so the rule splits on whether the engine had
started: a failure before it settles at zero, a failure in or after it
settles at the full reservation, and the one failure at it that provably
ran nothing โ EngineUnavailable, a subprocess that never started โ settles
at zero too. A failure whose envelope could still be read is the exception
on the other side: the CLI measured that run, so it settles at the
envelope's figure rather than the reservation. See
WORKER.md.
๐ The degradation ladder¶
A hard stop at 100 % is the wrong shape โ it makes the agent useless for the rest of the window with no warning. Instead:
| Worst utilisation | Mode | Behaviour |
|---|---|---|
| < 85 % | full |
admit |
| โฅ 85 % | mention_only |
stop auto-reviewing fresh pull requests; still honour an explicit @claude |
| โฅ 100 % | exhausted |
refuse everything |
A run is also refused, at any rung, when the tightest window's remainder is
below max_run_tokens. There is no partial reservation: a run that cannot be
afforded in full is not started, because a truncated review is still a spend.
The mode is recorded on every ledger row, so a thinner-than-usual review is explainable afterwards.
In practice exhausted is reached only by an overrun โ a run that settles
above what it reserved. Admission alone stalls just short of 100 %, because the
must-fit-whole rule refuses before the window fills. That is exactly the case a
hard stop exists to cover.
The 60 % rung of the original ladder โ drop to correctness and security dimensions only โ is not implemented. There is no engine to thin a review, so it would be a mode recorded and never acted on. It returns with the engine adapter.
๐งพ The ledger¶
Every posted comment must be traceable to a ledger row recording engine, model,
mode, token usage and usage_confidence (exact / estimated /
unavailable). That last field exists because
some engines report no token usage at all,
which forces the governor onto proxy controls โ run-count, wall-clock and turn
caps โ and that is a materially weaker guarantee an operator should be able to
see.
Ledger rows are never deleted, even when review content is purged: the rolling windows are computed from historical usage. Deleting one would silently hand back allowance that was genuinely spent.
engine and model are NULL between reserve and settle, because nothing
knows them until a run finishes. actor_id is recorded on every row but
nothing enforces against it yet โ see below.
๐ Kill switches¶
All limits live in one config.yaml, reloaded on SIGHUP:
budget.enabled: falsestops reviewing. It does not turn the budget checks off. The two readings of "kill switch" differ by catastrophe, so it is worth being explicit: the governor refuses every claim while it is false.publish.dry_run: trueexercises the full pipeline while posting nothing. It arrives with the publisher; the reload mechanism built here takes it without change.
SIGHUP re-reads the file and swaps only the budget section. A changed
repository or store path is logged as needing a restart rather than
half-applied, and a broken file leaves the previous configuration in force โ
crashing would turn the emergency brake into a way to take the service down
with a typo.
๐ The circuit breaker¶
Implemented in src/pr_review_agent/breaker.py, over the budget_state
table, with Governor.trip and Governor.breaker in budget.py as its only
callers. It sits beside the governor rather than inside it because the two
answer different questions โ the governor measures windows and decides what
may be spent, the breaker decides how far to trust the numbers those windows
are measured against. Design note:
the circuit breaker.
Every limit above is the operator's guess, because the plan publishes no
quota. A guess that is too low is harmless. A guess that is too high is the
one failure the rest of this design cannot see: the governor admits runs and
reports healthy utilisation while the real limit is already being hit, and the
worker retries into the same wall until max_attempts runs out,
spending each time. The breaker is the only feedback from reality into that
guess.
Trip. A usage-limit failure refuses every claim for TRIP_HOLD, which is
one SESSION. Five hours because it is the shortest window, and because it is
a duration rather than a reset time โ the only kind of answer available when
constraint 4 says the plan exposes none.
trip(window, resets_at), the signature this document used to sketch, is
therefore not what was built: trip(now) takes no reset time because there is
none to take.
If the weekly limit was the one that blew, the next attempt trips again and the
calibration keeps shrinking โ the design converges either way, so it never
needs the attribution to be right. A refused claim costs no attempt:
ReviewQueue.claim already skips a candidate admit rejects rather than
burning one, because a refusal is about the allowance, not about the trigger.
Decay, and recovery. The same trip multiplies a stored calibration by
DECAY_FACTOR (0.9), and every window's effective limit becomes configured ร
calibrated. So the ceiling drops a tenth each time reality disagrees with the
guess, converging downward over a handful of windows instead of hitting the
wall once per window forever.
Recovery is additive: one percentage point per clean SESSION, capped at the
configured value. Multiplicative down and additive up is what makes the series
converge rather than oscillate. It exists because the pool is shared โ a trip
does not always mean the guess was too high, it can equally mean the maintainer
had a heavy week, and a calibration that could only ever be revised downward
would leave the reviewer permanently crippled by one of those. That is the
mirror of the permanent floor the fit rejects.
Because a trip holds for a full SESSION and recovery accrues only in clean
ones, decay is self-rate-limiting: no second timer, and no way for a burst of
trips to collapse the calibration in an afternoon.
The calibration is an integer percentage floored at 1, never a float.
_headroom divides by the limit, so a calibration reaching zero would raise
where it should refuse โ the same reason reviewer_share_pct is validated
1..100. Decay truncates rather than rounds, because round(3.6) is 4 and a
rounded decay stalls at 4 % forever instead of converging.
What a tripped run settles at. Not the full reservation. A usage limit is one of the few failures where the spend is knowable: refused up front the CLI did no work and it is zero, hit mid-run the envelope measured it. Charging the ceiling would write tokens that were never spent into all three rolling windows, and โ the windows being rolling and ledger rows never deleted โ they would keep refusing real runs for up to a week after the account recovered.
Detection is a guess, and this is where to correct it¶
Issue #20 required the real usage-limit failure to be observed before the interface was fixed. It has not been. Manufacturing one means driving a live subscription into the wall this design exists to avoid.
So the guess is confined to one constant, _USAGE_LIMIT_MARKERS in
src/pr_review_agent/engine/claude.py, matched against both plausible
carriers โ a nonzero exit's stderr and the result envelope. The envelope is
read on a nonzero exit too, because that is how the CLI reports an errored
run: exit 1, the envelope on stdout, stderr empty. When the real
error is seen, editing that tuple is the whole fix: no signature changes and
no migration. A miss costs a retry; a false positive takes the reviewer
offline for five hours, which is why the markers are narrow.
๐ณ Not built yet¶
Recorded here so they are not rediscovered as omissions.
Nothing aborts a run that is exceeding its reservation. max_run_tokens
is reserved against and settled against, but a run that overruns it does so
undetected until settle, after the tokens are gone โ which is why the
exhausted rung is reachable only by an overrun. Layer 3
considered and rejected every way
to enforce it at a subprocess seam.
The breaker above is not that enforcement, and should not be read as it: it answers the account's limit being reached, not one run outspending its own reservation. What absorbs an overrun today is the pre-flight estimate, whose rate is fitted against what runs actually cost โ so an expensive run raises the predicted cost of the next comparable one, and a large enough pull request is refused before it starts. That is feedback after the fact rather than a ceiling, and the difference is a reservation's worth of tokens.
The ladder's 60 % rung likewise waits on a running engine to degrade.
๐งช What the tests pin¶
In the tests/test_budget_*.py family -- the windows, the ladder, the fit and
the concurrency load -- and the admit cases in tests/test_queue_admit.py:
- a synthetic concurrent load cannot breach any configured window;
- with
reviewer_share_pctconfigured, agent usage never exceeds its share; - daily pacing refuses a run the weekly window alone would have allowed;
- with
per_contributor_pctunset nothing is scoped to a contributor, and with it set one contributor's second run is refused while another's is still admitted, the refusal naming the contributor window; - the ladder is observed at 85 % and 100 %, and at 85 % a refused pull request
does not block a maintainer's
@claudebehind it in the queue; - a refusal costs no attempt and leaves the row
pending; - an unsettled reservation counts in full, and keeps counting past its lease;
- one trigger cannot hold two open reservations, under eight concurrent
workers or by hand, and re-admitting a trigger closes the reservation a
crash left open as
lostat its full ceiling without handing any allowance back; - a worker that crashes after its engine ran, and then re-claims its own row
under the same
owneronce the lease lapses, publishes the second review instead of discarding it, and leaves one settled row per attempt; settlereleases the remainder and records engine, model and confidence;budget.enabled: falseadmits nothing, andSIGHUPflips it without a restart;- a
SIGHUPagainst an unparseable file keeps the previous config in force; - an empty ledger estimates at the documented constant and still refuses an oversized pull request, the fit takes over at the tenth fittable row, and rows the engine could not measure are not fitted;
- a pre-flight refusal returns the window to exactly where it was, and a pull request with nothing left to review is refused for free;
- a trip refuses every claim without costing an attempt, admission resumes once the hold expires, and the calibration decays, bounds every window, floors above zero and survives a restart;
- repeated trips drive the calibration monotonically downward rather than oscillating, and a clean window recovers a point, capped at the configured value.
In tests/test_pacing.py, for the pacer โ every case leaves the windows
nowhere near spent, so only the pacer can be what refused:
- a second review of one pull request inside the interval is refused, and the
row is still
pendingwith no attempt spent, then claimed once the interval elapses; - a mention waits the shorter interval while an ordinary trigger at the same moment still waits;
- the interval binds one pull request only, so a busy one does not hold the queue for every other;
0disables the interval, spelled rather than approximated;- a run that reached an engine and recorded no findings still paces the
next one โ the ledger is what is measured, not
runs; - the cap refuses past its count, rolls out of its trailing day, and is absent by default;
- a ledger row from before migration 11 reads as no history rather than as a recent review.
In tests/test_cli_engine_outcomes.py and tests/test_worker_usage.py, for
the detector and the join:
- a usage limit on stderr and one in the envelope both raise
UsageLimited, the envelope carrying its measured usage and the stderr case carrying none, while an unrelated failure is still a protocol error; - an errored envelope that arrives with exit 1 is still asked, and a failure that is not a usage limit quotes the envelope's own error rather than the empty stderr;
- the envelope is only asked when it says it errored, and only its own
error-carrying fields are read -- a finding that quotes
rate_limit_errorparses as a completed review, because the review's prose is written about a tree an attacker can edit and must never be able to trip the breaker; - a
UsageLimitedrun trips the breaker, settles at what is known rather than at the reservation, leaves the row pending, and is not attempted again while the breaker holds.
And in tests/test_workspace.py and tests/test_exclusions.py, for the
exclusions half of layer 2:
- a vendored-only change over the line cap is refused without exclusions and admitted with them;
- an excluded path appears in neither the size count nor the diff;
- a binary file counts as one file and no lines.
And in tests/test_config_sections.py, tests/test_store_ledger.py and
tests/test_worker_failures.py, for layer 3:
- a review wall clock at or above the queue lease fails startup, pinned
against
DEFAULT_LEASEitself rather than against1800, so changing the lease cannot leave the check behind; - a run killed on the wall clock is distinguishable in the ledger from one
whose engine merely fell over, and both still settle at the full
reservation with
unavailableconfidence; - an adapter whose subprocess never started settles at zero and reads as
engine_unavailable, because the reservation exists to cover a spend that might have happened and this one could not have; - a pre-flight refusal reads as
refusedrather than as a failure; - a claim whose pull request was merged or closed while it waited settles at
zero and reads as
closed, having reached neither a checkout nor an engine.
Terminating a review that is over budget, as opposed to over time, is the gap named above. It is not tested here because it is not built.