Bug hunt: middleware and agent — prove defects with red-first tests #67
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Goal
Find real defects in
middleware/(Bun/Fastify) andagent/(Go) — not style issues — prove eachone with a failing test or a minimal repro, fix the high-confidence small ones, report the rest.
Hunt areas (priority order)
middleware/src/executions.ts): claim/start/result/expire races —double-claim, lease expiry mid-run (
safety.tsLeaseTable), timeout vs late result ordering,circuit-breaker half-open re-entry.
db/persist.ts): flush gaps — a mutate path that bypasses the in-memory Maps,hydrate/flush tearing under concurrent writes, dirty-sweep vs onSend ordering.
tenancy.ts,access.ts,authorization.ts): cross-org reads/writes byid (IDOR-shaped logic bugs),
filterVisibleleaking in list endpoints, instance-operator vsorg-role confusion.
webhooks.ts,pipeline.ts,decision.ts): dedupe window edge cases,kill-switch/circuit fail-open paths (any must be fail-closed), enqueue double-fire.
agent/): job-poll long-poll correctness, result POST retry/idempotency, capabilityreporting, install/upgrade path error handling (no daemon tests against real systemd).
control-agent.ts): dispatcher error paths, session expiry handling.Method
bun test/go testyou add (red first), the fix, thengreen. No finding without a repro.
UC-OP-02 /metrics endpoint,UC-MW-19 plan survives a second buildApp) — they are not yours; do not chase or "fix" them by weakening.Constraints
.skip. Regression tests are mandatory with fixes.Refs #<N>only — no closing keywords.once").
Claimed — branch
issue/67-bug-hunt. Doing: red-first hunt across execution lifecycle, persistence, tenancy/authz, pipeline/webhooks, agent, control plane; fix high-confidence small defects with regression tests.Issue #67 — Bug hunt: middleware + agent, red-first
Restated goal
Find real defects (not style) in
middleware/andagent/, prove each with a failing test or aminimal repro, fix the high-confidence small ones on branch
issue/67-bug-huntwith a regressiontest each, and report the rest as a findings table. "Done" for the operator: the middleware suite is
green except the two known pre-existing reds, every commit carries a red-first regression test, and
the issue thread holds the plan + outcome so a human can review without opening the worktree.
Scope
Will change (confirmed so far):
middleware/src/routes/agent-api.ts—/agent/:id/:token/resultcontinues into trust/circuit/incident/rollback side effects even when
applyResultreportsalready_done(UC-OP-04 / UC-AG-10say a duplicate report must not change state).
middleware/tests/uc-mw-122-execution-lifecycle.test.ts(or the cluster file that already coversUC-OP-04) — new regression test.
Will not touch:
middleware/src/app.tsroute registration,plans/(product content),dashboard/,agent/unless a scout finding is proven; the two known pre-existing reds(
UC-OP-02 /metrics,UC-MW-19 second buildApp) — reported, never weakened.UC decision
No new UC. The defect is a violation of two existing UCs:
UC-OP-04(idempotent/result) — "Second POST … without rewritingexec.steps"; Postconditions:"Execution remains at the terminal status from the first apply."
UC-AG-10(lost-ack retry is safe) — "Execution status and steps are unchanged by the retry."UC-MW-74(circuit breaker) — a duplicate success report currently callsrecordSuccess,which clears a tripped key (fail-open).
Implementation cite:
middleware/src/routes/agent-api.ts(POST /agent/:id/:token/result).Tests cite: existing
middleware/tests/uc-mw-cluster3-agent.test.ts(UC-AG-10) +middleware/tests/uc-mw-cluster5-ops.test.ts(UC-OP-04); new regression assertions live there.UC versions bumped only if behaviour text drifts (it does not — the code is the thing that is wrong).
Evidence (already reproduced, red-first)
middleware/tests/zz-red-probe.test.ts(throwaway) againstbuildApp({ seed: true }):So the "no-op" retry is not a no-op. Same code path also calls
circuit.recordSuccess('plan:…'|'host:…')on the duplicate, which deletes a tripped circuit key(fail-open), and
enqueueRollback/scheduleRecheckare reachable again on the retry.Ordered tasks
duplicate
POST /resultmust leaveplan.verifiedStreak,plan.totalVerifiedandstores.circuit.tripped()unchanged. Verify:cd middleware && bun test tests/uc-mw-cluster5-ops.test.ts tests/uc-mw-cluster3-agent.test.ts→ red.routes/agent-api.ts, return early onoutcome.status === 'already_done'(before anyplan/trust/circuit/rollback/incident side effect). Verify: same command → green.
own commit. Verify per finding with the targeted test file.
_check.py,check-ai-headers.py,check-type-mirror.py,cd middleware && bun test(expect the two known reds),rg 'Showing lines'.Independent commands are batched; one logical change per commit.
Open questions (assumption taken)
{ ok: true, status: 'already_done' }— that is what UC-OP-04 and UC-AG-10 already state and whatthe agent's retry loop (
UC-AG-54) requires; only the side effects are the bug.finding maps to; only add a UC if the behaviour is genuinely new.
Risk
already_donecould skip a legitimately needed side effect (e.g. anincident evidence attach that a retry was meant to repair). Fallback: scope the early return to
the trust/circuit/rollback/recheck block only, leaving purely additive idempotent writes alone.
agent-api.tsprompt strings would forceuc-mw-115-control-chat.test.ts; thischange touches no strings, but run that file anyway if the diff grows into it.
DONE (partial fix set) — Refs #67
Branch
issue/67-bug-hunt, 2 commits, local only (not pushed).Commits
1f81996fix(result): duplicate /result report is a true no-op30fa4e2fix(tenancy): gate bulk group ops and plan child collections on row scopeDefect 1 — duplicate
POST /agent/:id/:token/resultran the whole tail.applyResultreturned{ok:true,status:'already_done'}but the handler continued intorecordOutcome,verifiedStreak=0,circuit.recordSuccess(=state.delete(key), so a duplicate of a successful result cleared a tripped circuit key),enqueueRollback,scheduleRecheck,incidents.applyOutcome. Now early-returns foralready_done. Tests:tests/uc-mw-cluster5-ops.test.ts(2 regressions, red-first).Defect 2 — cross-org leaks.
POST /agents/bulkgroup-add/group-removechecked only the agent rows → any operator could rewrite another org's group hostnames. Now gated on the group row withassertEdit(...,'group').GET /plans/:id/{executions,compatibility,health}returned sibling-org rows for shared plans (disk-cleanup) — leakedorganizationId/host/agentId/stepCount. Now filtered via the existingmakeVisibility/executionVisible/filterVisible. Tests:tests/uc-mw-142-ownership.test.ts(2 regressions, red-first, verified RED withgit stash -- middleware/src).Gates (one line each)
python3 docs/use-cases/_check.py— exit 0 (358 files conform)python3 scripts/check-ai-headers.py— exit 0 (573 ok, 0 missing, 0 malformed)python3 scripts/check-type-mirror.py— exit 0 (3 informational lines)rg -n 'Showing lines' --glob '!node_modules' --glob '!.git'— only docs quoting the command; no footers in sourcecd middleware && bun test --timeout 20000 tests/uc-mw-cluster5-ops.test.ts tests/uc-mw-cluster3-agent.test.ts— 24 pass / 0 failcd middleware && bun test --timeout 20000 tests/uc-mw-114-agents-bulk.test.ts— 1 pass / 0 failcd middleware && bun test --timeout 20000 tests/uc-mw-142-ownership.test.ts— 6 pass / 1 fail; the failure (viewer invite can read but not write, line 136, 500≠403) is pre-existing, proved by re-running with the src fix stashed (3 pass / 4 fail). Not chased.agent/anddashboard/untouched → their gates not run.Reported, not fixed (full table in
meta/issue-67.report.md):assertEditdouble-send (access.ts:104, ~20 call sites → 500 instead of 403/404) ·incidents.ts:98evidence clobber ·safety.ts:145half-open latch (UC-MW-74 step 4) ·tripsForglobal key atplans.ts:1186·pipeline.ts:183/incidents.ts:79attempt counting ·pipeline.ts:35repeat-resolved webhook ignorestransition·agent/main.go:1055repo-job allowlist no-op ·agent/exectimeout vs process group ·agent/cmd.gounguarded Client mutation + flat backoff.Known reds left alone:
UC-OP-02 /metrics,UC-MW-19secondbuildApp. Bun's default 5s timeout is the source of thePGlite is closed/writeHeadcascade — always--timeout 20000. Report:meta/issue-67.report.md.REVIEW: FIXED 1 — Refs #67
Verdict FIXED 1 (1 review finding found, 1 fixed). Full report:
meta/issue-67.review-report.md.Verified true of the tree
routes/agent-api.ts:299-302(already_doneearly return,1f81996) androutes/agents.ts:123-125+routes/plans.ts:657-660,1163,1170-1171(30fa4e2). Red-first confirmed by merge-base source comparison: atb64f46athe result handler has noalready_donebranch, the bulk group block has noassertEdit, and the child queries are unfiltered.uc-mw-cluster5-ops+uc-mw-cluster3-agent→ 24 pass / 0 fail;uc-mw-114-agents-bulk→ 1 pass / 0 fail;uc-mw-142-ownership→ 6 pass / 1 fail.uc-mw-142red (viewer invite can read but not write, 500≠403) fails in isolation and the diff touches noPOST /planspath → pre-existing, by isolation + diff reasoning (no merge-base run; worktree-free constraint)._check.py0,check-ai-headers.py0 (573 ok),check-type-mirror.py0,rg 'Showing lines'= 9 doc-only matches. Plan Scope honoured: noapp.ts,plans/,dashboard/,agent/, known reds untouched. Tests additive only, no.skip,Refs #67only, nothing pushed.Fix (review commit)
0372b9breview(middleware): assert zero cross-org rows in shared-plan child collections— the new compatibility guard wascounted <= 1(passes while one sibling-org agent leaks) and/plans/:id/health, also changed by30fa4e2, had no assertion; now pinned torows.length === 0,counted === 0,health.executions === 0. Suite still 6 pass / 1 fail (same pre-existing red).Findings (mine)
30fa4e2bundles two defects in one commit — accepted as a tightly-coupled cluster (same class, same scout origin).plans.ts:1186; after its own diff that line isplans.ts:1190.safety.ts:144-155matches UC-MW-74 step 4 ("exactly one half-open execution"); the real gap is that a probe that never reports blocks the key untilPOST /ops/circuit/reset.db/persist.ts) and control plane (control-agent.ts) yielded zero findings — unexamined, not proven clean.UNFIXED: F1
access.ts:104assertEdit double-send → 500 instead of 403/404 (~20return;callers) · F2incidents.ts:98evidence clobber · F3 half-open probe has no timeout · F4pipeline.ts:183/incidents.ts:79attempt counting · F5plans.ts:1190globaltripsForleak · F6pipeline.ts:35repeat-resolved ignorestransition· F7agent/main.go:1055repo-job allowlist no-op · F8agent/exec/exec.go:39timeout spares grandchildren · F9agent/cmd.go:765/827unsynchronised Client + flat backoff · persistence/control-plane areas unexamined.PUBLISHED — branch issue/67-bug-hunt pushed, PR #74 opened to agent-develop: #74. 3 commits (2 defect fixes: duplicate /result no-op, tenancy gating on bulk group ops + shared-plan child collections; 1 review commit). Orchestrator gates on final tree: _check.py 0, check-ai-headers 0 (573 ok), check-type-mirror 0, full middleware bun suite 587 pass / 1 fail — the fail is UC-MW-19 (plan survives a second buildApp), the documented pre-existing red on a clean base; all new regression tests pass, including the uc-mw-142 viewer test that failed during the hunt (pre-existing, now green). Review: FIXED 1 (
0372b9b). Reported-not-fixed findings table lives in the DONE comment above — needs operator triage.