1. Executive summary
Build one bounded agent per ticket with an in-loop reflection stage, not a multi-agent fleet. The agent plans, edits in a sandbox, runs CI, and reflects on failures until tests pass or a retry budget is spent, then opens a pull request. It never merges: a human approves every merge. This is rung 3 of the escalation ladder, and the single-versus-multi-agent question is the crux of the design [knowledge/decision-trees.md].
Success is 30 percent of eligible tickets merged with under 10 minutes of human review, and zero production incidents attributable to the system. The safety posture is what makes that acceptable: sandboxed execution, no production credentials, no deploy access, and a hard rule that the system opens PRs but never merges them.
Interview notes
Strong answer. Lead with the escalation decision and the safety boundary, not the tool list. State the merged-rate target and the zero-incident bar in the first minute.
Interviewer probes. Why an agent at all rather than a scripted refactor tool? What is the blast radius if it goes wrong?
Tradeoffs to voice. Autonomy versus safety: more autonomy raises throughput but widens blast radius. The position is maximum autonomy inside a sandbox, zero autonomy at the merge boundary.
2. Requirements and NFRs
| NFR | Target |
|---|---|
| Volume | ~600 eligible tickets/week (bugs, small features), 40 concurrent jobs peak |
| Latency | Batch-tolerant: ticket to PR under 4 hours is fine |
| Success | 30% of eligible tickets merged with < 10 min human review; zero attributable prod incidents in Q1 |
| Cost | ≤ 20 USD per attempted ticket |
| Codebase | Monorepo, 30M LOC, existing CI (25 min full run) |
| Safety | Never merges, no prod credentials, no deploy access, sandboxed execution only |
| Compliance | SOC 2; code exfiltration is the chief concern |
Interview notes
Strong answer. Separate eligible tickets (bugs, small scoped features) from the long tail the system should refuse. The 30 percent target is of the eligible slice, not all tickets.
Interviewer probes. How do you decide a ticket is eligible? What stops it attempting a 3-week refactor?
Tradeoffs to voice. Coverage versus reliability: chasing hard tickets lowers the merged rate and burns budget. Scope tightly and hand back early.
3. Decision record
| Gate | Decision | Why, and what lost |
|---|---|---|
| Escalation ladder | Rung 3, single agent | The path is open-ended: the number of edit-test iterations depends on results [knowledge/building-effective-agents.md]. A fixed workflow cannot absorb that. Rejected: rung 2. |
| Single vs multi-agent | Single, with reflection | The reflection pattern (self-review plus CI as ground truth) gives most of the benefit of a separate reviewer agent [knowledge/agentic-design-patterns.md]. A planner-coder-reviewer fleet costs several times the tokens and adds coordination failure modes for no measured reliability gain [knowledge/multi-agent-orchestration.md]. Rejected: multi-agent. |
| Autonomy | Tier 1 (open PR), merge out of scope | A PR is reversible and reviewed; a merge is not the agent's to make [knowledge/security-governance.md]. |
| Grounding | Repo as corpus, retrieval over code | The agent retrieves relevant files rather than loading 30M LOC; long-context on the whole repo is neither affordable nor necessary [knowledge/rag-patterns.md]. |
| Memory | Per-run checkpointed state | Long-running jobs need resumability; no cross-ticket memory is required [knowledge/context-memory.md]. |
Interview notes
Strong answer. Make the single-versus-multi-agent call explicitly and defend it with token economics, not taste. Name CI as the real reviewer.
Interviewer probes. When would multi-agent actually win here? What signal would make you switch?
Tradeoffs to voice. If tickets needed genuinely parallel independent subtasks, or separate privilege domains, multi-agent would earn its cost. This workload does not.
4. System architecture
A ticket is admitted through a safety gate into a sandboxed runtime. The agent reasons via a model gateway, edits through repo tools, runs code in a sandbox and tests through CI, and reflects on failures. It checkpoints to run state, then opens a PR that a human reviews.
| Component | Serves | Scaling | Failure mode and fallback | Tech example |
|---|---|---|---|---|
| Safety gate | Admission, credential stripping | Stateless | Reject ticket if sandbox unavailable | Policy check + sandbox provisioner |
| Coding agent | Plan, edit, test loop | One per ticket, horizontal | Retry budget then hand back | Sonnet-class model |
| Repo tools | Read, edit, branch | Stateless | Abort on write conflict | MCP server over the VCS [knowledge/interoperability-observability.md] |
| Sandbox | Isolated execution | Ephemeral per job | Kill on resource or time cap | Container or microVM |
| CI runner | Ground-truth tests | Pooled, queued | Backpressure to the agent | Existing CI, 25 min suite |
| Test + self-review | In-loop reflection | Stateless | Bounded iterations | Evaluator-optimizer loop [knowledge/agentic-design-patterns.md] |
| Run state | Checkpoint, resume | Durable store | Resume from last checkpoint | Durable execution store [knowledge/context-memory.md] |
| Human review | Approves every merge | Human capacity | PR waits; never auto-merges | Existing code review |
Interview notes
Strong answer. Draw the sandbox boundary first and put the human at the merge line. Everything else is inside a box that cannot reach production.
Interviewer probes. What exactly is in the sandbox, and what secrets does the agent hold? How do you prevent code exfiltration?
Tradeoffs to voice. Isolation versus iteration speed: tighter sandboxes are safer but slower to provision. Ephemeral microVMs per job is the balance.
5. Code retrieval
The corpus is the monorepo. The agent does not load 30M LOC; it retrieves the files and symbols relevant to the ticket (code search plus embeddings over the repo), then works within that focused context [knowledge/rag-patterns.md]. A repo map plus retrieved files is cached across the agent's iterations to control token cost.
Interview notes
Strong answer. Frame code as retrieval, not context stuffing. The repo map is the cheap always-on context; specific files are retrieved just in time.
Interviewer probes. How does the agent find the right files in a 30M-LOC monorepo? What about cross-file changes?
Tradeoffs to voice. Recall versus token budget: retrieving more files raises the chance of catching cross-file effects but costs tokens on every iteration.
6. Tools and integrations
| Tool | Tier | Enforcement |
|---|---|---|
| read_code, search | 0 | Read-only, sandbox scope |
| edit_file, create_branch | 1 | Sandbox branch only, never the default branch |
| run_tests (CI) | 0 | Pooled runner, no prod targets |
| open_pull_request | 1 | PR only; merge permission not granted to the agent |
Interview notes
Strong answer. Enumerate tools by tier and point out the two that do not exist (merge, deploy). Absence is the enforcement.
Interviewer probes. Could the agent open a PR that quietly weakens a security check? How would a reviewer catch it?
Tradeoffs to voice. Autonomy versus reviewer load: broader edit permissions raise throughput but make human review harder. Keep diffs small and scoped.
7. State and memory
Each run is checkpointed so a job interrupted at hour two can resume rather than restart, which matters for cost on long tickets [knowledge/context-memory.md]. There is no cross-ticket memory: every ticket starts clean, which keeps behavior predictable and avoids leaking one team's code context into another's.
Interview notes
Strong answer. Checkpoint at natural boundaries (after a passing test run), not every token. Resumability is a cost lever, not just reliability.
Interviewer probes. What happens if the model or sandbox dies mid-edit? Do you replay tool calls or restore state?
Tradeoffs to voice. Checkpoint frequency versus overhead: more checkpoints resume better but cost storage and time.
8. Security and safety
Code exfiltration is the chief threat [knowledge/security-governance.md]. The lethal trifecta (private data, untrusted content, exfiltration channel) is broken by removing the channel: the sandbox has no outbound network beyond the model gateway and the VCS, and holds no production secrets.
| Control | Purpose |
|---|---|
| Sandbox with no prod creds or deploy access | Excessive agency, exfiltration |
| No merge or deploy tool | Irreversible-action prevention |
| Egress allowlist (model gateway, VCS only) | Code exfiltration |
| Prompt-injection handling on ticket text and code comments | LLM01, indirect injection from repo content |
| Full audit of tool calls and diffs | SOC 2, forensics |
Interview notes
Strong answer. Name code exfiltration as the top risk and show how the egress allowlist and credential-free sandbox close it. Injection can come from a malicious code comment.
Interviewer probes. A ticket description contains an injection telling the agent to add a backdoor. What stops the PR from shipping it?
Tradeoffs to voice. Convenience versus containment: an egress allowlist blocks some legitimate package fetches; mirror them internally rather than open egress.
9. Evaluation plan
Offline evals on a held-out set of historical tickets with known-good PRs, scored on tests-pass and diff similarity, before any live run [knowledge/evaluation.md]. Because tool agents are unreliable even at high single-run scores, measure pass^k across repeated attempts on the same ticket [knowledge/evaluation.md]. Online, track merged rate, human review minutes, and revert rate as the real signals.
Interview notes
Strong answer. CI is a built-in outcome eval: tests either pass or they do not. Layer trajectory review on top to catch tests that pass for the wrong reason.
Interviewer probes. How do you know a merged PR was actually good, not just green? What is your regression signal?
Tradeoffs to voice. Outcome versus process eval: green tests are necessary but not sufficient; revert rate and reviewer edits are the honest quality signal.
10. Observability
Every ticket run is a trace: plan, each edit, each test run, each reflection, as spans with token and cost attribution, using OpenTelemetry GenAI conventions [knowledge/interoperability-observability.md]. Dashboards track merged rate, cost per attempted ticket against the 20 USD ceiling, retry-loop depth, and abandonment reasons; a cost anomaly or a spike in abandonment pages a human.
Interview notes
Strong answer. Instrument cost per ticket and retry depth from day one: they are your early warning that the agent is grinding on tickets it should hand back.
Interviewer probes. How do you debug a ticket that burned the full budget and produced nothing?
Tradeoffs to voice. Detail versus noise: full trajectory traces are large; sample deep traces and always keep cost and outcome.
11. Scale and cost analysis
Model pricing used, [live-sourced 2026-07-14]: Sonnet 4.6 at 3.00 / 15.00 USD per million input / output tokens, prompt caching near 10% of input on cached tokens.
Per-ticket cost
A typical attempt runs roughly 15 to 25 model calls (plan, then edit-test iterations). Assume ~20 calls, each with ~10k cached context (repo map, system) plus ~20k fresh (retrieved files, diffs, test output) and ~1.5k output. Per call that is about 0.003 USD cached, 0.06 USD fresh input, and 0.023 USD output, near 0.086 USD. Twenty calls is roughly 1.70 USD, and hard tickets that iterate 50 times still land near 4 to 5 USD, comfortably under the 20 USD ceiling. CI and sandbox compute are the other cost line and are dominated by the 25 minute test suite.
The real cost lever
The ceiling is generous; the waste is tickets that will never land. Early abandonment (a retry budget and a confidence check after planning) protects both the per-ticket cost and the merged rate.
At 10x (6,000 tickets/week)
Token cost scales linearly and stays within the per-ticket ceiling, so the binding constraints move to CI throughput and sandbox provisioning, not model spend [knowledge/latency-cost-reliability.md]. A 25 minute suite times thousands of attempts needs a large CI pool or a smart subset-of-tests-first strategy, and sandbox provisioning must scale to hundreds of concurrent ephemeral environments.
Interview notes
Strong answer. Show that model cost is not the constraint: CI capacity is. That reframes the scaling conversation away from tokens.
Interviewer probes. At 10x, what breaks first? How do you keep CI cost sane when every attempt runs a 25 minute suite?
Tradeoffs to voice. Thoroughness versus throughput: running the full suite every iteration is safe but expensive; run an impacted-tests subset first, full suite before the PR.
12. Failure modes, degradation, and lifecycle
The ticket state machine makes the terminal outcomes explicit: merged, handed back, or abandoned. Handing back cleanly is a feature, not a failure.
| Failure | Handling |
|---|---|
| Tests keep failing | Bounded retries, then abandon with a summary of what was tried |
| Scope too large | Detected at planning, handed back to a human immediately |
| Flaky test | Retry once; if it flips, flag rather than chase |
| Model or sandbox crash | Resume from the last checkpoint [knowledge/latency-cost-reliability.md] |
| Reviewer requests changes | Re-enter the edit loop from the PR comments |
Interview notes
Strong answer. Treat handing back as a first-class success path. An agent that knows its limits is more valuable than one that always produces a PR.
Interviewer probes. How does the agent know when to give up? What stops an infinite edit-test loop?
Tradeoffs to voice. Persistence versus waste: a higher retry budget lands more hard tickets but burns cost on hopeless ones; tune the budget from the abandonment data.
13. Rollout plan
- Crawl: shadow mode on closed historical tickets, comparing the agent's PR to the human's. Gate: offline tests-pass and diff quality [knowledge/evaluation.md].
- Walk: live on a few low-risk services, one team, PRs clearly labeled as agent-authored. Gate: merged rate and review-time targets, zero incidents.
- Run: expand service by service. The abandonment and revert data continuously retunes eligibility and the retry budget [knowledge/genai-sysdesign-loop.md].
Interview notes
Strong answer. Start in shadow against known-good PRs: it is a free, safe eval with ground truth. Expand by service risk, not all at once.
Interviewer probes. How do you build reviewer trust so the 10-minute review target is realistic?
Tradeoffs to voice. Speed of rollout versus trust: moving fast risks a bad merge that poisons adoption; expand only as the revert rate stays near zero.
14. References
Knowledge base
- knowledge/decision-trees.md, building-effective-agents.md, agentic-design-patterns.md (rung 3, reflection, single vs multi-agent)
- knowledge/multi-agent-orchestration.md (the token economics behind rejecting multi-agent)
- knowledge/security-governance.md (sandboxing, exfiltration, autonomy tiers), rag-patterns.md (code retrieval)
- knowledge/evaluation.md (pass^k, CI as eval, shadow mode), context-memory.md (checkpointing)
- knowledge/latency-cost-reliability.md, interoperability-observability.md (CI scaling, tracing)
Live-sourced
- Anthropic API pricing, retrieved 2026-07-14, for the cost model in section 11.
Primary-source citations for each knowledge document are in agentify/knowledge/SOURCES.md.