1. Executive summary
TechFlow should build a single tool-using agent behind an intent router, not a multi-agent system and not a bare workflow. The use case has open-ended dialogue and a small, well-scoped tool set (retrieve, look up subscription, write ticket), which lands it at rung 3 of the escalation ladder [knowledge/decision-trees.md].
The two decisions that shape everything else are permission-aware retrieval (the internal KB is tier-gated and past tickets carry PII, so retrieval must run under the caller identity) and a hard autonomy ceiling at tier 1 (create and update tickets, never touch billing). At the target load of 1,800 conversations per day, the design runs at roughly 0.06 USD per conversation against the 0.15 USD ceiling, and about 4,500 USD per month all in against the 6,000 USD ceiling. Rollout is staged behind evaluation gates, starting read-only.
2. Requirements and NFRs
Functional scope, confirmed with the customer:
- Answer product questions from public docs and the tier-gated internal KB, with citations.
- Look up order and subscription status for the authenticated customer.
- Create and update support tickets.
- Escalate to a human when confidence is low or the request is out of scope.
Non-goals (hard boundaries): never issue refunds, never change billing, never delete data. English only at launch.
| NFR | Target | Source |
|---|---|---|
| Throughput | 1,800 conversations/day, 40 concurrent sessions peak | customer |
| Latency | First token < 2s, full answer < 10s p95 | customer |
| Availability | 99.9% (business-hours bias) | customer |
| RTO / RPO | 4h / 15min for conversation state | customer |
| Cost | ≤ 0.15 USD per conversation, ≤ 6,000 USD/month | customer |
| Compliance | SOC 2 Type II, GDPR (EU customers), no HIPAA | customer |
| Quality | 40% ticket deflection with CSAT ≥ 4.2/5 | customer |
3. Decision record
Each decision tree from [knowledge/decision-trees.md] was walked. Answers and the strongest rejected alternative:
| Gate | Decision | Why, and what lost |
|---|---|---|
| Need generative AI? | Yes | Free-text questions over unstructured docs need synthesis. Rejected: pure search (cannot compose an answer or drive ticket actions). |
| Escalation ladder | Rung 3, single agent | The path is not predetermined: some turns need retrieval, some a subscription lookup, some a ticket write, in an order that depends on the question [knowledge/building-effective-agents.md]. Rejected: a fixed workflow (rung 2), which cannot handle the branching without an unwieldy router tree. |
| Single vs multi-agent | Single | Three tools with no overlap and one coherent task do not justify multi-agent token cost (roughly an order of magnitude more) [knowledge/multi-agent-orchestration.md]. Anti-escalation rule applied. |
| Knowledge strategy | RAG, permission-aware | KB changes weekly and is tier-gated; retrieval must filter by entitlement [knowledge/rag-patterns.md]. Rejected: fine-tuning (facts change weekly; no style need) and long-context (2,000 pages exceed a sane per-call budget). |
| Autonomy tiers | Tier 0 to 1 only | Reads and reversible ticket writes are safe to automate; billing and deletion are out of scope by design [knowledge/security-governance.md]. |
| Memory tier | Session state + compaction | Multi-turn within a session, no cross-session personalization required at launch [knowledge/context-memory.md]. |
4. System architecture
An intent router (a cheap model) admits and classifies each turn, then hands to the agent, which reasons in a bounded tool loop against a retriever, a subscription-lookup tool, and a ticket tool. Input and output guardrails bracket the model. The agent runs under the caller identity so retrieval and tools inherit the user's entitlements.
| Component | Serves | Scaling | Failure mode and fallback | Tech example |
|---|---|---|---|---|
| Input guardrail | Injection + PII defense | Stateless, horizontal | Fail closed: block the turn | Rules + small classifier + moderation [knowledge/security-governance.md] |
| Intent router | Cheap triage, cost control | Stateless, horizontal | Fall through to agent on error | Haiku-class model, cached |
| Agent | Reasoning and tool loop | Stateless workers, horizontal | Max 8 steps then escalate | Sonnet-class model |
| Model gateway | Routing, provider fallback | Stateless, horizontal | Secondary provider on outage | Gateway with multi-provider routing [knowledge/latency-cost-reliability.md] |
| Retriever | Permission-aware RAG | Stateless, horizontal | Degrade to docs-only, flag | Hybrid search + reranker [knowledge/rag-patterns.md] |
| Vector index | KB + ticket embeddings | Stateful, read replicas | Rebuildable from source | pgvector or OpenSearch-class |
| Session memory | Multi-turn state | Stateful, partitioned | RPO 15min via replication | Redis-class + durable log |
| Ticket tools | Subscription read, ticket write | Stateless, horizontal | Retry idempotently, then escalate | MCP server over internal APIs [knowledge/interoperability-observability.md] |
| Human review | Escalation target | Human capacity | Queue with SLA | Existing support desk |
| Output guardrail | Groundedness, safety | Stateless, horizontal | Suppress ungrounded claims | Groundedness check [knowledge/evaluation.md] |
5. Data and retrieval
Four sources with different trust and freshness profiles: public product docs (~2,000 pages, weekly refresh), the internal KB (tier-gated), the order and subscription database (per-tenant, strictly isolated), and past tickets (PII present). The retrieval design follows advanced RAG [knowledge/rag-patterns.md]: hybrid dense plus keyword search, reranking to the top passages, and query rewriting on follow-up turns.
Permission-aware retrieval
This is the crux. Every chunk is indexed with its entitlement metadata (public, customer tier, tenant id). The retriever receives the caller identity from the agent runtime and filters candidates by entitlement before ranking, so a customer never retrieves another tenant's data or a higher tier's KB [knowledge/security-governance.md]. PII in past tickets is scrubbed at ingestion and access to ticket embeddings is tenant-scoped.
Freshness
Docs reindex weekly on publish; the KB and tickets stream incremental updates with delete propagation so revoked content leaves the index promptly.
6. Tools and integrations
Three tools, each with an explicit autonomy tier and enforcement gate [knowledge/security-governance.md]. Tools are exposed as an MCP server so the interface is typed and versioned [knowledge/interoperability-observability.md].
| Tool | Contract | Tier | Enforcement |
|---|---|---|---|
| search_knowledge | query, identity, filters → passages | 0 (read) | Entitlement filter in the retriever |
| get_subscription | identity → status, plan, renewal | 0 (read) | Tenant-scoped credential, read-only role |
| create_or_update_ticket | identity, fields → ticket id | 1 (reversible write) | Idempotency key, full audit log, rate cap per session |
7. State and memory
Conversation state is held per session with summarization-based compaction once the turn history approaches the context budget [knowledge/context-memory.md]. No long-term cross-session memory ships at launch, which keeps the PII surface small. State is replicated to meet the 15 minute RPO, and a durable event log allows a session to resume after a worker restart.
8. Security, identity, and guardrails
Threat surfaces present here [knowledge/security-governance.md]: user input (prompt injection), retrieved content (indirect injection from KB or ticket text), and tool outputs. The lethal-trifecta test applies, since the agent has private data access, exposure to untrusted content, and a write tool; the mitigation is to keep writes reversible (tier 1) and to bracket the model with guardrails.
| Layer | Mitigates (OWASP) |
|---|---|
| Input validation + classifier | LLM01 prompt injection, LLM02 sensitive-info disclosure |
| Entitlement-filtered retrieval | LLM02, excessive data exposure |
| Tool tiering + absent billing tools | LLM06 excessive agency (ASI: uncontrolled autonomy) |
| Output groundedness check | LLM09 misinformation |
| Full audit log of prompt, response, tool calls | Forensics, SOC 2 evidence |
Identity: the agent acts with the authenticated user's permissions, not a broad service account, so a compromised turn cannot reach data the user could not already see. Tenancy: vector index, memory, and caches are tenant-scoped. Governance: maps to NIST AI RMF MEASURE (evals) and MANAGE (audit, escalation) [knowledge/security-governance.md].
9. Evaluation plan
Eval before launch, gates between phases [knowledge/evaluation.md].
- Golden set: 150 to 300 real questions, stratified by intent and customer tier, including negative and out-of-scope cases, refreshed monthly from production traces.
- Component metrics: retrieval recall@k and reranked precision measured separately from generation; generation scored for faithfulness and answer relevance (RAGAS-style, plus the TruLens triad) [knowledge/evaluation.md].
- Agent metrics: task completion, tool-call correctness, and pass^k for reliability across repeated runs, since tool agents are inconsistent even at high single-run scores [knowledge/evaluation.md].
- Judge: LLM-as-judge with rubric prompting and position and verbosity bias controls; judge version pinned.
| Gate | Unlocks |
|---|---|
| Offline faithfulness and no confidentiality leak on the golden set | Shadow |
| Shadow parity with human answers, zero entitlement violations | Canary (5%) |
| Canary CSAT ≥ 4.2 and deflection trending to target | General availability |
10. Observability
Every turn emits a trace with the full reasoning chain as spans (router, agent step, each retrieval, each tool call) using OpenTelemetry GenAI conventions, capturing tokens, latency, and cost per span [knowledge/interoperability-observability.md]. Dashboards track deflection, CSAT, p50/p95 latency per stage, cost per conversation, guardrail trigger rate, and eval-score drift. Alerts fire on cost anomalies, latency breach, a spike in escalations, or any entitlement-filter error. Sampled production traces flow back into the golden set.
11. Scale and cost analysis
Model pricing used, [live-sourced 2026-07-14]: Haiku 4.5 at 1.00 USD input and 5.00 USD output per million tokens; Sonnet 4.6 at 3.00 USD input and 15.00 USD output per million tokens, with prompt caching at roughly 10% of input price on cached tokens.
Per-conversation cost (4 turns, ~3 agent calls)
| Element | Tokens | Cost |
|---|---|---|
| Guardrails + router (Haiku) | ~3k in, ~150 out | ~0.004 USD |
| Agent, 3 calls (Sonnet) | ~2k cached + ~4k fresh in, ~400 out each | ~0.056 USD |
| Embedding + rerank | query-scale | ~0.001 USD |
| Total per conversation | ~0.06 USD (ceiling 0.15) |
Monthly: 1,800/day times 30 is 54,000 conversations at ~0.06 USD is ~3,240 USD in model spend. Add vector index hosting, reranker, memory, and observability for roughly 4,500 USD all in, under the 6,000 USD ceiling.
Latency budget (p95 ≤ 10s)
Input guardrail ~0.3s, router ~0.5s, retrieval ~0.7s, agent reasoning across 3 calls ~6s (streaming gives first token under 2s), tool calls ~0.8s, output guardrail ~0.4s. Tail latency compounds across the agent loop [knowledge/latency-cost-reliability.md], so the 8-step cap and per-stage timeouts protect the SLO.
At 10x (18,000 conversations/day)
Model spend alone would reach ~32,000 USD/month, over budget. Levers, in order: semantic caching (support questions repeat heavily, a 40 to 50% hit rate is realistic and cuts cost proportionally), route more turns to Haiku, and widen prompt caching. Concurrency rises from 40 to ~400, which the stateless router, agent, and guardrail tiers absorb horizontally; the vector index needs read replicas and the memory store needs partitioning.
12. Failure modes and degradation
| Failure | Degradation |
|---|---|
| Model provider outage | Gateway fails over to secondary provider; if both down, static help + human queue [knowledge/latency-cost-reliability.md] |
| Retrieval degraded | Answer from public docs only, flag reduced confidence, offer escalation |
| Tool (ticket API) failure | Idempotent retry, then escalate with context preserved |
| Runaway loop | 8-step cap ends the turn and escalates |
| Guardrail false positive | Route to human rather than block silently; track rate |
For any tier 1 write, the system fails closed: if the audit log write fails, the ticket action does not proceed.
13. Rollout plan
- Crawl (read-only): answers with citations, no ticket writes, internal users only. Gate: offline faithfulness and zero entitlement leaks.
- Walk (canary): 5% of real customers, ticket creation enabled at tier 1, subscription reads live. Gate: shadow parity, CSAT hold, zero confidentiality violations [knowledge/evaluation.md].
- Run (GA): full traffic, ticket updates enabled, human escalation always available. Field feedback and sampled traces feed the golden set and the next iteration [knowledge/genai-sysdesign-loop.md].
14. Request walkthrough
The primary path for a single customer turn, from admission through grounded response.
15. References
Knowledge base
- knowledge/decision-trees.md, building-effective-agents.md, multi-agent-orchestration.md (pattern selection)
- knowledge/rag-patterns.md (permission-aware retrieval), context-memory.md (state)
- knowledge/security-governance.md (guardrails, identity, OWASP and NIST mappings)
- knowledge/evaluation.md (eval-driven development, gates), interoperability-observability.md (MCP, tracing)
- knowledge/latency-cost-reliability.md, enterprise-architecture.md (capacity, cost, DR)
Live-sourced
- Anthropic API pricing, retrieved 2026-07-14, for the cost model in section 11.
Each knowledge document carries its own primary-source citations; see agentify/knowledge/SOURCES.md for provenance.