Agents That Wake Up When the World Does
Most agent demos share a birth defect: the loop starts with a user typing a message and ends when the model stops. Real work doesn't. Invoices arrive at 3am, a monitor flips to red while the agent is mid-task, a customer replies six hours after the thread went quiet. The moment your agent must react to events rather than answer requests, you are building a different kind of system — and a handful of system primitives decide whether it survives.
Event-Driven Means a Different Kind of Loop
A request-driven agent owns its lifecycle: start, think, act, finish. An event-driven agent is a resident process: it subscribes to sources (webhooks, queues, email, monitoring), wakes on an event, runs a task, and goes back to sleep. The architecture consequences are immediate:
- State must live outside the process. Any task can be interrupted and resumed; the conversation history and working artifacts belong in durable storage, not in memory.
- Events are not prompts. Raw events need normalization, deduplication, and prioritization before they become agent tasks — otherwise one flapping monitor spawns a hundred urgent sessions.
- The loop needs a supervisor. Retries, timeouts, and deadlock detection are the harness's job. A task that wakes, fails, and re-queues itself forever is a new failure class with no chatbot equivalent.
Safety Points: The Missing Feature in Every Demo
When an agent acts continuously, "wait for user confirmation" can't be the only safety mechanism — sometimes thousands of events arrive between two human glances. The working pattern is safety points: defined boundaries where the agent must pause and yield to an external judgment before proceeding.
- Pre-irreversibility gates. Send, delete, pay, deploy — these transitions always cross a gate, no matter how confident the agent is.
- Budget gates. Per-task and per-day spending caps that block the loop and escalate instead of silently continuing.
- Anomaly gates. Statistical guards on the agent's own behavior: a sudden change in action distribution or a burst of repeated tool calls trips the gate automatically.
Cancellation and Preemption
Two operations that feel trivial until you implement them:
- Cancellation must be cooperative and immediate. The user says "stop" — the agent should abort at the next tool boundary, roll back partial side effects where possible, and record what it was about to do. Killing a process mid-write is how corruption happens. Design every long-running tool call with a cancellation token and an idempotency key.
- Preemption needs priority classes. A P1 incident should not wait behind a batch report. Give agents the same scheduling discipline as operating systems: priority classes, preemption points, and a rule for what happens to the preempted task (suspend with state, never abandon).
Fast Path / Slow Path
| Fast path (reflexes) | Slow path (reasoning) | |
|---|---|---|
| Job | Acknowledge, triage, run a pre-approved playbook | Full context assembly, multi-step planning |
| Model | Small, tight prompts, hard time bounds | Expensive models, human review where warranted |
| Judged on | Latency, false-positive rate | Quality of reasoning |
| Fails when | Conflated with the slow path: slow to acknowledge, shallow in thought | Burns budget on routine deltas that need no reasoning |
The cleanest architecture for mixed workloads splits reactions by latency requirement. The fast path handles reflexes — acknowledge, triage, execute a pre-approved playbook, reply "on it" — with a small model, tight prompts, and hard time bounds. The slow path handles reasoning — full context assembly, multi-step planning, expensive models, human review where warranted. The fast path is judged on latency and false- positive rate; the slow path on quality. Conflating them gives you an agent that is both slow to acknowledge and shallow in thought — the worst of both.
A Worked Example: A Supervisor for Scheduled Reports
Concrete case: an agent that digests nightly CRM exports and posts a morning summary. The naive version is a cron job that fires an LLM call on every file. The production version looks like this:
- Normalize at the door. Each export lands as a normalized event (source, kind, timestamp, checksum). Duplicate deliveries collapse by checksum; a missed 02:00 export raises a "stale source" event at 02:15 instead of silently missing.
- Gate with safety points. The draft summary is reversible (a message to an internal channel). The post to the customer channel is irreversible — it passes an irreversibility gate: anomaly scan on the numbers (deltas beyond 5σ block), budget check (LLM spend today under the cap), and a freshness check on every input source.
- Split the paths. The fast path renders the template summary in under a second when all deltas are routine. The slow path — an LLM analysis — runs only when a delta crosses the anomaly threshold, and it costs 20× more, so it must earn its invocation.
- Cancel cleanly. A corrected export arriving at 06:50 preempts the in-flight slow-path analysis; the agent finishes the current tool call, drops the stale task, and logs the preemption for the weekly reliability review.
The failure modes this design kills: double-posted summaries (deduplication), silently stale dashboards (stale-source events), runaway spend (budget gates), and stale conclusions (preemption on corrected input). None of these show up in a demo — all of them show up in week two of production.
Key Takeaways
- Event-driven agents are resident systems with supervisors, not request handlers with loops.
- Normalize, deduplicate, and prioritize events before they become tasks.
- Safety points — irreversibility, budget, anomaly gates — replace per-action confirmations at scale.
- Cancellation is cooperative at tool boundaries; preemption needs real priority classes.
- Split reflexes (fast path) from reasoning (slow path) and judge each by its own metrics.
