Case study / 05

Agent Runtime

Reliable execution infrastructure for asynchronous AI workflows.

Provider-agnostic execution infrastructure built with FastAPI, Redis/RQ, and SQLite. The API persists jobs before dispatch; asynchronous workers execute provider calls independently of the HTTP connection. Durable state coordinates idempotent submission, bounded retries, structured-output validation, and cancellation. A dispatcher recovers lost queue deliveries, while execution logs and metrics make attempts and outcomes inspectable. Cancellation protects runtime state and result publication; it cannot undo an external provider call.

FastAPI
Redis / RQ
SQLite
Pydantic
React
Docker

GitHub

Async workersBounded retriesSchema-validated results

Asynchronous Execution

Provider work continues beyond the HTTP request.

Idempotent Submission

Caller key + validated request hash resolve duplicates.

Bounded Recovery

Retryable failures use persisted backoff and a finite budget.

Validated Results

Schema validation gates every persisted success.

System architecture

Request and state layer leading to queued worker execution, output validation, and persisted results, with reliability mechanisms and provider failure handling. — Accept quickly. Execute durably.
View full diagram
SQLite owns execution state. Redis transports work. Workers publish validated results.
**Implementation detail.** The dispatcher is required for recovery. Invalid output and permanent errors fail without automatic retry; only normalized transient provider failures retry.

Engineering decisions

Decision	Why	Tradeoff
Async workers over request-bound execution	Provider work must outlive an open HTTP connection.	Explicit job state and queue coordination.
Redis/RQ over heavier orchestration	Python-native workers, attempt timeouts, and manageable operations.	Larger workloads may need stronger orchestration and recovery coordination.
Explicit provider reliability semantics	Control duplicate submissions, transient failures, invalid output, and late results.	More state transitions and failure paths to model and test.

Under the hood

API / State

FastAPI submission, status, and cancellation endpoints
SQLite records persisted before queue dispatch
Caller key + request hash; changed request returns 409

Queue / Execution

Redis/RQ workers atomically claim an attempt
Provider execution outside the HTTP lifetime
Dispatcher redelivers due work and expires abandoned runs

Reliability / Telemetry

Capped exponential backoff; SDK retries disabled
Pydantic validation before result publication
Structured execution logs and Prometheus snapshot gauges

Technical deep dives

Job flow from submission through queue and worker claim to validation and success, with branches for retry, failure, cancellation, and lost delivery recovery. — Job Lifecycle & Failure Recovery
View full diagram
Durable records coordinate claims, retries, cancellation, and delivery recovery.
**Implementation detail.** The API uses `queued → running → completed`, plus `failed`, `cancelled`, and `timed_out`. Intermediate diagram stages are conceptual. Abandoned running attempts time out; queued deliveries can be recovered. Cancellation blocks future attempts and late publication, not in-flight external effects.

Submission deduplication and normalized provider error classification, branching into a scheduled retry or terminal failure, with validation and cancellation safeguards. — Idempotency & Retry Semantics
View full diagram
One logical job per caller key; a single, observable budget for transient failures.
**Implementation detail.** Deduplication requires a caller key; matching payloads alone create new jobs. A changed request with the same key returns 409. Acceptance returns 202. Backoff is capped exponential without jitter; `max_retries` defaults to 2 and is capped at 5. There is no 24-hour retry TTL or exactly-once provider-call guarantee.

Reliability proof

Automated tests cover concurrent idempotency, duplicate delivery, retry budgets, cancellation races, invalid output, and recovery. Temporary databases, fake providers, and queue stubs isolate correctness; live Redis/provider integration is a separate check.

Automated runtime tests
CI lint, tests & builds
Docker packaging
Schema validation
Execution telemetry

Inspect tests

System checks

Same caller key + valid request resolves to the existing job.
Automatic and manual retries share a bounded budget.
Invalid provider output cannot persist as success.
Cancellation prevents further attempts and late publication.
HTTP disconnects do not cancel accepted jobs.

Next at scale

Revisit SQLite for concurrent, multi-host writes; add tenant isolation and quotas. Fleet-wide retries need jitter and provider-aware coordination. Operational recovery needs stronger readiness checks and integration validation across the database, dispatcher, queue, and workers.