Cloudflare details D1 control plane architecture for scaling to millions of serverless SQLite databases
Cloudflare published an engineering deep dive on how the control plane of D1, its managed serverless database built on SQLite, scales to millions of databases, with hundreds of thousands active at any time. The post reports 15x growth in live databases and 18x growth in daily active databases over the past year (as of October 8, 2026). It is product and architecture news with no vulnerability, incident or threat activity described. The design keeps queries off centralized single-region components and relies on eventual consistency and self-correcting processes.
Technical Details
- Each D1 database is backed by one SQLite Durable Object. The Edge API Worker handles HTTP API requests, and the Eyeball Worker proxies all query traffic to the Durable Objects.
- CRUD endpoints use Hyperdrive to reach a central Postgres cluster, with read replicas and caching where possible. Workers KV holds routing metadata so that queries rarely touch Postgres.
- Database creation initializes the Durable Object first, then writes the metadata row to Postgres as the creation checkpoint, with retries using new IDs. Durable Objects heartbeat every 30 seconds (plus jitter) to self-correct state and clean up orphaned objects.
- Storage limit enforcement uses a Kafka-like queue to smooth writes, delta tables and a scheduled job that runs every second. The job pre-aggregates per-account usage with a Postgres CTE (
DELETE ... RETURNING) to avoid row lock contention and linear scans. The totals are eventually consistent. - Stated limits are 5 GB per account on the Free plan and 1 TB on Workers Paid.
Remediation
No remediation is applicable. The article describes design principles: design for failure and idempotency, avoid single-region components in hot paths, use KV for routing metadata, precompute aggregations, and load-test with realistic workloads.
Linha do tempo
-
Cloudflare details D1 control plane architecture for scaling to millions of serverless SQLite databases
ler a análise desta fonte ocultar
Cloudflare published an engineering deep dive on how the control plane of D1, its managed serverless database built on SQLite, scales to millions of databases, with hundreds of thousands active at any time. The post reports 15x growth in live databases and 18x growth in daily active databases over the past year (as of October 8, 2026). It is product and architecture news with no vulnerability, incident or threat activity described. The design keeps queries off centralized single-region components and relies on eventual consistency and self-correcting processes.
Technical Details
- Each D1 database is backed by one SQLite Durable Object. The Edge API Worker handles HTTP API requests, and the Eyeball Worker proxies all query traffic to the Durable Objects.
- CRUD endpoints use Hyperdrive to reach a central Postgres cluster, with read replicas and caching where possible. Workers KV holds routing metadata so that queries rarely touch Postgres.
- Database creation initializes the Durable Object first, then writes the metadata row to Postgres as the creation checkpoint, with retries using new IDs. Durable Objects heartbeat every 30 seconds (plus jitter) to self-correct state and clean up orphaned objects.
- Storage limit enforcement uses a Kafka-like queue to smooth writes, delta tables and a scheduled job that runs every second. The job pre-aggregates per-account usage with a Postgres CTE (
DELETE ... RETURNING) to avoid row lock contention and linear scans. The totals are eventually consistent. - Stated limits are 5 GB per account on the Free plan and 1 TB on Workers Paid.
Remediation
No remediation is applicable. The article describes design principles: design for failure and idempotency, avoid single-region components in hot paths, use KV for routing metadata, precompute aggregations, and load-test with realistic workloads.