Cloudflare details data pipeline optimizations that cut about 70 GB/s of backbone traffic and defer hardware growth
Cloudflare's Data team describes how it keeps hardware demand down for a pipeline that delivered roughly two billion events per second in May 2026. Over five years its changes have been equivalent to deferring about two and a half years of hardware expansion. This is an engineering blog post with no security incident, vulnerability or threat activity. It is relevant to CTI only as context on large-scale telemetry and log pipeline design.
Technical Details
- Pipeline: Logfwdr collects data from distributed sources (Workers, FL) and sends it to centralized receivers. The receivers feed the Buftee buffering system, and consumers deliver the data to destinations such as ClickHouse.
- 2022: The monolithic schema was split into product-specific schemas, cutting backbone bandwidth by 35-40%.
- 2024: Compression moved from LZ4 to zstd, cutting compressed throughput by about 40% with little CPU impact.
- Spring 2026: Double-writing to two regions was replaced by a shared ingest pool with late replication from Buftee. This eliminated about 43 GB/s of transit traffic and added about 7 GB/s of replication, a net saving of 36 GB/s.
- Summer 2026: Sampling was added in Logfwdr on top of receiver-side sampling. This saved about 25 GB/s (over 2 PB/day) across the top datasets. The sampled row count scales roughly as x0.319 instead of x0.485.
- Fall 2026: Aggregation at every pipeline stage cut the HTTP dataset from 9+ GB/s to under 2 GB/s (about 80%). A generic Ready-Aggregates product was also created.
- Combined effect: About 70 GB/s of compressed backbone traffic was eliminated. The Ready-Analytics receivers regained about 750 CPUs and 3.5 TiB of memory.
Linha do tempo
-
Cloudflare details data pipeline optimizations that cut about 70 GB/s of backbone traffic and defer hardware growth
ler a análise desta fonte ocultar
Cloudflare's Data team describes how it keeps hardware demand down for a pipeline that delivered roughly two billion events per second in May 2026. Over five years its changes have been equivalent to deferring about two and a half years of hardware expansion. This is an engineering blog post with no security incident, vulnerability or threat activity. It is relevant to CTI only as context on large-scale telemetry and log pipeline design.
Technical Details
- Pipeline: Logfwdr collects data from distributed sources (Workers, FL) and sends it to centralized receivers. The receivers feed the Buftee buffering system, and consumers deliver the data to destinations such as ClickHouse.
- 2022: The monolithic schema was split into product-specific schemas, cutting backbone bandwidth by 35-40%.
- 2024: Compression moved from LZ4 to zstd, cutting compressed throughput by about 40% with little CPU impact.
- Spring 2026: Double-writing to two regions was replaced by a shared ingest pool with late replication from Buftee. This eliminated about 43 GB/s of transit traffic and added about 7 GB/s of replication, a net saving of 36 GB/s.
- Summer 2026: Sampling was added in Logfwdr on top of receiver-side sampling. This saved about 25 GB/s (over 2 PB/day) across the top datasets. The sampled row count scales roughly as x0.319 instead of x0.485.
- Fall 2026: Aggregation at every pipeline stage cut the HTTP dataset from 9+ GB/s to under 2 GB/s (about 80%). A generic Ready-Aggregates product was also created.
- Combined effect: About 70 GB/s of compressed backbone traffic was eliminated. The Ready-Analytics receivers regained about 750 CPUs and 3.5 TiB of memory.