Cloudflare Hyperdrive MySQL GA: shared-cache capability mismatch caused rare empty query responses and hangs
Cloudflare announced general availability of Hyperdrive support for MySQL, its database connection pooler for Cloudflare Workers, and detailed a rare caching bug found during the beta. A high-volume customer saw empty query responses in about 0.001% of cases, only on instances with caching enabled. The root cause was a flawed assumption inherited from the Postgres implementation, that the Client node can cache exactly the bytes it writes back to the Worker. This is an engineering reliability post-mortem, not a security incident, and the article reports no exploitation, data exposure or attacker activity.
Technical Details
- Each Hyperdrive instance has a Client node (runs beside the Worker, handles local caching) and an Endpoint node (holds the connection pool, placed near the origin database). When both run in the same data center they share the CDN cache.
- Hyperdrive speaks to the origin with a fixed set of MySQL capability flags, but must translate responses to the capabilities of the Worker's driver. The mysql2 Node.js library does not support
CLIENT_DEPRECATE_EOF, so Hyperdrive rewrites result sets to add EOF packets and converts OK packets to EOFs. - Because the Client cached the translated bytes, the shared cache could contain responses in the wrong format. Scenario 1: the Endpoint caches a Hyperdrive-encoded response and the Client serves it to the Worker without translation. Scenario 2: the Client caches a Worker-encoded response, the Endpoint reads it and passes it to a different Client, which mistranslates it and sends an early EOF or hangs.
- Other beta bugs: the handshake negotiated
utf8(an alias ofutf8mb3), which broke emojis and was fixed by usingutf8mb4.COM_RESET_CONNECTIONreverted the connection to the server default charset (e.g.latin1), fixed by re-settingutf8mb4after each reset.
Remediation
Hyperdrive now always writes a canonical version of the result bytes, encoded with Hyperdrive's capabilities, into the cache regardless of which node stores it. After the fix, the empty-row problem disappeared and the Client connection success rate reached 99.99%. No customer action is described.
Linha do tempo
-
Cloudflare Hyperdrive MySQL GA: shared-cache capability mismatch caused rare empty query responses and hangs
ler a análise desta fonte ocultar
Cloudflare announced general availability of Hyperdrive support for MySQL, its database connection pooler for Cloudflare Workers, and detailed a rare caching bug found during the beta. A high-volume customer saw empty query responses in about 0.001% of cases, only on instances with caching enabled. The root cause was a flawed assumption inherited from the Postgres implementation, that the Client node can cache exactly the bytes it writes back to the Worker. This is an engineering reliability post-mortem, not a security incident, and the article reports no exploitation, data exposure or attacker activity.
Technical Details
- Each Hyperdrive instance has a Client node (runs beside the Worker, handles local caching) and an Endpoint node (holds the connection pool, placed near the origin database). When both run in the same data center they share the CDN cache.
- Hyperdrive speaks to the origin with a fixed set of MySQL capability flags, but must translate responses to the capabilities of the Worker's driver. The mysql2 Node.js library does not support
CLIENT_DEPRECATE_EOF, so Hyperdrive rewrites result sets to add EOF packets and converts OK packets to EOFs. - Because the Client cached the translated bytes, the shared cache could contain responses in the wrong format. Scenario 1: the Endpoint caches a Hyperdrive-encoded response and the Client serves it to the Worker without translation. Scenario 2: the Client caches a Worker-encoded response, the Endpoint reads it and passes it to a different Client, which mistranslates it and sends an early EOF or hangs.
- Other beta bugs: the handshake negotiated
utf8(an alias ofutf8mb3), which broke emojis and was fixed by usingutf8mb4.COM_RESET_CONNECTIONreverted the connection to the server default charset (e.g.latin1), fixed by re-settingutf8mb4after each reset.
Remediation
Hyperdrive now always writes a canonical version of the result bytes, encoded with Hyperdrive's capabilities, into the cache regardless of which node stores it. After the fix, the empty-row problem disappeared and the Client connection success rate reached 99.99%. No customer action is described.