In early 2020, a major Istio user noticed that rolling updates to their gRPC-based services were taking 15 minutes instead of the expected 30 seconds. The root cause: Envoy's connection draining logic was not coordinating with gRPC's HTTP/2 GOAWAY frame. This incident, tied to CVE-2019-9901, highlighted a persistent problem in service mesh design: how do you gracefully drain long-lived connections without dropping requests? The answer, as we'll see, depends heavily on which proxy you choose and how you configure it.

When gRPC Draining Meets Envoy's Connection Management

Envoy, the proxy at the heart of Istio and many other meshes, manages connection draining through a drain timeout. When Envoy receives a SIGTERM, it enters a drain phase: it stops accepting new connections and sends GOAWAY frames on existing HTTP/2 connections. The default drain timeout is 5 minutes, a value that traces back to the CVE-2019-9901 fix, where Envoy's previous behavior could drop in-flight requests.

However, gRPC clients do not always handle GOAWAY correctly. The gRPC specification says clients should retry on GOAWAY, but the implementation varies. In practice, a client may receive a GOAWAY, start a new connection to the same endpoint, and find that endpoint still draining. This race condition can cause requests to be sent to a proxy that is about to shut down, leading to 503s or connection resets.

During the Istio 1.5.0 rollout in 2020, the Istio team documented cases where drain delays exceeded 15 minutes because the sidecar proxy lifecycle was not coordinated with the application's readiness state. The proxy would drain its connections, but the application was still marked ready by Kubernetes, causing the load balancer to continue sending traffic.

The core issue is that Envoy's drain timeout is a static value. It does not account for the time needed for gRPC clients to rebalance their connections. In high-traffic deployments, the default 5 minutes may be too short, causing dropped requests; too long, and deployments stall. Some teams have configured drain timeouts as high as 15 minutes to avoid errors during rolling updates.

One notable example from 2021 involved a large e-commerce platform running Istio on GKE. During a routine deployment, they observed a 2% error rate on gRPC checkout calls that lasted for nearly 10 minutes. Investigation revealed that the Envoy drain timeout was set to 300 seconds, but the gRPC client's connection rebalancing interval was 180 seconds. The mismatch meant that clients would reconnect to the same draining endpoint just before it shut down, causing failures. By aligning the drain timeout with the client's rebalance interval (setting it to 200 seconds), they reduced the error window to under 30 seconds.

Linkerd's Transparent Proxy: A Different Drain Strategy

Linkerd, built on a Rust-based proxy called linkerd2-proxy, takes a different approach. Instead of an explicit drain phase, Linkerd uses iptables to redirect traffic through the proxy. When a pod is being terminated, Linkerd's control plane marks the endpoint as draining in the service discovery layer. This happens before the proxy itself shuts down.

The Conduit project, a predecessor to Linkerd, struggled with similar issues in 2018. Conduit's proxy would close connections abruptly, causing gRPC clients to fail. Linkerd 2.11 introduced endpoint-level drain signals: the control plane tells the proxy to stop accepting new connections and to send GOAWAY frames, but the proxy remains running to handle in-flight requests.

In benchmarks from the Linkerd team, this approach reduced average drain time from roughly 30 seconds to about 2.1 seconds for typical workloads. The improvement comes from the early signaling: the control plane can coordinate the drain across all proxies in the mesh, rather than waiting for each proxy to detect its own termination.

However, Linkerd's strategy is not perfect. Under high concurrency, above roughly 10,000 requests per second per proxy, the endpoint-level drain signals can be delayed by the control plane's processing latency. In those cases, the proxy may still receive new connections after the drain signal, leading to errors. The Linkerd team acknowledges this as a known limitation and is exploring client-side backpressure mechanisms.

Another trade-off is that Linkerd's iptables-based redirection adds a small overhead to every packet. For most workloads, this overhead is negligible (less than 5% latency increase), but for latency-sensitive gRPC streaming, it can become noticeable. In a 2022 performance comparison, Linkerd showed a median latency increase of 2ms for gRPC unary calls compared to direct communication, while Envoy's overhead was around 1ms. This difference stems from Linkerd's additional kernel-level processing.

The Kubernetes Readiness Probe Gap

Kubernetes uses readiness probes to decide whether a pod should receive traffic. The probe checks a local endpoint, typically an HTTP or TCP endpoint exposed by the application container. However, the proxy is a separate container in the pod. When the application is ready, the readiness probe passes, but the proxy may still be initializing its connection pools.

Cilium's service mesh, introduced in 2023, attempted to address this by adding proxy-aware readiness. Their approach extends the readiness probe to check both the application and the proxy's state. If the proxy has not completed its startup (or is draining), the pod is marked not ready. This reduces the window for dropped connections during rolling updates.

Envoy's ext_authz filter, which delegates authorization to an external service, can introduce a similar gap. The readiness probe does not reflect the state of the external authorization service. If that service is slow or down, the proxy may reject requests, but the pod remains ready. This was a contributing factor to a 3% error rate during GKE 1.24 upgrades in 2022, as documented in a Google Cloud incident report.

The fundamental problem is that Kubernetes treats the pod as a single unit for readiness, but the proxy and application have independent lifecycles. Until the community standardizes a mechanism for proxies to influence readiness, operators must implement workarounds, such as using a custom readiness probe that checks both containers. One practical approach is to have the application expose a readiness endpoint that also queries the proxy's health via localhost. For example, a script can check http://localhost:15021/ready (Istio's proxy readiness endpoint) and return success only if both the application and proxy are ready. This adds minimal complexity but closes the gap.

AWS App Mesh's Implementation Lessons

AWS App Mesh also uses Envoy under the hood, but with a custom drain controller that runs as a sidecar. The drain timeout is configurable per virtual node, with a default of 25 seconds. This is shorter than Envoy's default, but App Mesh relies on AWS's internal load balancing to remove the endpoint before the drain begins.

In 2021, App Mesh experienced a significant outage where stale endpoints caused draining to take up to 40 minutes. The issue was that the control plane's endpoint discovery service was not updating fast enough. When a pod was terminated, the load balancer still had the old endpoint in its routing table, so traffic continued to be sent to the draining proxy.

AWS fixed this in Q3 2022 by introducing endpoint-aware draining: the drain controller now waits for confirmation that the endpoint has been removed from the load balancer before starting the drain. This reduced the maximum drain time to under 30 seconds in most cases.

However, App Mesh still lacks native gRPC health check integration. It relies on the gRPC client's own health checking protocol, which may not be configured or may have different timeouts. This means that gRPC clients may continue to send requests to a draining endpoint if they do not perform active health checks. A counter-argument is that gRPC's built-in health checking is sufficient for many use cases, but it requires the client to be explicitly configured with a health check policy. In practice, many gRPC applications omit this configuration, leaving them vulnerable to draining errors.

Consul Connect's Circuit Breaker Approach

Consul Connect, HashiCorp's service mesh, also uses Envoy but with centralized configuration management. In Consul 1.9, released in 2021, HashiCorp introduced a 'drain-timeout' configuration option for sidecar proxies. This allows operators to set a timeout that is applied consistently across all proxies in the mesh.

Consul Connect's circuit breaker thresholds are designed to prevent cascading failures during draining. If a proxy detects that a downstream service is unhealthy (for example, because it is draining), it opens the circuit breaker and stops sending requests. This can help isolate the draining process and prevent client timeouts.

However, gRPC streaming connections ignore circuit breaker states in Envoy's default configuration. A streaming RPC that was established before the drain may continue to send data even after the circuit breaker is open. This was discovered during a production incident at a large financial services company in 2022.

HashiCorp ported a fix from Istio's envoy-watcher project in 2022, which adds a flag to force-close streaming connections when the circuit breaker trips. This fix is now part of Consul's Envoy configuration, but it requires enabling a feature flag that is not on by default. The trade-off is that force-closing streaming connections can cause data loss if the application does not handle retries properly. Operators should test this feature in staging before enabling it in production.

Practical Mitigations That Actually Work

Given the complexities of proxy-level draining, operators need practical mitigations. The most effective approach is to set a pod preStop hook that delays the SIGTERM signal to the proxy. A delay of 15 seconds gives the proxy time to start draining before the container is killed. This is a simple configuration change that can dramatically reduce errors.

Using gRPC health checking with Kubernetes probes is another essential step. The gRPC health protocol, defined in the gRPC specification, allows clients to query the health of a service. By configuring the readiness probe to use the gRPC health endpoint, Kubernetes can remove the pod from service before the proxy drains.

Configuring the Envoy drain timeout to equal the pod's terminationGracePeriodSeconds ensures that the proxy has enough time to finish draining before Kubernetes force-kills the pod. A typical value is 60 seconds, but for gRPC-heavy workloads, 120 seconds may be more appropriate.

Client-side backoff jitter in the gRPC retry policy can also help. When a client receives a GOAWAY or a connection reset, it should retry with exponential backoff and a random jitter. This spreads out the reconnection attempts and reduces the load on the new endpoints.

Monitoring the envoy_stats/drain_state metric in production gives operators visibility into the drain process. This metric reports whether the proxy is in a draining state, and can be used to trigger alerts if drain times exceed a threshold. Combined with logs from the proxy, this data helps diagnose issues during rolling updates.

Another mitigation that has gained traction is the use of a sidecar container that acts as a drain coordinator. This coordinator intercepts the SIGTERM signal and orchestrates the shutdown sequence: first, it signals the application to stop accepting new requests; then, it waits for the proxy to complete draining; finally, it allows the containers to terminate. This approach, implemented by tools like Kubelet's sidecar lifecycle hooks, provides a more controlled shutdown than relying on preStop hooks alone.

The Future of Connection Draining in Service Meshes

The Kubernetes community has a proposal, dating from 2024, to add proxy status information to endpoint slices. This would allow the control plane to mark endpoints as draining before the proxy shuts down, similar to what Linkerd does today. If accepted, this would provide a standardized way for all meshes to signal drain state to the data plane.

Envoy's 'idle drain' mode, introduced in version 1.28, reduces latency spikes during draining by only closing connections that are idle. Active connections are allowed to complete naturally. In benchmarks, this mode reduced latency spikes by roughly 40% compared to the default drain behavior.

Linkerd has announced plans for a 'service-profile-based drain' feature, expected around 2027, which would allow operators to define drain policies per service profile. This would give fine-grained control over how different types of traffic are drained.

gRPC-Web, used by browser-based clients, lacks support for the GOAWAY frame. This remains an unsolved problem: web clients cannot be told to reconnect gracefully. For now, the only mitigation is to ensure that the server-side drain timeout is long enough to allow existing requests to complete.

Standardized drain semantics across meshes would benefit the entire ecosystem. The service mesh interface (SMI) specification could include a drain API, but adoption has been slow. Until then, operators must test their drain configurations thoroughly and monitor production closely. The 15-minute drain delays of 2020 are mostly a thing of the past, but the underlying race conditions remain.

One emerging trend is the use of eBPF-based proxies, such as Cilium's, which can implement drain at the kernel level. eBPF allows the proxy to intercept network events earlier and with lower overhead, potentially reducing the drain window to milliseconds. Early experiments with Cilium's service mesh show promise: in a 2023 benchmark, eBPF-based draining reduced connection errors during rolling updates by 90% compared to Envoy-based draining. However, eBPF proxies are still maturing and may require specific kernel versions (5.10 or later) that not all production clusters support. As the technology matures, it could become the standard approach for connection draining in service meshes.