How to Fix Gin Server Graceful Shutdown Timing Out on Kubernetes Pod Termination

When Kubernetes terminates a pod running your Gin service, requests get abruptly cut off instead of finishing cleanly, and you may see connection reset errors on the client side during every deployment or scale-down event. This means your application isn't correctly handling the termination signal Kubernetes sends before it forcibly kills the process.

The Problem

During a rolling deployment or pod scale-down, in-flight requests fail instead of completing:

client error: connection reset by peer

Checking pod events shows Kubernetes forcibly killing the container after its grace period expired, rather than the application exiting cleanly on its own:

$ kubectl describe pod myapp-7d9f8b6c5-x2k9p
Events:
  Warning  Killing  2m  kubelet  Stopping container myapp
  # process didn't exit within terminationGracePeriodSeconds, force-killed with SIGKILL

Why It Happens

Kubernetes terminates a pod by sending SIGTERM first, then waits for terminationGracePeriodSeconds (30 seconds by default) before sending an unconditional SIGKILL if the process hasn't exited on its own. If your Gin application doesn't explicitly listen for SIGTERM and shut down gracefully within that window, one of two things happens: either the process ignores the signal and gets forcibly killed mid-request when the grace period expires, or your shutdown logic exists but takes longer than the grace period to actually finish. Common specific causes:

• No signal handling at all — the application has no code listening for SIGTERM, so Kubernetes' default Go signal behavior (immediate termination) applies, cutting off in-flight requests instantly rather than letting them finish.
• Graceful shutdown logic exists but has too short an internal timeout, or no timeout at all paired with slow in-flight requests that never finish within the available window.
• The grace period itself is too short for how long your application's slowest legitimate requests can take, especially for endpoints doing meaningful background work.
• The Kubernetes Service isn't given time to stop routing new traffic to the terminating pod before SIGTERM is sent, so new requests keep arriving even as shutdown begins.

The Fix

Implement explicit graceful shutdown in your Gin server, listening for SIGTERM and giving in-flight requests a chance to finish before exiting:

func main() {
    router := gin.Default()
    router.GET("/health", healthHandler)

    srv := &http.Server{
        Addr:    ":8080",
        Handler: router,
    }

    go func() {
        if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
            log.Fatalf("listen: %s\n", err)
        }
    }()

    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGTERM, syscall.SIGINT)
    <-quit
    log.Println("shutting down gracefully...")

    ctx, cancel := context.WithTimeout(context.Background(), 25*time.Second)
    defer cancel()

    if err := srv.Shutdown(ctx); err != nil {
        log.Fatalf("forced shutdown: %s", err)
    }
    log.Println("server exited cleanly")
}

srv.Shutdown() stops accepting new connections immediately but lets existing in-flight requests finish, up to the context's timeout — set this timeout comfortably shorter than your pod's terminationGracePeriodSeconds, giving a safety margin for the process to actually exit cleanly afterward.

Explicitly configure a generous enough grace period in your Kubernetes deployment manifest to accommodate your application's realistic longest in-flight request duration:

# deployment.yaml
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 30
      containers:
      - name: myapp
        ...

To handle the gap between Kubernetes deciding to terminate a pod and its Service actually stopping traffic routing to it, add a brief preStop hook that sleeps for a couple of seconds before the application itself even receives SIGTERM — this gives the Service's endpoint controller time to remove the pod from rotation first:

lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 5"]

Still Not Working?

If graceful shutdown is implemented correctly but still occasionally gets force-killed, check whether specific long-running requests (large uploads, slow downstream calls, streaming responses) are legitimately exceeding your shutdown timeout — log the shutdown process's actual duration to see how close it's running to the limit under real traffic:

start := time.Now()
if err := srv.Shutdown(ctx); err != nil {
    log.Printf("forced shutdown after %v: %s", time.Since(start), err)
} else {
    log.Printf("graceful shutdown completed in %v", time.Since(start))
}

If shutdown durations are consistently close to your timeout, either extend terminationGracePeriodSeconds further, or investigate whether those specific slow requests should have their own shorter server-side timeout so they can't indefinitely delay a clean shutdown in the first place.