# Kubernetes: NGINX/PHP-FPM graceful shutdown and 502 errors

---

We have a PHP application running with Kubernetes in pods with two dedicated containers — NGINX и PHP-FPM.

The problem is that during downscaling clients get 502 errors. E.g. when a pod is stopping, its containers can not correctly close existing connections.

So, in this post, we will take a closer look at the pods’ termination process in general, and NGINX and PHP-FPM containers in particular.

Testing will be performed on the AWS Elastic Kubernetes Service by the [Yandex.Tank](https://rtfm.co.ua/en/yandex-tank-load-testing-tool-an-overview-configuration-and-examples/) utility.

Ingress resource will create an AWS Application Load Balancer with the [AWS ALB Ingress Controller](https://rtfm.co.ua/en/aws-elastic-kubernetes-service-running-alb-ingress-controller/).

As a runtime on Kubernetes, WorkerNodes Docker is used.

### Pod Lifecycle — Termination of Pods

So, let’s take an overview of the pods’ stopping and termination process.

Basically, a pod is a set of processes running on a Kubernetes WorkerNode, which are stopped by standard IPC ([*Inter-Process Communication*](https://en.wikipedia.org/wiki/Inter-process_communication)) [signals](https://en.wikipedia.org/wiki/Signal_(IPC)).

To give the pod the ability to finish all its operations, a [container runtime](https://kubernetes.io/docs/setup/production-environment/container-runtimes/) at first ties softly stop it (*graceful shutdown*) by sending a `SIGTERM` signal to a PID 1 in each container of this pod (see [docker stop](https://docs.docker.com/engine/reference/commandline/stop/)). Also, a cluster starts counting a *grace period* before force kill this pod by sending a `SIGKILL` signal.

The `SIGTERM` can be overridden by using the [`STOPSIGNAL`](https://docs.docker.com/engine/reference/builder/#stopsignal) in an image used to spin up a container.

Thus, the whole flow of the pod’s deleting is (actually, the part below is a kinda copy of the [official documentation](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination)):

1. a user issues a `kubectl delete pod` or `kubectl scale deployment` command which triggers the flow and the cluster start a countdown of the grace period with the default value set to the 30 seconds
    
2. the API server of the cluster updates the pod’s status — from the *Running* state, it becomes the *Terminating* (see [Container states](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-states)). A `kubelet` on the WorkerNode where this pod is running receives this status update and starts the pod's termination process:
    
3. if a container(s) in the pod has a [preStop hook](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#container-hooks) - `kubelet` will run it. If the hook is still running, the default 30 seconds on the grace period - another 2 seconds will be added. The grace period can be set with the [terminationGracePeriodSeconds](https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/#hook-handler-execution)
    
4. when a `preStop` hook is finished, a `kubelet` will send a notification to the Docker runtime to stop containers related to the pod. The Docker daemon will send the `SIGTERM` signal to a process with the PID 1 in each container. Containers will get the signal in random order.
    
5. **simultaneously** with the beginning of the graceful shutdown — Kubernetes Control Plane (its [`kube-controller-manager`](https://kubernetes.io/docs/concepts/overview/components/#kube-controller-manager)) will remove the pod from the endpoints (see [Kubernetes – Endpoints](https://theithollow.com/2019/02/04/kubernetes-endpoints/)) and a corresponding Service will stop sending traffic to this pod
    
6. after the grace period countdown is finished, a `kubelet` will start *force shutdown* - Docker will send the `SIGKILL` signal to all remaining processes in all containers of the pod which can not be ignored, and those processes will be immediately terminated without change to correctly finish their operations
    
7. `kubelet` triggers deletion of the pod from the API server
    
8. API server deletes a record about this pod from the `etcd`
    

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1692512405912/93fe1d94-d458-408e-8f34-90c4a8a7ba0f.png align="left")

Actually, there are two issues:

1. the NGINX and PHP-FPM receives the `SIGTERM` signal as a force как "brutal murder" and will finish their processes immediately, и завершают работу немедленно, without concern about existing connections (see [Controlling nginx](http://nginx.org/en/docs/control.html) and [php-fpm(8) - Linux man page](https://linux.die.net/man/8/php-fpm))
    
2. the 2 and 3 steps — sending the `SIGTERM` and an endpoint deletion - are performed at the same time. Still, an Ingress Service will update its data about endpoints not momentarily and a pod can be killed before then an Ingress will stop sending traffic to it causing 502 error for clients as the pod can not accept new connections
    

E.g. if we have a connection to an NGINX server, the NGINX master process during the fast shutdown will just drop this connection and our client will receive the 502 error, see the [Avoiding dropped connections in nginx containers with “STOPSIGNAL SIGQUIT”](https://ubuntu.com/blog/avoiding-dropped-connections-in-nginx-containers-with-stopsignal-sigquit).

### NGINX `STOPSIGNAL` and 502

Okay, now we got some understanding of how it’s going — let’s try to reproduce the first issue with NGINX.

The example below is taken from the post above and will be deployed to a Kubernetes cluster.

Prepare a Dockerfile:

```plaintext
FROM nginx

RUN echo 'server {\n\
    listen 80 default_server;\n\
    location / {\n\
      proxy_pass [http://httpbin.org/delay/10;\n\](http://httpbin.org/delay/10;%5Cn%5C)
    }\n\
}' > /etc/nginx/conf.d/default.conf

CMD ["nginx", "-g", "daemon off;"]
```

Here NGINX will `proxy_pass` a request to the [http://httpbin.org](http://httpbin.org) which will respond with a 10 seconds delay emulating a PHP backend.

Build an image and push it to a repository:

```plaintext
$ docker build -t setevoy/nginx-sigterm .
$ docker push setevoy/nginx-sigterm
```

Now, add a Deployment manifest to spin up 10 pods from this image.

Here is the full file with a Namespace, Service, and Ingress, in the following part of this post, will add only updated parts of the manifest:

```plaintext
---
apiVersion: v1
kind: Namespace
metadata:
  name: test-namespace
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-deployment
  namespace: test-namespace
  labels:
    app: test
spec:
  replicas: 10
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
      - name: web
        image: setevoy/nginx-sigterm
        ports:
        - containerPort: 80
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
        readinessProbe:
          tcpSocket:
            port: 80
---
apiVersion: v1
kind: Service
metadata:
  name: test-svc
  namespace: test-namespace
spec:
  type: NodePort
  selector:
    app: test
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ingress
  namespace: test-namespace
  annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}]'
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: test-svc
          servicePort: 80
```

Deploy it:

```plaintext
$ kubectl apply -f test-deployment.yaml
namespace/test-namespace created
deployment.apps/test-deployment created
service/test-svc created
ingress.extensions/test-ingress created
```

Check the Ingress:

```plaintext
$ curl -I aadca942-testnamespace-tes-5874–698012771.us-east-2.elb.amazonaws.com
HTTP/1.1 200 OK
```

And we have 10 pods running:

```plaintext
$ kubectl -n test-namespace get pod
NAME READY STATUS RESTARTS AGE
test-deployment-ccb7ff8b6–2d6gn 1/1 Running 0 26s
test-deployment-ccb7ff8b6–4scxc 1/1 Running 0 35s
test-deployment-ccb7ff8b6–8b2cj 1/1 Running 0 35s
test-deployment-ccb7ff8b6-bvzgz 1/1 Running 0 35s
test-deployment-ccb7ff8b6-db6jj 1/1 Running 0 35s
test-deployment-ccb7ff8b6-h9zsm 1/1 Running 0 20s
test-deployment-ccb7ff8b6-n5rhz 1/1 Running 0 23s
test-deployment-ccb7ff8b6-smpjd 1/1 Running 0 23s
test-deployment-ccb7ff8b6-x5dc2 1/1 Running 0 35s
test-deployment-ccb7ff8b6-zlqxs 1/1 Running 0 25s
```

Prepare a `load.yaml` for the [Yandex.Tank](https://rtfm.co.ua/en/yandex-tank-load-testing-tool-an-overview-configuration-and-examples/):

```plaintext
phantom:
  address: aadca942-testnamespace-tes-5874-698012771.us-east-2.elb.amazonaws.com
  header_http: "1.1"
  headers:
     - "[Host: aadca942-testnamespace-tes-5874-698012771.us-east-2.elb.amazonaws.com]"
  uris:
    - /    
  load_profile:
    load_type: rps
    schedule: const(100,30m)
  ssl: false
console:
  enabled: true
telegraf:
  enabled: false
  package: yandextank.plugins.Telegraf
  config: monitoring.xml
```

Here, we will perform 1 request per second to pods behind our Ingress.

Run tests:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1692512407005/50c8c472-c61a-4d71-b6d9-d724ea3f1783.png align="left")

All good so far.

Now, scale down the Deployment to only one pod:

```plaintext
$ kubectl -n test-namespace scale deploy test-deployment — replicas=1
deployment.apps/test-deployment scaled
```

Pods became *Terminating*:

```plaintext
$ kubectl -n test-namespace get pod
NAME READY STATUS RESTARTS AGE
test-deployment-647ddf455–67gv8 1/1 Terminating 0 4m15s
test-deployment-647ddf455–6wmcq 1/1 Terminating 0 4m15s
test-deployment-647ddf455-cjvj6 1/1 Terminating 0 4m15s
test-deployment-647ddf455-dh7pc 1/1 Terminating 0 4m15s
test-deployment-647ddf455-dvh7g 1/1 Terminating 0 4m15s
test-deployment-647ddf455-gpwc6 1/1 Terminating 0 4m15s
test-deployment-647ddf455-nbgkn 1/1 Terminating 0 4m15s
test-deployment-647ddf455-tm27p 1/1 Running 0 26m
…
```

And we got our 502 errors:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1692512407992/a06e82e2-ad5e-4377-b0ac-90aaddc63e18.png align="left")

Next, update the Dockerfile  —  add the `STOPSIGNAL SIGQUIT`:

```plaintext
FROM nginx

RUN echo 'server {\n\
    listen 80 default_server;\n\
    location / {\n\
      proxy_pass [http://httpbin.org/delay/10;\n\](http://httpbin.org/delay/10;%5Cn%5C)
    }\n\
}' > /etc/nginx/conf.d/default.conf

STOPSIGNAL SIGQUIT

CMD ["nginx", "-g", "daemon off;"]
```

Build, push:

```plaintext
$ docker build -t setevoy/nginx-sigquit .
docker push setevoy/nginx-sigquit
```

Update the Deployment with the new image:

```plaintext
...
    spec:
      containers:
      - name: web
        image: setevoy/nginx-sigquit
        ports:
        - containerPort: 80
...
```

Redeploy, and check again.

Run tests:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1692512409255/fc888310-e951-4b76-a84c-a92bf0899cc7.png align="left")

Scale down the deployment again:

```plaintext
$ kubectl -n test-namespace scale deploy test-deployment — replicas=1
deployment.apps/test-deployment scaled
```

And no errors this time:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1692512410198/c95b0143-01ff-4284-9792-c45a413d85ab.png align="left")

Great!

### Traffic, `preStop`, and `sleep`

But still, if repeat tests a few times we still can get some 502 errors:

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1692512411264/e08a1ebe-eb0a-4490-9c32-f33d8ece4b2d.png align="left")

This time most likely we are facing the second issue — endpoints update is performed at the same time when the `SIGTERM` Is sent.

Let’s add a `preStop` hook with the sleep to give some time to update endpoints and our Ingress, so after the cluster will receive a request to stop a pod, a `kubelet` on a WorkerNode will wait for 5 seconds before sending the `SIGTERM`:

```plaintext
...
    spec:
      containers:
      - name: web
        image: setevoy/nginx-sigquit
        ports:
        - containerPort: 80
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sleep","5"]
...
```

Repeat tests  —  and now everything is fine

Our PHP-FPM had no such issue as its [image](https://hub.docker.com/layers/phpdockerio/php74-fpm/latest/images/sha256-bd812f269663fb369c4c7d810e99f52ae939e45412b600c88fabacb982de858d?context=explore) was initially built with the `STOPSIGNAL SIGQUIT`.

### Other possible solutions

And of course, during debugging, I’ve tried some other approaches to mitigate the issue.

See links at the end of this post, and here I’ll describe them in short terms.

#### `preStop` and `nginx -s quit`

One of the solutions was to add a `preStop` hook which will send `QUIT` to NGINX:

```plaintext
lifecycle:
  preStop:
    exec:
      command:
      - /usr/sbin/nginx
      - -s
      - quit
```

Or:

```plaintext
...
        lifecycle:
          preStop:
            exec:
              command:
              - /bin/sh
              - -SIGQUIT
              - 1
....
```

But it didn’t help. Not sure why, as the idea seems to be correct — instead of waiting for the TERM from Kubernetes/Docker - we are gracefully stopping the NGINX master process by sending QUIT.

You can also run the `strace` utility to check which signal is really received by the NGINX.

#### NGINX + PHP-FPM, `supervisord`, and `stopsignal`

Our application is running in two containers in one pod, but during the debugging, I’ve also tried to use a single container with both NGINX and PHP-FPM, for example, [trafex/alpine-nginx-php7](https://hub.docker.com/r/trafex/alpine-nginx-php7).

There I’ve tried to add to [`stopsignal`](http://supervisord.org/configuration.html#program-x-section-values) to the `supervisor.conf` for both NGINX and PHP-FPM with the `QUIT` value, but this also didn't help, although the idea also seems to be correct.

Still, one can try this way.

#### PHP-FPM, and `process_control_timeout`

In the [Graceful shutdown in Kubernetes is not always trivial](https://medium.com/flant-com/kubernetes-graceful-shutdown-nginx-php-fpm-d5ab266963c2) and on the Stack Overflow in the [Nginx / PHP FPM graceful stop (SIGQUIT): not so graceful](https://stackoverflow.com/questions/36564074/nginx-php-fpm-graceful-stop-sigquit-not-so-graceful) question is a note that FPM’s master process is killed before its child and this can lead to the 502 as well.

Not our current case, but pay your attention to the [`process_control_timeout`](https://www.php.net/manual/ru/install.fpm.configuration.php).

#### NGINX, HTTP, and keep-alive session

Also, it can be a good idea to use the `[Connection: close]` header - then the client will close its connection right after a request is finished and this can decrease the 502 errors count.

But anyway, they will persist if NGINX will get the `SIGTERM` during processing a request.

See the [HTTP persistent connection](https://en.wikipedia.org/wiki/HTTP_persistent_connection).

### Useful links

* [Graceful shutdown in Kubernetes is not always trivial](https://medium.com/flant-com/kubernetes-graceful-shutdown-nginx-php-fpm-d5ab266963c2) ([перевод на Хабре](https://habr.com/ru/company/flant/blog/489994/))
    
* [Gracefully Shutting Down Pods in a Kubernetes Cluster](https://blog.gruntwork.io/gracefully-shutting-down-pods-in-a-kubernetes-cluster-328aecec90d) — the nginx -s quit in the preStop solution, also there is a good description of the issue with the traffic being sent to terminated pods
    
* [Kubernetes best practices: terminating with grace](https://cloud.google.com/blog/products/containers-kubernetes/kubernetes-best-practices-terminating-with-grace)
    
* [Termination of Pods](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination)
    
* [Kubernetes’ dirty endpoint secret and Ingress](https://philpearl.github.io/post/k8s_ingress/)
    
* [Avoiding dropped connections in nginx containers with “STOPSIGNAL SIGQUIT”](https://ubuntu.com/blog/avoiding-dropped-connections-in-nginx-containers-with-stopsignal-sigquit) — actually, here I’ve found our solution plus an idea of how to reproduce it
    

*Originally published at* [*RTFM: Linux, DevOps and system administration*](https://rtfm.co.ua/en/kubernetes-nginx-php-fpm-graceful-shutdown-and-502-errors/)*.*

---