--- title: "Troubleshooting GKE Pods in CrashLoopBackOff: How to Start with Strace" date: 2025-03-30 author: "Pengzhan Hao" category: "Tech" tags: ["Container","CrashLoopBackOff","Debug","Docker","GKE","Kubernetes","Strace"] cover: "/posts-assets/troubleshooting-gke-pods-in-crashloopbackoff-how-to-start-with-strace/images/cover.jpg" excerpt: "In most real-world Kubernetes cases, infrastructure engineers don't need to understand how a container itself works. This is reasonable because container image developers are typic..." --- In most real-world *Kubernetes* cases, infrastructure engineers don't need to understand how a container itself works. This is reasonable because container image developers are typically responsible for it. However, a problem often arises when a container is deployed on *Kubernetes*, which is such a complex deployment and orchestration system. Sometimes, unexpected components can influence a container's behavior. Debugging an unexpected container is feasible for self-hosted *Kubernetes*, where engineers have better observability and testing capability. However, if you are using *GKE* or another cloud provider's *Kubernetes* engine, the situation can become complicated. Here's a problem I recently faced. ## The Challenge: Debugging a Crashing vLLM Container on GKE Recently, I deployed a *Large Language Model (LLM)* model serving container on *GKE* as a Deployment. When I ran the serving framework in a non-containerized environment, everything worked perfectly, but it consistently crashed when running inside a *Kubernetes* pod. The most challenging aspect of the situation was that no application stdout or logs were being generated by the application. The only information I had was from `kubectl`, which showed that the container had exited with status code 1.‘ ```bash $ kubectl get pods NAME READY STATUS RESTARTS AGE meta-deployment-789d98c8f7-4fsz4 0/1 CrashLoopBackOff 5 10m ``` To better understand why this issue was happening, I tried the same container with different entry points and commands. I found that the issue was highly related to the keyword "*vllm*". Every time a command contained the string "*vllm*", it would crash (except when I used `echo vllm`). It seemed that `echo` worked because it's a shell built-in command, but any command that involved `exec` crashed. With this context, I decided to use `gdb` it to debug the container. To do that, I needed to rebuild the container image on top of the existing one. However, even running `gdb --args python -m vllm.entrypoints.api_server ....` still resulted in a crash. To move forward, I decided to rely on **[*strace*](https://man7.org/linux/man-pages/man1/strace.1.html)**, a kernel tracing utility, to understand what was happening. Here's the tricky part: *GKE*'s default worker node OS image is ***[COS (Container-Optimized OS)](https://cloud.google.com/container-optimized-os/docs)*** , a Debian-based Linux OS with many restrictions. For example, it lacks binary/package managers, and most paths under its file system are non-executable. So, how could I easily download and use *strace*? ## Step-by-Step Guide: Debugging GKE CrashLoopBackOff with Strace Since we were already on a containerized platform, the simplest approach was to deploy another container and perform all operations within it. This container also needed to fulfill three criteria: - The *strace* pod must be privileged. - The *strace* pod must have the `SYS_PTRACE` Linux capability. - The *strace* pod must be in the host PID namespace to view all processes. Here's an example of pod I used ```bash $ cat &1 | tee strace.log # pid to the target container, e.g. 4128 # Go back to target pod's shell root@meta-deployment-789d98c8f7-4fsz4:/$ python -m vllm.entrypoints.api_server .... # Inside strace pod wait(-1, fork(.... ``` After that, I was able to examine the *strace* log to understand why and when my container was crashing. ## Summary In this post, I explained a complicated scenario and the limitations of *GKE* when debugging it. I also described the solution to overcome these debugging difficulties by enabling the powerful *strace* utility on a *GKE* node. As for the reason behind this strange behavior, that will be discussed in a later post. It's a different but interesting problem. ## Cross-references [1] vLLM deployment example from [GCP Vertex Model Garden](https://cloud.google.com/model-garden?hl=en) ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: meta-deployment spec: replicas: 1 selector: matchLabels: app: meta-server template: metadata: labels: app: meta-server ai.gke.io/model: Llama-3-1-8B-Instruct ai.gke.io/inference-server: vllm examples.ai.gke.io/source: model-garden spec: containers: - name: inference-server image: us-docker.pkg.dev/vertex-ai/vertex-vision-model-garden-dockers/pytorch-vllm-serve:20240821_1034_RC00 resources: requests: cpu: 8 memory: 29Gi ephemeral-storage: 80Gi nvidia.com/gpu : 1 limits: cpu: 8 memory: 29Gi ephemeral-storage: 80Gi nvidia.com/gpu : 1 command: args: - python - -m - vllm.entrypoints.api_server - --host=0.0.0.0 - --port=7080 - --swap-space=16 - --gpu-memory-utilization=0.9 - --max-model-len=32768 - --trust-remote-code - --disable-log-stats - --model=gs://vertex-model-garden-public-us/llama3.1/Meta-Llama-3.1-8B-Instruct - --tensor-parallel-size=1 - --max-num-seqs=12 - --enforce-eager - --disable-custom-all-reduce - --enable-chunked-prefill env: - name: MODEL_ID value: "meta-llama/Llama-3.1-8B-Instruct" - name: DEPLOY_SOURCE value: "UI_NATIVE_MODEL" volumeMounts: - mountPath: /dev/shm name: dshm volumes: - name: dshm emptyDir: medium: Memory nodeSelector: cloud.google.com/gke-accelerator: nvidia-l4 --- apiVersion: v1 kind: Service metadata: name: meta-service spec: selector: app: meta-server type: ClusterIP ports: - protocol: TCP port: 8000 targetPort: 7080 ```