haopengzhan/CoderSherlock.github.io
1
0
Fork 0
mirror of https://github.com/CoderSherlock/CoderSherlock.github.io.git synced 2026-06-13 08:08:10 -07:00
Code Issues Packages Projects Releases Wiki Activity

Added new post and update some misc

Browse Source
This commit is contained in:
CoderSherlock
2024-04-11 03:51:19 +00:00
parent 477c178ca1
commit 4fc7f42ff4
17 changed files with 207 additions and 75 deletions
Download Patch File Download Diff File Expand all files Collapse all files
_posts/2024-04-11-Debug-kubelet.md
+30
View File
@@ -0,0 +1,30 @@
---
title: "Debug Kubelet"
date: 2024-04-10 03:34:00 -0400
tags: ["Kubernetes", "Kubelet", "Debug"]
author: Pengzhan Hao
cover: '/static/2024-04/Kubelet.webp'
---
## Debug logs
Like all others program's debugging, the most straightforward way for newbies and the easiest way for advanced developer is relying on logs. Same to debug `kubelet`, bumping up verbosity to show more logs is the most intuitive approach when facing an issue. Like most component in Kubernetes, `kubelet` uses `klog` for logging and there are 10 verbosity levels(0-9).
TL;DR: Bumping up to level 5 would satisfy most debugging needs.
| Level | Meaning | Example |
| ----- | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| 0 | Always on (Warning, Error, Fatal) | https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/kubelet.go#L757-L757 |
| 1 | Default level logs when don't want any verbosity | https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/kubelet.go#L2527 |
| 2 | Most important logs when major operations happen, also the default verbosity level | https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/kubelet.go#L483-L483 |
| 3 | Extended information | https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/kubelet.go#L2176 |
| 4 | Debug level | https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/kubelet.go#L1731 |
| 5 | Trace level | https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/kubelet.go#L2821-L2821 |
| 6 | Display requested resources | https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/cm/cgroup_manager_linux.go#L401 |
| 7 | Display HTTP request headers | https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/logs/container_log_manager.go#L299 |
| 8 | Display HTTP request payload | https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/prober/prober_manager.go#L192 |
By the time, this note was written. In `kubelet` related code, level 8 was only used in `pkg/kubelet/prober/prober_manager.go` and level 7 was only used in `pkg/kubelet/logs/container_log_manager.go`. And there are 11 occurrences that level 6 was used, and all of them are not part of workload lifecycle related.
## Further readings
[Inotify watcher leaks in Kubelet]