CoderSherlock.github.io/_posts/2024-04-11-Debug-kubelet.md

title, date, tags, author, cover

title	date	tags	author	cover
Debug Kubelet	2024-04-10 03:34:00 -0400	Kubernetes Kubelet Debug	Pengzhan Hao	/static/2024-04/kubelet.png

Debug logs

Like all others program's debugging, the most straightforward way for newbies and the easiest way for advanced developer is relying on logs. Same to debug kubelet, bumping up verbosity to show more logs is the most intuitive approach when facing an issue. Like most component in Kubernetes, kubelet uses klog for logging and there are 10 verbosity levels(0-9).

TL;DR: Bumping up to level 5 would satisfy most debugging needs.

Level	Meaning	Example
0	Always on (Warning, Error, Fatal)	https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/kubelet.go#L757-L757
1	Default level logs when don't want any verbosity	https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/kubelet.go#L2527
2	Most important logs when major operations happen, also the default verbosity level	https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/kubelet.go#L483-L483
3	Extended information	https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/kubelet.go#L2176
4	Debug level	https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/kubelet.go#L1731
5	Trace level	https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/kubelet.go#L2821-L2821
6	Display requested resources	https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/cm/cgroup_manager_linux.go#L401
7	Display HTTP request headers	https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/logs/container_log_manager.go#L299
8	Display HTTP request payload	https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/prober/prober_manager.go#L192

By the time, this note was written. In kubelet related code, level 8 was only used in pkg/kubelet/prober/prober_manager.go and level 7 was only used in pkg/kubelet/logs/container_log_manager.go. And there are 11 occurrences that level 6 was used, and all of them are not part of workload lifecycle related.

Further readings

Inotify watcher leaks in Kubelet