Recently, I faced an issue where Kubelet on a node reported error message failed to create file descriptors.
error creating file watcher: too many open files
@@ -61,7 +61,7 @@ Inode Pathname
258855 /etc/srv/kubernetes/pki/ca-certificates.crt
Put all processes above into one single script, I can retrieve all target files, that would help to understand if there’s a real leakage. Also, I count the unique inode amount, this could also help to know which inode are monitored multiple times.
+Put all processes above into one single script(please see the updated version in the appendix), I can retrieve all target files, that would help to understand if there’s a real leakage. Also, I count the unique inode amount, this could also help to know which inode are monitored multiple times.
cat << EOF | sudo tee -a test.sh
echo "kubelet pid="${PID}
@@ -134,6 +134,61 @@ Count: 7491
debugfs
+Appendix
+
+Updated script to get inotify watchers initiated by kubelet
+
+Thanks for yujuhong@’s momentum and helps in finishing updated script.
+
+PID=$(echo $(ps -aux | grep "/home/kubernetes/bin/kubelet" | head -1) | cut -d " " -f 2)
+echo "Kubelet Pid:" ${PID}
+
+inums_raw=$(find /proc/${PID}/fdinfo -type f 2>/dev/null | xargs grep ^inotify)
+# echo ${inums_raw}
+echo "Count: $(find /proc/${PID}/fdinfo -type f 2>/dev/null | xargs grep ^inotify | wc -l)"
+
+while read -r line;
+do
+ reg="ino:([0-9a-f]*) sdev:([0-9a-f]*)"
+ if [[ ${line} =~ $reg ]]; then
+ ino="${BASH_REMATCH[1]}"
+ sdev="${BASH_REMATCH[2]}"
+ # echo $ino $sdev
+ else
+ echo "wrong line"
+ fi
+
+ sdev_in_dec=$((16#$sdev))
+ minor=$((sdev_in_dec % 256))
+ major=$((sdev_in_dec / 256))
+ # echo "${major}:${minor}"
+
+ in_fds_sdev+=("${ino}-${major}:${minor}")
+done <<< "${inums_raw}"
+
+uniq_pairs=($(echo "${in_fds_sdev[@]}" | sort | uniq))
+echo "Unique target" ${#uniq_pairs[@]}
+
+printf "%-10s %-10s %-6s %s\n" "INUM" "DEV" "COUNT" "TARGET"
+for pair in "${uniq_pairs[@]}"
+do
+ count=$(echo "${in_fds_sdev[@]}" | grep -o "${pair}" | wc -l)
+ fd_hex=$(echo ${pair} | cut -d "-" -f 1)
+ dev=$(echo ${pair} | cut -d "-" -f 2)
+ fd_dec="$((16#${fd_hex}))"
+
+ mount_info=$(grep ${dev} /proc/$PID/mountinfo)
+ if [[ -z $mount_info ]]; then
+ echo "Can't find mount info for" $dev
+ else
+ tmpfs_path=$(echo $mount_info | cut -d " " -f 5)
+ # echo $tmpfs_path
+ loc=$(find ${tmpfs_path} -inum ${fd_dec})
+ printf "%-10s %-10s %-6s %s\n" "${fd_dec}" "${dev}" "${count}" "${loc}"
+ fi
+done
+
+
References
]]>Pengzhan Hao Debug Kubelet 2024-04-10T03:34:00-04:00 2024-04-19T00:41:23-04:00 https://blog.pengzhan.dev/posts/Debug-kubelet Debug logs
+ Like all others program’s debugging, the most straightforward way for newbies and the easiest way for advanced developer is relying on logs. Same to debug kubelet, bumping up verbosity to show more logs is the most intuitive approach when facing an issue. Like most component in Kubernetes, kubelet uses klog for logging and there are 10 verbosity levels(0-9).
By the time, this note was written. In kubelet related code, level 8 was only used in pkg/kubelet/prober/prober_manager.go and level 7 was only used in pkg/kubelet/logs/container_log_manager.go. And there are 11 occurrences that level 6 was used, and all of them are not part of workload lifecycle related.
Further readings
-]]>The reason why I am writing this down is because it has been a week and no students ask for the solution of the last Lab. I realise that learning gap between students are huge, especially when a non-profit university is admitting more and more students. @@ -567,7 +622,7 @@ It’s time to switch from the running scheduler to the selected process. Wait f
-
This generated word cloud figure reflects the most popular economy news’ keyword in the week started 06-28-2020. Two largest words in the figure are “新冠” and “新冠病毒”, both means “Covid-19” (This figure was in the week of the second covid spur in Beijing, China). The size of the image fits my phone screen and I can use an app to automatic sync it to my phone’s wallpaper. However, in this image, too many location nouns are presented. This will be something I can make progress on in the future.
]]>This generated word cloud figure reflects the most popular economy news’ keyword in the week started 06-28-2020. Two largest words in the figure are “新冠” and “新冠病毒”, both means “Covid-19” (This figure was in the week of the second covid spur in Beijing, China). The size of the image fits my phone screen and I can use an app to automatic sync it to my phone’s wallpaper. However, in this image, too many location nouns are presented. This will be something I can make progress on in the future.
]]>@@ -982,7 +1037,7 @@ Using ssh may connect to different physical devices under same domain name, this
target remote localhost:28467
# target remote [ip-addr]:28467
-2016-10
@@ -1092,7 +1147,7 @@ unxz ubuntu-14.04lts-xubuntu-odroid-xu-20140714.img.xz # dump file su dd if=/dev/block/mmcblk0p37 of=/sdcard/boot.img -]]>+]]>
My current solution is using AP to forward all SSL traffic to a proxy, charles proxy is my first choice (Prof asked). It’s a non-free software which still update new versions now. So mainly, I’ll talk about how to charles SSL proxy. @@ -1121,4 +1176,4 @@ You also need to save charles Root Certificate, it also contains in the same men
- Set Proxy and SSL Proxy -