Updated inotify leak in kubelet post

_posts/2024-04-11-inotify-watcher-leaks-in-kubelet.md

@@ -77,7 +77,7 @@ Inode	Pathname
 258855	/etc/srv/kubernetes/pki/ca-certificates.crt
 ```
 
-Put all processes above into one single script, I can retrieve all target files, that would help to understand if there's a real leakage. Also, I count the unique inode amount, this could also help to know which inode are monitored multiple times.
+Put all processes above into one single script(please see the [updated version in the appendix](#updated-script-to-get-inotify-watchers-initiated-by-kubelet)), I can retrieve all target files, that would help to understand if there's a real leakage. Also, I count the unique inode amount, this could also help to know which inode are monitored multiple times.
 
 ```bash
 cat << EOF | sudo tee -a test.sh
@@ -149,6 +149,62 @@ This turns things easy, because I can just use pod ID to compare between running
 - [How Kubelet leaked inotify watchers?]()
 - [debugfs]()
 
+## Appendix
+
+### Updated script to get inotify watchers initiated by kubelet
+
+Thanks for [yujuhong@](https://github.com/yujuhong)'s momentum and helps in finishing updated script.
+
+```bash
+PID=$(echo $(ps -aux | grep "/home/kubernetes/bin/kubelet" | head -1) |  cut -d " " -f 2)
+echo "Kubelet Pid:" ${PID}
+
+inums_raw=$(find /proc/${PID}/fdinfo -type f 2>/dev/null | xargs grep ^inotify)
+# echo ${inums_raw}
+echo "Count: $(find /proc/${PID}/fdinfo -type f 2>/dev/null | xargs grep ^inotify | wc -l)"
+
+while read -r line;
+do
+        reg="ino:([0-9a-f]*) sdev:([0-9a-f]*)"
+        if [[ ${line} =~ $reg ]]; then
+                ino="${BASH_REMATCH[1]}"
+                sdev="${BASH_REMATCH[2]}"
+                # echo $ino $sdev
+        else
+                echo "wrong line"
+        fi
+
+        sdev_in_dec=$((16#$sdev))
+        minor=$((sdev_in_dec % 256))
+        major=$((sdev_in_dec / 256))
+        # echo "${major}:${minor}"
+
+        in_fds_sdev+=("${ino}-${major}:${minor}")
+done <<< "${inums_raw}"
+
+uniq_pairs=($(echo "${in_fds_sdev[@]}" | sort | uniq))
+echo "Unique target" ${#uniq_pairs[@]}
+
+printf "%-10s %-10s %-6s %s\n" "INUM" "DEV" "COUNT" "TARGET"
+for pair in "${uniq_pairs[@]}"
+do
+        count=$(echo "${in_fds_sdev[@]}" | grep -o "${pair}" | wc -l)
+        fd_hex=$(echo ${pair} | cut -d "-" -f 1)
+        dev=$(echo ${pair} | cut -d "-" -f 2)
+        fd_dec="$((16#${fd_hex}))"
+
+        mount_info=$(grep ${dev} /proc/$PID/mountinfo)
+        if [[ -z $mount_info ]]; then
+                echo "Can't find mount info for" $dev
+        else
+                tmpfs_path=$(echo $mount_info | cut -d " " -f 5)
+                # echo $tmpfs_path
+                loc=$(find ${tmpfs_path} -inum ${fd_dec})
+                printf "%-10s %-10s %-6s %s\n" "${fd_dec}" "${dev}" "${count}" "${loc}"
+        fi
+done
+```
+
 ## References
 
 [^flbit_ino]: [Fluentbit error "cannot adjust chunk size" on GKE](https://stackoverflow.com/a/76712244)

_site/posts/charles-is-not-a-good-tool.html

@@ -34,7 +34,7 @@
 <meta name="twitter:card" content="summary" />
 <meta property="twitter:title" content="Using charles proxy to monitor mobile SSL traffics" />
 <script type="application/ld+json">
-{"@context":"https://schema.org","@type":"BlogPosting","author":{"@type":"Person","name":"Pengzhan Hao"},"dateModified":"2024-04-19T00:41:23-04:00","datePublished":"2016-10-27T22:50:33-04:00","description":"In this blog, I will generally talk about how to use proper tools to monitor SSL traffics of a mobile devices. Currently, I only can dealing with those SSL traffics which use an obviously certification. Some applications may not using system root cert or they doesn’t provide us a method to modify their own certs. For these situation, I still didn’t find a good solutions for it. But I’ll keep updating this if I get one. My current solution is using AP to forward all SSL traffic to a proxy, charles proxy is my first choice (Prof asked). It’s a non-free software which still update new versions now. So mainly, I’ll talk about how to charles SSL proxy.","headline":"Using charles proxy to monitor mobile SSL traffics","mainEntityOfPage":{"@type":"WebPage","@id":"https://blog.pengzhan.dev/posts/charles-is-not-a-good-tool"},"url":"https://blog.pengzhan.dev/posts/charles-is-not-a-good-tool"}</script>
+{"@context":"https://schema.org","@type":"BlogPosting","author":{"@type":"Person","name":"Pengzhan Hao"},"dateModified":"2024-04-24T00:32:03-04:00","datePublished":"2016-10-27T22:50:33-04:00","description":"In this blog, I will generally talk about how to use proper tools to monitor SSL traffics of a mobile devices. Currently, I only can dealing with those SSL traffics which use an obviously certification. Some applications may not using system root cert or they doesn’t provide us a method to modify their own certs. For these situation, I still didn’t find a good solutions for it. But I’ll keep updating this if I get one. My current solution is using AP to forward all SSL traffic to a proxy, charles proxy is my first choice (Prof asked). It’s a non-free software which still update new versions now. So mainly, I’ll talk about how to charles SSL proxy.","headline":"Using charles proxy to monitor mobile SSL traffics","mainEntityOfPage":{"@type":"WebPage","@id":"https://blog.pengzhan.dev/posts/charles-is-not-a-good-tool"},"url":"https://blog.pengzhan.dev/posts/charles-is-not-a-good-tool"}</script>
 <!-- End Jekyll SEO tag -->
 
 <!-- end custom head snippets -->

_site/posts/eddl-how-do-we-train-on-limited-edge-devices.html

@@ -34,7 +34,7 @@
 <meta name="twitter:card" content="summary" />
 <meta property="twitter:title" content="EDDL: How do we train neural networks on limited edge devices - PART 1" />
 <script type="application/ld+json">
-{"@context":"https://schema.org","@type":"BlogPosting","author":{"@type":"Person","name":"Pengzhan Hao"},"dateModified":"2024-04-19T00:41:23-04:00","datePublished":"2021-10-13T16:53:20-04:00","description":"This post introduces our previous milestone in project “Edge trainer”, as the paper “EDDL: A Distributed Deep Learning System for Resource-limited Edge Computing Environment.” was published. As the first part of the introductions, I focus only on the motivation and summary of our works. More details in design and implementation can be found in late posts.","headline":"EDDL: How do we train neural networks on limited edge devices - PART 1","mainEntityOfPage":{"@type":"WebPage","@id":"https://blog.pengzhan.dev/posts/eddl-how-do-we-train-on-limited-edge-devices"},"url":"https://blog.pengzhan.dev/posts/eddl-how-do-we-train-on-limited-edge-devices"}</script>
+{"@context":"https://schema.org","@type":"BlogPosting","author":{"@type":"Person","name":"Pengzhan Hao"},"dateModified":"2024-04-24T00:32:03-04:00","datePublished":"2021-10-13T16:53:20-04:00","description":"This post introduces our previous milestone in project “Edge trainer”, as the paper “EDDL: A Distributed Deep Learning System for Resource-limited Edge Computing Environment.” was published. As the first part of the introductions, I focus only on the motivation and summary of our works. More details in design and implementation can be found in late posts.","headline":"EDDL: How do we train neural networks on limited edge devices - PART 1","mainEntityOfPage":{"@type":"WebPage","@id":"https://blog.pengzhan.dev/posts/eddl-how-do-we-train-on-limited-edge-devices"},"url":"https://blog.pengzhan.dev/posts/eddl-how-do-we-train-on-limited-edge-devices"}</script>
 <!-- End Jekyll SEO tag -->
 
 <!-- end custom head snippets -->

_site/posts/generate-word-cloud-with-chinese-fenci.html

@@ -34,7 +34,7 @@
 <meta name="twitter:card" content="summary" />
 <meta property="twitter:title" content="Generate Word Cloud Figures with Chinese-Tokenization and WordCloud python libraries" />
 <script type="application/ld+json">
-{"@context":"https://schema.org","@type":"BlogPosting","author":{"@type":"Person","name":"Pengzhan Hao"},"dateModified":"2024-04-19T00:41:23-04:00","datePublished":"2020-09-15T22:00:14-04:00","description":"Let’s generate a word cloud like this. Don’t understand the language is not a big deal. If your written language is based on latin alphabet(or other language has space between words), skip tokenization.","headline":"Generate Word Cloud Figures with Chinese-Tokenization and WordCloud python libraries","mainEntityOfPage":{"@type":"WebPage","@id":"https://blog.pengzhan.dev/posts/generate-word-cloud-with-chinese-fenci"},"url":"https://blog.pengzhan.dev/posts/generate-word-cloud-with-chinese-fenci"}</script>
+{"@context":"https://schema.org","@type":"BlogPosting","author":{"@type":"Person","name":"Pengzhan Hao"},"dateModified":"2024-04-24T00:32:03-04:00","datePublished":"2020-09-15T22:00:14-04:00","description":"Let’s generate a word cloud like this. Don’t understand the language is not a big deal. If your written language is based on latin alphabet(or other language has space between words), skip tokenization.","headline":"Generate Word Cloud Figures with Chinese-Tokenization and WordCloud python libraries","mainEntityOfPage":{"@type":"WebPage","@id":"https://blog.pengzhan.dev/posts/generate-word-cloud-with-chinese-fenci"},"url":"https://blog.pengzhan.dev/posts/generate-word-cloud-with-chinese-fenci"}</script>
 <!-- End Jekyll SEO tag -->
 
 <!-- end custom head snippets -->

_site/posts/intro-xv6.html

@@ -34,7 +34,7 @@
 <meta name="twitter:card" content="summary" />
 <meta property="twitter:title" content="Xv6 introduction" />
 <script type="application/ld+json">
-{"@context":"https://schema.org","@type":"BlogPosting","author":{"@type":"Person","name":"Pengzhan Hao"},"dateModified":"2024-04-19T00:41:23-04:00","datePublished":"2017-07-28T14:56:55-04:00","description":"In this post, you will learn a few basic concepts of xv6. Learning path will be closed coupled to first project assignment I gave when I assisted in teaching OS classes. Understand system call and know how to implement a simple one will be coved as the first half. In the second half of this post, I will discuss a little bit more on how to debug xv6 using gdb.","headline":"Xv6 introduction","mainEntityOfPage":{"@type":"WebPage","@id":"https://blog.pengzhan.dev/posts/intro-xv6"},"url":"https://blog.pengzhan.dev/posts/intro-xv6"}</script>
+{"@context":"https://schema.org","@type":"BlogPosting","author":{"@type":"Person","name":"Pengzhan Hao"},"dateModified":"2024-04-24T00:32:03-04:00","datePublished":"2017-07-28T14:56:55-04:00","description":"In this post, you will learn a few basic concepts of xv6. Learning path will be closed coupled to first project assignment I gave when I assisted in teaching OS classes. Understand system call and know how to implement a simple one will be coved as the first half. In the second half of this post, I will discuss a little bit more on how to debug xv6 using gdb.","headline":"Xv6 introduction","mainEntityOfPage":{"@type":"WebPage","@id":"https://blog.pengzhan.dev/posts/intro-xv6"},"url":"https://blog.pengzhan.dev/posts/intro-xv6"}</script>
 <!-- End Jekyll SEO tag -->
 
 <!-- end custom head snippets -->

_site/posts/some-of-my-previews-exper-work.html

@@ -34,7 +34,7 @@
 <meta name="twitter:card" content="summary" />
 <meta property="twitter:title" content="Some of my previews experiment works: 2016" />
 <script type="application/ld+json">
-{"@context":"https://schema.org","@type":"BlogPosting","author":{"@type":"Person","name":"Pengzhan Hao"},"dateModified":"2024-04-19T00:41:23-04:00","datePublished":"2016-10-28T12:27:33-04:00","description":"This blog contains only some basic record of my works. For some details, I will write a unique blog just for some specific topics.","headline":"Some of my previews experiment works: 2016","mainEntityOfPage":{"@type":"WebPage","@id":"https://blog.pengzhan.dev/posts/some-of-my-previews-exper-work"},"url":"https://blog.pengzhan.dev/posts/some-of-my-previews-exper-work"}</script>
+{"@context":"https://schema.org","@type":"BlogPosting","author":{"@type":"Person","name":"Pengzhan Hao"},"dateModified":"2024-04-24T00:32:03-04:00","datePublished":"2016-10-28T12:27:33-04:00","description":"This blog contains only some basic record of my works. For some details, I will write a unique blog just for some specific topics.","headline":"Some of my previews experiment works: 2016","mainEntityOfPage":{"@type":"WebPage","@id":"https://blog.pengzhan.dev/posts/some-of-my-previews-exper-work"},"url":"https://blog.pengzhan.dev/posts/some-of-my-previews-exper-work"}</script>
 <!-- End Jekyll SEO tag -->
 
 <!-- end custom head snippets -->

_site/posts/welcome-to-my-blog.html

@@ -34,7 +34,7 @@
 <meta name="twitter:card" content="summary" />
 <meta property="twitter:title" content="STSD: Stop Talking Start Doing" />
 <script type="application/ld+json">
-{"@context":"https://schema.org","@type":"BlogPosting","author":{"@type":"Person","name":"Pengzhan Hao"},"dateModified":"2024-04-19T00:41:23-04:00","datePublished":"2016-10-26T22:50:33-04:00","description":"My personal blog, some contents are useful, the others are not. Just like my mediocre life.","headline":"STSD: Stop Talking Start Doing","mainEntityOfPage":{"@type":"WebPage","@id":"https://blog.pengzhan.dev/posts/welcome-to-my-blog"},"url":"https://blog.pengzhan.dev/posts/welcome-to-my-blog"}</script>
+{"@context":"https://schema.org","@type":"BlogPosting","author":{"@type":"Person","name":"Pengzhan Hao"},"dateModified":"2024-04-24T00:32:03-04:00","datePublished":"2016-10-26T22:50:33-04:00","description":"My personal blog, some contents are useful, the others are not. Just like my mediocre life.","headline":"STSD: Stop Talking Start Doing","mainEntityOfPage":{"@type":"WebPage","@id":"https://blog.pengzhan.dev/posts/welcome-to-my-blog"},"url":"https://blog.pengzhan.dev/posts/welcome-to-my-blog"}</script>
 <!-- End Jekyll SEO tag -->
 
 <!-- end custom head snippets -->

_site/sitemap.xml

@@ -2,43 +2,43 @@
 <urlset xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd" xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
 <url>
 <loc>https://blog.pengzhan.dev/posts/welcome-to-my-blog</loc>
-<lastmod>2024-04-19T00:41:23-04:00</lastmod>
+<lastmod>2024-04-24T00:32:03-04:00</lastmod>
 </url>
 <url>
 <loc>https://blog.pengzhan.dev/posts/charles-is-not-a-good-tool</loc>
-<lastmod>2024-04-19T00:41:23-04:00</lastmod>
+<lastmod>2024-04-24T00:32:03-04:00</lastmod>
 </url>
 <url>
 <loc>https://blog.pengzhan.dev/posts/some-of-my-previews-exper-work</loc>
-<lastmod>2024-04-19T00:41:23-04:00</lastmod>
+<lastmod>2024-04-24T00:32:03-04:00</lastmod>
 </url>
 <url>
 <loc>https://blog.pengzhan.dev/posts/intro-xv6</loc>
-<lastmod>2024-04-19T00:41:23-04:00</lastmod>
+<lastmod>2024-04-24T00:32:03-04:00</lastmod>
 </url>
 <url>
 <loc>https://blog.pengzhan.dev/posts/generate-word-cloud-with-chinese-fenci</loc>
-<lastmod>2024-04-19T00:41:23-04:00</lastmod>
+<lastmod>2024-04-24T00:32:03-04:00</lastmod>
 </url>
 <url>
 <loc>https://blog.pengzhan.dev/posts/eddl-how-do-we-train-on-limited-edge-devices</loc>
-<lastmod>2024-04-19T00:41:23-04:00</lastmod>
+<lastmod>2024-04-24T00:32:03-04:00</lastmod>
 </url>
 <url>
 <loc>https://blog.pengzhan.dev/posts/eddl-how-do-we-train-on-limited-edge-devices-part2</loc>
-<lastmod>2024-04-19T00:41:23-04:00</lastmod>
+<lastmod>2024-04-24T00:32:03-04:00</lastmod>
 </url>
 <url>
 <loc>https://blog.pengzhan.dev/posts/cs350-labs</loc>
-<lastmod>2024-04-19T00:41:23-04:00</lastmod>
+<lastmod>2024-04-24T00:32:03-04:00</lastmod>
 </url>
 <url>
 <loc>https://blog.pengzhan.dev/posts/Debug-kubelet</loc>
-<lastmod>2024-04-19T00:41:23-04:00</lastmod>
+<lastmod>2024-04-24T00:32:03-04:00</lastmod>
 </url>
 <url>
 <loc>https://blog.pengzhan.dev/posts/inotify-watcher-leaks-in-kubelet</loc>
-<lastmod>2024-04-18T21:14:15-04:00</lastmod>
+<lastmod>2024-04-24T00:32:03-04:00</lastmod>
 </url>
 <url>
 <loc>https://blog.pengzhan.dev/archive</loc>