<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en"><generator uri="https://jekyllrb.com/" version="4.2.2">Jekyll</generator><link href="https://blog.pengzhan.dev/feed.xml" rel="self" type="application/atom+xml" /><link href="https://blog.pengzhan.dev/" rel="alternate" type="text/html" hreflang="en" /><updated>2024-04-13T17:17:15-04:00</updated><id>https://blog.pengzhan.dev/feed.xml</id><title type="html">STSD</title><subtitle>My personal blog, some contents are useful, the others are not. Just like my mediocre life.
</subtitle><author><name>Pengzhan Hao</name><email>haopengzhan@gmail.com</email></author><entry><title type="html">Debug Kubelet</title><link href="https://blog.pengzhan.dev/posts/Debug-kubelet" rel="alternate" type="text/html" title="Debug Kubelet" /><published>2024-04-10T03:34:00-04:00</published><updated>2024-04-10T23:51:19-04:00</updated><id>https://blog.pengzhan.dev/posts/Debug-kubelet</id><content type="html" xml:base="https://blog.pengzhan.dev/posts/Debug-kubelet"><![CDATA[<h2 id="debug-logs">Debug logs</h2>

<p>Like all others program’s debugging, the most straightforward way for newbies and the easiest way for advanced developer is relying on logs. Same to debug <code class="language-plaintext highlighter-rouge">kubelet</code>, bumping up verbosity to show more logs is the most intuitive approach when facing an issue. Like most component in Kubernetes, <code class="language-plaintext highlighter-rouge">kubelet</code> uses <code class="language-plaintext highlighter-rouge">klog</code> for logging and there are 10 verbosity levels(0-9).</p>

<p>TL;DR: Bumping up to level 5 would satisfy most debugging needs.</p>

<table>
  <thead>
    <tr>
      <th>Level</th>
      <th>Meaning</th>
      <th>Example</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>0</td>
      <td>Always on (Warning, Error, Fatal)</td>
      <td>https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/kubelet.go#L757-L757</td>
    </tr>
    <tr>
      <td>1</td>
      <td>Default level logs when don’t want any verbosity</td>
      <td>https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/kubelet.go#L2527</td>
    </tr>
    <tr>
      <td>2</td>
      <td>Most important logs when major operations happen, also the default verbosity level</td>
      <td>https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/kubelet.go#L483-L483</td>
    </tr>
    <tr>
      <td>3</td>
      <td>Extended information</td>
      <td>https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/kubelet.go#L2176</td>
    </tr>
    <tr>
      <td>4</td>
      <td>Debug level</td>
      <td>https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/kubelet.go#L1731</td>
    </tr>
    <tr>
      <td>5</td>
      <td>Trace level</td>
      <td>https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/kubelet.go#L2821-L2821</td>
    </tr>
    <tr>
      <td>6</td>
      <td>Display requested resources</td>
      <td>https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/cm/cgroup_manager_linux.go#L401</td>
    </tr>
    <tr>
      <td>7</td>
      <td>Display HTTP request headers</td>
      <td>https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/logs/container_log_manager.go#L299</td>
    </tr>
    <tr>
      <td>8</td>
      <td>Display HTTP request payload</td>
      <td>https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/prober/prober_manager.go#L192</td>
    </tr>
  </tbody>
</table>

<p>By the time, this note was written. In <code class="language-plaintext highlighter-rouge">kubelet</code> related code, level 8 was only used in <code class="language-plaintext highlighter-rouge">pkg/kubelet/prober/prober_manager.go</code> and level 7 was only used in <code class="language-plaintext highlighter-rouge">pkg/kubelet/logs/container_log_manager.go</code>.  And there are  11 occurrences that level 6 was used, and all of them are not part of workload lifecycle related.</p>

<h2 id="further-readings">Further readings</h2>
<p>[Inotify watcher leaks in Kubelet]</p>]]></content><author><name>Pengzhan Hao</name></author><category term="Kubernetes" /><category term="Kubelet" /><category term="Debug" /><summary type="html"><![CDATA[Debug logs Like all others program’s debugging, the most straightforward way for newbies and the easiest way for advanced developer is relying on logs. Same to debug kubelet, bumping up verbosity to show more logs is the most intuitive approach when facing an issue. Like most component in Kubernetes, kubelet uses klog for logging and there are 10 verbosity levels(0-9). TL;DR: Bumping up to level 5 would satisfy most debugging needs. Level Meaning Example 0 Always on (Warning, Error, Fatal) https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/kubelet.go#L757-L757 1 Default level logs when don’t want any verbosity https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/kubelet.go#L2527 2 Most important logs when major operations happen, also the default verbosity level https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/kubelet.go#L483-L483 3 Extended information https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/kubelet.go#L2176 4 Debug level https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/kubelet.go#L1731 5 Trace level https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/kubelet.go#L2821-L2821 6 Display requested resources https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/cm/cgroup_manager_linux.go#L401 7 Display HTTP request headers https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/logs/container_log_manager.go#L299 8 Display HTTP request payload https://github.com/kubernetes/kubernetes/blob/d9c54f69d4bb7ae1bb655e1a2a50297d615025b5/pkg/kubelet/prober/prober_manager.go#L192 By the time, this note was written. In kubelet related code, level 8 was only used in pkg/kubelet/prober/prober_manager.go and level 7 was only used in pkg/kubelet/logs/container_log_manager.go. And there are 11 occurrences that level 6 was used, and all of them are not part of workload lifecycle related. Further readings [Inotify watcher leaks in Kubelet]]]></summary></entry><entry><title type="html">Labs of CS350</title><link href="https://blog.pengzhan.dev/posts/cs350-labs" rel="alternate" type="text/html" title="Labs of CS350" /><published>2022-02-22T16:08:17-05:00</published><updated>2022-05-04T19:45:56-04:00</updated><id>https://blog.pengzhan.dev/posts/cs350-labs</id><content type="html" xml:base="https://blog.pengzhan.dev/posts/cs350-labs"><![CDATA[<p>This will be a series regarding lab I gave during the spring 2022 semester.</p>

<p>The reason why I am writing this down is because it has been a week and no students ask for the solution of the last Lab.
I realise that learning gap between students are huge, especially when a non-profit university is admitting more and more students.
To help all students in understanding concepts of modern OS, I decided to write this post.</p>

<p>It starts with the past lab content I have (as the skelton), and will be amended with extra materials I think it helps.
Remember, it’s for helping in learning. DON’T COPY &amp; PASTE CODE!</p>

<h2 id="index">Index</h2>
<p><a href="#lab1-introduction">Lab1: Introduction of Makefile and Xv6.</a><br />
<a href="#lab3-process">Lab3: System calls for process management.</a><br />
<a href="#lab4-ipc">Lab4: Inter-processes communication.</a><br />
<a href="#lab6-7-scheduling">Lab6/7: CPU scheduling.</a></p>

<h2 id="lab1-introduction">Lab1-Introduction</h2>

<h2 id="lab3-process">Lab3-Process</h2>

<h2 id="lab4-ipc">Lab4-IPC</h2>

<h2 id="lab6-7-scheduling">Lab6-7-Scheduling</h2>

<h3 id="first-user-process-in-xv6">First user process in xv6</h3>
<h4 id="kernel-works">Kernel works</h4>
<p>In xv6, as the same as conventional linux OS, the very first user-level process is <strong>init</strong>.
Before <strong>init</strong>’s running, all the OS bootstraps happen in a highly privileged mode(kernel level).</p>

<p>Xv6’s kernel has the entry point as the main function located in the file <em>main.c</em>.
The main function invokes 17 functions to set up kernel page tables, interrupt handlers, I/O devices and etc.
When all kernel preparations are done, by calling the function <strong><em>userinit()</em></strong>, kernel will boot up process init.</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">int</span>
<span class="nf">main</span><span class="p">(</span><span class="kt">void</span><span class="p">)</span>
<span class="p">{</span>
  <span class="n">kinit1</span><span class="p">(</span><span class="n">end</span><span class="p">,</span> <span class="n">P2V</span><span class="p">(</span><span class="mi">4</span><span class="o">*</span><span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="p">));</span> <span class="c1">// phys page allocator</span>
  <span class="n">kvmalloc</span><span class="p">();</span>      <span class="c1">// kernel page table</span>
  <span class="n">mpinit</span><span class="p">();</span>        <span class="c1">// collect info about this machine</span>
  <span class="n">lapicinit</span><span class="p">();</span>
  <span class="n">seginit</span><span class="p">();</span>       <span class="c1">// set up segments</span>
  <span class="n">cprintf</span><span class="p">(</span><span class="s">"</span><span class="se">\n</span><span class="s">cpu%d: starting xv6</span><span class="se">\n\n</span><span class="s">"</span><span class="p">,</span> <span class="n">cpu</span><span class="o">-&gt;</span><span class="n">id</span><span class="p">);</span>
  <span class="n">picinit</span><span class="p">();</span>       <span class="c1">// interrupt controller</span>
  <span class="n">ioapicinit</span><span class="p">();</span>    <span class="c1">// another interrupt controller</span>
  <span class="n">consoleinit</span><span class="p">();</span>   <span class="c1">// I/O devices &amp; their interrupts</span>
  <span class="n">uartinit</span><span class="p">();</span>      <span class="c1">// serial port</span>
  <span class="n">pinit</span><span class="p">();</span>         <span class="c1">// process table</span>
  <span class="n">tvinit</span><span class="p">();</span>        <span class="c1">// trap vectors</span>
  <span class="n">binit</span><span class="p">();</span>         <span class="c1">// buffer cache</span>
  <span class="n">fileinit</span><span class="p">();</span>      <span class="c1">// file table</span>
  <span class="n">ideinit</span><span class="p">();</span>       <span class="c1">// disk</span>
  <span class="k">if</span><span class="p">(</span><span class="o">!</span><span class="n">ismp</span><span class="p">)</span>
    <span class="n">timerinit</span><span class="p">();</span>   <span class="c1">// uniprocessor timer</span>
  <span class="n">startothers</span><span class="p">();</span>   <span class="c1">// start other processors</span>
  <span class="n">kinit2</span><span class="p">(</span><span class="n">P2V</span><span class="p">(</span><span class="mi">4</span><span class="o">*</span><span class="mi">1024</span><span class="o">*</span><span class="mi">1024</span><span class="p">),</span> <span class="n">P2V</span><span class="p">(</span><span class="n">PHYSTOP</span><span class="p">));</span> <span class="c1">// must come after startothers()</span>
  <span class="n">userinit</span><span class="p">();</span>      <span class="c1">// first user process</span>
  <span class="c1">// Finish setting up this processor in mpmain.</span>
  <span class="n">mpmain</span><span class="p">();</span>
<span class="p">}</span>
</code></pre></div></div>
<p>It’s tricky since that <strong>init</strong> is a user process, but kernel can’t call any user-level system calls to create it.
Why? 1. Kernel has all privileges to create a user process. So it doesn’t need to call system calls such as <strong><em>fork()</em></strong>. 
And 2. All other user processes can be created by forking from its parent. 
Forking including clone the whole user virtual memory layout. However, the first process has no parent to fork from.
That’s why it makes the creation of the first user process becomes so unique.</p>

<p>In <em>proc.c</em>, <strong><em>userinit()</em></strong> define there gives us the whole procedure of creating <strong>init</strong>.
Similar to the <strong><em>fork()</em></strong>, but more simple.
Process control block(structures for storing the process status) was created at the very first by calling <strong><em>allocproc()</em></strong>.
After then, by invoking <strong><em>setupkvm()</em></strong>(defined in <em>vm.c</em>), kernel memory map was setup for the process.
During setting up kernel memory map, a page size virtual memory will be assigned to the process as ready. 
And later, this page size memory will be used to store instructions of <strong>init</strong>.</p>

<p>Followed by setup kernel stack for the <strong>init</strong> process, calling <strong><em>inituvm()</em></strong> will load <strong>init</strong>’s text into the page that is just being allocated.
<strong><em>inituvm()</em></strong> takes 3 arguments: a pointer to the process’s page directory (p-&gt;pgdir),
a char-type pointer declared from external which point to <strong>init</strong>’s text segment(_binary_initcode_start), and
a char-type pointer which points to an external integer as the size of the <strong>init</strong>’s text segment(_binary_initcode_size).
Simply put, it will load instructions of <strong>init</strong> into the memory.</p>

<p>So now, the problem becomes when and where did instructions for <strong>init</strong> have compiled into the kernel?</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">void</span>
<span class="nf">userinit</span><span class="p">(</span><span class="kt">void</span><span class="p">)</span>
<span class="p">{</span>
  <span class="k">struct</span> <span class="n">proc</span> <span class="o">*</span><span class="n">p</span><span class="p">;</span>
  <span class="k">extern</span> <span class="kt">char</span> <span class="n">_binary_initcode_start</span><span class="p">[],</span> <span class="n">_binary_initcode_size</span><span class="p">[];</span>
  
  <span class="n">p</span> <span class="o">=</span> <span class="n">allocproc</span><span class="p">();</span>
  <span class="n">initproc</span> <span class="o">=</span> <span class="n">p</span><span class="p">;</span>
  <span class="k">if</span><span class="p">((</span><span class="n">p</span><span class="o">-&gt;</span><span class="n">pgdir</span> <span class="o">=</span> <span class="n">setupkvm</span><span class="p">())</span> <span class="o">==</span> <span class="mi">0</span><span class="p">)</span>
    <span class="n">panic</span><span class="p">(</span><span class="s">"userinit: out of memory?"</span><span class="p">);</span>
  <span class="n">inituvm</span><span class="p">(</span><span class="n">p</span><span class="o">-&gt;</span><span class="n">pgdir</span><span class="p">,</span> <span class="n">_binary_initcode_start</span><span class="p">,</span> <span class="p">(</span><span class="kt">int</span><span class="p">)</span><span class="n">_binary_initcode_size</span><span class="p">);</span>
  <span class="n">p</span><span class="o">-&gt;</span><span class="n">sz</span> <span class="o">=</span> <span class="n">PGSIZE</span><span class="p">;</span>
  <span class="n">memset</span><span class="p">(</span><span class="n">p</span><span class="o">-&gt;</span><span class="n">tf</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="o">*</span><span class="n">p</span><span class="o">-&gt;</span><span class="n">tf</span><span class="p">));</span>
  <span class="n">p</span><span class="o">-&gt;</span><span class="n">tf</span><span class="o">-&gt;</span><span class="n">cs</span> <span class="o">=</span> <span class="p">(</span><span class="n">SEG_UCODE</span> <span class="o">&lt;&lt;</span> <span class="mi">3</span><span class="p">)</span> <span class="o">|</span> <span class="n">DPL_USER</span><span class="p">;</span>
  <span class="n">p</span><span class="o">-&gt;</span><span class="n">tf</span><span class="o">-&gt;</span><span class="n">ds</span> <span class="o">=</span> <span class="p">(</span><span class="n">SEG_UDATA</span> <span class="o">&lt;&lt;</span> <span class="mi">3</span><span class="p">)</span> <span class="o">|</span> <span class="n">DPL_USER</span><span class="p">;</span>
  <span class="n">p</span><span class="o">-&gt;</span><span class="n">tf</span><span class="o">-&gt;</span><span class="n">es</span> <span class="o">=</span> <span class="n">p</span><span class="o">-&gt;</span><span class="n">tf</span><span class="o">-&gt;</span><span class="n">ds</span><span class="p">;</span>
  <span class="n">p</span><span class="o">-&gt;</span><span class="n">tf</span><span class="o">-&gt;</span><span class="n">ss</span> <span class="o">=</span> <span class="n">p</span><span class="o">-&gt;</span><span class="n">tf</span><span class="o">-&gt;</span><span class="n">ds</span><span class="p">;</span>
  <span class="n">p</span><span class="o">-&gt;</span><span class="n">tf</span><span class="o">-&gt;</span><span class="n">eflags</span> <span class="o">=</span> <span class="n">FL_IF</span><span class="p">;</span>
  <span class="n">p</span><span class="o">-&gt;</span><span class="n">tf</span><span class="o">-&gt;</span><span class="n">esp</span> <span class="o">=</span> <span class="n">PGSIZE</span><span class="p">;</span>
  <span class="n">p</span><span class="o">-&gt;</span><span class="n">tf</span><span class="o">-&gt;</span><span class="n">eip</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>  <span class="c1">// beginning of initcode.S</span>

  <span class="n">safestrcpy</span><span class="p">(</span><span class="n">p</span><span class="o">-&gt;</span><span class="n">name</span><span class="p">,</span> <span class="s">"initcode"</span><span class="p">,</span> <span class="k">sizeof</span><span class="p">(</span><span class="n">p</span><span class="o">-&gt;</span><span class="n">name</span><span class="p">));</span>
  <span class="n">p</span><span class="o">-&gt;</span><span class="n">cwd</span> <span class="o">=</span> <span class="n">namei</span><span class="p">(</span><span class="s">"/"</span><span class="p">);</span>

  <span class="n">p</span><span class="o">-&gt;</span><span class="n">state</span> <span class="o">=</span> <span class="n">RUNNABLE</span><span class="p">;</span>
<span class="p">}</span>
</code></pre></div></div>
<h4 id="where-the-user-level-code-was-integrated">Where the user-level code was integrated?</h4>
<p>If you search the keyword “_binary_initcode_start” in the source code, you can’t find any references.
The clue comes from the <em>Makefile</em>.</p>

<p>In the makefile, <strong>initcode</strong> is a prerequisites to compile the kernel image.
<strong>Step 1</strong>: Before kernel was compiled, <em>initcode.S</em> was first compiled to a runnable binary <em>initcode</em>.
This binary was very odd because it was not supposed to let any other OS to run it.
<em>Initcode.s</em> was first compiled without any standard including, and generating the intermediate file <em>initcode.o</em>.</p>

<p><strong>Step 2</strong>: <em>Initcode.o</em> then linked to <em>Initcode.out</em> with two uncommon settings. 
First it specify the entry of this binary file as when “start” symbol points to.
This “start” symbol was declared in the assembly code.
Second it specify a absolute address(0) for the text segments. 
By doing this, text segments will be placed at the start of the binary file (except the header of the ELF)<sup id="fnref:ldman" role="doc-noteref"><a href="#fn:ldman" class="footnote" rel="footnote">1</a></sup>.</p>

<p><strong>Step 3</strong>: <em>Initcode.out</em> is already a minimized binary but it’s not enough.
That’s why when using <strong>objcopy</strong> to copy it to the file <em>initcode</em>, it further strip all headers and debug information<sup id="fnref:objcopyman" role="doc-noteref"><a href="#fn:objcopyman" class="footnote" rel="footnote">2</a></sup>.
At this point, we have a minimal binary file <em>initcode</em>.
From the first byte of this file, it’s only includes runnable instructions.
And the size of the file is only 44 bytes.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>initcode: initcode.S
	<span class="si">$(</span>CC<span class="si">)</span> <span class="si">$(</span>CFLAGS<span class="si">)</span> <span class="nt">-nostdinc</span> <span class="nt">-I</span><span class="nb">.</span> <span class="nt">-c</span> initcode.S                         <span class="c"># Step 1</span>
	<span class="si">$(</span>LD<span class="si">)</span> <span class="si">$(</span>LDFLAGS<span class="si">)</span> <span class="nt">-N</span> <span class="nt">-e</span> start <span class="nt">-Ttext</span> 0 <span class="nt">-o</span> initcode.out initcode.o    <span class="c"># Step 2</span>
	<span class="si">$(</span>OBJCOPY<span class="si">)</span> <span class="nt">-S</span> <span class="nt">-O</span> binary initcode.out initcode                       <span class="c"># Step 3</span>
	<span class="si">$(</span>OBJDUMP<span class="si">)</span> <span class="nt">-S</span> initcode.o <span class="o">&gt;</span> initcode.asm
</code></pre></div></div>

<p>This binary later were appended to the kernel using following commands. 
And during this appending, 3 symbols were generated and added to the symbol table of the <em>kernel</em><sup id="fnref:ldman:1" role="doc-noteref"><a href="#fn:ldman" class="footnote" rel="footnote">1</a></sup>.
<strong>“_binary_initcode_start”</strong> contains the address of where the initcode segment was appended to.
<strong>“_binary_initcode_end”</strong> contains the address of where the initcode segment was ended at.
<strong>“_binary_initcode_size”</strong> is a *ABS* type symbol with value 0x2C(45) that specify the size of the initcode segment is 45 bytes.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kernel: <span class="si">$(</span>OBJS<span class="si">)</span> entry.o entryother initcode kernel.ld
	<span class="si">$(</span>LD<span class="si">)</span> <span class="si">$(</span>LDFLAGS<span class="si">)</span> <span class="nt">-T</span> kernel.ld <span class="nt">-o</span> kernel entry.o <span class="si">$(</span>OBJS<span class="si">)</span> <span class="nt">-b</span> binary initcode entryother <span class="c"># &lt;- This Line</span>
	<span class="si">$(</span>OBJDUMP<span class="si">)</span> <span class="nt">-S</span> kernel <span class="o">&gt;</span> kernel.asm
	<span class="si">$(</span>OBJDUMP<span class="si">)</span> <span class="nt">-t</span> kernel | <span class="nb">sed</span> <span class="s1">'1,/SYMBOL TABLE/d; s/ .* / /; /^$$/d'</span> <span class="o">&gt;</span> kernel.sym
</code></pre></div></div>
<p><strong>In short summary</strong>,
using objdump, we can verify that source code <em>initcode.S</em> has been compiled and loaded into the kernel.
Also the segment of initcode’s instructions was located by the pointer “_binary_initcode_start”.
That’s explain when calling <strong><em>inituvm(p-&gt;pgdir, _binary_initcode_start, (int)_binary_initcode_size);</em></strong>, 
functionalities implemented in initcode.S will be loaded into the runtime of the first process within xv6.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Header of the file kernel</span>
kernel:     file format elf32-i386
kernel
architecture: i386, flags 0x00000112:
EXEC_P, HAS_SYMS, D_PAGED
start address 0x0010000c

Program Header:
    LOAD off    0x00001000 vaddr 0x80100000 paddr 0x00100000 align 2<span class="k">**</span>12
         filesz 0x00008c6a memsz 0x00008c6a flags r-x
...
Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .text         00008586  80100000  00100000  00001000  2<span class="k">**</span>2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
...
SYMBOL TABLE:
...
8010b50c g       .data	00000000 _binary_initcode_end
...
8010b4e0 g       .data	00000000 _binary_initcode_start
...
0000002c g       <span class="k">*</span>ABS<span class="k">*</span>	00000000 _binary_initcode_size
...
</code></pre></div></div>

<h4 id="user-level-code">User-level code</h4>

<p>Take a look of content in the <em>initcode.S</em>, you will find the code can explain itself well.
There are no other jobs but just calling system call <strong>exec</strong> to run a user-level binary <strong>“init”</strong>.</p>

<p><em>Initcode.S</em>:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Initial process execs /init.</span>

<span class="c">#include "syscall.h"</span>
<span class="c">#include "traps.h"</span>


<span class="c"># exec(init, argv)</span>
.globl start
start:
  pushl <span class="nv">$argv</span>
  pushl <span class="nv">$init</span>
  pushl <span class="nv">$0</span>  // where <span class="nb">caller </span>pc would be
  movl <span class="nv">$SYS_exec</span>, %eax
  int <span class="nv">$T_SYSCALL</span>

<span class="c"># for(;;) exit();</span>
<span class="nb">exit</span>:
  movl <span class="nv">$SYS_exit</span>, %eax
  int <span class="nv">$T_SYSCALL</span>
  jmp <span class="nb">exit</span>

<span class="c"># char init[] = "/init\0";</span>
init:
  .string <span class="s2">"/init</span><span class="se">\0</span><span class="s2">"</span>

<span class="c"># char *argv[] = { init, 0 };</span>
.p2align 2
argv:
  .long init
  .long 0
</code></pre></div></div>

<p>The <strong>“init”</strong> mentioned above is not a pure user-level binary executable that compiled from the source code <em>init.c</em>.
Within <em>init.c</em>, a file named <em>console</em> will be created at the runtime for saving standard outputs and errors.
Then it will forked a child process(the second user process), and let it run program <strong>“sh”</strong>.</p>

<p><strong>“sh”</strong> is the xv6’s default shell, a user-level program that generated from source <em>sh.c</em>.
After the shell boots up, you can interactive with the xv6. 
This’s how first process (and second process) was started in the xv6.</p>

<p><em>init.c</em>:</p>
<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// init: The initial user-level program</span>

<span class="cp">#include</span> <span class="cpf">"types.h"</span><span class="cp">
#include</span> <span class="cpf">"stat.h"</span><span class="cp">
#include</span> <span class="cpf">"user.h"</span><span class="cp">
#include</span> <span class="cpf">"fcntl.h"</span><span class="cp">
</span>
<span class="kt">char</span> <span class="o">*</span><span class="n">argv</span><span class="p">[]</span> <span class="o">=</span> <span class="p">{</span> <span class="s">"sh"</span><span class="p">,</span> <span class="mi">0</span> <span class="p">};</span>

<span class="kt">int</span>
<span class="nf">main</span><span class="p">(</span><span class="kt">void</span><span class="p">)</span>
<span class="p">{</span>
  <span class="kt">int</span> <span class="n">pid</span><span class="p">,</span> <span class="n">wpid</span><span class="p">;</span>

  <span class="k">if</span><span class="p">(</span><span class="n">open</span><span class="p">(</span><span class="s">"console"</span><span class="p">,</span> <span class="n">O_RDWR</span><span class="p">)</span> <span class="o">&lt;</span> <span class="mi">0</span><span class="p">){</span>
    <span class="n">mknod</span><span class="p">(</span><span class="s">"console"</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">1</span><span class="p">);</span>
    <span class="n">open</span><span class="p">(</span><span class="s">"console"</span><span class="p">,</span> <span class="n">O_RDWR</span><span class="p">);</span>
  <span class="p">}</span>
  <span class="n">dup</span><span class="p">(</span><span class="mi">0</span><span class="p">);</span>  <span class="c1">// stdout</span>
  <span class="n">dup</span><span class="p">(</span><span class="mi">0</span><span class="p">);</span>  <span class="c1">// stderr</span>

  <span class="k">for</span><span class="p">(;;){</span>
    <span class="n">printf</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s">"init: starting sh</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
    <span class="n">pid</span> <span class="o">=</span> <span class="n">fork</span><span class="p">();</span>
    <span class="k">if</span><span class="p">(</span><span class="n">pid</span> <span class="o">&lt;</span> <span class="mi">0</span><span class="p">){</span>
      <span class="n">printf</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s">"init: fork failed</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
      <span class="n">exit</span><span class="p">();</span>
    <span class="p">}</span>
    <span class="k">if</span><span class="p">(</span><span class="n">pid</span> <span class="o">==</span> <span class="mi">0</span><span class="p">){</span>
      <span class="n">exec</span><span class="p">(</span><span class="s">"sh"</span><span class="p">,</span> <span class="n">argv</span><span class="p">);</span>
      <span class="n">printf</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s">"init: exec sh failed</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
      <span class="n">exit</span><span class="p">();</span>
    <span class="p">}</span>
    <span class="k">while</span><span class="p">((</span><span class="n">wpid</span><span class="o">=</span><span class="n">wait</span><span class="p">())</span> <span class="o">&gt;=</span> <span class="mi">0</span> <span class="o">&amp;&amp;</span> <span class="n">wpid</span> <span class="o">!=</span> <span class="n">pid</span><span class="p">)</span>
      <span class="n">printf</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="s">"zombie!</span><span class="se">\n</span><span class="s">"</span><span class="p">);</span>
  <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<h3 id="xv6s-round-robin-schduler">Xv6’s round robin schduler</h3>

<p>The Scheduler is the core of an operating system. 
With the scheduling of processes, the kernel can achieve near-real-time execution of multiple workloads.
The scheduling problem is also an active aspect of computer science research. 
You can’t have one algorithm to fit all scenarios.</p>

<p>Xv6 by default has a round-robin scheduler. 
It’s controlled using two-level for-loops, where the top-level for-loop is an endless loop that will keep the scheduler busy running. 
The second-level nested for-loop will iterate a data structure named Ptable where all control information for processes is stored. 
Information including pid, process name, etc. is stored in a structure called proc. Ptable is an array of processes. 
Every runnable process in the Ptable will run strictly 1 time tick until the for-loop reached the last process in the Ptable. 
Then it will loop back to the top-level for-loop for the next iteration of processes.</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// In file proc.c</span>
<span class="k">struct</span> <span class="p">{</span>
  <span class="k">struct</span> <span class="n">spinlock</span> <span class="n">lock</span><span class="p">;</span>
  <span class="k">struct</span> <span class="n">proc</span> <span class="n">proc</span><span class="p">[</span><span class="n">NPROC</span><span class="p">];</span>
<span class="p">}</span> <span class="n">ptable</span><span class="p">;</span>

<span class="c1">// In file proc.h</span>
<span class="k">struct</span> <span class="n">proc</span> <span class="p">{</span>
  <span class="n">uint</span> <span class="n">sz</span><span class="p">;</span>                     <span class="c1">// Size of process memory (bytes)</span>
  <span class="n">pde_t</span><span class="o">*</span> <span class="n">pgdir</span><span class="p">;</span>                <span class="c1">// Page table</span>
  <span class="kt">char</span> <span class="o">*</span><span class="n">kstack</span><span class="p">;</span>                <span class="c1">// Bottom of kernel stack for this process</span>
  <span class="k">enum</span> <span class="n">procstate</span> <span class="n">state</span><span class="p">;</span>        <span class="c1">// Process state</span>
  <span class="kt">int</span> <span class="n">pid</span><span class="p">;</span>                     <span class="c1">// Process ID</span>
  <span class="k">struct</span> <span class="n">proc</span> <span class="o">*</span><span class="n">parent</span><span class="p">;</span>         <span class="c1">// Parent process</span>
  <span class="k">struct</span> <span class="n">trapframe</span> <span class="o">*</span><span class="n">tf</span><span class="p">;</span>        <span class="c1">// Trap frame for current syscall</span>
  <span class="k">struct</span> <span class="n">context</span> <span class="o">*</span><span class="n">context</span><span class="p">;</span>     <span class="c1">// swtch() here to run process</span>
  <span class="kt">void</span> <span class="o">*</span><span class="n">chan</span><span class="p">;</span>                  <span class="c1">// If non-zero, sleeping on chan</span>
  <span class="kt">int</span> <span class="n">killed</span><span class="p">;</span>                  <span class="c1">// If non-zero, have been killed</span>
  <span class="k">struct</span> <span class="n">file</span> <span class="o">*</span><span class="n">ofile</span><span class="p">[</span><span class="n">NOFILE</span><span class="p">];</span>  <span class="c1">// Open files</span>
  <span class="k">struct</span> <span class="n">inode</span> <span class="o">*</span><span class="n">cwd</span><span class="p">;</span>           <span class="c1">// Current directory</span>
  <span class="kt">char</span> <span class="n">name</span><span class="p">[</span><span class="mi">16</span><span class="p">];</span>               <span class="c1">// Process name (debugging)</span>
<span class="p">};</span>
</code></pre></div></div>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// In file proc.c</span>
<span class="kt">void</span>
<span class="nf">scheduler</span><span class="p">(</span><span class="kt">void</span><span class="p">)</span>
<span class="p">{</span>
  <span class="k">struct</span> <span class="n">proc</span> <span class="o">*</span><span class="n">p</span><span class="p">;</span>

  <span class="k">for</span><span class="p">(;;){</span>
    <span class="c1">// Enable interrupts on this processor.</span>
    <span class="n">sti</span><span class="p">();</span>

    <span class="c1">// Loop over process table looking for process to run.</span>
    <span class="n">acquire</span><span class="p">(</span><span class="o">&amp;</span><span class="n">ptable</span><span class="p">.</span><span class="n">lock</span><span class="p">);</span>
    <span class="k">for</span><span class="p">(</span><span class="n">p</span> <span class="o">=</span> <span class="n">ptable</span><span class="p">.</span><span class="n">proc</span><span class="p">;</span> <span class="n">p</span> <span class="o">&lt;</span> <span class="o">&amp;</span><span class="n">ptable</span><span class="p">.</span><span class="n">proc</span><span class="p">[</span><span class="n">NPROC</span><span class="p">];</span> <span class="n">p</span><span class="o">++</span><span class="p">){</span>
      <span class="k">if</span><span class="p">(</span><span class="n">p</span><span class="o">-&gt;</span><span class="n">state</span> <span class="o">!=</span> <span class="n">RUNNABLE</span><span class="p">)</span>
        <span class="k">continue</span><span class="p">;</span>

      <span class="c1">// Switch to chosen process.  It is the process's job</span>
      <span class="c1">// to release ptable.lock and then reacquire it</span>
      <span class="c1">// before jumping back to us.</span>
      <span class="n">proc</span> <span class="o">=</span> <span class="n">p</span><span class="p">;</span>
      <span class="n">switchuvm</span><span class="p">(</span><span class="n">p</span><span class="p">);</span>
      <span class="n">p</span><span class="o">-&gt;</span><span class="n">state</span> <span class="o">=</span> <span class="n">RUNNING</span><span class="p">;</span>
      <span class="n">swtch</span><span class="p">(</span><span class="o">&amp;</span><span class="n">cpu</span><span class="o">-&gt;</span><span class="n">scheduler</span><span class="p">,</span> <span class="n">proc</span><span class="o">-&gt;</span><span class="n">context</span><span class="p">);</span>
      <span class="n">switchkvm</span><span class="p">();</span>

      <span class="c1">// Process is done running for now.</span>
      <span class="c1">// It should have changed its p-&gt;state before coming back.</span>
      <span class="n">proc</span> <span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
    <span class="p">}</span>
    <span class="n">release</span><span class="p">(</span><span class="o">&amp;</span><span class="n">ptable</span><span class="p">.</span><span class="n">lock</span><span class="p">);</span>

  <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<p>It’s not hard to understand why this logic makes a round-robin manner. 
This is very important to understand how to pick a process to run because scheduling is about always picking the appropriate process to achieve higher performance.</p>

<p>You can always come up with some new ideas for designing a good scheduler policy. 
Understanding how to switch from one process to another is equivalently important.</p>

<p>Once the process for the next time tick is selected. 
It’s time to switch from the running scheduler to the selected process. Wait for a second, there are two questions we haven’t answered.</p>
<ol>
  <li>What is the running scheduler?</li>
  <li>How did the last running process stop running and give the CPU back to the scheduler?</li>
</ol>

<h3 id="lab">Lab</h3>

<div class="footnotes" role="doc-endnotes">
  <ol>
    <li id="fn:ldman" role="doc-endnote">
      <p><a href="https://linux.die.net/man/1/ld">ld(1) - Linux man page</a> <a href="#fnref:ldman" class="reversefootnote" role="doc-backlink">&#8617;</a> <a href="#fnref:ldman:1" class="reversefootnote" role="doc-backlink">&#8617;<sup>2</sup></a></p>
    </li>
    <li id="fn:objcopyman" role="doc-endnote">
      <p><a href="https://sourceware.org/binutils/docs/binutils/objcopy.html">3 objcopy - binutils mannual</a> <a href="#fnref:objcopyman" class="reversefootnote" role="doc-backlink">&#8617;</a></p>
    </li>
  </ol>
</div>]]></content><author><name>Pengzhan Hao</name></author><category term="Xv6" /><category term="Teaching" /><category term="Operating system" /><category term="Binghamton university" /><summary type="html"><![CDATA[This will be a series regarding lab I gave during the spring 2022 semester. The reason why I am writing this down is because it has been a week and no students ask for the solution of the last Lab. I realise that learning gap between students are huge, especially when a non-profit university is admitting more and more students. To help all students in understanding concepts of modern OS, I decided to write this post. It starts with the past lab content I have (as the skelton), and will be amended with extra materials I think it helps. Remember, it’s for helping in learning. DON’T COPY &amp; PASTE CODE! Index Lab1: Introduction of Makefile and Xv6. Lab3: System calls for process management. Lab4: Inter-processes communication. Lab6/7: CPU scheduling. Lab1-Introduction Lab3-Process Lab4-IPC Lab6-7-Scheduling First user process in xv6 Kernel works In xv6, as the same as conventional linux OS, the very first user-level process is init. Before init’s running, all the OS bootstraps happen in a highly privileged mode(kernel level). Xv6’s kernel has the entry point as the main function located in the file main.c. The main function invokes 17 functions to set up kernel page tables, interrupt handlers, I/O devices and etc. When all kernel preparations are done, by calling the function userinit(), kernel will boot up process init. int main(void) { kinit1(end, P2V(4*1024*1024)); // phys page allocator kvmalloc(); // kernel page table mpinit(); // collect info about this machine lapicinit(); seginit(); // set up segments cprintf("\ncpu%d: starting xv6\n\n", cpu-&gt;id); picinit(); // interrupt controller ioapicinit(); // another interrupt controller consoleinit(); // I/O devices &amp; their interrupts uartinit(); // serial port pinit(); // process table tvinit(); // trap vectors binit(); // buffer cache fileinit(); // file table ideinit(); // disk if(!ismp) timerinit(); // uniprocessor timer startothers(); // start other processors kinit2(P2V(4*1024*1024), P2V(PHYSTOP)); // must come after startothers() userinit(); // first user process // Finish setting up this processor in mpmain. mpmain(); } It’s tricky since that init is a user process, but kernel can’t call any user-level system calls to create it. Why? 1. Kernel has all privileges to create a user process. So it doesn’t need to call system calls such as fork(). And 2. All other user processes can be created by forking from its parent. Forking including clone the whole user virtual memory layout. However, the first process has no parent to fork from. That’s why it makes the creation of the first user process becomes so unique. In proc.c, userinit() define there gives us the whole procedure of creating init. Similar to the fork(), but more simple. Process control block(structures for storing the process status) was created at the very first by calling allocproc(). After then, by invoking setupkvm()(defined in vm.c), kernel memory map was setup for the process. During setting up kernel memory map, a page size virtual memory will be assigned to the process as ready. And later, this page size memory will be used to store instructions of init. Followed by setup kernel stack for the init process, calling inituvm() will load init’s text into the page that is just being allocated. inituvm() takes 3 arguments: a pointer to the process’s page directory (p-&gt;pgdir), a char-type pointer declared from external which point to init’s text segment(_binary_initcode_start), and a char-type pointer which points to an external integer as the size of the init’s text segment(_binary_initcode_size). Simply put, it will load instructions of init into the memory. So now, the problem becomes when and where did instructions for init have compiled into the kernel? void userinit(void) { struct proc *p; extern char _binary_initcode_start[], _binary_initcode_size[]; p = allocproc(); initproc = p; if((p-&gt;pgdir = setupkvm()) == 0) panic("userinit: out of memory?"); inituvm(p-&gt;pgdir, _binary_initcode_start, (int)_binary_initcode_size); p-&gt;sz = PGSIZE; memset(p-&gt;tf, 0, sizeof(*p-&gt;tf)); p-&gt;tf-&gt;cs = (SEG_UCODE &lt;&lt; 3) | DPL_USER; p-&gt;tf-&gt;ds = (SEG_UDATA &lt;&lt; 3) | DPL_USER; p-&gt;tf-&gt;es = p-&gt;tf-&gt;ds; p-&gt;tf-&gt;ss = p-&gt;tf-&gt;ds; p-&gt;tf-&gt;eflags = FL_IF; p-&gt;tf-&gt;esp = PGSIZE; p-&gt;tf-&gt;eip = 0; // beginning of initcode.S safestrcpy(p-&gt;name, "initcode", sizeof(p-&gt;name)); p-&gt;cwd = namei("/"); p-&gt;state = RUNNABLE; } Where the user-level code was integrated? If you search the keyword “_binary_initcode_start” in the source code, you can’t find any references. The clue comes from the Makefile. In the makefile, initcode is a prerequisites to compile the kernel image. Step 1: Before kernel was compiled, initcode.S was first compiled to a runnable binary initcode. This binary was very odd because it was not supposed to let any other OS to run it. Initcode.s was first compiled without any standard including, and generating the intermediate file initcode.o. Step 2: Initcode.o then linked to Initcode.out with two uncommon settings. First it specify the entry of this binary file as when “start” symbol points to. This “start” symbol was declared in the assembly code. Second it specify a absolute address(0) for the text segments. By doing this, text segments will be placed at the start of the binary file (except the header of the ELF)1. Step 3: Initcode.out is already a minimized binary but it’s not enough. That’s why when using objcopy to copy it to the file initcode, it further strip all headers and debug information2. At this point, we have a minimal binary file initcode. From the first byte of this file, it’s only includes runnable instructions. And the size of the file is only 44 bytes. initcode: initcode.S $(CC) $(CFLAGS) -nostdinc -I. -c initcode.S # Step 1 $(LD) $(LDFLAGS) -N -e start -Ttext 0 -o initcode.out initcode.o # Step 2 $(OBJCOPY) -S -O binary initcode.out initcode # Step 3 $(OBJDUMP) -S initcode.o &gt; initcode.asm This binary later were appended to the kernel using following commands. And during this appending, 3 symbols were generated and added to the symbol table of the kernel1. “_binary_initcode_start” contains the address of where the initcode segment was appended to. “_binary_initcode_end” contains the address of where the initcode segment was ended at. “_binary_initcode_size” is a *ABS* type symbol with value 0x2C(45) that specify the size of the initcode segment is 45 bytes. kernel: $(OBJS) entry.o entryother initcode kernel.ld $(LD) $(LDFLAGS) -T kernel.ld -o kernel entry.o $(OBJS) -b binary initcode entryother # &lt;- This Line $(OBJDUMP) -S kernel &gt; kernel.asm $(OBJDUMP) -t kernel | sed '1,/SYMBOL TABLE/d; s/ .* / /; /^$$/d' &gt; kernel.sym In short summary, using objdump, we can verify that source code initcode.S has been compiled and loaded into the kernel. Also the segment of initcode’s instructions was located by the pointer “_binary_initcode_start”. That’s explain when calling inituvm(p-&gt;pgdir, _binary_initcode_start, (int)_binary_initcode_size);, functionalities implemented in initcode.S will be loaded into the runtime of the first process within xv6. # Header of the file kernel kernel: file format elf32-i386 kernel architecture: i386, flags 0x00000112: EXEC_P, HAS_SYMS, D_PAGED start address 0x0010000c Program Header: LOAD off 0x00001000 vaddr 0x80100000 paddr 0x00100000 align 2**12 filesz 0x00008c6a memsz 0x00008c6a flags r-x ... Sections: Idx Name Size VMA LMA File off Algn 0 .text 00008586 80100000 00100000 00001000 2**2 CONTENTS, ALLOC, LOAD, READONLY, CODE ... SYMBOL TABLE: ... 8010b50c g .data 00000000 _binary_initcode_end ... 8010b4e0 g .data 00000000 _binary_initcode_start ... 0000002c g *ABS* 00000000 _binary_initcode_size ... User-level code Take a look of content in the initcode.S, you will find the code can explain itself well. There are no other jobs but just calling system call exec to run a user-level binary “init”. Initcode.S: # Initial process execs /init. #include "syscall.h" #include "traps.h" # exec(init, argv) .globl start start: pushl $argv pushl $init pushl $0 // where caller pc would be movl $SYS_exec, %eax int $T_SYSCALL # for(;;) exit(); exit: movl $SYS_exit, %eax int $T_SYSCALL jmp exit # char init[] = "/init\0"; init: .string "/init\0" # char *argv[] = { init, 0 }; .p2align 2 argv: .long init .long 0 The “init” mentioned above is not a pure user-level binary executable that compiled from the source code init.c. Within init.c, a file named console will be created at the runtime for saving standard outputs and errors. Then it will forked a child process(the second user process), and let it run program “sh”. “sh” is the xv6’s default shell, a user-level program that generated from source sh.c. After the shell boots up, you can interactive with the xv6. This’s how first process (and second process) was started in the xv6. init.c: // init: The initial user-level program #include "types.h" #include "stat.h" #include "user.h" #include "fcntl.h" char *argv[] = { "sh", 0 }; int main(void) { int pid, wpid; if(open("console", O_RDWR) &lt; 0){ mknod("console", 1, 1); open("console", O_RDWR); } dup(0); // stdout dup(0); // stderr for(;;){ printf(1, "init: starting sh\n"); pid = fork(); if(pid &lt; 0){ printf(1, "init: fork failed\n"); exit(); } if(pid == 0){ exec("sh", argv); printf(1, "init: exec sh failed\n"); exit(); } while((wpid=wait()) &gt;= 0 &amp;&amp; wpid != pid) printf(1, "zombie!\n"); } } Xv6’s round robin schduler The Scheduler is the core of an operating system. With the scheduling of processes, the kernel can achieve near-real-time execution of multiple workloads. The scheduling problem is also an active aspect of computer science research. You can’t have one algorithm to fit all scenarios. Xv6 by default has a round-robin scheduler. It’s controlled using two-level for-loops, where the top-level for-loop is an endless loop that will keep the scheduler busy running. The second-level nested for-loop will iterate a data structure named Ptable where all control information for processes is stored. Information including pid, process name, etc. is stored in a structure called proc. Ptable is an array of processes. Every runnable process in the Ptable will run strictly 1 time tick until the for-loop reached the last process in the Ptable. Then it will loop back to the top-level for-loop for the next iteration of processes. // In file proc.c struct { struct spinlock lock; struct proc proc[NPROC]; } ptable; // In file proc.h struct proc { uint sz; // Size of process memory (bytes) pde_t* pgdir; // Page table char *kstack; // Bottom of kernel stack for this process enum procstate state; // Process state int pid; // Process ID struct proc *parent; // Parent process struct trapframe *tf; // Trap frame for current syscall struct context *context; // swtch() here to run process void *chan; // If non-zero, sleeping on chan int killed; // If non-zero, have been killed struct file *ofile[NOFILE]; // Open files struct inode *cwd; // Current directory char name[16]; // Process name (debugging) }; // In file proc.c void scheduler(void) { struct proc *p; for(;;){ // Enable interrupts on this processor. sti(); // Loop over process table looking for process to run. acquire(&amp;ptable.lock); for(p = ptable.proc; p &lt; &amp;ptable.proc[NPROC]; p++){ if(p-&gt;state != RUNNABLE) continue; // Switch to chosen process. It is the process's job // to release ptable.lock and then reacquire it // before jumping back to us. proc = p; switchuvm(p); p-&gt;state = RUNNING; swtch(&amp;cpu-&gt;scheduler, proc-&gt;context); switchkvm(); // Process is done running for now. // It should have changed its p-&gt;state before coming back. proc = 0; } release(&amp;ptable.lock); } } It’s not hard to understand why this logic makes a round-robin manner. This is very important to understand how to pick a process to run because scheduling is about always picking the appropriate process to achieve higher performance. You can always come up with some new ideas for designing a good scheduler policy. Understanding how to switch from one process to another is equivalently important. Once the process for the next time tick is selected. It’s time to switch from the running scheduler to the selected process. Wait for a second, there are two questions we haven’t answered. What is the running scheduler? How did the last running process stop running and give the CPU back to the scheduler? Lab ld(1) - Linux man page &#8617; &#8617;2 3 objcopy - binutils mannual &#8617;]]></summary></entry><entry><title type="html">EDDL: How do we train neural networks on limited edge devices - PART 2</title><link href="https://blog.pengzhan.dev/posts/eddl-how-do-we-train-on-limited-edge-devices-part2" rel="alternate" type="text/html" title="EDDL: How do we train neural networks on limited edge devices - PART 2" /><published>2021-10-31T13:01:14-04:00</published><updated>2021-12-11T13:35:39-05:00</updated><id>https://blog.pengzhan.dev/posts/eddl-how-do-we-train-on-limited-edge-devices-part2</id><content type="html" xml:base="https://blog.pengzhan.dev/posts/eddl-how-do-we-train-on-limited-edge-devices-part2"><![CDATA[<p>In the last post, part1, our idea of distributed learning on edge environment was generally addressed.
I introduced the reason why edge distributed learning is needed and what improvements it can achieve.
In this post, I will talk about our motivation study and how our framework works.</p>

<h2 id="how-does-data-support-us-training-on-edge">How does data support us training on edge?</h2>

<p>Before designing and implementing our framework, we first need confirmation that training on edge resource-limited devices is worthwhile.
We were using a malware detection neural network to show why a small, customized neural network is better.</p>

<p>We collected 32000+ mobile apps feature as global data.
With these data records, we trained a multilayer perceptron called “PerNet” to determine whether a given feature belongs to a benign or malware app.
We called this <strong>detection</strong>.
As well, PerNet can also classify malware apps into different types of attacks.
We called this <strong>classification</strong>.
The global model can achieve 93% above recall rate and 96.93% above accuracy.</p>

<p>With all these data, we selected two community app usage sub-dataset for local model generations.</p>

<ul>
  <li>
    <p>Large categories (Scenario 1)
  We chose the 5 largest categories of apps, including entertainment, tools, brain&amp;Puzzle, Lifestyle, and Education, as well as the 5 largest malware categories.
  All together, 12000+ apps were included in this sub-dataset, almost 50 to 50 between benign and malware.</p>
  </li>
  <li>
    <p>Campus-community categories (Scenario 2)
  We chose the 5 most downloaded categories from college students as benign groups, as well as a similar amount of 5 malware categories.
  To ensure that malware apps are included in 5 benign categories, we also considered synthesizing some other malware apps within categories of 5 most downloaded(benign) categories.</p>
  </li>
</ul>

<p>With these two types of sub-dataset, we used the same PerNet to generate multiple local models.
Under each scenarios experiment, we compared global and local models on the preserved test dataset.
In all classification performances, local beat global in every scenario.
In detection performances, local also share the same accuracy as global does.</p>

<p><img src="/static/2021-10/t.3_inference_result.png" alt="Inference results" /></p>

<p>In summary, local models were trained on special occasions.
Under the same circumstance, a global model can achieve no better accuracy than local models.
The reason why local is better might be because of overfitting.
I believe this issue also be considered in the machine learning communities that they brought <a href="https://en.wikipedia.org/wiki/Transfer_learning">transfer learning</a>,
a technique to optimize global models to special scenarios but performing more training to a global model once it’s shipped to local.</p>

<h2 id="design-and-implementation">Design and Implementation</h2>

<h3 id="overall-design">Overall design</h3>

<p>The basic EDDL distributed training setup consists of 3 parts.
<strong>EDDL training cluster</strong>, a device cluster that consists of edge or mobile devices that are participating in training.
<strong>EDDL manager</strong>, the initial driver program that works as collect training data, relay data to training devices and initial training clusters.
<strong>Training data entry (TDE)</strong>, a data storage for all training data.</p>

<h3 id="dynamic-training-data-distribution">Dynamic training data distribution</h3>

<p>Existing distributed DNN training solutions usually statically partition training data among workers.
It can be a problem when the training node joins and exits.
We designed our framework that can dynamically distribute training data during learning.
Before every training batch started, a batch of TDE will be sent to devices.</p>

<p>In our experiments, we found that by applying this design, overall training time was shortened by doing.
Especially in large amount devices cases, this optimization can be 50% less than statically divided.</p>

<h3 id="scaling-up-cluster-size">Scaling up cluster size</h3>

<p>Our framework was designed to have both sync and async parameter aggregation.
Asynchronous aggregation can allow a high outcome of training batch but with a sacrifice or converge time.
Synchronous aggregation allows a quick converge time in epochs, however can’t ensure performance when there’s a struggler worker.</p>

<p>As showed in experiments, we chose sync as default because the converging time is dominant in overall training time.
But, we also considered the possibilities of that async with more workers can achieve similar overall training time.</p>

<p>We introduced a formula to determine whether adding more training nodes can help or not.
Here we used bandwidth usage coefficient (BUC) as</p>

\[BUC = \dfrac{n}{T_{sync}}\]

<p>In this formula, \(n\) is the number of devices, and \(T_{sync}\) is the transmission time of parameters.
With an increasing number of workers, n increase linearly but transmission time does not.
When \(BUC\) increases, the cluster can speed up training time by adding workers.
Otherwise, adding more workers won’t help with overall training time.</p>

<h3 id="adaptive-leader-role-splitting">Adaptive leader role splitting</h3>

<p>The idea of role splitting is simple that a device can work as a worker as well leader.
The advantage of doing this is straightforward that we can transfer 1 less parameter and training time will be shortened.</p>

<p>However, in our current settings, it can’t perform much better help since only 1 leader role is in a cluster.
We can benefit from this in our future works.</p>

<h3 id="overall-architecture">Overall architecture</h3>

<p><img src="/static/2021-10/f.5_Impl_leader_worker.png" alt="Implementation" /></p>

<p>Details were given in the image.</p>

<h3 id="prototype-hardware-and-software">Prototype hardware and software</h3>

<p>EDDL was designed to be run on two single-board computer embedded platforms.
One such platform is <a href="https://www.hardkernel.com/shop/odroid-xu4-special-price/">ODROID-XU4</a>, which is equipped with a 2.1/1.4 GHz 32-bit ARM processor and 2GB memory.
The other platform is the <a href="https://www.raspberrypi.com/products/raspberry-pi-3-model-b/">Raspberry Pi 3 Model B board</a>, which comes with an ARM 1.2 GHz 64-bit quad-core processor and 1GB memory.</p>

<p>The operating system running on the above platforms is Ubuntu 18.04 with Linux kernel 4.14.
We used <a href="http://dlib.net/">Dlib</a>, a C++ library that provides implementations for a wide range of machine learning algorithms.
We chose the Dlib library because it is written in C/C++, and can be easily and natively used in embedded devices.</p>]]></content><author><name>Pengzhan Hao</name></author><category term="Research" /><category term="Edge computing" /><summary type="html"><![CDATA[In the last post, part1, our idea of distributed learning on edge environment was generally addressed. I introduced the reason why edge distributed learning is needed and what improvements it can achieve. In this post, I will talk about our motivation study and how our framework works. How does data support us training on edge? Before designing and implementing our framework, we first need confirmation that training on edge resource-limited devices is worthwhile. We were using a malware detection neural network to show why a small, customized neural network is better. We collected 32000+ mobile apps feature as global data. With these data records, we trained a multilayer perceptron called “PerNet” to determine whether a given feature belongs to a benign or malware app. We called this detection. As well, PerNet can also classify malware apps into different types of attacks. We called this classification. The global model can achieve 93% above recall rate and 96.93% above accuracy. With all these data, we selected two community app usage sub-dataset for local model generations. Large categories (Scenario 1) We chose the 5 largest categories of apps, including entertainment, tools, brain&amp;Puzzle, Lifestyle, and Education, as well as the 5 largest malware categories. All together, 12000+ apps were included in this sub-dataset, almost 50 to 50 between benign and malware. Campus-community categories (Scenario 2) We chose the 5 most downloaded categories from college students as benign groups, as well as a similar amount of 5 malware categories. To ensure that malware apps are included in 5 benign categories, we also considered synthesizing some other malware apps within categories of 5 most downloaded(benign) categories. With these two types of sub-dataset, we used the same PerNet to generate multiple local models. Under each scenarios experiment, we compared global and local models on the preserved test dataset. In all classification performances, local beat global in every scenario. In detection performances, local also share the same accuracy as global does. In summary, local models were trained on special occasions. Under the same circumstance, a global model can achieve no better accuracy than local models. The reason why local is better might be because of overfitting. I believe this issue also be considered in the machine learning communities that they brought transfer learning, a technique to optimize global models to special scenarios but performing more training to a global model once it’s shipped to local. Design and Implementation Overall design The basic EDDL distributed training setup consists of 3 parts. EDDL training cluster, a device cluster that consists of edge or mobile devices that are participating in training. EDDL manager, the initial driver program that works as collect training data, relay data to training devices and initial training clusters. Training data entry (TDE), a data storage for all training data. Dynamic training data distribution Existing distributed DNN training solutions usually statically partition training data among workers. It can be a problem when the training node joins and exits. We designed our framework that can dynamically distribute training data during learning. Before every training batch started, a batch of TDE will be sent to devices. In our experiments, we found that by applying this design, overall training time was shortened by doing. Especially in large amount devices cases, this optimization can be 50% less than statically divided. Scaling up cluster size Our framework was designed to have both sync and async parameter aggregation. Asynchronous aggregation can allow a high outcome of training batch but with a sacrifice or converge time. Synchronous aggregation allows a quick converge time in epochs, however can’t ensure performance when there’s a struggler worker. As showed in experiments, we chose sync as default because the converging time is dominant in overall training time. But, we also considered the possibilities of that async with more workers can achieve similar overall training time. We introduced a formula to determine whether adding more training nodes can help or not. Here we used bandwidth usage coefficient (BUC) as \[BUC = \dfrac{n}{T_{sync}}\] In this formula, \(n\) is the number of devices, and \(T_{sync}\) is the transmission time of parameters. With an increasing number of workers, n increase linearly but transmission time does not. When \(BUC\) increases, the cluster can speed up training time by adding workers. Otherwise, adding more workers won’t help with overall training time. Adaptive leader role splitting The idea of role splitting is simple that a device can work as a worker as well leader. The advantage of doing this is straightforward that we can transfer 1 less parameter and training time will be shortened. However, in our current settings, it can’t perform much better help since only 1 leader role is in a cluster. We can benefit from this in our future works. Overall architecture Details were given in the image. Prototype hardware and software EDDL was designed to be run on two single-board computer embedded platforms. One such platform is ODROID-XU4, which is equipped with a 2.1/1.4 GHz 32-bit ARM processor and 2GB memory. The other platform is the Raspberry Pi 3 Model B board, which comes with an ARM 1.2 GHz 64-bit quad-core processor and 1GB memory. The operating system running on the above platforms is Ubuntu 18.04 with Linux kernel 4.14. We used Dlib, a C++ library that provides implementations for a wide range of machine learning algorithms. We chose the Dlib library because it is written in C/C++, and can be easily and natively used in embedded devices.]]></summary></entry><entry><title type="html">EDDL: How do we train neural networks on limited edge devices - PART 1</title><link href="https://blog.pengzhan.dev/posts/eddl-how-do-we-train-on-limited-edge-devices" rel="alternate" type="text/html" title="EDDL: How do we train neural networks on limited edge devices - PART 1" /><published>2021-10-13T16:53:20-04:00</published><updated>2021-12-11T13:35:39-05:00</updated><id>https://blog.pengzhan.dev/posts/eddl-how-do-we-train-on-limited-edge-devices</id><content type="html" xml:base="https://blog.pengzhan.dev/posts/eddl-how-do-we-train-on-limited-edge-devices"><![CDATA[<p>This post introduces our previous milestone in project “Edge trainer”, as the paper “EDDL: A Distributed Deep Learning System for Resource-limited Edge Computing Environment.” was published.
As the first part of the introductions, I focus only on the motivation and summary of our works.
More details in design and implementation can be found in late posts.</p>

<p><img src="/static/2021-10/edgelearn-1.png" height="250" />
<!--more--></p>

<h2 id="why-do-we-need-training-on-edge">Why do we need training on edge?</h2>

<p>Cloud is not trustworthy anymore. More and more facts support that breach on the cloud happens frequently than before.
Nowadays, with more generated personal sensitive data has been uploaded to the cloud center, tech companies know better to someones than the user.</p>

<p>Researchers, no matter in the industry on academia, are working in a way that still learning from users’ data but also keeping raw sensitive data under users’ control.
Many publications have already shown the feasibility of only sharing the after-trained model instead of raw data.
One recent popular study on this is google’s <a href="https://ai.googleblog.com/2017/04/federated-learning-collaborative.html">federated learning</a>.</p>

<p>During investigating this problem, we found that letting end-user train their data is safe, but sacrifice efficiency.
Since one end device has limited resources, training time and power consumption can be disappointing.
We believe there must have leverage between privacy and efficiency in some target scenarios.</p>

<p>Fortunately, we observed that users who belong to the same campus, plant, firm, and community always share similar interests.
Therefore, these co-located users have similar demands in using AI-involved routines.
Also, co-located users are easily targeted by the same type of threats, such as ransomware to financial practitioners.</p>

<p>Think about this, sending features of a new malware app to cloud services to train neural networks used by antivirus programs.
This process may take a long time and a small number of samples may not be recognized by the global neural networks model.
A customized local model trained and deployed on the edge can successfully counter the problem.
With edge training as a supplement to the cloud training can achieve better response time and let the whole system more flexible.</p>

<h2 id="why-training-on-edge-is-hard">Why training on edge is hard?</h2>

<p>Since all co-located users’ devices can be used for edge training, issues and challenges occur as deploying this distributed system.</p>

<p>The first challenge is <strong>struggling workers</strong>.
Training devices are heterogeneous, from limited IoT cameras to high-end media centers with powerful GPUs.
They are not designed to do machine learning.
So, a good edge-based distributed learning framework must be able to handle a variety of speeds in training tasks.</p>

<p>The second challenge is how to <strong>scale up</strong> clusters.
On a campus, thousands and more devices may contribute computing resources to the same training tasks.
However, these devices may be located far no matter in physical or in network topology.
The question of how can we well use them well, without struggling with endless transmission time remains a challenge.</p>

<p>The third issue is frequently <strong>joining and exiting</strong> of devices.
We can’t rely on each device to faithfully work on training tasks rather than their original workload.
Smartly schedule work balance and handle join/exit issues also need under consideration.</p>

<h2 id="our-proposal">Our proposal</h2>

<ul>
  <li>
    <p>Dynamic training data distribution and runtime profiler</p>

    <p>We design a dynamic training data distribution mechanism that helps both the first and the third challenges.
  Preprocessing data can be transmitted without leakage of raw and sensitive information.
  This can help struggling workers who can train small batches in order to upload parameters with a similar training time.
  Also, for extremely slow devices, join and exit of devices cases, dynamic data distribution and profiler can help with keeping global training parameters from pollution and staleness.</p>

    <p>To counter heterogeneity, more approaches were applied in our later research.
  More details were introduced to the runtime profiler in the later works.</p>
  </li>
  <li>
    <p>Asynchronous and synchronous aggregation enabled</p>

    <p>In our findings, asynchronous and synchronous parameter update have their pros and cons.
  Keeping sync all the time leads to struggling worker issues unsolvable.
  However, async’s harm to accuracy and convergence time also needs attention.
  To carefully choose between these two update policies at the runtime is what we proposed to make use of their own advantages.</p>
  </li>
  <li>
    <p>Leader role splitting</p>

    <p>The idea is to let worker devices with higher bandwidth take leader-role during training.
  Parameter updating does not require much computation but only needs a great of bandwidth.
  Devices with sufficient bandwidth can also work as virtual leader devices.
  This approach helps minimize physical devices we used and more leaders can further scale up workers’ limits.</p>
  </li>
</ul>]]></content><author><name>Pengzhan Hao</name></author><category term="Research" /><category term="Edge computing" /><summary type="html"><![CDATA[This post introduces our previous milestone in project “Edge trainer”, as the paper “EDDL: A Distributed Deep Learning System for Resource-limited Edge Computing Environment.” was published. As the first part of the introductions, I focus only on the motivation and summary of our works. More details in design and implementation can be found in late posts.]]></summary></entry><entry><title type="html">Generate Word Cloud Figures with Chinese-Tokenization and WordCloud python libraries</title><link href="https://blog.pengzhan.dev/posts/generate-word-cloud-with-chinese-fenci" rel="alternate" type="text/html" title="Generate Word Cloud Figures with Chinese-Tokenization and WordCloud python libraries" /><published>2020-09-15T22:00:14-04:00</published><updated>2021-12-11T13:35:39-05:00</updated><id>https://blog.pengzhan.dev/posts/generate-word-cloud-with-chinese-fenci</id><content type="html" xml:base="https://blog.pengzhan.dev/posts/generate-word-cloud-with-chinese-fenci"><![CDATA[<p>Let’s generate a word cloud like this. 
Don’t understand the language is not a big deal.
If your written language is based on latin alphabet(or other language has space between words), skip tokenization.</p>

<p><img src="/static/2020-09/2020-06-28.png" height="250" />
<!--more--></p>

<h2 id="background">Background</h2>

<p>Recently, I set up a web-based RSS client for retrieving and organizing everyday news. I used <a href="https://tt-rss.org/">TinyTinyRSS</a>, or as ttrss, a popular RSS client which friendly to docker. Thanks to developer <a href="https://ttrss.henry.wang/#about">HenryQW</a>, a well-written Nginx-based docker configuration is already available in docker hub. With more feeds were added, I found some feeds does not need to be checked everyday. Thus I was thinking to create a script to automatically list all keywords appears in a last period and generate a heat map kind figure of it.</p>

<p>Before you go further, I’ll tell you all my settings to give readers a general overview.</p>

<p>My first step is to read all text-based information from TTRSS’s PostgreSQL database. With information, I used a Chinese-NLP library, <a href="https://github.com/fxsjy/jieba">jieba</a>, to extract all keyword with their occurrences frequency. By using <a href="https://github.com/amueller/word_cloud">WordCloud</a>, a python library, word cloud figure is generated and present. More details will be discussed in later sections.</p>

<h2 id="get-rss-feeds-text">Get RSS feeds’ text</h2>

<p>My first thought is generating a keyword heat map for economy news of a last week. Since this blog post are more skewed to Chinese tokenization and draw the word cloud figure. I’ll leave my code here just in case. The SQL connector I used is <a href="https://pypi.org/project/psycopg2/">psycopg2</a>, an easy-use PostgreSQL library.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
	<span class="bp">self</span><span class="p">.</span><span class="n">dbe</span> <span class="o">=</span> <span class="n">psycopg2</span><span class="p">.</span><span class="n">connect</span><span class="p">(</span>
    	<span class="n">host</span><span class="o">=</span><span class="n">DB_HOST</span><span class="p">,</span> <span class="n">port</span><span class="o">=</span><span class="n">DB_PORT</span><span class="p">,</span> <span class="n">database</span><span class="o">=</span><span class="n">DB_NAME</span><span class="p">,</span> <span class="n">user</span><span class="o">=</span><span class="n">DB_USER</span><span class="p">,</span> <span class="n">password</span><span class="o">=</span><span class="n">DB_PASS</span><span class="p">)</span>

<span class="k">def</span> <span class="nf">get_1w_of_feed_byid</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="nb">id</span><span class="o">=</span><span class="mi">1</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">list</span><span class="p">:</span>
	<span class="n">cur</span> <span class="o">=</span> <span class="bp">self</span><span class="p">.</span><span class="n">dbe</span><span class="p">.</span><span class="n">cursor</span><span class="p">()</span>
    <span class="n">cur</span><span class="p">.</span><span class="n">execute</span><span class="p">(</span><span class="s">'SELECT content FROM public.ttrss_entries </span><span class="se">\
</span><span class="s">    	where date_updated &gt; now() - interval </span><span class="se">\'</span><span class="s">1 week</span><span class="se">\'</span><span class="s"> AND id in ( </span><span class="se">\
</span><span class="s">        select int_id from DB_TABLE_NAME </span><span class="se">\
</span><span class="s">        where feed_id='</span> <span class="o">+</span> <span class="nb">str</span><span class="p">(</span><span class="nb">id</span><span class="p">)</span> <span class="o">+</span> <span class="s">' </span><span class="se">\
</span><span class="s">        ) </span><span class="se">\
</span><span class="s">        ORDER BY id ASC '</span>
        <span class="p">)</span>
	<span class="n">rows</span> <span class="o">=</span> <span class="n">cur</span><span class="p">.</span><span class="n">fetchall</span><span class="p">()</span>
	<span class="k">return</span> <span class="n">rows</span>
</code></pre></div></div>

<p>Most arguments are intuitive and easy to understand. The only exception is argument of function <em>get_1w_of_feed_byid</em>. This <strong>id</strong> is the feed index of my subscriptions.</p>

<h2 id="tokenize-with-frequency">Tokenize with frequency</h2>

<p>Two popular tokenization library were used, and I chose <a href="https://github.com/fxsjy/jieba">jieba</a> after a few comparison. Before cutting the sentence, we first need to remove all punctuation marks.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">remove_biaodian</span><span class="p">(</span><span class="n">text</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
    <span class="n">punct</span> <span class="o">=</span> <span class="nb">set</span><span class="p">(</span><span class="sa">u</span><span class="s">''':!),.:;?]}¢'"、。〉》」』】〕〗〞︰︱︳﹐､﹒
                ﹔﹕﹖﹗﹚﹜﹞！），．：；？｜｝︴︶︸︺︼︾﹀﹂﹄﹏､～￠
                々‖•·ˇˉ―--′’”([{£¥'"‵〈《「『【〔〖（［｛￡￥〝︵︷︹︻
                ︽︿﹁﹃﹙﹛﹝（｛“‘-—_…'''</span><span class="p">)</span>
    <span class="n">ret</span> <span class="o">=</span> <span class="s">""</span>
    <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">text</span><span class="p">:</span>
        <span class="k">if</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">punct</span><span class="p">:</span>
            <span class="n">ret</span> <span class="o">+=</span> <span class="s">''</span>
        <span class="k">else</span><span class="p">:</span>
            <span class="n">ret</span> <span class="o">+=</span> <span class="n">x</span>
    <span class="k">return</span> <span class="n">ret</span>
</code></pre></div></div>

<p>After we have an all characters string, we can call jieba. By using the function <em>jieba.posseg.cut</em> with or without paddle, we can have a word list and their “part of speech”.  As you can see in the following code, I also did two more works.</p>

<p>First, in the if statement, I only kept all nouns with some categories. Category abbreviation such as “nr” and “ns” represent different “part of speech”, I attached with categories I used in the following table. For more details you can find in this <a href="https://github.com/fxsjy/jieba">link</a>.</p>

<p>The second work is only keeping words with length longer than 2 characters. In Chinese, there’s no space between words such as Latin writing systems. Since then, some single-character-words such as conjunction words are easy to be misrecognized as specialty-noun.  And this misrecognition will cause more single-character being regarded as specialty-noun. I am not able to improve NLP method, so I used a easy way to fix this by removing any words less than 2 characters.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">import</span> <span class="nn">jieba.posseg</span> <span class="k">as</span> <span class="n">pseg</span>

<span class="k">def</span> <span class="nf">get_noun_jieba</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">content</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">list</span><span class="p">:</span>
	<span class="n">content</span> <span class="o">=</span> <span class="n">remove_biaodian</span><span class="p">(</span><span class="n">content</span><span class="p">)</span>
	<span class="n">words</span> <span class="o">=</span> <span class="n">pseg</span><span class="p">.</span><span class="n">cut</span><span class="p">(</span><span class="n">content</span><span class="p">)</span>	<span class="c1"># Invoking jieba.posseg.cut function 
</span>
	<span class="n">ret</span> <span class="o">=</span> <span class="p">[]</span>
	<span class="k">for</span> <span class="n">word</span><span class="p">,</span> <span class="n">flag</span> <span class="ow">in</span> <span class="n">words</span><span class="p">:</span>
		<span class="c1"># print(word, flag)
</span>		<span class="k">if</span> <span class="n">flag</span> <span class="ow">in</span> <span class="p">[</span><span class="s">'nr'</span><span class="p">,</span> <span class="s">'ns'</span><span class="p">,</span> <span class="s">'nt'</span><span class="p">,</span> <span class="s">'nw'</span><span class="p">,</span> <span class="s">'nz'</span><span class="p">,</span> <span class="s">'PER'</span><span class="p">,</span> <span class="s">'ORG'</span><span class="p">,</span> <span class="s">'x'</span><span class="p">]:</span>   <span class="c1"># LOC
</span>			<span class="n">ret</span><span class="p">.</span><span class="n">append</span><span class="p">(</span><span class="n">word</span><span class="p">)</span>
	<span class="k">return</span> <span class="p">[</span><span class="n">remove_biaodian</span><span class="p">(</span><span class="n">i</span><span class="p">)</span> <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="n">ret</span> <span class="k">if</span> <span class="n">i</span><span class="p">.</span><span class="n">strip</span><span class="p">()</span> <span class="o">!=</span> <span class="s">""</span> <span class="ow">and</span> <span class="nb">len</span><span class="p">(</span><span class="n">remove_biaodian</span><span class="p">(</span><span class="n">i</span><span class="p">.</span><span class="n">strip</span><span class="p">()))</span> <span class="o">&gt;=</span> <span class="mi">2</span><span class="p">]</span>
</code></pre></div></div>

<ul>
  <li>Word category names and abbreviations</li>
</ul>

<table>
  <thead>
    <tr>
      <th>Abbreviation</th>
      <th>Category name/ Part of speech</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>nr</td>
      <td>People name noun</td>
    </tr>
    <tr>
      <td>ns</td>
      <td>Location name noun</td>
    </tr>
    <tr>
      <td>nt</td>
      <td>Organization name noun</td>
    </tr>
    <tr>
      <td>nw</td>
      <td>Arts work noun</td>
    </tr>
    <tr>
      <td>nz</td>
      <td>Other noun</td>
    </tr>
    <tr>
      <td>PER</td>
      <td>People name noun</td>
    </tr>
    <tr>
      <td>ORG</td>
      <td>Location name noun</td>
    </tr>
    <tr>
      <td>x</td>
      <td>Non-morpheme word</td>
    </tr>
  </tbody>
</table>

<p>With all words extracted, we can easily calculate their frequencies.  After this, we can using the following line of code to print a sorted result to verify correctness.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">noun</span> <span class="o">=</span> <span class="n">seg</span><span class="p">.</span><span class="n">get_noun_jieba</span><span class="p">(</span><span class="n">test_content</span><span class="p">)</span>
<span class="c1"># ... Calculate frequency of above word list ...
</span><span class="k">print</span><span class="p">(</span><span class="nb">sorted</span><span class="p">(</span><span class="n">a_dict</span><span class="p">.</span><span class="n">items</span><span class="p">(),</span> <span class="n">key</span><span class="o">=</span><span class="k">lambda</span> <span class="n">x</span><span class="p">:</span> <span class="n">x</span><span class="p">[</span><span class="mi">1</span><span class="p">]))</span>
</code></pre></div></div>

<h2 id="draw-word-cloud">Draw word cloud</h2>

<p>With a keyword and frequency dictionary(data structure), we can just call built-in functions from wordcloud library to generate the figure.</p>

<p>First we need to initialize an instance of wordcloud class. As you can see in my code, I set it with 6 parameters. Width and Height of the canvas, maximum amount of words used to generate the figure, the font of words, background color and margin between any two words.</p>

<p>After having the instance, we call function <em>generate_from_frequencies</em> and pass keyword dictionary to it. The return value of this function is an bitmap image, which we can use <a href="https://matplotlib.org/">matplotlib</a> to plot it to your screen.</p>

<p>I tested my plot on ubuntu-subsystem on Windows 10, unfortunately matplotlib under subsystem depends on x11 window manager and its not default available on windows. We need to install an  x11 manager to support. <a href="https://sourceforge.net/projects/xming/">Xming</a> is the one I used.</p>

<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kn">from</span> <span class="nn">wordcloud</span> <span class="kn">import</span> <span class="n">WordCloud</span>
<span class="kn">import</span> <span class="nn">matplotlib.pyplot</span> <span class="k">as</span> <span class="n">plt</span>

<span class="n">font_path</span> <span class="o">=</span> <span class="s">"./font/haipai.ttf"</span>
<span class="n">output_path</span> <span class="o">=</span> <span class="s">"./font/out.png"</span>


<span class="k">def</span> <span class="nf">show_figure_with_frequency</span><span class="p">(</span><span class="n">keywords</span><span class="p">:</span> <span class="nb">dict</span><span class="p">):</span>
    <span class="n">wc</span> <span class="o">=</span> <span class="n">WordCloud</span><span class="p">(</span><span class="n">width</span><span class="o">=</span><span class="mi">828</span><span class="p">,</span> <span class="n">height</span><span class="o">=</span><span class="mi">1792</span><span class="p">,</span> <span class="n">max_words</span><span class="o">=</span><span class="mi">200</span><span class="p">,</span> <span class="n">font_path</span><span class="o">=</span><span class="n">font_path</span><span class="p">,</span>
                   <span class="n">background_color</span><span class="o">=</span><span class="s">"white"</span><span class="p">,</span> <span class="n">margin</span><span class="o">=</span><span class="mi">1</span><span class="p">).</span><span class="n">generate_from_frequencies</span><span class="p">(</span><span class="n">keywords</span><span class="p">)</span>
    <span class="n">plt</span><span class="p">.</span><span class="n">imshow</span><span class="p">(</span><span class="n">wc</span><span class="p">)</span>
    <span class="n">plt</span><span class="p">.</span><span class="n">axis</span><span class="p">(</span><span class="s">'off'</span><span class="p">)</span>
    <span class="n">plt</span><span class="p">.</span><span class="n">show</span><span class="p">()</span>
</code></pre></div></div>

<p>If everything work fine, a word cloud figure will show up in a new window. My version looks like this.</p>

<p><img src="/static/2020-09/2020-06-28.png" height="150" /></p>

<p>This generated word cloud figure reflects the most popular economy news’ keyword in the week started 06-28-2020. Two largest words in the figure are “新冠” and “新冠病毒”, both means “Covid-19” (This figure was in the week of the second covid spur in Beijing, China). The size of the image fits my phone screen and I can use an app to automatic sync it to my phone’s wallpaper. However, in this image, too many location nouns are presented. This will be something I can make progress on in the future.</p>]]></content><author><name>Pengzhan Hao</name></author><category term="Visualization" /><summary type="html"><![CDATA[Let’s generate a word cloud like this. Don’t understand the language is not a big deal. If your written language is based on latin alphabet(or other language has space between words), skip tokenization.]]></summary></entry><entry><title type="html">Xv6 introduction</title><link href="https://blog.pengzhan.dev/posts/intro-xv6" rel="alternate" type="text/html" title="Xv6 introduction" /><published>2017-07-28T14:56:55-04:00</published><updated>2021-12-11T13:35:39-05:00</updated><id>https://blog.pengzhan.dev/posts/intro-xv6</id><content type="html" xml:base="https://blog.pengzhan.dev/posts/intro-xv6"><![CDATA[<p>In this post, you will learn a few basic concepts of xv6. Learning path will be closed coupled to first project assignment I gave when I assisted in teaching OS classes.
Understand system call and know how to implement a simple one will be coved as the first half.
In the second half of this post, I will discuss a little bit more on how to debug xv6 using gdb.<br />
<!--more--></p>

<h2 id="xv6-systemcall">Xv6 Systemcall</h2>

<p>To invoke a system call, we have to first define a user mode function to be the interface of the kernel instruction  in file <em>user.h</em>.</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kt">void</span> <span class="nf">function</span> <span class="p">(</span><span class="kt">void</span><span class="p">);</span>
</code></pre></div></div>

<p>This interface-like function will then pass the function name, in this case function, to <em>usys.S</em>. When using user mode function in programs, <em>usys.S</em> will generate a reference to SYS_function and push system call number of this function into %eax. After that, system can know from <em>syscall.c</em> and determining whether this system call is available. We must define same name system function and add it into <em>syscall.h</em> and <em>syscall.c</em>.</p>

<div class="language-c highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="cp">#define SYS_function ##  // ## is the system call number
</span><span class="p">[</span><span class="n">SYS_function</span><span class="p">]</span>  <span class="n">sys_function</span> <span class="c1">// real system function name</span>
<span class="k">extern</span> <span class="kt">int</span> <span class="nf">sys_function</span><span class="p">(</span><span class="kt">void</span><span class="p">);</span> <span class="c1">// real system function declaration</span>
</code></pre></div></div>

<p>After adding these sentences to syscall files, we can implement real function in specific place where you want to make the function works well.</p>

<p>Sometimes, we need to pass variables among system calls. In this case, variables’ values are not necessary and even can’t be pass directly into system_function. When invoke a system call function, all variables of this system call will be pushed into current process’ stack. In file <em>syscall.c</em>, multiple functions are provided to get these variables from the process. I won’t waste time on explaining how to use these functions especially when elegant and detailed comments were written in source codes. However, I will explain concepts and how process organized and works in xv6 in future articles.</p>

<h2 id="debug-xv6-with-gdb">Debug xv6 with gdb</h2>

<p>Please make sure that you have used gdb before.
If you never used gdb, you may write a simple 50-100 lines c code and practice how to use gdb first.</p>

<ul>
  <li><a href="https://sourceware.org/gdb/current/onlinedocs/gdb/">GDB Manual</a></li>
  <li><a href="https://darkdust.net/files/GDB%20Cheat%20Sheet.pdf">GDB cheatsheet (pdf)</a></li>
</ul>

<p>To make sure xv6 gdb enabled, please check if <em>.gdbinit.tmpl</em> file exist.
This file is used for generate <em>.gdbinit</em> file which you can late consider it as a configuration for gdb.</p>

<p>Before running the xv6 instance in QEMU, one more thing you need to know is that using gdb to debug xv6 must be attached remotely.
This is because xv6 was running within QEMU, and emulator is virtually gapped from the host device.
Later when you start debugging, QEMU will open a gdb server to let gdb client connect to.</p>

<p>Once you want to start, using following command to compile and run xv6</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>make qemu-nox-gdb
<span class="k">***</span> Now run <span class="s1">'gdb'</span><span class="nb">.</span>
qemu-system-i386 <span class="nt">-nographic</span> <span class="nt">-drive</span> <span class="nv">file</span><span class="o">=</span>fs.img,index<span class="o">=</span>1,media<span class="o">=</span>disk,format<span class="o">=</span>raw <span class="nt">-drive</span> <span class="nv">file</span><span class="o">=</span>xv6.img,index<span class="o">=</span>0,media<span class="o">=</span>disk,format<span class="o">=</span>raw <span class="nt">-smp</span> 2 7
</code></pre></div></div>

<p>At this moment, it feels xv6 was stuck, this is because QEMU is ready to be connected by the gdb client.
You may use the <em>.gdbinit</em> to automatically finish this remote connection by simple typein following command in another terminal.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span>gdb <span class="nt">-x</span> .gdbinit
GNU gdb <span class="o">(</span>Debian 8.2.1-2+b3<span class="o">)</span> 8.2.1

...

The target architecture is assumed to be i8086
<span class="o">[</span>f000:fff0]    0xffff0: ljmp   <span class="nv">$0x3630</span>,<span class="nv">$0xf000e05b</span>
0x0000fff0 <span class="k">in</span> ?? <span class="o">()</span>
+ symbol-file kernel
warning: A handler <span class="k">for </span>the OS ABI <span class="s2">"GNU/Linux"</span> is not built into this configuration
of GDB.  Attempting to <span class="k">continue </span>with the default i8086 settings.

<span class="o">(</span>gdb<span class="o">)</span> 
</code></pre></div></div>

<p>Now within this gdb client shell, type ‘c’ to continue the xv6, and you will see xv6 start execution in the first terminal.</p>

<p>At this moment, you may add breakpoints to your code to see if your code is correctly implemented or not.</p>

<p><strong>One more thing</strong>, if you open <em>.gdbinit</em> file, you’ll find that it by default connect to a localhost target.
If you are working on some other environment that target and client were not placed in the same device, change the localhost to ip address correspondingly.
Using ssh may connect to different physical devices under same domain name, this is because load balancer were used. To check ip address, search command <em>ip</em>.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>target remote localhost:28467
<span class="c"># target remote [ip-addr]:28467</span>
</code></pre></div></div>]]></content><author><name>Pengzhan Hao</name></author><category term="Xv6" /><category term="Teaching" /><category term="Operating system" /><summary type="html"><![CDATA[In this post, you will learn a few basic concepts of xv6. Learning path will be closed coupled to first project assignment I gave when I assisted in teaching OS classes. Understand system call and know how to implement a simple one will be coved as the first half. In the second half of this post, I will discuss a little bit more on how to debug xv6 using gdb.]]></summary></entry><entry><title type="html">Some of my previews experiment works: 2016</title><link href="https://blog.pengzhan.dev/posts/some-of-my-previews-exper-work" rel="alternate" type="text/html" title="Some of my previews experiment works: 2016" /><published>2016-10-28T12:27:33-04:00</published><updated>2021-12-11T13:35:39-05:00</updated><id>https://blog.pengzhan.dev/posts/some-of-my-previews-exper-work</id><content type="html" xml:base="https://blog.pengzhan.dev/posts/some-of-my-previews-exper-work"><![CDATA[<p>This blog contains only some basic record of my works. For some details, I will write a unique blog just for some specific topics.
<!--more--></p>

<h1 id="2016-10">2016-10</h1>

<h2 id="time-experiment-of-rsync">Time Experiment of rsync</h2>

<p>Patch is based on rsync with version 3.1.2. [<a href="https://download.samba.org/pub/rsync/rsync-3.1.2.tar.gz">Rsync</a>|<a href="/static/2016-10/rsync/rsync-3.1.2-time.patch">Patch</a>]</p>

<h3 id="how-to-collect-data">How to collect data</h3>

<p>Basically, everything of transmission time and computation time will be output with overall time will be printed on the console.
But we also need some bash script to collect data through different size of random size and with different modification through them.</p>

<ul>
  <li>Start from 8K to 64M, modify at beginning, [<a href="/static/2016-10/rsync/small2Big_change_at_begin.sh">Bash script</a>]</li>
  <li>Start from 8K to 64M, modify at last, [<a href="/static/2016-10/rsync/small2Big_change_at_last.sh">Bash script</a>]</li>
  <li>Start from 8K to 64M, modify at random place with a (slow) python script, [<a href="/static/2016-10/rsync/small2Big_change_at_anyplace.sh">Bash script</a>|<a href="/static/2016-10/rsync/addbyte.py">Python program</a>]</li>
</ul>

<h2 id="time-experiment-of-seafile">Time Experiment of seafile</h2>

<p>Patch is based on seafile 5.1.4. You can find the release from <a href="https://github.com/haiwen/seafile/releases">seafile official repo</a>. You may follow official compile instructions from <a href="https://manual.seafile.com/build_seafile/linux.html">here</a>. [<a href="">Patch <strong>no longer avaiable, new version at following sections</strong></a>]</p>

<h3 id="how-to-collect-data-1">How to collect data</h3>

<p>We also need everything be done using scripting. But this time I only design added some distance between two increasing files’ sizes.</p>

<ul>
  <li>Start from 8K to 16M, 4 times increasing, modify at beginning/ at 1024 different places with python script. [<a href="/static/2016-11/seafile/trans.sh">Bash Script</a>|<a href="/static/2016-11/seafile/addbyte.py">Python program</a>]</li>
  <li>After using this auto testing script, everything of output will be marked in log files of seafile, which located in <strong>~/.ccnet/log/seafile.log</strong></li>
  <li>We need to use this simple awk code and vim operation to extract data.</li>
</ul>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># CDC: content defined chucks</span>
<span class="c"># HUT: Http upload traffic</span>
<span class="c"># ALL: overall time of one commit &amp; upload</span>
<span class="nb">awk</span> <span class="s1">'/CDC|HUT|ALL/ {print $4,$5}'</span> ~/.ccnet/log/seafile.log <span class="o">&gt;</span> results.stat
</code></pre></div></div>

<h3 id="install-seafile-on-odroid-xu">Install Seafile on odroid xu</h3>

<p>Due to failure of my cross-compile to seafile on android. I used develop board as a replacement experiment platform for ARM-seafile testing. I used a <a href="http://www.hardkernel.com/main/products/prdt_info.php?g_code=G137510300620">odroid xu</a> as hardware standard. Because all I need is an ARM platform, only an ARM-Ubuntu is enough for me. But develop prototype on a board is much fun than coding, I won’t address much this time. But I’ll start a blog telling some really cool stuff I made for a strange aim.</p>

<p>To install a ubuntu with GUI is my all preparation work. I found to way to do this.</p>

<ul>
  <li>
    <p><a href="http://www.armhf.com/boards/odroid-xu/">armhf</a> is a website for arm-based ubuntu. It has a detailed instruction to follow at <a href="http://www.armhf.com/boards/odroid-xu/odroid-sd-install/">here</a>. They also provide ubuntu 12.04/ 14.04 and debian 7.5 to choose. But unfortunately odroid xu’s hdmi output doesn’t supported by ubuntu native firmware. So install ubuntu-desktop might can’t be boot up for video output.</p>
  </li>
  <li>
    <p>Burn images is much easy to install a pre-complied ubuntu system. I found this on odroid xu’s forum, which contains xubuntu image [<a href="http://odroid.in/ubuntu_14.04lts/ubuntu-14.04lts-xubuntu-odroid-xu-20140714.img.xz">download</a>] for odroid xu. With this image, you just need to use dd command to write whole system mirror into sdcard.</p>
  </li>
</ul>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># If .img end with xz, use this command to uncompress first</span>
unxz ubuntu-14.04lts-xubuntu-odroid-xu-20140714.img.xz    
<span class="c"># Burn image into SD-card</span>
<span class="nb">sudo dd </span><span class="k">if</span><span class="o">=</span>ubuntu-14.04lts-xubuntu-odroid-xu-20140714.img <span class="nv">of</span><span class="o">=</span>/dev/sdb <span class="nv">bs</span><span class="o">=</span>1M <span class="nv">conv</span><span class="o">=</span>fsync
<span class="nb">sync</span>
</code></pre></div></div>

<h1 id="2016-11">2016-11</h1>

<h2 id="android-kernel">Android Kernel</h2>

<h3 id="how-to-build-an-android-kernel">How to build an Android Kernel?</h3>

<p>Generally, I won’t tell anything in this parts, just mark some related links, and point out some mistakes or error solutions.</p>

<ul>
  <li>
    <p><a href="http://source.android.com/source/building-kernels.html#figuring-out-which-kernel-to-build">Google Official Guide</a>
– If you don’t have AOSP sources, you have to download prebuilt toolchains which recommended in this guide might not be correct. Use following links to choose your fitting tools.
— <a href="https://android.googlesource.com/?format=HTML">ASOP git root</a>, under sub class “/platform/prebuilts/gcc”</p>
  </li>
  <li>
    <p><a href="https://softwarebakery.com/building-the-android-kernel-on-linux">Packing and Flashing a Boot.img</a> <strong>[highly recommend]</strong></p>
  </li>
</ul>

<h1 id="2016-12">2016-12</h1>

<h2 id="android-kernel-1">Android Kernel</h2>

<h3 id="how-to-compile-with-ftrace">How to compile with ftrace?</h3>

<p>If we want to debug under android, ftrace is a great tool for working. But, ftrace is not available in android if we used default configure file. Android kernel configuration is in <strong>arch/arm64/kernel/configs</strong>. We need to add few lines under that.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">CONFIG_STRICT_MEMORY_RWX</span><span class="o">=</span>y
<span class="nv">CONFIG_FUNCTION_TRACER</span><span class="o">=</span>y
<span class="nv">CONFIG_FUNCTION_GRAPH_TRACER</span><span class="o">=</span>y
<span class="nv">CONFIG_DYNAMIC_FTRACE</span><span class="o">=</span>y
<span class="nv">CONFIG_PERSISTENT_TRACER</span><span class="o">=</span>y
<span class="nv">CONFIG_IRQSOFF_TRACER</span><span class="o">=</span>y
<span class="nv">CONFIG_PREEMPT_TRACER</span><span class="o">=</span>y
<span class="nv">CONFIG_SCHED_TRACER</span><span class="o">=</span>y
<span class="nv">CONFIG_STACK_TRACER</span><span class="o">=</span>y
</code></pre></div></div>

<h3 id="how-to-extract-android-images-dump-an-image">How to extract android images: Dump an image</h3>

<p>If we want to hold a rooted status after flashing boot, we need to extract an image from android devices. We can first use following command to find which blocks belongs to. According to some references, <a href="http://forum.xda-developers.com/showthread.php?t=2450045">this article</a> provide three ways to dump an image, I picked one for easy using.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>adb shell
<span class="nb">ls</span> <span class="nt">-al</span> /dev/block/platform/<span class="nv">$SOME</span><span class="se">\_</span>DEVICE../../by-name <span class="c"># {Partitions} -&gt; {Device Block}</span>

<span class="c"># dump file</span>
su
<span class="nb">dd </span><span class="k">if</span><span class="o">=</span>/dev/block/mmcblk0p37 <span class="nv">of</span><span class="o">=</span>/sdcard/boot.img
</code></pre></div></div>]]></content><author><name>Pengzhan Hao</name></author><category term="Research" /><category term="Log" /><category term="Miscellanies" /><summary type="html"><![CDATA[This blog contains only some basic record of my works. For some details, I will write a unique blog just for some specific topics.]]></summary></entry><entry><title type="html">Using charles proxy to monitor mobile SSL traffics</title><link href="https://blog.pengzhan.dev/posts/charles-is-not-a-good-tool" rel="alternate" type="text/html" title="Using charles proxy to monitor mobile SSL traffics" /><published>2016-10-27T22:50:33-04:00</published><updated>2021-12-11T13:35:39-05:00</updated><id>https://blog.pengzhan.dev/posts/charles-is-not-a-good-tool</id><content type="html" xml:base="https://blog.pengzhan.dev/posts/charles-is-not-a-good-tool"><![CDATA[<p>In this blog, I will generally talk about how to use proper tools to monitor SSL traffics of a mobile devices. Currently, I only can dealing with those SSL traffics which use an obviously certification. Some applications may not using system root cert or they doesn’t provide us a method to modify their own certs. For these situation, I still didn’t find a good solutions for it. But I’ll keep updating this if I get one.<br />
My current solution is using AP to forward all SSL traffic to a proxy, <a href="https://www.charlesproxy.com/">charles proxy</a> is my first choice (Prof asked). It’s a non-free software which still update new versions now. So mainly, I’ll talk about how to charles SSL proxy.
<!--more--></p>

<h3 id="preparations">Preparations</h3>
<ul>
  <li>Monitor device situation: Linux Machine with wireless adapter</li>
  <li>Download the newest version(4.0.1) of charles</li>
  <li>Target android devices with root privilege</li>
</ul>

<h3 id="install-charles-and-configuration">Install Charles and Configuration</h3>

<ul>
  <li>You have to install charles first. After downloading the charles proxy, you have to unzip it and configure some basic settings.</li>
</ul>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># open charles first</span>
./bin/charles  
</code></pre></div></div>
<ul>
  <li>Save charles’ private key and public key</li>
</ul>

<p>In Help -&gt; SSL Proxying -&gt; Export Charles Root Certificate and Private Key, enter a password and save the public and private key in *.p12 format.<br />
You also need to save charles Root Certificate, it also contains in the same menu. For convience, save it as *.pem format.</p>

<ul>
  <li>Set Proxy and SSL Proxy</li>
</ul>]]></content><author><name>Pengzhan Hao</name></author><category term="Network" /><category term="Charles proxy" /><summary type="html"><![CDATA[In this blog, I will generally talk about how to use proper tools to monitor SSL traffics of a mobile devices. Currently, I only can dealing with those SSL traffics which use an obviously certification. Some applications may not using system root cert or they doesn’t provide us a method to modify their own certs. For these situation, I still didn’t find a good solutions for it. But I’ll keep updating this if I get one. My current solution is using AP to forward all SSL traffic to a proxy, charles proxy is my first choice (Prof asked). It’s a non-free software which still update new versions now. So mainly, I’ll talk about how to charles SSL proxy.]]></summary></entry><entry><title type="html">STSD: Stop Talking Start Doing</title><link href="https://blog.pengzhan.dev/posts/welcome-to-my-blog" rel="alternate" type="text/html" title="STSD: Stop Talking Start Doing" /><published>2016-10-26T22:50:33-04:00</published><updated>2024-04-03T18:23:31-04:00</updated><id>https://blog.pengzhan.dev/posts/welcome-to-my-blog</id><content type="html" xml:base="https://blog.pengzhan.dev/posts/welcome-to-my-blog"><![CDATA[]]></content><author><name>Pengzhan Hao</name><email>haopengzhan@gmail.com</email></author><category term="Diary" /><summary type="html"><![CDATA[]]></summary></entry></feed>