diff --git a/._config.yml.swp b/._config.yml.swp deleted file mode 100644 index 9b7449d..0000000 Binary files a/._config.yml.swp and /dev/null differ diff --git a/README.md b/README.md index 8298e76..03e89ee 100644 --- a/README.md +++ b/README.md @@ -1,55 +1,3 @@ -# jekyll-theme-EasyBook +My Personal Blog -Another Jekyll Blog Theme "EasyBook" - -Start blogging with a simple but powerful theme. - -## Features - -- Everything Basic Jekyll has. -- **Pagination** is enabled. -- **Comments** using Disqus or 多说. -- **TOC** makes reading easier. -- **Profile** including your links and avatar on the sidebar. -- *And more...* - -## Install - -Read [Quick Guide] to learn how to install and publish. - -Read [Welcome to Jekyll and EasyBook][helloPost] and [Jekyll][jekyll] for further information. - -**If need help, please:** - - - [**Read Wiki pages**](https://github.com/laobubu/jekyll-theme-EasyBook/wiki) - - **Google it** - - [**Submit an issue**](https://github.com/laobubu/jekyll-theme-EasyBook/issues/new) - -## Donate - -Buy me a coffee if you find EasyBook useful for you :smiley: - -Visit - -## Demo - -[laobubu's blog](http://blog.laobubu.net) | [empty site](http://laobubu.github.io/jekyll-theme-EasyBook) - -![Homepage](https://ooo.0o0.ooo/2016/02/11/56bc997c65daf.png) - -## License - -``` -The MIT License (MIT) -Copyright (c) 2016 laobubu - -Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -``` - -[Quick Guide]: https://github.com/laobubu/jekyll-theme-EasyBook/wiki/Quick-Guide -[helloPost]: http://laobubu.github.io/jekyll-theme-EasyBook/archivers/hello -[jekyll]: http://jekyllrb.com/ +Nothing new diff --git a/_includes/head.html b/_includes/head.html index ab7a653..da2a2b1 100644 --- a/_includes/head.html +++ b/_includes/head.html @@ -9,4 +9,14 @@ + diff --git a/_posts/2016-10-27-charles-is-not-a-good-tool.md b/_posts/2016-10-27-charles-is-not-a-good-tool.md new file mode 100644 index 0000000..9093d54 --- /dev/null +++ b/_posts/2016-10-27-charles-is-not-a-good-tool.md @@ -0,0 +1,16 @@ +--- +layout: post +title: "Using charles proxy to monitor mobile SSL traffics" +date: 2016-10-27 22:50:33 -0400 +categories: Network +--- + +In this blog, I will generally talk about how to use proper tools to monitor SSL traffics of a mobile devices. Currently, I only can dealing with those SSL traffics which use an obviously certification. Some applications may not using system root cert or they doesn't provide us a method to modify their own certs. For these situation, I still didn't find a good solutions for it. But I'll keep updating this if I get one. +My current solution is using AP to forward all SSL traffic to a proxy, [charles proxy](https://www.charlesproxy.com/) is my first choice (Prof asked). It's a non-free software which still update new versions now. So mainly, I'll talk about how to charles SSL proxy. + +### Preparations +- Monitor device situation: Linux Machine with wireless adapter +- Download the newest version(4.0.1) of charles +- Target android devices with root privilege + + diff --git a/_posts/2016-10-28-some-of-my-previews-exper-work.md b/_posts/2016-10-28-some-of-my-previews-exper-work.md new file mode 100644 index 0000000..5e15d92 --- /dev/null +++ b/_posts/2016-10-28-some-of-my-previews-exper-work.md @@ -0,0 +1,27 @@ +--- +layout: post +title: "Some of my previews experiment works" +date: 2016-10-28 12:27:33 -0400 +categories: Research +--- +# Time series + + +## 2016-10 + +### Time Experiment of rsync + +Patch is based on rsync with version 3.1.2. \[[Rsync](https://download.samba.org/pub/rsync/rsync-3.1.2.tar.gz)\|[Patch](/static/2016-10/rsync/rsync-3.1.2-time.patch)\] + +#### How to collect data + +Basically, everything of transmission time and computation time will be output with overall time will be printed on the console. +But we also need some bash script to collect data through different size of random size and with different modification through them. + +- Start from 8K to 64M, modify at beginning, \[[Bash script](/static/2016-10/rsync/small2Big_change_at_begin.sh)\] +- Start from 8K to 64M, modify at last, \[[Bash script](/static/2016-10/rsync/small2Big_change_at_last.sh)\] +- Start from 8K to 64M, modify at random place with a (slow) python script, \[[Bash script](/static/2016-10/rsync/small2Big_change_at_anyplace.sh)\|[Python program](/static/2016-10/rsync/addbyte.py)\] + +### Time Experiment of seafile + + diff --git a/_site/404.html b/_site/404.html index 2ab962d..75e7c04 100644 --- a/_site/404.html +++ b/_site/404.html @@ -13,6 +13,16 @@ + @@ -85,6 +95,10 @@
Newest Posts
diff --git a/_site/README.md b/_site/README.md index 8298e76..03e89ee 100644 --- a/_site/README.md +++ b/_site/README.md @@ -1,55 +1,3 @@ -# jekyll-theme-EasyBook +My Personal Blog -Another Jekyll Blog Theme "EasyBook" - -Start blogging with a simple but powerful theme. - -## Features - -- Everything Basic Jekyll has. -- **Pagination** is enabled. -- **Comments** using Disqus or 多说. -- **TOC** makes reading easier. -- **Profile** including your links and avatar on the sidebar. -- *And more...* - -## Install - -Read [Quick Guide] to learn how to install and publish. - -Read [Welcome to Jekyll and EasyBook][helloPost] and [Jekyll][jekyll] for further information. - -**If need help, please:** - - - [**Read Wiki pages**](https://github.com/laobubu/jekyll-theme-EasyBook/wiki) - - **Google it** - - [**Submit an issue**](https://github.com/laobubu/jekyll-theme-EasyBook/issues/new) - -## Donate - -Buy me a coffee if you find EasyBook useful for you :smiley: - -Visit - -## Demo - -[laobubu's blog](http://blog.laobubu.net) | [empty site](http://laobubu.github.io/jekyll-theme-EasyBook) - -![Homepage](https://ooo.0o0.ooo/2016/02/11/56bc997c65daf.png) - -## License - -``` -The MIT License (MIT) -Copyright (c) 2016 laobubu - -Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: - -The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. - -THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. -``` - -[Quick Guide]: https://github.com/laobubu/jekyll-theme-EasyBook/wiki/Quick-Guide -[helloPost]: http://laobubu.github.io/jekyll-theme-EasyBook/archivers/hello -[jekyll]: http://jekyllrb.com/ +Nothing new diff --git a/_site/about/index.html b/_site/about/index.html index 79c83f8..abe7ee4 100644 --- a/_site/about/index.html +++ b/_site/about/index.html @@ -13,6 +13,16 @@ + @@ -53,10 +63,25 @@
-

Pengzhan Hao

-

Email:haopengzhan@gmail.com
-Address:Binghamton University
-Phone: (607)-II-III-II-83-VII-II

+

Pengzhan Hao

+
    +
  • Research Assistant in MOSLab
    • +
  • Master Student of SUNY at Binghamton
    • +
+ +
+
    +
  • +

    Email: phao3 at binghamton dot edu

    +
    • +
  • +

    MOSLab, H-6 Engineering Building, Binghamton University
    +4400 Vestal Parkway East
    +Binghamton, NY 13902

    +
    • +
+ +
@@ -90,6 +115,10 @@ Phone: (607)-II-III-II-83-VII-II

Newest Posts
diff --git a/_site/archivers/charles-is-not-a-good-tool.html b/_site/archivers/charles-is-not-a-good-tool.html new file mode 100644 index 0000000..fc4399a --- /dev/null +++ b/_site/archivers/charles-is-not-a-good-tool.html @@ -0,0 +1,135 @@ + + + + + + + + + Using charles proxy to monitor mobile SSL traffics « Stop Talking, Start Doing - 停止空想，开始行动 + + + + + + + + + + + +
+
+ Stop Talking, Start Doing - 停止空想，开始行动 + +
+
+ +
+
+
+
+ +
+

Using charles proxy to monitor mobile SSL traffics

+

Oct 27, 2016

+
+ +
+

In this blog, I will generally talk about how to use proper tools to monitor SSL traffics of a mobile devices. Currently, I only can dealing with those SSL traffics which use an obviously certification. Some applications may not using system root cert or they doesn’t provide us a method to modify their own certs. For these situation, I still didn’t find a good solutions for it. But I’ll keep updating this if I get one.
+My current solution is using AP to forward all SSL traffic to a proxy, charles proxy is my first choice (Prof asked). It’s a non-free software which still update new versions now. So mainly, I’ll talk about how to charles SSL proxy.

+ +

Preparations

+
    +
  • Monitor device situation: Linux Machine with wireless adapter
    • +
  • Download the newest version(4.0.1) of charles
    • +
  • Target android devices with root privilege
    • +
+ + +
+ + + + + + +
+ +
+
+
+ Pengzhan Hao - 碾子 +
Pengzhan Hao - 碾子
+

+

+ + GitHub + + + Twitter + + + Email + +

+
+ +
+
Newest Posts
+ +
+ +
+
TOC
+
+
+
+
+ +
+
+© 2016 Pengzhan Hao - 碾子 +
+
+ + + + + + diff --git a/_site/archivers/hello.html b/_site/archivers/hello.html index a9029a3..5e14dbf 100644 --- a/_site/archivers/hello.html +++ b/_site/archivers/hello.html @@ -12,6 +12,16 @@ + @@ -87,6 +97,10 @@
Newest Posts
diff --git a/_site/archivers/some-of-my-previews-exper-work.html b/_site/archivers/some-of-my-previews-exper-work.html new file mode 100644 index 0000000..2245605 --- /dev/null +++ b/_site/archivers/some-of-my-previews-exper-work.html @@ -0,0 +1,146 @@ + + + + + + + + + Some of my previews experiment works « Stop Talking, Start Doing - 停止空想，开始行动 + + + + + + + + + + + +
+
+ Stop Talking, Start Doing - 停止空想，开始行动 + +
+
+ +
+
+
+
+ +
+

Some of my previews experiment works

+

Oct 28, 2016

+
+ +
+

Time series

+ +

2016-10

+ +

Time Experiment of rsync

+ +

Patch is based on rsync with version 3.1.2. [Rsync|Patch]

+ +

How to collect data

+ +

Basically, everything of transmission time and computation time will be output with overall time will be printed on the console.
+But we also need some bash script to collect data through different size of random size and with different modification through them.

+ + + +

Time Experiment of seafile

+ + +
+ + + + + + +
+ +
+
+
+ Pengzhan Hao - 碾子 +
Pengzhan Hao - 碾子
+

+

+ + GitHub + + + Twitter + + + Email + +

+
+ +
+
Newest Posts
+ +
+ +
+
TOC
+
+
+
+
+ +
+
+© 2016 Pengzhan Hao - 碾子 +
+
+ + + + + + diff --git a/_site/category/index.html b/_site/category/index.html index f2549d7..562075a 100644 --- a/_site/category/index.html +++ b/_site/category/index.html @@ -13,6 +13,16 @@ + @@ -55,11 +65,27 @@ +

NETWORK

+ + +

NONSENSE

+ + +

RESEARCH

+ @@ -90,6 +116,10 @@
Newest Posts
diff --git a/_site/feed.xml b/_site/feed.xml index c8f5109..09007a0 100644 --- a/_site/feed.xml +++ b/_site/feed.xml @@ -6,10 +6,65 @@ https://codersherlock.github.com// - Thu, 27 Oct 2016 01:09:32 -0400 - Thu, 27 Oct 2016 01:09:32 -0400 + Mon, 31 Oct 2016 17:08:36 -0400 + Mon, 31 Oct 2016 17:08:36 -0400 Jekyll v3.0.1 + + Some of my previews experiment works + <h1 id="time-series">Time series</h1> + +<h2 id="section">2016-10</h2> + +<h3 id="time-experiment-of-rsync">Time Experiment of rsync</h3> + +<p>Patch is based on rsync with version 3.1.2. [<a href="https://download.samba.org/pub/rsync/rsync-3.1.2.tar.gz">Rsync</a>|<a href="/static/2016-10/rsync/rsync-3.1.2-time.patch">Patch</a>]</p> + +<h4 id="how-to-collect-data">How to collect data</h4> + +<p>Basically, everything of transmission time and computation time will be output with overall time will be printed on the console.<br /> +But we also need some bash script to collect data through different size of random size and with different modification through them.</p> + +<ul> + <li>Start from 8K to 64M, modify at beginning, [<a href="/static/2016-10/rsync/small2Big_change_at_begin.sh">Bash script</a>]</li> + <li>Start from 8K to 64M, modify at last, [<a href="/static/2016-10/rsync/small2Big_change_at_last.sh">Bash script</a>]</li> + <li>Start from 8K to 64M, modify at random place with a (slow) python script, [<a href="/static/2016-10/rsync/small2Big_change_at_anyplace.sh">Bash script</a>|<a href="/static/2016-10/rsync/addbyte.py">Python program</a>]</li> +</ul> + +<h3 id="time-experiment-of-seafile">Time Experiment of seafile</h3> + + + Fri, 28 Oct 2016 12:27:33 -0400 + https://codersherlock.github.com//archivers/some-of-my-previews-exper-work + https://codersherlock.github.com//archivers/some-of-my-previews-exper-work + + + Research + + + + + Using charles proxy to monitor mobile SSL traffics + <p>In this blog, I will generally talk about how to use proper tools to monitor SSL traffics of a mobile devices. Currently, I only can dealing with those SSL traffics which use an obviously certification. Some applications may not using system root cert or they doesn’t provide us a method to modify their own certs. For these situation, I still didn’t find a good solutions for it. But I’ll keep updating this if I get one. <br /> +My current solution is using AP to forward all SSL traffic to a proxy, <a href="https://www.charlesproxy.com/">charles proxy</a> is my first choice (Prof asked). It’s a non-free software which still update new versions now. So mainly, I’ll talk about how to charles SSL proxy.</p> + +<h3 id="preparations">Preparations</h3> +<ul> + <li>Monitor device situation: Linux Machine with wireless adapter</li> + <li>Download the newest version(4.0.1) of charles</li> + <li>Target android devices with root privilege</li> +</ul> + + + Thu, 27 Oct 2016 22:50:33 -0400 + https://codersherlock.github.com//archivers/charles-is-not-a-good-tool + https://codersherlock.github.com//archivers/charles-is-not-a-good-tool + + + Network + + + Stop Talking is the worst title of one blog diff --git a/_site/index.html b/_site/index.html index 114cbfd..1e30a7a 100644 --- a/_site/index.html +++ b/_site/index.html @@ -13,6 +13,16 @@ + @@ -52,6 +62,41 @@
    +
  • +

    + Some of my previews experiment works +

    + +
    Oct 28, 2016
    + +
    +

    Time series

    + + +

    + Read More » +

    +
    +
    • + +
  • +

    + Using charles proxy to monitor mobile SSL traffics +

    + +
    Oct 27, 2016
    + +
    +

    In this blog, I will generally talk about how to use proper tools to monitor SSL traffics of a mobile devices. Currently, I only can dealing with those SSL traffics which use an obviously certification. Some applications may not using system root cert or they doesn’t provide us a method to modify their own certs. For these situation, I still didn’t find a good solutions for it. But I’ll keep updating this if I get one.
    +My current solution is using AP to forward all SSL traffic to a proxy, charles proxy is my first choice (Prof asked). It’s a non-free software which still update new versions now. So mainly, I’ll talk about how to charles SSL proxy.

    + + +

    + Read More » +

    +
    +
    • +

  • Stop Talking is the worst title of one blog @@ -106,6 +151,10 @@
    Newest Posts
    diff --git a/_site/static/2016-10/rsync/addbyte.py b/_site/static/2016-10/rsync/addbyte.py new file mode 100644 index 0000000..5c156f8 --- /dev/null +++ b/_site/static/2016-10/rsync/addbyte.py @@ -0,0 +1,57 @@ +import sys +from subprocess import call +import os +import time +import math +import random + + + +# print time.localtime() +hostfile = open(sys.argv[1],'r') +# print "cp -f "+sys.argv[1]+" "+sys.argv[3] +# call(["cp","-f",sys.argv[1],sys.argv[3]]) +guestfile = open(sys.argv[2],'r') +targetfile = open(sys.argv[3],'w+') +hostsize = os.path.getsize(sys.argv[1]) +guestsize = os.path.getsize(sys.argv[2]) + +# print hostsize +""" +print mean +if mean!=0: + for i in range(0,guestsize): + guestfile.seek(i) + target.seek(mean*i) + #print str(mean*i)+":"+target.read(1)+"\t"+str(i)+":"+guestfile.read(1) + temp = target.read(1)+guestfile.read(1) + target.seek(mean*i) + target.write(temp) + +print guestfile,target +""" +mean = 1 +c_sz = [12, 24,47,94,188,375,729,1024,1449,2048,2897,4096,5794,8192] +ndx = int(math.log(float(hostsize)/8192,2)) +dist = int(float(hostsize)/c_sz[ndx] + 0.5) +# print dist + +if mean!=0: + guest = guestfile.read() + target = hostfile.read() + byte_ndx = 0 + flt_byte = 0 + for i in range(0,c_sz[ndx]): + # loc = random.randint(i*dist+flt_byte, (i+1)*dist+flt_byte) + loc = 10+i*dist+flt_byte + # print loc, guest[byte_ndx] + target = target[:loc]+guest[byte_ndx]+target[loc:] + flt_byte += 1 + byte_ndx = (byte_ndx+1)%guestsize +target = target[:-3]+"X"+target[-3:] + +targetfile.write(target) +hostfile.close() +guestfile.close() +targetfile.close() +# print time.localtime() diff --git a/_site/static/2016-10/rsync/rsync-3.1.2-time.patch b/_site/static/2016-10/rsync/rsync-3.1.2-time.patch new file mode 100644 index 0000000..fd96f21 --- /dev/null +++ b/_site/static/2016-10/rsync/rsync-3.1.2-time.patch @@ -0,0 +1,585 @@ +diff -uNr rsync/flist.c rsync-3.1.2/flist.c +--- rsync/flist.c 2016-09-17 00:14:38.448855376 -0400 ++++ rsync-3.1.2/flist.c 2015-08-08 15:47:03.000000000 -0400 +@@ -2431,7 +2431,7 @@ + if (DEBUG_GTE(FLIST, 3)) + rprintf(FINFO, "[%s] flist_eof=1\n", who_am_i()); + } +- //printf("how long build flist takes %ld\n", stats.flist_buildtime); // HPZ: I should not add this trace code, TODO: Delete ++ + return flist; + } + +diff -uNr rsync/main.c rsync-3.1.2/main.c +--- rsync/main.c 2016-10-07 15:10:04.094266860 -0400 ++++ rsync-3.1.2/main.c 2015-08-24 00:58:18.000000000 -0400 +@@ -23,12 +23,10 @@ + #include "rsync.h" + #include "inums.h" + #include "io.h" +-#include //HPZ: + #if defined CONFIG_LOCALE && defined HAVE_LOCALE_H + #include + #endif + +- + extern int dry_run; + extern int list_only; + extern int io_timeout; +@@ -106,8 +104,6 @@ + mode_t orig_umask = 0; + int batch_gen_fd = -1; + int sender_keeps_checksum = 0; +-struct timeval tv, td; // HPZ: only solution +- + + /* There's probably never more than at most 2 outstanding child processes, + * but set it higher, just in case. */ +@@ -353,7 +349,7 @@ + write_batch < 0 ? " (BATCH ONLY)" : dry_run ? " (DRY RUN)" : ""); + } + +- fflush(stdout); ++ fflush(stdout); + fflush(stderr); + } + +@@ -527,12 +523,11 @@ + batch_gen_fd = from_gen_pipe[0]; + *f_out_p = from_gen_pipe[1]; + *f_in_p = batch_fd; +- //printf("1***\n"); + pid = (pid_t)-1; /* no child pid */ + #ifdef ICONV_CONST + setup_iconv(); + #endif +- } else if (local_server) { // HPZ: Local Branch ++ } else if (local_server) { + /* If the user didn't request --[no-]whole-file, force + * it on, but only if we're not batch processing. */ + if (whole_file < 0 && !write_batch) +@@ -543,7 +538,7 @@ + setup_iconv(); + #endif + } else { +- pid = piped_child(args, f_in_p, f_out_p); // HPZ: Remote Branch ++ pid = piped_child(args, f_in_p, f_out_p); + #ifdef ICONV_CONST + setup_iconv(); + #endif +@@ -1112,9 +1107,9 @@ + + /* This is called once the connection has been negotiated. It is used + * for rsyncd, remote-shell, and local connections. */ +-int client_run(int f_in, int f_out, pid_t pid, int argc, char *argv[]) ++int client_run(int f_in, int f_out, pid_t pid, int argc, char *argv[]) + { +- struct file_list *flist = NULL; ++ struct file_list *flist = NULL; + int exit_code = 0, exit_code2 = 0; + char *local_name = NULL; + +@@ -1156,55 +1151,18 @@ + send_filter_list(f_out); + if (filesfrom_host) + filesfrom_fd = f_in; ++ + if (write_batch && !am_server) + start_write_batch(f_out); + flist = send_file_list(f_out, argc, argv); + if (DEBUG_GTE(FLIST, 3)) + rprintf(FINFO,"file list sent\n"); ++ + if (protocol_version < 31 && filesfrom_host && protocol_version >= 23) + io_start_multiplex_in(f_in); + + io_flush(NORMAL_FLUSH); +- +- //gettimeofday(&td, NULL); // HPZ +- long sec = 0, usec = 0; +- float time = 0; +- /* +- if((td.tv_usec-tv.tv_usec)<0){ +- sec = td.tv_sec - tv.tv_sec - 1; +- usec = 1000000 +td.tv_usec - tv.tv_usec; +- }else{ +- sec = td.tv_sec - tv.tv_sec; +- usec = td.tv_usec - tv.tv_usec; +- } +- time = (float)sec +(float)usec/1000000; +- printf("BE:%.4f\n", time); // HPZ: Print when end running +- */ +- +- +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "send_files start", logtime.tv_sec, logtime.tv_usec); +- + send_files(f_in, f_out); +- gettimeofday(&td, NULL); // HPZ +- +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "send_files finish", logtime.tv_sec, logtime.tv_usec); +- +- /* +- //printf("%ld.%ld\n", td.tv_sec, td.tv_usec); +- //long sec = 0, usec = 0; +- if((td.tv_usec-tv.tv_usec)<0){ +- sec = td.tv_sec - tv.tv_sec - 1; +- usec = 1000000 +td.tv_usec - tv.tv_usec; +- }else{ +- sec = td.tv_sec - tv.tv_sec; +- usec = td.tv_usec - tv.tv_usec; +- } +- time = (float)sec +(float)usec/1000000; +- printf("AF:%.4f\n", time); // HPZ: Print when end running +- */ +- + io_flush(FULL_FLUSH); + handle_stats(-1); + if (protocol_version >= 24) +@@ -1217,24 +1175,9 @@ + } + output_summary(); + io_flush(FULL_FLUSH); +- //struct timeval tv; +- gettimeofday(&td, NULL); // HPZ +- //long sec = 0, usec = 0; +- if((td.tv_usec-tv.tv_usec)<0){ +- sec = td.tv_sec - tv.tv_sec - 1; +- usec = 1000000 +td.tv_usec - tv.tv_usec; +- }else{ +- sec = td.tv_sec - tv.tv_sec; +- usec = td.tv_usec - tv.tv_usec; +- } +- time = (float)sec +(float)usec/1000000; +- //printf("%.4f,\n", time); // HPZ: Print when end running +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "RSYNC finish", logtime.tv_sec, logtime.tv_usec); +- printf("-----------------------------------\n"); + exit_cleanup(exit_code); +- + } ++ + if (!read_batch) { + if (protocol_version >= 23) + io_start_multiplex_in(f_in); +@@ -1245,6 +1188,7 @@ + } + + send_filter_list(read_batch ? -1 : f_out); ++ + if (filesfrom_fd >= 0) { + start_filesfrom_forwarding(filesfrom_fd); + filesfrom_fd = -1; +@@ -1299,7 +1243,7 @@ + * + * Calls either start_socket_client (for sockets) or do_cmd and + * client_run (for ssh). */ +-static int start_client(int argc, char *argv[]) ++static int start_client(int argc, char *argv[]) + { + char *p, *shell_machine = NULL, *shell_user = NULL; + char **remote_argv; +@@ -1429,9 +1373,8 @@ + } + } + +- if (daemon_over_rsh < 0){ ++ if (daemon_over_rsh < 0) + return start_socket_client(shell_machine, remote_argc, remote_argv, argc, argv); +- } + + if (password_file && !daemon_over_rsh) { + rprintf(FERROR, "The --password-file option may only be " +@@ -1471,11 +1414,8 @@ + if (tmpret < 0) + return tmpret; + } +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "client_run start", logtime.tv_sec, logtime.tv_usec); ++ + ret = client_run(f_in, f_out, pid, argc, argv); +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "client_run finish", logtime.tv_sec, logtime.tv_usec); + + fflush(stdout); + fflush(stderr); +@@ -1581,11 +1521,6 @@ + int ret; + int orig_argc = argc; + char **orig_argv = argv; +- gettimeofday(&tv, NULL); // HPZ +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "RSYNC start", logtime.tv_sec, logtime.tv_usec); +- +- + #ifdef HAVE_SIGACTION + # ifdef HAVE_SIGPROCMASK + sigset_t sigmask; +diff -uNr rsync/match.c rsync-3.1.2/match.c +--- rsync/match.c 2016-10-07 15:03:51.783262780 -0400 ++++ rsync-3.1.2/match.c 2015-08-08 15:47:03.000000000 -0400 +@@ -37,8 +37,6 @@ + static int total_false_alarms; + static int total_hash_hits; + static int total_matches; +-static int64 transfer_time = 0; // HPZ: This is used to detect how many times invoking a match +-static int64 meet_time = 0; // HPZ: This is used to detect how many times it hit the buffer size + + extern struct stats stats; + +@@ -87,12 +85,6 @@ + hash_table[t] = i; + } + } +- /* +- uint32 q; +- for(q=0; q < tablesize; q++){ // HPZ: Print hash table +- if(hash_table[q]!=-1) +- printf("%d:\t%d\n", q, hash_table[q]); +- }*/ + } + + +@@ -116,18 +108,14 @@ + int32 n = (int32)(offset - last_match); /* max value: block_size (int32) */ + int32 j; + +- /* +- //if (DEBUG_GTE(DELTASUM, 2) && i >= 0) { +- if(i >= 0 ){ ++ if (DEBUG_GTE(DELTASUM, 2) && i >= 0) { + rprintf(FINFO, + "match at %s last_match=%s j=%d len=%ld n=%ld\n", + big_num(offset), big_num(last_match), i, + (long)s->sums[i].len, (long)n); + } +- */ + + send_token(f, i, buf, last_match, n, i < 0 ? 0 : s->sums[i].len); +- transfer_time ++; // HPZ + data_transfer += n; + + if (i >= 0) { +@@ -347,13 +335,11 @@ + match. The 3 reads are caused by the + running match, the checksum update and the + literal send. */ +- if (backup >= s->blength+CHUNK_SIZE && end-offset > CHUNK_SIZE){ ++ if (backup >= s->blength+CHUNK_SIZE && end-offset > CHUNK_SIZE) + matched(f, s, buf, offset - s->blength, -2); +- meet_time++; // HPZ +- } + } while (++offset < end); + +- matched(f, s, buf, len, -1); // HPZ : This send only 0-token-literal data ++ matched(f, s, buf, len, -1); + map_ptr(buf, len-1, 1); + } + +@@ -379,8 +365,6 @@ + hash_hits = 0; + matches = 0; + data_transfer = 0; +- struct timeval tv2; // HPZ start +- gettimeofday(&tv2, NULL); // HPZ + + sum_init(checksum_seed); + +@@ -450,21 +434,6 @@ + total_false_alarms += false_alarms; + total_matches += matches; + stats.literal_data += data_transfer; +- +- struct timeval td2; // HPZ +- gettimeofday(&td2, NULL); +- long sec = 0, usec = 0; +- float time = 0; +- if((td2.tv_usec-tv2.tv_usec)<0){ +- sec = td2.tv_sec - tv2.tv_sec - 1; +- usec = 1000000 +td2.tv_usec - tv2.tv_usec; +- }else{ +- sec = td2.tv_sec - tv2.tv_sec; +- usec = td2.tv_usec - tv2.tv_usec; +- } +- time = (float)sec +(float)usec/1000000; +- //printf("%.4f,", time); // HPZ +- + } + + void match_report(void) +@@ -472,10 +441,6 @@ + if (!DEBUG_GTE(DELTASUM, 1)) + return; + +- printf("matched has been invoked %ld times.\n", transfer_time); +- printf("windows size has been hitted %ld times.\n", meet_time); +- +- + rprintf(FINFO, + "total: matches=%d hash_hits=%d false_alarms=%d data=%s\n", + total_matches, total_hash_hits, total_false_alarms, +diff -uNr rsync/pipe.c rsync-3.1.2/pipe.c +--- rsync/pipe.c 2016-10-05 12:41:00.931943173 -0400 ++++ rsync-3.1.2/pipe.c 2015-08-08 15:47:03.000000000 -0400 +@@ -175,7 +175,6 @@ + + *f_in = from_child_pipe[0]; + *f_out = to_child_pipe[1]; +- //printf("f_in:\t%d, f_out:\t%d",*f_in,*f_out); //HPZ : This never changed + + return pid; + } +diff -uNr rsync/proto.h rsync-3.1.2/proto.h +--- rsync/proto.h 2016-10-07 16:25:14.123350335 -0400 ++++ rsync-3.1.2/proto.h 2015-12-21 15:22:53.000000000 -0500 +@@ -253,7 +253,7 @@ + void read_del_stats(int f); + int child_main(int argc, char *argv[]); + void start_server(int f_in, int f_out, int argc, char *argv[]); +-int client_run(int f_in, int f_out, pid_t pid, int argc, char *argv[]) ; ++int client_run(int f_in, int f_out, pid_t pid, int argc, char *argv[]); + void remember_children(UNUSED(int val)); + const char *get_panic_action(void); + int main(int argc,char *argv[]); +@@ -399,7 +399,6 @@ + const char *sum_as_hex(const char *sum); + NORETURN void out_of_memory(const char *str); + NORETURN void overflow_exit(const char *str); +-void print(char *msg); + void free_xattr(stat_x *sxp); + int get_xattr(const char *fname, stat_x *sxp); + int copy_xattrs(const char *source, const char *dest); +diff -uNr rsync/rsync.c rsync-3.1.2/rsync.c +--- rsync/rsync.c 2016-10-07 15:50:15.478230583 -0400 ++++ rsync-3.1.2/rsync.c 2015-08-08 15:47:03.000000000 -0400 +@@ -317,7 +317,6 @@ + + read_loop: + while (1) { +- + ndx = read_ndx(f_in); + + if (ndx >= 0) +@@ -369,7 +368,7 @@ + flist->parent_ndx = ndx; + stop_flist_forward(); + } +- ++ + iflags = protocol_version >= 29 ? read_shortint(f_in) + : ITEM_TRANSFER | ITEM_MISSING_DATA; + +@@ -750,4 +749,3 @@ + : am_receiver ? "receiver" + : "Receiver"; /* pre-forked receiver */ + } +- +diff -uNr rsync/rsync.h rsync-3.1.2/rsync.h +--- rsync/rsync.h 2016-10-07 16:24:23.365209158 -0400 ++++ rsync-3.1.2/rsync.h 2015-08-08 15:47:03.000000000 -0400 +@@ -705,8 +705,6 @@ + const char basename[1]; /* The basename (AKA filename) follows */ + }; + +-struct timeval logtime; // HPZ: This is very useful log timestamp +- + extern int file_extra_cnt; + extern int inc_recurse; + extern int uid_ndx; +diff -uNr rsync/sender.c rsync-3.1.2/sender.c +--- rsync/sender.c 2016-10-07 15:31:56.633777127 -0400 ++++ rsync-3.1.2/sender.c 2015-09-07 13:07:17.000000000 -0400 +@@ -206,8 +206,6 @@ + if (DEBUG_GTE(SEND, 1)) + rprintf(FINFO, "send_files starting\n"); + +- int tmpflag=0; // HPZ: I'm so stupid +- struct timeval tv3, td3; + while (1) { + if (inc_recurse) { + send_extra_file_list(f_out, MIN_FILECNT_LOOKAHEAD); +@@ -215,16 +213,9 @@ + } + + /* This call also sets cur_flist. */ +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "rw_ndx start", logtime.tv_sec, logtime.tv_usec); + ndx = read_ndx_and_attrs(f_in, f_out, &iflags, &fnamecmp_type, +- xname, &xlen); // HPZ: Figure out this! +- ++ xname, &xlen); + extra_flist_sending_enabled = False; +- //printf("ndx:%d\n", ndx); +- /* +- From here +- */ + + if (ndx == NDX_DONE) { + if (!am_server && INFO_GTE(PROGRESS, 2) && cur_flist) { +@@ -241,24 +232,16 @@ + continue; + } + } +- if (++phase > max_phase){ ++ if (++phase > max_phase) + break; +- } + if (DEBUG_GTE(SEND, 1)) + rprintf(FINFO, "send_files phase=%d\n", phase); +- write_ndx(f_out, NDX_DONE); // HPZ: Due to phase, it will run twice. so WHAT's WRITE_NDX ++ write_ndx(f_out, NDX_DONE); + continue; +- } +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "rw_ndx finish", logtime.tv_sec, logtime.tv_usec); +- +- /* +- to here +- */ +- ++ } ++ + if (inc_recurse) + send_extra_file_list(f_out, MIN_FILECNT_LOOKAHEAD); +- // printf("****************************%d\n",ndx); + + if (ndx - cur_flist->ndx_start >= 0) + file = cur_flist->files[ndx - cur_flist->ndx_start]; +@@ -273,14 +256,16 @@ + if (!change_pathname(file, NULL, 0)) + continue; + f_name(file, fname); ++ + if (DEBUG_GTE(SEND, 1)) + rprintf(FINFO, "send_files(%d, %s%s%s)\n", ndx, path,slash,fname); +- ++ + #ifdef SUPPORT_XATTRS + if (preserve_xattrs && iflags & ITEM_REPORT_XATTR && do_xfers + && !(want_xattr_optim && BITS_SET(iflags, ITEM_XNAME_FOLLOWS|ITEM_LOCAL_CHANGE))) + recv_xattr_request(file, f_in); + #endif ++ + if (!(iflags & ITEM_TRANSFER)) { + maybe_log_item(file, iflags, itemizing, xname); + write_ndx_and_attrs(f_out, ndx, iflags, fname, file, +@@ -308,6 +293,7 @@ + who_am_i()); + exit_cleanup(RERR_PROTOCOL); + } ++ + if (file->flags & FLAG_FILE_SENT) { + if (csum_length == SHORT_SUM_LENGTH) { + /* For inplace: redo phase turns off the backup +@@ -325,6 +311,7 @@ + if (iflags & ITEM_IS_NEW) + stats.created_files++; + } ++ + updating_basis_file = inplace && (protocol_version >= 29 + ? fnamecmp_type == FNAMECMP_FNAME : make_backups <= 0); + +@@ -341,13 +328,13 @@ + fnamecmp_type, xname, xlen); + continue; + } +- ++ + if (!(s = receive_sums(f_in))) { + io_error |= IOERR_GENERAL; + rprintf(FERROR_XFER, "receive_sums failed\n"); + exit_cleanup(RERR_PROTOCOL); + } +- ++ + fd = do_open(fname, O_RDONLY, 0); + if (fd == -1) { + if (errno == ENOENT) { +@@ -368,6 +355,7 @@ + send_msg_int(MSG_NO_SEND, ndx); + continue; + } ++ + /* map the local file */ + if (do_fstat(fd, &st) != 0) { + io_error |= IOERR_GENERAL; +@@ -383,7 +371,7 @@ + } else + mbuf = NULL; + +- if (DEBUG_GTE(DELTASUM, 2)){ ++ if (DEBUG_GTE(DELTASUM, 2)) { + rprintf(FINFO, "send_files mapped %s%s%s of size %s\n", + path,slash,fname, big_num(st.st_size)); + } +@@ -391,6 +379,7 @@ + write_ndx_and_attrs(f_out, ndx, iflags, fname, file, + fnamecmp_type, xname, xlen); + write_sum_head(f_xfer, s); ++ + if (DEBUG_GTE(DELTASUM, 2)) + rprintf(FINFO, "calling match_sums %s%s%s\n", path,slash,fname); + +@@ -400,15 +389,8 @@ + rprintf(FCLIENT, "%s\n", fname); + + set_compression(fname); +- +- +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "match_sum start", logtime.tv_sec, logtime.tv_usec); +- match_sums(f_xfer, s, mbuf, st.st_size); +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "match_sum finish", logtime.tv_sec, logtime.tv_usec); +- match_report(); + ++ match_sums(f_xfer, s, mbuf, st.st_size); + if (INFO_GTE(PROGRESS, 1)) + end_progress(st.st_size); + +@@ -424,6 +406,7 @@ + } + } + close(fd); ++ + free_sums(s); + + if (DEBUG_GTE(SEND, 1)) +@@ -432,7 +415,6 @@ + /* Flag that we actually sent this entry. */ + file->flags |= FLAG_FILE_SENT; + } +- + if (make_backups < 0) + make_backups = -make_backups; + +@@ -442,7 +424,7 @@ + if (DEBUG_GTE(SEND, 1)) + rprintf(FINFO, "send files finished\n"); + +- // Match_report suppose to be here. ++ match_report(); + + write_ndx(f_out, NDX_DONE); + } +diff -uNr rsync/util2.c rsync-3.1.2/util2.c +--- rsync/util2.c 2016-10-07 16:25:30.710777685 -0400 ++++ rsync-3.1.2/util2.c 2015-08-08 15:47:03.000000000 -0400 +@@ -111,8 +111,3 @@ + rprintf(FERROR, "ERROR: buffer overflow in %s [%s]\n", str, who_am_i()); + exit_cleanup(RERR_MALLOC); + } +- +-void print(char *msg) +-{ +- printf("%s\n", msg); +- } +diff -uNr rsync/util.c rsync-3.1.2/util.c +--- rsync/util.c 2016-10-07 16:16:49.961215701 -0400 ++++ rsync-3.1.2/util.c 2015-12-21 13:54:02.000000000 -0500 +@@ -1645,4 +1645,3 @@ + } + return (char*)lp->items + (lp->count++ * item_size); + } +- diff --git a/_site/static/2016-10/rsync/small2Big_change_at_anyplace.sh b/_site/static/2016-10/rsync/small2Big_change_at_anyplace.sh new file mode 100644 index 0000000..748db07 --- /dev/null +++ b/_site/static/2016-10/rsync/small2Big_change_at_anyplace.sh @@ -0,0 +1,31 @@ +FILESIZE=8192 +let CHANGESIZE=1024 + +UNI_TARGET='phao3@bingsuns.binghamton.edu:~' +UNI_PASSWD=Pengzhan2015USA + +LAB_TARGET='moslab@10.42.0.1:~/Desktop/server' +LAB_PASSWD=1 + +LOC_TARGET='..' + +temp=1 +for i in `seq 1 14`; +do + dd if=/dev/urandom of=$FILESIZE.dat bs=8192 count=$temp status=none + sshpass -p $LAB_PASSWD ./rsync $FILESIZE.dat $LAB_TARGET + # ./rsync -v -v $FILESIZE.dat $LOC_TARGET --no-whole-file + # echo $FILESIZE + dd if=/dev/urandom of=$CHANGESIZE.dat bs=$CHANGESIZE count=1 status=none + # cat $CHANGESIZE.dat $FILESIZE.dat > temp.dat + python ../addbyte.py $FILESIZE.dat $CHANGESIZE.dat temp.dat + # cp -f $FILESIZE.dat temp.dat + sshpass -p $LAB_PASSWD ./rsync temp.dat $LAB_TARGET/$FILESIZE.dat + # ./rsync temp.dat $LOC_TARGET/$FILESIZE.dat --no-whole-file + + + let temp=$temp*2 + let FILESIZE=$FILESIZE*2 +done +date + diff --git a/_site/static/2016-10/rsync/small2Big_change_at_begin.sh b/_site/static/2016-10/rsync/small2Big_change_at_begin.sh new file mode 100644 index 0000000..96233b2 --- /dev/null +++ b/_site/static/2016-10/rsync/small2Big_change_at_begin.sh @@ -0,0 +1,29 @@ +FILESIZE=8192 +let CHANGESIZE=1024 + +UNI_TARGET='phao3@bingsuns.binghamton.edu:~' +UNI_PASSWD=Pengzhan2015USA + +LAB_TARGET='moslab@10.42.0.1:~/Desktop/server' +LAB_PASSWD=1 + +LOC_TARGET='..' + +temp=1 +for i in `seq 1 14`; +do + dd if=/dev/urandom of=$FILESIZE.dat bs=8192 count=$temp status=none + sshpass -p $LAB_PASSWD ./rsync $FILESIZE.dat $LAB_TARGET + # ./rsync $FILESIZE.dat $LOC_TARGET --no-whole-file + #echo $FILESIZE + dd if=/dev/urandom of=$CHANGESIZE.dat bs=$CHANGESIZE count=1 status=none + cat $CHANGESIZE.dat $FILESIZE.dat > temp.dat + sshpass -p $LAB_PASSWD ./rsync temp.dat $LAB_TARGET/$FILESIZE.dat + # ./rsync temp.dat $LOC_TARGET/$FILESIZE.dat --no-whole-file + + + let temp=$temp*2 + let FILESIZE=$FILESIZE*2 +done +date + diff --git a/_site/static/2016-10/rsync/small2Big_change_at_last.sh b/_site/static/2016-10/rsync/small2Big_change_at_last.sh new file mode 100644 index 0000000..a10b38b --- /dev/null +++ b/_site/static/2016-10/rsync/small2Big_change_at_last.sh @@ -0,0 +1,23 @@ +FILESIZE=134217728 +let CHANGESIZE=16777216 + +UNI_TARGET='phao3@bingsuns.binghamton.edu:~' +UNI_PASSWD=Pengzhan2015USA + +LAB_TARGET='moslab@10.42.0.1:~/Desktop/server' +LAB_PASSWD=1 +temp=16384 +for i in `seq 1 4`; +do + dd if=/dev/urandom of=$FILESIZE.dat bs=8192 count=$temp status=none + sshpass -p $LAB_PASSWD ./rsync $FILESIZE.dat $LAB_TARGET + #echo $FILESIZE + dd if=/dev/urandom of=$CHANGESIZE.dat bs=1024 count=$temp status=none + cat $FILESIZE.dat $CHANGESIZE.dat > temp.dat + sshpass -p $LAB_PASSWD ./rsync temp.dat $LAB_TARGET/$FILESIZE.dat + + let temp=$temp*2 + let FILESIZE=$FILESIZE*2 +done +date + diff --git a/_site/static/rsync-3.1.2-time.patch b/_site/static/rsync-3.1.2-time.patch new file mode 100644 index 0000000..fd96f21 --- /dev/null +++ b/_site/static/rsync-3.1.2-time.patch @@ -0,0 +1,585 @@ +diff -uNr rsync/flist.c rsync-3.1.2/flist.c +--- rsync/flist.c 2016-09-17 00:14:38.448855376 -0400 ++++ rsync-3.1.2/flist.c 2015-08-08 15:47:03.000000000 -0400 +@@ -2431,7 +2431,7 @@ + if (DEBUG_GTE(FLIST, 3)) + rprintf(FINFO, "[%s] flist_eof=1\n", who_am_i()); + } +- //printf("how long build flist takes %ld\n", stats.flist_buildtime); // HPZ: I should not add this trace code, TODO: Delete ++ + return flist; + } + +diff -uNr rsync/main.c rsync-3.1.2/main.c +--- rsync/main.c 2016-10-07 15:10:04.094266860 -0400 ++++ rsync-3.1.2/main.c 2015-08-24 00:58:18.000000000 -0400 +@@ -23,12 +23,10 @@ + #include "rsync.h" + #include "inums.h" + #include "io.h" +-#include //HPZ: + #if defined CONFIG_LOCALE && defined HAVE_LOCALE_H + #include + #endif + +- + extern int dry_run; + extern int list_only; + extern int io_timeout; +@@ -106,8 +104,6 @@ + mode_t orig_umask = 0; + int batch_gen_fd = -1; + int sender_keeps_checksum = 0; +-struct timeval tv, td; // HPZ: only solution +- + + /* There's probably never more than at most 2 outstanding child processes, + * but set it higher, just in case. */ +@@ -353,7 +349,7 @@ + write_batch < 0 ? " (BATCH ONLY)" : dry_run ? " (DRY RUN)" : ""); + } + +- fflush(stdout); ++ fflush(stdout); + fflush(stderr); + } + +@@ -527,12 +523,11 @@ + batch_gen_fd = from_gen_pipe[0]; + *f_out_p = from_gen_pipe[1]; + *f_in_p = batch_fd; +- //printf("1***\n"); + pid = (pid_t)-1; /* no child pid */ + #ifdef ICONV_CONST + setup_iconv(); + #endif +- } else if (local_server) { // HPZ: Local Branch ++ } else if (local_server) { + /* If the user didn't request --[no-]whole-file, force + * it on, but only if we're not batch processing. */ + if (whole_file < 0 && !write_batch) +@@ -543,7 +538,7 @@ + setup_iconv(); + #endif + } else { +- pid = piped_child(args, f_in_p, f_out_p); // HPZ: Remote Branch ++ pid = piped_child(args, f_in_p, f_out_p); + #ifdef ICONV_CONST + setup_iconv(); + #endif +@@ -1112,9 +1107,9 @@ + + /* This is called once the connection has been negotiated. It is used + * for rsyncd, remote-shell, and local connections. */ +-int client_run(int f_in, int f_out, pid_t pid, int argc, char *argv[]) ++int client_run(int f_in, int f_out, pid_t pid, int argc, char *argv[]) + { +- struct file_list *flist = NULL; ++ struct file_list *flist = NULL; + int exit_code = 0, exit_code2 = 0; + char *local_name = NULL; + +@@ -1156,55 +1151,18 @@ + send_filter_list(f_out); + if (filesfrom_host) + filesfrom_fd = f_in; ++ + if (write_batch && !am_server) + start_write_batch(f_out); + flist = send_file_list(f_out, argc, argv); + if (DEBUG_GTE(FLIST, 3)) + rprintf(FINFO,"file list sent\n"); ++ + if (protocol_version < 31 && filesfrom_host && protocol_version >= 23) + io_start_multiplex_in(f_in); + + io_flush(NORMAL_FLUSH); +- +- //gettimeofday(&td, NULL); // HPZ +- long sec = 0, usec = 0; +- float time = 0; +- /* +- if((td.tv_usec-tv.tv_usec)<0){ +- sec = td.tv_sec - tv.tv_sec - 1; +- usec = 1000000 +td.tv_usec - tv.tv_usec; +- }else{ +- sec = td.tv_sec - tv.tv_sec; +- usec = td.tv_usec - tv.tv_usec; +- } +- time = (float)sec +(float)usec/1000000; +- printf("BE:%.4f\n", time); // HPZ: Print when end running +- */ +- +- +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "send_files start", logtime.tv_sec, logtime.tv_usec); +- + send_files(f_in, f_out); +- gettimeofday(&td, NULL); // HPZ +- +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "send_files finish", logtime.tv_sec, logtime.tv_usec); +- +- /* +- //printf("%ld.%ld\n", td.tv_sec, td.tv_usec); +- //long sec = 0, usec = 0; +- if((td.tv_usec-tv.tv_usec)<0){ +- sec = td.tv_sec - tv.tv_sec - 1; +- usec = 1000000 +td.tv_usec - tv.tv_usec; +- }else{ +- sec = td.tv_sec - tv.tv_sec; +- usec = td.tv_usec - tv.tv_usec; +- } +- time = (float)sec +(float)usec/1000000; +- printf("AF:%.4f\n", time); // HPZ: Print when end running +- */ +- + io_flush(FULL_FLUSH); + handle_stats(-1); + if (protocol_version >= 24) +@@ -1217,24 +1175,9 @@ + } + output_summary(); + io_flush(FULL_FLUSH); +- //struct timeval tv; +- gettimeofday(&td, NULL); // HPZ +- //long sec = 0, usec = 0; +- if((td.tv_usec-tv.tv_usec)<0){ +- sec = td.tv_sec - tv.tv_sec - 1; +- usec = 1000000 +td.tv_usec - tv.tv_usec; +- }else{ +- sec = td.tv_sec - tv.tv_sec; +- usec = td.tv_usec - tv.tv_usec; +- } +- time = (float)sec +(float)usec/1000000; +- //printf("%.4f,\n", time); // HPZ: Print when end running +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "RSYNC finish", logtime.tv_sec, logtime.tv_usec); +- printf("-----------------------------------\n"); + exit_cleanup(exit_code); +- + } ++ + if (!read_batch) { + if (protocol_version >= 23) + io_start_multiplex_in(f_in); +@@ -1245,6 +1188,7 @@ + } + + send_filter_list(read_batch ? -1 : f_out); ++ + if (filesfrom_fd >= 0) { + start_filesfrom_forwarding(filesfrom_fd); + filesfrom_fd = -1; +@@ -1299,7 +1243,7 @@ + * + * Calls either start_socket_client (for sockets) or do_cmd and + * client_run (for ssh). */ +-static int start_client(int argc, char *argv[]) ++static int start_client(int argc, char *argv[]) + { + char *p, *shell_machine = NULL, *shell_user = NULL; + char **remote_argv; +@@ -1429,9 +1373,8 @@ + } + } + +- if (daemon_over_rsh < 0){ ++ if (daemon_over_rsh < 0) + return start_socket_client(shell_machine, remote_argc, remote_argv, argc, argv); +- } + + if (password_file && !daemon_over_rsh) { + rprintf(FERROR, "The --password-file option may only be " +@@ -1471,11 +1414,8 @@ + if (tmpret < 0) + return tmpret; + } +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "client_run start", logtime.tv_sec, logtime.tv_usec); ++ + ret = client_run(f_in, f_out, pid, argc, argv); +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "client_run finish", logtime.tv_sec, logtime.tv_usec); + + fflush(stdout); + fflush(stderr); +@@ -1581,11 +1521,6 @@ + int ret; + int orig_argc = argc; + char **orig_argv = argv; +- gettimeofday(&tv, NULL); // HPZ +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "RSYNC start", logtime.tv_sec, logtime.tv_usec); +- +- + #ifdef HAVE_SIGACTION + # ifdef HAVE_SIGPROCMASK + sigset_t sigmask; +diff -uNr rsync/match.c rsync-3.1.2/match.c +--- rsync/match.c 2016-10-07 15:03:51.783262780 -0400 ++++ rsync-3.1.2/match.c 2015-08-08 15:47:03.000000000 -0400 +@@ -37,8 +37,6 @@ + static int total_false_alarms; + static int total_hash_hits; + static int total_matches; +-static int64 transfer_time = 0; // HPZ: This is used to detect how many times invoking a match +-static int64 meet_time = 0; // HPZ: This is used to detect how many times it hit the buffer size + + extern struct stats stats; + +@@ -87,12 +85,6 @@ + hash_table[t] = i; + } + } +- /* +- uint32 q; +- for(q=0; q < tablesize; q++){ // HPZ: Print hash table +- if(hash_table[q]!=-1) +- printf("%d:\t%d\n", q, hash_table[q]); +- }*/ + } + + +@@ -116,18 +108,14 @@ + int32 n = (int32)(offset - last_match); /* max value: block_size (int32) */ + int32 j; + +- /* +- //if (DEBUG_GTE(DELTASUM, 2) && i >= 0) { +- if(i >= 0 ){ ++ if (DEBUG_GTE(DELTASUM, 2) && i >= 0) { + rprintf(FINFO, + "match at %s last_match=%s j=%d len=%ld n=%ld\n", + big_num(offset), big_num(last_match), i, + (long)s->sums[i].len, (long)n); + } +- */ + + send_token(f, i, buf, last_match, n, i < 0 ? 0 : s->sums[i].len); +- transfer_time ++; // HPZ + data_transfer += n; + + if (i >= 0) { +@@ -347,13 +335,11 @@ + match. The 3 reads are caused by the + running match, the checksum update and the + literal send. */ +- if (backup >= s->blength+CHUNK_SIZE && end-offset > CHUNK_SIZE){ ++ if (backup >= s->blength+CHUNK_SIZE && end-offset > CHUNK_SIZE) + matched(f, s, buf, offset - s->blength, -2); +- meet_time++; // HPZ +- } + } while (++offset < end); + +- matched(f, s, buf, len, -1); // HPZ : This send only 0-token-literal data ++ matched(f, s, buf, len, -1); + map_ptr(buf, len-1, 1); + } + +@@ -379,8 +365,6 @@ + hash_hits = 0; + matches = 0; + data_transfer = 0; +- struct timeval tv2; // HPZ start +- gettimeofday(&tv2, NULL); // HPZ + + sum_init(checksum_seed); + +@@ -450,21 +434,6 @@ + total_false_alarms += false_alarms; + total_matches += matches; + stats.literal_data += data_transfer; +- +- struct timeval td2; // HPZ +- gettimeofday(&td2, NULL); +- long sec = 0, usec = 0; +- float time = 0; +- if((td2.tv_usec-tv2.tv_usec)<0){ +- sec = td2.tv_sec - tv2.tv_sec - 1; +- usec = 1000000 +td2.tv_usec - tv2.tv_usec; +- }else{ +- sec = td2.tv_sec - tv2.tv_sec; +- usec = td2.tv_usec - tv2.tv_usec; +- } +- time = (float)sec +(float)usec/1000000; +- //printf("%.4f,", time); // HPZ +- + } + + void match_report(void) +@@ -472,10 +441,6 @@ + if (!DEBUG_GTE(DELTASUM, 1)) + return; + +- printf("matched has been invoked %ld times.\n", transfer_time); +- printf("windows size has been hitted %ld times.\n", meet_time); +- +- + rprintf(FINFO, + "total: matches=%d hash_hits=%d false_alarms=%d data=%s\n", + total_matches, total_hash_hits, total_false_alarms, +diff -uNr rsync/pipe.c rsync-3.1.2/pipe.c +--- rsync/pipe.c 2016-10-05 12:41:00.931943173 -0400 ++++ rsync-3.1.2/pipe.c 2015-08-08 15:47:03.000000000 -0400 +@@ -175,7 +175,6 @@ + + *f_in = from_child_pipe[0]; + *f_out = to_child_pipe[1]; +- //printf("f_in:\t%d, f_out:\t%d",*f_in,*f_out); //HPZ : This never changed + + return pid; + } +diff -uNr rsync/proto.h rsync-3.1.2/proto.h +--- rsync/proto.h 2016-10-07 16:25:14.123350335 -0400 ++++ rsync-3.1.2/proto.h 2015-12-21 15:22:53.000000000 -0500 +@@ -253,7 +253,7 @@ + void read_del_stats(int f); + int child_main(int argc, char *argv[]); + void start_server(int f_in, int f_out, int argc, char *argv[]); +-int client_run(int f_in, int f_out, pid_t pid, int argc, char *argv[]) ; ++int client_run(int f_in, int f_out, pid_t pid, int argc, char *argv[]); + void remember_children(UNUSED(int val)); + const char *get_panic_action(void); + int main(int argc,char *argv[]); +@@ -399,7 +399,6 @@ + const char *sum_as_hex(const char *sum); + NORETURN void out_of_memory(const char *str); + NORETURN void overflow_exit(const char *str); +-void print(char *msg); + void free_xattr(stat_x *sxp); + int get_xattr(const char *fname, stat_x *sxp); + int copy_xattrs(const char *source, const char *dest); +diff -uNr rsync/rsync.c rsync-3.1.2/rsync.c +--- rsync/rsync.c 2016-10-07 15:50:15.478230583 -0400 ++++ rsync-3.1.2/rsync.c 2015-08-08 15:47:03.000000000 -0400 +@@ -317,7 +317,6 @@ + + read_loop: + while (1) { +- + ndx = read_ndx(f_in); + + if (ndx >= 0) +@@ -369,7 +368,7 @@ + flist->parent_ndx = ndx; + stop_flist_forward(); + } +- ++ + iflags = protocol_version >= 29 ? read_shortint(f_in) + : ITEM_TRANSFER | ITEM_MISSING_DATA; + +@@ -750,4 +749,3 @@ + : am_receiver ? "receiver" + : "Receiver"; /* pre-forked receiver */ + } +- +diff -uNr rsync/rsync.h rsync-3.1.2/rsync.h +--- rsync/rsync.h 2016-10-07 16:24:23.365209158 -0400 ++++ rsync-3.1.2/rsync.h 2015-08-08 15:47:03.000000000 -0400 +@@ -705,8 +705,6 @@ + const char basename[1]; /* The basename (AKA filename) follows */ + }; + +-struct timeval logtime; // HPZ: This is very useful log timestamp +- + extern int file_extra_cnt; + extern int inc_recurse; + extern int uid_ndx; +diff -uNr rsync/sender.c rsync-3.1.2/sender.c +--- rsync/sender.c 2016-10-07 15:31:56.633777127 -0400 ++++ rsync-3.1.2/sender.c 2015-09-07 13:07:17.000000000 -0400 +@@ -206,8 +206,6 @@ + if (DEBUG_GTE(SEND, 1)) + rprintf(FINFO, "send_files starting\n"); + +- int tmpflag=0; // HPZ: I'm so stupid +- struct timeval tv3, td3; + while (1) { + if (inc_recurse) { + send_extra_file_list(f_out, MIN_FILECNT_LOOKAHEAD); +@@ -215,16 +213,9 @@ + } + + /* This call also sets cur_flist. */ +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "rw_ndx start", logtime.tv_sec, logtime.tv_usec); + ndx = read_ndx_and_attrs(f_in, f_out, &iflags, &fnamecmp_type, +- xname, &xlen); // HPZ: Figure out this! +- ++ xname, &xlen); + extra_flist_sending_enabled = False; +- //printf("ndx:%d\n", ndx); +- /* +- From here +- */ + + if (ndx == NDX_DONE) { + if (!am_server && INFO_GTE(PROGRESS, 2) && cur_flist) { +@@ -241,24 +232,16 @@ + continue; + } + } +- if (++phase > max_phase){ ++ if (++phase > max_phase) + break; +- } + if (DEBUG_GTE(SEND, 1)) + rprintf(FINFO, "send_files phase=%d\n", phase); +- write_ndx(f_out, NDX_DONE); // HPZ: Due to phase, it will run twice. so WHAT's WRITE_NDX ++ write_ndx(f_out, NDX_DONE); + continue; +- } +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "rw_ndx finish", logtime.tv_sec, logtime.tv_usec); +- +- /* +- to here +- */ +- ++ } ++ + if (inc_recurse) + send_extra_file_list(f_out, MIN_FILECNT_LOOKAHEAD); +- // printf("****************************%d\n",ndx); + + if (ndx - cur_flist->ndx_start >= 0) + file = cur_flist->files[ndx - cur_flist->ndx_start]; +@@ -273,14 +256,16 @@ + if (!change_pathname(file, NULL, 0)) + continue; + f_name(file, fname); ++ + if (DEBUG_GTE(SEND, 1)) + rprintf(FINFO, "send_files(%d, %s%s%s)\n", ndx, path,slash,fname); +- ++ + #ifdef SUPPORT_XATTRS + if (preserve_xattrs && iflags & ITEM_REPORT_XATTR && do_xfers + && !(want_xattr_optim && BITS_SET(iflags, ITEM_XNAME_FOLLOWS|ITEM_LOCAL_CHANGE))) + recv_xattr_request(file, f_in); + #endif ++ + if (!(iflags & ITEM_TRANSFER)) { + maybe_log_item(file, iflags, itemizing, xname); + write_ndx_and_attrs(f_out, ndx, iflags, fname, file, +@@ -308,6 +293,7 @@ + who_am_i()); + exit_cleanup(RERR_PROTOCOL); + } ++ + if (file->flags & FLAG_FILE_SENT) { + if (csum_length == SHORT_SUM_LENGTH) { + /* For inplace: redo phase turns off the backup +@@ -325,6 +311,7 @@ + if (iflags & ITEM_IS_NEW) + stats.created_files++; + } ++ + updating_basis_file = inplace && (protocol_version >= 29 + ? fnamecmp_type == FNAMECMP_FNAME : make_backups <= 0); + +@@ -341,13 +328,13 @@ + fnamecmp_type, xname, xlen); + continue; + } +- ++ + if (!(s = receive_sums(f_in))) { + io_error |= IOERR_GENERAL; + rprintf(FERROR_XFER, "receive_sums failed\n"); + exit_cleanup(RERR_PROTOCOL); + } +- ++ + fd = do_open(fname, O_RDONLY, 0); + if (fd == -1) { + if (errno == ENOENT) { +@@ -368,6 +355,7 @@ + send_msg_int(MSG_NO_SEND, ndx); + continue; + } ++ + /* map the local file */ + if (do_fstat(fd, &st) != 0) { + io_error |= IOERR_GENERAL; +@@ -383,7 +371,7 @@ + } else + mbuf = NULL; + +- if (DEBUG_GTE(DELTASUM, 2)){ ++ if (DEBUG_GTE(DELTASUM, 2)) { + rprintf(FINFO, "send_files mapped %s%s%s of size %s\n", + path,slash,fname, big_num(st.st_size)); + } +@@ -391,6 +379,7 @@ + write_ndx_and_attrs(f_out, ndx, iflags, fname, file, + fnamecmp_type, xname, xlen); + write_sum_head(f_xfer, s); ++ + if (DEBUG_GTE(DELTASUM, 2)) + rprintf(FINFO, "calling match_sums %s%s%s\n", path,slash,fname); + +@@ -400,15 +389,8 @@ + rprintf(FCLIENT, "%s\n", fname); + + set_compression(fname); +- +- +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "match_sum start", logtime.tv_sec, logtime.tv_usec); +- match_sums(f_xfer, s, mbuf, st.st_size); +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "match_sum finish", logtime.tv_sec, logtime.tv_usec); +- match_report(); + ++ match_sums(f_xfer, s, mbuf, st.st_size); + if (INFO_GTE(PROGRESS, 1)) + end_progress(st.st_size); + +@@ -424,6 +406,7 @@ + } + } + close(fd); ++ + free_sums(s); + + if (DEBUG_GTE(SEND, 1)) +@@ -432,7 +415,6 @@ + /* Flag that we actually sent this entry. */ + file->flags |= FLAG_FILE_SENT; + } +- + if (make_backups < 0) + make_backups = -make_backups; + +@@ -442,7 +424,7 @@ + if (DEBUG_GTE(SEND, 1)) + rprintf(FINFO, "send files finished\n"); + +- // Match_report suppose to be here. ++ match_report(); + + write_ndx(f_out, NDX_DONE); + } +diff -uNr rsync/util2.c rsync-3.1.2/util2.c +--- rsync/util2.c 2016-10-07 16:25:30.710777685 -0400 ++++ rsync-3.1.2/util2.c 2015-08-08 15:47:03.000000000 -0400 +@@ -111,8 +111,3 @@ + rprintf(FERROR, "ERROR: buffer overflow in %s [%s]\n", str, who_am_i()); + exit_cleanup(RERR_MALLOC); + } +- +-void print(char *msg) +-{ +- printf("%s\n", msg); +- } +diff -uNr rsync/util.c rsync-3.1.2/util.c +--- rsync/util.c 2016-10-07 16:16:49.961215701 -0400 ++++ rsync-3.1.2/util.c 2015-12-21 13:54:02.000000000 -0500 +@@ -1645,4 +1645,3 @@ + } + return (char*)lp->items + (lp->count++ * item_size); + } +- diff --git a/about.md b/about.md index 4ccc79a..f0e1d59 100644 --- a/about.md +++ b/about.md @@ -6,7 +6,15 @@ public: true --- Pengzhan Hao -============ -Email:haopengzhan@gmail.com -Address:Binghamton University -Phone: (607)-II-III-II-83-VII-II +------------ +- Research Assistant in MOSLab +- Master Student of SUNY at Binghamton + +------------ +- Email: phao3 at binghamton dot edu + +- MOSLab, H-6 Engineering Building, Binghamton University +4400 Vestal Parkway East +Binghamton, NY 13902 + +----------- diff --git a/static/2016-10/rsync/addbyte.py b/static/2016-10/rsync/addbyte.py new file mode 100644 index 0000000..5c156f8 --- /dev/null +++ b/static/2016-10/rsync/addbyte.py @@ -0,0 +1,57 @@ +import sys +from subprocess import call +import os +import time +import math +import random + + + +# print time.localtime() +hostfile = open(sys.argv[1],'r') +# print "cp -f "+sys.argv[1]+" "+sys.argv[3] +# call(["cp","-f",sys.argv[1],sys.argv[3]]) +guestfile = open(sys.argv[2],'r') +targetfile = open(sys.argv[3],'w+') +hostsize = os.path.getsize(sys.argv[1]) +guestsize = os.path.getsize(sys.argv[2]) + +# print hostsize +""" +print mean +if mean!=0: + for i in range(0,guestsize): + guestfile.seek(i) + target.seek(mean*i) + #print str(mean*i)+":"+target.read(1)+"\t"+str(i)+":"+guestfile.read(1) + temp = target.read(1)+guestfile.read(1) + target.seek(mean*i) + target.write(temp) + +print guestfile,target +""" +mean = 1 +c_sz = [12, 24,47,94,188,375,729,1024,1449,2048,2897,4096,5794,8192] +ndx = int(math.log(float(hostsize)/8192,2)) +dist = int(float(hostsize)/c_sz[ndx] + 0.5) +# print dist + +if mean!=0: + guest = guestfile.read() + target = hostfile.read() + byte_ndx = 0 + flt_byte = 0 + for i in range(0,c_sz[ndx]): + # loc = random.randint(i*dist+flt_byte, (i+1)*dist+flt_byte) + loc = 10+i*dist+flt_byte + # print loc, guest[byte_ndx] + target = target[:loc]+guest[byte_ndx]+target[loc:] + flt_byte += 1 + byte_ndx = (byte_ndx+1)%guestsize +target = target[:-3]+"X"+target[-3:] + +targetfile.write(target) +hostfile.close() +guestfile.close() +targetfile.close() +# print time.localtime() diff --git a/static/2016-10/rsync/rsync-3.1.2-time.patch b/static/2016-10/rsync/rsync-3.1.2-time.patch new file mode 100644 index 0000000..fd96f21 --- /dev/null +++ b/static/2016-10/rsync/rsync-3.1.2-time.patch @@ -0,0 +1,585 @@ +diff -uNr rsync/flist.c rsync-3.1.2/flist.c +--- rsync/flist.c 2016-09-17 00:14:38.448855376 -0400 ++++ rsync-3.1.2/flist.c 2015-08-08 15:47:03.000000000 -0400 +@@ -2431,7 +2431,7 @@ + if (DEBUG_GTE(FLIST, 3)) + rprintf(FINFO, "[%s] flist_eof=1\n", who_am_i()); + } +- //printf("how long build flist takes %ld\n", stats.flist_buildtime); // HPZ: I should not add this trace code, TODO: Delete ++ + return flist; + } + +diff -uNr rsync/main.c rsync-3.1.2/main.c +--- rsync/main.c 2016-10-07 15:10:04.094266860 -0400 ++++ rsync-3.1.2/main.c 2015-08-24 00:58:18.000000000 -0400 +@@ -23,12 +23,10 @@ + #include "rsync.h" + #include "inums.h" + #include "io.h" +-#include //HPZ: + #if defined CONFIG_LOCALE && defined HAVE_LOCALE_H + #include + #endif + +- + extern int dry_run; + extern int list_only; + extern int io_timeout; +@@ -106,8 +104,6 @@ + mode_t orig_umask = 0; + int batch_gen_fd = -1; + int sender_keeps_checksum = 0; +-struct timeval tv, td; // HPZ: only solution +- + + /* There's probably never more than at most 2 outstanding child processes, + * but set it higher, just in case. */ +@@ -353,7 +349,7 @@ + write_batch < 0 ? " (BATCH ONLY)" : dry_run ? " (DRY RUN)" : ""); + } + +- fflush(stdout); ++ fflush(stdout); + fflush(stderr); + } + +@@ -527,12 +523,11 @@ + batch_gen_fd = from_gen_pipe[0]; + *f_out_p = from_gen_pipe[1]; + *f_in_p = batch_fd; +- //printf("1***\n"); + pid = (pid_t)-1; /* no child pid */ + #ifdef ICONV_CONST + setup_iconv(); + #endif +- } else if (local_server) { // HPZ: Local Branch ++ } else if (local_server) { + /* If the user didn't request --[no-]whole-file, force + * it on, but only if we're not batch processing. */ + if (whole_file < 0 && !write_batch) +@@ -543,7 +538,7 @@ + setup_iconv(); + #endif + } else { +- pid = piped_child(args, f_in_p, f_out_p); // HPZ: Remote Branch ++ pid = piped_child(args, f_in_p, f_out_p); + #ifdef ICONV_CONST + setup_iconv(); + #endif +@@ -1112,9 +1107,9 @@ + + /* This is called once the connection has been negotiated. It is used + * for rsyncd, remote-shell, and local connections. */ +-int client_run(int f_in, int f_out, pid_t pid, int argc, char *argv[]) ++int client_run(int f_in, int f_out, pid_t pid, int argc, char *argv[]) + { +- struct file_list *flist = NULL; ++ struct file_list *flist = NULL; + int exit_code = 0, exit_code2 = 0; + char *local_name = NULL; + +@@ -1156,55 +1151,18 @@ + send_filter_list(f_out); + if (filesfrom_host) + filesfrom_fd = f_in; ++ + if (write_batch && !am_server) + start_write_batch(f_out); + flist = send_file_list(f_out, argc, argv); + if (DEBUG_GTE(FLIST, 3)) + rprintf(FINFO,"file list sent\n"); ++ + if (protocol_version < 31 && filesfrom_host && protocol_version >= 23) + io_start_multiplex_in(f_in); + + io_flush(NORMAL_FLUSH); +- +- //gettimeofday(&td, NULL); // HPZ +- long sec = 0, usec = 0; +- float time = 0; +- /* +- if((td.tv_usec-tv.tv_usec)<0){ +- sec = td.tv_sec - tv.tv_sec - 1; +- usec = 1000000 +td.tv_usec - tv.tv_usec; +- }else{ +- sec = td.tv_sec - tv.tv_sec; +- usec = td.tv_usec - tv.tv_usec; +- } +- time = (float)sec +(float)usec/1000000; +- printf("BE:%.4f\n", time); // HPZ: Print when end running +- */ +- +- +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "send_files start", logtime.tv_sec, logtime.tv_usec); +- + send_files(f_in, f_out); +- gettimeofday(&td, NULL); // HPZ +- +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "send_files finish", logtime.tv_sec, logtime.tv_usec); +- +- /* +- //printf("%ld.%ld\n", td.tv_sec, td.tv_usec); +- //long sec = 0, usec = 0; +- if((td.tv_usec-tv.tv_usec)<0){ +- sec = td.tv_sec - tv.tv_sec - 1; +- usec = 1000000 +td.tv_usec - tv.tv_usec; +- }else{ +- sec = td.tv_sec - tv.tv_sec; +- usec = td.tv_usec - tv.tv_usec; +- } +- time = (float)sec +(float)usec/1000000; +- printf("AF:%.4f\n", time); // HPZ: Print when end running +- */ +- + io_flush(FULL_FLUSH); + handle_stats(-1); + if (protocol_version >= 24) +@@ -1217,24 +1175,9 @@ + } + output_summary(); + io_flush(FULL_FLUSH); +- //struct timeval tv; +- gettimeofday(&td, NULL); // HPZ +- //long sec = 0, usec = 0; +- if((td.tv_usec-tv.tv_usec)<0){ +- sec = td.tv_sec - tv.tv_sec - 1; +- usec = 1000000 +td.tv_usec - tv.tv_usec; +- }else{ +- sec = td.tv_sec - tv.tv_sec; +- usec = td.tv_usec - tv.tv_usec; +- } +- time = (float)sec +(float)usec/1000000; +- //printf("%.4f,\n", time); // HPZ: Print when end running +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "RSYNC finish", logtime.tv_sec, logtime.tv_usec); +- printf("-----------------------------------\n"); + exit_cleanup(exit_code); +- + } ++ + if (!read_batch) { + if (protocol_version >= 23) + io_start_multiplex_in(f_in); +@@ -1245,6 +1188,7 @@ + } + + send_filter_list(read_batch ? -1 : f_out); ++ + if (filesfrom_fd >= 0) { + start_filesfrom_forwarding(filesfrom_fd); + filesfrom_fd = -1; +@@ -1299,7 +1243,7 @@ + * + * Calls either start_socket_client (for sockets) or do_cmd and + * client_run (for ssh). */ +-static int start_client(int argc, char *argv[]) ++static int start_client(int argc, char *argv[]) + { + char *p, *shell_machine = NULL, *shell_user = NULL; + char **remote_argv; +@@ -1429,9 +1373,8 @@ + } + } + +- if (daemon_over_rsh < 0){ ++ if (daemon_over_rsh < 0) + return start_socket_client(shell_machine, remote_argc, remote_argv, argc, argv); +- } + + if (password_file && !daemon_over_rsh) { + rprintf(FERROR, "The --password-file option may only be " +@@ -1471,11 +1414,8 @@ + if (tmpret < 0) + return tmpret; + } +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "client_run start", logtime.tv_sec, logtime.tv_usec); ++ + ret = client_run(f_in, f_out, pid, argc, argv); +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "client_run finish", logtime.tv_sec, logtime.tv_usec); + + fflush(stdout); + fflush(stderr); +@@ -1581,11 +1521,6 @@ + int ret; + int orig_argc = argc; + char **orig_argv = argv; +- gettimeofday(&tv, NULL); // HPZ +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "RSYNC start", logtime.tv_sec, logtime.tv_usec); +- +- + #ifdef HAVE_SIGACTION + # ifdef HAVE_SIGPROCMASK + sigset_t sigmask; +diff -uNr rsync/match.c rsync-3.1.2/match.c +--- rsync/match.c 2016-10-07 15:03:51.783262780 -0400 ++++ rsync-3.1.2/match.c 2015-08-08 15:47:03.000000000 -0400 +@@ -37,8 +37,6 @@ + static int total_false_alarms; + static int total_hash_hits; + static int total_matches; +-static int64 transfer_time = 0; // HPZ: This is used to detect how many times invoking a match +-static int64 meet_time = 0; // HPZ: This is used to detect how many times it hit the buffer size + + extern struct stats stats; + +@@ -87,12 +85,6 @@ + hash_table[t] = i; + } + } +- /* +- uint32 q; +- for(q=0; q < tablesize; q++){ // HPZ: Print hash table +- if(hash_table[q]!=-1) +- printf("%d:\t%d\n", q, hash_table[q]); +- }*/ + } + + +@@ -116,18 +108,14 @@ + int32 n = (int32)(offset - last_match); /* max value: block_size (int32) */ + int32 j; + +- /* +- //if (DEBUG_GTE(DELTASUM, 2) && i >= 0) { +- if(i >= 0 ){ ++ if (DEBUG_GTE(DELTASUM, 2) && i >= 0) { + rprintf(FINFO, + "match at %s last_match=%s j=%d len=%ld n=%ld\n", + big_num(offset), big_num(last_match), i, + (long)s->sums[i].len, (long)n); + } +- */ + + send_token(f, i, buf, last_match, n, i < 0 ? 0 : s->sums[i].len); +- transfer_time ++; // HPZ + data_transfer += n; + + if (i >= 0) { +@@ -347,13 +335,11 @@ + match. The 3 reads are caused by the + running match, the checksum update and the + literal send. */ +- if (backup >= s->blength+CHUNK_SIZE && end-offset > CHUNK_SIZE){ ++ if (backup >= s->blength+CHUNK_SIZE && end-offset > CHUNK_SIZE) + matched(f, s, buf, offset - s->blength, -2); +- meet_time++; // HPZ +- } + } while (++offset < end); + +- matched(f, s, buf, len, -1); // HPZ : This send only 0-token-literal data ++ matched(f, s, buf, len, -1); + map_ptr(buf, len-1, 1); + } + +@@ -379,8 +365,6 @@ + hash_hits = 0; + matches = 0; + data_transfer = 0; +- struct timeval tv2; // HPZ start +- gettimeofday(&tv2, NULL); // HPZ + + sum_init(checksum_seed); + +@@ -450,21 +434,6 @@ + total_false_alarms += false_alarms; + total_matches += matches; + stats.literal_data += data_transfer; +- +- struct timeval td2; // HPZ +- gettimeofday(&td2, NULL); +- long sec = 0, usec = 0; +- float time = 0; +- if((td2.tv_usec-tv2.tv_usec)<0){ +- sec = td2.tv_sec - tv2.tv_sec - 1; +- usec = 1000000 +td2.tv_usec - tv2.tv_usec; +- }else{ +- sec = td2.tv_sec - tv2.tv_sec; +- usec = td2.tv_usec - tv2.tv_usec; +- } +- time = (float)sec +(float)usec/1000000; +- //printf("%.4f,", time); // HPZ +- + } + + void match_report(void) +@@ -472,10 +441,6 @@ + if (!DEBUG_GTE(DELTASUM, 1)) + return; + +- printf("matched has been invoked %ld times.\n", transfer_time); +- printf("windows size has been hitted %ld times.\n", meet_time); +- +- + rprintf(FINFO, + "total: matches=%d hash_hits=%d false_alarms=%d data=%s\n", + total_matches, total_hash_hits, total_false_alarms, +diff -uNr rsync/pipe.c rsync-3.1.2/pipe.c +--- rsync/pipe.c 2016-10-05 12:41:00.931943173 -0400 ++++ rsync-3.1.2/pipe.c 2015-08-08 15:47:03.000000000 -0400 +@@ -175,7 +175,6 @@ + + *f_in = from_child_pipe[0]; + *f_out = to_child_pipe[1]; +- //printf("f_in:\t%d, f_out:\t%d",*f_in,*f_out); //HPZ : This never changed + + return pid; + } +diff -uNr rsync/proto.h rsync-3.1.2/proto.h +--- rsync/proto.h 2016-10-07 16:25:14.123350335 -0400 ++++ rsync-3.1.2/proto.h 2015-12-21 15:22:53.000000000 -0500 +@@ -253,7 +253,7 @@ + void read_del_stats(int f); + int child_main(int argc, char *argv[]); + void start_server(int f_in, int f_out, int argc, char *argv[]); +-int client_run(int f_in, int f_out, pid_t pid, int argc, char *argv[]) ; ++int client_run(int f_in, int f_out, pid_t pid, int argc, char *argv[]); + void remember_children(UNUSED(int val)); + const char *get_panic_action(void); + int main(int argc,char *argv[]); +@@ -399,7 +399,6 @@ + const char *sum_as_hex(const char *sum); + NORETURN void out_of_memory(const char *str); + NORETURN void overflow_exit(const char *str); +-void print(char *msg); + void free_xattr(stat_x *sxp); + int get_xattr(const char *fname, stat_x *sxp); + int copy_xattrs(const char *source, const char *dest); +diff -uNr rsync/rsync.c rsync-3.1.2/rsync.c +--- rsync/rsync.c 2016-10-07 15:50:15.478230583 -0400 ++++ rsync-3.1.2/rsync.c 2015-08-08 15:47:03.000000000 -0400 +@@ -317,7 +317,6 @@ + + read_loop: + while (1) { +- + ndx = read_ndx(f_in); + + if (ndx >= 0) +@@ -369,7 +368,7 @@ + flist->parent_ndx = ndx; + stop_flist_forward(); + } +- ++ + iflags = protocol_version >= 29 ? read_shortint(f_in) + : ITEM_TRANSFER | ITEM_MISSING_DATA; + +@@ -750,4 +749,3 @@ + : am_receiver ? "receiver" + : "Receiver"; /* pre-forked receiver */ + } +- +diff -uNr rsync/rsync.h rsync-3.1.2/rsync.h +--- rsync/rsync.h 2016-10-07 16:24:23.365209158 -0400 ++++ rsync-3.1.2/rsync.h 2015-08-08 15:47:03.000000000 -0400 +@@ -705,8 +705,6 @@ + const char basename[1]; /* The basename (AKA filename) follows */ + }; + +-struct timeval logtime; // HPZ: This is very useful log timestamp +- + extern int file_extra_cnt; + extern int inc_recurse; + extern int uid_ndx; +diff -uNr rsync/sender.c rsync-3.1.2/sender.c +--- rsync/sender.c 2016-10-07 15:31:56.633777127 -0400 ++++ rsync-3.1.2/sender.c 2015-09-07 13:07:17.000000000 -0400 +@@ -206,8 +206,6 @@ + if (DEBUG_GTE(SEND, 1)) + rprintf(FINFO, "send_files starting\n"); + +- int tmpflag=0; // HPZ: I'm so stupid +- struct timeval tv3, td3; + while (1) { + if (inc_recurse) { + send_extra_file_list(f_out, MIN_FILECNT_LOOKAHEAD); +@@ -215,16 +213,9 @@ + } + + /* This call also sets cur_flist. */ +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "rw_ndx start", logtime.tv_sec, logtime.tv_usec); + ndx = read_ndx_and_attrs(f_in, f_out, &iflags, &fnamecmp_type, +- xname, &xlen); // HPZ: Figure out this! +- ++ xname, &xlen); + extra_flist_sending_enabled = False; +- //printf("ndx:%d\n", ndx); +- /* +- From here +- */ + + if (ndx == NDX_DONE) { + if (!am_server && INFO_GTE(PROGRESS, 2) && cur_flist) { +@@ -241,24 +232,16 @@ + continue; + } + } +- if (++phase > max_phase){ ++ if (++phase > max_phase) + break; +- } + if (DEBUG_GTE(SEND, 1)) + rprintf(FINFO, "send_files phase=%d\n", phase); +- write_ndx(f_out, NDX_DONE); // HPZ: Due to phase, it will run twice. so WHAT's WRITE_NDX ++ write_ndx(f_out, NDX_DONE); + continue; +- } +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "rw_ndx finish", logtime.tv_sec, logtime.tv_usec); +- +- /* +- to here +- */ +- ++ } ++ + if (inc_recurse) + send_extra_file_list(f_out, MIN_FILECNT_LOOKAHEAD); +- // printf("****************************%d\n",ndx); + + if (ndx - cur_flist->ndx_start >= 0) + file = cur_flist->files[ndx - cur_flist->ndx_start]; +@@ -273,14 +256,16 @@ + if (!change_pathname(file, NULL, 0)) + continue; + f_name(file, fname); ++ + if (DEBUG_GTE(SEND, 1)) + rprintf(FINFO, "send_files(%d, %s%s%s)\n", ndx, path,slash,fname); +- ++ + #ifdef SUPPORT_XATTRS + if (preserve_xattrs && iflags & ITEM_REPORT_XATTR && do_xfers + && !(want_xattr_optim && BITS_SET(iflags, ITEM_XNAME_FOLLOWS|ITEM_LOCAL_CHANGE))) + recv_xattr_request(file, f_in); + #endif ++ + if (!(iflags & ITEM_TRANSFER)) { + maybe_log_item(file, iflags, itemizing, xname); + write_ndx_and_attrs(f_out, ndx, iflags, fname, file, +@@ -308,6 +293,7 @@ + who_am_i()); + exit_cleanup(RERR_PROTOCOL); + } ++ + if (file->flags & FLAG_FILE_SENT) { + if (csum_length == SHORT_SUM_LENGTH) { + /* For inplace: redo phase turns off the backup +@@ -325,6 +311,7 @@ + if (iflags & ITEM_IS_NEW) + stats.created_files++; + } ++ + updating_basis_file = inplace && (protocol_version >= 29 + ? fnamecmp_type == FNAMECMP_FNAME : make_backups <= 0); + +@@ -341,13 +328,13 @@ + fnamecmp_type, xname, xlen); + continue; + } +- ++ + if (!(s = receive_sums(f_in))) { + io_error |= IOERR_GENERAL; + rprintf(FERROR_XFER, "receive_sums failed\n"); + exit_cleanup(RERR_PROTOCOL); + } +- ++ + fd = do_open(fname, O_RDONLY, 0); + if (fd == -1) { + if (errno == ENOENT) { +@@ -368,6 +355,7 @@ + send_msg_int(MSG_NO_SEND, ndx); + continue; + } ++ + /* map the local file */ + if (do_fstat(fd, &st) != 0) { + io_error |= IOERR_GENERAL; +@@ -383,7 +371,7 @@ + } else + mbuf = NULL; + +- if (DEBUG_GTE(DELTASUM, 2)){ ++ if (DEBUG_GTE(DELTASUM, 2)) { + rprintf(FINFO, "send_files mapped %s%s%s of size %s\n", + path,slash,fname, big_num(st.st_size)); + } +@@ -391,6 +379,7 @@ + write_ndx_and_attrs(f_out, ndx, iflags, fname, file, + fnamecmp_type, xname, xlen); + write_sum_head(f_xfer, s); ++ + if (DEBUG_GTE(DELTASUM, 2)) + rprintf(FINFO, "calling match_sums %s%s%s\n", path,slash,fname); + +@@ -400,15 +389,8 @@ + rprintf(FCLIENT, "%s\n", fname); + + set_compression(fname); +- +- +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "match_sum start", logtime.tv_sec, logtime.tv_usec); +- match_sums(f_xfer, s, mbuf, st.st_size); +- gettimeofday(&logtime, NULL); +- printf("[%s]:%ld.%ld\n", "match_sum finish", logtime.tv_sec, logtime.tv_usec); +- match_report(); + ++ match_sums(f_xfer, s, mbuf, st.st_size); + if (INFO_GTE(PROGRESS, 1)) + end_progress(st.st_size); + +@@ -424,6 +406,7 @@ + } + } + close(fd); ++ + free_sums(s); + + if (DEBUG_GTE(SEND, 1)) +@@ -432,7 +415,6 @@ + /* Flag that we actually sent this entry. */ + file->flags |= FLAG_FILE_SENT; + } +- + if (make_backups < 0) + make_backups = -make_backups; + +@@ -442,7 +424,7 @@ + if (DEBUG_GTE(SEND, 1)) + rprintf(FINFO, "send files finished\n"); + +- // Match_report suppose to be here. ++ match_report(); + + write_ndx(f_out, NDX_DONE); + } +diff -uNr rsync/util2.c rsync-3.1.2/util2.c +--- rsync/util2.c 2016-10-07 16:25:30.710777685 -0400 ++++ rsync-3.1.2/util2.c 2015-08-08 15:47:03.000000000 -0400 +@@ -111,8 +111,3 @@ + rprintf(FERROR, "ERROR: buffer overflow in %s [%s]\n", str, who_am_i()); + exit_cleanup(RERR_MALLOC); + } +- +-void print(char *msg) +-{ +- printf("%s\n", msg); +- } +diff -uNr rsync/util.c rsync-3.1.2/util.c +--- rsync/util.c 2016-10-07 16:16:49.961215701 -0400 ++++ rsync-3.1.2/util.c 2015-12-21 13:54:02.000000000 -0500 +@@ -1645,4 +1645,3 @@ + } + return (char*)lp->items + (lp->count++ * item_size); + } +- diff --git a/static/2016-10/rsync/small2Big_change_at_anyplace.sh b/static/2016-10/rsync/small2Big_change_at_anyplace.sh new file mode 100644 index 0000000..748db07 --- /dev/null +++ b/static/2016-10/rsync/small2Big_change_at_anyplace.sh @@ -0,0 +1,31 @@ +FILESIZE=8192 +let CHANGESIZE=1024 + +UNI_TARGET='phao3@bingsuns.binghamton.edu:~' +UNI_PASSWD=Pengzhan2015USA + +LAB_TARGET='moslab@10.42.0.1:~/Desktop/server' +LAB_PASSWD=1 + +LOC_TARGET='..' + +temp=1 +for i in `seq 1 14`; +do + dd if=/dev/urandom of=$FILESIZE.dat bs=8192 count=$temp status=none + sshpass -p $LAB_PASSWD ./rsync $FILESIZE.dat $LAB_TARGET + # ./rsync -v -v $FILESIZE.dat $LOC_TARGET --no-whole-file + # echo $FILESIZE + dd if=/dev/urandom of=$CHANGESIZE.dat bs=$CHANGESIZE count=1 status=none + # cat $CHANGESIZE.dat $FILESIZE.dat > temp.dat + python ../addbyte.py $FILESIZE.dat $CHANGESIZE.dat temp.dat + # cp -f $FILESIZE.dat temp.dat + sshpass -p $LAB_PASSWD ./rsync temp.dat $LAB_TARGET/$FILESIZE.dat + # ./rsync temp.dat $LOC_TARGET/$FILESIZE.dat --no-whole-file + + + let temp=$temp*2 + let FILESIZE=$FILESIZE*2 +done +date + diff --git a/static/2016-10/rsync/small2Big_change_at_begin.sh b/static/2016-10/rsync/small2Big_change_at_begin.sh new file mode 100644 index 0000000..96233b2 --- /dev/null +++ b/static/2016-10/rsync/small2Big_change_at_begin.sh @@ -0,0 +1,29 @@ +FILESIZE=8192 +let CHANGESIZE=1024 + +UNI_TARGET='phao3@bingsuns.binghamton.edu:~' +UNI_PASSWD=Pengzhan2015USA + +LAB_TARGET='moslab@10.42.0.1:~/Desktop/server' +LAB_PASSWD=1 + +LOC_TARGET='..' + +temp=1 +for i in `seq 1 14`; +do + dd if=/dev/urandom of=$FILESIZE.dat bs=8192 count=$temp status=none + sshpass -p $LAB_PASSWD ./rsync $FILESIZE.dat $LAB_TARGET + # ./rsync $FILESIZE.dat $LOC_TARGET --no-whole-file + #echo $FILESIZE + dd if=/dev/urandom of=$CHANGESIZE.dat bs=$CHANGESIZE count=1 status=none + cat $CHANGESIZE.dat $FILESIZE.dat > temp.dat + sshpass -p $LAB_PASSWD ./rsync temp.dat $LAB_TARGET/$FILESIZE.dat + # ./rsync temp.dat $LOC_TARGET/$FILESIZE.dat --no-whole-file + + + let temp=$temp*2 + let FILESIZE=$FILESIZE*2 +done +date + diff --git a/static/2016-10/rsync/small2Big_change_at_last.sh b/static/2016-10/rsync/small2Big_change_at_last.sh new file mode 100644 index 0000000..a10b38b --- /dev/null +++ b/static/2016-10/rsync/small2Big_change_at_last.sh @@ -0,0 +1,23 @@ +FILESIZE=134217728 +let CHANGESIZE=16777216 + +UNI_TARGET='phao3@bingsuns.binghamton.edu:~' +UNI_PASSWD=Pengzhan2015USA + +LAB_TARGET='moslab@10.42.0.1:~/Desktop/server' +LAB_PASSWD=1 +temp=16384 +for i in `seq 1 4`; +do + dd if=/dev/urandom of=$FILESIZE.dat bs=8192 count=$temp status=none + sshpass -p $LAB_PASSWD ./rsync $FILESIZE.dat $LAB_TARGET + #echo $FILESIZE + dd if=/dev/urandom of=$CHANGESIZE.dat bs=1024 count=$temp status=none + cat $FILESIZE.dat $CHANGESIZE.dat > temp.dat + sshpass -p $LAB_PASSWD ./rsync temp.dat $LAB_TARGET/$FILESIZE.dat + + let temp=$temp*2 + let FILESIZE=$FILESIZE*2 +done +date +