<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 10, 2017 at 7:48 PM, Qiong Cai <span dir="ltr"><<a href="mailto:qiong.cai@gmail.com" target="_blank">qiong.cai@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Yes, I also figured out this way. I used the 2D array to speedup the reading. See the comparison below.<div><br></div><div>Thanks all!<br><div><br></div><div><br></div><div>// 1D results</div><div><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(213,59,211)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">\begin</span><span style="font-variant-ligatures:no-common-ligatures">{tabular}{</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)">ll</span><span style="font-variant-ligatures:no-common-ligatures">}</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(205,121,35)"><span style="font-variant-ligatures:no-common-ligatures">\toprule</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Number of Numbers </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">&</span><span style="font-variant-ligatures:no-common-ligatures"> Reading Time </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">\\</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(205,121,35)"><span style="font-variant-ligatures:no-common-ligatures">\midrule</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">10 </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">&</span><span style="font-variant-ligatures:no-common-ligatures"> 0.096s </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">\\</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">100 </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">&</span><span style="font-variant-ligatures:no-common-ligatures"> 0.096s </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">\\</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">1K </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">&</span><span style="font-variant-ligatures:no-common-ligatures"> 0.100s </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">\\</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">10K </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">&</span><span style="font-variant-ligatures:no-common-ligatures"> 0.463s </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">\\</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">100K </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">&</span><span style="font-variant-ligatures:no-common-ligatures"> 1m23s </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">\\</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">500K </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">&</span><span style="font-variant-ligatures:no-common-ligatures"> 30m21 </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">\\</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(205,121,35)"><span style="font-variant-ligatures:no-common-ligatures">\bottomrule</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(213,59,211)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">\end</span><span style="font-variant-ligatures:no-common-ligatures">{tabular}</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">\caption</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">{</span><span style="font-variant-ligatures:no-common-ligatures">The running time of readFromFile on my iMAC (3.2GHz, SKL). The solution</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">is not scaled after 10K data.</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">}</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)"><br></span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)"><br></span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">// 2D results, I used 1K for each subarray</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(213,59,211)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">\begin</span><span style="font-variant-ligatures:no-common-ligatures">{tabular}{</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)">ll</span><span style="font-variant-ligatures:no-common-ligatures">}</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(205,121,35)"><span style="font-variant-ligatures:no-common-ligatures">\toprule</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Number of Numbers </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">&</span><span style="font-variant-ligatures:no-common-ligatures"> Reading Time </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">\\</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(205,121,35)"><span style="font-variant-ligatures:no-common-ligatures">\midrule</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">100K </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">&</span><span style="font-variant-ligatures:no-common-ligatures"> 1.075s </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">\\</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">500K </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">&</span><span style="font-variant-ligatures:no-common-ligatures"> 5.434s </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">\\</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">1M </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">&</span><span style="font-variant-ligatures:no-common-ligatures"> 14s </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">\\</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">5M </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">&</span><span style="font-variant-ligatures:no-common-ligatures"> 6m27s </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">\\</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(205,121,35)"><span style="font-variant-ligatures:no-common-ligatures">\bottomrule</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(213,59,211)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">\end</span><span style="font-variant-ligatures:no-common-ligatures">{tabular}</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(213,59,211)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">\label{</span><span style="font-variant-ligatures:no-common-ligatures">table:fastReadFromFile</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">}</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">\caption</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">{</span><span style="font-variant-ligatures:no-common-ligatures">The running time of fastReadFromFile on my iMAC (3.2GHz, SKL). For 500K data points,</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">it improves the performance by 335x over readFromFile. However, the solution is not</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">scalable after 5M data points. I can further improve it by using multidimenstional arrays</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">
</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">when my data set size reaches 5M.</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">}</span></p><div><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)"><br></span></div><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)"><br></span></p></div><div><br></div></div></div></blockquote><div><br></div><div><br></div><div><div>%% test-032M.mp</div><div>string foo[][][][];</div><div><br></div><div>def set_foo(expr c, s) =</div><div> foo[c div 1000][c div 100][c div 10][c] := s ;</div><div>enddef ;</div><div>def get_foo(expr c) =</div><div> foo[c div 1000][c div 100][c div 10][c]</div><div>enddef ;</div><div><br></div><div>string s;</div><div>numeric c ; c := 0;</div><div>forever:</div><div> s := readfrom "test-032M.txt";</div><div> exitif s = EOF;</div><div> c := c + 1 ;</div><div> set_foo(c,s) ;</div><div>endfor ;</div><div>save l ; numeric l;</div><div>for i=1 upto c :</div><div> l := length(get_foo(c)) ;</div><div>endfor ;</div><div><br></div><div>end</div></div><div><br></div><div><br></div><div><br></div><div><div># nl test-032M.txt |tail </div><div>287991<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</div><div>287992<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</div><div>287993<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</div><div>287994<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</div><div>287995<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</div><div>287996<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</div><div>287997<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</div><div>287998<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</div><div>287999<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</div><div>288000<span class="gmail-Apple-tab-span" style="white-space:pre"> </span>xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</div></div><div><br></div><div><br></div><div><div># \time -v mpost --numbersystem=double test-032M.mp </div><div>This is MetaPost, version 1.9991 (TeX Live 2017/dev) (kpathsea version 6.2.3/dev)</div><div><br></div><div><br></div><div>Preloading the plain mem file, version 1.005) ) (./test-032M.mp )</div><div>Transcript written on test-032M.log.</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Command being timed: "mpost --numbersystem=double test-032M.mp"</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>User time (seconds): 5.98</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>System time (seconds): 0.04</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Percent of CPU this job got: 99%</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Elapsed (wall clock) time (h:mm:ss or m:ss): 0:06.03</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Average shared text size (kbytes): 0</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Average unshared data size (kbytes): 0</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Average stack size (kbytes): 0</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Average total size (kbytes): 0</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Maximum resident set size (kbytes): 79988</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Average resident set size (kbytes): 0</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Major (requiring I/O) page faults: 0</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Minor (reclaiming a frame) page faults: 19378</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Voluntary context switches: 175</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Involuntary context switches: 9</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Swaps: 0</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>File system inputs: 0</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>File system outputs: 0</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Socket messages sent: 0</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Socket messages received: 0</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Signals delivered: 0</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Page size (bytes): 4096</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>Exit status: 0</div></div></div><br clear="all"><div><br></div><div><br></div><div><br></div>-- <br><div class="gmail_signature">luigi<br></div>
</div></div>