<div dir="ltr">Yes, I also figured out this way. I used the 2D array to speedup the reading. See the comparison below.<div><br></div><div>Thanks all!<br><div><br></div><div><br></div><div>// 1D results</div><div><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(213,59,211)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">\begin</span><span style="font-variant-ligatures:no-common-ligatures">{tabular}{</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)">ll</span><span style="font-variant-ligatures:no-common-ligatures">}</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(205,121,35)"><span style="font-variant-ligatures:no-common-ligatures">\toprule</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Number of Numbers </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">&</span><span style="font-variant-ligatures:no-common-ligatures"> Reading Time </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">\\</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(205,121,35)"><span style="font-variant-ligatures:no-common-ligatures">\midrule</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">10 </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">&</span><span style="font-variant-ligatures:no-common-ligatures"> 0.096s </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">\\</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">100 </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">&</span><span style="font-variant-ligatures:no-common-ligatures"> 0.096s </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">\\</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">1K </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">&</span><span style="font-variant-ligatures:no-common-ligatures"> 0.100s </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">\\</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">10K </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">&</span><span style="font-variant-ligatures:no-common-ligatures"> 0.463s </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">\\</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">100K </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">&</span><span style="font-variant-ligatures:no-common-ligatures"> 1m23s </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">\\</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">500K </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">&</span><span style="font-variant-ligatures:no-common-ligatures"> 30m21 </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">\\</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(205,121,35)"><span style="font-variant-ligatures:no-common-ligatures">\bottomrule</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(213,59,211)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">\end</span><span style="font-variant-ligatures:no-common-ligatures">{tabular}</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">\caption</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">{</span><span style="font-variant-ligatures:no-common-ligatures">The running time of readFromFile on my iMAC (3.2GHz, SKL). The solution</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">is not scaled after 10K data.</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">}</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)"><br></span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)"><br></span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">// 2D results, I used 1K for each subarray</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(213,59,211)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">\begin</span><span style="font-variant-ligatures:no-common-ligatures">{tabular}{</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(0,0,0)">ll</span><span style="font-variant-ligatures:no-common-ligatures">}</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(205,121,35)"><span style="font-variant-ligatures:no-common-ligatures">\toprule</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">Number of Numbers </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">&</span><span style="font-variant-ligatures:no-common-ligatures"> Reading Time </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">\\</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(205,121,35)"><span style="font-variant-ligatures:no-common-ligatures">\midrule</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">100K </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">&</span><span style="font-variant-ligatures:no-common-ligatures"> 1.075s </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">\\</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">500K </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">&</span><span style="font-variant-ligatures:no-common-ligatures"> 5.434s </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">\\</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">1M </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">&</span><span style="font-variant-ligatures:no-common-ligatures"> 14s </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">\\</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">5M </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">&</span><span style="font-variant-ligatures:no-common-ligatures"> 6m27s </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">\\</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(205,121,35)"><span style="font-variant-ligatures:no-common-ligatures">\bottomrule</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(213,59,211)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">\end</span><span style="font-variant-ligatures:no-common-ligatures">{tabular}</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(213,59,211)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">\label{</span><span style="font-variant-ligatures:no-common-ligatures">table:fastReadFromFile</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">}</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(205,121,35)">\caption</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">{</span><span style="font-variant-ligatures:no-common-ligatures">The running time of fastReadFromFile on my iMAC (3.2GHz, SKL). For 500K data points,</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">it improves the performance by 335x over readFromFile. However, the solution is not</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">scalable after 5M data points. I can further improve it by using multidimenstional arrays</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">
</span></p><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">when my data set size reaches 5M.</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)">}</span></p><div><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)"><br></span></div><p style="margin:0px;font-size:12px;line-height:normal;font-family:menlo;color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures;color:rgb(213,59,211)"><br></span></p></div><div><br><div><img><br></div><div><img><br></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 10, 2017 at 10:14 AM, luigi scarso <span dir="ltr"><<a href="mailto:luigi.scarso@gmail.com" target="_blank">luigi.scarso@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Fri, Mar 10, 2017 at 3:58 PM, Hans Hagen <<a href="mailto:pragma@wxs.nl">pragma@wxs.nl</a>> wrote:<br>
> this runs in 3 sec on my machine which is way faster than storing at the mp<br>
> end where at some point you hit limitations in constructing names for<br>
> variables (keep in mind that a[i] is not an array but a hashed variable name<br>
> constructed from a and i)<br>
><br>
><br>
<br>
</span>We have played a bit, and the point is that<br>
is better to think to a[] as a tree, and lesser is the depth , the better.<br>
<br>
For example given c and foo[]<br>
one can use foo[c] to store s, but<br>
can also declare foo[][][][] and say<br>
foo[c div 1000][c div 100][c div 10][c] := s ;<br>
<br>
We don't lose items, because the last index of foo is unique, and<br>
lookup is better<br>
(of course we have now 3 operations, which cost).<br>
<br>
You can experiment with the following code by Hans:<br>
<br>
<br>
string foo[][][][];<br>
<br>
def set_foo(expr c, s) =<br>
foo[c div 1000][c div 100][c div 10][c] := s ;<br>
enddef ;<br>
def get_foo(expr c) =<br>
foo[c div 1000][c div 100][c div 10][c]<br>
enddef ;<br>
<br>
string s;<br>
<span class=""> numeric c ; c := 0;<br>
</span> forever:<br>
s := readfrom "foo3.tmp";<br>
exitif s = EOF;<br>
c := c + 1 ;<br>
set_foo(c,s) ;<br>
endfor ;<br>
save l ; numeric l;<br>
for i=1 upto c :<br>
l := length(get_foo(c)) ;<br>
endfor ;<br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
<br>
--<br>
luigi<br>
--<br>
<a href="http://tug.org/metapost/" rel="noreferrer" target="_blank">http://tug.org/metapost/</a><br>
</font></span></blockquote></div><br></div>