[tex-live] Rsync, a different question

[George N. White III]

On Mon, Nov 10, 2008 at 6:58 AM, Philip TAYLOR <P.Taylor at rhul.ac.uk> wrote:
>
>
> George N. White III wrote:
>
>> In my experience, simply retrying the same site rarely works, so a better
>> approach might be to use a list of different CTAN mirrors.  If they
>> all fail then you have a serious network problem, so looping may not be
>> much help.
>
> As others have already observed, there are many potential
> issues if one tries to build a local CTAN (or TeX Live)
> mirror from multiple CTAN mirrors rather than from one.

That needs to be fixed.  I think linux package managers rely on
having versions in filenames.  You get a master list of the
current versions, then depending on the tool, it can  go
shopping across a bunch of mirror sites for the version it
needs.  Problems occur if you get an old master list, as
mirror sites don't store old versions, but it is much more
robust than being tied to just one mirror.

> But in my experience (admittedly very limited), retrying
> Rsync.TeX.Ac.Uk usually /does/ work; my last full TeX
> Live mirror required at most five retries in total before
> the mirroring was complete, and as Norbert observed, each
> retry (usually) has less work to do than the one before.

Maybe there is some traffic shaping between you and the
CTAN site.
<http://en.wikipedia.org/wiki/Traffic_shaping>

You might avoid this by limiting the rsync rate yourself, but it
is a bit tricky since the external limit may depend on the
competing demand from external sources, so you can only
guess at what limit to impose on your own rsync.   There are
reports that some providers mark any IP that has used P2P
for shaping, and that both ftp and rsync are subject to shaping, so
your thruput may be affected by your usage history.  Shaping is often
imposed during peak hours, so you may find things work better
if you do the transfers at some carefully selected time.

<http://carlhutzler.com/blog/2008/02/07/traffic-shaping-and-off-site-backups/>
discusses using traffic shaping via a DD-WRT router to keep his backups from
slowing down his other traffic, but the same approach should work to reduce
your chances of being hit by external shaping.

Linux has <http://lartc.org/manpages/tc.txt> -- tc - show / manipulate traffic
control settings, and various more user friendly wrappers for tc.

There appear to be a number of "free" traffic shaping tools for Win32.

> But I haven't yet found an elegant syntax for
>
>        Repeat Until Rsync ...
>
> in Win/32 CMD.

Maybe <http://www.computerperformance.co.uk/powershell/powershell_loops.htm>
could help.   Another route to take advantage of linux tools is to run
linux in a small
VM or maybe colinux.

-- 
George N. White III <aa056 at chebucto.ns.ca>
Head of St. Margarets Bay, Nova Scotia