[tex-live] [luatex] [lltx] Location of recorder file

[Philipp Stephani]

Am 17.05.2011 um 01:27 schrieb Reinhard Kotucha:

> On 2011-05-15 at 11:45:01 +0200, Philipp Stephani wrote:
> 
>> Am 15.05.2011 um 00:36 schrieb Reinhard Kotucha:
>> 
>>> On 2011-05-14 at 02:43:31 +0200, Philipp Stephani wrote:
>>> 
>>>> [...] 
>>>> 
>>>> The Microsoft Visual C runtime is the current standard C runtime
>>>> and not at all old or outdated.
>>> 
>>> Hi Philipp,
>>> thank you very much for the comprehensive response.
>>> 
>>> I was confused by different behavior of cmd.exe and the Exploder,
>> 
>> What exactly is the different behavior? AFAIK cmd doesn't accept /
>> as a path delimiter (due to historical reasons, of course...), but
>> that's about it. Missing Unicode support is generally caused by the
>> non-Unicode font that is used by default.
> 
> There were differences regarding file permissions.  In cmd.exe I was
> sometimes unable to [re]move files.  I got a "permission denied" error
> though I could [re]move the files in the Exploder.  I don't know what
> caused the file permission problem, but I vaguely remember that it
> sometimes could be solved by setting the executable flag under Cygwin,
> though I don't understand why.

AFAIK there can't be any such differences since both the console and the Explorer only report back the errors reported by the kernel. Maybe you ran the Explorer with elevated privileges in that case?

> 
> And there is also the encoding mess.  We had TortoiseSVN, which was
> running in the Exploder and thus used CP1252.  When I committed files
> on the command line (cmd), CP850 was used and non-ASCII characters in
> log messages were not displayed properly in the GUI.  I tried "chcp
> 1252" in the CLI but cmd.exe obviously didn't support it.

As I explained, everything in Windows uses UTF-16, the old "codepages" are only for legacy applications. If you have encoding problems, they are *almost always* application-related. You can use any Unicode filename just fine in both the Explorer and the command prompt. So this is purely a TortoiseSVN problem.

> 
> There is also another issue, besides that reported by Mojca and Ulrike,
> regarding the display of filenames.  Put a Type 1 font into
> c:\windows\fonts.  The CLI (dir) lists both, the PFB and the PFM file,
> while the Exploder only shows the PFM file.

The "Fonts" directory lists fonts, not files.

> 
> That was on XP SP2, I've never used Vista or W7, but I don't expect
> much progress anyway.

XP was already a good operating system, and much progress in all areas has been made since then. Sure, there are a few issues, but most of them don't matter in practice for average users (and that's the target audience of Windows). Unfortunately there is a lot of backward-compatibility stuff that makes the system more complex than needed, but if you follow modern programming rules (e.g., none of the legacy *A functions, don't try to write in C:\Windows, ...), there shouldn't be too many problems. (I know that trying to code platform-independently adds several orders of magnitude.)

> It's the Exploder which doesn't accept them as path delimiters.  If
> you enter a file name with forward slashes, the Exploder prepends
> "http://" to the path, which is probably not what you expect. :) 

Well, why not? The backslash is the standard directory separator on Windows, and the slash is the URL component separator. I think this is quite a user-friendly choice.

> 
> Furthermore, the gap between the CLI and the GUI seems to increase.

There is no gap, these are just two different interfaces to the operating system, both of which are in active development.

> On a German XP there was a directory c:\Programme.  This directory was
> displayed by the Exploder too.  Under Vista, programs are installed
> under "c:\Program Files", but the Exploder shows the non-existent
> directory "Programme" instead.

This behavior was adopted from OS X, where several folders have localized names in the UI, but non-localized names for the kernel (e.g. /Applications is listed as "Programme" on my system). The idea is that a programmer wants a fixed path such as "/Applications", while the user prefers to see the folders in his native language.

> 
> I'm a bit wondering that you say that the only difference between the
> CLI and the GUI is how they handle forward slashes and file names
> beginning with a dot.

There are certainly more differences, but nothing spectacular, I think. (But I don't really know since I've never used the command prompt that much.)

>  I have the impression that Windows (please note
> the plural) are at least two distinct operating systems, NTVDM
> provides a third one, and I've heard that even some XP programs are
> virtualized too under W7...

Yes, there is a lot of virtualization of every kind, and a lot of legacy support. This is all for backwards compatibility, but not something to worry about in new programs.

> 
> However, I think that you, as a C programmer, don't have to worry so
> much about the limitations of cmd.exe.  But if you are using Perl or
> Emacs, you can't avoid shell escapes.  I encountered the problem with
> file permissions in Emacs, which calls "cmd /c delete ..." in order to
> delete a file.  When we wrote the installer, we encountered severe
> problems with pipes.  They are implemented in cmd.exe and don't
> support binary files reliably.  As a C programmer you would certainly
> avoid the shell and use popen() directly, which can be forced to
> binary mode.  The pipes implemented in cmd.exe are *guessing* whether
> a file is a text or binary file and try to "repair" line endings
> occasionally.

I must confess that I don't use cmd.exe much, and I think that's pretty common for Windows programmers. Cmd.exe is good for simple administrative tasks, but not so good for scripting, where there are more powerful application (e.g. PowerShell).
It is true that Windows doesn't really encourage command line work. Programs that would be text scripts on Linux would often be implemented as full-fledged binary programs on Windows. Put it another way, the gap between a normal user and a programmer is somewhat larger on Windows: programming usually means using Visual Studio, and the notion of "hacking together a small shell script" is almost not present.

> 
> You recently said:
> 
>> If I read perldoc perlfunc correctly, something like
>> open(HANDLE, "-|", "command", "arg1", "arg2", ...)
>> should do the job, provided that maps to the correct CreateProcessW
>> invocation (with conversion to UTF-16 and proper quoting) on Windows
>> and to a fork/exec or spawn_process invocation on Linux and OS X
>> without going through any shell or standard C function.
> 
> Well, since we painfully had to learn that even if you pass arguments
> as a list to execve(), you still have to quote arguments because
> Windows converts the argument list to a string and then back to a list
> internally, before passing the arguments to its child process, I doubt
> that even low-level functions avoid the shell.

No, you shouldn't have to quote anything, that's what the language's standard library is for. For example, Python converts an argument list to a command-line string using the list2cmdline functions:
>>> print subprocess.list2cmdline(["a b", r'c" d\""ef g'])
"a b" "c\" d\\\"\"ef g"
Indeed this is something that only the language runtime can do: the spawn* functions indeed don't quote their arguments properly, as discussed in http://msdn.microsoft.com/en-us/library/20y988d2.aspx. As usual, that behavior can't be changed because it could break anything. Therefore I think the spawn* and exec* functions should be avoided altogether: if you have to preprocess the argument vector in any case, directly calling CreateProcessW isn't such a big deal either.

> 
> Look at
> 
>  texmf/scripts/texlive/rungs.tlu 
> 
> in order to see how we solve the problem ATM.  It would be nice to
> have the annoying function fixwin() integrated in texlua, similarly as
> it's integrated in Perl.  However, Taco won't do that without a clear
> specification from Microsoft.  Understandably.  But we didn't find any
> specifications so far.

The rules are described here:
http://msdn.microsoft.com/en-us/library/17w5ykft(v=VS.100).aspx
However, I think adding them to luatex isn't such a good idea since it would widen the gap between stock Lua and LuaTeX even more. The best thing would be to write an extension module that wraps CreateProcessW directly.
If it's only about calculating the command line from an argument vector (i.e. the list2cmdline) function, then that can be done in pure Lua. For example, you could simply translate the list2cmdline function from http://hg.python.org/cpython/file/4b7c29201c60/Lib/subprocess.py.