[tlbuild] Fixes for LuaJIT on OpenBSD/clang

[Mojca Miklavec]

Hi,

Sending this to the list for feedback since automagic is somewhat beyond me ...

On Tue, 8 Jan 2019 at 13:38, luigi scarso wrote:
> On Tue, Jan 8, 2019 at 1:19 PM Mojca Miklavec wrote:
>> On Tue, 8 Jan 2019 at 12:37, luigi scarso  wrote:
>>> On Mon, 7 Jan 2019 at 13:14, Mojca Miklavec wrote:
>>>> On Mon, 7 Jan 2019 at 12:29, luigi scarso wrote:
>>>>> On Mon, Jan 7, 2019 at 12:16 PM Mojca Miklavec wrote:
>>>>>> On Mon, 7 Jan 2019 at 11:41, luigi scarso wrote:
>>>>>>> On Mon, Jan 7, 2019 at 11:27 AM Mojca Miklavec wrote:
>>>>>>>>
>>>>>>>> Also ... https://github.com/LuaJIT/LuaJIT/issues/453
>>>>>>>> http://build.contextgarden.net/builders/luatex.openbsd-amd64-6.3.prg/builds/271/steps/shell/logs/stdio
>>>>>>>>
>>>>>>> but these ones should be fixed, aren't they ?
>>>>>>
>>>>>> No.
>>>>>>
>>>>>> I believe that gcc works and clang fails (if those flags are missing)
>>>>>> and you were probably testing with gcc last time around?
>>>>>>
>>>>>> For the TeX Live builds on the build farm I'm currently passing
>>>>>>     --disable-luajittex --disable-mfluajit
>>>>>> to the OpenBSD builds just to avoid known failures.
>>>>>>
>>>>>> Yes, I could switch to gcc, but then again ... it's not that we have
>>>>>> users who need luajittex on OpenBSD, it's just that it would be nice
>>>>>> to add some autofoo magic to TeX Live to properly fix this also for
>>>>>> clang.
>>>>>
>>>>> we need only to add LDFLAGS as in
>>>>>  gmake CC=clang LDFLAGS="-lc++abi -lpthread"
>>>>> right ?
>>>>
>>>> Yes, but only in a single step, when linking luajit(tex), and only if
>>>> this is required. I would suggest creating a test. The code fails to
>>>> link stuff like
>>>> LuaJIT-src/src/lj_err.c:extern uintptr_t _Unwind_GetCFA(_Unwind_Context *);
>>>>
>>>> It would make sense to create some minimal example that tries to
>>>> compile something like
>>>>
>>>>   // _Unwind_Context *ctx, void *cf
>>>>   cf = (void *)_Unwind_GetCFA(ctx);
>>>>
>>>> and if it fails, tries again with "-lc++abi -lpthread".
>>>>
>>>> This needs to be done in autotools, and the flags applied only to luajit.
>>>
>>> fixed.
>>
>> Luigi, thank you very much.
>>
>> However there's one issue with this patch:
>>     https://github.com/TeX-Live/luatex/commit/273de1c6308ab9378905a5e1929b19207e2765af
>>     https://github.com/TeX-Live/luatex/commit/4da102e56b50dd65f122f80f101e79cb2af31121
>> You should at least test for
>>     *clang*
>> rather than
>>     clang
>> since one might specify the compiler as
>>     CC=/usr/bin/clang-5.0
>
> hm don't mess things with compilers names... not needed.

That's not messing up ;)

But in short, luajitex now fails to build with
    CC=cc
for example.

> The canonical way is not
> CC=/usr/bin/clang-5.0
> but
> export PATH=/opt/clang/clang+llvm-7.0.0-x86_64-linux-gnu-ubuntu-16.04/bin:$PATH
> export CC=clang

Your suggestion is a valid way to pick one particular compiler, but
that's not how most package managers work (and not how plenty of
developers use the tools). MacPorts for example is consistently using
things like
    CC=/opt/local/bin/gcc-mp-8 or
    CC=/opt/local/bin/clang-mp-7.0
and it apparently works for all the tens of thousands of packages :)

> (Ok, it's true that I have
> i686-linux-gnu-gcc-7 on my linux 64bit
> but cross-compile luajit from 64bit to 32bit is  black magic )

We are not talking about cross-compilation, but rather a legitimate
way to select a different native compiler. And there is absolutely no
guarantee that users will specify precisely "clang". When I'm building
for OS X, I call the compiler with both the full path and the strange
suffix.

> Keep in mind that we are providing binaries, not packages --

We are not even providing binaries for OpenBSD for TeX Live. What we
do provide are also sources that others (including packagers like
OpenBSD ports) can take and build themselves.

> and we are not testing compilers.

We are definitely not writing test suites for compilers, but we should
occasionally be able to test for compiler capabilities and act
accordingly, by either providing different compiler flags, on
enabling/disabling a certain feature. I'm already not exactly a fan of
testing whether compiler name matches "*clang*". For example, on my
main machine I happily get:

> cc --version
Apple LLVM version 10.0.0 (clang-1000.11.45.5)

> gcc --version
Apple LLVM version 10.0.0 (clang-1000.11.45.5)

Now explain to Apple that this is the wrong way of doing things and
how CC=cc should behave in a different way .... (of course a test for
whether compiler name matches clang is definitely going to fail in
this case.)

But even the OpenBSD system in question would have issues

> cc --version
OpenBSD clang version 5.0.1 (tags/RELEASE_501/final) (based on LLVM 5.0.1)

which means that setting CC=cc to build TeX Live will also fail.


I did a long run of trial-and-error now (did I even mention that I'm
not a huge fan of autotools?) and came up with the following
suggestion, which should probably go inside some "if this is OpenBSD"
loop:



AC_MSG_CHECKING([whether we need additional flags for linking luajit])
AC_LINK_IFELSE(
  [AC_LANG_PROGRAM(
    [[#include "stdint.h"
      typedef struct _Unwind_Context _Unwind_Context;
      extern uintptr_t _Unwind_GetCFA(_Unwind_Context *);]],
    [[_Unwind_Context *ctx;
      _Unwind_GetCFA(ctx);]]
  )],
  [AC_MSG_RESULT([no])],
  [
    jitlibs="-lc++abi -lpthread"
    save_LIBS=$LIBS
    LIBS="$LIBS $jitlibs"
    AC_LINK_IFELSE(
      [AC_LANG_PROGRAM(
        [[#include "stdint.h"
          typedef struct _Unwind_Context _Unwind_Context;
          extern uintptr_t _Unwind_GetCFA(_Unwind_Context *);]],
        [[_Unwind_Context *ctx;
          _Unwind_GetCFA(ctx);]]
      )],
      [
        AC_MSG_RESULT(["$jitlibs"])
        LIBLUAJIT_LDEXTRA="$jitlibs"
      ],
      [AC_MSG_FAILURE([luajit cannot be linked])]
    )
    LIBS=$save_LIBS
  ]
)


I tested this in a hello-world environment, but I can try to write a
full patch for LuaTeX / TeX Live if needed. The code should not be
executed, it's just a test for whether the functions are available.
I'm sure there's a better way to test for this, but I didn't know how
to quickly come up with something valid.

A much simpler alternative that I would also be OK with would be
checking whether __clang__ is defined by the preprocessor.

In any case, this sounds way more robust than comparing the value of
$CC with the hardcoded "clang" only.

One way where this might go wrong is perhaps OpenBSD on exotic
architectures. It could be that the code calling Unwind is #ifdefed
away, but I don't bother enough to try to find suitable hardware,
install OpenBSD, compile etc.

Mojca