Replacing the ICU-4.6 LESwaps.h code with the ICU-4.4 version does result in a xetex binary that can read Zapfinfo without crashing. (Whether it can actually typeset anything correctly, I don't know.) It doesn't seem like the right solution to me, but at least it's a possibility. k