[XeTeX] how to select fonts in math mode in plan TeX?

[Adam Russell]

When using xelatex I make use of the most excellent mathspec package.
However, I would like to do something similar in just plain xetex.
That is, how do I specify which font to use in math mode?
I tried the obvious which is:
     \font\cas="CaslonAntT"
     \cas puke %works! puke is typeset in the specified font
     $p=\int_0^1 f(x)\cas why is this not in the correct font?$
This, obviously,  is the way one would do it in plain TeX.
How to do the same thing in xetex?
Or, even better, how to globally set the math mode font?