[XeTeX] how to select fonts in math mode in plan TeX?

Sun Mar 27 04:36:40 CEST 2011

When using xelatex I make use of the most excellent mathspec package.
However, I would like to do something similar in just plain xetex.
That is, how do I specify which font to use in math mode?
I tried the obvious which is:
\font\cas="CaslonAntT"
\cas puke %works! puke is typeset in the specified font
$p=\int_0^1 f(x)\cas why is this not in the correct font?$
This, obviously,  is the way one would do it in plain TeX.
How to do the same thing in xetex?
Or, even better, how to globally set the math mode font?