So you want to host a Silverlight application (aka a .xap) on a server, making it available to be called from other servers. I just had to do this and figure I'll document the steps. It isn't rocket science, but here's the story.

First, you need to place a clientaccesspolicy.xml or a crossdomain.xml file at the root of your server. Read this MSDN documentation for more.

Then, you need to set the enableHTMLAccess attribute to true in the object tag:

<param name="enableHtmlAccess" value="true"   />

Lastly, make sure your server is sending the right mime type for the .xap (application/x-silverlight-app). Tip o' the hat to Joe Field for that one.

And, with that, walla -- you are good to go.  Here's an example of a music player being served from where I'm dynamically passing in the URL to the .mp3, which sits entirely on another server:

Get Microsoft Silverlight