I got it to work like this under XP...
I have an ASUS K8VSE motherboard with onboard Soundmax audio and a co-ax SPDIF output. Getting it to work reliably was really hard, and I could never get it to work at all in Windows media Player MP or Media Centre Edition because (I think) they use the DirectSound Audio Renderer by default and the ASUS Soundmax drivers only output SPDIF to the Waveout audio renderer (there are Via drivers that work instead, but they gave me other problems).
Anyway, this is the most reliable way I've got it to work (Graphedit is also a really useful tool to see what codecs / renderers are being inserted invisibly into the play chain!)
First, uninstall unwanted codecs (Cyberlink, Intervideo, AC3Filter, XVid, anything else you've installed) - the "cleaner" it is the better it works!
1. Install NVIDIA PureVideo decoder - I have the platinum version that includes SPDIF support, I'm not sure all versions do.
2. Download and install FFDShow codec (primarily for MPEG4/Xvid/DIVx). Don't worry too much about the options you use during install, we'll change them later.
3. In GBPVR Config Playback tab, set the Video codec to NVIDIA, the Audio codec to FFDShow and the audio renderer to Default Waveout device (or your soundcard driver Waveout if listed)
4. In GBPVR DVD tab, set Video codec to NVIDIA, Audio codec to NVIDIA and audio renderer to Defaullt Waveout device (or your soundcard driver Waveout if listed)
5. Go to Start, Programs, FFDShow, Audio Decoder Settings
a. Under the Codecs tab, set AC3 and DTS to SPDIF. If you want MP1/MP2/MP3 passed through (so you can let your amp recode surroundsound if required), set them to "disabled". This is what I prefer, but if you want to use use FFDShow to recode them into different surroundsound formats, leave them enabled (see c. below).
b. Under the Resample tab, turn resampling on and set it to 48KHz (98KHz may also work)
c. Under the Mixer tab, you can either leave it off, or turn it on and set the type to "same as input" to pass through to the amp. (You can play around with the options here and the codec settings you have enabled, if you want FFDShow to remix into different surroundsound formats.)
d. Under the Output tab, check the "AC3" box.
6. In your Soundcard drivers, make sure the soundcard output is set to SPDIF and AC3. How you do this will vary from soundcard to soundcard.
7. Start GBPVR / PVRX2 in windowed mode (not -FSE for the next bit!).
8. Play a DVD from DVD drive or Hard Drive files. The NVidia codec icon should pop up on the taskbar. Right click it to enter configuration mode.
a. For Video codec configuration, make sure hardware acceleration is on and choose VRM9 output as default. (I also find setting deinterlacing to "Auto" rather than "Smart" gives less jitter / tearing, don't know why)
b. For Audio codec, set it to Receiver / SPDIF mode.
9. Stop the DVD and play an AVI (MPEG4/DIVx/XVid). The FFDShow video and audio codec icons should pop up on the taskbar. If you need to "tweak" any settings, you can do it by righ clicking these.
10. Now stop GBPVR, restart the PC (not really necessary but always a good idea). SPDIF *should* now be working. Make sure GBPVR is set back to start -FSE if you changed it earlier.
If it's all working properly, DVDs should use the NVidia audio decoder and pass AC3/DTS through to the amplified via SPDIF. MP3 and MPeg 1/2 should either pass through to SPDIF unchanged, or use the FFDShow audio codec (depending on how you set it).
Give this a try anyway, and post back results - I'll try to help troubleshoot if I can.
P.S. If you have video jitter / tearing, always start GBPVR with the the -FSE switch; in your video card settings, for jitter reduce antialiasing (or turn it off), also anisotropic filtering; for tearing, force horizontal sync on.
Near-silent budget HTPC: X-case w/ ACBel 400W quiet PSU, 2x undervolted (5V) 80mm case fans, ASUS M2A-VM-HDMI, silent Asus nVidia 9400GT, AMD X2 4800+, 2GB Corsair DDR800, Windows XP/SP3, Hauppauge HVR-900 hybrid tuner + PVR150MCE analog, GBPVR PVRX2 (latest), CyberLink H.264 PDVD(7), Monogram AAC, FFDShow video & audio, VMR9-FSE
HDMI -> Panasonic PTAX100E 100" 720p/1080i projector, S-Video -> 27" CRT TV, SP/DIF -> Denon AVR3805 (7.1) + Q-Acoustics 1030 / 1010 speakers, Harmony 525 remote