Do you have a 5.1 soundcard and a 5.1 speakers hooked up to that soundcard, or are you trying to send the digital audio signal to an external decoder, like a home theater system?
Either way, you need to install either ac3filter 0.7 or higher or intervideo audio decoder 2.0 or higher. You can download ac3filter at
www.doom9.org, and the intervideo audio decoder is found in a lot of
codec packs such as Nimo codec pack or Tsumami codec pack--both of which are found on a lot of sites including
www.divx-digest.com.
BSPlayer has built in configuration options for intervideo audio decoder, right-click player>options>
preferences>intervideo AC3. There, you can choose either 6 speakers or SPDIF out for an external audio decoder.
For ac3filter, go to start menu>ac3filter>configuration, and you will see
the speakers options in the lower left corner, starting with 2 speaker mono (I think). Version 1.00 (I think) handles DTS.
If your soundcard does handle AC3, and you want to passthrough AC3 to an external audio decoder, there should be an option for digital our that you need to check (like mine, which is a Soundblaster Live 5.1 card).
Hope that helps