First you must have an AC3 filter installed
The easiest way is to install the one you'll find at doom9's (
www.doom9.net -> downloads -> filters -> audiofilters.zip ; [register iviaudio.ax and mpgaudio.ax only, you don't need mmswitch])
It is intervideo's AC3 filter
then you can set it up as you like in bsplayer's preferences (and when playing, right-click->properties->intervideo audio filter)
(I was told recently about another ac3 filter which can be found at doom9's too but I didn't have time to try it yet)