I assume you tried different choices of "For AC3 decoding use " and "Output device" on BSPlayer's Audio-preferences "main-page"?
(Also: settings of AC3filter ("Main" and "Mixer"-tabs) re Output/Speakers were correct?; also check options under AC3filter's "System"-tab)