Under (rightclick >) Options > Preferences > Audio
what did you select at "For AC3 recording use"?
Please check what filter BSPlayer actually uses (with a file loaded: rightclick > Options > Filters > Advanced)
You can influence that using filter management
(rightclick > Options > Preferences > Filter management)
(If AC3 filter: what version of AC3 filter are you using? I know there have been post 0.70b versions with buggy SPDIF. I still use good old 0.70b; what is setting of filter merit at its "system" tab?)