Just to be sure which AC3Filter setting ("Main"-tag) you selected:
"3/2 - 5 channels" or "3/2+SW 5.1 channels"?
(Noticed there are also 2 Dolby Surround settings?)
And what is selected as PCM just below that?
And "Use SPDIF"?
Is AC3Filter shown (with an appropriate file loaded) under
(rightclick >) Options > Filters > Advanced ...? resp.
(rightclick >) Options > Preferences > Audio ...(south east corner)?
(Perhaps also check the other AC3Filter-settings under its Mixer/Gains/System-tags)