You have the same problem as I do.
I've tried every possible rendering mode and with several BS.Player-versions yesterday and to-day.
If I use anything else then "Overlay mixer" the screen goes black (it does appear to step frame-by-frame, but only doesn't show it).
The only BSPlayer version which will happily and without any problem (and using the same DivX codec) do frame-by-frame stepping instead of stepping by seconds is (as I wrote before) version 0.86.501.
(You can install this version without any problem together with BSplayer 1.x or 2.x versions by the way; I have!)
(P.S.: I'm using Geforce4/Nvidia)