No overlay means that all overlay modes or left unchecked, is it that complicate?
In overlay mode, the picture is directly written in the video card memory, this is a write only process, so that no application can read it back.
I just tested the feature on one movie in full screen with no overlay and with Shift+P I got a 1280x533 JPG. [Alt+3] tells me it's a 576x240 DivX3 movie (that use YV12 surface).
PS: Maybe it doesn't work on every movies.
For instance I had to check Use-VMR9 to be able to take a screeshot of a DivX5 movie wich surface was "Unknown". Then I could only get a BMP and of the original size.
You have to make you own investigation and test on several movies. I thing that the main concern is the surface used for encoding the video.
TIP: When no overlay is selected, then you can use the [PrtScr] key to take a screenshot or [Alt+PrtScr] to restrict to current window. And paste the picture in your favorite photo editor. If in full screen, you'll have to crop the picture as black stripes will be included.[/i]