When SPDIF is enabled the signal must go thru an external decoder before it comes to the speakers.
You do not get any sound because you are sending a digital signal to the speakers.
When you choose "2 Channel Stereo" AC3Filter decodes the signal and puts out an analog signal from your soundcard to the speakers.
I am not sure what you meen with AC3 Amplifier but i'll guess it is a amlifier with Dolby Digital decoder (just what you need
).
Then if you want to use SPDIF the cable must be connected to a digital input on your Amplifier.
Perhaps you have to choose to decode Dolby Digital on your Amplifier but normally it changes by it self.
The cable you are using know is for analog signals and has a electrical resistence of 50 Ohm and you need a 75 Ohm cable for digital signals.
I hope there is something useful for you here or maybe I misunderstood your problem.
English isn't my native language so I hope you'll understand me :)
Bye and good luck!!