It sounds like you are using a graphic card that does not pack enough punch to run the Advanced Output.
What does the log report as the main OpenGL Renderer?
You can access the log by going into Resolume > Feedback > View Log and then scroll down to the part where it says 'Main OpenGL Renderer'
There's some screenshots on what to look for
in this thread.