It *IS* possible to do this in Resolume. You need at least 5 layers to do it (including one "background" content layer).
1. Matte shape layer(s). Start with ARGB content, or use an effect such as Auto Mask (which maps luma to the alpha channel) or the Luma as Alpha blend mode (requires more than one layer).
2. Content layer (use Multiply or similalr to cut out to the shape of the matte).
3. Layer Router layer. This is a kind of "flatten" layer. Set the source to "Composition" (which is your cut out content from the previous layers).
4. Background layer(s). Whatever you want the cut-out content to float above. Note that you probably want to make sure that you have something "solid" so your matte setup layers below don't show through. You can stick in a solid black layer for safety if you like.
5. Layer Router layer. This is the overlay! Set the source to the other Layer Router layer (whatever "3" is).