There was some uncertainty as to how to make this work in a discussion regarding this method on the VJ Union Global group on Facebook, so I set about making a more detailed example.
Explicit example
Note: layer 1 is at the bottom of the composition, layer 5 at the top. Start at the bottom and build your way up.
1 - overlay content: whatever you like.
Blend Mode = Alpha, Opacity = 1.0
2 - overlay shape: ordinary video content.
Blend Mode = Multiply, Opacity = 0.5 (full multiply)
Layer effect chain:
- Bright.Contrast - may or may not be required, depending on the clip contents;
- Colorize (Sat 1, Bright 1, Contrast 1) - to make the layer black & white only;
- Auto Mask - to make the black bits transparent.
3 - overlay result: Layer Router (found in the Browser Panel under Sources > Routing) with Input = Composition. This is a kind of "flatten" layer, or the result of the submix (of 1 and 2).
Blend Mode = Alpha, Opacity = 1.0
4 - background: whatever you want the overlay to float above. Make sure that it is not transparent if you sometimes don't want the overlay to be showing, since layers 1,2,3 will show through.
Blend Mode = Alpha, Opacity = 1.0
5 - overlay: Layer Router with Input = Layer 3 (the "flatten" layer). This is the floating overlay!
Blend Mode = Alpha, Opacity = 1.0
Notes
The Blend Mode and Opacity slider for
3 - overlay result don't really matter unless you allow the lower layers to show through layer 4.
The screenshot shows layer
2 - overlay shape in white (with transparent background) in the preview monitor. The output monitor shows this white area filled with the "kewbic flow" loop from Beeple (which is kind of blue background with wavy yellow lines).
Layer
2 - overlay shape is where you would use a live (possibly infra red) camera feed, most likely with some scale, perspective and so on adjustments to make it match the scene, to project a different texture on to a performer in front of a larger background screen.
In this example I put the
Auto Mask effect chain on the layer; in practice, I find it better to place the effect chain on each individual clip in the layer so you can tune the contrast and so on for each clip. Of course, if your mask shape clips are all nice and clean/uniform to start with, a single layer-based effect chain will probably be fine.
If you have a clip (or generative content) that already has alpha transparency, you may find it best to
not apply the
Auto Mask effect. You can get some interesting results from layer
2 - overlay shape having dark and light sections, with the existing alpha channel defining the shape and the lightness allowing for the video content and/or a silhouette/shadow shape (down to black). Non-transparent black is just nice sometimes!
Since making up this example, I decided that a better name for layer 3 would be "submix", or something to that effect. Indeed the whole concept could be referred to as submixing. I left the names as-is to be consistent with the screenshot.
Note on buggy behaviour
While I was recreating this effect as a new composition, I ran into problems getting things to work when switching the layers 1 and 2 about (and changing their blend modes, etc, as required); it seems there is some bug (or are some bugs) that make the result somehow different despite all the relevant parameters apparently being equal. It didn't make sense, and reloading the composition (as saved) seemed to iron out some kink(s).
On this basis I recommend choosing which layer arrangement you are going to use to make the transparent overlay and stick with it.
Cheers,
subpixel.