I'm assuming you are talking about creating a round mask in photoshop with transparency and use it on the top layer of your composition, right? You can also create a crop layer in [Advanced Output], select [Bezier] under [Point Mode] of the [Edit Points] in the output transformation and right click ...