The jagged edges you're seeing are referred to as "aliasing." Many graphics programs have an anti-aliasing filter, but Resolume does not because it would be a huge performance hit (low frame rate).
Photoshop is by far the cleanest way to do masking if you have the time.