For shapes other than squares you need to use bezier edit points/point mode, right click the mask and select "Make circle" at the bottom of context menu.
-the jagged edges, i think you are sitting too close.
"warping the image" Are you confusing mask with slice?
good luck