Is really hard to undestand what's wrong there.
Are you using Slice Transform on your clip, your comp, wich slice and mask working on it?
I think you're trying to use the Advanced Output to mix stacked layers. In that case, try to make all your layer mixes and masking at composition level.
Set a Slice Transform on your clip/layer with the mask/slice you need, and turn it off from advanced out.