Looks like it is: /composition/layers/2/maskmode int 0-2
I'm not sure if it's official though, so might change in the future.
Software developer, Sound Engineer,
Control Your show with ”Enter” - multiple Resolume servers at once - SMPTE/MTC column launch
try for free: http://programs.palffyzoltan.hu