I still believe you are running into the limits of your hardware. Your CPU supports PCIE 3.0, but your motherboard only supports 2.0. This restricts the amount of capture bandwidth you have available.
With the number of cards you have, your only available configuration would be, from top to bottom - 1:mini recorder, 2:5870, 3:(empty, covered by gpu), 4:empty, 5:intensity 4k, 6:empty, 7:mini recorder
Putting the Intensity 4k in slot 5 will cause the GPU in slot 2 to run at x8 instead of x16, also lowering the available capture bandwidth.
My main machine is a ASUS 9X79X WS, which supports PCIE 3.0 or 2.0. I originally had an i7 3930k in there, which has 40 PCIE lanes, but only supports PCIE 2.0. I kept running into low memory errors once I started doing more capturing. I upgraded the CPU to an i7 4930k which support PCIE 3.0 and the memory errors went away.