Software Overhead or Hardware Overhead?

[admiralperpetual]

Hi there,

Have a new "extreme" test system we put together in the shop to see how much we could push things a bit and having some interesting results.

Here's the specs:

Software: Resolume Arena 5.1.4
CPU: Intel Xeon E5-1620v4 (40 PCI-E lanes)
Motherboard: Gigabyte X99-UD3P
Ram: 32GB DDR4 (quad-channel)
GPU: EVGA GeForce GTX 1080ti Founders Edition 11GB
I/O: 3x BlackMagic DeckLink Quad2 SDI cards (24 outputs total)
Storage: 2x WD Blue 1TB SATA3 SSD in Raid-0 (can't spare the pci-e lanes for m.2 in this case)
OS: Windows 10 Home 64bit w/all updates (game mode disabled)

So, with all 3 of the Decklink Quad2 cards installed and 24-outs going from Resolume we get about 18fps at 720p60. I'm not really suprised, as it's a lot for the GPU to render and move across the PCI-E lanes (keep in mind all cards have the full 16x/8x lanes available because of the CPU).

However, when I check hardware managers I'm only seeing about 40-40% GPU usage, 25-40% CPU usage, and super low read-usage from the SSD RAID (though I find that sort of hard to believe as they are reading many separate DXV files).. is this all due to the bottle-neck of the GPU needing to move everything over the PCI-E lanes to the Decklink cards, or some sort of software bottleneck? I have gone through the usual Nvidia settings and set everything to max performance, vsync on or off doesn't seem to make much difference.

Interesting thing is, effects or resolume settings (such as "disable global effects") don't seem to make much difference. When only using 16-outputs the frame rate is about 35fps and turning on lots of effects only drops it to 33fps for example.

So, any input would be nice. For now I'll plan to only use 16 outs as that seems to be fine, but would be nice to use all 24!

thanks!

[Daniel]

Interesting test, thx for sharing.

I noticed this before, maybe it works with you as well. I had a laptop previously that was getting low fps (~30fps) after 20 HD layers. However when I added 3 more HD layers, than the fps spiked up again.

I never knew and didn't investigate why. What if you add a couple more layers? Maybe the fps will go up again? Would love to see you test that.

Also, if you have 16 outputs, I doubt you will be using 720p60 footage. What if you import a footage that uses the resolution of 16 outputs? At least use 8k

[Menno]

Unexpected low harddisk rates are usually caused by the os' File Caching mechanism. The OS detects which files are often used and puts them into free ram. You probably have lots of free ram with that 32GB, if you aren't running memory intensive applications next to resolume.
You'll see disk io go up if you make resolume use files that havn't been used for a while (trigger different clips), or just use very long videos that dont fit into free ram

The low cpu / gpu usage you're seeing has two causes. First these programs tend to show you an average over a certain amount of time. Resolume pretty much uses each hardware resource sequentially. So when we're using the cpu, the gpu will be idle. This idle time shows up in your measurement too.
The second cause is that they show an average for the entire system. Your cpu is built for multitasking, it can do 8 things at the same time. Resolume doesn't have 8 things it can do in parallel all the time, so a few of your cores/hyperthreads are just snoozing while the others are running as fast as they can. (you probably wont see this in your tool as the os switches threads between cores to distribute heat, so it'll show as averaged out between multiple cores) This comes down to resolume 5 preferring 4 fast cores over 8 slow cores, yours arent slow so you should be okay there.

In the end it's always hard to pinpoint directly where performance issues are coming from and it's always a combination between software and hardware.
If i have one gues concerning your hardware, it would be memory bandwidth. The video frames dont travel from gpu directly to the capture card. Instead they travel from gpu all the way to ram, and then from ram all the way to the capture card. It's actually pretty hard to saturate a PCI3x16 bus, but it's a lot easier to saturate the ram io. Since you said you have no disk io, i expect the os to use file caching, adding to the burden of the ram usage.
I havn't found a tool yet which supports proper memory profiling, so it's hard to actually verify if this is your bottleneck. If you find a good tool please let us know

[admiralperpetual]

Interesting!

I hadn't thought about all the videos being loaded into ram, makes sense.. I'll see if increasing the ram speed makes any sort of difference. Using the DeckLink outputs is much nicer for us since we don't need to worry about Windows moving the screens around or freaking out if it loses the connection with one of the monitors or anything like that. We do have a DeckLink Duo2 card as well, so we may try using 2x of the Quad2 and 1x of the Duo2 for 20 sdi outputs total and see if that stays above 24fps (the magic number for us).

thanks!

[admiralperpetual]

little update:

unfortunately I was already running at the max ram speed my bios lets me reach (2400mhz)..

I tried overclocking the CPU/GPU a bit, upping the CPU to 3.8ghz helped a tiny bit - seems upping the GPU makes no difference at all though - even enabling "K" mode in evga precision OCX to force max clock speeds and then boosting the core to 2100mhz and ram to 3900mhz. was tempted to try a GTX TITAN Xp but I doubt it would help much if OCing the 1080ti doesn't help.

too bad!

[lightbx]

It seems obvious to me! The low fps you are seeing is a result from sending video to 24 simultaneous outputs over 3 Decklink Quad cards (ha ha ha! ... )

[admiralperpetual]

It seems obvious to me! The low fps you are seeing is a result from sending video to 24 simultaneous outputs over 3 Decklink Quad cards (ha ha ha! ... )

dare to dream!

but yes, would make more sense if we were seeing anything in the system being "maxed out".. I am a bit surprised that we can get away with 20-outs.

[massta]

That's a lot of outputs. If you drop the # of outputs, does that help?

I'm running Resolume 6.0.3 and see significant frame rate drop from 50+ fps to 35fps when outputting 5 slices with edge blending to 5 projectors. This is just one layer of 2.5k x 2.5k pixels! Clips range from 1-4 minutes each, have about 10 movies on the layer. When I turn on a 2nd layer above it with a small movie on loop, I drop another 5-6 fps. This is still above 24fps and acceptable but man is it close. I'm running very little clip Fx for color adjustments. Sizing is scaled in some, so thinking of rendering them to exact size to free up Resolume doing it.
My CPU is running at 70%, might try overclocking it. I'll check ram speed also. Using NVidia NVS 810.

[Oaktown]

Using NVidia NVS 810

I think that's your bottleneck right there. Take a look at this thread.