Troubleshooting Signal Loss

Post your questions here and we'll all try to help.
Post Reply
User avatar
Wasko
Hasn't felt like this about software in a long time
Posts: 58
Joined: Wed Sep 23, 2015 00:20

Troubleshooting Signal Loss

Post by Wasko » Sun Feb 05, 2017 17:06

Last night I had a projector lose signal randomly during an event. After spending some time frantically trying to trouble shoot the problem, I opted to disconnect another part of the installation that was more decorative and less crucial and re-map the center stage through it's output. The show went on, but the excruciating few minutes of bluescreen at peak-show time has got me thinking about redundancy and how to make my rig reliable especially for larger scale, high expectation events.

How do you go about trouble shooting signal loss?
How do you ensure that problems like this don't happen to you?
What are some best practises for redundancy and ensuring reliable performance?

I have a suspicion the issue might be with a port on my GPU, but I am not sure how to test this and I am having trouble re-creating the problem. This might be too vague.

I want to maintain a system that is reliable and have peace of mind this issue won't arise again, but am unsure of the steps to take to get there.

Zoltán
Team Resolume
Posts: 3916
Joined: Thu Jan 09, 2014 13:08
Location: Székesfehérvár, Hungary

Re: Troubleshooting Signal Loss

Post by Zoltán » Sun Feb 05, 2017 17:43

that is a hard one.

In my experience the first to check is the cable you are using,

Your video card connector could also have loose pins.

If you play audio too it's best to use DI boxes, they can separate you from the audio equipment ground, preventing ground loops, the sound guy will surely have 2 channels of DI for you.

Make sure people don't step on long cables, as the change in cable geometry can cause changes in cable capacity, which can lead to signal distortion that your end equipment may not be able to handle.

Then make sure the grounding of the projector is happening only on the power cable/signal cable and they have the same grounding point, preferably at the same point where your pc is grounded.
I mean, not to use the wall outlets randomly at the venue for power, but run your power cables separately from a given point in a tree topology. Ground loops can cause your pieces of equipment to be at different ground potentials.
Also if you are rigging your projector on truss, then the high power switching of the lighting equipment can cause the ground voltage to shift, and if there is a galvanic connection to your projector's ground, then that shift will pull all your equipment. Especially if there is a faulty/wet lamp, with leaking current to ground.
(Once I worked at an open air event where the sound power came from one place, and the lighting power from an other connector 30 meters from the audio connector, I had to run some antenna cables for wireless audio equipment to a lighting truss, and the voltage difference between the antenna ground and the lighting ground/truss ground gave me a small shock, it was about 70VAC, and nothing was wet. )

Make sure you use a stable power source, surges or undervoltages can cause problems.
For example if you are on the same power outlet where 10 kW of lighting is switched on and off and you are on a relatively small gauge/long cable then the impedance of the cable will cause the end voltage to drop when the high load lamps are switched on.
Using an online ups will help with voltage stability, but only if you run separate power cables.

Back, when we used composite/analogue signals you could see streaks running in the picture, which was a clear indication for a ground loop. You don't get those any more with digital signals, but the loops can be there.

and there are surely other factors that didn't come to my mind now.
Theatre sound engineer, photographer, programmer, aka ravensc.
Control Your show with ”Enter” - multiple Resolume servers at once - SMPTE/MTC column launch
try for free: http://programs.palffyzoltan.hu
VJ footage: https://goo.gl/58oHy7

User avatar
Oaktown
Resolume honorary member
Posts: 2817
Joined: Tue May 08, 2012 15:19
Location: Oakland, CA

Re: Troubleshooting Signal Loss

Post by Oaktown » Sun Feb 05, 2017 17:51

Redundancy is expensive and creates complex installations because ideally in order to achieve close to 100% redundancy, you'd have to run two complete independent systems side by side that can be updated on the fly.

In most cases, that kind of setup is not achievable because of time and budget constraints!

So the next best thing is to identify what can go wrong in your system and have a plan in place to handle equipment failure. In my experience the two most likely places you will encounter failure will be your media server and your projectors so here is what we do to reduce the potential for failure.

Media server:
- Every server going out is always thoroughly tested before it goes out and our system are typically designed as media servers not computers that handle video as well.
- Never update a system or software right before a show.
- Make sure all your cables are securely connected to your computer. For instance I prefer to use DVI and DisplayPort than HDMI and MiniDP because the latter don't latch. So we always tie and tape everything connected so that it can come off.
- Monitor your system during the show so that you can be aware of usage and temperature problems. For instance if your GPU's temperature starts rising, you can manually force the fan to 100% and make sure you have proper ventilation around your system. Also, liquid-cooled GPU and CPUs are much safer than air cooled.
- Make sure you set your desktop to black.
- I also always make sure my content outputs are grouped together and located in the top right corner of my UI so that we don't see accidental pointers and window edges in the projections.
- I always use a power back up and test it before the show by unplugging the system. Make sure you turn the beeping of the power back up off!

Projectors:
- I never use single lamp projectors so all projectors we use have 2 or 4 lamps.
- Make sure you setup your projectors properly before every show. Here is the short list of what I check before every show: change blue background to black, remove logos on startup, turn off the sleep mode and set the projector to start on power
- I always use the native resolution so that it doesn't accidentally switch to a different resolution
- Make sure your cables are securely connected, tied and tapes.
- Make sure you get a dedicated non-dim circuit and don't overload your circuits.
- If you can, try to have one dedicated circuit per projector.
- Don't run your projectors off a dimmer pack set to non-dim profile board on the lighting board.
- Check the projector specs and use the max power consumption as your basis for calculations and assume an 80% max load.
- Check air filters before every show
- Check the hours on lamps and swap older lamps with new ones. I typically don't like to use projectors that have lamps older than 50% of estimated life span.
- If you have the budget, if you have to change a lamp, put all new matching lamps.
- All lamps are not created equal so always buy good lamps from reputable vendors. If the OEM lamp costs $500 and you paid $80 for your on eBay, chances are you are getting a bad lamp
- Always make sure you follow manufacturer's guidelines as far as mounting your projectors. Some units can't be mounted at serious angles and some units have different lamp requirements for landscape and portrait orientation
- Whenever possible, run your projectors on a network so that you can troubleshoot from a web browser without having to have a menu on the screen.
- Of course if you have the budget, set up a backup projector next to your primary one and make sure it's focused and ready to go. The best way to make this work is to split the signal into both projectors and douse the backup one either internally or mechanically.

Other tidbits:
- Always check your set up before the audience comes in.
- Never assume other teams (lighting, rigging, set, catering, etc...) have your back so double check and triple check
- Make sure you run all your cables away from foot traffic inasmuch as possible.
- Whenever possible, run a backup signal cable to your projector
- Use power conditioners for your server and broadcast gear
- Clean up your work station so that you can easily identify cables.
- Make sure every cable run has some slack on both ends
- Label all cables on both sides.
- Make sure all the technicians that are working on your gear know your gear and have been informed of your backup plan
- Whenever possible, use Gefen Detective units sot that your GPU doesn't resync when one screen gets accidentally unplugged.
- Make no one leaves any food or water near your station at any time.
- Always choose wired connections over wireless.
- Don't use wireless signal unless there is no other way but in my experience, there is always another way!
- I you use wireless, bring your own dedicated network.
- Check every cable before every show and I mean each and every one of them :-)

I'm sure there are many other things we do that is so ingrained in our workflow that I can't think of it so this by no mean an exhaustive list.

I'll be interested to hear what other people do.

User avatar
Wasko
Hasn't felt like this about software in a long time
Posts: 58
Joined: Wed Sep 23, 2015 00:20

Re: Troubleshooting Signal Loss

Post by Wasko » Sun Feb 05, 2017 19:05

So many useful tips here and things that I had never considered that will significantly up my game and rectify some of this problem. Thank you. I can't believe I hadn't thought to set my desktop and projector backgrounds to black, that is an essential piece of advice.

I just returned from striking the show. I realized that the projector in question was plugged into a power source that was also supplying power to the moving heads on a truss beam. Having read all this I am now thinking that the issue may have been inconsistent power supply to the projector because of the lights drawing power full force. This would explain the fact that everything was running smooth until halfway through the night when the lighting was really starting to pick up.

Ravensc - you mentioned having my projectors all stemming from the same power source in a tree topology. Are you suggesting that it would be more full-proof to have my computer and projectors all powered from the same wall outlet? I would have to ensure that the outlet is providing adequate power beforehand of course.

And I was going to make another thread for this but I will leave it here:


Is there a way to 'lock' your outputs to their appropriate screen / projector?
More specifically, I want to lock my toolbar, Resolume window, etc to my monitor / UI.
Every time I unplug or replug a connection is a gamble because I may or may not lose my interface to another output (a projector that I can't see to re-select main output, for instance :( ). The only guarantee I have that they will go where they are intended is when I first connect in sequential order. This could potentially leave me in a difficult situation in times like these where disconnecting or re-connecting an output is necessary.

Oaktown - is this function of the Gefen device you suggested? Is there a way to lock my UI output in windows 7?

Thanks again.

Zoltán
Team Resolume
Posts: 3916
Joined: Thu Jan 09, 2014 13:08
Location: Székesfehérvár, Hungary

Re: Troubleshooting Signal Loss

Post by Zoltán » Sun Feb 05, 2017 21:15

Wasko wrote: Ravensc - you mentioned having my projectors all stemming from the same power source in a tree topology. Are you suggesting that it would be more full-proof to have my computer and projectors all powered from the same wall outlet? I would have to ensure that the outlet is providing adequate power beforehand of course.
yes, obviously you'll need some kind of distribution, and I don't mean, that for example for 3 projectors on a truss, say 4 meters away you must have 3 complete cable runs, you can have one proper gauge cable run up the truss, then split 3 ways on the truss.
Having separate cabling adds to redundancy of course, but also cost and weight.

Running and knowing your own cables, means more work but it will make you independent from others, which simplifies troubleshooting, and adds a kind of redundancy to the whole power system.
The show may be able to go on if the projectors fall out, but the lighting and sound continues to work.
Oaktown wrote:- Never update a system or software right before a show.
adding, I never update a system that works as expected, ever.
I also run thru my shows, play every single cue, to check that all the media is present and playable, and all the software/midi connections work.
I also have a white flash+sound cue in the playlist for every show, that I launch just before the audience is allowed in, for a fast-check.
On my work laptop I have to switch the WD drive's head parking off on every single startup.

Don't trust one drive to hold your content/system, use a raid 1/5/6 if possible, yes even on SSD-s.
Never use raid 0 for important data, I mean use raid 0 for data you want to lose.
Raid is not a backup, if you accidentally delete something from any raid, consider it gone.
If you build a production pc, build it like you'd build a server, redundant power supply, a good raid, lots of ventilation, redundant network cards, sturdy rack case, ECC ram, etc.

Wiggling cables before the show can show you lose contacts.

yes, having black backgrounds is essential :)

I use velcro to fasten cables, I don't like tape, it gets messy. Also we already have a velcro on every cable, that holds them together when stored.
If you need to extend cables, a fast way to help them stay together is making a tie on them, if you don't have velcro or tape at hand.
18fbo3a935cu8jpg.jpg
18fbo3a935cu8jpg.jpg (38.21 KiB) Viewed 2312 times
Always have velcro or tape at hand :) as tying puts stress on the cable, and will wear out the cable at the connector.
Use the tape or velcro so, that the connectors don't hold weight, when the cable is pulled, for example plug the connectors, then make a loop and secure the cables together.

Never bend cables too much, even when reeling them up, use light force, if you are putting much force to a cable when reeling up you are reeling it up too narrow.
Running a cable upwards, secure it if you can so it doesn't have to hold its own weight.
Never step on the connector of the cables.
Keep cables and connectors clean and dry.
These can help to maintain the health of your cables.

And for the case that something goes wrong, keep calm, if you rush things you can make an error more easily.
Also don't run if the audience can see you, it will help them notice that something is not as planned.
Theatre sound engineer, photographer, programmer, aka ravensc.
Control Your show with ”Enter” - multiple Resolume servers at once - SMPTE/MTC column launch
try for free: http://programs.palffyzoltan.hu
VJ footage: https://goo.gl/58oHy7

t13swift
Hasn't felt like this about software in a long time
Posts: 149
Joined: Thu Mar 26, 2015 14:00

Re: Troubleshooting Signal Loss

Post by t13swift » Mon Feb 06, 2017 10:17

Another thing I thought I should add is that if you are running video (HDMI etc) over ethernet extenders, be sure to use shielded cables especially if they run anywhere near 3 phase power cables.

User avatar
Oaktown
Resolume honorary member
Posts: 2817
Joined: Tue May 08, 2012 15:19
Location: Oakland, CA

Re: Troubleshooting Signal Loss

Post by Oaktown » Mon Feb 06, 2017 15:20

Speaking of cables, keep in mind that the maximum recommended length for HDMI/DVI is 15ft so anything beyond that is risky. I use some high end 50ft DL/DVI & HDMI cables that work great especially paired with a Datapath X4 because of the built amp but I don't use them on shows that really matter! If you need to run long runs of HDMI/DVI cables, switch to coax, ethernet or fiber optic distribution or use amps (but only if you have to).

3G-SDI is super reliable but limited to SMPTE resolution so if you use and computer resolutions, it's not an option. If you do long runs of 3G-SDI, make sure you use high end cables & connectors and reclock your signal every 300ft

I'm a big fan of HDBaseT and so far I've had zero failure with it (knock on wood). Just like t13swift recommended, always use shielded cables and unless you really have to, try to stay away from connecting barrels. All of our cables are either Cat5e or Cat6a high end cables.

Typicallly for cable runs over 300ft, I'll use fiber optic. It's expensive but completely oblivious to any interference. Make sure you use tactical fiber optic cables or military grade.

And like I mentioned before, every cable we use in a show is test before it goes up.

Another thing you can do if you have time in pre-production is to work with the lighting team and have them add a few lights to cover your screens with patterns/colors so that in the event of equipment failure they can make your large blank canvas disappear into the show.

Post Reply