Automation 101: The Dangers Of Hand/Off/Auto

When we automate we invariably offer our clients and their operations staff the benefit of Hand/Off/Auto functionality for all the components on our systems. To those new to automation it allows an operator or maintenance worker to remove a piece of equipment, like a pump or blower for example, from the oversight and control automated controller (MCC, PLC, etc.) and operate it manually while it is still installed. It also gives them the ability to turn the component off altogether. We need this option to be available for commissioning as well as for maintenance. Having the freedom to shut down a step in the process, or simply remove it from the automated process, enables the operators to perform essential tasks. It is also a nice safety feature. It gives people on the ground a position of authority and autonomy in your preprogrammed process.

But with great power comes great responsibility. Hand/Off/Auto is not an exception. What we need to be constantly aware of is that we have a complete, integrated, interrelated system. We are never simply working on a pump. We are working on a pump, in a system. Every action that we perform on a piece of equipment is a type of industrial open heart surgery. Literally everything that we do, even to the smallest individual component, we are actually doing to the whole system.

A pump for example is pumping something from somewhere to somewhere else. It is also pumping through pipes, hoses, valves, heat exchanges, and various types of instrumentation. Those two sentences alone should make you cringe in the light of this discussion.

Imagine if you will that you have a pump that draws from a tank. It pushes that water through an ORP sensor and that sensor activates a bypass valve and a corresponding isolation valve, to send chlorinated water to drain. If the sensor does not activate, it will send your chlorinated water through your cartridge filter to feed your high pressure pumps on the reverse osmosis (RO) system. The tank will also have a butterfly valve to allow the pump to be removed for service. Now there are two ways of handling this dilemma. The first — and unfortunately the most common and costly — is to assume that the operator is fully trained and responsible and incapable of making a mistake. Good luck with that.

OK, now that we have set a small stage, let’s watch the play. What could go wrong in “Hand” mode with this pump in this system? You could drain the tank. It no doubt has a low level switch, but that doesn’t affect “Hand” mode. We have removed the component from the control circuit, but not from the physical process. Draining the tank down would create a vortex that will cavitate the pump. Maybe we drain it completely dry. You could forget to open a valve and cavitate the pump that way. You could have a high chlorine level and blow that salty water straight into your RO filters, which for those who do not work with RO, is deadly to them. You could be dumping feed water to drain at a ridiculous rate. You could blow your ORP sensor dry and lose it. There are plenty of other possibilities, especially in the complexity of a real system.

None of these problems are trivial, yet all can very easily happen if we do not put checks in place to protect our systems. Any one of these problems could shut down our line and cost a great deal of time and money in repairs. We can rely on the operators. They do good work, but they can miss something, especially in the hectic mayhem of trying to get a system back online that has gone down, or simply rushing through a maintenance cycle so they can meet production requirements.

So there is a second option, but it is a little more complicated. We can program our systems so that “Hand” mode is not exactly “hand” mode. We do not actually remove the component from the process. We really just remove the controller’s ability to start the component. We instead allow a manual start, within the parameters of the acceptable conditions of the system. We are still isolating the component so that a system start up, or attempt at it, will not energize the component. The operator or tradesman can manually energize the component as before. The only difference is that we will monitor the existing sensors. And with that feedback we will prevent anyone from initiating an action that will cause damage or harm to the component or the system. This will prevent the above problems.

You will also that this point need a specific set of alarms dedicated to the maintenance cycle. When the operator tries to bump the pump, and nothing happens, he/she has to be able to tell immediately whether there is an unsafe condition preventing the pump from energizing, or if it is a problem with the pump, VFD, motor, starter, etc. that is causing the issue. Seasoned operators will have an issue with this, but that is covered a couple of paragraphs down.

The simplest way to alleviate frustrations is to have a green light/red light signal at the component site. It will inform the operator at a glance if the system is in a safe mode to operate the component — a green light for a no-alarm, safe condition. If an alarm has been activated, like a low level switch for example, a red light would turn on. The operator can then easily determine from the alarm condition what his/her actions should be. The HMI at the panel must also have a more detailed alarm condition, however.

The detailed alarm condition must be preplanned. The programmed needs to identify all the conditions that make operation of the component undesirable and then identify which sensors will deliver that information to the controller. The HMI screen must then display those sensors and alarms to the operator.

The issue I mentioned above is: what if you want to bump a pump, motor, etc. to ensure that it is hooked up properly, running in the right direction, etc.? Bumping a motor for a split second will not cause many of the above problems. This is where the detailed screen comes into play. An operator that knows his system will be able to look at the screen and determine from the data if an alarm can be ignored for a particular event, such as bumping a motor. Then he/she will be able to override it with a push of a button. But it will be a conscious decision to do so.

However, what we really have is an option between fail-safe and fail, and hope for the best. Allowing full autonomous, isolated, manual control of a component while it is integrated into a system is “fail.” Any one of a number of errors that the operator may not be aware of can bring catastrophe.

Creating operational limits and alarms on components in the “Hand” position of the Hand/Off/Auto function produces a fail-safe functionality in your system. It actually increases your control of your components and system while providing an opportunity for data logging of maintenance events. You no longer force your operators to memorize every detail of everything that can go wrong and run around turning valves and flicking switches hoping to get them all right. Now you allow the pre-existing, fully automated and monitored system to do that for you, and your operator only has to make educated decisions as to the state and function of the alarms.

You do not need to add sensors. You do not need extra hardware or inputs on your PLC. You are simply taking what you already have and are using to operate your system and putting it to work for you in a different capacity. The small, upfront programming involved will be paid for in operating time and complexity alone. The Hand/Off/Auto functionality becomes simple and quick. Not having a major component prematurely and cataclysmically die at the absolute wrong time, well that is an added bonus.