Get Smart & Move From Preventive to Predictive Maintenance

Nelson Baxter, ABM Technical Services CEO, engages with Rafael Reyes, Everactive Director of Product Marketing, on a vibrant (no pun intended) overview on maintenance philosophies, its current challenges, and where the future of predictive maintenance is heading.

Okay, I started working in vibration analysis over 40 years ago, primarily in utilities and power plants, both fossil and nuclear. I’ve got a bachelor’s degree in Mechanical Engineering and a master’s degree in Nuclear Engineering. I worked in a nuclear project group, and worked on a nuclear project. In 1984, I started my own business and I’ve been doing vibration testing ever since.

I would go out to all kinds of different plants, pretty much within the manufacturing industry – a lot of power plants, like gas turbines, steam turbines, and nuclear power plants. I give advice on the reactor coolant pumps and turbine vibration problems. But, my primary business is to go out once a month and monitor machines to give people early warning of problems that are developing. Two primary reasons, which sound a little legalistic, are to reduce what are called consequential damages and consequential downtime.

Consequential damages are real simple – if you have a $500 bearing and you do not repair that $500 bearing, if it fails and wipes out the rotor, or wipes out the stator of a major motor, for instance, the $500 thing that had been maybe fixed in four hours, now becomes a $50,000 item and it takes six weeks to get the motor repaired. So, that’s one aspect to be considered regarding predictive maintenance, and that’s what a lot of people think about.

However, the really important thing to consider is consequential downtime. For most organizations, the consequential downtime may be 10 or 100 times the cost of the consequential damages. If you take a whole line down in a manufacturing facility, and you’re not prepared for that, then you might lose several hundred thousand dollars a day. A facility I was at yesterday would lose $10,000 an hour if it went down as a result of equipment failure. That’s not an uncommon amount of cost when you shut down an entire whole facility.

Because of the consequential damages and the consequential downtime, it’s a very good idea to do what we call predictive maintenance. There are several aspects of that. I work primarily in vibration analysis, there’s also oil analysis, PDMA testing on motors, and infrared imaging of electrical circuits. These are all situations where if you catch a problem early, you can save a lot of money. In fact, one major save at a plant will probably pay for the predictive maintenance program for maybe two or three years.

But, a lot of people don’t have the foresight, and they’re so busy that, as they say, “they fight alligators rather than drain the swamp”. Many people are just overwhelmed keeping up with the daily concerns of maintaining a facility. In the long run, it’s pretty obvious that you might have a small component which if it fails might take down your whole facility. Preventing failures is primarily the business that I’m in. We go in generally once a month, and test the major pieces of rotating equipment. That usually provides pretty good coverage; note there are some problems that progress quickly and that’s where it would be advantageous to have a system that monitors equipment health on a continuous basis. Such a system doesn’t have to tell you what’s wrong, it just has to inform the owner that something is different, and provide some warning allowing preemptive action to be taken.

I also teach digital signal processing for the Vibration Institute, an advanced level course that shows people the limitations of the technology and how they can get around some of them.

That’s pretty much my background, my whole career has been in troubleshooting. Again, I would say 80 or 90% of my time is spent on predictive maintenance and the other 10-20% is going out diagnosing difficult problems on turbines, motors, pumps, compressors and gearboxes.

The crudest maintenance approach is called breakdown maintenance, basically you don’t maintain it until it breaks. There are many machines where it is fine to use this approach. For example, you don’t want to spend a lot of money analyzing problems if you have two pumps that are only say five horsepower and are redundant. Obviously, breakdown maintenance is not an acceptable approach in regards to major equipment or even small equipment if that small equipment could take your facility down.

Another other type of maintenance is preventative maintenance and that was quite popular for a long time. Basically, you go in, for instance, every three years and overhaul a pump, whether it needs it or not. It’s based upon time and that approach may not be a good idea because you’re over maintaining and you’re working on things that often don’t need to be worked on.

1. You’re spending your limited maintenance budget on equipment that may not need to be repaired and this money is not available to repair those machines that may actually need it.
2. Once you work on the equipment, it may be worse than it was before you worked on it.

It falls under the old adage of “if it isn’t broken, then do not fix it”.

The two items above are the primary reasons that preventative maintenance is not the preferable option. That being said, there are cases when you might want to still use preventative maintenance. For instance, let’s say you have a plant where there is only a major outage every three years. In that case it might be advantageous to just go ahead and maintain this plant’s equipment rather than wait on it to show signs of failure. There are therefore cases for breakdown maintenance and cases for preventative maintenance. But for the majority of things, you’ll want to do predictive maintenance. In other words, you want to just work on the things that need to be worked on and leave everything else alone.

Yes. The key thing that a lot of people forget is they’ll jump into a program and just say we’re going to monitor everything, thus diluting their focus. They need to do what’s called a criticality assessment. They need to sit down and work closely with both operations and maintenance to perform this assessment. For example, let’s suppose your facility has 500 machines. First you talk to your production people and ask: Which machines, if they fail, can stop production? and then you talk to your maintenance people and ask: Which machines have caused problems in the past? You then put this information together to produce a strategic plan of action. Instead of using a shotgun approach, the criticality assessment allows your organization to be much more focused. The budget only contains so much money so it needs to be allocated on the machines that can do the most harm if they fail, or on the major pieces of equipment that have had a lot of failures in the past. Throwing a bunch of money in a shotgun approach is a waste of limited resources and it’s not going to be effective. Once a criticality assessment (with the assistance of operations and maintenance) has been performed, the equipment can be divided into breakdown, preventative and predictive maintenance categories.

Manufacturing facilities are going to know what it costs when a production line is down. For example, if we take this line down, we’re going to lose this much money per day. Power plants are certainly going to be able to get an estimate. Power plants are a little more difficult because it depends what backup power is available. For instance, if their most efficient plant goes down, and there’s a lot of demand on the system, they are going to have to purchase the replacement power at the most expensive rate, thus the costs go through the roof. So again, it’s a bit about timing. It’s hard to say how much it’s going to cost. But if you estimate, for example, that if we lose this plant, what’s the next one in line from a cost basis – normally, they can use that. That’s going to be very conservative because the utility may be forced into purchasing power at the most expensive rate in the middle of winter and the cost to purchase this power could go up by a factor of 10 to 20 times the normal cost. In other areas where you might have an environmental situation, for instance, what kinds of fines are we going to get if this equipment goes down? And that’s going to be pretty much known that it will cost, for instance, $10,000 a day if you’re polluting the water. Anyone who’s a businessman, should know the value of what they are producing and what it will cost if production ceases as a result of an unplanned downtime.

I think anybody who does a good assessment is going to have to do all that. Again, you are in the business and if you are violating environmental standards, you’re going to pay for it. If somebody gets hurt, that’s harder to define, but nobody wants that. There are actually some major costs involved if somebody gets hurt. Besides the moral issues of getting somebody hurt, or a business cost to that, you’re going to be fined by OSHA. There’s also other costs involved, for instance, some businesses will not do business with a company that has a poor safety record. So, you definitely have to know your business and its associated costs. You can experience costs if you disobey environmental rules, and you can get fined and lose business if you have a poor safety record. All of this has to be considered when performing a criticality assessment.

Once you get everybody involved, then you can see how your business can get affected whether it be consequential damages, consequential downtime, safety or environmental fines problems. This approach will allow a business to establish a realistic basis for how much they can afford to spend to monitor their vital equipment and keep their facilities from going down in an uncontrolled manner?

There’s going to be upfront costs, and again, it is a return-on-investment issue. The hardest thing of all about predictive maintenance is the scenario of “how much did I save by not running my car into the tree this morning”. There is no line item in any kind of accounting that accounts for the fact that you prevented something from happening. That is the absolute most difficult thing to come up with. So sometimes you have to think of it in terms of more like an investment in insurance. It’s hard to quantify how much your insurance policy saved. Yet it’s the same point: who’s going to go without insurance? That would be a bad business practice. So, it’s a difficult thing and it’s where upper management and strategic thinkers need to get involved. What kind of a business are we wanting to run here? What are acceptable risks and what are not acceptable? That gets to be non-scientific, sometimes. We throw science and we throw math into it, but then there are these other things that you have to consider – what kind of a business do we want to have here and how can we get hurt and what can we do to prevent it? And it’s hard, again as mentioned above, there’s no line item for when you prevented a failure from occurring. That’s the absolute most difficult thing you deal with in predictive maintenance. But, it’s also like the dentist – once a toothache is there, everybody is alert. You don’t want to wait until that point, right? Would you rather go to the doctor and find out a problem early on, or are you going to wait till it’s a catastrophe? You have to think about Predictive Maintenance in a similar manner.

I personally don’t get involved in insurance issues. I have, however, definitely heard people say that they have a Predictive Maintenance Program because it affects their insurance rates. For instance, insurance companies frown on having a house without smoke detectors. Having a predictive maintenance program is a lot like having smoke detectors for your individual machines. Why would you not do that?

I’m involved in a new company and their investors are actually requiring the implementation of a predictive maintenance program. The investors want to know if the startup company is doing predictive maintenance on their equipment. They are actually bringing this subject up in the investment phase, which is an encouraging thing to hear. Investors are becoming aware enough of it that they are asking: Are you maintaining your equipment properly? Because you cannot show a profit if your facility is offline due to equipment failures. This looks good until the car fails. Investors did not get rich by not performing their due diligence and an important part of due diligence of a manufacturing facility is how a business approaches maintenance of vital equipment.

The primary predictive technology I work in is vibration analysis for rotating equipment. That’s where you can get the most bang for your buck. However, in some cases oil analysis is equally or more effective at early-stage problem identification. This is particularly true for machines that have components that operate below 300 RPM. Since the vast majority of rotating equipment operates above 300 RPM, vibration is therefore still the most utilized form of predictive maintenance.

Another predictive technology is infrared imaging. This technology is primarily used on electrical circuits. It is utilized to scan fuses and electrical connections, to detect and eliminate hotspots before they can cause a fire or a failure. Infrared imaging doesn’t need to be done as frequently as vibration analysis. Usually once per year is sufficient for infrared scans.

Then there is also advanced motor analysis, where you’re looking for arcing or cracked rotor bars, or some other kind of problem within the motor that vibration monitoring might not catch.

One of the most promising developments in the field of Predictive Maintenance is with the use of software to assist in early identification of problems via Advanced Pattern Recognition (APR). In most facilities, critical parameters such as temperatures, speeds, flow rates, pressures and vibration levels are brought into a central computer. These systems are set up to alarm when a parameter goes out of its normal range. APR takes this approach another step farther. APR looks at more than one parameter at a time and builds a mathematical model of the relationships between different parameters. For example, the APR system detects that a motor is drawing 800 amps and in the past when it pulled 800 amps, the motor’s internal winding temperature was 180 degrees and now the winding temperature at the same current flow is 190 degrees. The APR system does not know why the temperature has increased, but detects something is different from the normal relationship between these two parameters. In this particular instance, the cause was simply that the air filters on the motor were clogged. Once an APR system learns what is normal by bringing in vibration readings, temperature, pressures and flow rates, then it can alarm on any deviations from normal that it sees. This is not Artificial Intelligence (AI). Everybody wants to jump on the AI band wagon. APR simply points out that something’s not right, now go out and investigate. I think if we can get these systems to just say something is wrong, that’s a huge leap over not knowing something’s wrong until it happens. If APR says we have 1,000 machines, but these 4 are really what you need to look at, then the terabyte haystack of data has suddenly become much smaller allowing us to concentrate on diagnosing problems. These systems allow a few talented experienced individuals to watch over thousands of vital machines.

Vibration analysis just sort of leveled out over the last few years, not really much new has happened. As a result of the introduction of portable spectrum analyzers, it is more efficient than it used to be. In the future, there are probably going to be more online monitoring systems installed. As noted above, I think the future is in APR. I think that’s the future. Automation of data acquisition combined with APR will allow for both efficient monitoring of the health of vital machines and the production of actionable advice thereby allowing for a much fewer number of machinery experts to detect and act on most problems at an early enough stage to avoid consequential damages and costly unscheduled down time.

You hear a lot about IoT, Internet of Things. IoT is setting up to potentially overwhelm us with data. Imagine getting terabytes of data, but having no way to filter it to identify what is really important. Hard drives out on the cloud will fill up with useless data. For example, one of the systems I work with is storing a terabyte a day of just vibration data. Fortunately for this organization, they have advanced pattern recognition. Otherwise, what would they do with the terabytes of vibration information that come in each day?

I think vibration data collection will become more automated.

The reasons being:

1. It is expensive and inefficient to have personnel travel to collect data.
2. Because of safety, environmental and now because of terrorist threats, it’s much more difficult to get into plants to collect data than it used to be.
3. There are fewer and fewer people experienced and qualified in vibration analysis so those that are left need to work more efficiently.

Again, I haven’t seen a lot of new things other than some of the online systems. It’s in a technology that is in a mature state. The FFT, the Fast Fourier transform, was a major change. Having the portable instrumentation to go around, that was a major change and now, the next change will be more online systems and more APR. I don’t see how it can go anywhere, but that.

Yes, and eventually probably some AI involved in it. The trouble is that everybody’s pushing AI because it sounds more futuristic. In my opinion, it would be more productive to concentrate on getting data collection automated than work on APR than try to jump into AI. We need to get accurate data then filter it using APR. Once these things occur AI could be developed for specific very vital machines. With only a very small percentage of machines having an online monitoring system, that issue needs to be resolved prior to even thinking about a computer system telling you what is wrong with your machines.

Reliability is everything. Most facilities do not have extra capacity so when a line is down, due to an unscheduled failure, that can be the difference between succeeding and failing. Also, machines are replacing people everywhere. When a machine fails the entire plant can be brought down. For example, A single paper machine can produce a 20 ft wide piece of paper that is coming out of the machine at 60 miles/hour. If that one machine goes down, it has a very large effect on production and possibly not being able to live up to contract delivery obligations. If a turbine at an 1100-Megawatt nuclear power plant comes down that means the power for an entire city of 250,000 people will not be available. These large machines have made our modern way of life possible. When they fail in an unexpected manner that can be disastrous.

Definitely a force multiplier. I’ve watched this industry for 40 years, I’ve seen promises of expert systems and artificial intelligence made every few years. It seems like these technologies are always in the future 20-30 years. Force multipliers on the other hand are most promising and available with today’s technology. This is like the multiplier effect of when people used to run around with a swept filter instrument versus a portable digital FFT data collector, the multiplier was huge. As you get more automated systems, and can access any machine from anywhere in the world, that’s a huge force multiplier. Then if you add advanced pattern recognition, so you don’t have to look at all the data that is coming in, that’s another great force multiplier. Individual force multipliers actually can multiply together and can thus vastly increase efficiency. For example, a single individual with an automated data collection system combined with Advanced Pattern Recognition software can effectively monitor the health of thousands of machines. You can have one qualified person do the job of 100 people in the past and that’s what we’re going to need because there are less and less people who actually know how machines work.

Right! Because of the IoT thing everybody keeps talking about the amount of data available is going to be potentially overwhelming. What are you going to do with all this data? It finally comes down to the plant manager saying, “Can I run this machine or not?” Somebody has to make that decision. Somehow, I don’t think that a computer is going to be the answer. Somebody’s going to have to put themselves on the line and say, “yes, it looks like we can make it until the next outage”. Somebody has to do that and that’s going to be the key for all of this to work. If we produce data, and there’s no way to filter it and analyze it, then we are just going to have a lot of data filling up hard drives out on the web.

I think the key is going to be a reduction in cost of remote monitoring systems. That’s the key to how far down the pyramid (as described below) you’re going to go. Think of the machines as a pyramid. If you have, for instance 1,000 machines then that pyramid of machines can be broken down into different levels. The top of the pyramid might consist of just 3-4 major machines, such as compressors, turbines and large fans. These are very vital high-cost machines. They would be set up with an on line monitoring system outfitted with proximity probes to monitor shaft vibration. These complex monitoring systems produce real time continuous data and can shut down the machine in a matter of milliseconds should the vibration get too high. The next level on the pyramid would contain 40-50 machines that are important but do not require proximity probes. Some of these machines will have accelerometers and the data will be transmitted to the PI system. The third step down the pyramid might consist of 300-400 mid-level machines. These machines along with the level 2 machines would be tested monthly using portable FFT analyzers and the data collected would be transferred to a computer and analyzed by vibration specialists. The remaining machines are small non vital pumps, motors and fans that could be run to failure. The second and third sections of the pyramid are where remote monitoring systems are starting to be installed. The key to how far down the pyramid these remote monitoring systems will go will be based upon the cost. As the cost decreases, you can go further down the pyramid. Other factors will also enter into the remote monitoring decision. A few of these considerations, as mentioned previously, will be safety concerns, environmental non compliance, plant security, IT access and availability of certified analysts.

Definitely. Remote monitoring is effective, but it still comes down to cost. Can you get the cost down to where remote monitoring is able to compete with somebody walking around doing it?

That’s been a challenge for quite a long time. We’ve had remote monitoring for a long, long time. I’ve personally been involved in it for almost 20 years, but it almost always dies out. It seems to sputter along and the cost is almost always the issue that kills it. Companies will install remote monitors on a few key machines, but it seldom gets very far down the pyramid.

One of the big challenges, of course, is education and educating managers, as to the benefits of it. One of the challenges is that you have to convince high level management personnel who control the purse strings. It’s nice to speak about it at a conference or talk to the maintenance guy. If there is a maintenance manager then it is of course advantageous to talk to him. But at some point, somebody above him has got to approve it, and make the case to others. Making the case to upper management and educating upper management on the effectiveness of this has always been the challenge. Maintenance is often just a necessary evil in their eyes and can be a bit of a step child when compared to other issues they face.

I know another challenge, going back forever is personnel. How can you retain a key guy who understands the technical specifics of this field? His value could be tremendous to the company, but he’s a diagnostic guy and often can’t get promoted so they move on to another job and the vibration program dies. There used to be a talk of a dual ladder approach, where you’re very highly technical people were worth as much as your management people, but it never happens. It always ends up, they take the guy and they turn him into a manager and you just lost a really great diagnostic person, and maybe also ended up with a poor manager. This happens every day. The expert, in order to get a promotion, leaves his position as a vibration specialist. That’s been a huge challenge for in house vibration programs. These are things that have been going on for 40 years. Educating upper management, and then retaining these expert people seems to be very difficult. Once you get a person trained in this area, how do you retain them? You have the right person, they can save you a lot of money, but it’s more advantageous for them to take a promotion as an area supervisor, than to continue to do what they’re doing now.

Therefore, educating management into the benefits of doing predictive maintenance and then maintaining personnel is very difficult. I go into facilities and find they have the data collection equipment there, they have an online system, but they’ve got nobody to use them because nobody understands it and the person who did understand it is gone. I don’t even know how many places I have gone into, and helped them set up the programs that eventually failed. They get a person who’s enthusiastic and they start up a program. But when I come back five years later, the equipment has dust all over it because the guy left or he got promoted into a position and now there’s nobody left to do the data collection and analytical work. These are problems you may not want to hear in this talk, but these are real problems.

The targets are going to be whoever’s responsible for maintenance and maybe a layer or two of management above them. Because of the popularity and all the talk about IOT and AI, you might get some traction in those areas. I think another group to approach would be production management. They hate it when things break down. So that would be another group to approach. Maybe you could get production management and maintenance management together to approach upper management regarding remote continuous monitoring.

I don’t think there’s anybody that would be totally against it. However, when remote monitoring is being considered, the IT department may be resistant in allowing access to their network.

I don’t know anybody who’s inherently against it other than bean counters, asking, “How much is the return on our investment in remote monitoring of our vital equipment?”. About the only hope you’ve got is if you had some things that broke in the previous year, they’ll then know what it costs when there is an unscheduled failure. That’s the only thing you have to go on because when you save something, nobody knows how much you saved, because it didn’t happen. There is no line item in accounting for how much you saved by preventing something from happening.

I don’t think the government really cares too much about that kind of thing. Insurance might get involved.

No, generally not, to be truthful. Maintenance employees will sometimes get behind it. Sometimes engineering will get behind it, and they’ll go to a course or something and say, “Wow, we ought to do this”.

Now sometimes you get a good guy called a White Knight. If you get a white knight that goes in and really convinces people, from the bottom up and if he’s a really good salesman, internally, then you can see programs getting started that way. But, for the program to be sustainable over time, you have to get higher levels of management involved. For it to survive, Predictive Maintenance has to become part of the corporate culture.

Preventative maintenance seems to go in cycles. It starts out with enthusiasm then withers with time. I have seen this cycle so many times I just start chuckling “Here we go again, they had a good guy, but they didn’t take care of him. He left and the program felt flat”. Then I come in and rescue the program. Or they get a guy and they say do this in your spare time. Because the last thing he wants to do is to walk around and collect data in a hot nasty place, he or she puts it off and the data never gets collected.

The technology is already here to do effective predictive maintenance, but structurally and corporate culture wise, it tends to be a problem. Sometimes programs go very well, but it just depends on the company. Keeping somebody internally in their organization to keep the program going is often really difficult to maintain on a long-term basis.

Yes. That is correct.
To get somebody to go out and collect data in a nasty situation is challenging. If they can do anything else, they won’t do that. Only certain people are willing to go test cooling towers in January. This is another good reason to change to remote monitoring.

If a guy gets sick, or he leaves the company, with an automated system the data still gets taken. You can bring a new person in, and the history is still there if it was collected automatically. If it’s a manual program, and a person leaves and they don’t replace him for say six months, you end up with six months of nothing. Another advantage of an online system is that if an analyst leaves, then another analyst at a sister plant of the same company can look at the data because it is available on the internet rather than being siloed on a computer at one location. So, there’s a lot of advantages to an online remote monitoring system as compared to a manual collection system.