Doing a Double-Take: SCADA replication software refines Carlsberg plant operations
Malcolm Wheatley, senior contributing editor, Leeds, U.K. -- Manufacturing Business Technology, 5/9/2008 6:00:00 AM
At the Leeds, U.K. brewing & bottling plant of Danish brewing group Carlsberg, an asynchronous fail-over and replication solution from Double-Take Software is keeping production going when server failures take critical systems off-line.Brewing three million hectoliters of beer each year—including the company’s popular British brand, Tetley's—and operating 24 hours a day, server failure used to hit production hard.
Over the last five years, explains Senior Systems Engineer Graeme Walker, server failure has knocked out the brewery’s SCADA system twice. On each occasion, he relates, it has taken IT engineers between two and four hours to replace the hardware and reinstate the system. And with its SCADA systems inactive, brewing operations would grind to a halt—with a four-hour interruption to brewing the equivalent of a loss in production of approximately 190,000 liters of beer.
In mid-2007, explains Walker, the decision was made to upgrade the plant’s Intellution SCADA to a more current version—now sold and supported under the GE Proficy banner. But while this contained an improved fail-over capability, the extent of the data replication wasn’t total. Recognizing that the upgrade would require new servers, Carlsberg began to explore other options.
"Given that the fail-over mechanism didn’t meet our requirements, we decided to look at an external solution,” says Walker. “We initially considered a Microsoft cluster, but it seemed very complex and the cost was prohibitive. Instead, Polestar, our IT partner, suggested we take a look at the data replication and fail-over capabilities of Double-Take Software."
A test system was set up to see if the Double-Take solution would deliver the required fail-over capability. A model of the proposed infrastructure was built by establishing an ethernet connection between a production unit on the plant floor and a "target" server located elsewhere in the facility, to which data would be replicated. Using a variety of methods to recreate realistic failure scenarios, Carlsberg engineers then caused the production server to fail—monitoring whether the fail-over process worked, and if all the required data was present.
“In every test, the Double-Take solution failed over and failed back seamlessly. After the trials were completed, we had no hesitation in giving Polestar the green light to implement the full solution,” recalls Walker.
It was an evaluation process that was fairly typical, adds Double-Take’s Worcester, U.K.-based Sales & Marketing Director, Ian Masters. Some of the company’s largest customers are in the manufacturing industry, he notes, with a common requirement being a low server load.
“With asynchronous replication, there’s no real performance impact: The CPU load is just 3 or 4 percent,” says Masters. “Asynchronous replication doesn’t guarantee total replication. Over IP networks, packet loss can happen, but the data on the target system will be usable, ‘crash-consistent’ data. We’re hardware- and application-agnostic, and all the customer needs is a Windows operating system.”
 | Carlsberg Senior Systems Engineer Graeme Walker says asynchronous fail-over and replication software from Double-Take is keeping production going when server failures take critical systems off-line. |
Carlsberg’s server upgrade took place in January, relates Walker, with the SCADA upgrade following immediately. From a previous installation that he characterizes as “15 servers and no redundancy”, the plant now has five primary SCADA servers; two secondary servers used for server databases, pictures, and recipes; and two servers dedicated to the role of replication target servers.
And in addition to their back-up role in the event of server failure, the replication servers also offer operational benefits, he adds. Server maintenance, for example, has been greatly simplified.
“If we need to investigate problems, install patches, or implement server or software upgrades, we manually fail over to the target units, leaving us free to work on the production servers without compromising performance,” says Walker. “Another benefit of having two identical datasets is that we are able to run our tape backup from the target servers. This means that the back-up process doesn't interfere with production performance, and we run the backups during standard working hours.”
Carlsberg recently had genuine need to call on its new fail-over capability—just weeks after going live. “Some plant engineers accidentally cut the power supply to a production server,” says Walker. “On the shop floor, the operators didn’t notice a thing.”
There are no other articles related to this article.
Talkback
Related Content
Related Content
Sponsored Links
More results on MBT Research Library