Rexam's agile IT initiative keeps plants running

A division of London-based Rexam, an $8-billion packaging manufacturer with 24,000 employees and operations in 20 countries, the business had invested heavily in automating its 17 North American beverage can plants, which supplied cans to customers such as Anheuser-Busch and Coca-Cola.

A single instance of SAP's R/3 enterprise application suite linked the plants to the administrative processes that replenished raw materials, determined the next orders to be manufactured, and invoiced customers. Sever that link—with a network outage, or systems being taken off-line for an upgrade—and life became not unlivable, but difficult. For with manufacturing plants running at full capacity, downtime wasn't acceptable.

Rexam addressed the challenge using the SAP xAPP Manufacturing Integration and Intelligence (xMII) to allow continued plant production when the network was unavailable. In addition, fault-tolerant, high-availability servers from Stratus Technologies ensured that local resources needed for production, including SAP xMII, were always available.

While use of high-availability servers has been fairly common in other industries, their use in production environments is growing today. Managements recognize that proliferation of productivity-enhancing applications in manufacturing—and tending to form into an integrated, mutually dependent “solution”—means the risks associated with downtime are increasingly severe.

The prior modus operandi adopted in the plants worked well, recalls Niemzyk, but wasn't ideal. Any unplanned system downtime was buffered by manual back-up processes. Product got shipped; customers got invoiced; business went on as usual—albeit at the expense of a little more effort than usual.

Meanwhile, planned downtime for system upgrades and patches took place over holiday periods. Although workable from a systems management perspective, it was less than fully satisfactory, notes Niemzyk. Christmas, for example, was the corporate financial year end, and not a time during which to introduce system changes. And it meant encroaching on employees' holiday time.

“Instead of implementing an initiative when it made sense for the business, we were tying 'go live' to the next convenient holiday, which was usually later than we wished,” says Niemzyk.

Researching better ways of dealing with planned and unplanned downtime threw up dispiritingly few alternatives. For a while, says Niemzyk, Plan 'A' was to use two parallel instances of SAP R/3.

“We'd take one down and carry out the upgrade while transacting in the other,” he explains. “Then we'd bring back the first instance, re-synchronize, and take the second instance off-line.”

The drawbacks were obvious. “It added to overhead, and didn't address communications issues,” says Niemzyk.

To deal with unplanned events, some way of continuing to transact on the plant floor during an outage was required. Re-synchronization of plant events with SAP R/3 could take place when connections were restored. But the prospect of a third-party manufacturing execution system (MES) with its own database running at each plant was unappealing. Rexam, says Niemzyk, was looking to strip out complexity—not add to it.

The answer emerged when Niemzyk's IT team began exploring SAP's newest offerings—and especially SAP xMII. Running SAP xMII on the plant floor, realized Niemzyk, would not only address unplanned network downtime through a “store-and-forward” synchronization capability, but also provide an MES-like facility that was presently lacking—including data aggregation, plant-level “dashboards,” and RF communications.

The so-called “MES” layer can include detailed functionality for work-in-process (WIP) tracking; scheduling; and industry-specific requirements typically not handled by ERP. In other cases, it simply connotes the means to connect the plant with the enterprise. SAP xMII is the result of that company's further development of the innovative, Web-based solution it obtained in the acquisition of Lighthammer.

SAP xMII on the plant floor—backed by a local database—would enable manufacturing operations to run locally not just for an hour or so, but long enough for even the most severe outage to be fixed. Complex patchings or system upgrades could be done in the data center during the normal work week.

Setting a target of 48 hours for what he terms “local sustainability,” Niemzyk began talking to systems integrator Tata Consultancy Services (TCS) about piloting an xMII application at a single Rexam plant.

But on what hardware should the plant-level xMII applications reside? For there was little point, Niemzyk realized, in implementing something that added to overall system vulnerability, rather than reducing it.

“Quite literally, we wanted to never lose a transaction, or miss a heartbeat on the shop floor. The goal was continuous availability,” says Niemzyk.

For awhile, installing xMII on server clusters was the plan. But clusters would need scripts to be written to manage the failover process. These scripts would need to be tested and maintained as hardware within the clusters changed. What's more, clusters often require applications to be modified to recognize the cluster environment that they're running on. And clusters might necessitate having IT people on-site to manage the process.

Niemzyk told his team to look harder, and soon after, discussions began with Stratus Technologies, which offers fault-tolerant, high-availability servers.

With global corporations benefiting from increasingly sophisticated ERP systems, opportunities exist for optimizing plant operations based on business priorities, resulting from so-called "top floor to shop floor" integration.

With certified 99.999+ percent uptime and no need for plant-level support, the Stratus boxes also looked like ordinary Windows servers from an application's point of view, eliminating requirements to modify applications to run on clusters.

But Niemzyk wasn't yet done optimizing availability. As TCS constructed the pilot, its brief included an important rider: At the plant-floor level, the xMII screens were to look exactly like the SAP R/3 screens they were replacing.

The logic, he admits, was chiefly driven by the wish to minimizing training at the plants. With screens that looked the same, operatives would transition smoothly from SAP R/3 to SAP xMII. But there was an availability dimension to the decision too.

“We've kept the original transactions 'live' in the SAP R/3 system,” says Niemzyk. “In the unlikely event that we lost the local system at a plant, we could go back to remotely signing on to SAP R/3 at the data center.”

By summer 2007, with tests showing the pilot plant was delivering expected levels of availability, the implementation was extended to Rexam Beverage Can NA's 16 other plants. All plants were live by mid-December 2007.

“We had great cooperation from Stratus, TCS, and our other partners, and we brought on one or two plants a week. It was that rapid,” says Niemzyk.

Fault-tolerant servers from Stratus Technologies are said to eliminate the operation complexity and high costs inherent in high-availability approaches such as clusters.

Rexam's requirement for continuous availability and operational simplicity didn't surprise Hill, who notes top-floor to shop-floor integration always includes requirements for high availability. Today, Stratus and SAP jointly offer such a combination under the “Perfect Plant” initiative, including the recently inaugurated Perfect Plant Center of Excellence (See sidebar).

Rexam, Hill says, has gained from transaction store-and-forward functionality.

“This is a tremendous benefit in terms of corporate IT flexibility and plant uptime,” says Hill. “It wasn't necessarily where we saw ourselves adding value in manufacturing—but there's little doubt that the demand is there for local uptime. Finding opportunities to take the data center off-line is getting tougher and tougher.”

Niemzyk concurs. “From a system availability perspective, we're getting the uptime we expected, and when network outages occur, we don't miss a heartbeat.” Not least, Rexam will be upgrading to SAP ECC 6.0 sometime soon, and it won't be at Christmas or Thanksgiving.

“No one likes to work over a holiday period,” says Niemzyk, “and it's a significant lifestyle and morale benefit to the IT team not to have to do it. Internally, we're calling it 'IT agility': the flexibility to implement upgrades and patches when it makes sense to do so—and not when we have to.”

What's more, SAP xMII has delivered a double benefit in terms of ease of upgrade. Much of the customization of SAP R/3, undertaken to maximize competitive advantage, was in the shop-floor modules, Niemzyk reveals. SAP xMII, which functionally replaces those shop-floor modules, delivers that same customization through an in-built tool kit, making maintenance easier—especially at major upgrade points.

As plans move forward within Rexam to roll out the solution in other geographies and divisions, Niemzyk admits one early characterization of the solution was wrong.

Rexam Beverage Can North America evaluated several options for ensuring needed process capabilities were available as needed before settling on use of SAP xMII and Stratus Technologies high-availability servers.