Design Solutions For Mission Critical Centers

"The design is as critical as the mission." That's what Freddy Padilla, Page Principal and MEP Engineer, wants people to understand about mission critical centers that support emergency response such as 9-1-1, police and fire.

He shared additional details with Mission Critical Magazine about the distinctive needs of these facilities serving the public, and how multidisciplinary firms ensure they're met. Freddy has more than two decades of experience as an engineer designing infrastructure for critical infrastructure across multiple market sectors. See what he has to say.

Call centers for 9-1-1 emergencies serve a vital and singularly unique need in the communities they support. These centers must be operational 24/7, during severe weather and other emergencies. Therefore, their systems are especially sophisticated with decisive layers of redundancy. Whether designing a new call center or renovating an existing facility, leaders and operators face highly specialized operational and technological challenges.

System Design

Design system sequencing to endure extreme conditions. Initial planning and design decisions for a 9-1-1 call center relate to technical issues that keep a center’s systems functioning during emergencies threatening its operation. These technical issues involve complex sequencing required to activate automatic system responses to seamlessly maintain operation.

It is critical that the control room and data center never fail, even in extreme duress. For example, if a center is supported by two substations and one utility is lost, the system automatically switches to the alternate utility. If the second utility fails, the center is designed to instantaneously access generator power with multiple generators providing redundancy in support of one another. 

These failures can happen all at once or in a rapid cascading fashion as the load transfers. An A-B system is a cost-effective way to manage this type of episode.

For non-critical loads, such as offices, generators are calibrated to decrease output or turn off to conserve power for critical functions when necessary.

Design a simple system. Because an emergency call center is inherently complex, it is especially important that the system be designed to be as simple as possible.

Incorporating simple solutions wherever possible makes the system easier to operate and upgrade. When technology improves and/or workarounds become obsolete, components of the system that are no longer needed can be removed.

Take static transfer switches as an example. The technology has evolved and switches are designed to transfer loads out of phase so uninterruptible power supplies (UPSs) don’t have to be synchronized, reducing UPS system complexity.

Interestingly, many operators unnecessarily continue to synchronize these devices because they are uncomfortable giving up the practice and allowing the new technology to perform.

Transformerless UPSs are another example of innovation that some operators understandably struggle to embrace. In the past, UPSs had transformers to ensure that the system was completely isolated and grounded. Removing the transformers saves money and improves reliability since there is one less device to fail.

Today, many centers have a “refresh rate” that calls for new equipment every three to five years to keep pace with changing technology, so isolation is not necessary to protect equipment that is replaced well before the end of its life.

Increasingly, operating manuals are becoming a vitally important component of the system.With higher turnover rates, the brain trust of long-term employees that endured in previous generations is no longer a reliable resource. And, combined with increasingly complex systems necessitating sophisticated redundancy, emergency response facilities are incorporating additional devices and more complex operating sequences.

A manual of operations for the facility is an invaluable resource for owners to track and analyze preventive maintenance and coordinate multiple pieces of equipment. An operation manual is also a valuable training tool for new employees, explaining the complex systems and their use, with documented procedures that don’t rely on institutional knowledge.

Testing + Commissioning

Multiple preliminary tests lead to a smooth final integrated system test at commissioning. One important consideration at the end of the construction process is testing the entire system to verify all components are functioning as designed.

Designers and contractors are partners in this process as the team integrates and operationalizes each of the components. Ideally, five levels of preliminary testing are conducted in order to ensure that final testing runs smoothly.

Various levels of testing can include equipment tests and a system test. As designers and contractors are active participants through all the levels of testing, they have an intimate knowledge of the system and are able to answer any questions that come up at the final integrated system test “level five.”

Empowered with this, operators can isolate user errors from system errors as well as have a better understanding of how the building works. While a final integrated system test is part of the commissioning process, established procedures ensure that problems will be addressed systematically if and when they emerge.

Testing under various temperature conditions provides more accurate resiliency results. In the age of climate change, with global temperatures on the rise, operators are finding it increasingly important to test equipment under more extreme temperature conditions.

For example, a generator placed outdoors may not operate at its full capacity or may even shut down if it gets too hot, rendering backup power support ineffective and obsolete. The solution for this challenge is testing protocols that require a wider range of temperatures at different times of the year. Tests may also involve working around building operations to verify systems that are already installed and functioning.

Conduct proactive, advanced equipment testing in the context of the building system. Another common but unanticipated testing challenge is whether operators test pieces of equipment individually or as a group synthesized to perform a task as a unit.

Today, project leaders simulate the unique conditions of the facility system in testing before introducing new equipment, even if a piece of equipment promises to be functional off-the-rack. As an example, the operating load, once installed, could be different than originally calculated, which drastically alters the way the equipment operates within the building. It is key to conduct proactive, advanced troubleshooting of equipment within the context of the building and system in which it will be installed.

Documentation of change

Consistently updating operating manuals and documenting changes offers an array of benefits. Documenting every change to a piece of equipment or the process is key to having the data when it is needed. This is no small task and is understandably much easier said than done. When a component fails, operators can replace it with a new, better device. However, a more sophisticated device may have different requirements, underscoring why documenting change is so critical. Having accurate data on equipment and operations not only helps avoid errors and equipment breakdown but also enhances safety. For example, current arc flash labels and newly revised documentation will tell operators what equipment to focus on both during a shutdown and an emergency.

Documenting change promotes upgrading to optimize operational efficiency. The quest for more energy-efficient operations is inspiring many operators to alter their configurations to achieve fewer losses. Systems are regularly evaluated in light of new technology/requirements to increase efficiency.

In electrical terms, for transformers, fewer losses typically mean fewer wires, which mean less impedance. Whenever a transformer or other source becomes more efficient, the impedance goes down and the ability of the source to generate a short circuit is increased. There are more efficient transformers today, but changing transformers without proper documentation can put employees at risk if the equipment lacks the proper protections. Systems are modified to account for the new conditions, and requirements are set for the market by governmental agencies.

Updated operations manuals including comprehensive and clear record keeping of change offer both maintenance personnel and vendors a holistic, up-to-date perspective of the mission critical facility. Because of the high turnover among maintenance personnel, operators are more likely to outsource tasks like preventive maintenance and associated documentation to vendors. This may seem efficient at first glance, but while vendor representatives are experts on their own equipment, they are often not experts on how it interacts with other equipment within the facility.

In some cases, vendors are not aware of what is going on in the data center, with training and operations schedules, potentially causing a scheduling conflict. Something as simple as leaving a door open can create issues for the nearby air handling units. Therefore, a process is required to ensure that vendors know exactly how the facility is operated.

Documentation helps operators walk the high-tech tightrope of decision making when renovating. Documentation significantly contributes to decision making when upgrading an existing data center, which can be more challenging than building new.

Many operators face the issue of growth outpacing their space and/or resources. Operators reaching maximum capacity seek solutions to how they can grow without significant system change. The high-tech tightrope operators walk in a renovation project is how to incorporate newer, more efficient technology while causing as little disruption to the building as possible.

For example, reassigning loads and re-sequencing is easier than replacing an entire system. In one project, a team used an A-B-C system with each part loaded at 66%, rather than the standard twin configuration with each side loaded 45% to 50%. This allowed them to increase capacity with minimum system changes without having any effect on the critical load.

To keep the system up and running, designers prioritize the backup system to maintain reliability, as changes are executed. Coordination of maintenance and upgrades, involvement of vendors, and an overall knowledge and transparency of the way the facility works will contribute to operations. This also supports the design decisions that were made early in the planning process.

Thorough documentation and better processes enable the efficiency required to operate around the clock, even while experiencing significant duress.

Supporting response

Given the significant and crucial support provided by call centers for 9-1-1 emergencies, the design of these centers is critical to the communities they serve. Designs for these facilities must ensure they can endure extreme conditions while remaining simple and understandable for users. It is imperative to deliver the appropriate solutions for mission critical centers that support emergency response teams.

To view Freddy's expertise in the Mission Critical Magazine article, click here.