Experts are turning their attention towards Hurricane Preparedness this week as we officially start the hurricane season for 2017.

Experts are turning their attention towards Hurricane Preparedness this week as we officially start the hurricane season for 2017.

During a hurricane situation there are several ways you can prepare a data center. Pergravis is going to outline two options; shut down and emergency operations.

Regardless of the exact strength of a storm, it may be a potentially life threatening event and all necessary preparations should be made to safeguard data center hardware, data and personnel.

Shut Down

Shutting down the data center will protect personnel and systems from a variety of storm related damage.  For the enterprise, it will be protected from electrical surges, and power outages.  Personnel will be able to seek refuge in shelters designed for hurricanes rather than maintaining operations in a facility that may be at risk from hurricane force winds.  If you chose to shut down, Pergravis recommends the following actions be taken.

  • Declare an emergency early. The declaration should be communicated senior executives, employees, partner and customers.
  • Communicate with your employees, user base, including remote users, partners and customers that utilize enterprise resources that there will be a shut down. Prepare the user base for a potentially long term outage.
  • Execute a complete backup as soon as possible. If a full back up is not possible, prioritize backups so the most critical data is protected.  This would include items such as ERP databases, email databases and critical application configuration files.  Full application backups are not necessary, the configuration files are critical however.
  • If backups are written to tape, place the backup tapes in proper containers and store them in a location that will ensure they are protected during the storm. This may be some place other than their normal storage location.  Be careful to avoid magnetic fields and heat during transport.
  • Shut down and secure the enterprise. Following completion of the backups and after confirming that no transactional operations are taking place on the enterprise, execute a graceful shut down of the enterprise. A graceful shutdown will ensure the integrity of databases.  Please be aware that there may be a specific order in which hardware is powered down.  Likewise, there may be a specific order in which hardware is restarted.  This is particularly true for SAN environments.
  • Following the shutdown of the enterprise, physically secure the data center. This includes shutting down UPS units so they do not deplete their batteries while the enterprise is powered down.  All main circuit breakers to the data center, UPS systems, cooling systems should be opened to prevent the possibility of damage from power spikes or lightning.
  • Remote facility recovery preparations. If applicable, coordinate with remote sites to begin preparations for recovering to the remote site should there be catastrophic loss of the primary data center.  This would include identifying available hardware, personnel and vendors who can travel to the remote site, with backup tapes to begin the restoration process.
  • Plan for restoration of normal services following the event. Line up all information technology and facilities staff that will be necessary to restore normal operations once the hurricane has passed. Facility staff should be available to restore power to UPS systems, cooling units, etc. while the IT staff should be prepared to carefully power up the enterprise
  • Develop a Communications Plan. Maintaining communications among the information technology and facilities team will be critical.  It is important for data center management and staff to know how to reach one another through all the normal channels, including home phones, cell phones and personal email accounts. However, these forms of communication could be interrupted, so it might be useful to know spouses’ contact information or to have out-of-region locations where people can leave messages for one another. These forms of communication can be confirmed in advance of predictable bad weather. Often, when one channel is blocked another one can be still be used; it pays to know all of them in advance.

Emergency Operations

If operations must be maintained during the storm, take all necessary measures to protect human life. If there is any doubt whether operations can be maintained safely, shut down the enterprise and ensure all personnel are able to seek shelter in an appropriate location. If you chose emergency operations, Pergravis recommends the following actions be taken.

  • Declare an emergency early. The declaration should be communicated senior executives, employees, partner and customers.
  • Communicate with your employees, user base, including remote users, partners and customers that utilize enterprise resources that enterprise services may not be available due to the storm. Prepare the user base for a potentially long term outage.
  • Staffing Plan. Develop a staffing plan based on staff availability and necessary skill sets.  Keep in mind that during a major emergency it is entirely possible that staff will choose to safeguard their family and homes and sacrifice their employment in the process.
  • Shelter in Place Preparations. Plan for a minimum of seven days operations following the event.  Measures should be taken to provide for the well-being of staff that will be sheltering in place during the hurricane and in the days following.  Plan on providing the following necessities:
    • Food
    • Water
    • Cots & bedding
    • Personal Hygiene Articles
    • Sanitation
    • Changes of Clothing
    • Rain Gear
    • Battery Operated Radio
    • Flashlights with spare batteries
  • We assume the facility is equipped with an emergency generator. The generator is the heart of emergency operations and all necessary measures should be taken to protect it.  We recommend operating the data center on generator power during the hurricane.  Assuming the generator is properly maintained, it will likely be a more reliable source of power than utility supplied electricity.  In preparation for the storm, the following should be done at a minimum:
    • Refuel the generator and ensure that there is a sufficient fuel supply stored onsite to run the generator continuously until utility power is restored.
    • Inspect the generator including the following items:
      • Sample the fuel and ensure it is free of all contaminants including water, particulates and biologics (algae).
      • Engine oil level
      • Coolant level
      • Air filter condition
      • Fan belt condition
      • Ensure the starting battery is charged and serviceable
    • Contact the fuel supplier and ensure that the data center is placed on the priority refueling list. This will likely incur a fee, potentially a sizable fee.
    • Test the generator to confirm normal operation. Extended operation without a load is not recommended.
    • Assuming the generator is regularly maintained and there is confidence in the data center UPS systems to support the enterprise during a load transfer from utility power to generator, we recommend transferring the data center load to the generator in advance of Tropical Storm force winds (39 MPH) reaching the area. Prior to the transfer, ensure that all necessary steps have been take to ensure that sufficient fuel is in the generator tanks and that the generator is in good operating condition.
  • Develop a Communications Plan. Maintaining communications among the information technology and facilities team will be critical.  It is important for data center management and staff to know how to reach one another through all the normal channels, including home phones, cell phones and personal email accounts. However, these forms of communication could be interrupted, so it might be useful to know spouses’ contact information or to have out-of-region locations where people can leave messages for one another. These forms of communication can be confirmed in advance of predictable bad weather. Often, when one channel is blocked another one can be still be used; it pays to know all of them in advance.

Pergravis has the expertise and knowledge to create a formal data center disaster recovery plan for any client.  This article is not a formal disaster recovery plan. The recommendations are not based on any analysis or understanding of a specific enterprise or facility. For more information regarding data center disaster recovery or for a consultation please contact Pergravis at info@pergravis.com.

Tags: ,