Minimizing downtime and ensuring cyber resilience are vital for industries dealing with critical infrastructure that depends on real-time operations.
MSPs servicing manufacturing, power and energy, oil and gas, healthcare, and retail customers know firsthand how minutes of downtime comes at a significant cost to their customers. Those customers frequently have complicated system setups that require complex configurations. Therefore, recovering the system to its previously known working state is crucial to avoid lengthy and costly downtime.
One of the solutions to minimize downtime in those environments is bare metal recovery.
Using Bare Metal Recovery
Bare metal recovery involves restoring the entire production system, the operating system, applications, settings, and data, to new or existing hardware. Unlike basic file and folder backup, bare metal recovery requires a full image replica of the system, and the ability to inject system drivers on recovery to dissimilar hardware.
MSPs should implement frequent full-image backups to local or network-attached storage, and replication to the cloud to be able to access the replicas online or recover in case of a disaster impacting local storage. In addition to scheduled backups, it is crucial to create full system replicas before any critical changes — operating system updates, application patches, configuration changes — to have the ability to roll back to the previously working state in case of any failures.
When a system experiences catastrophic hardware failure, ransomware attacks, or a natural disaster, a properly implemented bare metal recovery procedure and tools can reduce downtime from days or hours to even minutes.
Choose the Right Tool
When selecting the right tool, MSPs must consider the hardware and software supported. In many environments with critical infrastructure, old hardware and outdated operating systems are abundant.
Another consideration is performance and reliability. It’s important to do careful testing and evaluation of:
- Recovery time objectives (RTOs), showing how fast you can recover
- Recovery point objectives (RPOs), showing how much data loss is tolerable in specific environments
Hardware, storage, and network configuration can significantly influence the performance of bare metal recovery tools. A solution that works well in one environment may not be suitable for another.
The last consideration is the efficiency of compression of the backup data. In some cases, the system size is small, so storage capacity is not an issue. However, if you use centralized storage and multiple systems to store multiple replicas, storage capacity can quickly become an issue.
Implementing Bare Metal Recovery Processes
The best practice for MSPs is to create a standardized deployment procedure for specific types of client environments. The procedure should cover hardware and software inventory, network configuration, recovery sequence, and testing protocol.
Standardization and documentation of the procedures simplify employee and customer training, reduce human error, and provide consistent recovery experiences across client environments. It is especially important if the selected solution enables customers to do self-service for recovery without IT personnel on site. Develop a simple recovery guidance for the systems’ users and train users on the procedure.
Gaidar Magdanurov
Test, Verify, Repeat
The most common pitfall in bare metal recovery is failing to execute regular test recovery. Extended periods of time without failure can often lead to complacency, yet a backup that is not tested is the same as no backup.
MSPs should perform automated backup verification and weekly or monthly recovery testing on production systems during planned maintenance downtime or on spare hardware. Further, customers often require reports on recoverability, so the testing procedure also enables MSPs to provide these reports to meet compliance requirements.
Automation
Automation reduces human error and accelerates recovery times. Typical scenarios for automation with scripting of built-in tools include:
- Backup scheduling
- Backup verification
- Recovery environment preparation and testing
- Post-recovery validation and reporting
Modern backup tools offer either built-in capabilities, APIs, or command-line interfaces for automation.
However, a note of caution: Malfunctioning automation can lead to disastrous results and unbelievable speed. Therefore, every modification to automation scripts or workflows requires sandbox testing before production deployment.
Typical Challenges with Bare Metal Recovery
The most frequent issue during recovery to new hardware is device driver incompatibility. To avoid this issue, the images must be tested on spare hardware before implementing the recovery process on new hardware. The drivers for the latest hardware should be readily available and stored together with the backup images.
The other challenge when using network storage is the complexity of network configuration and unexpected network configuration changes. Automated testing of backup recoverability and network monitoring allow for spotting issues before they become a problem. Also, if network storage is used, the full testing procedure for bare metal recovery should be executed after any network configuration changes.
Lastly, consider degradation of storage performance. Monitoring and frequent retesting with measurements of RTO and RPO would allow for early detection and prevention of issues.
Implementing a Client-specific Offering
When implementing bare metal recovery, you must balance cost and performance. Tailoring the system to the customer’s needs helps in negotiating a reasonable deal for the cost of the solution and services. It also demonstrates the MSP’s expertise to their end customers.
To develop a client-specific offering, MSPs should assess the business impact in terms of downtime cost and risk. Then, determine requirements for hardware, network bandwidth, storage, and recovery procedures. Based on that, MSPs can determine the cost of the solution and propose client pricing based on the target profit margin.
A few hours of downtime with the lower cost of storage and network may be suitable for one customer. But another client may be willing to pay for extra spare hardware, multiple storage arrays, and technical personnel available on site to avoid downtime.
Additionally, it is crucial to successfully implement bare-metal recovery procedures to educate customers on their roles and set realistic expectations on the recovery time and downtime risks. As part of customer interactions, regular reporting and review of business continuity plans should include an analysis of the risk and cost for the scenarios of bare metal recovery.
Conclusion
Bare metal recovery can be a strong competitive advantage for MSPs while working with customers serving critical infrastructure or dealing with the cost of downtime. However, the software solution for each environment should be carefully tested, and procedures should be documented and trained to provide a high quality of service as expected by customers.
Gaidar Magdanurov is president of Acronis.
Featured image: iStock
