May 17, 2012

Best Practices For A Cloud Service Outage?

By Balaji Viswanathan

Cloud Service Outage

If you are using one of the major CSPs (cloud service providers) you may already be used to major service outages. Amazon EC2, Rackspace, Google Apps and Microsoft Azure have all had their fair share of outages in the past 18 months, and some of them have been huge failures (Amazon’s April 2011 outage lasted 47 hours for some customers), which have brought down sites such as Reddit. However, companies such as Netflix have been able to survive the outages well. In this post, I will cover details on how you can effectively manage a service outage by taking note of best practices from Netflix and other companies that have successfully weathered the storm.

Create a disaster management fire-drill

Fire drills should be an essential part of cloud management. Once a month, have a fire drill to simulate failures in different parts of the system to see how your systems hold up in case of failures. This includes preparing your PR and customer service personnel, instituting quality control processes and executing an executive-level contingency plan to prevent panic from gripping the company.

Have all your data backed up securely

Periodically back up data and store it away from your primary CSP. For example, you could have an Amazon S3 Instance to back up your Rackspace cloud installation. This will mitigate against a single point of failure.

Keep another service provider ready

Have another CSP ready to run an instance of your server at short notice, if needed. Even if it doesn’t provide full features for the site, this plan B should provide a minimum working subset. (In an email application, for instance, the service must allow you to send and receive email, even if contracts and archive access is not restored.)

Create stateless systems

One of the lessons Netflix offered was to build stateless systems where possible. That means a new request from the client can be served by any of the available servers, even if the original server to which the client made the request is down. This requires very careful planning during development.

Work on graceful degradation

Your system has a graceful degradation when a certain percentage of failure causes only an equivalent percentage drop in performance instead of bringing down the whole system. To enable graceful degradation, you must detect failures quickly (set  quick timeouts when the system recognizes a failure) and shut down all non-essential features of the system (to save precious resources for critical features).

Create a communication plan to keep customers in the loop

If there is an outage at your CSP, your customers will also be affected. Imagine you are running an online store and the outage has prevented you from shipping. Even if none of the previous disaster management steps worked, you could save some of the bad press by keeping the customers and other stakeholders regularly informed of the status. Identify proper communication channels and create a plan to keep all of them in the loop. This requires having a backup of customer contact data, writing FAQs and preparing your employees to handle questions appropriately.

Having a proper service outage plan is essential to the survival of your business in the long-term. It could save a lot of headaches, not to mention your brand value, when failures do happen.

By Balaji Viswanathan

Balaji Viswanathan

Balaji is the CEO of Invento Robots. Balaji has been writing about technology for several years and is currently featured in many publications around the globe.
Lex Hegt

How Can Organizations Effectively Monitor and Analyze Their Azure Billing Data?

Monitor and Analyze Azure Billing Data With the ever-increasing investments in Azure, many organizations struggle [...]
Read more
Sushil Kumar

Generative AI and Cloud Computing: The Greatest Infusion

Generative AI The fusion of cloud computing, app modernization, and artificial intelligence (AI) drives digital [...]
Read more
Michael Kleef

Akamai’s Michael Kleef Reveals Key Shifts in Cloud Computing Landscape

Welcome to a conversation with Michael Kleef, Vice President of Product Marketing, Developer Advocacy, and [...]
Read more

Innovative Solutions Ensuring Cybersecurity in Cloud-Native Deployments

Innovative Solutions Ensuring Cybersecurity The digital landscape is evolving at a breakneck pace, and organizations [...]
Read more
David Cantor

Impact of AI in Storytelling and Creativity 

These are monumental topics that command volumes of diligent research, backed by empirical evidence and [...]
Read more

5 Azure Cost Management Strategies

What Is Azure Cost Management? Azure cost management refers to the practices and processes that [...]
Read more

SPONSOR PARTNER

Explore top-tier education with exclusive savings on online courses from MIT, Oxford, and Harvard through our e-learning sponsor. Elevate your career with world-class knowledge. Start now!
© 2024 CloudTweaks. All rights reserved.