IT - IT Management

A few weeks ago we lived a blackout that affected almost all our customers to a greater or lesser extent. I would like in this post to go through some of the most notable points of those hours of uncertainty.

Of these points there are good, bad and ugly. All related to our own IT services and our customers, the operation and plans that should have worked differently.

And I speak of our own because within the incident we also saw part of our operation affected. These things we have to learn were identified during the crisis and analyzed later.

From the first minutes we got in touch with our clients, no matter what type of service they had contracted with us. We wanted to identify the following points:

1- If they had affected services.

2- Status of the business operation.

3- What was the action plan, if they had.

4- The criticality to return the service during a Sunday.

Because we have clients in the most diverse verticals, some had to operate on Sunday mandatory, and others could wait for Monday morning.

From these communications with customers and the answers we obtained, we created this list.

Let’s start with the good, which I think worked well within the entire uncertainty process.

We are not as bad as it seems

This is one of the good points that I want to highlight. Despite what one may think, few of our clients were completely affected by their services.

At least from the point of view of the primary services of the data center.

In almost all cases, by one method or another, the services continued to be delivered with minimal interruption.

The plans and investments made in recent years proved to be efficient.

Communications are still running

This is another good point we can learn from. Although there were many who had electricity in their data centers, they did not have access to them due to failures at some point in the communications.

Something that did not fail at all until a few hours inside the blackout were cellular data services. This means that we can take advantage of modern communications technologies such as SDWAN and leverage them through backup services such as cell phone lines.

It is clear that it is not to replace remote access services, but to offer an additional path.

If cellular data services proved to work as they did, we should consider adopting them as backup.

Access to information vital to the operation

Clearly this is one of the bad points we have to learn from.

Having the information stored in our web services and not being able to access them for lack of communications is something that should not be repeated.

From not having a copy in another online service of that document with the VPN configuration procedure or, much more seriously, the data center shutdown procedure.

We had a case where we were within minutes of having to shut down the entire data center and there was no access to the orderly shutdown procedure.

If we have the information published from our data center, we should have some of this information replicated in another service that allows access if necessary.

Decisions are made before

Within the bad points I also want to talk about the lack of planning for this type of scenarios. While the most common thing to analyze, and create, is a disaster recovery plan, different scenarios have to be raised.

Within the practice of DRP we always have to analyze different scenarios, such as a blackout or a prolonged event that will impact the service.

These scenarios arise during the stage of creating the plan and before each of these, the actions to be taken are defined. Doing this with such anticipation allows you to make much better decisions.

During the blackout we found many cases where there were no definitions made before the events and we found a lot of improvisation.

This event should immediately trigger a review of your disaster recovery plan if there is, or otherwise, the creation of one.

Tests are necessary

For the end, I leave the ugly, the worst point of all points to my liking.

We found a case, of which we made initial contact, that when consulting the status of their services informed us that the UPS had supported the load and that they would start the generator manually manually.

He informed us that they had enough fuel to operate for hours once the generator started and that they were without difficulties.

The generator never started, the batteries ran out and the rest … is history.

In this case there was a quarterly generator test plan.

A plan that was not executed two years ago. Yes, two years.

During the incident, in some cases, there were moments of crisis that could have been avoided with planning, with evidence and with a disaster recovery plan that had not even been fully executed.

Do not wait for another blackout to plan and make the definitions now, when we are calm.

From all this we have to learn. The good, the bad and the ugly.

Are you interested in knowing more about good DRP practices? Contact us https://www.wetcom.com/page/contactus

This post was first published in Spanish at 5 cosas que podemos aprender del apagón.

As some of you may know, wetcom turned 13 a few weeks back. But with the birthday celebrations another thing happen that made us prouder than ever before.

Well, what happened?, you may be asking. vmware awarded wetcom as the partner of the year for the first time.

That made me think what happened, what we did in order to get there starting with only 2 laptops and 2 backpacks. 13 years is a bunch of time and we never got the awarded like this.

Finally I identified 3 pillars that contributed to this achievement that I would like to share with you today.

The first pillar that contributed is teamwork.

Wetcom’s team integration and cohesion is the result of several initiatives we encouraged during the last few years, that made every team member contribute from their position or internal team.

Without the team we have today it would be imposible for us to get recognized as we have been.

What make this team so special? It’s special because the team over-performed the individualities present within.

And when the teams over-perform individualities great things come true, things like this.

Starting from professional services, marketing, human resources, sales and back office all played a strategic role that allowed wetcom to achieve the award.

Focus is the second pillar that I identified to be crucial to this success.

How many technology vendors have you seen developing cloud and virtualization solutions?. How many solutions around this ecosystem?

It would have been completely easy for us to pick a lot of those technologies during 13 years.

Provide services of the quality that we do with a broad spectrum of technologies and solutions would be hard for our standards.

Quality over quantity.

We started operations as a vmware business partner from day zero. And along with a small chosen vendors we kept working with vmware technologies all this time.

Being focused on a small group of technologies allowed us to provide services and solutions at a level not seen very often.

And the third one, the third pillar, is the ability to hand in there.

The main reason why companies fail to achieve great results, is that they give up at trying when things don’t get the way they want.

We waited 13 years to get to the point we are today, this ability to work knowing that the efforts we were making will eventually payoff kept us moving forward in tough situations.

A certain level of maturity is required to understand and behave like this and it’s not always easy to do.

If you master this and don’t get desperate for short term results I’m confident you are in the right path.

To recap, if you have an awesome team, you work laser-focused, and have the will to hang in there long enough, great things can happen to your organization.

Of course that there are more pillars that contributed to Wetcom’s successes in the last few years, but hey we have plenty of time so I will tell you in another post.

Thanks for reading.

Almost after 2 years of my last published post here I decided that it was time to get back to roots, but this time, in English.

If you are one of the old time readers may be asking why the switch and the answer is simple. Altho blogging allowed me to make friends and grow a business is time to broaden the audience in order to help me take the business to other shores.

Not better, but new.

If the time allows me to translate the content published here believe me that it will be republished in Spanish, have no doubt about that.

My idea is to restart writing about technology, businesses, and of course something that will help others take their business to the next level.

So stay tuned to read great content in the next few days!