Over Easter weekend, an outage happened to the largest real estate technology company in America. They recovered quickly and elegantly. On another side of life, a broker client of ours with 400 agents was given one-week notice that the company who provided Microsoft Exchange Mail services is going out of business. The realty is that software systems and hardware systems break. And when it happens, it is very disruptive. It is a good idea to think about your disaster plan every year, and review your vendor contracts. You should have a document that outlines everything that happens when systems fail.
Redundancy, Back Up, and Co-dependence
Redundancy and Back Up are loaded terms. It is one thing to have copies of your critical systems (Back Up) – it is another think to have those systems online and available (Redundancy). Moreover, a lot of online software is co-dependent on other software. Here are a few examples.
When the MLS system went down, other services did not. However, since many of those systems were only accessed through the MLS single-sign-on system – agents did not know where to go to access them. This could mean different things for different MLSs. For example, if the sign on security system or the MLS is not available, services like Instanet, ZipForms, or Realist may be working just fine – but agents do not know how to access those services without going through the MLS.
The solution to all of this is to have a plan in place, and have staff ready to manage DNS so that users trying to access services can get information on what happened, what is happening to fix it, what systems are impacted, and how to access solutions that were not impacted. (DNS is “Domain Name System” – it directs consumers to specific webpages and allows traffic to be redirected as necessary).
Stop and Think
Many agents lost their minds when the MLS service was interrupted. Strangely, I don’t know why. Aside from a temporary inability to add or edit a listing – all or most of the information that an agent needs was available through other sources – including their own website. I know that showing instructions and agent remarks are invaluable to agents. But you can always pick up the phone and call the listing agent if you are stuck. Sold data and tax data are also quite available online. Sure, it is not as convenient or efficient to find workarounds for service outages – but it is doable. Make the most of the challenge. You may be surprised at how resourceful you can be.
Communications Plan
The most important plan you need to have when you have any type of system outage is a plan for communications. Many MLSs turned to Facebook to advise customers of the problem and the solution. Remember, understanding why something is broken is often hard. Users ask themselves, “Is is me or is it the system?” Moreover, Murphy’s Law is that the outage will happen at the very worst possible time, like a holiday weekend. Make sure that there is someone appointed to orchestrate the communications plan every night, every weekend, and every holiday. Something as simple as being able to record a service announcement on the phone system can go a long way toward calming everyone down. Some MLSs like MRIS even offer a text messaging option that is used not only for advising on routine maintenance, but also communicating emergency status.
If you have not taken a look at your emergency plan – pull it out and dust it off. If you need a review – let us know. We are more than capable of giving you a hand.
This is a great article Victor. I love the Stop and Think part mostly. I had several agents contact me on Sunday during the outage and I gave them all similar advice. The best thing we did as an Association was to have our Facebook / Twitter Page updating what was happening and that kept a lot of folks happy with knowing we were aware of the issue. Corelogic did a good job of keeping us on top of the issues, the timing and eventually getting all systems restored. These types of issues happen, and what makes a great company better is to review their current practices and make them better so they can prevent something like this in the future!
Thanks Peter. I was watching the entire thing unfold and we were on the phone with a few MLS customers providing them support. A lot of key folks were traveling or at church. One thing that I can tell you is that many MLS staff members need social media training on handling public relations problems. You need to begin with the understanding that some customers will be unreasonable and behave badly in these types of circumstances. Don’t be surprised when an agent goes bizerk. Keep your cool, listen, empathize. Never fight fire with fire.
On another note – CoreLogic did a great job recovering. It takes an insane amount of energy to respond as well as they did. I noticed that your team did a great job too. So did most of the large MLS providers. The small companies did not fair so well. A few of the 2 or 3 person Associaton owned MLSs absolutely failed. If they were reachable, they did not know what to do. For the life of me, I do not understand how brokers tollerate that level of service.
I don’t see how being out for 48 hours is acceptable in any terms. It is easy to be elegant if the recovery time is 2 days. I am sure Amazon and google if down for 2 days would not call a 48 hour recovery elegant.
Hi Ben, I understand the pain. For me, going without my computer or phone for a day or two is a personal challenge. I presume that you are an agent from the upstate new york region. Was there a particular action that you needed to take in the MLS which you could not wait to perform, or do another way?
From our standpoint it was down for 24 hours. I am not sure what market Ben is in but we were down early Sunday morning and back up early Monday morning.
A 48hr. “Timeout” for Corelogic is one thing. To compare Corelogic to Facebook or Twitter
is irrelevant. Realtors do not count on Facebook to make a living.
These ‘events’ are not limited to Corelogic but are happening here in North Texas with increasing frequency. Backing up Data has never been more critical.