Crisis Management takes note on blogging the Rockies

by mike

In a fun twist the blogging on the recent Colorado Rockies ticket snafu got noticed by some folks involved at Bernstein Crisis Management PR Firm.

These folks have taken an interesting angle on blogging with regards to the immediate availability of news through the blogging community over status quo news agencies. Basically if you want to find out what is happening you might be better off doing a google or technorati search instead of just hitting the standard news channels. The reasoning is simple, with cell phone cameras (both photo and video) and sites such as youtube or personal blogs regular joes like you and me can be giving the rest of the world news faster than organized news channels. Interesting and fun for this blog to have gotten a little press so I hope you don’t mind my sharing.

Here is their article…

CRISIS MANAGER UNIVERSITY

Editor’s Note: When I received a short description of this situation from the author, a long-time “Crisis Manager” reader, I asked if he could develop it into a case history. He did so, superbly!

The Rockies Were Rookies at Crisis Management
By Robert Austin, APR

With its proximity to the Rocky Mountains, 300 days of sunshine per year, 650 miles of urban bike trails and over 20,000 acres of public parks and open space, Denver is a perfect setting for sports of every kind imaginable. Add in eight professional teams and you have a town where sport can be a religious experience.

When a team in such a place makes it to a championship, it is, for many, more rare and significant than a virgin birth. Tens of thousands will make whatever pilgrimage necessary to touch the holy grail of a ticket. It follows that when the Colorado Rockies made it to the World Series, fans prepared to camp out at Coors Field for the lottery that might lead them to salvation. Five days before the sale, however, the Rockies’ front office committed a blasphemy that put fans up and down the Rocky Mountain Front Range in crisis mode:tickets would be sold only online.

Immediately, fans and analysts began to question the logistics of the decision. Local news stations invited computer experts to give their opinions on how any system could handle the expected onslaught of Internet users looking for tickets from around the world. ESPN easily found non-experts to comment, quoting one hopeful fan, “If 250,000 people are online trying to get tickets, I can’t imagine the Rockies’ Web server can handle that sort of load.”

After thoroughly debating the technical aspects, attention turned to the fairness of the process. Some speculated that scalpers would use automated software that would enable them to process requests much faster than a human could. Others pointed to the number of fans disadvantaged with slow, dial-up connections or no Internet access at all. Still others pointed to the fact that anyone, anywhere in the world was able to stand in line, a situation that a queue at Coors Field might have averted.

Throughout these debates, the Rockies’ front office stayed on message. The fears of technical snafus were met with confidence. Both the club and Major League Baseball gave reassurances that all would be fine. Rockies spokesperson Jay Alves insisted the club’s computers were ready to go and said the staff prepared for any crashes. “We don’t anticipate that, but if something happens, we’re ready for that too,” he said. Matthew Gould, vice president of corporate communications for MLB, allayed fears by pointing to experience. “Obviously, this is not a first-time thing for us,” he said. No one gave any details about contingency plans.

On Monday, October 22 at 10:00 a.m., the sale began. Tens of thousands of fans sat in front of computer screens, aimlessly clicking the “refresh” button only to be repeatedly faced with the ominous message “site not available.” Two hours into it, Mike Miller, author of the blog The Urban Brain performed a “whois” search for evenue.net, the apparent host for the online sale. From that search, he obtained a phone number of the parent company, Paciolan, and called it. Miller then scooped news outlets with a blog post describing the developing crisis:

“The very nice lady who was answering phones caught on to what I was up to right away as she asked ‘are you calling about Rockies tickets’.

‘Yes’ I replied. So she gave me the scoop.

They are having issues, big issues, on the hosting side with their ISP. Whatever is going on, they can’t resolve traffic between their servers and the outside ISP’s.

They have contacted their ISP and are hoping for some resolution soon. They don’t know if it is something really big or something really easy to fix, but either way they are dead in the water and the update they gave me is that fewer than 100 tickets have been sold so far.

She said the IT guys just were not expecting this mass amount of response and they whole system is overwhelmed.

Well duh, what did they think would happen? Someone is going to get a little dose of reality about this whole thing.”

Three hours after Miller’s post, Rockies Spokesperson Alves appeared at a press conference in front of eager reporters. Enough fans had shown up at the ballpark for furious chants of “we want tickets; we want tickets” to be heard over the spokesman. Alves announced that ticket sales had been suspended due to a system crash brought on by 8.5 million hits in 90 minutes. The crash was so bad that all of Paciolan’s North American customers were affected.

At 4:36 p.m., the Rockies posted a press release on the club Web site announcing suspension of the sale. Prior to that, the Web site still featured a banner announcing the sale was in progress. It wasn’t until the 10:00 p.m. newscast that the club announced the system was fixed, that sales would go back online at noon the following day and that a contingency plan was in place should anything go wrong again, although no details of that plan were given.

By this time, the club and Paciolan had also figured out the problem: a malicious attack. “Our Web site, and ultimately, our fans and our organization, were the victim of an external malicious attack on our Web site that shut down the system and kept our fans from being able to purchase their World Series tickets,” Alves said.

Yet, even fans with no IT background doubted that claim. One fan, who had been a victim of such an attack said flat out, “They’re lying.” Drew Curtis, the owner of Fark.com, told The Denver Channel. Com: “If they notified their upstream provider that they were under attack, the upstream provider could have shut that off in no time flat.”

The following day ticket sales went off without a hitch and some 60,000 tickets were sold within a few hours. At this point, news outlets pointed out that no investigation was being launched for the perpetrators of the attack. Analysts and computer experts began a new discussion on this topic.
The Aftermath

In the end, tickets sold out and, inevitably, thousands ended up disappointed and ticketless. So what was the actual damage?

* Confidence in the club’s management faltered. On blogs and news outlet Web sites, fans wrote at length about the poor treatment the club gave its fans.

* Business lost immeasurable productivity. Many fans took the day off to try for tickets online. Others used office computers. In fact, a Denver Post news report blogged live from the offices of Blue Cross/Blue Shield detailed how several employees tied up office computers for several hours trying to access the site. The fact that the entire process was repeated the next day may have doubled the loss.

* Paciolan’s entire North American customer base were thrown into crisis mode as well. They, too, likely suffered loss of sales. For example, the Denver Center for Performing Arts also uses Paciolan for online ticket sales. The DCPA first started to experience a slowdown shortly after the Rockies sale began. Soon after, the system failed entirely. During the outage, the DCPA posted a notice on its web site advising ticket buyers to order their tickets by phone or in person. Those were options Rockies fans didn’t have. Anita Edwards, the Center’s Web services manager said, “harm done.” It would take another few weeks before the Center could assess any financial damages.

The Lessons

This crisis began long before the servers crashed. It began the moment the Rockies made the announcement that tickets would be sold only online. It was at that moment that the club’s public, its fans, went into crisis mode. Many made immediate plans to take the day off, to have multiple computers set up to try to up their odds of scoring a ticket. After the announcement, news outlets and computer experts began to ask questions that were answered too easily and almost flippantly: “Obviously, we’ve done this before.” Questions on the fairness of the decision were left unanswered. In announcing the decision, the club simply stated that it was the most fair way for fans to get tickets - period, no discussion.

As communicators, we’ve all heard the tips on how to manage crises. We learn about the first, golden hour of a developing situation. We list out steps to be taken, messages to be disseminated and follow-up to be carried out. We practice how to talk to the media to convey those messages and how to think on our feet with microphones pressed in our faces. We train our employees not to make unauthorized statements to members of the media (or others, for that matter).

Each of these lessons is reinforced by the Rockies’ ticket woes. Among them:

* The club was slow to respond to the media and to its fans. Blogger Mike Miller exposed the crash at least two hours before the club made any statement. It took five hours to get a statement on the Web site - the very place where thousands of fans had been camped out for information.

* When the club and its partners did give an explanation, a “malicious attack,” no details were given and no investigations were launched. To skeptics, this was further evidence that the club, MLB and Paciolan simply weren’t taking responsibility for messing up.

Ultimately, however, the biggest lesson lies in the failure to think the decision to sell tickets exclusively online all the way through in the first place. Our job, as communicators, is to try to anticipate the perceptions, attitudes and actions of our publics. That means examining every policy decision and asking the “what if” questions.

Sure, it is possible that our colleagues at the Rockies don’t have a seat at the table where these questions can be asked, which merely underscores another lesson of crisis management. Nonetheless, someone at the club should have been asking the questions. What if we sold the tickets using a system tested by other clubs in the past? What if the servers can’t handle the load? What if there was a back-up plan to shift to in-person or phone sales if the server did crash? What if we’re wrong and this system isn’t fair to local fans? What if we did have a lottery system, even if it were for online only sales? What if a rookie performance in ticket sales somehow translates to a rookie performance on the field? We may never know.

Robert Austin, APR, is Director of Professional and Public Relations for the Rocky Mountain Lions Eye Bank in Denver, CO. Contact: raustin@corneas.org.