T Minus 12 Hours…and NO network!

New store openings in retail are a big deal (hello, Mr. Obvious)! Let me restate, new store openings in retail are a VERY big deal! I don’t know if we have ever had to delay a store opening in our history, I certainly know it has never happened during my tenure. So when we were 12 hours away from a grand opening and still didn’t have network connectivity to the store I was sweating bullets…big time!

How we came to be in this position is not the point, but I will give a little context. We are in the midst of a redesign of our WAN (wide area network for those that hate acronyms). Rather than going with our current platform, we made the decision to go with a new platform (ok, when I say “we made the decision”…I really mean “I made the decision”). There was some confusion in placing the initial order that caused a delay. We then learned the build out would take six weeks (did I mention we were now three weeks out from opening?)! Never fear, our new platform included redundancy in the form of a 4G failover. We would just use that as our primary until the permanent connection was established. With Plan B firmly in hand we (again, I mean I) decided to move forward.

But, this is not a story about network connectivity, per se, this is a story about how a team of vendor partners came to our aid and saved our bacon! In a recent post, I wrote about the keys to a long lasting vendor relationship. These partners truly exhibited those keys and then some!

T minus 96 hours – no network connection

The signal for the 4G connection was not strong enough inside the building. Stringing a 100 foot coax across the floor and out the back door, we were able to get a solid signal. We called one of our partners to ask them to mount an external antennae. They dropped everything to come out and do a site survey (late on a Friday afternoon, mind you) and order the parts. However, they took it a step further. They took it upon themselves to contact one of THEIR partners to implement a Plan C, dropping a high speed DSL line in as a temporary solution in case Plan B did not work out. Have you ever contacted a telecom supplier and requested service in under 48 hours? Amazingly, with several strings pulled, installation was scheduled for that Monday morning.

T minus 24 hours – still no network connection

Monday morning dawned and the parts needed for the antennae installation arrived and our partner went about drilling holes in concrete walls and installing the antennae. Mid morning the telecom vendor arrived to install Plan C, the DSL line. About this same time, I was assuring my boss that we would have connectivity and the store WOULD open on time.

T minus 18 hours – STILL no network connection

I (and several other VPs and my boss) arrived on-site for the customary grand opening open house for employees and families. I made may way back to the telecom closet where I found half my team, our cabling partner, the telecom technician, and two or three others. Tensions were high! The antennae was mounted, but we were still unable to get the connection stable. The 100 foot coax was still strung out across the floor , out the door to an antennae duct-taped to a traffic pole (no I am not kidding!). Everyone was concentrating on getting the DSL (remember, that is plan C) connection established.

Time for Plan D – call our existing partner to see if the line from our old location could be swung to this one. (a week minium, that would not do). Plan E – establish a line of site connection to one of our other locations (nope – believe it or not there is a 200 foot elevation difference…who knew there was a hill in central Indiana?). Plan F – ride on the wireless of another retailer in the area – with permission of course (so what are the odds Walmart would agree to THAT request?)

T minus 16 hours – the sweat is rolling

At this point, I took a risk. I texted one of the vendors that had been courting us for a year. We had done a little business (one other time when we got in a jam) but nothing that you would write home about. (did I mention it is 5pm?…I should also mention the telecom technician was still there lending a hand, incredible service above and beyond his normal duties). Within seconds, I had an answer back from my text. We jumped on a call. He rallied a couple of his senior engineers and they jumped on a call with our engineers.

T minus 12 hours – time to brush off the resume

T minus 11 hours – 

T minus 10 hours – 

T minus 9 hours 47 minutes – WE HAVE LIFT OFF! (Well we have a network connection, at any rate)

Man that was too close! Working through all the issues together we were finally able (we in this case does not mean ME) to establish the connection and begin testing the store configurations. All was well with the world…until the next morning when Murphy struck (man, we have to fire that guy) and unrelated to the store opening, our wireless controller crashed. One of the team, installed a personal wireless router to enable the store to open on time, while others got back on the phone with the same engineers that had worked through the evening the night before to troubleshoot and resolve the issue.

Without our partners (and in some cases, their partners) we would have faced disaster. If you will bear with me on this day before Thanksgiving, I would like to thank them here: Thanks to Patrick at TWTelecom, Thanks to Scott, Tim and others at BD Managed Services, Thanks to Kelly at Lingotek, Thanks to Dwight, Cale, Tom, Greg and Richard at Sinewave, thanks to the technician from AT&T (I failed to even get his name). I would also like to thank my team for their incredible efforts under pressure: Ed, Kelvin, Jeff and Patrick THANKS GUYS!!

If anything you read here or in other posts strikes a chord, I would love to hear from you. Leave a comment, hit me up on Twitter (@jtongici), find me on LinkedIn, or Google +.