The Power of Community, the Importance of Contingency Planning and a Lesson in Humility

In 2019, real estate developer United Properties, owned by Pohlad Companies, began construction on a new 37-story building in downtown Minneapolis. Gateway Tower would be home to a Four Seasons Hotel, the U.S. Headquarters of RBC Wealth Management, and 34 private condo residences. With 1.2 million square feet of space, stunning architecture, and a historic setting, the building would help revitalize downtown Minneapolis.

I was tasked with leading the technology workstream for the build-out of the Pohlad Companies office space at the new building. We began by selecting vendors to assist us with planning and designing the network architecture and AV solutions.

We worked through the planning and design process just as the COVID-19 pandemic was picking up and it became apparent that it wasn’t going to fade away after a few months. We began having conversations about how the network and AV needs of the new space would be different in a post-pandemic world. We applied the lessons learned from a year of remote work and the prospect of hybrid work models to ensure that all conference rooms would have videoconferencing capabilities instead of just one or two dedicated videoconference rooms per floor. Furthermore, we made sure that those rooms would be platform agnostic and leverage a “bring your own meeting” approach, as it became clear that user preference and flexibility would overrule any desire for standardization and consistency.

We finalized the design stage and placed orders for all the network and AV equipment in July 2021, more than 8 months before we were scheduled to move into the new building.

Now, it’s confession time. Once we placed those orders, I admit to bragging about how early we were. Our equipment was all scheduled to arrive in December 2021, almost 5 months before our actual move-in date. We were in great shape! I had heard rumors that other tenants had not ordered their network equipment yet, and I gloated about the fact that we were so proactive compared to everyone else. We were confident that we would not run into delays due to the chip shortages and logistics challenges that were just starting to cause problems globally by that time.

That is why I must accept full and total responsibility for what happened next.

Most of our equipment arrived on time in December 2021, but we were notified by Cisco that four line items would be delayed until February 4, 2022. Hmmm, that’s a long delay. But, our target network go-live date was not until mid-March, so that still left us plenty of time, right?

I’m generally a very optimistic person, but when building teams, I know that I need to surround myself with people who have complementary personalities. I intentionally look for people who are detail-oriented, cautious, and even a bit cynical to balance out my optimistic, big-picture, visionary approach. In fact, my most valued team member is someone I affectionately refer to as Eeyore. I often say that I need those people to dot my T’s and cross my I’s.

So, in early January when my team suggested that we should begin developing a contingency plan just in case the delayed equipment did not ship on time, my gut (and 25 years of experience) told me to listen to them. In our next weekly project status meeting, I began pushing our vendor to help us develop a contingency plan. At first, they thought we were overreacting. They assured us that no customer has ever seen their ship dates bumped twice, and we would definitely receive our equipment on February 4, 2022. But we insisted, and the team began looking at options.

Sure enough, on February 4, we learned that the remaining equipment was not shipped on time, and new ship dates were given for the back-ordered equipment: two firewall switches, twelve access switches, two wireless controllers, and two outdoor access points.

For the sake of the non-technical audience, the two firewall switches were necessary to connect the entire network, securely, to the Internet. The 12 access switches would enable about 550 individual computers, printers, and other endpoints to plug into the network. Together, the 14 back-ordered switches were necessary devices to allow employees to connect to the Internet from their desks. The wireless controllers were necessary to provide a Wi-Fi network for employees and guests. And, the two outdoor access points would extend that wireless signal out onto the private terrace of our fifth-floor offices. (OK, so the outdoor access points weren’t mission critical. But the rest of it was!)

Originally, the ship dates were early March, right about the same date as our target network go-live date. But, within days, our ship dates were bumped again, all to various dates in May and June 2022. Our move-in date was set in stone as April 22, 2022.

On February 11, we met with Cisco’s director of U.S. client operations. This poor guy. I’m pretty sure his entire job for the entirety of 2021 and 2022 was joining Zoom calls with people like me to explain why we hadn’t gotten our stuff and why we shouldn’t expect to see it any time soon. He explained the reasons for the delay included the expected culprits: logistics problems, chip shortages, and staffing challenges as a result of COVID. But, he also explained that Cisco’s global supply chain issues were exacerbated by the snowstorms in Houston in February 2021, the blockage of the Suez canal in March 2021, and the rash of airline flight cancellations during the Omicron surge. I was just waiting for him to round out the list of excuses with murder hornets and Godzilla.

In the executive meeting that week, I shared the high-level update about the equipment delays. My boss, Matt asked, “So, what does that mean for our network and move-in date?” I replied, “Well, Matt. There’s been a lot of talk lately about the ‘future of work’ and the ‘new normal.’ It’s time to embrace the ideology that we come into the office to connect with people, not with the Internet.” Matt was not amused.

Fortunately, my team and I had already begun developing the contingency plan that would allow us to bring the network online by March 14. That would give our security vendor sufficient time to complete their work before the construction management company handed over the keys to our office space. Our AV vendor also confirmed that they would be able to have all AV equipment installed, tested, and ready for move-in by April 22. (Shoutout, AVI Systems!)

Over the last ten years, I have established an internal peer mentoring group consisting of the IT leaders from each of the Pohlad Companies operating businesses. We call ourselves the Pohlad IT Council, and the group has become a tightknit community of trusted colleagues and friends. We meet regularly for the purposes of knowledge sharing, collaboration and helping each other out when needed. Everyone in the group was closely following the progress of the Gateway construction and our technology design and implementation.

When the technology workstream began looking into contingency plans, one group email to the IT Council was all it took to find what we needed. We quickly learned that Carousel Motor Group had two Cisco Catalyst 2960 switches they were no longer using that would work as temporary substitutes for the two delayed firewall switches. And then, the MN Twins dug under the couch cushions in the player’s lounge and found 17 Cisco Catalyst 3750 switches. I have no idea why they were hanging on to that many old switches, but I was glad they did because those devices could be used as temporary substitutes for the 12 access switches we were missing.

Now, don’t get me wrong, this is not ideal. The devices were end of sale and end of support three years earlier. We asked our network partner to perform a health check on the switches to ensure that 12 of them were working optimally and do any maintenance that might be needed. We also worked with Cisco to find out if the devices could be properly licensed so they would be supported while in use. We needed to replace the temporary switches with our new switches as soon as possible once they were delivered. We had to invest an additional $15,000 in different cables and connectors to attach the older devices to the new network. We experienced a few days delay trying to source the right connectors.

But, in the end, our wired network went live on time, using the borrowed equipment, but still allowing our security and AV vendors to complete their work in time for our target move-in dates. We determined that we could install the wireless controllers on virtual servers instead of dedicated appliances, so our Wi-Fi network would also be live by move-in date.

There are a lot of lessons to be learned from our Gateway Network project, including the power of community in problem solving, the value of creating contingency plans early, and importance of not tempting fate by boasting about how early you’ve ordered equipment when a pandemic is wreaking havoc on global supply chains.

And now, the final update in this multi-year saga: the last of our backordered switches finally arrived on February 13, 2023 a full 18 months after the original order date, and 10 months after we moved into the new building.