Unlocking Network Potential: The Open Networking Revolution

Graham: Thank you Kelly. I'm Graham Wooden. I'm a Senior Director of Engineering here at Uniti Fiber. We are the operating arm of Uniti Group, which is a REIT, a Real Estate Investment Trust. We are publicly traded and we are the second-largest independent fiber provider in the United States. Our REIT consists of fiber infrastructure for cell towers, backhaul, wholesale, and we have over 8 million strand miles of fiber all throughout the U.S. We're here today to explain our disaggregated network solution. You know, it's been a great road. We've learned a lot. There have been a lot of pioneers in front of us that we've been able to examine, and I'd love to talk to you here shortly about all that.

Graham: Thanks Kelly, and thanks Alan, for giving the brief overview of disaggregated networking. As Alan mentioned, the COVID pandemic acted as a catalyst for us to move towards a new networking topology and design. We observed a significant increase in both wholesale and enterprise sales, which made it clear that our IP backbone was on the verge of falling short of the necessary 100G and 400G interfaces. To stay ahead, especially with supply chain delays from new manufacturers and the cost of gray market items quadrupling overnight, we realized the need to explore new technologies.

Our existing infrastructure, particularly our monolithic chassis 9006s, were already equipped with several 100G line cards. However, upgrading to the next generation line card, a 12 port, would cost us $60,000 to $80,000 on the gray market. Although this could temporarily keep us ahead of any surge, we understood it wasn't a sustainable long-term solution. With 40 IP backbone nodes carrying traffic throughout the Southeast, the economics of updating three or four line cards per chassis, considering both CapEx and OpEx, including user licensing, were simply not feasible.

This is where the benefits of disaggregated networking became apparent. By separating out our infrastructure, we began exploring routers from UfiSpace, which offered a 5x increase in interfaces at one-third the cost of a gray market line card. We also looked into how AT&T was implementing disaggregated networking by acquiring Avada, a move that, while not groundbreaking since Avada has been used on Linux servers for years, signaled a significant shift in the industry towards this type of infrastructure.

Our research led us to IP Infusion, which stood out due to its role as an extensive development partner for Broadcom on the StrataDNX ASIC chip. We were impressed by the supportability across UfiSpace and Edgecore routers and their internal development and external growth. EPS played a crucial role in connecting us with the right contacts at UfiSpace and Edgecore for hardware inquiries and developing our relationship with IP Infusion for software support.

Our IP backbone operates as a completely separate network from our Ethernet, necessitating specific considerations. We took extra time to thoroughly explore our options and use cases, ultimately forming what we believe to be a strong relationship with IP Infusion, UfiSpace, and Edgecore, with EPS's assistance.

Graham: Absolutely. I'll start with my experience with EPS Global. From the moment we were introduced, it was clear they were going to be a solid partner. The sense of assurance came not just from our account manager but also from the service and sales engineering team. They demonstrated a readiness to answer any questions we had, and if they didn't have the answers immediately, they were quick to find them. I always felt well taken care of, without any hesitation or unresolved issues. EPS showed a genuine interest in our business.

Working with EPS, especially given our larger customer status and footprint, has been seamless. We've collaborated on having hardware readily available in staging warehouses throughout the United States for any necessary RMAs. Fortunately, we haven't experienced any hardware failures to date, but it's reassuring to know that EPS has hardware staged in RMA status ready to ship if needed.

Graham: Yeah. So, and that's a great question Kelly. That was initially our, not hold back, but, you know, item that we wanted to address quickly was, you know, we're used to traditional support methods from larger organizations, you know, how we were going to receive the support from IP Infusion. So, you know, luckily here at Uniti, we have a pretty large lab. It's about 40 racks, plus the 40 racks. And each of those racks encompasses equipment and hardware and software that we run out in the field. So anything that we run out in the field, we're able to stage it properly in our lab first. We were able to, with help of IP Infusion, we were able to stage a lot of our gear in the lab and run through lots of test cases of how we were deploying out individual markets, how we're deploying out to the backbone and out to our edges, and then recreating various test scenarios. And while we did uncover a few little bugs, IP Infusion was quote unquote Johnny on the spot of getting them fixed, getting issues, you know, quickly handled. And it was reassuring that for a company our size, that IP Infusion was right there ready to support.

Graham: Yes, so when you examine our network and topology, it's evident that we operate hardware that is somewhat more expensive to run and consumes more power compared to smaller operators. This isn't to suggest any differences in behavior, but rather it's a reflection of our structural setup. As we began to assess how to meet our demand and ensure we had the necessary capacity for our wholesale and enterprise operations, power consumption wasn't initially a primary concern. We operate in a multitude of central offices where power regulation is somewhat lax, and we had resigned ourselves to the reality that some aspects would be regulated fairly while others, like power, would not. Our initial focus was on driving CapEx savings, contemplating the expenditure of $60,000 on an eight-port, 100G line card to push the limits of our monolithic chassis. We encountered situations where, after installing an eight-port line card, we found ourselves needing to upgrade to a 12-port card within a month or two, only to discover there were no available slots in the chassis to accommodate this change. We recognized we were approaching a capacity crunch.

Although we could have opted for a larger slot chassis, this would likely have increased our power demands. Re-evaluating our options led us to consider alternatives like the UfiSpace S9600-56DX, which offers 48x 100G ports and 8x 400G ports. The port density of this compared to our traditionally deployed eight-port line cards represented a significant saving, reducing the cost from $60,000 for a single line card to $20,000 for the router, while quintupling the number of ports. The operational savings, including software and licensing, were also considerably lower than with our traditional vendors. This realization allowed us to allocate our budget more effectively, enabling growth in other areas of the network.

The reduction in power consumption emerged as an unexpected benefit. Initially, we had accepted power costs as a given, but we soon discovered that our monolithic chassis were consuming up to 56-60 amps of DC power, which could be reduced to about 6-7 amps. This represented a significant cost saving, far exceeding our initial expectations. Moreover, aligning with our ESG and environmental goals became an additional advantage. By reducing our power consumption, we not only achieved financial savings but also contributed to more sustainable and environmentally friendly operations.

It's important to note that in these locations, we deploy two chassis for redundancy, with one handling the northeast and the other the south-southwest. Previously, we were drawing close to 120 amps of DC power to support our IP backbone. With the new setup, we've managed to reduce this to approximately 14 amps for both chassis combined. This drastic reduction in power consumption has led to substantial savings on our monthly bills to central office operators or data centers and has allowed us to be better stewards of our community.

Graham: Absolutely, Kelly. It does. And you know, there's other parts of Uniti as well that our ESG is very important. It's just that on the IP backbone side, we never, never real realized how much important it will be just because we just succumbed to the feeling that we would have to run these big routers forever. So it was a blessing in disguise that we were able to move to smaller chassis with more ports and have the same software capabilities that we're running before. So it was a win across the board.

Graham: Graham: Sure. Yeah. Well, I will quote the immortal words of Olivia Newton-John and say, "Let's get physical." The biggest lesson we learned wasn't about software, differences in the CLI, or any perceived notions of protocol changes. All that was straightforward. Switching from one vendor to OcNOS was much easier than I anticipated. What I didn't really anticipate were the physical changes. Our line cards were running optics with SC connectors, and these new chassis are running QSFPs with LC connectors. During our first deployment, we had the physical dimensions of the box spec'd out in a rack, and power cables run, but it quickly became evident that our SC connectors, the jumpers we previously used for the old chassis, couldn't be used on the new chassis due to the change from SC to LC connectors. We had to quickly redo our cabling to have LC ends.

Some might say, "Well, shoot, Graham, you should have caught that before." We were so hyper-focused on the physical size changes of the chassis, going from a 14RU box to a 2RU box, and rerunning the DC power because the lugs were a bit different. We just thought we'd move from this 100G optic on the old chassis to the new optic, then realized we were going from SC connectors to LC. The lesson learned is really about the physical layer. It's important to review all your physical plant rack space before you deploy. That way, you're not caught off guard at the 11th hour when you're trying to do maintenance and then realize you have to spend another 30-40 minutes running cables, which could have been done beforehand.

Graham: Certainly. The next steps in our journey, as I've touched on earlier, revolve around the strategic deployment of our backbone locations. We've opted for deploying two nodes primarily to diversify our failure domain. This approach is akin to not putting all our eggs in one basket. Our decision to deploy two nodes is driven by the need to mitigate the risk of a single point of failure affecting our entire network. Our IP backbone is entirely optical, eschewing traditional switch networks between nodes for a purely optical connection. This design choice necessitates a careful consideration of failure scenarios at the physical layer, drawing inspiration from the insights of living new John on the importance of planning for failure.

From the outset, we committed to standardizing on two backbone nodes to ensure redundancy. This setup ensures that an outage at one backbone node doesn't incapacitate the entire market or disrupt traffic flow, thanks to the presence of a secondary node. Our journey has seen us grow comfortable with IP infusion and Qualcomm's two-box solution. However, with the advent of Jericho2, we're exploring further diversification away from a single vendor. We've already incorporated Edgecore into our setup, but we're now looking to expand our use of Jericho2 ASICs in our backbone nodes. This strategy aims to safeguard against potential issues with either Qualcomm or Jericho2 components, ensuring our backbone remains operational.

Beyond the backbone routers, our focus extends to other critical components of our IP infrastructure. Our distribution routers, which interface directly with customer networks and handle BGP routing with customers, cache servers, and external IX connections, are under review. Traditionally, these have been 9904 or 9906 chassis, but we're considering transitioning to UfiSpace or Edgecore routers. Additionally, we're evaluating changes to our route reflectors and special service edge routers, which are integral to our service delivery.

This evolution, initially centered on backbone nodes, now spans the entire IP service delivery framework for Uniti, encompassing over a hundred routers. Our goal is to implement these changes across the board, enhancing the resilience and efficiency of our network infrastructure.

Graham: One aspect that particularly stood out and resonated with me regarding IP-Infusion was its history of previous licensing agreements with other vendors. This was through its license-only offering system, which I had encountered while using products from vendors such as F5 and Fujitsu. For years, I had been utilizing these vendors' offerings, unaware that the systems they incorporated were actually licensed components from IP-Infusion. This realization came to light during my recent exploration into IP-Infusion's offerings.

IP-Infusion has demonstrated a strong presence since the early 2000s, possibly even earlier, by licensing its components to other vendors. This fact served as a reminder of IP-Infusion's long-term commitment to the industry. It highlighted that IP-Infusion wasn't a newcomer that emerged in the late 2000s or 2010s. While that might be the case for its OcNOS side, the essential components that constitute OcNOS have been actively supported and developed by IP-Infusion since the year 2000.

These components have been operational in various vendors' offerings that we have been utilizing in different capacities for many years. This knowledge assured me that we were adopting software that would be integral to our lifecycle for many years to come.

Graham: So, reflecting on my IT career, which spans over 26 years now, I recall starting in a very small dial-up ISP back in 1997. Fast forward to 2023, I find myself working for a significantly larger ISP. One pivotal lesson I've absorbed early in my career, which continues to guide me in every new endeavor, is the application of what's known as the KISS (Keep It Stupid Simple) method, an acronym for Keep It Stupid Simple. The essence of this approach is to avoid overthinking and overanalyzing, which often leads to drowning in one's own thoughts and unnecessarily complicating matters. I consistently strive to adopt a simplistic perspective. While it's acknowledged that the end result might be complex, the initial review should aim for simplicity. The goal is to keep it stupid simple, as advocated by the KISS principle.

For those contemplating moving away from monolithic chassis types or transitioning from a current vendor to alternatives like IP Infusion or a disaggregated offering system, my advice is to focus on the very basics. Consider what protocols you're using and the physical outcomes you aim to achieve, such as size reduction or reducing the number of interfaces. Perhaps there's a need to transition from a multitude of 1G or 10G interfaces to more advanced 100G or 400G options. At this stage, I would recommend not dwelling too much on the protocol specifics or the appearance of the CLI (Command Line Interface). Initially, we anticipated challenges in these areas, only to discover that the CLI was quite user-friendly and similar to what we were accustomed to. Furthermore, all the protocols we traditionally relied on were available. In essence, we realized that our initial concerns were misplaced.

Having the support of companies like EPS and IP Infusion was instrumental in steering our focus towards what truly needed to be addressed. Their guidance, coupled with the application of the KISS method, contributed to what I believe was a successful deployment. My hope is that participants of today's webinar can apply these insights and achieve similar positive outcomes.