The Critical Lowdown Podcast Episode 30
Unlocking Network Potential: The Open Networking Revolution
In the evolving landscape of telecommunications, the shift towards Open Networking represents a significant pivot from traditional, proprietary systems. This transition is not merely a change in technology but a transformation in how networks are conceptualized, designed, and managed. Open Networking, with its emphasis on flexibility, cost efficiency, and innovation, is reshaping the infrastructure of the internet, making it more accessible and adaptable to the needs of today's digital world.
In this podcast, we delve into the practical aspects of transitioning to disaggregated networks, a journey undertaken by Uniti, EPS Global, and IP Infusion. Uniti, a leading fiber infrastructure provider, alongside EPS Global, a distributor specializing in Open Networking solutions, and IP Infusion, a pioneer in network operating systems, come together to share their insights. The discussion covers the motivations behind moving to Open Networking, the challenges encountered, and the tangible benefits realized, including significant reductions in both capital and operational expenditures.
Subscribe to The Critical Lowdown from EPS Global wherever you get your podcasts:
Chief Marketing and Product Officer, IP Infusion
Director Sales, North America, EPS Global
Sr. Director - Core/Backbone Engineering, Uniti Fiber
If you have any questions about or need advice or tech support for your upcoming project, don’t hesitate to get in touch.
Transcript of Podcast
Kelly: Hello everyone, and welcome to today's webinar featuring a practical guide to disaggregated networks where Uniti, EPS Global, and IP Infusion present keys to success. I'm Kelly LeBlanc, Chief Marketing and Product Officer for IP Infusion based at our headquarters here in Santa Clara, California. Today I'm joined by two very special guests, Graham Wooden at Uniti Fiber and Alan Fagan at EPS Global. Graham, can you please introduce yourself and your company and briefly explain what you do and why you're qualified to speak on Open Networking?
Graham: Thank you Kelly. I'm Graham Wooden. I'm a Senior Director of Engineering here at Uniti Fiber. We are the operating arm of Uniti Group, which is a REIT, a Real Estate Investment Trust. We are publicly traded and we are the second-largest independent fiber provider in the United States. Our REIT consists of fiber infrastructure for cell towers, backhaul, wholesale, and we have over 8 million strand miles of fiber all throughout the U.S. We're here today to explain our disaggregated network solution. You know, it's been a great road. We've learned a lot. There have been a lot of pioneers in front of us that we've been able to examine, and I'd love to talk to you here shortly about all that.
Alan: My name is Alan Fagan, and I manage the North American sales team for EPS Global. We're a value-added distributor specializing in Open Networking solutions. Our partnerships span across software providers like IP Infusion and hardware manufacturers such as Edgecore, UfiSpace, and Coherent (formerly Finisar). We aim to build comprehensive solutions for clients, including Uniti. With nearly a decade of experience in Open Networking, we possess both the technical and logistical capabilities necessary to guide customers through understanding and deploying Open Networking solutions.
Kelly: Alan, can you please explain what is Open Networking and how it differs from traditional networking?
Alan: Sure. Open Networking emerged in the early 2000s as a response to the dominance of proprietary networking vendors like Cisco and Juniper, who controlled the underlying protocols, software, and hardware for networks. This control made it difficult for organizations to innovate or switch to new technologies. Open Networking, in contrast, emphasizes open standards, open source software, and commodity hardware, making the underlying protocols, software, and hardware publicly available for anyone to develop and use. This approach offers several benefits, including increased flexibility, reduced costs, and supply chain flexibility.
Open Networking allows organizations more flexibility to choose the best hardware and software combination for their needs, without being locked into a monolithic solution. It reduces costs by using commodity hardware, offering the freedom to choose hardware suppliers, which can lead to cost leverage. This approach also offers supply chain flexibility, allowing for the use of different hardware if one option is not available, a benefit that became particularly clear during the recent supply chain challenges experienced during COVID.
Another advantage of Open Networking is faster innovation, as it allows anyone to contribute to the development of new technologies. There are also reductions in power costs, which Graham will discuss in more detail later. Initially, Open Networking was adopted by hyperscalers with significant resources, like Amazon, Facebook, and Microsoft. However, with advancements in technology and support from companies like IP Infusion, EdgeCore, and UfiSpace, Tier-2 and Tier-3 operators are now also benefiting from Open Networking.
With that, I'll hand over to Kelly and Graham, who will discuss Uniti's journey from choosing IP Infusion to implementation and the benefits they've seen from this transition.
Kelly: Thanks a lot, Alan. In today's webinar, we're diving into Uniti's adoption of IP Infusion's disaggregated networking solution, OcNOS. OcNOS is a versatile network operating system compatible with various hardware options, including those from EdgeCore and UfiSpace. It's designed to offer a unified software platform for diverse networking needs, supported by EPS Global's value-added distribution services. OcNOS caters to a range of applications from mobile backhaul and access, aggregation and metro networks, to routed optical networks and data center networks. The solution's scalability is impressive, starting at 32 gigabits per second and extending up to 14.4 terabits per second, all while maintaining compatibility with Open Networking hardware. Additionally, it supports a wide selection of optics, including 400G ZR and ZR+ optics.
Before we delve into Graham's insights on Uniti's journey towards Open Networking, let's set the stage. Uniti, a publicly traded infrastructure provider boasting 8 million strands of fiber, has embarked on this path with the support of IP Infusion and EPS Global. The transition to OcNOS, facilitated by hardware from EdgeCore and UfiSpace and serviced by EPS Global, aims to enhance and expand Uniti's network capabilities.
In today's session, Graham will discuss the tangible benefits of network disaggregation, highlighting significant reductions in capital and operational expenditures, alongside decreased power consumption—illustrated by the comparison to the power usage of a fully equipped Cisco ASR 9006 chassis.
Without further delay, let's welcome Graham to share his experience with Open Networking at Uniti. Graham, could you enlighten us on why Uniti opted for Open Networking?
Graham: Thanks Kelly, and thanks Alan, for giving the brief overview of disaggregated networking. As Alan mentioned, the COVID pandemic acted as a catalyst for us to move towards a new networking topology and design. We observed a significant increase in both wholesale and enterprise sales, which made it clear that our IP backbone was on the verge of falling short of the necessary 100G and 400G interfaces. To stay ahead, especially with supply chain delays from new manufacturers and the cost of gray market items quadrupling overnight, we realized the need to explore new technologies.
Our existing infrastructure, particularly our monolithic chassis 9006s, were already equipped with several 100G line cards. However, upgrading to the next generation line card, a 12 port, would cost us $60,000 to $80,000 on the gray market. Although this could temporarily keep us ahead of any surge, we understood it wasn't a sustainable long-term solution. With 40 IP backbone nodes carrying traffic throughout the Southeast, the economics of updating three or four line cards per chassis, considering both CapEx and OpEx, including user licensing, were simply not feasible.
This is where the benefits of disaggregated networking became apparent. By separating out our infrastructure, we began exploring routers from UfiSpace, which offered a 5x increase in interfaces at one-third the cost of a gray market line card. We also looked into how AT&T was implementing disaggregated networking by acquiring Avada, a move that, while not groundbreaking since Avada has been used on Linux servers for years, signaled a significant shift in the industry towards this type of infrastructure.
Our research led us to IP Infusion, which stood out due to its role as an extensive development partner for Broadcom on the StrataDNX ASIC chip. We were impressed by the supportability across UfiSpace and Edgecore routers and their internal development and external growth. EPS played a crucial role in connecting us with the right contacts at UfiSpace and Edgecore for hardware inquiries and developing our relationship with IP Infusion for software support.
Our IP backbone operates as a completely separate network from our Ethernet, necessitating specific considerations. We took extra time to thoroughly explore our options and use cases, ultimately forming what we believe to be a strong relationship with IP Infusion, UfiSpace, and Edgecore, with EPS's assistance.
Kelly: Well, you've conducted extensive research and put considerable thought into this deployment, collaborating closely with both of our companies. Could you share your experience of working with IP Infusion and EPS Global?
Graham: Absolutely. I'll start with my experience with EPS Global. From the moment we were introduced, it was clear they were going to be a solid partner. The sense of assurance came not just from our account manager but also from the service and sales engineering team. They demonstrated a readiness to answer any questions we had, and if they didn't have the answers immediately, they were quick to find them. I always felt well taken care of, without any hesitation or unresolved issues. EPS showed a genuine interest in our business.
Working with EPS, especially given our larger customer status and footprint, has been seamless. We've collaborated on having hardware readily available in staging warehouses throughout the United States for any necessary RMAs. Fortunately, we haven't experienced any hardware failures to date, but it's reassuring to know that EPS has hardware staged in RMA status ready to ship if needed.
Kelly: Support is a critical aspect, particularly with Open Networking. It's been a leading topic as we aim to lower the barrier of entry and ensure a smooth, comfortable transition. Before we delve into the benefits in terms of CapEx, OpEx, and power savings, I believe it's important to first address the topic of support.
Graham: Yeah. So, and that's a great question Kelly. That was initially our, not hold back, but, you know, item that we wanted to address quickly was, you know, we're used to traditional support methods from larger organizations, you know, how we were going to receive the support from IP Infusion. So, you know, luckily here at Uniti, we have a pretty large lab. It's about 40 racks, plus the 40 racks. And each of those racks encompasses equipment and hardware and software that we run out in the field. So anything that we run out in the field, we're able to stage it properly in our lab first. We were able to, with help of IP Infusion, we were able to stage a lot of our gear in the lab and run through lots of test cases of how we were deploying out individual markets, how we're deploying out to the backbone and out to our edges, and then recreating various test scenarios. And while we did uncover a few little bugs, IP Infusion was quote unquote Johnny on the spot of getting them fixed, getting issues, you know, quickly handled. And it was reassuring that for a company our size, that IP Infusion was right there ready to support.
Alan: Yeah. And I think it's a really good point that you made there, Kelly, about support being a big issue. You know, for us as EPS Global out evangelizing Open Networking, it is one of the biggest concerns that customers have. Customers are not familiar with Open Networking. Will they receive the levels of support that they're used to? In our experience with customers where we've deployed, most of the time they actually find the support better. They have better access to the technical resources within like an IP Infusion than they would with a traditional vendor. And typically they're pleasantly surprised.
Kelly: Yeah. Thank you both. Now that we've addressed that, let us call it the barrier to entry. I think it's a good time to deep dive into the topics at hand and about these benefits. These have been some pretty bold claims. So Graham, let's deep dive into the CapEx and OpEx savings topic. So how were you able to get these sort of metrics?
Graham: Yes, so when you examine our network and topology, it's evident that we operate hardware that is somewhat more expensive to run and consumes more power compared to smaller operators. This isn't to suggest any differences in behavior, but rather it's a reflection of our structural setup. As we began to assess how to meet our demand and ensure we had the necessary capacity for our wholesale and enterprise operations, power consumption wasn't initially a primary concern. We operate in a multitude of central offices where power regulation is somewhat lax, and we had resigned ourselves to the reality that some aspects would be regulated fairly while others, like power, would not. Our initial focus was on driving CapEx savings, contemplating the expenditure of $60,000 on an eight-port, 100G line card to push the limits of our monolithic chassis. We encountered situations where, after installing an eight-port line card, we found ourselves needing to upgrade to a 12-port card within a month or two, only to discover there were no available slots in the chassis to accommodate this change. We recognized we were approaching a capacity crunch.
Although we could have opted for a larger slot chassis, this would likely have increased our power demands. Re-evaluating our options led us to consider alternatives like the UfiSpace S9600-56DX, which offers 48x 100G ports and 8x 400G ports. The port density of this compared to our traditionally deployed eight-port line cards represented a significant saving, reducing the cost from $60,000 for a single line card to $20,000 for the router, while quintupling the number of ports. The operational savings, including software and licensing, were also considerably lower than with our traditional vendors. This realization allowed us to allocate our budget more effectively, enabling growth in other areas of the network.
The reduction in power consumption emerged as an unexpected benefit. Initially, we had accepted power costs as a given, but we soon discovered that our monolithic chassis were consuming up to 56-60 amps of DC power, which could be reduced to about 6-7 amps. This represented a significant cost saving, far exceeding our initial expectations. Moreover, aligning with our ESG and environmental goals became an additional advantage. By reducing our power consumption, we not only achieved financial savings but also contributed to more sustainable and environmentally friendly operations.
It's important to note that in these locations, we deploy two chassis for redundancy, with one handling the northeast and the other the south-southwest. Previously, we were drawing close to 120 amps of DC power to support our IP backbone. With the new setup, we've managed to reduce this to approximately 14 amps for both chassis combined. This drastic reduction in power consumption has led to substantial savings on our monthly bills to central office operators or data centers and has allowed us to be better stewards of our community.
Kelly: That's excellent. I think I would say being a publicly traded company, you have many social responsibilities and this power consumption savings would absolutely address those requirements.
Graham: Absolutely, Kelly. It does. And you know, there's other parts of Uniti as well that our ESG is very important. It's just that on the IP backbone side, we never, never real realized how much important it will be just because we just succumbed to the feeling that we would have to run these big routers forever. So it was a blessing in disguise that we were able to move to smaller chassis with more ports and have the same software capabilities that we're running before. So it was a win across the board.
Kelly: Wow, this is an incredible story. So with all stories, you know, every story has its benefits and its lessons learned. And so, you know, this has been a journey for you. And so I would like to talk a little bit about your lessons learned a lot along this journey and anything, anything you could do to share with the community about these lessons.
Graham: Graham: Sure. Yeah. Well, I will quote the immortal words of Olivia Newton-John and say, "Let's get physical." The biggest lesson we learned wasn't about software, differences in the CLI, or any perceived notions of protocol changes. All that was straightforward. Switching from one vendor to OcNOS was much easier than I anticipated. What I didn't really anticipate were the physical changes. Our line cards were running optics with SC connectors, and these new chassis are running QSFPs with LC connectors. During our first deployment, we had the physical dimensions of the box spec'd out in a rack, and power cables run, but it quickly became evident that our SC connectors, the jumpers we previously used for the old chassis, couldn't be used on the new chassis due to the change from SC to LC connectors. We had to quickly redo our cabling to have LC ends.
Some might say, "Well, shoot, Graham, you should have caught that before." We were so hyper-focused on the physical size changes of the chassis, going from a 14RU box to a 2RU box, and rerunning the DC power because the lugs were a bit different. We just thought we'd move from this 100G optic on the old chassis to the new optic, then realized we were going from SC connectors to LC. The lesson learned is really about the physical layer. It's important to review all your physical plant rack space before you deploy. That way, you're not caught off guard at the 11th hour when you're trying to do maintenance and then realize you have to spend another 30-40 minutes running cables, which could have been done beforehand.
Alan: Yeah, I'd like to talk a little bit as well, Kelly, about some of the lessons that we learned as EPS from the Uniti experience. Graham has talked quite a bit about the power savings, the really dramatic power savings. We were aware that there were power savings, but seeing them so significant was a lesson learned for us. But I think it's a common thread that we see with the customers who have implemented Open Networking. They may implement it because of lead time or cost, but once they have it implemented, they find all these other benefits that they were not aware of. And there's no going back to the old way once they've done it. But I think, yeah, the dramatic power piece was a real lesson for us.
Kelly: That's good feedback as well, Alan. Thank you. And thank you for deep diving on these lessons learned. I think there are a couple of other areas where the audience could benefit because there's always next steps in a journey. So, Graham, could you share with the community what the next steps in your journey could be?
Graham: Certainly. The next steps in our journey, as I've touched on earlier, revolve around the strategic deployment of our backbone locations. We've opted for deploying two nodes primarily to diversify our failure domain. This approach is akin to not putting all our eggs in one basket. Our decision to deploy two nodes is driven by the need to mitigate the risk of a single point of failure affecting our entire network. Our IP backbone is entirely optical, eschewing traditional switch networks between nodes for a purely optical connection. This design choice necessitates a careful consideration of failure scenarios at the physical layer, drawing inspiration from the insights of living new John on the importance of planning for failure.
From the outset, we committed to standardizing on two backbone nodes to ensure redundancy. This setup ensures that an outage at one backbone node doesn't incapacitate the entire market or disrupt traffic flow, thanks to the presence of a secondary node. Our journey has seen us grow comfortable with IP infusion and Qualcomm's two-box solution. However, with the advent of Jericho2, we're exploring further diversification away from a single vendor. We've already incorporated Edgecore into our setup, but we're now looking to expand our use of Jericho2 ASICs in our backbone nodes. This strategy aims to safeguard against potential issues with either Qualcomm or Jericho2 components, ensuring our backbone remains operational.
Beyond the backbone routers, our focus extends to other critical components of our IP infrastructure. Our distribution routers, which interface directly with customer networks and handle BGP routing with customers, cache servers, and external IX connections, are under review. Traditionally, these have been 9904 or 9906 chassis, but we're considering transitioning to UfiSpace or Edgecore routers. Additionally, we're evaluating changes to our route reflectors and special service edge routers, which are integral to our service delivery.
This evolution, initially centered on backbone nodes, now spans the entire IP service delivery framework for Uniti, encompassing over a hundred routers. Our goal is to implement these changes across the board, enhancing the resilience and efficiency of our network infrastructure.
Kelly: That's fantastic. It's what we refer to as total network disaggregation here at IP Infusion. Essentially, you're breaking down everything - the hardware, the chipset, the optics. It's a fascinating story, and hearing about these next steps is incredibly interesting. Alan, I understand you've played a significant role in this journey towards the next steps. I'd love to hear your perspective on this.
Alan: Yeah, I think for us, the next step really begins today with this event and getting the word out there. I'm very proud of the impact that we've had on Uniti and how we've helped them. I'm also very proud of the relationship that we have with IP Infusion and with the hardware manufacturers that we work with. Additionally, I'm very proud of my team, including the sales team involved with Graham and Uniti, Bob Crane, and George Say, who is our technical director. I believe there's no one out there with more knowledge of this marketplace than George.
I really feel we have the tools in place—we have the software, we have the hardware. The biggest job we have now is evangelizing, getting the word out there, and helping people understand the benefits they can reap from implementing this. Graham is a tremendous example of that. We're all aware of the need to get broadband connectivity out there everywhere it's required. We believe that what we have is a way for that to be done in an economical and very powerful way that will work. We just have to keep getting the word out there.
Kelly: Excellent. So, you know, one of the things I think could be really helpful, Graham, is if you could summarize the one key criteria that led you to select IP-Infusion, EPS Global, and Open Networking.
Graham: One aspect that particularly stood out and resonated with me regarding IP-Infusion was its history of previous licensing agreements with other vendors. This was through its license-only offering system, which I had encountered while using products from vendors such as F5 and Fujitsu. For years, I had been utilizing these vendors' offerings, unaware that the systems they incorporated were actually licensed components from IP-Infusion. This realization came to light during my recent exploration into IP-Infusion's offerings.
IP-Infusion has demonstrated a strong presence since the early 2000s, possibly even earlier, by licensing its components to other vendors. This fact served as a reminder of IP-Infusion's long-term commitment to the industry. It highlighted that IP-Infusion wasn't a newcomer that emerged in the late 2000s or 2010s. While that might be the case for its OcNOS side, the essential components that constitute OcNOS have been actively supported and developed by IP-Infusion since the year 2000.
These components have been operational in various vendors' offerings that we have been utilizing in different capacities for many years. This knowledge assured me that we were adopting software that would be integral to our lifecycle for many years to come.
Kelly: Excellent. I thought maybe you would bring it back to Olivia Newton-John, but I think that's been a theme throughout this entire interview.
Okay. Let's go back to Alan.
Alan, can you share some advice on how to begin a transition to Open Networking?
Alan: Yeah, I think the most important piece of advice that I would give is to find the right partner to work with, right? And I will say that I believe that EPS Global is the right partner. I think Graham would back me up on that one as well. And the reason that I say that is that we have the technical knowledge, we have technical people on our team, and we have the relationships. We have the software relationships, we have the hardware relationships, and the ability to pull all those together as well.
Moreover, where we don't have the answers, we have the relationships with IP Infusion. We have the relationships with Edgecore, with UfiSpace, and we allow our customers to leverage those relationships as well. We do the introductions directly. So, Graham and his peers can talk directly to the hardware and the software people and figure out solutions. We're all about figuring out the solutions.
If I think of the customers, the potential customers out there, one of the fears that they have is having to try and navigate this landscape on their own. What's the right software? What's the right hardware that goes with that? We can help in that. And I think that's a really key thing. Find the right partner, work with them, and we can make this easy.
Kelly: Excellent Alan. Graham, how about from your perspective, some advice on how to begin a transition to Open Networking for those who are maybe still on the fence or just getting started?
Graham: So, reflecting on my IT career, which spans over 26 years now, I recall starting in a very small dial-up ISP back in 1997. Fast forward to 2023, I find myself working for a significantly larger ISP. One pivotal lesson I've absorbed early in my career, which continues to guide me in every new endeavor, is the application of what's known as the KISS (Keep It Stupid Simple) method, an acronym for Keep It Stupid Simple. The essence of this approach is to avoid overthinking and overanalyzing, which often leads to drowning in one's own thoughts and unnecessarily complicating matters. I consistently strive to adopt a simplistic perspective. While it's acknowledged that the end result might be complex, the initial review should aim for simplicity. The goal is to keep it stupid simple, as advocated by the KISS principle.
For those contemplating moving away from monolithic chassis types or transitioning from a current vendor to alternatives like IP Infusion or a disaggregated offering system, my advice is to focus on the very basics. Consider what protocols you're using and the physical outcomes you aim to achieve, such as size reduction or reducing the number of interfaces. Perhaps there's a need to transition from a multitude of 1G or 10G interfaces to more advanced 100G or 400G options. At this stage, I would recommend not dwelling too much on the protocol specifics or the appearance of the CLI (Command Line Interface). Initially, we anticipated challenges in these areas, only to discover that the CLI was quite user-friendly and similar to what we were accustomed to. Furthermore, all the protocols we traditionally relied on were available. In essence, we realized that our initial concerns were misplaced.
Having the support of companies like EPS and IP Infusion was instrumental in steering our focus towards what truly needed to be addressed. Their guidance, coupled with the application of the KISS method, contributed to what I believe was a successful deployment. My hope is that participants of today's webinar can apply these insights and achieve similar positive outcomes.
Kelly: That's great advice. Thank you, Graham.
Glossary of Terms