The Critical Lowdown Podcast Episode 18
Powering Up Broadband: How Electrical Co-Ops Are Transforming Into ISPs Through Disaggregated Networking
Vinnie Maniola, Sales Director, IP Infusion; Kevin Myers, Senior Network Architect, IP ArchiTechs
in Conversation With Alan Fagan, Sales Director North America, EPS Global
The need for high speed broadband connections in rural areas is firmly in the spotlight. The shift to online services and working from home has accelerated an already unprecedented demand. Governments are responding in kind - with $42.5bn being pledged in the US alone through its Broadband Equity, Access, and Deployment Program. The need for rapid deployments means electrical coops - who have existing infrastructure in these areas - are turning their hand to broadband deployment. Why?
- The electricity customer base is not growing meaning the revenue model has plateaued, but Co-Ops are looking to return value to their members still. Becoming an ISP offers a massive new revenue share, it provides new revenue streams for Internet, Private Circuits, and Co-Location.
- Major vendors like Juniper have dominated the market for Electric Co-Ops but inflated licencing costs and and a lack of knowing where the costs are going to go in the future has prompted a need for supply chain diversification.
- For Electrical Co-Ops there are complex problems to solve in every vertical of networking: Enterprise, OT for Electric Grid, and Service Provider for ISP.
- IPv6 strategies are few and far between - forcing CAPEX spend on large blocks of IPv4.
All of which makes Electrical Co-Ops the perfect candidate for disaggregated or open networking. It can help to reduce CAPEX and Operational expenses immediately and for the future, making projections more attractive, and the technology is there to enable multitenancy to use the network for the 3 different models. Open Networking hardware is more readily available so a new network can be deployed quicker, and the support available from open networking vendors is responsive and dependable.
Today we look at how can Electrical Co-Ops make the transition to becoming Service Providers with ease through Open Networking. Let's jump right in!
Subscribe to The Critical Lowdown from EPS Global wherever you get your podcasts:
If you have any questions about or need advice or tech support for your upcoming project, don’t hesitate to get in touch. Or check out our Service Provider Solutions here.
[AF] My name is Alan Fagan, and I manage North American Sales for EPS Global. We're a value-added distributor and we work with the best software and hardware manufacturers to provide Open Networking solutions. Today, we're going to talk about Electric Co-Ops becoming Internet Service Providers and bringing broadband to their customers. Joining me for this discussion are Vinnie Maniola, Sales Director for IP Infusion; and Kevin Myers, Senior Network Architect with IP ArchiTechs.
To give a bit of background, electric cooperatives have been around for a long time. They were set up in the US in the early 20th century to provide electricity to rural areas that were not being served by private utilities. There are now over 900 of these cooperatives in the US, providing electricity to more than 42 million people in 47 states. 200 of these Co-Ops currently offer Internet service, and that number is growing rapidly as more cooperatives recognize the benefits of providing broadband access to their members. They're also taking advantage of the $42.5 billion available through the BEAD Program (Broadband Equity Access and Deployment). It's expected that up to 150 more will offer Internet service over the next 3 years.
Electric Co-Ops are well positioned to provide Internet service as they already have the infrastructure in place and a strong reputation for customer service. Providing Internet service can be a profitable business for electrical cooperatives and can help improve their members’ lives by giving them access to the same opportunities as those in more urban areas.
There are several initial hurdles that need to be crossed by Co-Ops considering becoming ISPs. NRECA recently had a podcast addressing some of the concerns and talked about how crucial feasibility studies are and how they must include legal and regulatory issues.
In this podcast, we're going to assume that the initial hurdles have already been crossed, and we'll focus on the network infrastructure. We will talk about how disaggregated networking can be a perfect fit for Co-Ops and bring substantial savings in CapEx, OpEx, and also dramatically reduced deployment times due to much shorter lead times than traditional vendors.
So, Kevin and Vinnie, both of you have experience of either being part of an electrical co-op or helping a co-op become an Internet Service Provider. Can you tell us a little bit about yourselves and then tell us why you think Co-Ops are perfectly positioned to bring broadband to the communities? Kevin, we'll start with you.
[KM] Thanks Alan. My name is Kevin Myers, I'm a Senior Network Architect with IP ArchiTechs, and I've spent the last couple of decades building networks of all types, a lot of them in the Service Provider space. I've also done some work in various fields, such as Fortune 500 publicly traded companies and data centers, so I've had experience with a little bit of everything. I am currently leading the engineering team at IP ArchiTechs, primarily focusing on consulting and constructing various types of networks, including those for electric cooperatives.
I believe Electric Co-Ops are ideally positioned to capitalize on this opportunity because they have experience in infrastructure, heavy construction, and the necessary components to become an ISP, as well as managing a utility-type infrastructure with customers and subscribers.
[AF] What they are doing is quite similar to their original purpose, isn't it? They were established to provide electricity to underserved areas, and now they can essentially do the same with broadband, delivering it to regions that are not adequately served by existing providers.
[KM] Definitely. A century ago, the focus was on providing financial and structural support to develop electricity. Today, as the Internet has become an essential utility in our daily lives, access in rural areas is crucial. I recall a network we constructed in the US for an electric cooperative. After completing the project, they created a promotional video featuring interviews with local residents. One individual, who had grown up in the area, was contemplating relocating to the nearest city due to the lack of broadband. However, with the arrival of broadband, they could remain in their hometown, work remotely for a major corporation, and stay close to their family. So, witnessing the impact on people's lives like this shows that the opportunity here goes far beyond merely establishing Internet connectivity; it is truly transformative.
[AF] Thanks Kevin.
Vinnie, I believe that you are actually part of an electrical co-op. Can you tell us what that's like?
[VM] Certainly, thank you for inviting me, I am Vinnie Maniola, and I oversee the Western region sales for IP Infusion. I have had the opportunity to work with all three types of electric utilities: municipal, investor-owned, and cooperative utilities. My projects have involved various tasks such as connecting transport and access between sub-stations, and implementing Fiber-to-the-Home (FTTH) deployments in cooperatives and other regional areas. My spouse and I are constructing a house in Portal, Arizona, as members of a cooperative, and our electricity connection process begins well before the actual construction. It's an exciting new journey for us.
Cooperatives are naturally suited to provide broadband due to the advancements in metering technology, evolving from manual meter reading to Automated Meter Reading (AMR) and Advanced Metering Infrastructure (AMI). This shift has led to a transition from one-way communication, where customers receive kilowatt-hours and pay bills, to two-way communication that allows for household interactions and electric bill management. As the Internet of Things becomes more integrated, cooperatives are exploring ways to manage this traffic. By upgrading their communication systems, they can leverage their investments and offer additional services, making them well-positioned to provide broadband.
[AF] Thanks, Vinnie. I'm aware that there are several diverse types of networks that can be typically deployed in a Co-Op. Kevin, could you discuss these various network types?
[KM] Absolutely. Electric cooperatives are interesting because they generally run three different types of networks. Sometimes these are three completely physically separate networks, depending on regulatory constraints, and sometimes they are one physical network with logical separation of the three different networks. For the most part, what you'll see is:
- An enterprise-style network that exists to serve the employees of the cooperative. This includes offices, service locations, and anywhere that requires connection to the corporate network and access to corporate applications and resources. It is built very much like an enterprise network that you would find in any other enterprise in the world.
- An operational technology (OT) network that controls the electrical grid. This IP network is designed to control and manage things like life and safety, grid control, and access to metering. We see SCADA a lot, like in manufacturing and industrial environments. These networks have very specific security requirements and design and build requirements.
- ISP networks for many cooperatives, which are Service Provider carrier networks. This is not new for some cooperatives, but very new for others. Learning and understanding what this network looks like and what the design build looks like is a big challenge and a process to go through.
One of the unique design challenges is having three very different networks that need to work together and, most importantly, figuring out a way to justify the budget for them.
[AF] So, what do you think are the most critical steps that Co-Ops need to take to make sure they're prepared for the future of broadband deployment?
[KM] I believe the crucial aspect is a significant mental shift that needs to occur. I have observed that network designs in the cooperative domain are still evolving. There is a notion that introducing Internet mainly serves to provide residential Internet access to electricity consumers and small businesses. However, when constructing a Service Provider with access to infrastructure like fiber, it is beneficial to envision it as an asset and a Service Provider capable of delivering various services, including large businesses, community police and fire, or cell tower companies. Therefore, the primary focus should be on seeing the network as a comprehensive Service Provider with the capacity to offer SLA services in addition to providing Internet access to residential areas. Adopting this design approach allows for making informed decisions that enable more than just Internet connectivity for individuals, resulting in a broader community impact.
[AF] Do you think that offering Internet service can contribute to enhancing the financial health of Co-Ops?
[KM] When considering this as a designer, it's important to acknowledge the turbulence experienced in the last two or three years, such as the pandemic and global supply shortages in IT. This has made it difficult to obtain equipment, and major vendors like Juniper, which is heavily used in the Electric Co-op space, have changed their value proposition. Licensing costs have increased significantly, and what used to be included in a license has become more expensive and à la carte.
As cooperatives try to add revenue streams, these massive CapEx purchases for network infrastructure can erode their profits, making it hard to maintain and deliver value to members. With infrastructure and CapEx costs increasing two or three times, it becomes difficult to get approval from the electric cooperative board.
This situation leads to a conversation around disaggregated networking, which has become a more viable option for electric cooperatives and other ISPs. Using major vendors has become untenable due to the lack of ROI. Instead, we are increasingly using disaggregated networking with companies like IP Infusion, Edgecore, and UfiSpace, which offer better value, the same SLA, and faster equipment delivery.
[AF] Thanks, Kevin. Vinnie, maybe you could talk a little bit about IP Infusion and how you guys address cost issues and support providers with equipment at a really good cost compared to what they've traditionally been paying?
[VM] I'd be happy to. Initially, IP Infusion provided our network operating system, OcNOS, to OEM vendors. Recently, there has been a significant increase in disaggregated Open Networking, and our focus is on collaborating with high-performance switch manufacturers such as Edgecore. In terms of cost, we can compare our performance with traditional vendors, but it's our licensing model that sets us apart. We offer a perpetual license, which means you purchase it once for a single copy, without turning it into multiple recurring licenses. As a result, when evaluating ROI, our total cost of ownership over five years is highly competitive compared to the traditional single vendor approach.
[AF] It's important to note that IP Infusion has been around for a long time, and the hardware it's been deployed on from Edgecore and UfiSpace is solid and proven. Now, regarding the financial health of Co-Ops, the electrical customer base is not growing, so the revenue model is plateauing.
Kevin, could you talk about how becoming an ISP can help providers with that?
[KM] Absolutely. Cooperatives are just now learning how to monetize an IP network as an asset. It's crucial to be deliberate in designing the network to get the flexibility you want. That's why we spend so much time selecting the right components for the value proposition because you don't always know what you're going to be asked to do. When you bring bandwidth and the ability to deliver private circuits to an area, you don't know what opportunities will arise.
For example, we had a company approach a regional ISP we were working with that needed tens of thousands of grain silos connected with private circuits. By building a flexible network that takes advantage of the better ROI that disaggregation brings, they were able to deliver those circuits at a price point that brought significant revenue into the ISP and shortened the ROI period.
It's essential to be deliberate in designing the network so that you can leverage every opportunity to bring in revenue. For many cooperatives, increases in licensing costs have made it a tough uphill climb. Being deliberate in the design and using disaggregated networking can help cooperatives add value and better monetize their IP networks.
[AF] Great, thanks Kevin. Earlier, you mentioned the supply chain issues we've seen over the last couple of years and the increase in lead times. In a previous broadcast in this series, we discussed how MetaLINK was able to deploy their network within a matter of months using IP Infusion, Edgecore, and UfiSpace, despite being quoted an 18-month to 2-year lead time from legacy vendors.
We are witnessing the advantages of reduced lead times from UfiSpace, Edgecore, and EPS Global, as we have inventory ready for rapid deployment. Now, let's discuss the significance of supply chain diversification for Co-Ops. As providers of disaggregated networking, how can we assist them in this endeavor?
[KM] It's interesting because network operators might not have thought much about supply chain diversification 3 or 4 years ago. They assumed the supply chain was fine and didn't need to worry about it. However, that's not the case now. One of the strong positives for disaggregated networking is the ability to choose from multiple ODMs, such as UfiSpace, Edgecore, or others. This gives a significant advantage in getting equipment in a timely manner, especially when major vendors like Juniper and Cisco have long lead times due to custom ASICs and Just-in-Time inventory practices.
The advantage of disaggregation in the supply chain is not only the multiple ODMs to work with, but also the simpler network components. We design networks differently using smaller boxes, which are easier to manufacture and stock. This allows companies like EPS Global to have inventory stocked in warehouses around the world and available for quick deployment. For example, a common customer of EPS and IP ArchiTechs was able to receive their equipment ahead of schedule, which is not something we see with other major vendors.
In summary, supply chain diversification is important for Co-Ops, and disaggregated networking offers multiple ODM options, simpler network components, and shorter lead times, making it an attractive solution for network operators.
[AF] Yes, I think it works for us in terms of our inventory stocking positions because we can use one switch for multiple different things. It's really helped us, particularly in the last couple of years with this supply chain crunch.
Vinnie, I’m going to throw a couple of questions your way now. If a customer has existing infrastructure, how does the IPI solution fit in with that?
[VM] Thank you for asking that. IPI has been deployed since 1999. We're deployed in some of the small to mid-size WISPs all the way up to the fourth largest carrier in the world, NTT. We have a staff of hundreds of engineers who have evolved the software over time to where in a brownfield deployment, it will work flawlessly with all of the major players out in the networking world.
We have a modern MPLS stack that Kevin can talk to even more than I can, but we do segment routing, we do all of the basic features, and we do them well. The longevity of being out there for 24 years speaks to when you put this out in a field, it will work. I would add that we're part of a mature ecosystem with these hardware partners where we rigorously test our software with each hardware platform that we're going to support. And just as important, we will take the support call. We will offer hardware bundles with these platforms, and we will support the hardware and the software. We'll take that frontline call.
[AF] Great. So, Kevin, maybe talk a little bit about your experience in the field of putting IPI solutions in and having them interoperate with what's already in place.
[KM] We actually do. I think you guys know this because you've had Vince Schuele, a Network Architect with IP ArchiTechs, on previous podcasts. He has talked about the interop testing that we do. We have a lab where we perform these tests because it's crucial for a Service Provider to ensure compatibility with other vendors. Unlike enterprises that typically stick to one vendor, Service Providers must work with multiple vendors to fulfill various roles. Ensuring smooth interoperability is essential, and our experience has been positive with IP Infusion and ODMs participating in numerous open standards interop tests, such as the annual event in Berlin.
In our practical experience deploying this technology in the US, Africa, and Latin America, we have found that it works well with other vendors. And even when you do have problems, as every network vendor is going to have quirks and bugs, that's just the nature of the beast, the support has been fantastic with IP Infusion. We usually get great support, and it's very timely. In fact, I would rather go talk to IP Infusion's TAC and let them work with the ODMs if needed. I've called into Cisco and Juniper TAC many times, and lately, the TAC experiences have not been as great. It's refreshing to work with IP Infusion, and when there is a bug, they are very timely about getting it fixed and patching code. When you're running a network, that's very important to not be left out to dry when dealing with those kinds of challenges in integration.
[AF] I definitely agree. You touched on an important point there, Kevin, about the global support aspect. We're talking specifically about electrical Co-Ops in the US, but everything we're discussing can apply equally to people who want to become Service Providers globally. Now, I want to talk specifically about reliability and support. Most electrical Co-Ops have a strong reputation for customer service, and support is crucial to them. Vinnie, could you discuss how support from IPI compares to support from companies like Juniper and Cisco?
[VM] Sure. We offer support contracts with options like 8/5 or 24/7 phone support, and we troubleshoot the bundled equipment, both hardware and software. From a cost standpoint, our maintenance and support are significantly less expensive than others, but we provide the same level of service because we understand that uptime is vital, especially for utilities.
Reliability is extremely important, and the product must be supported by the company that provides it. I think that's a differentiator between IPI and some other newer software vendors in open networking.
[AF] You mentioned earlier that you offer a bundled product, which includes a switch with software loaded on it.
[VM] Yes, we work with customers to determine their requirements. The switches we've been discussing, from Edgecore and UfiSpace, have capacities ranging from 32Gb to 14TB. Customers provide us with their application needs, and we match them with the appropriate ODM partners. Based on cost and lead time, we deploy across multiple platforms. We provide the bundle, and once it's sold, we support it. Our lead times typically range from four to 12 weeks, which is generally faster than legacy vendors.
[AF] So, it sounds like you're providing all the benefits of disaggregated networking while maintaining the convenience of a single vendor, where everything is included and ready to go.
[VM]Yes, and utilities appreciate having one point of contact for support, without any finger-pointing.
[AF] Kevin, anything to add?
[KM] I'd like to emphasize that the relationship between disaggregated vendors and operating systems like IP Infusion, Edgecore, and UfiSpace is collaborative. People who may not be familiar with the current state of disaggregated networking might worry about dealing with multiple parties, but that's not the case. It's important to understand that the industry has evolved, and the collaboration between parties in disaggregated networking is strong.
[AF] That's a really great point, Kevin, and it's something we also covered on the MetaLINK podcast. The biggest misconception about Open Networking is that people think they have to figure everything out themselves - the software, the hardware, and taking responsibility for everything. But there's lots of help available from companies like IP ArchiTechs, IP Infusion, and ourselves. We're here to help people deploy Open Networking and achieve all the advantages we've been discussing.
Before we wrap up, I want to make sure we haven't missed anything. Is there anything else either of you would like to add?
[KM] One more driver for disaggregated networking is IPv6, which electric cooperatives are just now realizing. Getting the right equipment to support IPv6 successfully isn't always easy, but IP Infusion has made great strides. We have many networks deployed on IPv6, and for those unfamiliar with the difference between IPv4 and IPv6, we're going through a major shift in the fundamental protocol and foundation of the Internet, similar to the switch from analog to digital television in the U.S.
Many ISPs haven't made this shift yet, and we've found that disaggregated networking makes it more practical to adopt IPv6. In many cases, major vendors charge high prices for advanced IPv6 services and licenses. For example, one provider was going to spend half a million dollars buying IPv4 space instead of looking at an Open Networking architecture with disaggregated components and using solutions like IP Infusion and NetElastic. By investing in IPv6 deployment and technologies like Carrier Grade NAT, disaggregated networking can be more economical and avoid sinking capital into legacy IP infrastructure and licensing.
In conclusion, disaggregated networking can facilitate the adoption of new technologies like IPv6 very nicely and economically.
[AF] Thanks, Kevin. Any final thoughts, Vinnie?
[VM] Sure, I believe that for those who are just hearing about this, it's important to know that it has been in existence for quite some time. Major data center operators, such as Amazon, Facebook, and Google, have been using white box solutions for over a decade. This is not an unexplored area, as some international Service Providers have been implementing it for more than five years. Considering the Telecom Infra Project (TIP) and global initiatives, this is not a new concept by any means.
While it may not have been employed in certain segments previously, it is worth exploring. I would also like to mention GigaOm, an analyst firm that conducts research on all network operating systems, not just within the disaggregated space. As far as confidence in the technology goes, it is a proven and established solution that is here to stay.
[AF] You're right Vinnie. The hyperscale guys have been doing this for years for a very good reason - they can save a ton of money and have much more control over the network. The challenge for smaller providers is that they don't have the army of engineers that an Amazon or a Facebook has, but they have us, and we can help them achieve the same goals with software, hardware, and consultancy. We can do that. We can help them.
This was a great talk, guys. I really appreciate it. And I think the message is to any Co-Ops who are considering going down this road, we're here to help. Contact us and we can help you navigate this. And with that, I'll say thank you and goodbye.
Glossary of Terms:
- Electric Co-Ops: Electric cooperatives established to provide electricity to rural areas not served by private utilities.
- Broadband: High-speed Internet access that provides fast, reliable connections.
- BEAD: Broadband Equity Access and Deployment program, which provides funding for broadband infrastructure.
- Disaggregated Networking: A networking approach that separates hardware and software components, allowing for more flexibility and cost savings.
- CapEx: Capital expenditure, the funds used by a company to acquire or upgrade physical assets.
- OpEx: Operational expenditure, the ongoing cost of running a business.
- Network Infrastructure: The hardware and software resources of an entire network that enable network connectivity and communication.
- MPLS: Multi-Protocol Label Switching, a technique for creating virtual links between distant nodes on a network.
- IPv6: Internet Protocol version 6, the most recent version of the Internet Protocol, which provides an identification and location system for computers on networks and routes traffic across the Internet.
- Carrier Grade NAT: A technology used by Service Providers to extend the life of IPv4 addresses, by allowing multiple devices to share a single public IP address.
- Open Networking: An approach to networking that involves the use of open standards, open-source software, and disaggregated hardware, allowing for greater flexibility and customization.
- ODM: Original Design Manufacturer, a company that designs and manufactures products for other companies to rebrand and sell.
- IPI/IP Infusion: IP Infusion, a company that provides network operating systems and solutions for disaggregated networking.
- Edgecore: A company that produces open networking hardware, such as switches and routers.
- UfiSpace: A company that designs and manufactures networking equipment for disaggregated networks.
- SCADA: Supervisory Control and Data Acquisition, a system used for monitoring and controlling industrial processes and infrastructure.
- SLA: Service Level Agreement, a contract between a Service Provider and a customer that defines the level of service and performance expected.
- GigaOm: An analyst firm that conducts research on network operating systems and other technology topics.
- Telecom Infra Project: A global initiative to accelerate the development of innovative network infrastructure technologies and solutions.
- NetElastic: A company that provides software-based routing and network functions virtualization (NFV) solutions for Service Providers.
- WISP: Wireless Internet Service Provider
- OEM: Original Equipment Manufacturer, a company that produces components or products that are used in the products of another company.
- NTT: Nippon Telegraph and Telephone Corporation, the fourth largest telecommunications provider in the world.
- AMR: Automated Meter Reading, a technology used to remotely collect data from utility meters.
- AMI: Advanced Metering Infrastructure, a system that enables two-way communication between utility meters and a utility company, allowing for real-time monitoring and management of energy usage.