In this episode of The Critical Lowdown, Alan Fagan, who manages Sales for North America at EPS Global, moderates a panel featuring Jeff Hummel, Technical Services Supervisor and Nathan Gerencser, Senior Network Engineer both from MetaLINK; and Vince Schuele, Senior Network Architect from IP ArchiTechs.

If you would prefer to listen to the full podcast, you can listen here.
 

Transcript of: Beating the Supply Chain Crunch - A Case Study for Open Networking

Alan Fagan: Hi, I'm Alan Fagan and I manage North American sales for EPS Global. We're a value-added distributor with a focus on Open Networking and we partner with the best hardware and software manufacturers to build solutions for our customers. Currently we're seeing a huge interest in Open Networking from Service Providers, and today I'm delighted to be joined by one of our customers, MetaLINK; and a partner of ours, IP ArchiTechs, and we're going to talk about a project for MetaLINK that we all collaborated on and we hope to pass on valuable lessons from it. I’ll now hand over to Jeff and Nathan to tell us a little bit about MetaLINK.

Jeff Hummel: Hi, my name is Jeff Hummel and I’ve been with MetaLINK for 25 years. MetaLINK is an Internet Service Provider serving northwest Ohio, and with our full service footprint extends into southern Michigan and Northeast Indiana as well. Over the last 25 years we have grown exponentially - leading up to the project that we're going to be discussing here shortly. Nathan Gerencser is our Senior Network Engineer and he's been instrumental, along with Vince from IP ArchiTechs in getting our project off the ground here, along with working with Alan and the good folks from EPS.

So, Nate, I'll let you give a little intro.

Nathan Gerencser: I am Nathan Gerencser and I’m the Senior Network Engineer at MetaLINK, and I’ve been here for 16 years. I have done about every role in the company from climbing towers for our fixed wireless deployment, to doing tech support, and deploying equipment and designing networks. My current role is now Senior Network Engineer.

Alan Fagan: Thanks, Nate. And Vince, if you could tell us a little about yourself and a little bit about IP ArchiTechs?

Vince Schuele: Thanks Alan. My name Vince Schuele and I'm a Senior Network Architect with IP ArchiTechs. We're an independent consulting firm that specializes in Disaggregated Solutions. We don't actually sell any hardware, we focus on the business objectives and goals to help people realize their return on investment faster.

AF: Thanks Vince. Getting into the specifics around this project, what problems were you having and how were you trying to solve them?

JH: The main impetus for the project that we've been working on has been the exponential growth that we've had and the need to essentially get a larger upstream connection to the Internet for our clients and ourselves, which led to requiring a need for larger and larger bandwidth.

We partnered with a carrier in order to achieve that, and we were given a timeline which was very short. That's when we reached out to IP ArchiTechs to ask: "What are the possibilities in this short amount of time?"

NG: At this point, before we started working with IP ArchiTechs on this project, we hadn’t considered Whitebox at all to solve some of the issues we were facing. We talked to some of the big players in the game, and they put forth some designs for us and everything came with a 12-18 month wait. Either the hardware didn’t exist; or it came with these astronomical price tags to do what we needed to do.

AF: Thanks, Nate. You talked about the growth that you've seen and the need for additional bandwidth. The last couple of years has hit home how important broadband access is, with the pandemic and people working from home. Did you see a direct impact from that from that on your business?

JH: Absolutely. Within the first few months of the country going into lockdown, everybody was sent home to do work from home, school from home, etc. We saw a 400% increase in new orders within the first 2 months of lockdown. Over the past 2 years, I'd be comfortable saying that we had an additional 10-15G worth of traffic onto our ingress/egress points.

AF: Broadband has gone from being from what would have probably been considered a luxury pre-pandemic, something nice to have; to now it's something that's absolutely essential, it's like needing electricity or light.

JH: It absolutely is. In our interactions with our customers these days, they treat it just like any other public utility. I sometimes fear to say that out loud, but the writing's on the wall, eventually it will be that sort of entity. But for the time being, it's just all about getting the best and most reliable service out to our clients that are providing that demand.

AF: It sounds like the main drivers for you guys approaching IP ArchiTechs was lead time? 18 months? Which wasn't going to work, and then the cost on top of this.

Vince, when MetaLINK approached you, what did you think of the issue that they had, and how you did you think you could resolve it?

VS: They came to us and said they had a pretty tight timeline to get this capacity upgrade and they were trying to do this within a few months, which required a specific set of protocols.

We went to the drawing board and said, what's available and can still deliver the services that they need, and allow them to upgrade and achieve that timeline?

We knew they needed BGP and MPLS, we had to deliver Layer-2 VPN. There's a wide suite of Whitebox options that meet that requirement, and in the Service Provider realm - OcNOS from IP Infusion has a pretty feature rich deployment to satisfy all those requirements.

EPS Global has done a great job to date of delivering hardware to meet to meet our customers requirements across the globe. That was why we looked to you. We said "we need this hardware solution and this Operating System, can you deliver it in time to meet the customer's requirement?"

AF: I think we engaged with you for the first time in December of 2021, and correct me if I'm wrong, but I believe we had the network deployed by the end of February or at least part of it, is that correct?

JH: By the end of February, the switches were showing up at our dock. I would say, taking into account people being out over the holidays, it was about 3 months to deploy this network from start to finish.

AF: Obviously that compares very favorably to the legacy vendors that you had been talking to?

JH: Yes. We were hearing lead times and delivery times anywhere from a year to all the way up to 2 years.

AF: Judging by that, we were able to solve the lead time issue; but also from a cost perspective, I think you were able save a lot of money as well. Once you got into the Open Networking space, were there any other things that you found out that were pleasant surprises, or even nasty surprises? Hopefully there was more pleasant than nasty!

JH: I'll just throw in the big dog here and then let Nate take it from here! Firstly the licensing is a different scope than what you might expect from the larger providers that are out there. That was a pleasant surprise for me. I also love the flexibility of everything that we're able to accomplish with this. Nate, do you have anything to add?

NG: There is so much flexibility, with the IP Infusion software we're running, it’s the same software or slightly different variations of it can run across 3 different switch models and 2 different manufacturers, it’s all the same set up.

We have found along the way that not necessarily every switch model has the exact same underlying firmware, there's a couple different bugs to fix with one of the models that the other models don't have. But when we encountered some of these bugs, the IP Infusion guys have been able to turn around a patch for it in record time.

I can think of some legacy hardware where we've had bugs that have been ongoing for a year, 2 years, or even some that are still outstanding, whereas with these guys they're like: "we can work between the Whitebox switch and our software to figure out what's wrong, and we can get that patched up in pretty short order". That has been refreshing, and their support has been a lot easier to engage with compared to some of the legacy guys that we've worked with before. Sometimes you're jumping through 85 levels of support hell to get to where you need to be.

AF: A lesson I learned years ago was always do business with people who your business is important to. And I think for IPI, someone like MetaLINK is a very important customer. It's tougher to be a really important customer for someone like a Cisco or a Juniper, someone at that scale. It's tough to really get the answers.

Could you tell me a little bit about what the solution looked like that we came up with, what hardware, what software?

NG: We were looking to increase our transport links between all our different core sites. We ended up with the solution where originally we had a whole bunch of 10GbE DWDM circuits that were just lagged together and it was very difficult to manage. What we landed on was using the Edgecore 5916 switch running IP Infusion software, and then building 100G links instead, so that all of our core sites are now interconnected at 100Gbps, and we can start dropping our circuits we have at those core sites.

Most of the AS5916-54X’s end up with a few extra 100G ports, but then we also ended up with, 48x 10G ports. We can add a whole bunch of 10 Gig services or our ports at 10G speed at least to the solution. And, keep building that transport network up until we get to the point where, maybe we go to 200G or 400G, or whatever is the best when we hit that that next bump in performance.

VS:If I can jump into the technical details a little bit here. One of the big things originally was we're running a ERPS ring which could be cumbersome to manage, especially as it continues to grow, and you try to add new nodes. How do we still deliver those Layer-2 services but use routing? That's where we blended on segment routing to go with using EVPN for Layer-2 extensions or Layer-2 VPN so we can go and we can kind of simplify the protocol stack and make the operation and maintenance a lot easier and cleaner across the board. So you can focus on getting to that next step faster because you have less time spent just doing normal maintenance of the older solutions, for example I go and I lag together a whole bunch of DWDM circuits now if I just turn off another routed link and it's in place. It's not this big, long drawn-out process.

NG:We've actually put that scenario to the test. After we built our first site, we built out a ring. There were 12 locations that we added to 2 spokes of this ring, with the old solution it would have been a whole bunch of work to do. In this case it was we've raised in the switch we had it online and in 15 minutes. It was just a matter of you programming it and powering it on, we were able to test that pretty well. It's quite the time saver to just drop another switch in somewhere, route it into the same network, and away you go.

AF: Do you have advice for other companies that look like yourselves and are considering Open Networking? I'll open that question for you as well, Vince. I mean, what advice would you give to Service Providers? There's so many Service Providers that are faced with the same issues that you guys were faced with last year - having loads more bandwidth to provide, but dealing with the supply chain crunch and wondering about how to manage it.

NG: The biggest takeaway for me is: don't be afraid to try something new. Vince helped us to see that, just because this is a totally different solution than you're used to, you go to one company - you buy the switch, you buy the software, you buy the support, you buy all of the licenses, whereas with Whitebox, it's a little bit different. But don't be afraid to investigate that solution because it could check a lot of boxes for you.

JH: I'd add to that, Vince and IP ArchiTechs were very instrumental in helping us jump over that hurdle. There's lots of preconceived notions in the industry about "you got to have this if you want to play in this arena", but for different sized companies, that's not even a possibility. Like Nathan said, don't be afraid. Have a look at it. At the very least, go out and give it a test. It's affordable enough for you to pick up a couple switches and go to town.

VS: That would be the biggest thing that I would say too is, don't be afraid to look. Not everybody has the problems of the Tier-1’s, we're not all Lunin and Telia or early on. That set of equipment and Service Provider that's geared towards the Tier-1’s looks very different than what is a very valid and robust solution for the Tier-2’s and regional ISPs.

Being able to still deliver that same Quality-of-Service out to their customers without having to go through that same sort of traditional experience that Nathan was talking about, where you go to the vendor, you get the support, you get the hardware. It's a little different here, but if one switch model doesn't work, I can pull from a whole other set of switches that's on the same silicon, it might be a different manufacturer, but there's other options on the shelf.

I don't have to wait 18 months to deliver bandwidth, I can do it in 3-4 weeks, once we had their core moved over, now we're talking weeks to add new switches and new routers, not months or whatever the lead time is on "insert vendor here".

AF: I think that's a key point Vince. Jeff and Nate’s experience really shows this, that you're not on your own. I think that's a fear that some people have when they're entering into the Open Networking space, that they're on their own, and that no one really owns this, because the switch comes from somewhere; the software comes in somewhere else. I have to figure this out by myself.

That's not the case. if you're working with someone like IP ArchiTechs who understands the landscape and understands the solutions and then can work with someone like ourselves at EPS Global who actually has the inventory on hand, have the switches on hand, have the partnerships with the software companies, and can actually deliver in a very timely manner. You're not on your own, there's help out there. We're here to help, and I think that's a that's a really important message.

This probably sounds like a bit of a dumb question now at the stage that we're at in the discussion, but would you ever go back to the old way of doing things?

JH: We're still early in it. I can't say that it wouldn't have a necessity in certain places, but given the experience that we've had so far, I honestly don't see a reason we ever would. Going back to what you said about not being alone in this process, the co-operation that we've seen among yourselves at EPS, IP ArchiTechs, and then also even the software developer and the switch manufacturer. It was just amazing to see.

Nathan kind of touched on it earlier, the experience you have with the larger providers, maybe waiting a week or more to even be addressed by their support team, we immediately had a subject matter expert from the software company working directly with the hardware manufacturer and whenever they needed to get particular information or whatever, Nathan was able to work directly with Vince, and that all took place within days. I mean, literally days. And by the end of those days we had a solution. To me it's amazing.

Going back would be very hard for me to turn tail and say, well, I want to go back to the old way. It's like, no, I've had better now!

AF: Anything you want to add to that Nate?

NG: I guess just back maybe to the previous point about working with IP Infusion. We're working with a single point of contact. I don't have to contact Edgecore and say I've got this problem. I'm working with IP Infusion and then IP Infusion takes on the task of working with the underlying hardware manufacturer. That streamlines that process as well, you don't have to make a whole bunch of support tickets to get your issue addressed.

AF: Again, I think that really speaks to the idea that you're not on your own or left to figure this out alone. I really feel from talking to customers that it’s a concern that they have, particularly if they've been used to the Cisco service comfort blanket that they’re paying huge amounts of money for. But your advice earlier is very important - now that there's a certain level of comfort with Open Networking, it is worth giving it a try, don’t be afraid to take that first step., it's really important advice, give it a try.

VS: If I could jump in here... I think one of the things we kind of keep touching on here is the idea that in disaggregated networking there's no support channels, but that’s just completely false. In fact I'd say it's exactly the opposite. There's more support - that level of support and coordination that Nathan and Jeff have been talking about throughout this podcast.

AF: Yes, absolutely. I think the proof is in the experience that they've had. So what's next, guys? What's next for MetaLINK?

JH: On to bigger and better, I suppose! As the world of the Internet and Service Providers evolves into what the new reality is, obviously everything is going to be more. It's all about more. For MetaLINK particularly, it’s expanding our fiber networks and obviously, that's going to mean more and bigger packages for our customers, which is going to necessitate the need for a big iron solution to transport all of that and get it out to the Internet, and get that data back from the Internet.

AF: Thanks a lot, guys. We talked at the top of this about hoping that there's valuable lessons to be learned from the experience, and I really think there is. I think there's so many companies like yourselves out there, there's so much demand for an Open Networking and equipment, and we're here to help. Anyone who's listening to this who wants to find out more, please feel free to reach out to IP ArchiTechs or EPS Global and we can help you achieve the results that our customer MetaLINK has over the last number of months.

So thanks again, guys. Really appreciate the time.

If you have any questions about or need advice or tech support for your upcoming project, don’t hesitate to get in touch. Or check out our Service Provider Solutions here.