Thanks to ChatGPT, there is a lot of somewhat confusing or basic information out there on the topic of Network Transport. To help make sense of it all, here’s an overview of the Network Transport options—written by a real engineer! Although partial to Flexential products, the themes, strategies, and solutions presented here are mainly uniform across all customers and vendors. Hopefully, this information will help you understand the Network Transport business and be better informed when buying products to connect critical locations.
What is Network Transport?
To start, let’s define Network Transport. Network Transport is connectivity to connect specific locations privately. This contrasts with Transit or Internet, which is connectivity to connect any location publicly.
With that understanding, many, many products meet the definition of Network Transport, each with its own costs, benefits, features, and underlying technologies:
- Layer 2 Point-to-Point Connectivity
- Layer 2 Multipoint Connectivity
- Layer 3 Multipoint Connectivity
- Layer 1 Wavelengths or Dark Fiber
- SD-WAN, IPSEC, and Tunnel-based approaches
This category breakdown and product list are not exhaustive but represent the most common approaches used today.
Layer 2 Point-to-Point
Several products and technologies today offer Layer 2 Point-to-Point transport, such as Ethernet Private Line (EPL) and L2VPN Pseudowire. You’ll also see iterations like Ethernet Virtual Private Line (EVPL), which can support multiple Point-to-Point connections on the same physical interface. These products have slight differences, but functionally they achieve a similar goal of connecting two sites. Connecting more than 2-3 sites with point-to-point solutions requires increasing complexity as it scales.
The benefit of this approach is that these technologies have been around for a long time, are highly reliable, and are among the most cost-effective. They are an ideal solution when connecting discrete locations or workloads.
For example, Flexential has offered multiple products in this category for over ten years and today delivers an L2VPN Pseudowire called Data Center Interconnect (DCI). We see this product used regularly for Disaster Recovery scenarios, where they are connecting a primary and backup site. In this scenario, workloads are critical and capacity needs are high, and there is typically no need for a major expansion of other sites and locations.
Standard DCI DR use case and deployment
The primary drawback of this approach is site or location scaling, especially when clients require a fully meshed network (i.e., all sites must talk directly to all other sites). Here, this complexity is highlighted.
Point-to-point at scale
If you want to know the math behind it, the formula [c = s * (s – 1) / 2] captures this scenario, where ‘c’ refers to total connections, and ‘s’ refers to the total number of sites. So, for example, if you wanted to connect ten sites with point-to-point technology, you would need… 45 connections! And this is also assuming a single connection hand-off!
Layer 2 Multipoint Connectivity
Layer 2 Multipoint is often described with the umbrella category of E-LAN. Still, you may also see references to Ethernet Virtual Private LAN (EVPLAN), or you might see references to some of the technologies that support the service, such as VPLS. The naming and marketing conventions of this product category can be confusing, but functionally, they provide layer 2 multipoint-to-multipoint connectivity. The operator experience is as if all your sites are connected to the same switch on the same VLAN ID.
IX Mesh – Layer 2 operator experience
Multipoint-to-multipoint connectivity has at least two major benefits. First, scalability is far superior with this type of product, as new sites can be added without any changes or new connections to the existing sites. Secondly, many clients find managing failover and redundancy easier with this type of product. With DCI or point-to-point, you must route to different DCI circuits and/or devices to utilize failover, as each DCI is its unique network. With multipoint, all connections are on the same network. Especially for clients with active/passive firewall clusters at their edge, multipoint offers superior ease of use when it comes to ‘built-in failover.’
Flexential offers a Layer 2 product in this category called Interconnection Mesh. In the diagram below, we’ve highlighted the problems we see regularly with clients trying to perform failover with traditional point-to-point products.
Limitations of Point-to-Point Failover
Regarding drawbacks, Interconnection Mesh or multipoint products are more expensive than point-to-point solutions in scenarios where clients only connect 2-3 sites.
Lastly, a critical technical decision when looking at multipoint products is whether you want Layer 2 or 3. This is not a cost vs. benefit analysis but a tradeoff in management vs. control.
- With Layer 2 Multipoint, clients have complete control (and responsibility) for all networking routing.
- With Layer 3 Multipoint, clients participate in routing but rely on the vendor as a centralized router for all connected sites.
Clients who want multipoint connectivity will want to consider how much—or how little—control they want over their network routing.
Layer 3 Multipoint Connectivity
Layer 3 Multipoint Connectivity is often labeled L3VPN or IPVPN. There is also an industry-wide misnomer where some folks refer to this as ‘MPLS,’ but MPLS is an underlying protocol used to support this product, not the product itself. The operator experience is like E-LAN described above, but rather than the vendor giving you a Layer 2 switch between sites, they are giving you a Layer 3 router. As a solution, there is a lot of potential overlap between Layer 2 Multipoint and Layer 3 Multipoint, as many problems could be solved with either solution. The choice usually comes down to preference, control, and specific use cases.
For clients with limited resources or knowledge of multi-site routing, especially if they are new to dynamic protocols like BGP, L3VPN can add value by situating the provider as a centralized router. This allows clients to manage a single BGP peering session with the provider, rather than peering or routing directly with all remote sites, and can easily make changes or expansions. With Layer 2 multipoint, if you add a site, you do not need to do anything at existing sites physically, but you will need to update your configs at existing sites logically. With Layer 3 multipoint, you often do not need to do anything—physical or logical—at existing sites when a new site comes online.
The tradeoff is control and features. Clients may want a provider to refrain from participating in their routing, or they may be looking for advanced BGP features that the provider cannot support. Most L3VPN products do not support advanced BGP features like community-based traffic engineering, granular route filters, etc. For clients who need specific and complete control over routing, Layer 2 solutions are preferable.
Flexential offers Layer 2 (E-LAN) and Layer 3 (L3VPN) multipoint iterations under the product family Interconnection Mesh. As mentioned, the general theme we observe is that clients are focused on management and control when selecting their deployment and that both are equally suited for large multi-site deployments.
Lastly, we see an advantage with L3VPN in one specific use case. Some customers, especially those connecting to remote clouds and large numbers of Layer 2 devices, prefer L3VPN to circumvent protocols like spanning-tree. This is a hyper-specific and niche concern, but for any network engineer that has worked on spanning-tree, this likely triggers some awareness. Whatever your product or use case, if you use Layer 2 connectivity, you must understand and account for spanning-tree. For some customer deployments, it is easier to avoid it by using a Layer 3 product. Flexential generally recommends L3VPN for mixed and cloud connectivity for just this reason.
Layer 2 vs. Layer 3 IX Mesh
Layer 1 Wavelengths or Dark Fiber
Clients can also buy Network Transport down at its lowest layers or physical layer. Dark Fiber is literally as it sounds, with clients being required to supply the optical gear necessary to power the fiber end-to-end. Wavelengths are carrier products that mimic a Layer 1 connection like Dark Fiber. Still, the provider is powering the optical light and providing a limited range of the optical spectrum instead of the whole spectrum. The operator's experience is as if you have a physical cable—not a switch or router—between your devices.
The primary use case and value of this type of Network Transport is when clients want a large capacity. For example, a standard wavelength product is almost always 100Gbps, and it is impossible to offer concepts or restrictions like “committed rates.” You must buy the entire pipe.
Most Flexential clients do not need this level of capacity and control, so we only offer limited products in this area. In dense and major metros like Hillsboro, Flexential does maintain some Dark Fiber offerings, but it is not a product line we have extended to all markets.
SD-WAN, IPSEC, and Tunnel-based approaches
SD-WAN is a bit of a black box. Today, while there is some active consolidation in the marketplace, there remain 20+ active vendors and no RFC or industry-wide standard. “SD-WAN” does not mean a specific technology but a loose category of approaches.
Generally, one of the primary approaches is for SD-WAN to create IPSEC tunnels over the internet between sites. IPSEC tunnels connect 1.) specific sites and 2.) do it privately with encryption, so we think of this as a kind of ‘Network Transport.’ Historically, no one loved tunnels for transport because the internet was unreliable. Today, reliable internet is more available, and these SD-WAN providers have all developed proprietary protocols in error detection and error correction to patch the gaps where the internet fails. Once unthinkable, some clients are now actively retiring “dedicated transport” and using internet-based tunnels to connect their sites. This used to be driven by cost, but with dedicated transport dropping in cost so heavily compared to the internet, it is unclear that cost is still a significant factor in this shift.
The lack of consistent internet performance—and a lack of industry standards in SD-WAN—means that this approach to transport can be good or bad.
For clients taking this approach, one of the best investments available is to get ‘good’ internet, which Flexential offers via its IP Bandwidth product line in all facilities. IP Bandwidth blends 3 x Major Tier I ISPs, provides full N+1 physical and logical diversity at all sites, and offers a 100% uptime SLA to all redundant clients. Even if you have the best SD-WAN with the best error-correcting protocols, tunnel transport will not function if your underlying internet does not work.
Although there is much more to write about SD-WAN and tunnel transport, for the sake of keeping this within the context of ‘Network Transport,’ we’ll leave it this: if you’re going to put your transport over the internet, get good internet!
Conclusion
Finding a partner that can provide the choices of services to allow you to build and expand is crucial. There is no one size fits all solution offering. Flexential offers a wide range of interconnection services that include the following:
- Interconnection Mesh, a multi-point, any to any network with Layer 2 and Layer 3 options
- DCI, a layer 2 point-to-point circuit
- Dark Fiber cross connects to connect customers between destinations.
There is a lot to consider when connecting your sites. The biggest takeaway is to keep these high-level categories in mind, especially as you explore options for site-to-site connectivity. Planning and optimizing networks takes work, and Flexential Professional Services offers several workshops to assist you in developing your network strategy. If you have questions or need more details, your Flexential representative can dive deeper into our specific interconnection products.
Thanks for reading! Happy networking and remember, wearing a tank top is the best way to cable in the hot aisle.
Learn more about Network Transport and the full portfolio of Flexential Interconnection offerings, register for a workshop, or contact us for more information.
