ISP Design Guide:
Overview and introduction to separated functions
Core - The job of the network core is to NAT - Network Address translation is QoE - Quality of Experience or QoE is a term Shaping - Shaping of traffic is an important
What are network functions? connect all other devices and functions as increasingly used by service providers as that's become popular within the last 5 element in end-to-end delivery of bandwidth.
simply as possible. IPv4 has become more scarce. years or so. It generally refers to a shaping It can help to smooth issues with capacity,
Network Functions - The major tasks in the data plane that must be performed by an L2/ L3 network device to appliance that has advanced traffic backhaul quality and even wireless
ensure smooth delivery of the Internet from the border of an ISP down to the subscriber last mile. Examples are Ideally, the core has a very simple L2/ L3 Typically CG-NAT in a NAT444 configuration identification capabilities at L7 and data problems inside the subscriber's home or
border routers, core switches and aggregation routers. config and enough ports to connect the to support a dual stack deployment with about the health of the network. The business.
current prod devices and have room for IPv6 is the most common. appliance will normally sit logically or
What are operational support functions? growth.
This is a market segment that's grown
physically inline between routers in the core
and the aggregation layer.
Without going into an enormous amount of
detail around shaping, it differs from
This is a great place for Layer 3 switches significantly in the last year due to the policing/ rate limiting in that it queues a
Operational Support Functions - The major tasks in the control and management plane that must be because they are fairly inexpensive these bandwidth explosion caused by the Most QoE appliances use Active Queue portion of traffic and tries to hit a target rate
performed by a device or service to facilitate and support the operation of network functions. Examples are DHCP, days and come with a variety of port pandemic and a move to working remotely. Management (AQM) shapers like fq_codel before it's exceeded.
DNS, Applications/ Servers, Billing Systems, Corporate VPNs and connectivity. layouts, densities and speeds. This function can also use NAT64 or or cake to manage throughput to each
464XLAT for single stack networks that subscriber. Whereas policing/ rate limiting is a hard limit
Why separate them? need IPv4 connectivity. that drops traffic as soon as the max rate is
reached. This function is commonly found
It's tempting for new and even experienced ISPs to pile all of the functions into one router, switch or server - and on aggregation and last mile routers but can
then add another for "redundancy". This generally creates problems with complexity, failure domains and growth. also be a separate appliance which will be
Separating functions allows for network designs to be modular, repeatable and more scalable. Automation is expanded on in the QoE description.
easier beacuse templating is easier. The end result is better uptime, lower opex, easier growth and lowered risk.
Call or e-mail for professional Network Functions
iparchitechs.com +1(855)645-7684 consulting@iparchitechs.com
assistance with your network
Network Functions Network Functions
Core
NAT QoE/ Shaping
ISP #1
RF Last Mile
ISP #2
FTTH
Operational Support
OOB Functions
Management
Network Functions Firewall/ VPN Network Functions
Border Aggregation/ BNG
Applications
MPLSPE
Servers
VPN
Border - Upstream connectivity to the
Storage
Internet.
Commonly referred to as IP transit, this Out of Band Aggregation/BNG - These devices terminate
typically involves a BGP peering between services for subscribers and are often the
ASNs to advertise prefixes and receive the L3 gateway for the last mile.
full global routing table and/ or partial routes
and a default route. This device is typically where a lot of
complexity is pushed to in an ISP network.
Internet Exchange or IX is another type of
peering that border routers establish to
connect directly with other ASNs and
ISP - LTE It's not uncommon to put more complex
security rules, RADIUS billing integration,
bypass transit. overlay termination and complex VLAN
tagging operations - as well as other types
Peering and failver can become more of config that wouldn't generally go into a
complex as the ISP grows, so keeping the border or core router.
border layer separate reduces the amount of
churn when a peering needs to be turned up Firewall - One of the more common Applications - ISPs often require a variety of Servers/Storage - Most ISPs will run their Out of Band (OOB) - This is one of the most The role can also combine other functions
or down because the border layer and any mistakes that new ISPs make is using the applications and systems to operate the own servers and storage to support overlooked additions to an ISP network. like NAT and Shaping.
failover or path changes are insulated from routers that carry subscriber traffic as the network and manage both sales and applications and corporate needs. OOB is a critical element to executing
the customer gateways on the agg/ bng security and L3 gateway for applications technical operations for the subscriber. low-risk maintenance windows. This is sometimes called a Broadband
routers by the core layer. and servers. This can be as simple as a single server Network Gateway or BNG which is a term
Examples of ISP applications are with a single disk or as complex as several It's also incredibly helpful to reduce the from the days of PPPoE and usually implies
This also allows for connection tracking and This creates a challenging environment for monitoring, billing, management controllers, racks of dedicated compute and storage. number of trips to the data center and can dynamic service provisioning of shaping
maintaining "state" to be turned off on the server/ application failover, becuase it's IPAM and DNS/ DHCP. provide access during an unplanned outage. polices, IP assignment and other attributes
border and core functions so they can directly tied to subscriber failover and Whether the servers are intended to be used via RADIUS integration with DHCP and a
defend against low level attacks and put uptime. Designing an application infrastructure can as bare metal, hypervisors or container Out of band networks typically are billing system.
more resources into throughput and not be a daunting task for some and if it's not platforms, building them in the "data center" standalone with a router, switch, console
unnecessary "border" firewalls. It's easy to take down the entire network separated from the service provider data zone as a separate function keeps the server and possibly a laptop or small server. MPLS PE - Similiar to agg/ bng, the role of
when working with firewall and NAT rules so plane routers, it can be very hard to grow servers from becoming a dependency in the an MPLS PE is to terminate MPLS services
Not all startup ISPs will peer with their creating a separate "data center" zone for and change. prod ISP routers. IP enabled PDUs are an optional but very like VPLS and L3VPN and forward the traffic
upstream, so for the purposes of this operational support functions makes sense. desirable item to have in an OOB network to to non-MPLS enabled routers/ switches.
drawing, Direct Internet Access or DIA is With the advent of cloud services, it's One common and often costly mistake is remotely power cycle equipment.
assumed to fall under border as well. Using a router as a firewall or an actual UTM sometimes more advantageous to begin running virtual routers that are intended to Typically the MPLS PE will be using an IGP
firewall helps to separate the applications with cloud hosted applications or split some be used in a network function role like The internet connection used to feed the like OSPF or IS-IS and label distrbution via
and services needed to support the ISP of the resources between the cloud and on border or BNG on top of the same infra OOB network should be separate (if LDP or SR-MPLS. Some ASNs will also run
from subscriber traffic. premises. that's used to deliver applications. possible) from the transit or DIA iBGP to advertise prefixes to the border.
connections used to connect subscribers.
VPN - Separate from the OOB VPN, an in By separating applications into their own Network Function Virtualization or NFV is Sometimes the BNG function is mixed with
band VPN can be used for both techincal function within operational support, it's intended to be used on a standalone In this example, LTE securely connects the the MPLS PE function and MPLS services
and corporate access to the networks as much easier to scale and move applications server/ hypervisor that's dedicated to the OOB network with users and might use a are dynamically provisioned, shaped, etc.
well as apps and services. to and from the cloud without impacting the data plane and will not be running mesh VPN like ZeroTier.
production data plane. applications or clustering.
This also makes security patches much The performance tweaks for apps. vs.
easier as the impact of upgrading is lower routers are different and can interfere with
when the entire ISP isn't on one device. each other. Keep these roles separate.
