Network configuration management at

CERN
Arkadiy Shevrikuko
Stefan Stancu

01/18/2022 1
 Outline
• Network overview
• Current solution: cfmgr
• Overview of open-source platforms
• Evolution plan

01/18/2022 2
 Network at CERN
• Around 4,4 k network devices
• Multi-vendor network (CERN procures equipment with open call for tenders)
• Specific configuration for each environment.
4000 switches
400 routers

01/18/2022 3
Network configuration automation
workflow at CERN

CFMGR

LanDB

01/18/2022 4
CFMGR – configuration
management tool

01/18/2022 5
 What is cfmgr?
• Perl-based command-line tool
• Helps to ensure consistent configuration for the network
• Uses central DB as a source of truth
• Features:
• Multi-vendor support
• Supports multiple operations
• Basic configuration management (load, align)
• Automatic configuration of ACLs and PoE
• Large scale firmware management
• etc…
• Integration with Spectrum CA
• Sends cron job status notifications

01/18/2022 6
CFMGR architecture

01/18/2022 7
 Evolution motivation
• Need to support a new router platform
• different configuration workflow
• Decrease in popularity of Perl
• Most network automation libraries (vendor or open-source) use
Python
• Large code base with non-uniform/outdated coding practices
• No clear separation between different modules of the system

01/18/2022 8
Open-source automation solutions

01/18/2022 9
Network automation stack
• A network automation stack comprises
multiple layers:
• Orchestration: triggering the action of
reconfiguring network devices
• Model: device independent network
configuration data
• Driver and device Interface: translate the
model to the appropriate device specific
format and enforce it on the device

01/18/2022 10
 N.A.P.A.L.M.
• N.A.P.A.L.M. (Network Automation and Programmability Abstraction Layer with Multivendor
support)
• Python library that implements a set of functions to interact with different network device Operating Systems using a unified API.
• N.A.P.A.L.M. API covers two aspects:
• Configuration management
• Getters (getting information from the device)

01/18/2022 11
• Python-based tool • For every device type,
• All orchestration is based on vendor, generation separate
playbooks: programs, written set of playbooks should be
in human readable language written
• Generates configuration • Only one node controls whole
using template language network, which can cause
(jinja2) and yaml data files concurrency issues during
multiprocessing operations
• No need to install additional
software on the network
• Simple set of operations
devices supported

01/18/2022 12
• Generation of the • Master-slave
configuration using architecture
templates
• Python-based
• Works over SSH in
• Reactive behaviour “headless” mode
• Scalable architecture • Provide layer of
• Supports ACL generation abstraction over
out of the box N.A.P.A.L.M.

01/18/2022 13
• Python based. • No multi-vendor support
• StackStorm is a • Was created to
IFTTT(if-this-then-that) orchestrate servers, but
orchestration platform not real network
• There are basic devices.
workflows for network • N.A.P.A.L.M. is the
configuration based on main source of network
N.A.P.A.L.M. device automation

01/18/2022 14
Open-source platform applicability
• There is no perfect automation platform
which can cover full network automation
stack.
• All network configuration capabilities are build
on top of N.A.P.A.L.M.
• Some of them have scalability issues
• Lack of multi-vendor support
• Generation of the configuration is still our duty
• Configuration models are vendor specific
• Partial cover of configuration generation with
OpenConfig

01/18/2022 15
Evolution plan

01/18/2022 16
CFMGR – current state

01/18/2022 17
 Evolution plan (1)
Not covered by any automation platform.
Should be developed

N.A.P.A.L.M.

Python libraries, provided by vendors

01/18/2022 18
 Evolution plan (2)

Configuration
Reuse CFMGR generator
logic forshould be
created
creatingusing Python
general and
configuration Use N.A.P.A.L.M. for diff
templates (eventually)
in universal for creating
format (YAML, operation and interaction
device specific configuration from
XML, JSON) with the network device
generic one

01/18/2022 19
 Evolution plan (3)
• Decouple concerns:
• Configuration generation: in-house
• Generate vendor independent configuration (Perl 
Python)
• Gradually use REST interface for retrieving database information
• Generate vendor specific configuration (use XSD/YANG
schemas if available)
• Configuration enforcement
• Use N.A.P.A.L.M. and vendor libraries (Python)

01/18/2022 20
 Evolution plan (4)
• Transition phase, need Perl – Python interaction
• Use ‘system’ to call full programs
• Structured input/output data (JSON and/or YAML)
• Proxy server for communicating to devices:
• Expose methods via XML-RPC

• Start with new vendor support

• Back-port to vendors still in production once the platform becomes
complete

01/18/2022 21
 Timeline

Autumn 2018: Early 2019: End of 2019: 2020+

Beginning of the NAPALM driver and Python Full Python-based
development. vendor specific implementation configuration
configuration of core vendor generation
generator. Support independent Additional features
of Juniper configuration (e.g. VLANs)
configuration elements Integrate other
workflow vendors
 Summary

Network configuration automation is a must

Due to the scale and diversity of CERN’s networks

Today there is no commercial or open-source product, capable of replacing cfmgr

We are evolving the tool in order to leverage open-source tools with multi-vendor
support:

Faster and easier integration of the new vendors

Easier maintenance of the tool

Faster implementation of new features

New calls for tender will most likely require programmatic configuration APIs

01/18/2022 23
 Questions?

01/18/2022 24
01/18/2022 25
 Things to improve
• Make use of open-source libraries for automation
and integrate them to the existing system
• Switch to modern programming language
• Integrate vendor specific APIs for generation of the
configuration
• Clear border between different modules of the
cfmgr

01/18/2022 26
 Update plan
• Different parts of cfmgr will be replaced with the new
ones
• Python-based modules which are going to interact with
Perl cfmgr using XML RPC mechanism
• Initial plan is to have support only for new vendors
(Juniper)
• Once architecture is clear and robust – implement
support for other vendors

01/18/2022 27
 Update plan
• Use NAPALM to organize communication between cfmgr and
network device
• Using RPC calls from perl to python
• Generation of the configuration
• Using XSD or yang schemas provided by vendors, to generate
appropriate classes
• Convert class based configuration to text version
• Replacement of the platform, which communicates with LANDB
• Rest API is available for deriving required data from LANDB

01/18/2022 28