First, let’s start with prerequisites. You should be familiar with vxlan, leaf-spine, evpn, cumulus switches. If you are not or you want to refresh your memory you can check the links below:
What is a VXLAN?
You can find a lot of info on this over the internet, but here are 2 articles:
https://www.sdxcentral.com/networking/virtualization/definitions/what-is-vxlan/
https://medium.com/@NTTICT/vxlan-explained-930cc825a51
What is evpn?
Here is a good answer: https://rickmur.com/evpn-rfc-7432-explained/
What is Cumulus?
Cumulus Linux is an open source Linux-based networking operating system for bare metal switches. Cumulus Networks is the software company that has designed it.
More info, here:
https://en.wikipedia.org/wiki/Cumulus_Networks
https://cumulusnetworks.com/about/
Leaf-Spine arhitecture:
https://blog.westmonroepartners.com/a-beginners-guide-to-understanding-the-leaf-spine-network-topology/
https://blog.mellanox.com/2018/04/why-leaf-spine-networks-taking-off/
I’ve used GNS3 as a lab environment. Also, I’ve tried to make the topology as simple as possible so it won’t take too many resources from the host.
I’ve created 2 leaves and one spine. If you have enough resources you can expand the topology to have more spines and leaves. The Cumulus OS version I’ve used is 3.7.10.
The Cumulus documentation i’ve used is this:
https://docs.cumulusnetworks.com/version/cumulus-linux-37/Network-Virtualization/
I’ve configured a VXLAN between the 2 leaves. Leaf 1 is made of Cumulus switch1 and 2, which are configured with a LAG towards cumulus 3 switch, and also an anycast IP so they act as one VTEP ( https://docs.cumulusnetworks.com/version/cumulus-linux-37/Network-Virtualization/VXLAN-Active-Active-Mode/ ). Also, Cumulus 1 and 2 switches are configured with VRR (Virtual Router Redundancy). VRR is similar to VRRP or HSRP. Cumulus 3 switch role is only so I could configure a LAG.
Leaf 2 has only one Cumulus switch. It should have been similar to the other leaf but to keep the topology as light as possible I’ve used just one switch, which is enough to configure BGP,evpn,vxlan.
There is only 1 VLAN and 1 vxlan(id 100 for both) in the lab, but that’s enough as an example.
EVPN is used as a control-plane mechanism to learn and share the MAC addresses. EVPN is often referred to as the means of implementing controllerless VXLAN.
Important: we need an anycast IP attached to the loopback on each clag pair, which is for northbound traffic, and an anycast IP for each clag pair for vrr, which is for southbound traffic.
So let’s start with the creation of VLANs, LAG, and VRR:
net commands============switch1# net add clag peer sys-mac 44:38:39:FF:01:01 interface swp5-6 primary backup-ip 10.0.0.2switch1# net add vlan 100-200switch1# net add clag port bond bond1 interface swp1 clag-id 1switch1# net add bond bond1 bridge trunk vlans 100,200switch1# net add vlan 100 ip address 192.168.1.252/24switch1# net add vlan 100 ip address-virtual 00:00:5e:00:01:00 192.168.1.254/24switch1# net pendingswitch1# net commitswitch2# net add clag peer sys-mac 44:38:39:FF:01:01 interface swp5-6 secondary backup-ip 10.0.0.1 backup-ip 10.0.0.1switch2# net add vlan 100-200switch2# net add clag port bond bond1 interface swp1 clag-id 1switch2# net add bond bond1 bridge trunk vlans 100,200switch2# net add vlan 100 ip address 192.168.1.253/24switch2# net add vlan 100 ip address-virtual 00:00:5e:00:01:00 192.168.1.254/24switch2# net pendingswitch2# net commitswitch3# net add bridge bridge ports swp5-6switch3# net add bridge bridge vids 100,200switch3# net add bridge bridge pvid 1switch3# net add interface swp5-6 bridge access 100switch3# net add bond bond1 bond slaves swp1-2switch3# net add bond bond1 bridge trunk vlans 100,200switch3# net pendingswitch3# net commitswitch5# net add bridge bridge ports swp5-6switch5# net add bridge bridge vids 100,200switch5# net add bridge bridge pvid 1switch5# net add interface swp5-6 bridge access 100switch5# net add vlan 100 ip address 192.168.1.251/24Verification============switch1# net show interfaceswitch1# net show clagclagctl -v |
Next, is the BGP, evpn part. I’ve started with configuring the loopback interfaces and their anycast IPs (anycast only for switches 1 and 2).
Then I moved to BGP and evpn and advertising the loopback IP.
You will notice that there are no neighbor statements. This is because Cumulus can use a feature called BGP unnumbered. This is using the ipv6 link-local addresses to establish the neighbor relationship.
More on this, here: https://cumulusnetworks.com/blog/bgp-unnumbered-overview/
############# BGP routing ##################/etc/frr/daemonssudo systemctl enable frr.servicesudo systemctl start frr.service### EVPN Configurationswitch1# net add loopback lo ip address 10.1.1.1/32switch1# net add loopback lo clag vxlan-anycast-ip 10.10.10.10switch1# net add bgp autonomous-system 65001switch1# net add bgp neighbor swp3 interface remote-as externalswitch1# net add bgp l2vpn evpn neighbor swp3 activateswitch1# net add bgp l2vpn evpn advertise-all-vniswitch1# net add bgp network 10.1.1.1/32switch1# net add bgp network 10.10.10.10/32switch1# net pendingswitch1# net commitswitch2# net add loopback lo ip address 10.1.1.2/32switch2# net add loopback lo clag vxlan-anycast-ip 10.10.10.10switch2# net add bgp autonomous-system 65002switch2# net add bgp neighbor swp3 interface remote-as externalswitch2# net add bgp l2vpn evpn neighbor swp3 activateswitch2# net add bgp l2vpn evpn advertise-all-vniswitch2# net add bgp network 10.1.1.2/32switch2# net add bgp network 10.10.10.10/32switch2# net pendingswitch2# net commitswitch4# net add loopback lo ip address 10.1.1.4/32switch4# net add bgp autonomous-system 65004switch4# net add bgp neighbor swp1-2 interface remote-as externalswitch4# net add bgp l2vpn evpn neighbor swp1-2 activateswitch4# net add bgp network 10.1.1.4/32switch4# net pendingswitch4# net commitswitch5# net add loopback lo ip address 10.1.1.5/32switch5# net add bgp autonomous-system 65005switch5# net add bgp neighbor swp1 interface remote-as externalswitch5# net add bgp l2vpn evpn neighbor swp1 activateswitch5# net add bgp l2vpn evpn advertise-all-vniswitch5# net add bgp network 10.1.1.5/32switch5# net pendingswitch5# net commitVerification============net show route net show route ipv4 |
Next is the VXLAN part. There is only 1 vlan and 1 vlxan, id 100. This will provide connections between PC1 to PC2 . The VTEP’s will be: switch 1 and 2 , which will act as a single VTEP using the anycast ip of 10.10.10.10, and another VTEP on switch 5.
#### VxLAN Configurationswitch1# net add vxlan vni100 bridge access 100switch1# net add vxlan vni100 vxlan id 100switch1# net add vxlan vni100 bridge learning offswitch1# net add vxlan vni100 stp bpduguardswitch1# net add vxlan vni100 stp portbpdufilterswitch1# net add vxlan vni100 vxlan local-tunnelip 10.1.1.1switch1# net add vxlan vni100 bridge arp-nd-suppress onswitch2# net add vxlan vni100 bridge access 100switch2# net add vxlan vni100 vxlan id 100switch2# net add vxlan vni100 bridge learning offswitch2# net add vxlan vni100 stp bpduguardswitch2# net add vxlan vni100 stp portbpdufilterswitch2# net add vxlan vni100 vxlan local-tunnelip 10.1.1.2switch2# net add vxlan vni100 bridge arp-nd-suppress onswitch5# net add vxlan vni100 bridge access 100switch5# net add vxlan vni100 vxlan id 100switch5# net add vxlan vni100 bridge learning offswitch5# net add vxlan vni100 stp bpduguardswitch5# net add vxlan vni100 stp portbpdufilterswitch5# net add vxlan vni100 vxlan local-tunnelip 10.1.1.5switch5# net add vxlan vni100 bridge arp-nd-suppress onVerification============net show route ipv4net show evpn vninet show bgp l2vpn evpn vni 100net show bgp l2vpn evpn routenet show evpn mac vni 100 |
The GNS3 project files and the config of the switches can be found in the link below:
https://github.com/czirakim/VxLAN-s-on-Cumulus
Great job, chap!
Thanks, mate!