Inline EVPN between VMware NSX and SONIC
As promised , a more detailed blog on how to configure EVPN between a VMware NSX T0 Gateway and a switch running SONIC!
As a recap, this is the topology I have built, using NSX running on a vSphere Cluster, while using GNS3 on an ESXi VM to emulate the ‘physical’ switch fabric. For more information on how I got this working, check out my previous blog here.
I wanted to emulate a typical deployment where the ‘physical’ fabric is used as an ‘Underlay’ for NSX, while also providing connectivity for T0 Gateways, instantiated on NSX Edge Nodes. My ‘physical’ fabric was limited to a leaf (’Leaf-2’ in above picture) that was used to connect to the NSX T0 Gateway running EVPN , a leaf that will be used as the EVPN remote peer (‘Leaf-1’ in the above picture) , and a Spine interconnecting both Leaf Switches. The idea was to keep the switch fabric routing table small ie., to not have any VPN routes in them, with of course only ‘Leaf-1’ containing any VPN routes.
I decided to use IBGP for the underlay IPv4 routing, with the Spine switch functioning as the BGP Route Reflector, and IPv6 Unnumbered Interfaces on the Leaf-to-Spine links, which support IPv4 forwarding. These choices were mostly chosen to end up with a clean and easy configuration (or perhaps because I’m a network geek - or both :D).
In my case Interfaces Ethernet 7 and Ethernet 8 were the Spine interfaces that connected to the Leaf Switches. I also added a Loopback interface on the Spine switch, as well as the relevant BGP configuration, thus ending up with a very simple configuration as follows (default configuration omitted for clarity) :
Spine# show running-configuration interface Loopback 0
!
interface Loopback 0
ip address 192.168.254.254/32
Spine# show running-configuration interface e7
!
interface Ethernet7
description Connection to Leaf-2
no shutdown
ipv6 enable
Spine# show running-configuration interface e8
!
interface Ethernet8
description Connection to Leaf-1
no shutdown
ipv6 enable
Spine# show running-configuration bgp
!
router bgp 65535
router-id 192.168.254.254
!
address-family ipv4 unicast
maximum-paths 1
maximum-paths ibgp 1
network 192.168.254.254/32
!
neighbor interface Ethernet7
remote-as 65535
update-source 192.168.254.254
!
address-family ipv4 unicast
activate
next-hop-self force
route-reflector-client
!
neighbor interface Ethernet8
remote-as 65535
update-source 192.168.254.254
!
address-family ipv4 unicast
activate
next-hop-self force
route-reflector-clientThe Underlay configurations for Leaf-1 and Leaf-2 were equally simple, essentially a LoopBack interface configuration, and an IPv6 configuration on the Spine facing interfaces, and the relevant iBGP configuration on both Leaf Switches to announce Loopback’s, and an EBGP/BFD configuration to peer Leaf-1 to the NSX T0 Gateway, over an SVI interface tied to VLAN 1224.
Leaf-1:
Leaf-1# show running-configuration interface Loopback 0
!
interface Loopback 0
ip address 192.168.254.1/32
Leaf-1# show running-configuration interface Ethernet 8
!
interface Ethernet8
no shutdown
ipv6 enable
Leaf-1# show running-configuration bgp
router bgp 65535
router-id 192.168.254.1
!
address-family ipv4 unicast
maximum-paths 1
maximum-paths ibgp 1
network 192.168.254.1/32
!
neighbor interface Ethernet8
remote-as 65535
update-source 192.168.254.1
!
address-family ipv4 unicast
activateLeaf-2:
Leaf-2# show running-configuration interface Loopback 0
!
interface Loopback 0
ip address 192.168.254.2/32
Leaf-2# show running-configuration interface Ethernet 7
!
interface Ethernet8
no shutdown
ipv6 enable
Leaf-2# show running-configuration interface Ethernet 0
!
interface Ethernet0
no shutdown
switchport access Vlan 1224
Leaf-2# show running-configuration interface Vlan 1224
!
interface Vlan1224
ip address 192.168.124.200/24
Leaf-2# show running-configuration bgp
router bgp 65535
router-id 192.168.254.2
!
address-family ipv4 unicast
maximum-paths 1
maximum-paths ibgp 1
network 192.168.254.2/32
!
neighbor 192.168.124.1
remote-as 65000
update-source 192.168.124.200
bfd
!
address-family ipv4 unicast
activate
!
neighbor interface Ethernet7
remote-as 65535
update-source 192.168.254.2
!
address-family ipv4 unicast
activateOn the NSX side It was a very simple EBGP and BFD configuration to get the NSX T0 Gateway peering with Leaf-2, as can be seen on the screenshot below. I kept the standard IP route Filter, which allows the IPv4 address family. In Route Re-distribution I chose to only announce the T0 connected segments/uplinks/service/loopback interfaces, and static routes to the ‘physical’ fabric underlay. The idea here is that the parent T0 does not carry any tenant routes.
So far so good, now for the real stuff: BGP EVPN over VXLAN between a Tunnel End Point (TEP) on NSX and, a Tunnel End Point on Leaf-1, with the intention to route traffic between a workload on a VLAN on Leaf-1, and an workload on an overlay segment inside NSX, and this in the context of two different routing domains (red and blue), for example supporting individual tenants. For more details on Topology, VXLAN VNIs, VLAN id, VRF names, and subnet addresses, please refer to the drawing in the beginning of this blog.
The first thing that needs to happen is define an EVPN/VXLAN VNI Pool in NSX, as shown in this screenshot:
Then apply this VNI pool on your T0 Gateway, and set the EVPN mode to ‘Inline’, as shown in this screenshot. Note that you also need to define the local EVPN Tunnel End Points. I used 192.168.254.200 and 192.168.254.201, see the second screenshot. I had to include the EVPN TEP IP Tier-0 Subnet category under Route Re-distribution on the main page of the T0 Gateway as well, so that it would announce this /32 route to the ‘physical’ fabric.
I also built an NSX topology using two T0-VRFs:
VRF-RED (with VNI 75001, Route Distinguisher 65000:1, and a Route Target of 65535:1), and
VRF-BLUE (with VNI 75002, Route Distinguisher 65000:2, and a Route Target of 65535:2)
Then I needed some routes to advertise in those VRFs, so I created for each VRF a Tier-1 Gateway, and a Segment connected to it. Again refer to the above diagram for a topology and IP information.
Now that the NSX topology is built, the only remaining thing is to create the actual BGP EVPN neighbour. I had to set the Max Hop Limit to 4, as the BGP neighbour on Leaf-1 is a couple of routed hops away. I also had to add an additional Route Filter, to activate the L2VPN EVPN address family. I tried removing the IPv4 address family, but for some reason it kept on coming back. This is probably an implementation choice, allthough this BGP neighbour only needs to carry the L2VPn EVPN address family.
This completed the NSX side of things, now lets jump to the EVPN neighbour , the SONIC switch Leaf-1. I created two VLAN interfaces (10/20), added them to the right VRFs, and assigned two Ethernet Ports to the VLANs.
!
interface Ethernet1
no shutdown
switchport access Vlan 10
!
interface Ethernet2
no shutdown
switchport access Vlan 20
!
ip vrf Vrf-blue
!
interface Vlan10
ip vrf forwarding Vrf-blue
ip address 172.31.101.254/24
!
ip vrf Vrf-red
!
interface Vlan20
ip vrf forwarding Vrf-red
ip address 172.31.102.254/24Now lets add the VXLAN Tunnel End Point information, which maps the VLAN to the VNIs, as well as maps the VRFs to the VNIs:
!
interface vxlan vtep-leaf1
source-ip Loopback0
qos-mode pipe dscp 0
map vni 75001 vlan 10
map vni 75002 vlan 20
map vni 75001 vrf Vrf-blue
map vni 75002 vrf Vrf-red
!Then I need to add the router bgp configuration for the L2VPN EVPN address family. (Note that I omitted the already configured part of the IBGP configuration!)
!
router bgp 65535
!
address-family l2vpn evpn
advertise-all-vni
dup-addr-detection
!
neighbor 192.168.124.1
ebgp-multihop 4
remote-as 65000
update-source 192.168.254.1
!
address-family l2vpn evpn
activate
!This brought up the EVPN BGP peer with NSX, but no routes were being announced yet, as I still needed to add the VRF specific BGP configuration, which maps Route Distinguishers and Route Targets, as follows:
!
router bgp 65535 vrf Vrf-blue
log-neighbor-changes
timers 60 180
!
address-family ipv4 unicast
redistribute connected
maximum-paths 1
maximum-paths ibgp 1
!
address-family l2vpn evpn
advertise ipv4 unicast
rd 65535:1
route-target import 65000:1
route-target export 65535:1
dup-addr-detection
!
router bgp 65535 vrf Vrf-red
log-neighbor-changes
timers 60 180
!
address-family ipv4 unicast
redistribute connected
maximum-paths 1
maximum-paths ibgp 1
!
address-family l2vpn evpn
advertise ipv4 unicast
rd 65535:2
route-target import 65000:2
route-target export 65535:2
dup-addr-detection
!The next screenshots show that everything is working:
and of course we need to prove it with some pings :)
Leaf-1# ping vrf Vrf-blue 172.31.1.1
ping: Warning: source address might be selected on device other than: Vrf-blue
PING 172.31.1.1 (172.31.1.1) from 172.31.101.254 Vrf-blue: 56(84) bytes of data.
64 bytes from 172.31.1.1: icmp_seq=1 ttl=62 time=4.78 ms
64 bytes from 172.31.1.1: icmp_seq=2 ttl=62 time=4.65 ms
64 bytes from 172.31.1.1: icmp_seq=3 ttl=62 time=4.66 ms
64 bytes from 172.31.1.1: icmp_seq=4 ttl=62 time=4.24 ms
^C
--- 172.31.1.1 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 4.244/4.581/4.777/0.201 ms
Leaf-1# ping vrf Vrf-red 172.31.2.1
ping: Warning: source address might be selected on device other than: Vrf-red
PING 172.31.2.1 (172.31.2.1) from 172.31.102.254 Vrf-red: 56(84) bytes of data.
64 bytes from 172.31.2.1: icmp_seq=1 ttl=62 time=9.59 ms
64 bytes from 172.31.2.1: icmp_seq=2 ttl=62 time=4.85 ms
^C
--- 172.31.2.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 4.851/7.218/9.585/2.367 msSo, I hope this helps anyone out there who wants to use SONIC as their Underlay NOS, in conjunction with NSX EVPN. Cheers and see you next time!