VXLAN-EVPN with IRB QoS
Overview
An EVPN-based Integrated Routing and Bridging solution used for forwarding of intra-subnets and inter-subnets traffic. Here QoS is applied on IRB solution for L3 packets.
VXLAN quality of service (QoS) provides differentiated service in VXLAN applications. A device implements mapping between QoS priorities in original packets, internal priorities (local precedence assigned by the device to differentiate service classes of packets), and priorities of encapsulated packets. In this way, the switch provides the differentiated QoS service based on original packets.
Topology
The procedures in this section use the topology in Figure 10-11
VxLAN_EVPN_IRB
Note: In the above topology TS1, TS2 are the tenant systems. The blue and red color denotes different subnets in the Tenant systems.
Base Configuration - L2 VXLAN
VTEP1
(Multi-homed group1) - Part of both Multi-homed with po1(MH2).
Hardware profile and generic configuration:
#configure terminal | Enter Configure mode. |
(config)#hardware-profile filter vxlan enable | Enable hardware-profile filter for VXLAN. |
(config)#hardware-profile filter vxlan-mh enable | Enable hardware-profile filter for VXLAN multi-homing. |
(config)#hardware-profile filter egress-ipv4 enable | Enable hardware-profile filter for egress IPv4. |
(config)#evpn vxlan multihoming enable | Enable Multihoming, save configs and reboot the board for multihoming to be effective |
(config)#hardware-profile statistics ac-lif enable | Enable ac-lif for vxlan access-if port counters |
(config)#qos enable | Enabling qos |
Interface and loopback configuration:
(config)#interface po1 | Enter Interface mode for po1 (MH2) |
(config-if)#switchport | Make it L2 interface |
(config-if)# evpn multi-homed system-mac 0000.0000.2222 | Configure system mac as ESI value for Lag (po1) interface. VTEP1 and VTEP2 should have same ESI value |
(config-if)#exit | Exit Interface mode and return to Configure mode. |
(config)#interface xe3 | Enter Interface mode for xe3 |
(config-if)#channel-group 1 mode active | Make it member port of po1 |
(config-if)#exit | Exit Interface mode and return to Configure mode. |
(config)#interface lo | Enter Interface mode for lo |
(config-if)#ip address 1.1.1.1/32 secondary | Configure loopback ip address as 1.1.1.1 for VTEP1 |
(config-if)#exit | Exit Interface mode and return to Configure mode. |
(config)#interface ce52 | Enter Interface mode for network side port |
(config-if)#ip address 100.11.11.1/24 | Configure ip address as 100.11.11.1 on network side of Spine-P3 |
(config-if)#exit | Exit Interface mode and return to Configure mode. |
OSPF configuration:
(config)#router ospf 100 | Enter into router OSPF mode |
(config-router)#ospf router-id 1.1.1.1 | Configure router-id as 1.1.1.1 (lo ip address) |
(config-router)#network 1.1.1.1/32 area 0.0.0.0 | Add 1.1.1.1 (lo ip address) network into area 0 |
(config-router)#network 100.11.11.0/24 area 0.0.0.0 | Add 100.11.11.0(Spine-P3) network into area 0 |
(config-router)#bfd all-interfaces | Enabling bfd on all ospf interface for fast convergence |
(config-if)#exit | Exit Interface mode and return to Configure mode. |
BGP configuration:
(config)#router bgp 5000 | Enter into Router BGP mode |
(config-router)#bgp router-id 1.1.1.1 | Configure router-id as 1.1.1.1 (lo ip address) |
(config-router)#neighbor 2.2.2.2 remote-as 5000 | Specify a VTEP2 loopback ip address and remote-as defined |
(config-router)#neighbor 2.2.2.2 update-source lo | Configure update as loopback for VTEP2 |
(config-router)#neighbor 2.2.2.2 advertisement-interval 0 | Configure advertisement-interval as 0 for fast convergence for VTEP2 |
(config-router)#neighbor 4.4.4.4 remote-as 5000 | Specify a VTEP4 loopback ip address and remote-as defined |
(config-router)#neighbor 4.4.4.4 update-source lo | Configure update as loopback for VTEP4 |
(config-router)#neighbor 4.4.4.4 advertisement-interval 0 | Configure advertisement-interval as 0 for fast convergence for VTEP4 |
(config-router)#neighbor 5.5.5.5 remote-as 5000 | Specify a VTEP5 loopback ip address and remote-as defined |
(config-router)#neighbor 5.5.5.5 update-source lo | Configure update as loopback for VTEP5 |
(config-router)#neighbor 5.5.5.5 advertisement-interval 0 | Configure advertisement-interval as 0 for fast convergence for VTEP5 |
(config-router)#address-family l2vpn evpn | Enter into l2vpn EVPN address family mode |
(config-router-af)#neighbor 2.2.2.2 activate | Activate 2.2.2.2(VTEP2) into l2vpn evpn address family mode |
(config-router-af)#neighbor 4.4.4.4 activate | Activate 3.3.3.3(VTEP4) into l2vpn evpn address family mode |
(config-router-af)#neighbor 5.5.5.5 activate | Activate 5.5.5.5(VTEP5) into l2vpn evpn address family mode |
(config-router-af)#exit-address-family | Exit from l2vpn address family mode |
(config-router)#exit | Exit from Router BGP mode and enter into config mode |
L2 MAC VRF Configuration:
(config)#mac vrf L2VRF1 | Create mac routing/forwarding instance with L2VRF1 name and enter into vrf mode |
(config-vrf)#rd 1.1.1.1:11 | Assign RD value |
(config-vrf)#description MAC VRF RED | Give description to L2VRF1 as RED |
(config-vrf)#route-target both 9.9.9.9:100 | Assign route-target value for same for import and export. Should be same on all node for L2VRF1 |
(config-vrf)#exit | Exit from vrf mode |
(config)#mac vrf L2VRF2 | Create mac routing/forwarding instance with L2VRF2 name and enter into vrf mode |
(config-vrf)#rd 1.1.1.1:21 | Assign RD value |
(config-vrf)#route-target both 90.90.90.90:100 | Assign route-target value for same for import and export |
(config-vrf)#description MAC VRF BLUE | Give description to L2VRF2 as BLUE |
(config-vrf)#exit | Exit from vrf mode |
L2 VXLAN configuration:
(config)#nvo vxlan enable | Enable VXLAN |
(config)#evpn esi hold-time 90 | Configure ESI hold time to allow tunnel to come up at the time of vxlan initialization before making the ESI up. It should be same on both VTEP1 and VTEP2 |
(config)#nvo vxlan vtep-ip-global 1.1.1.1 | Configure Source vtep-ip-global configuration - Use loopback ip address |
(config)#nvo vxlan id 101 ingress-replication inner-vid-disabled | Configure VXLAN Network identifier with/without inner-vid- disabled configure and enter into VXLAN tenant mode |
(config-nvo)#vxlan host-reachability-protocol evpn-bgp L2VRF1 | Assign vrf for evpn-bgp to carry EVPN route |
(config-nvo)# vni-name VNI-101 | Configure VNI name as VNI-101 |
(config-nvo)#exit | Exit from VXLAN tenant mode and enter into configuration mode. |
(config)#nvo vxlan id 201 ingress-replication inner-vid-disabled | Configure VXLAN Network identifier with/without inner-vid- disabled configure and enter into VXLAN tenant mode |
(config-nvo)#vxlan host-reachability-protocol evpn-bgp L2VRF2 | Assign vrf for evpn-bgp to carry EVPN route |
(config-nvo)# vni-name VNI-201 | Configure VNI name as VNI-201 |
(config-nvo)#exit | Exit from VXLAN tenant mode and enter into configuration mode. |
(config)#nvo vxlan access-if port-vlan po1 10 | Enable port-vlan mapping i.e. access port to outer-vlan (SVLAN) - Multihomed access port |
(config-nvo-acc-if)#map vnid 101 | Map VXLAN Identified to access-port for VXLAN |
(config-nvo-acc-if)# mac 0000.2222.1010 ip 11.11.11.51 | Configure static mac-ip |
(config-nvo-acc-if)#exit | Exit from VXLAN access-interface mode and enter into configuration mode |
(config)#nvo vxlan access-if port-vlan po1 20 | Enable port-vlan mapping i.e. access port to outer-vlan (SVLAN) - Multihomed access port |
(config-nvo-acc-if)#map vnid 201 | Map VXLAN Identified to access-port for VXLAN |
(config-nvo-acc-if)# mac 0000.2222.1020 ip 21.21.21.51 | Configure static mac-ip |
(config-nvo-acc-if)#commit | Commit the candidate configuration to the running configuration |
(config-nvo-acc-if)#exit | Exit from VXLAN access-interface mode and enter into configuration mode |
(config)#exit | Exit from configuration mode |
VTEP2
(Multi-homed group1) - Part of both Multi-homed with po1(MH1).
Hardware profile and generic configuration:
#configure terminal | Enter Configure mode. |
(config)#hardware-profile filter vxlan enable | Enable hardware-profile filter for VXLAN. |
(config)#hardware-profile filter vxlan-mh enable | Enable hardware-profile filter for VXLAN multi-homing. |
(config)#hardware-profile filter egress-ipv4 enable | Enable hardware-profile filter for egress IPv4. |
(config)#evpn vxlan multihoming enable | Enable Multihoming, save configs and reboot the board for multihoming to be effective |
(config)#hardware-profile statistics ac-lif enable | Enable ac-lif for vxlan access-if port counters |
(config)#qos enable | Enabling qos |
Interface and loopback configuration:
(config)#interface po1 | Enter Interface mode for po1 (MH2) |
(config-if)#switchport | Make it L2 interface |
(config-if)# evpn multi-homed system-mac 0000.0000.2222 | Configure system mac as ESI value for Lag (po1) interface. VTEP1 and VTEP2 should have same ESI value |
(config-if)#exit | Exit Interface mode and return to Configure mode. |
(config)#interface xe2 | Enter Interface mode for xe2 |
(config-if)#channel-group 1 mode active | Make it member port of po1 |
(config-if)#exit | Exit Interface mode and return to Configure mode. |
(config)#interface lo | Enter Interface mode for lo |
(config-if)#ip address 2.2.2.2/32 secondary | Configure loopback ip address as 2.2.2.2 for VTEP2 |
(config-if)#exit | Exit Interface mode and return to Configure mode. |
(config) interface xe29 | Enter into network side interface |
(config-if)#ip address 100.12.12.1/24 | Configure ip address as 100.12.12.1 on network side of Spine-P3 |
(config-if)#exit | Exit Interface mode and return to Configure mode. |
OSPF configuration:
(config)#router ospf 100 | Enter into router OSPF mode |
(config-router)#ospf router-id 2.2.2.2 | Configure router-id as 2.2.2.2 (lo ip address) |
(config-router)#network 2.2.2.2/32 area 0.0.0.0 | Add 2.2.2.2 (lo ip address) network into area 0 |
(config-router)#network 100.12.12.0/24 area 0.0.0.0 | Add 100.12.12.0(Spine-P3) network into area 0 |
(config-router)#bfd all-interfaces | Enabling bfd on all ospf interface for fast convergence |
(config-if)#exit | Exit Interface mode and return to Configure mode. |
BGP configuration:
(config)#router bgp 5000 | Enter into Router BGP mode |
(config-router)#bgp router-id 2.2.2.2 | Configure router-id as 2.2.2.2 (lo ip address) |
(config-router)#neighbor 1.1.1.1 remote-as 5000 | Specify a VTEP1 loopback ip address and remote-as defined |
(config-router)#neighbor 1.1.1.1 update-source lo | Configure update as loopback for VTEP1 |
(config-router)#neighbor 1.1.1.1 advertisement-interval 0 | Configure advertisement-interval as 0 for fast convergence for VTEP1 |
(config-router)#neighbor 4.4.4.4 remote-as 5000 | Specify a VTEP4 loopback ip address and remote-as defined |
(config-router)#neighbor 4.4.4.4 update-source lo | Configure update as loopback for VTEP4 |
(config-router)#neighbor 4.4.4.4 advertisement-interval 0 | Configure advertisement-interval as 0 for fast convergence for VTEP4 |
(config-router)#neighbor 5.5.5.5 remote-as 5000 | Specify a VTEP5 loopback ip address and remote-as defined |
(config-router)#neighbor 5.5.5.5 update-source lo | Configure update as loopback for VTEP5 |
(config-router)#neighbor 5.5.5.5 advertisement-interval 0 | Configure advertisement-interval as 0 for fast convergence for VTEP5 |
(config-router)#address-family l2vpn evpn | Enter into l2vpn EVPN address family mode |
(config-router-af)#neighbor 1.1.1.1 activate | Activate 1.1.1.1(VTEP1) into l2vpn evpn address family mode |
(config-router-af)#neighbor 4.4.4.4 activate | Activate 4.4.4.4(VTEP4) into l2vpn evpn address family mode |
(config-router-af)#neighbor 5.5.5.5 activate | Activate 5.5.5.5(VTEP5) into l2vpn evpn address family mode |
(config-router-af)#exit-address-family | Exit from l2vpn address family mode |
(config-router)#exit | Exit from Router BGP mode and enter into config mode |
VRF Configuration:
(config)#mac vrf L2VRF1 | Create mac routing/forwarding instance with L2VRF1 name and enter into vrf mode |
(config-vrf)#rd 2.2.2.2:11 | Assign RD value |
(config-vrf)#description MAC VRF RED | Give description to L2VRF1 as RED |
(config-vrf)#route-target both 9.9.9.9:100 | Assign route-target value for same for import and export. Should be same on all node for L2VRF1 |
(config-vrf)#exit | Exit from vrf mode |
(config)#mac vrf L2VRF2 | Create mac routing/forwarding instance with L2VRF2 name and enter into vrf mode |
(config-vrf)#rd 2.2.2.2:21 | Assign RD value |
(config-vrf)#route-target both 90.90.90.90:100 | Assign route-target value for same for import and export |
(config-vrf)#description MAC VRF BLUE | Give description to L2VRF2 as BLUE |
(config-vrf)#exit | Exit from vrf mode |
VXLAN configuration:
(config)#nvo vxlan enable | Enable VXLAN |
(config)#evpn esi hold-time 90 | Configure ESI hold time to allow tunnel to come up at the time of vxlan initialization before making the ESI up.It should be same on both VTEP1 and VTEP2 |
(config)#nvo vxlan vtep-ip-global 2.2.2.2 | Configure Source vtep-ip-global configuration - Use loopback ip address |
(config)#nvo vxlan id 101 ingress-replication inner-vid-disabled | Configure VXLAN Network identifier with/without inner-vid- disabled configure and enter into VXLAN tenant mode |
(config-nvo)#vxlan host-reachability-protocol evpn-bgp L2VRF1 | Assign vrf for evpn-bgp to carry EVPN route |
(config-nvo)# vni-name VNI-101 | Configure VNI name as VNI-101 |
(config-nvo)#exit | Exit from VXLAN tenant mode and enter into configuration mode. |
(config)#nvo vxlan id 201 ingress-replication inner-vid-disabled | Configure VXLAN Network identifier with/without inner-vid- disabled configure and enter into VXLAN tenant mode |
(config-nvo)#vxlan host-reachability-protocol evpn-bgp L2VRF2 | Assign vrf for evpn-bgp to carry EVPN route |
(config-nvo)# vni-name VNI-201 | Configure VNI name as VNI-201 |
(config-nvo)#exit | Exit from VXLAN tenant mode and enter into configuration mode. |
(config)#nvo vxlan access-if port-vlan po1 10 | Enable port-vlan mapping i.e. access port to outer-vlan (SVLAN) - Multihomed access port |
(config-nvo-acc-if)#map vnid 101 | Map VXLAN Identified to access-port for VXLAN |
(config-nvo-acc-if)# mac 0000.2222.1010 ip 11.11.11.51 | Configure static mac-ip |
(config-nvo-acc-if)#exit | Exit from VXLAN access-interface mode and enter into configuration mode |
(config)#nvo vxlan access-if port-vlan po1 20 | Enable port-vlan mapping i.e. access port to outer-vlan (SVLAN) - Multihomed access port |
(config-nvo-acc-if)#map vnid 201 | Map VXLAN Identified to access-port for VXLAN |
(config-nvo-acc-if)# mac 0000.2222.1020 ip 21.21.21.51 | Configure static mac-ip |
(config-nvo-acc-if)#commit | Commit the candidate configuration to the running configuration |
(config-nvo-acc-if)#exit | Exit from VXLAN access-interface mode and enter into configuration mode |
(config)#exit | Exit from configuration mode |
VTEP4
Single Home -SH5.
Hardware profile and generic configuration:
#configure terminal | Enter Configure mode. |
(config)#hardware-profile filter vxlan enable | Enable hardware-profile filter for VXLAN. |
(config)#hardware-profile filter vxlan-mh enable | Enable hardware profile mh in SH VTEP also |
(config)#hardware-profile filter egress-ipv4 enable | Enable hardware-profile filter for egress IPv4. |
(config)#hardware-profile statistics ac-lif enable | Enable ac-lif for vxlan access-if port counters |
(config)#evpn vxlan multihoming enable | Enable Multihoming, save configs and reboot the board for multihoming to be effective |
(config)#qos enable | Enabling qos |
Interface and loopback configuration:
(config)#interface xe3 | Enter Interface mode for xe3 |
(config-if)#switchport | Make it L2 interface |
(config-if)#exit | Exit Interface mode and return to Configure mode. |
(config)#interface lo | Enter Interface mode for lo |
(config-if)#ip address 4.4.4.4/32 secondary | Configure loopback ip address as 4.4.4.4 for VTEP4 |
(config-if)#exit | Exit Interface mode and return to Configure mode. |
(config) interface ce49 | Enter interface towards network side |
(config-if)#ip address 100.14.14.1/24 | Configure ip address as 100.14.14.1 on network side of Spine-P3 |
(config-if)#exit | Exit Interface mode and return to Configure mode. |
OSPF configuration:
(config)#router ospf 100 | Enter into router OSPF mode |
(config-router)#ospf router-id 4.4.4.4 | Configure router-id as 4.4.4.4 (lo ip address) |
(config-router)#network 4.4.4.4/32 area 0.0.0.0 | Add 4.4.4.4 (lo ip address) network into area 0 |
(config-router)#network 100.14.14.0/24 area 0.0.0.0 | Add 100.14.14.0(Spine-P3) network into area 0 |
(config-router)#bfd all-interfaces | Enabling bfd on all ospf interface for fast convergence |
(config-if)#exit | Exit Interface mode and return to Configure mode. |
BGP configuration:
(config)#router bgp 5000 | Enter into Router BGP mode |
(config-router)#bgp router-id 4.4.4.4 | Configure router-id as 4.4.4.4 (lo ip address) |
(config-router)#neighbor 1.1.1.1 remote-as 5000 | Specify a VTEP1 loopback ip address and remote-as defined |
(config-router)#neighbor 1.1.1.1 update-source lo | Configure update as loopback for VTEP1 |
(config-router)#neighbor 1.1.1.1 advertisement-interval 0 | Configure advertisement-interval as 0 for fast convergence for VTEP1 |
(config-router)#neighbor 2.2.2.2 remote-as 5000 | Specify a VTEP2 loopback ip address and remote-as defined |
(config-router)#neighbor 2.2.2.2 update-source lo | Configure update as loopback for VTEP2 |
(config-router)#neighbor 2.2.2.2 advertisement-interval 0 | Configure advertisement-interval as 0 for fast convergence for VTEP2 |
(config-router)#neighbor 5.5.5.5 remote-as 5000 | Specify a VTEP5 loopback ip address and remote-as defined |
(config-router)#neighbor 5.5.5.5 update-source lo | Configure update as loopback for VTEP5 |
(config-router)#neighbor 5.5.5.5 advertisement-interval 0 | Configure advertisement-interval as 0 for fast convergence for VTEP5 |
(config-router)#address-family l2vpn evpn | Enter into l2vpn EVPN address family mode |
(config-router-af)#neighbor 1.1.1.1 activate | Activate 1.1.1.1(VTEP1) into l2vpn evpn address family mode |
(config-router-af)#neighbor 2.2.2.2 activate | Activate 2.2.2.2(VTEP2) into l2vpn evpn address family mode |
(config-router-af)#neighbor 5.5.5.5 activate | Activate 5.5.5.5(VTEP5) into l2vpn evpn address family mode |
(config-router-af)#exit-address-family | Exit from l2vpn address family mode |
(config-router)#exit | Exit from Router BGP mode and enter into config mode |
VRF Configuration:
(config)#mac vrf L2VRF1 | Create mac routing/forwarding instance with L2VRF1 name and enter into vrf mode |
(config-vrf)#rd 4.4.4.4:11 | Assign RD value |
(config-vrf)#description MAC VRF RED | Give description to L2VRF1 as RED |
(config-vrf)#route-target both 9.9.9.9:100 | Assign route-target value for same for import and export. Should be same on all node for L2VRF1 |
(config-vrf)#exit | Exit from vrf mode |
(config)#mac vrf L2VRF2 | Create mac routing/forwarding instance with L2VRF2 name and enter into vrf mode |
(config-vrf)#rd 4.4.4.4:21 | Assign RD value |
(config-vrf)#route-target both 90.90.90.90:100 | Assign route-target value for same for import and export |
(config-vrf)#description MAC VRF BLUE | Give description to L2VRF2 as BLUE |
(config-vrf)#exit | Exit from vrf mode |
VXLAN configuration:
(config)#nvo vxlan enable | Enable VXLAN |
(config)#nvo vxlan vtep-ip-global 4.4.4.4 | Configure Source vtep-ip-global configuration. Use loopback ip address |
(config)#nvo vxlan id 101 ingress-replication inner-vid-disabled | Configure VXLAN Network identifier with/without inner-vid- disabled configure and enter into VXLAN tenant mode |
(config-nvo)#vxlan host-reachability-protocol evpn-bgp L2VRF1 | Assign vrf for evpn-bgp to carry EVPN route |
(config-nvo)# vni-name VNI-101 | Configure VNI name as VNI-101 |
(config-nvo)#exit | Exit from VXLAN tenant mode and enter into configuration mode. |
(config)#nvo vxlan id 201 ingress-replication inner-vid-disabled | Configure VXLAN Network identifier with/without inner-vid- disabled configure and enter into VXLAN tenant mode |
(config-nvo)#vxlan host-reachability-protocol evpn-bgp L2VRF2 | Assign vrf for evpn-bgp to carry EVPN route |
(config-nvo)# vni-name VNI-201 | Configure VNI name as VNI-201 |
(config-nvo)#exit | Exit from VXLAN tenant mode and enter into configuration mode. |
(config)# nvo vxlan access-if port-vlan xe3 20 | Enable port-vlan mapping i.e. access port to outer-vlan (SVLAN) |
(config-nvo-acc-if)#map vnid 201 | Map VXLAN Identified to access-port for VXLAN |
(config-nvo-acc-if)# mac 0000.5555.1020 ip 21.21.21.101 | Configure static mac-ip |
(config-nvo-acc-if)#commit | Commit the candidate configuration to the running configuration |
(config-nvo-acc-if)#exit | Exit from VXLAN access-interface mode and enter into configuration mode |
(config)#exit | Exit from configuration mode |
VTEP5
Single Home -SH3
Hardware profile and generic configuration:
#configure terminal | Enter Configure mode. |
(config)#hardware-profile filter vxlan enable | Enable hardware-profile filter for VXLAN. |
(config)#hardware-profile filter vxlan-mh enable | Enable hardware profile mh in SH VTEP also |
(config)#hardware-profile filter egress-ipv4 enable | Enable hardware-profile filter for egress IPv4. |
(config)#hardware-profile statistics ac-lif enable | Enable ac-lif for vxlan access-if port counters |
(config)#evpn vxlan multihoming enable | Enable Multihoming, save configs and reboot the board for multihoming to be effective |
(config)#qos enable | Enabling qos |
Interface and loopback configuration:
(config)#interface xe1 | Enter Interface mode for xe1 (SH5) |
(config-if)#switchport | Make it L2 interface |
(config-if)#exit | Exit Interface mode and return to Configure mode. |
(config)#interface lo | Enter Interface mode for lo |
(config-if)#ip address 5.5.5.5/32 secondary | Configure loopback ip address as 5.5.5.5 for VTEP5 |
(config-if)#exit | Exit Interface mode and return to Configure mode. |
(config) interface ce53 | Enter interface mode |
(config-if)#ip address 100.15.15.1/24 | Configure ip address as 100.15.15.1 on network side of Spine-P3 |
(config-if)#exit | Exit Interface mode and return to Configure mode. |
OSPF configuration:
(config)#router ospf 100 | Enter into router OSPF mode |
(config-router)#ospf router-id 5.5.5.5 | Configure router-id as 5.5.5.5 (lo ip address) |
(config-router)#network 5.5.5.5/32 area 0.0.0.0 | Add 5.5.5.5 (lo ip address) network into area 0 |
(config-router)#network 100.15.15.0/24 area 0.0.0.0 | Add 100.15.15.0(Spine-P3) network into area 0 |
(config-router)#bfd all-interfaces | Enabling bfd on all ospf interface for fast convergence |
(config-if)#exit | Exit Interface mode and return to Configure mode. |
BGP configuration:
(config)#router bgp 5000 | Enter into Router BGP mode |
(config-router)#bgp router-id 5.5.5.5 | Configure router-id as 5.5.5.5(lo ip address) |
(config-router)#neighbor 1.1.1.1 remote-as 5000 | Specify a VTEP1 loopback ip address and remote-as defined |
(config-router)#neighbor 1.1.1.1 update-source lo | Configure update as loopback for VTEP1 |
(config-router)#neighbor 1.1.1.1 advertisement-interval 0 | Configure advertisement-interval as 0 for fast convergence for VTEP1 |
(config-router)#neighbor 2.2.2.2 remote-as 5000 | Specify a VTEP2 loopback ip address and remote-as defined |
(config-router)#neighbor 2.2.2.2 update-source lo | Configure update as loopback for VTEP2 |
(config-router)#neighbor 2.2.2.2 advertisement-interval 0 | Configure advertisement-interval as 0 for fast convergence for VTEP2 |
(config-router)#neighbor 4.4.4.4 remote-as 5000 | Specify a VTEP4 loopback ip address and remote-as defined |
(config-router)#neighbor 4.4.4.4 update-source lo | Configure update as loopback for VTEP4 |
(config-router)#neighbor 4.4.4.4 advertisement-interval 0 | Configure advertisement-interval as 0 for fast convergence for VTEP4 |
(config-router)#address-family l2vpn evpn | Enter into l2vpn EVPN address family mode |
(config-router-af)#neighbor 1.1.1.1 activate | Activate 1.1.1.1(VTEP1) into l2vpn evpn address family mode |
(config-router-af)#neighbor 2.2.2.2 activate | Activate 2.2.2.2(VTEP2) into l2vpn evpn address family mode |
(config-router-af)#neighbor 4.4.4.4 activate | Activate 4.4.4.4(VTEP4) into l2vpn evpn address family mode |
(config-router-af)#exit-address-family | Exit from l2vpn address family mode |
(config-router)#exit | Exit from Router BGP mode and enter into config mode |
VRF Configuration:
(config)#mac vrf L2VRF1 | Create mac routing/forwarding instance with L2VRF1 name and enter into vrf mode |
(config-vrf)#rd 5.5.5.5:11 | Assign RD value |
(config-vrf)#description MAC VRF RED | Give description to L2VRF1 as RED |
(config-vrf)#route-target both 9.9.9.9:100 | Assign route-target value for same for import and export. Should be same on all node for L2VRF1 |
(config-vrf)#exit | Exit from vrf mode |
(config)#mac vrf L2VRF2 | Create mac routing/forwarding instance with L2VRF2 name and enter into vrf mode |
(config-vrf)#rd 5.5.5.5:21 | Assign RD value |
(config-vrf)#route-target both 90.90.90.90:100 | Assign route-target value for same for import and export |
(config-vrf)#description MAC VRF BLUE | Give description to L2VRF2 as BLUE |
(config-vrf)#exit | Exit from vrf mode |
VXLAN configuration:
(config)#nvo vxlan enable | Enable VXLAN |
(config)#nvo vxlan vtep-ip-global 5.5.5.5 | Configure Source vtep-ip-global configuration. Use loopback ip address |
(config)#nvo vxlan id 101 ingress-replication inner-vid-disabled | Configure VXLAN Network identifier with/without inner-vid- disabled configure and enter into VXLAN tenant mode |
(config-nvo)#vxlan host-reachability-protocol evpn-bgp L2VRF1 | Assign vrf for evpn-bgp to carry EVPN route |
(config-nvo)# vni-name VNI-101 | Configure VNI name as VNI-101 |
(config-nvo)#exit | Exit from VXLAN tenant mode and enter into configuration mode. |
(config)#nvo vxlan id 201 ingress-replication inner-vid-disabled | Configure VXLAN Network identifier with/without inner-vid- disabled configure and enter into VXLAN tenant mode |
(config-nvo)#vxlan host-reachability-protocol evpn-bgp L2VRF2 | Assign vrf for evpn-bgp to carry EVPN route |
(config-nvo)# vni-name VNI-201 | Configure VNI name as VNI-201 |
(config-nvo)#exit | Exit from VXLAN tenant mode and enter into configuration mode. |
(config)# nvo vxlan access-if port-vlan xe1 10 | Enable port-vlan mapping i.e. access port to outer-vlan (SVLAN) |
(config-nvo-acc-if)#map vnid 101 | Map VXLAN Identified to access-port for VXLAN |
(config-nvo-acc-if)# mac 0000.4444.1010 ip 11.11.11.201 | Configure static mac-ip |
(config-nvo-acc-if)#commit | Commit the candidate configuration to the running configuration |
(config-nvo-acc-if)#exit | Exit from VXLAN access-interface mode and enter into configuration mode |
(config)#exit | Exit from configuration mode |
Switch1 (MH2)
Multihomed to 2-VTEPs (VTEP1 and VTEP2). It acts as Tenant system for vlan1.20.
#configure terminal | Enter Configure mode. |
(config)# bridge 1 protocol rstp vlan-bridge | Configure rstp vlan bridge |
(config)# vlan database | Enter vlan database config mode |
(config)#vlan 2-20 bridge 1 state enable | Configure vlans from 2-20 and associate with bridge 1 |
(config)#interface xe5 | Enter Interface mode for xe5 which is connected to TG |
(config-if)#switchport | Make as L2 port by configuring switchport |
(config-if)#bridge-group 1 | Associate bridge 1 into interface |
(config-if)# bridge-group 1 spanning-tree disable | Configure interface as stp disable |
(config-if)# switchport mode trunk | Mode as trunk |
(config-if)# switchport trunk allowed vlan add 10,20 | Trunk allowed vlan as 10.20 |
(config-if)#exit | Exit Interface mode and return to Configure mode. |
(config)#interface po1 | Enter Interface mode for po1 |
(config-if)#switchport | Make po1 as L2 port by configuring switchport |
(config-if)#bridge-group 1 | Associate po1 to bridge 1 |
(config-if)# bridge-group 1 spanning-tree disable | Configure po1 as stp disable |
(config-if)# switchport mode trunk | Mode as trunk |
(config-if)# switchport trunk allowed vlan add 10,20 | Trunk allowed vlan as 2.10.20 |
(config-if)#exit | Exit Interface mode and return to Configure mode. |
(config)#interface xe3 | Enter Interface mode for xe3 |
(config-if)#channel-group 1 mode active | Make it member port of po1 |
(config)#interface xe2 | Enter Interface mode for xe2 |
(config-if)#channel-group 1 mode active | Make it member port of po1 |
(config-if)#commit | Commit the candidate configuration to the running configuration |
(config-if)#exit | Exit from configuration mode |
Spine-P3
Spine node where all VTEPs are connected.
Generic configuration:
#configure terminal | Enter Configure mode. |
(Config)#qos enable | Enabling qos |
Interface and loopback configuration:
(config)#interface lo | Enter Interface mode for lo |
(config-if)#ip address 100.100.100.100/32 secondary | Configure loopback ip address as 100.100.100.100 for Spine-P3 |
(config-if)#exit | Exit Interface mode and return to Configure mode. |
(config)#interface ce16/1 | Enter Interface mode for ce16/1 |
(config-if)#ip address 100.11.11.2/24 | Configure ip address as 100.11.11.2 on network side of VTEP1 |
(config-if)#exit | Exit Interface mode and return to Configure mode. |
(config) interface ce5/1 | Enter into ce5/1 interface mode |
(config-if)#ip address 100.12.12.2/24 | Configure ip address as 100.12.12.2 on network side of VTEP2 |
(config-if)#exit | Exit Interface mode and return to Configure mode. |
(config) interface ce9/1 | Enter ce9/1 interface mode |
(config-if)#ip address 100.14.14.2/24 | Configure ip address as 100.14.14.12 on network side of VTEP4 |
(config-if)#exit | Exit Interface mode and return to Configure mode. |
(config) interface ce23/1 | Enter interface mode |
(config-if)#ip address 100.15.15.1/24 | Configure ip address as 100.15.15.1 on network side of VTEP5 |
(config-if)#exit | Exit Interface mode and return to Configure mode. |
OSPF configuration:
(config)#router ospf 100 | Enter into router OSPF mode |
(config-router)#ospf router-id 100.100.100.100 | Configure router-id as 100.100.100.100 (lo ip address) |
(config-router)#network 100.100.100.100/32 area 0.0.0.0 | Add 100.100.100.100 (lo ip address) network into area 0 |
(config-router)#network 100.11.11.0/24 area 0.0.0.0 | Add 100.11.11.0 (VTEP1) network into area 0 |
(config-router)#network 100.12.12.0/24 area 0.0.0.0 | Add 100.12.12.0 (VTEP2) network into area 0 |
(config-router)#network 100.14.14.0/24 area 0.0.0.0 | Add 100.14.14.0 (VTEP4) network into area 0 |
(config-router)#network 100.15.15.0/24 area 0.0.0.0 | Add 100.15.15.0 (VTEP5) network into area 0 |
(config-router)#bfd all-interfaces | Enabling bfd on all ospf interface for fast convergence |
(config-router)#commit | Commit the candidate configuration to the running configuration |
(config-router)#exit | Exit Interface mode and return to Configure mode. |
Centralized Gateway
In Centralized gateway approach, when two TS belonging to two different subnets connected to the same/different VTEP node, wanted to communicate with each other, their traffic needed to be back hauled from the VTEP node to the centralized gateway node where inter- subnet switching is performed and then back to the VTEP node.
IRB Configuration for Centralized Gateway
Configure from Base Configuration-L2 VXLAN section, then configure below commands for centralized gateway approach. Here VTEP4 is the centralized GW. In VTEP4, dscp-to-queue qos profile should be applied on the particular incoming L2VNID IRB interface and dscp-encap qos profile should be applied on vxlan tunnel egress.
Note: For L3 traffic, when L2VNID is sent in the traffic, then dscp-to-queue qos profile mapped at IRB interface of that particular L2VNID takes effect.
VTEP1
(config)#qos profile cos-to-queue COS_QUEUE | Create QoS profile for mapping traffic towards tunnel from access-if |
(config-ingress-cos-map)#cos 1 queue 5 | Configure particular COS value to queue value for the profile |
(config-ingress-cos-map)#exit | Exit from qos profile mode |
(config)#qos profile queue-color-to-dscp QUEUE_DSCP | Create QoS profile for attaching in vxlan tunnel egress |
(config-egress-dscp-map)#queue 5 dscp 34 | Configure queue value to DSCP value for the profile |
(config)#nvo vxlan tunnel qos-map-mode cos-dscp egress QUEUE_DSCP | Map the configured QoS profile to vxlan tunnel egress |
(config)#nvo vxlan access-if port-vlan po1 20 | Enter into vxlan access port mode |
(config-nvo-acc-if)#map qos-profile cos-to-queue COS_QUEUE | Map the qos profile in vxlan access-if |
(config-nvo-acc-if)#commit | Commit the candidate configuration to the running configuration |
(config-nvo-acc-if)#end | Exit from vxlan access port |
VTEP2
(config)#qos profile cos-to-queue COS_QUEUE | Create QoS profile for mapping traffic towards tunnel from access-if |
(config-ingress-cos-map)#cos 1 queue 5 | Configure particular COS value to queue value for the profile |
(config-ingress-cos-map)#exit | Exit from qos profile mode |
(config)#qos profile queue-color-to-dscp QUEUE_DSCP | Create QoS profile for attaching in vxlan tunnel egress |
(config-egress-dscp-map)#queue 5 dscp 34 | Configure queue value to DSCP value for the profile |
(config)#nvo vxlan tunnel qos-map-mode cos-dscp egress QUEUE_DSCP | Map the configured QoS profile to vxlan tunnel egress |
(config)#nvo vxlan access-if port-vlan po1 20 | Enter into vxlan access port mode |
(config-nvo-acc-if)#map qos-profile cos-to-queue COS_QUEUE | Map the qos profile in vxlan access-if |
(config-nvo-acc-if)#commit | Commit the candidate configuration to the running configuration |
(config-nvo-acc-if)#end | Exit from vxlan access port |
VTEP4
(config)#nvo vxlan irb | Enable VXLAN irb |
(config)#ip vrf L3VRF1 | Create mac routing/forwarding instance with L3VRF1 name and enter into vrf mode |
(config-vrf)#rd 51000:11 | Assign RD value |
(config-vrf)# route-target both 100:100 | Assign route-target value for same for import and export. |
(config-vrf)# l3vni 1000 | Configure L3VNI as 1000 for L3VRF1 |
(config-vrf)#exit | Exit from vrf mode |
(config)# interface irb1001 | Configure IRB interface 1001 |
(config-if)ip vrf forwarding L3VRF1 | Configure L3VRF1 |
(config-if)ip address 11.11.11.1/24 | Configure ip address |
(config-if)ipv6 address 1111::1/64 | Configure ipv6 address |
(config-if)exit | Exit from interface config mode |
(config)# interface irb2001 | Configure IRB interface 2001 |
(config-if)ip vrf forwarding L3VRF1 | Configure L3VRF1 |
(config-if)ip address 21.21.21.1/24 | Configure ip address |
(config-if)ipv6 address 2121::1/64 | Configure ipv6 address |
(config-if)exit | Exit from interface config mode |
(config)# nvo vxlan id 101 ingress-replication inner-vid-disabled | Configure VXLAN Network identifier with/without inner-vid- disabled configure and enter into VXLAN tenant mode |
(config-nvo)#vxlan host-reachability-protocol evpn-bgp L2VRF1 | Assign vrf for evpn-bgp to carry EVPN route |
(config-nvo)# evpn irb1001 | Configure irb1001 under vxlan id 101 |
(config-nvo)#exit | Exit from VXLAN tenant mode and enter into configuration mode. |
(config)#nvo vxlan id 201 ingress-replication inner-vid-disabled | Configure VXLAN Network identifier with/without inner-vid- disabled configure and enter into VXLAN tenant mode |
(config-nvo)#vxlan host-reachability-protocol evpn-bgp L2VRF2 | Assign vrf for evpn-bgp to carry EVPN route |
(config-nvo)# evpn irb2001 | Configure irb2001 under vxlan id 201 |
(config-nvo)#exit | Exit from VXLAN tenant mode and enter into configuration mode. |
(config)#qos profile dscp-to-queue DSCP_QUEUE | Configure the QoS profile to match the inner dscp value of the traffic at the IRB interface |
(config-ingress-dscp-map)#dscp 20 queue 1 | Configure particular dscp to a queue value. Configure particular dscp to a queue value. Here classification at the IRB L3 interface is based on customer dscp value. |
(config-ingress-dscp-map)#exit | Exit from qos profile config mode |
(config)#qos profile dscp-encap DSCP_ENCAP | Configure the QoS profile to remark the overlay dscp value of the traffic in vxlan tunnel egress. |
(config-egress-dscp-encap-map)#l3 dscp 20 dscpEncap 56 | Egress remarking of the customer dscp packet to overlay dscp. Here classification at the egress vtep is based on customer dscp value |
(config-egress-dscp-encap-map)#exit | Exit from qos profile mode |
config)#int irb2001 | Enter IRB L3 interface mode |
(config-irb-if)#qos map-profile dscp-to-queue DSCP_QUEUE | Map the qos profile in the IRB interface |
(config-irb-if)#exit | Exit from interface mode |
(config)#nvo vxlan tunnel qos-map-mode cos-dscp egress DSCP_ENCAP | Map the qos profile in vxlan tunnel egress |
(config)#commit | Commit the candidate configuration to the running configuration |
(config)#end | Exit from global config mode |
VTEP5
(config)#qos profile queue-color-to-cos QUEUE_COS | Create QoS profile for remark the queue value to COS value |
VTEP5(config-egress-cos-map)#queue 2 cos 5 | Configure particular queue value to COS value for the profile |
(config-ingress-cos-map)#exit | Exit from qos profile mode |
(config)# qos profile dscp-to-queue DSCP_QUEUE | Create QoS profile for attaching in vxlan tunnel ingress |
(config-ingress-dscp-map)#dscp 56 queue 2 | Configure DSCP value to queue value for the profile |
(config)#nvo vxlan tunnel qos-map-mode cos-dscp ingress DSCP_QUEUE | Map the configured QoS profile to vxlan tunnel ingress |
(config)#nvo vxlan access-if port-vlan xe1 10 | Enter into vxlan access port mode |
(config-nvo-acc-if)#map qos-profile queue-color-to-cos QUEUE_COS | Map the qos profile in vxlan access-if |
(config-nvo-acc-if)#commit | Commit the candidate configuration to the running configuration |
(config-nvo-acc-if)#end | Exit from vxlan access port |
Validation
Send traffic from TS2-21 to MH2 access-if with dscp value 20 and COS value 1(vlan20) and verify traffic reaceived at TS1-11 with dscp value 32 and COS value 5(vlan10) at the VTEP5 access-if.
VTEP1
VTEP1#show running-config qos
qos enable
!
qos profile cos-to-queue COS_QUEUE
cos 1 queue 5
!
qos profile queue-color-to-dscp QUEUE_DSCP
queue 5 color all dscp 34
!
!
!
VTEP1#show running-config nvo vxlan
!
nvo vxlan enable
!
evpn vxlan multihoming enable
!
nvo vxlan vtep-ip-global 1.1.1.1
!
nvo vxlan tunnel qos-map-mode cos-dscp egress QUEUE_DSCP
!
nvo vxlan id 101 ingress-replication inner-vid-disabled
vxlan host-reachability-protocol evpn-bgp L2VRF1
vni-name VNI-101
!
nvo vxlan id 201 ingress-replication inner-vid-disabled
vxlan host-reachability-protocol evpn-bgp L2VRF2
vni-name VNI-201
!
nvo vxlan access-if port-vlan po1 10
map vnid 101
mac 0000.2222.1010 ip 11.11.11.51
!
nvo vxlan access-if port-vlan po1 20
map vnid 201
mac 0000.2222.1020 ip 21.21.21.51
map qos-profile cos-to-queue COS_QUEUE
!
!
VTEP1#show nvo vxlan tunnel
VXLAN Network tunnel Entries
Source Destination Status Up/Down Update
============================================================================
1.1.1.1 5.5.5.5 Installed 01:15:13 01:15:13
1.1.1.1 4.4.4.4 Installed 01:15:28 01:15:28
1.1.1.1 2.2.2.2 Installed 01:11:40 01:11:40
Total number of entries are 3
VTEP1#show interface ce52 counters queue-stats
E - Egress, I - Ingress, Q-Size is in bytes
+--------------------+--------+-----------------+-------------------+-----------------+-------------------+
| Queue/Class-map | Q-Size | Tx pkts | Tx bytes | Dropped pkts | Dropped bytes |
+--------------------+--------+-----------------+-------------------+-----------------+-------------------+
q0 (E) 125304832 0 0 0 0
q1 (E) 125304832 0 0 0 0
q2 (E) 125304832 0 0 0 0
q3 (E) 125304832 0 0 0 0
q4 (E) 125304832 0 0 0 0
q5 (E) 125304832 1316880 1316879000 0 0
q6 (E) 125304832 0 0 0 0
q7 (E) 125304832 0 0 0 0
VTEP1#show qos-profile COS_QUEUE
profile name: COS_QUEUE
profile type: cos-to-queue
profile attached to 1 instances
configured mapping:
cos 1 queue 5
Detailed mapping:
---------------+----------------- | ---------------+-----------------
INPUT | OUTPUT | INPUT | OUTPUT
---------------+----------------- | ---------------+-----------------
COS | DEI | Queue | Color | COS | DEI | Queue | Color
-------+-------+-------+--------- | -------+-------+-------+---------
0 0 0 green | 0 1 0 yellow
1 0 5 green | 1 1 5 yellow
2 0 2 green | 2 1 2 yellow
3 0 3 green | 3 1 3 yellow
4 0 4 green | 4 1 4 yellow
5 0 5 green | 5 1 5 yellow
6 0 6 green | 6 1 6 yellow
7 0 7 green | 7 1 7 yellow
VTEP1#show qos-profile QUEUE_DSCP
profile name: QUEUE_DSCP
profile type: queue-color-to-dscp
profile attached to 1 instances
configured mapping:
queue 5 color all dscp 34
Detailed mapping:
----------------+-------- | ----------------+-------- | ----------------+--------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
----------------+-------- | ----------------+-------- | ----------------+--------
Queue | Color | DSCP | Queue | Color | DSCP | Queue | Color | DSCP
-------+--------+-------- | -------+--------+-------- | -------+--------+--------
0 green 0 | 0 yellow 0 | 0 red 0
1 green 10 | 1 yellow 12 | 1 red 14
2 green 18 | 2 yellow 20 | 2 red 22
3 green 26 | 3 yellow 28 | 3 red 30
4 green 34 | 4 yellow 36 | 4 red 38
5 green 34 | 5 yellow 34 | 5 red 34
6 green 48 | 6 yellow 48 | 6 red 48
7 green 56 | 7 yellow 56 | 7 red 56
VTEP4
VTEP4#show running-config qos
qos enable
!
qos profile dscp-to-queue DSCP_QUEUE
dscp 20 queue 1
!
qos profile dscp-encap DSCP_ENCAP
l3 dscp 20 dscpEncap 56
!
!
!
!
interface irb2001
qos map-profile dscp-to-queue DSCP_QUEUE
!
VTEP4#show running-config nvo vxlan
!
nvo vxlan enable
!
nvo vxlan irb
!
evpn vxlan multihoming enable
!
nvo vxlan vtep-ip-global 4.4.4.4
!
nvo vxlan tunnel qos-map-mode cos-dscp egress DSCP_ENCAP
!
nvo vxlan id 101 ingress-replication inner-vid-disabled
vxlan host-reachability-protocol evpn-bgp L2VRF1
evpn irb1001
vni-name VNI-101
!
nvo vxlan id 201 ingress-replication inner-vid-disabled
vxlan host-reachability-protocol evpn-bgp L2VRF2
evpn irb2001
vni-name VNI-201
!
nvo vxlan access-if port-vlan xe3 20
map vnid 201
mac 0000.5555.1020 ip 21.21.21.101
!
!
VTEP4#show nvo vxlan tunnel
VXLAN Network tunnel Entries
Source Destination Status Up/Down Update
============================================================================
4.4.4.4 2.2.2.2 Installed 00:08:40 00:08:40
4.4.4.4 1.1.1.1 Installed 00:12:28 00:12:28
4.4.4.4 5.5.5.5 Installed 00:12:13 00:12:13
Total number of entries are 3
VTEP4#show nvo vxlan
VXLAN Information
=================
Codes: NW - Network Port
AC - Access Port
(u) - Untagged
VNID VNI-Name VNI-Type Type Interface ESI VLAN DF-Status Src-Addr Dst-Addr
_______________________________________________________________________________________________________________________________
101 VNI-101 L2 NW ---- ---- ---- ---- 4.4.4.4 2.2.2.2
101 VNI-101 L2 NW ---- ---- ---- ---- 4.4.4.4 1.1.1.1
101 VNI-101 L2 NW ---- ---- ---- ---- 4.4.4.4 5.5.5.5
201 VNI-201 L2 NW ---- ---- ---- ---- 4.4.4.4 2.2.2.2
201 VNI-201 L2 NW ---- ---- ---- ---- 4.4.4.4 1.1.1.1
201 VNI-201 L2 NW ---- ---- ---- ---- 4.4.4.4 5.5.5.5
201 VNI-201 -- AC xe3 --- Single Homed Port --- 20 ---- ---- ----
Total number of entries are 7
VTEP4#show nvo vxlan arp-cache
VXLAN ARP-CACHE Information
===========================
VNID Ip-Addr Mac-Addr Type Age-Out Retries-Left
____________________________________________________________________________
201 21.21.21.51 0000.2222.1020 Static Remote ----
201 21.21.21.1 3c2c.991a.da7a Static Local ----
201 21.21.21.101 0000.5555.1020 Static Local ----
101 11.11.11.51 0000.2222.1010 Static Remote ----
101 11.11.11.1 3c2c.991a.da7a Static Local ----
101 11.11.11.201 0000.4444.1010 Static Remote ----
Total number of entries are 6
VTEP4#show nvo vxlan nd-cache
VXLAN ND-CACHE Information
===========================
VNID Ip-Addr Mac-Addr Type Age-Out Retries-Left
____________________________________________________________________________________________________
201 2121::1 3c2c.991a.da7a Static Local ----
101 1111::1 3c2c.991a.da7a Static Local ----
Total number of entries are 2
VTEP4#show nvo vxlan l3vni-map
L3VNI L2VNI IRB-interface
===================================
1000 101 irb1001
1000 201 irb2001
VTEP4#show ipv4 route vrf L3VRF1
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2,
ia - IS-IS inter area, E - EVPN,
v - vrf leaked
* - candidate default
IP Route Table for VRF "L3VRF1"
C 11.11.11.0/24 is directly connected, irb1001, 00:01:35
C 21.21.21.0/24 is directly connected, irb2001, 00:01:16
C 127.0.0.0/8 is directly connected, lo.L3VRF1, 00:06:12
Gateway of last resort is not set
VTEP4#show ipv6 route vrf L3VRF1
IPv6 Routing Table
Codes: K - kernel route, C - connected, S - static, D- DHCP, R - RIP,
O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
E2 - OSPF external type 2, E - EVPN N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type 2, i - IS-IS, B - BGP,
v - vrf leaked
Timers: Uptime
IP Route Table for VRF "L3VRF1"
C ::1/128 via ::, lo.L3VRF1, 00:06:29
C 1111::/64 via ::, irb1001, 00:01:52
C 2121::/64 via ::, irb2001, 00:01:33
C fe80::/64 via ::, irb2001, 00:01:33
VTEP4#show interface ce49 counters queue-stats
E - Egress, I - Ingress, Q-Size is in bytes
+--------------------+--------+-----------------+-------------------+-----------------+-------------------+
| Queue/Class-map | Q-Size | Tx pkts | Tx bytes | Dropped pkts | Dropped bytes |
+--------------------+--------+-----------------+-------------------+-----------------+-------------------+
q0 (E) 125304832 0 0 0 0
q1 (E) 125304832 1422755 1488201730 0 0
q2 (E) 125304832 0 0 0 0
q3 (E) 125304832 0 0 0 0
q4 (E) 125304832 0 0 0 0
q5 (E) 125304832 0 0 0 0
q6 (E) 125304832 0 0 0 0
q7 (E) 125304832 0 0 0 0
VTEP4#show qos-profile DSCP_QUEUE
profile name: DSCP_QUEUE
profile type: dscp-to-queue
profile attached to 1 instances
configured mapping:
dscp 20 queue 1
Detailed mapping:
-------+--------------------------- | -------+--------------------------- | -------+--------------------------- | -------+---------------------------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
-------+--------------------------- | -------+--------------------------- | -------+--------------------------- | -------+---------------------------
DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP
-------+-------+--------+---------- | -------+-------+--------+---------- | -------+-------+--------+---------- | -------+-------+--------+----------
0 0 green 0 | 16 2 green 16 | 32 4 green 32 | 48 6 green 48
1 0 green 1 | 17 2 green 17 | 33 4 green 33 | 49 6 green 49
2 0 green 2 | 18 2 green 18 | 34 4 green 34 | 50 6 green 50
3 0 green 3 | 19 2 green 19 | 35 4 green 35 | 51 6 green 51
4 0 green 4 | 20 1 yellow 20 | 36 4 yellow 36 | 52 6 green 52
5 0 green 5 | 21 2 green 21 | 37 4 green 37 | 53 6 green 53
6 0 green 6 | 22 2 yellow 22 | 38 4 yellow 38 | 54 6 green 54
7 0 green 7 | 23 2 green 23 | 39 4 green 39 | 55 6 green 55
8 1 green 8 | 24 3 green 24 | 40 5 green 40 | 56 7 green 56
9 1 green 9 | 25 3 green 25 | 41 5 green 41 | 57 7 green 57
10 1 green 10 | 26 3 green 26 | 42 5 green 42 | 58 7 green 58
11 1 green 11 | 27 3 green 27 | 43 5 green 43 | 59 7 green 59
12 1 yellow 12 | 28 3 yellow 28 | 44 5 green 44 | 60 7 green 60
13 1 green 13 | 29 3 green 29 | 45 5 green 45 | 61 7 green 61
14 1 yellow 14 | 30 3 yellow 30 | 46 5 green 46 | 62 7 green 62
15 1 green 15 | 31 3 green 31 | 47 5 green 47 | 63 7 green 63
VTEP4#show qos-profile DSCP_ENCAP
profile name: DSCP_ENCAP
profile type: dscp-encap
profile attached to 1 instances
configured mapping:
l3 dscp 20 dscpEncap 56
Detailed mapping:
L3 DSCP to DSCP-ENCAP
-------+-------- | -------+-------- | -------+-------- | -------+--------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
-------|-------- | -------|-------- | -------|-------- | -------|--------
DSCP | DSCP | DSCP | DSCP | DSCP | DSCP | DSCP | DSCP
-------+-------- | -------+-------- | -------+-------- | -------+--------
0 0 | 16 16 | 32 32 | 48 48
1 1 | 17 17 | 33 33 | 49 49
2 2 | 18 18 | 34 34 | 50 50
3 3 | 19 19 | 35 35 | 51 51
4 4 | 20 56 | 36 36 | 52 52
5 5 | 21 21 | 37 37 | 53 53
6 6 | 22 22 | 38 38 | 54 54
7 7 | 23 23 | 39 39 | 55 55
8 8 | 24 24 | 40 40 | 56 56
9 9 | 25 25 | 41 41 | 57 57
10 10 | 26 26 | 42 42 | 58 58
11 11 | 27 27 | 43 43 | 59 59
12 12 | 28 28 | 44 44 | 60 60
13 13 | 29 29 | 45 45 | 61 61
14 14 | 30 30 | 46 46 | 62 62
15 15 | 31 31 | 47 47 | 63 63
L2 Queue + Color to DSCP-ENCAP
---------------+--------| ----------------+--------| --------------+--------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
---------------+--------| ----------------+--------| --------------+--------
Queue | Color | DSCP | Queue | Color | DSCP | Queue | Color | DSCP
-------+-------+--------| -------+--------+--------| ------+-------+--------
0 green 0 | 0 yellow 0 | 0 red 0
1 green 8 | 1 yellow 8 | 1 red 8
2 green 16 | 2 yellow 16 | 2 red 16
3 green 24 | 3 yellow 24 | 3 red 24
4 green 32 | 4 yellow 32 | 4 red 32
5 green 40 | 5 yellow 40 | 5 red 40
6 green 48 | 6 yellow 48 | 6 red 48
7 green 56 | 7 yellow 56 | 7 red 56
VTEP4#show qos-profile interface irb2001
profile name: DSCP_QUEUE
profile type: dscp-to-queue (Ingress)
mapping:
-------+--------------------------- | -------+--------------------------- | -------+--------------------------- | -------+---------------------------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
-------+--------------------------- | -------+--------------------------- | -------+--------------------------- | -------+---------------------------
DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP
-------+-------+--------+---------- | -------+-------+--------+---------- | -------+-------+--------+---------- | -------+-------+--------+----------
0 0 green 0 | 16 2 green 16 | 32 4 green 32 | 48 6 green 48
1 0 green 1 | 17 2 green 17 | 33 4 green 33 | 49 6 green 49
2 0 green 2 | 18 2 green 18 | 34 4 green 34 | 50 6 green 50
3 0 green 3 | 19 2 green 19 | 35 4 green 35 | 51 6 green 51
4 0 green 4 | 20 1 yellow 20 | 36 4 yellow 36 | 52 6 green 52
5 0 green 5 | 21 2 green 21 | 37 4 green 37 | 53 6 green 53
6 0 green 6 | 22 2 yellow 22 | 38 4 yellow 38 | 54 6 green 54
7 0 green 7 | 23 2 green 23 | 39 4 green 39 | 55 6 green 55
8 1 green 8 | 24 3 green 24 | 40 5 green 40 | 56 7 green 56
9 1 green 9 | 25 3 green 25 | 41 5 green 41 | 57 7 green 57
10 1 green 10 | 26 3 green 26 | 42 5 green 42 | 58 7 green 58
11 1 green 11 | 27 3 green 27 | 43 5 green 43 | 59 7 green 59
12 1 yellow 12 | 28 3 yellow 28 | 44 5 green 44 | 60 7 green 60
13 1 green 13 | 29 3 green 29 | 45 5 green 45 | 61 7 green 61
14 1 yellow 14 | 30 3 yellow 30 | 46 5 green 46 | 62 7 green 62
15 1 green 15 | 31 3 green 31 | 47 5 green 47 | 63 7 green 63
profile name: default
profile type: dscp-to-dscp (Egress)
Status: Inactive
mapping:
----------------+---------- | ----------------+---------- | ----------------+----------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
----------------+---------- | ----------------+---------- | ----------------+----------
DSCP | Color | Out DSCP | DSCP | Color | Out DSCP | DSCP | Color | Out DSCP
------+---------+---------- | ------+---------+---------- | ------+---------+----------
0 green 0 | 0 yellow 0 | 0 red 0
1 green 1 | 1 yellow 1 | 1 red 1
2 green 2 | 2 yellow 2 | 2 red 2
3 green 3 | 3 yellow 3 | 3 red 3
4 green 4 | 4 yellow 4 | 4 red 4
5 green 5 | 5 yellow 5 | 5 red 5
6 green 6 | 6 yellow 6 | 6 red 6
7 green 7 | 7 yellow 7 | 7 red 7
8 green 8 | 8 yellow 8 | 8 red 8
9 green 9 | 9 yellow 9 | 9 red 9
10 green 10 | 10 yellow 12 | 10 red 14
11 green 11 | 11 yellow 11 | 11 red 11
12 green 12 | 12 yellow 12 | 12 red 14
13 green 13 | 13 yellow 13 | 13 red 13
14 green 14 | 14 yellow 14 | 14 red 14
15 green 15 | 15 yellow 15 | 15 red 15
16 green 16 | 16 yellow 16 | 16 red 16
17 green 17 | 17 yellow 17 | 17 red 17
18 green 18 | 18 yellow 20 | 18 red 22
19 green 19 | 19 yellow 19 | 19 red 19
20 green 20 | 20 yellow 20 | 20 red 22
21 green 21 | 21 yellow 21 | 21 red 21
22 green 22 | 22 yellow 22 | 22 red 22
23 green 23 | 23 yellow 23 | 23 red 23
24 green 24 | 24 yellow 24 | 24 red 24
25 green 25 | 25 yellow 25 | 25 red 25
26 green 26 | 26 yellow 28 | 26 red 30
27 green 27 | 27 yellow 27 | 27 red 27
28 green 28 | 28 yellow 28 | 28 red 30
29 green 29 | 29 yellow 29 | 29 red 29
30 green 30 | 30 yellow 30 | 30 red 30
31 green 31 | 31 yellow 31 | 31 red 31
32 green 32 | 32 yellow 32 | 32 red 32
33 green 33 | 33 yellow 33 | 33 red 33
34 green 34 | 34 yellow 36 | 34 red 38
35 green 35 | 35 yellow 35 | 35 red 35
36 green 36 | 36 yellow 36 | 36 red 38
37 green 37 | 37 yellow 37 | 37 red 37
38 green 38 | 38 yellow 38 | 38 red 38
39 green 39 | 39 yellow 39 | 39 red 39
40 green 40 | 40 yellow 40 | 40 red 40
41 green 41 | 41 yellow 41 | 41 red 41
42 green 42 | 42 yellow 42 | 42 red 42
43 green 43 | 43 yellow 43 | 43 red 43
44 green 44 | 44 yellow 44 | 44 red 44
45 green 45 | 45 yellow 45 | 45 red 45
46 green 46 | 46 yellow 46 | 46 red 46
47 green 47 | 47 yellow 47 | 47 red 47
48 green 48 | 48 yellow 48 | 48 red 48
49 green 49 | 49 yellow 49 | 49 red 49
50 green 50 | 50 yellow 50 | 50 red 50
51 green 51 | 51 yellow 51 | 51 red 51
52 green 52 | 52 yellow 52 | 52 red 52
53 green 53 | 53 yellow 53 | 53 red 53
54 green 54 | 54 yellow 54 | 54 red 54
55 green 55 | 55 yellow 55 | 55 red 55
56 green 56 | 56 yellow 56 | 56 red 56
57 green 57 | 57 yellow 57 | 57 red 57
58 green 58 | 58 yellow 58 | 58 red 58
59 green 59 | 59 yellow 59 | 59 red 59
60 green 60 | 60 yellow 60 | 60 red 60
61 green 61 | 61 yellow 61 | 61 red 61
62 green 62 | 62 yellow 62 | 62 red 62
63 green 63 | 63 yellow 63 | 63 red 63
VTEP5
VTEP5#show running-config qos
qos enable
!
qos profile queue-color-to-cos QUEUE_COS
queue 2 color all cos 5
!
qos profile dscp-to-queue DSCP_QUEUE
dscp 56 queue 2
!
!
!
VTEP5#show running-config nvo vxlan
!
nvo vxlan enable
!
evpn vxlan multihoming enable
!
nvo vxlan vtep-ip-global 5.5.5.5
!
nvo vxlan tunnel qos-map-mode cos-dscp ingress DSCP_QUEUE
!
nvo vxlan id 101 ingress-replication inner-vid-disabled
vxlan host-reachability-protocol evpn-bgp L2VRF1
vni-name VNI-101
!
nvo vxlan id 201 ingress-replication inner-vid-disabled
vxlan host-reachability-protocol evpn-bgp L2VRF2
vni-name VNI-201
!
nvo vxlan access-if port-vlan xe1 10
map vnid 101
mac 0000.4444.1010 ip 11.11.11.201
map qos-profile queue-color-to-cos QUEUE_COS
!
!
VTEP5#show nvo vxlan tunnel
VXLAN Network tunnel Entries
Source Destination Status Up/Down Update
============================================================================
5.5.5.5 2.2.2.2 Installed 01:11:17 01:11:17
5.5.5.5 4.4.4.4 Installed 01:14:50 01:14:50
5.5.5.5 1.1.1.1 Installed 01:14:50 01:14:50
Total number of entries are 3
VTEP5#show interface xe1 counters queue-stats
E - Egress, I - Ingress, Q-Size is in bytes
+--------------------+--------+-----------------+-------------------+-----------------+-------------------+
| Queue/Class-map | Q-Size | Tx pkts | Tx bytes | Dropped pkts | Dropped bytes |
+--------------------+--------+-----------------+-------------------+-----------------+-------------------+
q0 (E) 12517376 0 0 0 0
q1 (E) 12517376 0 0 0 0
q2 (E) 12517376 1005800 1052066800 0 0
q3 (E) 12517376 0 0 0 0
q4 (E) 12517376 0 0 0 0
q5 (E) 12517376 0 0 0 0
q6 (E) 12517376 0 0 0 0
q7 (E) 12517376 0 0 0 0
VTEP5#sh qos-profile QUEUE_COS
profile name: QUEUE_COS
profile type: queue-color-to-cos
profile attached to 1 instances
configured mapping:
queue 2 color all cos 5
Detailed mapping:
----------------+-------- | ----------------+-------- | ----------------+--------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
----------------+-------- | ----------------+-------- | ----------------+--------
Queue | Color | COS | Queue | Color | COS | Queue | Color | COS
-------+--------+-------- | -------+--------+-------- | -------+--------+--------
0 green 0 | 0 yellow 0 | 0 red 0
1 green 1 | 1 yellow 1 | 1 red 1
2 green 5 | 2 yellow 5 | 2 red 5
3 green 3 | 3 yellow 3 | 3 red 3
4 green 4 | 4 yellow 4 | 4 red 4
5 green 5 | 5 yellow 5 | 5 red 5
6 green 6 | 6 yellow 6 | 6 red 6
7 green 7 | 7 yellow 7 | 7 red 7
VTEP5#show qos-profile DSCP_QUEUE
profile name: DSCP_QUEUE
profile type: dscp-to-queue
profile attached to 1 instances
configured mapping:
dscp 56 queue 2
Detailed mapping:
-------+--------------------------- | -------+--------------------------- | -------+--------------------------- | -------+---------------------------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
-------+--------------------------- | -------+--------------------------- | -------+--------------------------- | -------+---------------------------
DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP
-------+-------+--------+---------- | -------+-------+--------+---------- | -------+-------+--------+---------- | -------+-------+--------+----------
0 0 green 0 | 16 2 green 16 | 32 4 green 32 | 48 6 green 48
1 0 green 1 | 17 2 green 17 | 33 4 green 33 | 49 6 green 49
2 0 green 2 | 18 2 green 18 | 34 4 green 34 | 50 6 green 50
3 0 green 3 | 19 2 green 19 | 35 4 green 35 | 51 6 green 51
4 0 green 4 | 20 2 yellow 20 | 36 4 yellow 36 | 52 6 green 52
5 0 green 5 | 21 2 green 21 | 37 4 green 37 | 53 6 green 53
6 0 green 6 | 22 2 yellow 22 | 38 4 yellow 38 | 54 6 green 54
7 0 green 7 | 23 2 green 23 | 39 4 green 39 | 55 6 green 55
8 1 green 8 | 24 3 green 24 | 40 5 green 40 | 56 2 green 56
9 1 green 9 | 25 3 green 25 | 41 5 green 41 | 57 7 green 57
10 1 green 10 | 26 3 green 26 | 42 5 green 42 | 58 7 green 58
11 1 green 11 | 27 3 green 27 | 43 5 green 43 | 59 7 green 59
12 1 yellow 12 | 28 3 yellow 28 | 44 5 green 44 | 60 7 green 60
13 1 green 13 | 29 3 green 29 | 45 5 green 45 | 61 7 green 61
14 1 yellow 14 | 30 3 yellow 30 | 46 5 green 46 | 62 7 green 62
15 1 green 15 | 31 3 green 31 | 47 5 green 47 | 63 7 green 63
Anycast Gateway
For today's large multi-tenant data center, centralized L3 gateway scheme is very inefficient and sometimes impractical. In order to overcome the drawback of centralized L3GW approach, anycast mode is used.
In Anycast gateway approach, all the VTEPs acts as default gateway for all the VNIDs. We will configure same anycast mac in all VTEPs.
IRB Configuration for Anycast
Configure from Base Configuration-L2 VXLAN section, then configure below commands for Anycast gateway approach.
Note: For L2 traffic, always dscp-to-queue qos profile at tunnel ingress takes effect.
Note: For L3 traffic in the local VTEP, routing is done at IRB level and also QoS applied at the IRB interface and it sends with l2vnid.
VTEP1
(config)#nvo vxlan irb | Enable VXLAN irb |
(config)#ip vrf L3VRF1 | Create mac routing/forwarding instance with L3VRF1 name and enter into vrf mode |
(config-vrf)#rd 11000:11 | Assign RD value |
(config-vrf)# route-target both 100:100 | Assign route-target value for same for import and export. |
(config-vrf)# l3vni 1000 | Configure L3VNI as 1000 for L3VRF1 |
(config-vrf)#exit | Exit from vrf mode |
(config)# evpn irb-forwarding anycast-gateway-mac 0000.0000.1111 | Configure anycast mac address |
(config)# interface irb1001 | Configure IRB interface 1001 |
(config-if)ip vrf forwarding L3VRF1 | Configure L3VRF1 |
(config-if)ip address 11.11.11.1/24 | Configure ip address |
(config-if)ipv6 address 1111::1/64 | Configure ipv6 address |
(config-if) evpn irb-if-forwarding anycast-gateway-mac | Configure anycast mac address |
(config-if)exit | Exit from interface config mode |
(config)# interface irb2001 | Configure IRB interface 2001 |
(config-if)ip vrf forwarding L3VRF1 | Configure L3VRF1 |
(config-if)ip address 21.21.21.1/24 | Configure ip address |
(config-if)ipv6 address 2121::1/64 | Configure ipv6 address |
(config-if) evpn irb-if-forwarding anycast-gateway-mac | Configure anycast mac address |
(config-if)exit | Exit from interface config mode |
(config)# nvo vxlan id 101 ingress-replication inner-vid-disabled | Configure VXLAN Network identifier with/without inner-vid- disabled configure and enter into VXLAN tenant mode |
(config-nvo)#vxlan host-reachability-protocol evpn-bgp L2VRF1 | Assign vrf for evpn-bgp to carry EVPN route |
(config-nvo)# evpn irb1001 | Configure irb1001 under vxlan id 101 |
(config-nvo)#exit | Exit from VXLAN tenant mode and enter into configuration mode. |
(config)#nvo vxlan id 201 ingress-replication inner-vid-disabled | Configure VXLAN Network identifier with/without inner-vid- disabled configure and enter into VXLAN tenant mode |
(config-nvo)#vxlan host-reachability-protocol evpn-bgp L2VRF2 | Assign vrf for evpn-bgp to carry EVPN route |
(config-nvo)# evpn irb2001 | Configure irb2001 under vxlan id 201 |
(config-nvo)#exit | Exit from VXLAN tenant mode and enter into configuration mode. |
(config)#qos profile dscp-to-queue DSCP_QUEUE | Configure QoS profile to remark the traffic received from access-if |
(config-ingress-dscp-map)#dscp 20 queue 4 | Configure particular dscp value to queue value in the profile |
(config-ingress-dscp-map)#exit | Exit from qos profile mode |
(config)#qos profile dscp-encap DSCP_ENCAP | Configure the qos profile to map the overlay dscp value at tunnel egress |
(config-egress-dscp-encap-map)#l3 dscp 20 dscpEncap 34 | Egress remarking of the customer dscp packet to overlay dscp |
(config-egress-dscp-encap-map)#exit | Exit from the qos profile mode |
(config)#int irb2001 | Enter IRB L3 interface |
(config-irb-if)#qos map-profile dscp-to-queue DSCP_QUEUE | Map the qos profile |
(config-irb-if)#exit | Exit from interface mode |
(config)#nvo vxlan tunnel qos-map-mode cos-dscp egress DSCP_ENCAP | Map the qos profile at vxlan tunnel egress |
(config)#commit | Commit the candidate configuration to the running configuration |
(config)#end | Exit from global config mode |
VTEP2
(config)#nvo vxlan irb | Enable VXLAN IRB |
(config)#ip vrf L3VRF1 | Create mac routing/forwarding instance with L3VRF1 name and enter into vrf mode |
(config-vrf)#rd 21000:11 | Assign RD value |
(config-vrf)# route-target both 100:100 | Assign route-target value for same for import and export. |
(config-vrf)# l3vni 1000 | Configure L3VNI as 1000 for L3VRF1 |
(config-vrf)#exit | Exit from vrf mode |
(config)# evpn irb-forwarding anycast-gateway-mac 0000.0000.1111 | Configure anycast mac address |
(config)# interface irb1001 | Configure IRB interface 1001 |
(config-if)ip vrf forwarding L3VRF1 | Configure L3VRF1 |
(config-if)ip address 11.11.11.1/24 | Configure ip address |
(config-if)ipv6 address 1111::1/64 | Configure ipv6 address |
(config-if) evpn irb-if-forwarding anycast-gateway-mac | Configure anycast mac address |
(config-if)exit | Exit from interface config mode |
(config)# interface irb2001 | Configure IRB interface 2001 |
(config-if)ip vrf forwarding L3VRF1 | Configure L3VRF1 |
(config-if)ip address 21.21.21.1/24 | Configure ip address |
(config-if)ipv6 address 2121::1/64 | Configure ipv6 address |
(config-if) evpn irb-if-forwarding anycast-gateway-mac | Configure anycast mac address |
(config-if)exit | Exit from interface config mode |
(config)# nvo vxlan id 101 ingress-replication inner-vid-disabled | Configure VXLAN Network identifier with/without inner-vid- disabled configure and enter into VXLAN tenant mode |
(config-nvo)#vxlan host-reachability-protocol evpn-bgp L2VRF1 | Assign vrf for evpn-bgp to carry EVPN route |
(config-nvo)# evpn irb1001 | Configure irb1001 under vxlan id 101 |
(config-nvo)#exit | Exit from VXLAN tenant mode and enter into configuration mode. |
(config)#nvo vxlan id 201 ingress-replication inner-vid-disabled | Configure VXLAN Network identifier with/without inner-vid- disabled configure and enter into VXLAN tenant mode |
(config-nvo)#vxlan host-reachability-protocol evpn-bgp L2VRF2 | Assign vrf for evpn-bgp to carry EVPN route |
(config-nvo)# evpn irb2001 | Configure irb2001 under vxlan id 201 |
(config-nvo)#exit | Exit from VXLAN tenant mode and enter into configuration mode. |
(config)#qos profile dscp-to-queue DSCP_QUEUE | Configure QoS profile to remark the traffic at remote vtep at the ingress tunnel. |
(config-ingress-dscp-map)#dscp 20 queue 4 | Configure particular dscp value to queue value in the profile |
(config-ingress-dscp-map)#exit | Exit from qos profile mode |
(config)#qos profile dscp-encap DSCP_ENCAP | Configure the qos profile to map the overlay dscp value at tunnel egress |
(config-egress-dscp-encap-map)#l3 dscp 20 dscpEncap 34 | Egress remarking of the customer dscp packet to overlay dscp |
(config-egress-dscp-encap-map)#exit | Exit from the qos profile mode |
(config)#int irb2001 | Enter IRB L3 interface |
(config-irb-if)#qos map-profile dscp-to-queue DSCP_QUEUE | Map the qos profile |
(config-irb-if)#exit | Exit from interface mode |
(config)#nvo vxlan tunnel qos-map-mode cos-dscp egress DSCP_ENCAP | Map the qos profile at vxlan tunnel egress |
(config)#commit | Commit the candidate configuration to the running configuration |
(config)#end | Exit from global config mode |
VTEP4
(config)#nvo vxlan irb | Enable VXLAN IRB |
(config)#ip vrf L3VRF1 | Create mac routing/forwarding instance with L3VRF1 name and enter into vrf mode |
(config-vrf)#rd 41000:11 | Assign RD value |
(config-vrf)# route-target both 100:100 | Assign route-target value for same for import and export. |
(config-vrf)# l3vni 1000 | Configure L3VNI as 1000 for L3VRF1 |
(config-vrf)#exit | Exit from vrf mode |
(config)# evpn irb-forwarding anycast-gateway-mac 0000.0000.1111 | Configure anycast mac address |
(config)# interface irb1001 | Configure IRB interface 1001 |
(config-if)ip vrf forwarding L3VRF1 | Configure L3VRF1 |
(config-if)ip address 11.11.11.1/24 | Configure ip address |
(config-if)ipv6 address 1111::1/64 | Configure ipv6 address |
(config-if) evpn irb-if-forwarding anycast-gateway-mac | Configure anycast mac address |
(config-if)exit | Exit from interface config mode |
(config)# interface irb2001 | Configure IRB interface 2001 |
(config-if)ip vrf forwarding L3VRF1 | Configure L3VRF1 |
(config-if)ip address 21.21.21.1/24 | Configure ip address |
(config-if)ipv6 address 2121::1/64 | Configure ipv6 address |
(config-if) evpn irb-if-forwarding anycast-gateway-mac | Configure anycast mac address |
(config-if)exit | Exit from interface config mode |
(config)# nvo vxlan id 101 ingress-replication inner-vid-disabled | Configure VXLAN Network identifier with/without inner-vid- disabled configure and enter into VXLAN tenant mode |
(config-nvo)#vxlan host-reachability-protocol evpn-bgp L2VRF1 | Assign vrf for evpn-bgp to carry EVPN route |
(config-nvo)# evpn irb1001 | Configure irb1001 under vxlan id 101 |
(config-nvo)#exit | Exit from VXLAN tenant mode and enter into configuration mode. |
(config)#nvo vxlan id 201 ingress-replication inner-vid-disabled | Configure VXLAN Network identifier with/without inner-vid- disabled configure and enter into VXLAN tenant mode |
(config-nvo)#vxlan host-reachability-protocol evpn-bgp L2VRF2 | Assign vrf for evpn-bgp to carry EVPN route |
(config-nvo)# evpn irb2001 | Configure irb2001 under vxlan id 201 |
(config-nvo)#commit | Commit the candidate configuration to the running configuration |
(config-nvo)#exit | Exit from VXLAN tenant mode and enter into configuration mode. |
VTEP5
(config)#nvo vxlan irb | Enable VXLAN IRB |
(config)#ip vrf L3VRF1 | Create mac routing/forwarding instance with L3VRF1 name and enter into vrf mode |
(config-vrf)#rd 51000:11 | Assign RD value |
(config-vrf)# route-target both 100:100 | Assign route-target value for same for import and export. |
(config-vrf)# l3vni 1000 | Configure L3VNI as 1000 for L3VRF1 |
(config-vrf)#exit | Exit from vrf mode |
(config)# evpn irb-forwarding anycast-gateway-mac 0000.0000.1111 | Configure anycast mac address |
(config)# interface irb1001 | Configure IRB interface 1001 |
(config-if)ip vrf forwarding L3VRF1 | Configure L3VRF1 |
(config-if)ip address 11.11.11.1/24 | Configure ip address |
(config-if)ipv6 address 1111::1/64 | Configure ipv6 address |
(config-if) evpn irb-if-forwarding anycast-gateway-mac | Configure anycast mac address |
(config-if)exit | Exit from interface config mode |
(config)# interface irb2001 | Configure IRB interface 2001 |
(config-if)ip vrf forwarding L3VRF1 | Configure L3VRF1 |
(config-if)ip address 21.21.21.1/24 | Configure ip address |
(config-if)ipv6 address 2121::1/64 | Configure ipv6 address |
(config-if) evpn irb-if-forwarding anycast-gateway-mac | Configure anycast mac address |
(config-if)exit | Exit from interface config mode |
(config)# nvo vxlan id 101 ingress-replication inner-vid-disabled | Configure VXLAN Network identifier with/without inner-vid- disabled configure and enter into VXLAN tenant mode |
(config-nvo)#vxlan host-reachability-protocol evpn-bgp L2VRF1 | Assign vrf for evpn-bgp to carry EVPN route |
(config-nvo)# evpn irb1001 | Configure irb1001 under vxlan id 101 |
(config-nvo)#exit | Exit from VXLAN tenant mode and enter into configuration mode. |
(config)#nvo vxlan id 201 ingress-replication inner-vid-disabled | Configure VXLAN Network identifier with/without inner-vid- disabled configure and enter into VXLAN tenant mode |
(config-nvo)#vxlan host-reachability-protocol evpn-bgp L2VRF2 | Assign vrf for evpn-bgp to carry EVPN route |
(config-nvo)# evpn irb2001 | Configure irb2001 under vxlan id 201 |
(config-nvo)#exit | Exit from VXLAN tenant mode and enter into configuration mode. |
(config)#qos profile dscp-to-queue DSCP_QUEUE | Configure the qos profile to remark outer dscp to queue of the the ingress tunnel traffic. Here classification at the ingress vtep is based on overlay dscp value. |
(config-ingress-dscp-map)#dscp 34 queue 2 | Configure particular dscp value to queue value in the profile. Here dscp to <queue, dscp> is not applicable. I.e ingress remarking of dscp at the ingress tunnel is not applicable. |
(config-ingress-dscp-map)#exit | Exit from qos profile mode |
(config)#qos profile queue-color-to-cos QUEUE_COS | Configure qos profile to remark the queue value to COS value |
(config-egress-cos-map)#queue 2 cos 6 | Configure particular queue value to COS value |
(config-egress-cos-map)#exit | Exit from qos profile config mode |
(config)#nvo vxlan tunnel qos-map-mode cos-dscp ingress DSCP_QUEUE | Map the qos profile in tunnel ingress |
(config)#nvo vxlan access-if port-vlan xe1 10 | Enter to vxlan access port config mode |
(config-nvo-acc-if)#map qos-profile queue-color-to-cos QUEUE_COS | Map the qos profile in vxlan access port |
(config-nvo-acc-if)#commit | Commit the candidate configuration to the running configuration |
(config-nvo-acc-if)#end | Exit from config mode |
Validations
Send traffic from TS2-21 to MH2 access-if with dscp value 20 and COS value 1(vlan20) and verify traffic received at TS1-11 with dscp value 20 and COS value 6(vlan10) at the VTEP5 access-if.
VTEP1
VTEP1#show running-config qos
qos enable
!
qos profile dscp-to-queue DSCP_QUEUE
dscp 20 queue 4
!
qos profile dscp-encap DSCP_ENCAP
l3 dscp 20 dscpEncap 34
!
!
!
!
interface irb2001
qos map-profile dscp-to-queue DSCP_QUEUE
!
VTEP1#show running-config nvo vxlan
!
nvo vxlan enable
!
nvo vxlan irb
!
evpn vxlan multihoming enable
!
evpn irb-forwarding anycast-gateway-mac 0000.0000.1111
!
nvo vxlan vtep-ip-global 1.1.1.1
!
nvo vxlan tunnel qos-map-mode cos-dscp egress DSCP_ENCAP
!
nvo vxlan id 101 ingress-replication inner-vid-disabled
vxlan host-reachability-protocol evpn-bgp L2VRF1
evpn irb1001
vni-name VNI-101
!
nvo vxlan id 201 ingress-replication inner-vid-disabled
vxlan host-reachability-protocol evpn-bgp L2VRF2
evpn irb2001
vni-name VNI-201
!
nvo vxlan access-if port-vlan po1 10
map vnid 101
mac 0000.2222.1010 ip 11.11.11.51
!
nvo vxlan access-if port-vlan po1 20
map vnid 201
mac 0000.2222.1020 ip 21.21.21.51
!
!
VTEP1#show nvo vxlan tunnel
VXLAN Network tunnel Entries
Source Destination Status Up/Down Update
============================================================================
1.1.1.1 5.5.5.5 Installed 01:15:13 01:15:13
1.1.1.1 4.4.4.4 Installed 01:15:28 01:15:28
1.1.1.1 2.2.2.2 Installed 01:11:40 01:11:40
Total number of entries are 3
VTEP1#show nvo vxlan
VXLAN Information
=================
Codes: NW - Network Port
AC - Access Port
(u) - Untagged
VNID VNI-Name VNI-Type Type Interface ESI VLAN DF-Status Src-Addr Dst-Addr
_______________________________________________________________________________________________________________________________
101 VNI-101 L2 NW ---- ---- ---- ---- 1.1.1.1 5.5.5.5
101 VNI-101 L2 NW ---- ---- ---- ---- 1.1.1.1 4.4.4.4
101 VNI-101 L2 NW ---- ---- ---- ---- 1.1.1.1 2.2.2.2
101 VNI-101 -- AC po1 00:00:00:00:00:22:22:00:00:00 10 DF ---- ----
201 VNI-201 L2 NW ---- ---- ---- ---- 1.1.1.1 5.5.5.5
201 VNI-201 L2 NW ---- ---- ---- ---- 1.1.1.1 4.4.4.4
201 VNI-201 L2 NW ---- ---- ---- ---- 1.1.1.1 2.2.2.2
201 VNI-201 -- AC po1 00:00:00:00:00:22:22:00:00:00 20 DF ---- ----
Total number of entries are 8
VTEP1#show nvo vxlan arp-cache
VXLAN ARP-CACHE Information
===========================
VNID Ip-Addr Mac-Addr Type Age-Out Retries-Left
____________________________________________________________________________
201 21.21.21.51 0000.2222.1020 Static Local ----
201 21.21.21.1 0000.0000.1111 Static Local ----
201 21.21.21.101 0000.5555.1020 Static Remote ----
101 11.11.11.51 0000.2222.1010 Static Local ----
101 11.11.11.10 0010.9400.0002 Dynamic Remote ----
101 11.11.11.1 0000.0000.1111 Static Local ----
101 11.11.11.201 0000.4444.1010 Static Remote ----
Total number of entries are 7
VTEP1#show nvo vxlan nd-cache
VXLAN ND-CACHE Information
===========================
VNID Ip-Addr Mac-Addr Type Age-Out Retries-Left
____________________________________________________________________________________________________
201 2121::1 0000.0000.1111 Static Local ----
101 1111::10 0010.9400.0002 Dynamic Remote ----
101 1111::1 0000.0000.1111 Static Local ----
Total number of entries are 3
VTEP1#show ipv4 route vrf L3VRF1
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2,
ia - IS-IS inter area, E - EVPN,
v - vrf leaked
* - candidate default
IP Route Table for VRF "L3VRF1"
C 11.11.11.0/24 is directly connected, irb1001, 00:13:19
C 21.21.21.0/24 is directly connected, irb2001, 00:12:56
C 127.0.0.0/8 is directly connected, lo.L3VRF1, 00:17:13
Gateway of last resort is not set
VTEP1#show ipv6 route vrf L3VRF1
IPv6 Routing Table
Codes: K - kernel route, C - connected, S - static, D- DHCP, R - RIP,
O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
E2 - OSPF external type 2, E - EVPN N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type 2, i - IS-IS, B - BGP,
v - vrf leaked
Timers: Uptime
IP Route Table for VRF "L3VRF1"
C ::1/128 via ::, lo.L3VRF1, 00:17:23
C 1111::/64 via ::, irb1001, 00:13:29
C 2121::/64 via ::, irb2001, 00:13:06
C fe80::/64 via ::, irb2001, 00:13:06
VTEP1#show interface ce52 counters queue-stats
E - Egress, I - Ingress, Q-Size is in bytes
+--------------------+--------+-----------------+-------------------+-----------------+-------------------+
| Queue/Class-map | Q-Size | Tx pkts | Tx bytes | Dropped pkts | Dropped bytes |
+--------------------+--------+-----------------+-------------------+-----------------+-------------------+
q0 (E) 125304832 0 0 0 0
q1 (E) 125304832 0 0 0 0
q2 (E) 125304832 0 0 0 0
q3 (E) 125304832 0 0 0 0
q4 (E) 125304832 0 0 0 0
q5 (E) 125304832 0 0 0 0
q6 (E) 125304832 0 0 0 0
q7 (E) 125304832 0 0 0 0
VTEP1#show nvo vxlan l3vni-map
L3VNI L2VNI IRB-interface
===================================
1000 101 irb1001
1000 201 irb2001
VTEP1#show qos-profile DSCP_QUEUE
profile name: DSCP_QUEUE
profile type: dscp-to-queue
profile attached to 1 instances
configured mapping:
dscp 20 queue 4
Detailed mapping:
-------+--------------------------- | -------+--------------------------- | -------+--------------------------- | -------+---------------------------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
-------+--------------------------- | -------+--------------------------- | -------+--------------------------- | -------+---------------------------
DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP
-------+-------+--------+---------- | -------+-------+--------+---------- | -------+-------+--------+---------- | -------+-------+--------+----------
0 0 green 0 | 16 2 green 16 | 32 4 green 32 | 48 6 green 48
1 0 green 1 | 17 2 green 17 | 33 4 green 33 | 49 6 green 49
2 0 green 2 | 18 2 green 18 | 34 4 green 34 | 50 6 green 50
3 0 green 3 | 19 2 green 19 | 35 4 green 35 | 51 6 green 51
4 0 green 4 | 20 4 yellow 20 | 36 4 yellow 36 | 52 6 green 52
5 0 green 5 | 21 2 green 21 | 37 4 green 37 | 53 6 green 53
6 0 green 6 | 22 2 yellow 22 | 38 4 yellow 38 | 54 6 green 54
7 0 green 7 | 23 2 green 23 | 39 4 green 39 | 55 6 green 55
8 1 green 8 | 24 3 green 24 | 40 5 green 40 | 56 7 green 56
9 1 green 9 | 25 3 green 25 | 41 5 green 41 | 57 7 green 57
10 1 green 10 | 26 3 green 26 | 42 5 green 42 | 58 7 green 58
11 1 green 11 | 27 3 green 27 | 43 5 green 43 | 59 7 green 59
12 1 yellow 12 | 28 3 yellow 28 | 44 5 green 44 | 60 7 green 60
13 1 green 13 | 29 3 green 29 | 45 5 green 45 | 61 7 green 61
14 1 yellow 14 | 30 3 yellow 30 | 46 5 green 46 | 62 7 green 62
15 1 green 15 | 31 3 green 31 | 47 5 green 47 | 63 7 green 63
VTEP1#show qos-profile DSCP_ENCAP
profile name: DSCP_ENCAP
profile type: dscp-encap
profile attached to 1 instances
configured mapping:
l3 dscp 20 dscpEncap 34
Detailed mapping:
L3 DSCP to DSCP-ENCAP
-------+-------- | -------+-------- | -------+-------- | -------+--------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
-------|-------- | -------|-------- | -------|-------- | -------|--------
DSCP | DSCP | DSCP | DSCP | DSCP | DSCP | DSCP | DSCP
-------+-------- | -------+-------- | -------+-------- | -------+--------
0 0 | 16 16 | 32 32 | 48 48
1 1 | 17 17 | 33 33 | 49 49
2 2 | 18 18 | 34 34 | 50 50
3 3 | 19 19 | 35 35 | 51 51
4 4 | 20 34 | 36 36 | 52 52
5 5 | 21 21 | 37 37 | 53 53
6 6 | 22 22 | 38 38 | 54 54
7 7 | 23 23 | 39 39 | 55 55
8 8 | 24 24 | 40 40 | 56 56
9 9 | 25 25 | 41 41 | 57 57
10 10 | 26 26 | 42 42 | 58 58
11 11 | 27 27 | 43 43 | 59 59
12 12 | 28 28 | 44 44 | 60 60
13 13 | 29 29 | 45 45 | 61 61
14 14 | 30 30 | 46 46 | 62 62
15 15 | 31 31 | 47 47 | 63 63
L2 Queue + Color to DSCP-ENCAP
---------------+--------| ----------------+--------| --------------+--------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
---------------+--------| ----------------+--------| --------------+--------
Queue | Color | DSCP | Queue | Color | DSCP | Queue | Color | DSCP
-------+-------+--------| -------+--------+--------| ------+-------+--------
0 green 0 | 0 yellow 0 | 0 red 0
1 green 8 | 1 yellow 8 | 1 red 8
2 green 16 | 2 yellow 16 | 2 red 16
3 green 24 | 3 yellow 24 | 3 red 24
4 green 32 | 4 yellow 32 | 4 red 32
5 green 40 | 5 yellow 40 | 5 red 40
6 green 48 | 6 yellow 48 | 6 red 48
7 green 56 | 7 yellow 56 | 7 red 56
VTEP1#show qos-profile interface irb2001
profile name: DSCP_QUEUE
profile type: dscp-to-queue (Ingress)
mapping:
-------+--------------------------- | -------+--------------------------- | -------+--------------------------- | -------+---------------------------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
-------+--------------------------- | -------+--------------------------- | -------+--------------------------- | -------+---------------------------
DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP
-------+-------+--------+---------- | -------+-------+--------+---------- | -------+-------+--------+---------- | -------+-------+--------+----------
0 0 green 0 | 16 2 green 16 | 32 4 green 32 | 48 6 green 48
1 0 green 1 | 17 2 green 17 | 33 4 green 33 | 49 6 green 49
2 0 green 2 | 18 2 green 18 | 34 4 green 34 | 50 6 green 50
3 0 green 3 | 19 2 green 19 | 35 4 green 35 | 51 6 green 51
4 0 green 4 | 20 4 yellow 20 | 36 4 yellow 36 | 52 6 green 52
5 0 green 5 | 21 2 green 21 | 37 4 green 37 | 53 6 green 53
6 0 green 6 | 22 2 yellow 22 | 38 4 yellow 38 | 54 6 green 54
7 0 green 7 | 23 2 green 23 | 39 4 green 39 | 55 6 green 55
8 1 green 8 | 24 3 green 24 | 40 5 green 40 | 56 7 green 56
9 1 green 9 | 25 3 green 25 | 41 5 green 41 | 57 7 green 57
10 1 green 10 | 26 3 green 26 | 42 5 green 42 | 58 7 green 58
11 1 green 11 | 27 3 green 27 | 43 5 green 43 | 59 7 green 59
12 1 yellow 12 | 28 3 yellow 28 | 44 5 green 44 | 60 7 green 60
13 1 green 13 | 29 3 green 29 | 45 5 green 45 | 61 7 green 61
14 1 yellow 14 | 30 3 yellow 30 | 46 5 green 46 | 62 7 green 62
15 1 green 15 | 31 3 green 31 | 47 5 green 47 | 63 7 green 63
profile name: default
profile type: dscp-to-dscp (Egress)
Status: Inactive
mapping:
----------------+---------- | ----------------+---------- | ----------------+----------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
----------------+---------- | ----------------+---------- | ----------------+----------
DSCP | Color | Out DSCP | DSCP | Color | Out DSCP | DSCP | Color | Out DSCP
------+---------+---------- | ------+---------+---------- | ------+---------+----------
0 green 0 | 0 yellow 0 | 0 red 0
1 green 1 | 1 yellow 1 | 1 red 1
2 green 2 | 2 yellow 2 | 2 red 2
3 green 3 | 3 yellow 3 | 3 red 3
4 green 4 | 4 yellow 4 | 4 red 4
5 green 5 | 5 yellow 5 | 5 red 5
6 green 6 | 6 yellow 6 | 6 red 6
7 green 7 | 7 yellow 7 | 7 red 7
8 green 8 | 8 yellow 8 | 8 red 8
9 green 9 | 9 yellow 9 | 9 red 9
10 green 10 | 10 yellow 12 | 10 red 14
11 green 11 | 11 yellow 11 | 11 red 11
12 green 12 | 12 yellow 12 | 12 red 14
13 green 13 | 13 yellow 13 | 13 red 13
14 green 14 | 14 yellow 14 | 14 red 14
15 green 15 | 15 yellow 15 | 15 red 15
16 green 16 | 16 yellow 16 | 16 red 16
17 green 17 | 17 yellow 17 | 17 red 17
18 green 18 | 18 yellow 20 | 18 red 22
19 green 19 | 19 yellow 19 | 19 red 19
20 green 20 | 20 yellow 20 | 20 red 22
21 green 21 | 21 yellow 21 | 21 red 21
22 green 22 | 22 yellow 22 | 22 red 22
23 green 23 | 23 yellow 23 | 23 red 23
24 green 24 | 24 yellow 24 | 24 red 24
25 green 25 | 25 yellow 25 | 25 red 25
26 green 26 | 26 yellow 28 | 26 red 30
27 green 27 | 27 yellow 27 | 27 red 27
28 green 28 | 28 yellow 28 | 28 red 30
29 green 29 | 29 yellow 29 | 29 red 29
30 green 30 | 30 yellow 30 | 30 red 30
31 green 31 | 31 yellow 31 | 31 red 31
32 green 32 | 32 yellow 32 | 32 red 32
33 green 33 | 33 yellow 33 | 33 red 33
34 green 34 | 34 yellow 36 | 34 red 38
35 green 35 | 35 yellow 35 | 35 red 35
36 green 36 | 36 yellow 36 | 36 red 38
37 green 37 | 37 yellow 37 | 37 red 37
38 green 38 | 38 yellow 38 | 38 red 38
39 green 39 | 39 yellow 39 | 39 red 39
40 green 40 | 40 yellow 40 | 40 red 40
41 green 41 | 41 yellow 41 | 41 red 41
42 green 42 | 42 yellow 42 | 42 red 42
43 green 43 | 43 yellow 43 | 43 red 43
44 green 44 | 44 yellow 44 | 44 red 44
45 green 45 | 45 yellow 45 | 45 red 45
46 green 46 | 46 yellow 46 | 46 red 46
47 green 47 | 47 yellow 47 | 47 red 47
48 green 48 | 48 yellow 48 | 48 red 48
49 green 49 | 49 yellow 49 | 49 red 49
50 green 50 | 50 yellow 50 | 50 red 50
51 green 51 | 51 yellow 51 | 51 red 51
52 green 52 | 52 yellow 52 | 52 red 52
53 green 53 | 53 yellow 53 | 53 red 53
54 green 54 | 54 yellow 54 | 54 red 54
55 green 55 | 55 yellow 55 | 55 red 55
56 green 56 | 56 yellow 56 | 56 red 56
57 green 57 | 57 yellow 57 | 57 red 57
58 green 58 | 58 yellow 58 | 58 red 58
59 green 59 | 59 yellow 59 | 59 red 59
60 green 60 | 60 yellow 60 | 60 red 60
61 green 61 | 61 yellow 61 | 61 red 61
62 green 62 | 62 yellow 62 | 62 red 62
63 green 63 | 63 yellow 63 | 63 red 63
VTEP2
VTEP2#show running-config qos
qos enable
!
qos profile dscp-to-queue DSCP_QUEUE
dscp 20 queue 4
!
qos profile dscp-encap DSCP_ENCAP
l3 dscp 20 dscpEncap 34
!
!
!
!
interface irb2001
qos map-profile dscp-to-queue DSCP_QUEUE
!
VTEP2#show running-config nvo vxlan
!
nvo vxlan enable
!
nvo vxlan irb
!
evpn esi hold-time 90
!
evpn vxlan multihoming enable
!
evpn irb-forwarding anycast-gateway-mac 0000.0000.1111
!
nvo vxlan vtep-ip-global 2.2.2.2
!
nvo vxlan tunnel qos-map-mode cos-dscp egress DSCP_ENCAP
!
nvo vxlan id 101 ingress-replication inner-vid-disabled
vxlan host-reachability-protocol evpn-bgp L2VRF1
evpn irb1001
vni-name VNI-101
!
nvo vxlan id 201 ingress-replication inner-vid-disabled
vxlan host-reachability-protocol evpn-bgp L2VRF2
evpn irb2001
vni-name VNI-201
!
nvo vxlan access-if port-vlan po1 10
map vnid 101
mac 0000.2222.1010 ip 11.11.11.51
!
nvo vxlan access-if port-vlan po1 20
map vnid 201
mac 0000.2222.1020 ip 21.21.21.51
!
!
VTEP2#show nvo vxlan tunnel
VXLAN Network tunnel Entries
Source Destination Status Up/Down Update
============================================================================
2.2.2.2 4.4.4.4 Installed 01:13:43 01:13:43
2.2.2.2 1.1.1.1 Installed 01:13:43 01:13:43
2.2.2.2 5.5.5.5 Installed 01:13:43 01:13:43
Total number of entries are 3
VTEP2#show nvo vxlan
VXLAN Information
=================
Codes: NW - Network Port
AC - Access Port
(u) - Untagged
VNID VNI-Name VNI-Type Type Interface ESI VLAN DF-Status Src-Addr Dst-Addr
_______________________________________________________________________________________________________________________________
101 VNI-101 L2 NW ---- ---- ---- ---- 2.2.2.2 4.4.4.4
101 VNI-101 L2 NW ---- ---- ---- ---- 2.2.2.2 1.1.1.1
101 VNI-101 L2 NW ---- ---- ---- ---- 2.2.2.2 5.5.5.5
101 VNI-101 -- AC po1 00:00:00:00:00:22:22:00:00:00 10 NON-DF ---- ----
201 VNI-201 L2 NW ---- ---- ---- ---- 2.2.2.2 4.4.4.4
201 VNI-201 L2 NW ---- ---- ---- ---- 2.2.2.2 1.1.1.1
201 VNI-201 L2 NW ---- ---- ---- ---- 2.2.2.2 5.5.5.5
201 VNI-201 -- AC po1 00:00:00:00:00:22:22:00:00:00 20 NON-DF ---- ----
Total number of entries are 8
VTEP2#show nvo vxlan arp-cache
VXLAN ARP-CACHE Information
===========================
VNID Ip-Addr Mac-Addr Type Age-Out Retries-Left
____________________________________________________________________________
201 21.21.21.51 0000.2222.1020 Static Local ----
201 21.21.21.1 0000.0000.1111 Static Local ----
201 21.21.21.101 0000.5555.1020 Static Remote ----
101 11.11.11.51 0000.2222.1010 Static Local ----
101 11.11.11.10 0010.9400.0002 Dynamic Remote ----
101 11.11.11.1 0000.0000.1111 Static Local ----
101 11.11.11.201 0000.4444.1010 Static Remote ----
Total number of entries are 7
VTEP2#show nvo vxlan nd-cache
VXLAN ND-CACHE Information
===========================
VNID Ip-Addr Mac-Addr Type Age-Out Retries-Left
____________________________________________________________________________________________________
201 2121::1 0000.0000.1111 Static Local ----
101 1111::10 0010.9400.0002 Dynamic Remote ----
101 1111::1 0000.0000.1111 Static Local ----
Total number of entries are 3
VTEP2#show ipv4 route vrf L3VRF1
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2,
ia - IS-IS inter area, E - EVPN,
v - vrf leaked
* - candidate default
IP Route Table for VRF "L3VRF1"
C 11.11.11.0/24 is directly connected, irb1001, 00:10:20
C 21.21.21.0/24 is directly connected, irb2001, 00:09:55
C 127.0.0.0/8 is directly connected, lo.L3VRF1, 00:13:30
Gateway of last resort is not set
VTEP2#show ipv6 route vrf L3VRF1
IPv6 Routing Table
Codes: K - kernel route, C - connected, S - static, D- DHCP, R - RIP,
O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
E2 - OSPF external type 2, E - EVPN N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type 2, i - IS-IS, B - BGP,
v - vrf leaked
Timers: Uptime
IP Route Table for VRF "L3VRF1"
C ::1/128 via ::, lo.L3VRF1, 00:13:36
C 1111::/64 via ::, irb1001, 00:10:26
C 2121::/64 via ::, irb2001, 00:10:01
C fe80::/64 via ::, irb2001, 00:10:01
VTEP2#show nvo vxlan l3vni-map
L3VNI L2VNI IRB-interface
===================================
1000 101 irb1001
1000 201 irb2001
VTEP2#show interface xe29 counters queue-stats
E - Egress, I - Ingress, Q-Size is in bytes
+--------------------+--------+-----------------+-------------------+-----------------+-------------------+
| Queue/Class-map | Q-Size | Tx pkts | Tx bytes | Dropped pkts | Dropped bytes |
+--------------------+--------+-----------------+-------------------+-----------------+-------------------+
q0 (E) 12517376 0 0 0 0
q1 (E) 12517376 0 0 0 0
q2 (E) 12517376 0 0 0 0
q3 (E) 12517376 0 0 0 0
q4 (E) 12517376 2998022 2998045000 0 0
q5 (E) 12517376 0 0 0 0
q6 (E) 12517376 0 0 0 0
q7 (E) 12517376 0 0 0 0
VTEP2#show qos-profile DSCP_QUEUE
profile name: DSCP_QUEUE
profile type: dscp-to-queue
profile attached to 1 instances
configured mapping:
dscp 20 queue 4
Detailed mapping:
-------+--------------------------- | -------+--------------------------- | -------+--------------------------- | -------+---------------------------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
-------+--------------------------- | -------+--------------------------- | -------+--------------------------- | -------+---------------------------
DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP
-------+-------+--------+---------- | -------+-------+--------+---------- | -------+-------+--------+---------- | -------+-------+--------+----------
0 0 green 0 | 16 2 green 16 | 32 4 green 32 | 48 6 green 48
1 0 green 1 | 17 2 green 17 | 33 4 green 33 | 49 6 green 49
2 0 green 2 | 18 2 green 18 | 34 4 green 34 | 50 6 green 50
3 0 green 3 | 19 2 green 19 | 35 4 green 35 | 51 6 green 51
4 0 green 4 | 20 4 yellow 20 | 36 4 yellow 36 | 52 6 green 52
5 0 green 5 | 21 2 green 21 | 37 4 green 37 | 53 6 green 53
6 0 green 6 | 22 2 yellow 22 | 38 4 yellow 38 | 54 6 green 54
7 0 green 7 | 23 2 green 23 | 39 4 green 39 | 55 6 green 55
8 1 green 8 | 24 3 green 24 | 40 5 green 40 | 56 7 green 56
9 1 green 9 | 25 3 green 25 | 41 5 green 41 | 57 7 green 57
10 1 green 10 | 26 3 green 26 | 42 5 green 42 | 58 7 green 58
11 1 green 11 | 27 3 green 27 | 43 5 green 43 | 59 7 green 59
12 1 yellow 12 | 28 3 yellow 28 | 44 5 green 44 | 60 7 green 60
13 1 green 13 | 29 3 green 29 | 45 5 green 45 | 61 7 green 61
14 1 yellow 14 | 30 3 yellow 30 | 46 5 green 46 | 62 7 green 62
15 1 green 15 | 31 3 green 31 | 47 5 green 47 | 63 7 green 63
VTEP2#show qos-profile DSCP_ENCAP
profile name: DSCP_ENCAP
profile type: dscp-encap
profile attached to 1 instances
configured mapping:
l3 dscp 20 dscpEncap 34
Detailed mapping:
L3 DSCP to DSCP-ENCAP
-------+-------- | -------+-------- | -------+-------- | -------+--------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
-------|-------- | -------|-------- | -------|-------- | -------|--------
DSCP | DSCP | DSCP | DSCP | DSCP | DSCP | DSCP | DSCP
-------+-------- | -------+-------- | -------+-------- | -------+--------
0 0 | 16 16 | 32 32 | 48 48
1 1 | 17 17 | 33 33 | 49 49
2 2 | 18 18 | 34 34 | 50 50
3 3 | 19 19 | 35 35 | 51 51
4 4 | 20 34 | 36 36 | 52 52
5 5 | 21 21 | 37 37 | 53 53
6 6 | 22 22 | 38 38 | 54 54
7 7 | 23 23 | 39 39 | 55 55
8 8 | 24 24 | 40 40 | 56 56
9 9 | 25 25 | 41 41 | 57 57
10 10 | 26 26 | 42 42 | 58 58
11 11 | 27 27 | 43 43 | 59 59
12 12 | 28 28 | 44 44 | 60 60
13 13 | 29 29 | 45 45 | 61 61
14 14 | 30 30 | 46 46 | 62 62
15 15 | 31 31 | 47 47 | 63 63
L2 Queue + Color to DSCP-ENCAP
---------------+--------| ----------------+--------| --------------+--------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
---------------+--------| ----------------+--------| --------------+--------
Queue | Color | DSCP | Queue | Color | DSCP | Queue | Color | DSCP
-------+-------+--------| -------+--------+--------| ------+-------+--------
0 green 0 | 0 yellow 0 | 0 red 0
1 green 8 | 1 yellow 8 | 1 red 8
2 green 16 | 2 yellow 16 | 2 red 16
3 green 24 | 3 yellow 24 | 3 red 24
4 green 32 | 4 yellow 32 | 4 red 32
5 green 40 | 5 yellow 40 | 5 red 40
6 green 48 | 6 yellow 48 | 6 red 48
7 green 56 | 7 yellow 56 | 7 red 56
VTEP2#show qos-profile interface irb2001
profile name: DSCP_QUEUE
profile type: dscp-to-queue (Ingress)
mapping:
-------+--------------------------- | -------+--------------------------- | -------+--------------------------- | -------+---------------------------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
-------+--------------------------- | -------+--------------------------- | -------+--------------------------- | -------+---------------------------
DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP
-------+-------+--------+---------- | -------+-------+--------+---------- | -------+-------+--------+---------- | -------+-------+--------+----------
0 0 green 0 | 16 2 green 16 | 32 4 green 32 | 48 6 green 48
1 0 green 1 | 17 2 green 17 | 33 4 green 33 | 49 6 green 49
2 0 green 2 | 18 2 green 18 | 34 4 green 34 | 50 6 green 50
3 0 green 3 | 19 2 green 19 | 35 4 green 35 | 51 6 green 51
4 0 green 4 | 20 4 yellow 20 | 36 4 yellow 36 | 52 6 green 52
5 0 green 5 | 21 2 green 21 | 37 4 green 37 | 53 6 green 53
6 0 green 6 | 22 2 yellow 22 | 38 4 yellow 38 | 54 6 green 54
7 0 green 7 | 23 2 green 23 | 39 4 green 39 | 55 6 green 55
8 1 green 8 | 24 3 green 24 | 40 5 green 40 | 56 7 green 56
9 1 green 9 | 25 3 green 25 | 41 5 green 41 | 57 7 green 57
10 1 green 10 | 26 3 green 26 | 42 5 green 42 | 58 7 green 58
11 1 green 11 | 27 3 green 27 | 43 5 green 43 | 59 7 green 59
12 1 yellow 12 | 28 3 yellow 28 | 44 5 green 44 | 60 7 green 60
13 1 green 13 | 29 3 green 29 | 45 5 green 45 | 61 7 green 61
14 1 yellow 14 | 30 3 yellow 30 | 46 5 green 46 | 62 7 green 62
15 1 green 15 | 31 3 green 31 | 47 5 green 47 | 63 7 green 63
profile name: default
profile type: dscp-to-dscp (Egress)
Status: Inactive
mapping:
----------------+---------- | ----------------+---------- | ----------------+----------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
----------------+---------- | ----------------+---------- | ----------------+----------
DSCP | Color | Out DSCP | DSCP | Color | Out DSCP | DSCP | Color | Out DSCP
------+---------+---------- | ------+---------+---------- | ------+---------+----------
0 green 0 | 0 yellow 0 | 0 red 0
1 green 1 | 1 yellow 1 | 1 red 1
2 green 2 | 2 yellow 2 | 2 red 2
3 green 3 | 3 yellow 3 | 3 red 3
4 green 4 | 4 yellow 4 | 4 red 4
5 green 5 | 5 yellow 5 | 5 red 5
6 green 6 | 6 yellow 6 | 6 red 6
7 green 7 | 7 yellow 7 | 7 red 7
8 green 8 | 8 yellow 8 | 8 red 8
9 green 9 | 9 yellow 9 | 9 red 9
10 green 10 | 10 yellow 12 | 10 red 14
11 green 11 | 11 yellow 11 | 11 red 11
12 green 12 | 12 yellow 12 | 12 red 14
13 green 13 | 13 yellow 13 | 13 red 13
14 green 14 | 14 yellow 14 | 14 red 14
15 green 15 | 15 yellow 15 | 15 red 15
16 green 16 | 16 yellow 16 | 16 red 16
17 green 17 | 17 yellow 17 | 17 red 17
18 green 18 | 18 yellow 20 | 18 red 22
19 green 19 | 19 yellow 19 | 19 red 19
20 green 20 | 20 yellow 20 | 20 red 22
21 green 21 | 21 yellow 21 | 21 red 21
22 green 22 | 22 yellow 22 | 22 red 22
23 green 23 | 23 yellow 23 | 23 red 23
24 green 24 | 24 yellow 24 | 24 red 24
25 green 25 | 25 yellow 25 | 25 red 25
26 green 26 | 26 yellow 28 | 26 red 30
27 green 27 | 27 yellow 27 | 27 red 27
28 green 28 | 28 yellow 28 | 28 red 30
29 green 29 | 29 yellow 29 | 29 red 29
30 green 30 | 30 yellow 30 | 30 red 30
31 green 31 | 31 yellow 31 | 31 red 31
32 green 32 | 32 yellow 32 | 32 red 32
33 green 33 | 33 yellow 33 | 33 red 33
34 green 34 | 34 yellow 36 | 34 red 38
35 green 35 | 35 yellow 35 | 35 red 35
36 green 36 | 36 yellow 36 | 36 red 38
37 green 37 | 37 yellow 37 | 37 red 37
38 green 38 | 38 yellow 38 | 38 red 38
39 green 39 | 39 yellow 39 | 39 red 39
40 green 40 | 40 yellow 40 | 40 red 40
41 green 41 | 41 yellow 41 | 41 red 41
42 green 42 | 42 yellow 42 | 42 red 42
43 green 43 | 43 yellow 43 | 43 red 43
44 green 44 | 44 yellow 44 | 44 red 44
45 green 45 | 45 yellow 45 | 45 red 45
46 green 46 | 46 yellow 46 | 46 red 46
47 green 47 | 47 yellow 47 | 47 red 47
48 green 48 | 48 yellow 48 | 48 red 48
49 green 49 | 49 yellow 49 | 49 red 49
50 green 50 | 50 yellow 50 | 50 red 50
51 green 51 | 51 yellow 51 | 51 red 51
52 green 52 | 52 yellow 52 | 52 red 52
53 green 53 | 53 yellow 53 | 53 red 53
54 green 54 | 54 yellow 54 | 54 red 54
55 green 55 | 55 yellow 55 | 55 red 55
56 green 56 | 56 yellow 56 | 56 red 56
57 green 57 | 57 yellow 57 | 57 red 57
58 green 58 | 58 yellow 58 | 58 red 58
59 green 59 | 59 yellow 59 | 59 red 59
60 green 60 | 60 yellow 60 | 60 red 60
61 green 61 | 61 yellow 61 | 61 red 61
62 green 62 | 62 yellow 62 | 62 red 62
63 green 63 | 63 yellow 63 | 63 red 63
VTEP4
VTEP4#show nvo vxlan tunnel
VXLAN Network tunnel Entries
Source Destination Status Up/Down Update
============================================================================
4.4.4.4 2.2.2.2 Installed 01:14:47 01:14:47
4.4.4.4 1.1.1.1 Installed 01:18:35 01:18:35
4.4.4.4 5.5.5.5 Installed 01:18:20 01:18:20
Total number of entries are 3
VTEP4#show nvo vxlan
VXLAN Information
=================
Codes: NW - Network Port
AC - Access Port
(u) - Untagged
VNID VNI-Name VNI-Type Type Interface ESI VLAN DF-Status Src-Addr Dst-Addr
_______________________________________________________________________________________________________________________________
101 VNI-101 L2 NW ---- ---- ---- ---- 4.4.4.4 2.2.2.2
101 VNI-101 L2 NW ---- ---- ---- ---- 4.4.4.4 1.1.1.1
101 VNI-101 L2 NW ---- ---- ---- ---- 4.4.4.4 5.5.5.5
201 VNI-201 L2 NW ---- ---- ---- ---- 4.4.4.4 2.2.2.2
201 VNI-201 L2 NW ---- ---- ---- ---- 4.4.4.4 1.1.1.1
201 VNI-201 L2 NW ---- ---- ---- ---- 4.4.4.4 5.5.5.5
201 VNI-201 -- AC xe3 --- Single Homed Port --- 20 ---- ---- ----
Total number of entries are 7
VTEP4#show nvo vxlan arp-cache
VXLAN ARP-CACHE Information
===========================
VNID Ip-Addr Mac-Addr Type Age-Out Retries-Left
____________________________________________________________________________
201 21.21.21.51 0000.2222.1020 Static Remote ----
201 21.21.21.1 0000.0000.1111 Static Local ----
201 21.21.21.101 0000.5555.1020 Static Local ----
101 11.11.11.51 0000.2222.1010 Static Remote ----
101 11.11.11.10 0010.9400.0002 Dynamic Remote ----
101 11.11.11.1 0000.0000.1111 Static Local ----
101 11.11.11.201 0000.4444.1010 Static Remote ----
Total number of entries are 7
VTEP4#show nvo vxlan nd-cache
VXLAN ND-CACHE Information
===========================
VNID Ip-Addr Mac-Addr Type Age-Out Retries-Left
____________________________________________________________________________________________________
201 2121::1 0000.0000.1111 Static Local ----
101 1111::10 0010.9400.0002 Dynamic Remote ----
101 1111::1 0000.0000.1111 Static Local ----
Total number of entries are 3
VTEP4#show ipv4 route vrf L3VRF1
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2,
ia - IS-IS inter area, E - EVPN,
v - vrf leaked
* - candidate default
IP Route Table for VRF "L3VRF1"
C 11.11.11.0/24 is directly connected, irb1001, 00:07:24
C 21.21.21.0/24 is directly connected, irb2001, 00:07:14
C 127.0.0.0/8 is directly connected, lo.L3VRF1, 00:10:16
Gateway of last resort is not set
VTEP4#show ipv6 route vrf L3VRF1
IPv6 Routing Table
Codes: K - kernel route, C - connected, S - static, D- DHCP, R - RIP,
O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
E2 - OSPF external type 2, E - EVPN N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type 2, i - IS-IS, B - BGP,
v - vrf leaked
Timers: Uptime
IP Route Table for VRF "L3VRF1"
C ::1/128 via ::, lo.L3VRF1, 00:10:21
C 1111::/64 via ::, irb1001, 00:07:29
C 2121::/64 via ::, irb2001, 00:07:19
C fe80::/64 via ::, irb2001, 00:07:19
VTEP4#show nvo vxlan l3vni-map
L3VNI L2VNI IRB-interface
===================================
1000 101 irb1001
1000 201 irb2001
VTEP5
VTEP5#show running-config qos
qos enable
!
qos profile queue-color-to-cos QUEUE_COS
queue 2 color all cos 6
!
qos profile dscp-to-queue DSCP_QUEUE
dscp 34 queue 2
!
!
!
VTEP5#show running-config nvo vxlan
!
nvo vxlan enable
!
nvo vxlan irb
!
evpn vxlan multihoming enable
!
evpn irb-forwarding anycast-gateway-mac 0000.0000.1111
!
nvo vxlan vtep-ip-global 5.5.5.5
!
nvo vxlan tunnel qos-map-mode cos-dscp ingress DSCP_QUEUE
!
nvo vxlan id 101 ingress-replication inner-vid-disabled
vxlan host-reachability-protocol evpn-bgp L2VRF1
evpn irb1001
vni-name VNI-101
!
nvo vxlan id 201 ingress-replication inner-vid-disabled
vxlan host-reachability-protocol evpn-bgp L2VRF2
evpn irb2001
vni-name VNI-201
!
nvo vxlan access-if port-vlan xe1 10
map vnid 101
mac 0000.4444.1010 ip 11.11.11.201
map qos-profile queue-color-to-cos QUEUE_COS
!
!
VTEP5#show nvo vxlan tunnel
VXLAN Network tunnel Entries
Source Destination Status Up/Down Update
============================================================================
5.5.5.5 2.2.2.2 Installed 01:11:17 01:11:17
5.5.5.5 4.4.4.4 Installed 01:14:50 01:14:50
5.5.5.5 1.1.1.1 Installed 01:14:50 01:14:50
Total number of entries are 3
VTEP5#sh nvo vxlan
VXLAN Information
=================
Codes: NW - Network Port
AC - Access Port
(u) - Untagged
VNID VNI-Name VNI-Type Type Interface ESI VLAN DF-Status Src-Addr Dst-Addr
_______________________________________________________________________________________________________________________________
101 VNI-101 L2 NW ---- ---- ---- ---- 5.5.5.5 2.2.2.2
101 VNI-101 L2 NW ---- ---- ---- ---- 5.5.5.5 4.4.4.4
101 VNI-101 L2 NW ---- ---- ---- ---- 5.5.5.5 1.1.1.1
101 VNI-101 -- AC xe1 --- Single Homed Port --- 10 ---- ---- ----
201 VNI-201 L2 NW ---- ---- ---- ---- 5.5.5.5 2.2.2.2
201 VNI-201 L2 NW ---- ---- ---- ---- 5.5.5.5 4.4.4.4
201 VNI-201 L2 NW ---- ---- ---- ---- 5.5.5.5 1.1.1.1
Total number of entries are 7
VTEP5#show nvo vxlan arp-cache
VXLAN ARP-CACHE Information
===========================
VNID Ip-Addr Mac-Addr Type Age-Out Retries-Left
____________________________________________________________________________
201 21.21.21.51 0000.2222.1020 Static Remote ----
201 21.21.21.1 0000.0000.1111 Static Local ----
201 21.21.21.101 0000.5555.1020 Static Remote ----
101 11.11.11.51 0000.2222.1010 Static Remote ----
101 11.11.11.10 0010.9400.0002 Dynamic Local ----
101 11.11.11.1 0000.0000.1111 Static Local ----
101 11.11.11.201 0000.4444.1010 Static Local ----
Total number of entries are 7
VTEP5#show nvo vxlan nd-cache
VXLAN ND-CACHE Information
===========================
VNID Ip-Addr Mac-Addr Type Age-Out Retries-Left
____________________________________________________________________________________________________
201 2121::1 0000.0000.1111 Static Local ----
101 1111::10 0010.9400.0002 Dynamic Local ----
101 1111::1 0000.0000.1111 Static Local ----
Total number of entries are 3
VTEP5#show ipv4 route vrf L3VRF1
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2,
ia - IS-IS inter area, E - EVPN,
v - vrf leaked
* - candidate default
IP Route Table for VRF "L3VRF1"
C 11.11.11.0/24 is directly connected, irb1001, 00:05:08
C 21.21.21.0/24 is directly connected, irb2001, 00:04:57
C 127.0.0.0/8 is directly connected, lo.L3VRF1, 00:07:30
Gateway of last resort is not set
VTEP5#show ipv6 route vrf L3VRF1
IPv6 Routing Table
Codes: K - kernel route, C - connected, S - static, D- DHCP, R - RIP,
O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
E2 - OSPF external type 2, E - EVPN N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type 2, i - IS-IS, B - BGP,
v - vrf leaked
Timers: Uptime
IP Route Table for VRF "L3VRF1"
C ::1/128 via ::, lo.L3VRF1, 00:07:35
C 1111::/24 via ::, irb1001, 00:05:13
C 2121::/64 via ::, irb2001, 00:05:02
C fe80::/64 via ::, irb2001, 00:05:02
VTEP5#show nvo vxlan l3vni-map
L3VNI L2VNI IRB-interface
===================================
1000 101 irb1001
1000 201 irb2001
VTEP5#show interface xe1 counters queue-stats
E - Egress, I - Ingress, Q-Size is in bytes
+--------------------+--------+-----------------+-------------------+-----------------+-------------------+
| Queue/Class-map | Q-Size | Tx pkts | Tx bytes | Dropped pkts | Dropped bytes |
+--------------------+--------+-----------------+-------------------+-----------------+-------------------+
q0 (E) 12517376 0 0 0 0
q1 (E) 12517376 0 0 0 0
q2 (E) 12517376 1170696 1224549062 0 0
q3 (E) 12517376 0 0 0 0
q4 (E) 12517376 0 0 0 0
q5 (E) 12517376 0 0 0 0
q6 (E) 12517376 0 0 0 0
q7 (E) 12517376 0 0 0 0
VTEP5#show qos-profile QUEUE_COS
profile name: QUEUE_COS
profile type: queue-color-to-cos
profile attached to 1 instances
configured mapping:
queue 2 color all cos 6
Detailed mapping:
----------------+-------- | ----------------+-------- | ----------------+--------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
----------------+-------- | ----------------+-------- | ----------------+--------
Queue | Color | COS | Queue | Color | COS | Queue | Color | COS
-------+--------+-------- | -------+--------+-------- | -------+--------+--------
0 green 0 | 0 yellow 0 | 0 red 0
1 green 1 | 1 yellow 1 | 1 red 1
2 green 6 | 2 yellow 6 | 2 red 6
3 green 3 | 3 yellow 3 | 3 red 3
4 green 4 | 4 yellow 4 | 4 red 4
5 green 5 | 5 yellow 5 | 5 red 5
6 green 6 | 6 yellow 6 | 6 red 6
7 green 7 | 7 yellow 7 | 7 red 7
VTEP5#show qos-profile DSCP_QUEUE
profile name: DSCP_QUEUE
profile type: dscp-to-queue
profile attached to 1 instances
configured mapping:
dscp 34 queue 2
Detailed mapping:
-------+--------------------------- | -------+--------------------------- | -------+--------------------------- | -------+---------------------------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
-------+--------------------------- | -------+--------------------------- | -------+--------------------------- | -------+---------------------------
DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP
-------+-------+--------+---------- | -------+-------+--------+---------- | -------+-------+--------+---------- | -------+-------+--------+----------
0 0 green 0 | 16 2 green 16 | 32 4 green 32 | 48 6 green 48
1 0 green 1 | 17 2 green 17 | 33 4 green 33 | 49 6 green 49
2 0 green 2 | 18 2 green 18 | 34 2 green 34 | 50 6 green 50
3 0 green 3 | 19 2 green 19 | 35 4 green 35 | 51 6 green 51
4 0 green 4 | 20 2 yellow 20 | 36 4 yellow 36 | 52 6 green 52
5 0 green 5 | 21 2 green 21 | 37 4 green 37 | 53 6 green 53
6 0 green 6 | 22 2 yellow 22 | 38 4 yellow 38 | 54 6 green 54
7 0 green 7 | 23 2 green 23 | 39 4 green 39 | 55 6 green 55
8 1 green 8 | 24 3 green 24 | 40 5 green 40 | 56 7 green 56
9 1 green 9 | 25 3 green 25 | 41 5 green 41 | 57 7 green 57
10 1 green 10 | 26 3 green 26 | 42 5 green 42 | 58 7 green 58
11 1 green 11 | 27 3 green 27 | 43 5 green 43 | 59 7 green 59
12 1 yellow 12 | 28 3 yellow 28 | 44 5 green 44 | 60 7 green 60
13 1 green 13 | 29 3 green 29 | 45 5 green 45 | 61 7 green 61
14 1 yellow 14 | 30 3 yellow 30 | 46 5 green 46 | 62 7 green 62
15 1 green 15 | 31 3 green 31 | 47 5 green 47 | 63 7 green 63
Distributed Gateway
In distributed gateway approach, VTEP will act as default gateways for one or more VNIDs,
Each VTEP having its own default gateway IP and MAC configuration for a given VNID.
IRB QoS Configuration for Distributed
Configure from Base Configuration-L2 VXLAN section, then configure below commands for centralized distributed approach.
Note: For L3 traffic, when l3vni is sent in the traffic, then dscp-to-queue qos profile mapped at tunnel ingress takes effect.
VTEP4
Unconfigure vnid 101 from nvo vxlan.
(config)#nvo vxlan irb | Enable VXLAN irb |
(config)#ip vrf L3VRF1 | Create mac routing/forwarding instance with L3VRF1 name and enter into vrf mode |
(config-vrf)#rd 41000:11 | Assign RD value |
(config-vrf)# route-target both 100:100 | Assign route-target value for same for import and export. |
(config-vrf)# l3vni 1000 | Configure L3VNI as 1000 for L3VRF1 |
(config-vrf)#exit | Exit from vrf mode |
(config)# interface irb2001 | Configure IRB interface 2001 |
(config-if)ip vrf forwarding L3VRF1 | Configure L3VRF1 |
(config-if)ip address 21.21.21.1/24 | Configure ip address |
(config-if)ipv6 address 2121::1/64 | Configure ipv6 address |
(config-if)exit | Exit from interface config mode |
(config)router bgp 5000 | Enter into bgp router mode |
(config-router)#address-family ipv4 vrf L3VRF1 | Enter into address-family mode for L3VRF1 |
(config-router-af)#redistribute connected | Redistribute connected |
(config-router-af)#exit-address-family | Exit form address-family |
(config-router)#address-family ipv6 vrf L3VRF1 | Enter into address-family mode for L3VRF1 |
(config-router-af)#redistribute connected | Redistribute connected |
(config-router-af)#exit-address-family | Exit form address-family |
(config)#nvo vxlan id 201 ingress-replication inner-vid-disabled | Configure VXLAN Network identifier with/without inner-vid- disabled configure and enter into VXLAN tenant mode |
(config-nvo)#vxlan host-reachability-protocol evpn-bgp L2VRF2 | Assign vrf for evpn-bgp to carry EVPN route |
(config-nvo)# evpn irb2001 | Configure irb2001 under vxlan id 201 |
(config-nvo)#exit | Exit from VXLAN tenant mode and enter into configuration mode. |
(config)#qos remark dscp | Enable qos dscp remark for dscp-to-dscp mapping if required |
(config)#qos profile dscp-to-queue DSCP_QUEUE | Configure qos profile to remark at vxlan ingress tunnel. Here dscp to <queue, dscp> is not applicable. I.e ingress remarking of dscp at the ingress tunnel is not applicable. |
(config-ingress-dscp-map)#dscp 56 queue 6 | Configure particular outer dscp value to queue value. Here classification at the ingress vtep is based on overlay dscp value |
(config-ingress-dscp-map)#exit | Exit from qos profile config mode |
(config)#qos profile dscp-to-dscp DSCP_DSCP | Configure qos profile for dscp value remark if required. Here classification at the egress vtep is based on customer dscp value |
(config-egress-dscp-map)#dscp 20 dscp 32 | Configure particular dscp value to dscp value |
(config-egress-dscp-map)#exit | Exit from qos profile config mode |
(config)#qos profile queue-color-to-cos QUEUE_COS | Configure qos profile for remark at vxlan access-if |
(config-egress-cos-map)#queue 6 cos 2 | Configure particular queue value to COS value |
(config-egress-cos-map)#exit | Exit from qos profile config mode |
(config)#nvo vxlan tunnel qos-map-mode cos-dscp ingress DSCP_QUEUE | Map the qos profile to vxlan tunnel ingress |
(config)#int irb2001 | Enter IRB interface mode |
(config-irb-if)#qos map-profile dscp-to-dscp DSCP_DSCP | Map Qos profile for dscp remark if required |
(config-irb-if)#exit | Exit from interface mode |
(config)#nvo vxlan access-if port-vlan xe3 20 | Enter vxlan access-if mode |
(config-nvo-acc-if)#map qos-profile queue-color-to-cos QUEUE_COS | Map qos profile |
(config-nvo-acc-if)#commit | Commit the candidate configuration to the running configuration |
(config-nvo-acc-if)#end | Exit from global configuration mode |
VTEP5
Unconfigure vnid 201 from nvo vxlan.
(config)#nvo vxlan irb | Enable VXLAN IRB |
(config)#ip vrf L3VRF1 | Create mac routing/forwarding instance with L3VRF1 name and enter into vrf mode |
(config-vrf)#rd 51000:11 | Assign RD value |
(config-vrf)# route-target both 100:100 | Assign route-target value for same for import and export. |
(config-vrf)# l3vni 1000 | Configure L3VNI as 1000 for L3VRF1 |
(config-vrf)#exit | Exit from vrf mode |
(config)# interface irb1001 | Configure IRB interface 1001 |
(config-if)ip vrf forwarding L3VRF1 | Configure L3VRF1 |
(config-if)ip address 11.11.11.1/24 | Configure ip address |
(config-if)ipv6 address 1111::1/64 | Configure ipv6 address |
(config-if)exit | Exit from interface config mode |
(config)router bgp 5000 | Enter into bgp router mode |
(config-router)#address-family ipv4 vrf L3VRF1 | Enter into address-family mode for L3VRF1 |
(config-router-af)#redistribute connected | Redistribute connected |
(config-router-af)#exit-address-family | Exit form address-family |
(config-router)#address-family ipv6 vrf L3VRF1 | Enter into address-family mode for L3VRF1 |
(config-router-af)#redistribute connected | Redistribute connected |
(config-router-af)#exit-address-family | Exit form address-family |
(config)# nvo vxlan id 101 ingress-replication inner-vid-disabled | Configure VXLAN Network identifier with/without inner-vid- disabled configure and enter into VXLAN tenant mode |
(config-nvo)#vxlan host-reachability-protocol evpn-bgp L2VRF1 | Assign vrf for evpn-bgp to carry EVPN route |
(config-nvo)# evpn irb1001 | Configure irb1001 under vxlan id 101 |
(config-nvo)#exit | Exit from VXLAN tenant mode and enter into configuration mode. |
(config)#qos profile dscp-to-queue DSCP_QUEUE | Configure qos profile for dscp to queue for ingress traffic |
(config-ingress-dscp-map)# dscp 20 queue 4 | Configure particular dscp value to queue value |
(config-ingress-dscp-map)#qos profile dscp-encap DSCP_ENCAP | Configure qos profile for overlay dscp remark in vxlan tunnel egress |
(config-egress-dscp-encap-map)# l3 dscp 20 dscpEncap 56 | Egress remarking of the customer dscp packet to overlay dscp |
(config-egress-dscp-encap-map)#exit | Exit from qos profile config mode |
(config)#nvo vxlan tunnel qos-map-mode cos-dscp egress DSCP_ENCAP | Map the qos profile in vxlan tunnel egress |
(config)#interface irb1001 | Enter IRB L3 interface |
(config-irb-if)#qos map-profile dscp-to-queue DSCP_QUEUE | Map qos profile |
(config-irb-if)#commit | Commit the candidate configuration to the running configuration |
(config-irb-if)#end | Exit from global conf mode |
Validations
Send traffic from TS1-11 to VTEP5 access-if with dscp value 20 COS value 1(vlan10) and verify traffic received at TS2-21 with dscp value 32 and COS value 2(vlan20) at the VTEP4 access-if.
VTEP5
VTEP5#show running-config qos
qos enable
!
qos profile dscp-to-queue DSCP_QUEUE
dscp 20 queue 4
!
qos profile dscp-encap DSCP_ENCAP
l3 dscp 20 dscpEncap 56
!
!
!
!
interface irb1001
qos map-profile dscp-to-queue DSCP_QUEUE
!
VTEP5#show run nvo vxlan
!
nvo vxlan enable
!
nvo vxlan irb
!
evpn vxlan multihoming enable
!
nvo vxlan vtep-ip-global 5.5.5.5
!
nvo vxlan tunnel qos-map-mode cos-dscp egress DSCP_ENCAP
!
nvo vxlan id 101 ingress-replication inner-vid-disabled
vxlan host-reachability-protocol evpn-bgp L2VRF1
evpn irb1001
vni-name VNI-101
!
nvo vxlan id 201 ingress-replication inner-vid-disabled
vxlan host-reachability-protocol evpn-bgp L2VRF2
vni-name VNI-201
!
nvo vxlan access-if port-vlan xe1 10
map vnid 101
mac 0000.4444.1010 ip 11.11.11.201
!
!
VTEP5#show nvo vxlan tunnel
VXLAN Network tunnel Entries
Source Destination Status Up/Down Update
============================================================================
5.5.5.5 2.2.2.2 Installed 00:29:54 00:29:54
5.5.5.5 4.4.4.4 Installed 00:29:54 00:29:54
5.5.5.5 1.1.1.1 Installed 00:29:54 00:29:54
Total number of entries are 3
VTEP5#show nvo vxlan
VXLAN Information
=================
Codes: NW - Network Port
AC - Access Port
(u) - Untagged
VNID VNI-Name VNI-Type Type Interface ESI VLAN DF-Status Src-Addr Dst-Addr
_______________________________________________________________________________________________________________________________
101 VNI-101 L2 NW ---- ---- ---- ---- 5.5.5.5 2.2.2.2
101 VNI-101 L2 NW ---- ---- ---- ---- 5.5.5.5 4.4.4.4
101 VNI-101 L2 NW ---- ---- ---- ---- 5.5.5.5 1.1.1.1
101 VNI-101 -- AC xe1 --- Single Homed Port --- 10 ---- ---- ----
201 VNI-201 L2 NW ---- ---- ---- ---- 5.5.5.5 2.2.2.2
201 VNI-201 L2 NW ---- ---- ---- ---- 5.5.5.5 4.4.4.4
201 VNI-201 L2 NW ---- ---- ---- ---- 5.5.5.5 1.1.1.1
1000 ---- L3 NW ---- ---- ---- ---- 5.5.5.5 4.4.4.4
Total number of entries are 8
VTEP5#show nvo vxlan arp-cache
VXLAN ARP-CACHE Information
===========================
VNID Ip-Addr Mac-Addr Type Age-Out Retries-Left
____________________________________________________________________________
201 21.21.21.51 0000.2222.1020 Static Remote ----
201 21.21.21.1 3c2c.991a.da7a Static Remote ----
201 21.21.21.101 0000.5555.1020 Static Remote ----
101 11.11.11.51 0000.2222.1010 Static Remote ----
101 11.11.11.1 04f8.f82f.8eee Static Local ----
101 11.11.11.201 0000.4444.1010 Static Local ----
Total number of entries are 6
VTEP5#show nvo vxlan nd-cache
VXLAN ND-CACHE Information
===========================
VNID Ip-Addr Mac-Addr Type Age-Out Retries-Left
____________________________________________________________________________________________________
201 2121::1 3c2c.991a.da7a Static Remote ----
101 1111::1 04f8.f82f.8eee Static Local ----
Total number of entries are 2
VTEP5#show nvo vxlan l3vni-map
L3VNI L2VNI IRB-interface
===================================
1000 101 irb1001
VTEP5#show ipv4 route vrf L3VRF1
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2,
ia - IS-IS inter area, E - EVPN,
v - vrf leaked
* - candidate default
IP Route Table for VRF "L3VRF1"
B 4.4.4.4/32 [0/0] is directly connected, tunvxlan1000, 00:04:09
C 11.11.11.0/24 is directly connected, irb1001, 00:02:38
B 21.21.21.0/24 [200/0] via 4.4.4.4 (recursive is directly connected, tunvxlan1000), 00:04:09
C 127.0.0.0/8 is directly connected, lo.L3VRF1, 00:04:46
Gateway of last resort is not set
VTEP5#show ipv6 route vrf L3VRF1
IPv6 Routing Table
Codes: K - kernel route, C - connected, S - static, D- DHCP, R - RIP,
O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
E2 - OSPF external type 2, E - EVPN N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type 2, i - IS-IS, B - BGP,
v - vrf leaked
Timers: Uptime
IP Route Table for VRF "L3VRF1"
C ::1/128 via ::, lo.L3VRF1, 00:04:50
B ::ffff:404:404/128 [0/0] via ::, tunvxlan1000, 00:04:13
C 1111::/64 via ::, irb1001, 00:02:42
B 2121::/64 [200/0] via ::ffff:404:404 (recursive via ::, unknown), 00:04:13
C fe80::/64 via ::, irb1001, 00:02:42
VTEP5#show interface ce53 counter queue-stats
E - Egress, I - Ingress, Q-Size is in bytes
+--------------------+--------+-----------------+-------------------+-----------------+-------------------+
| Queue/Class-map | Q-Size | Tx pkts | Tx bytes | Dropped pkts | Dropped bytes |
+--------------------+--------+-----------------+-------------------+-----------------+-------------------+
q0 (E) 50069504 0 0 0 0
q1 (E) 50069504 0 0 0 0
q2 (E) 50069504 0 0 0 0
q3 (E) 50069504 0 0 0 0
q4 (E) 50069504 498796 698314400 0 0
q5 (E) 50069504 0 0 0 0
q6 (E) 50069504 0 0 0 0
q7 (E) 50069504 0 0 0 0
VTEP5#show qos-profile DSCP_QUEUE
profile name: DSCP_QUEUE
profile type: dscp-to-queue
profile attached to 1 instances
configured mapping:
dscp 20 queue 4
Detailed mapping:
-------+--------------------------- | -------+--------------------------- | -------+--------------------------- | -------+---------------------------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
-------+--------------------------- | -------+--------------------------- | -------+--------------------------- | -------+---------------------------
DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP
-------+-------+--------+---------- | -------+-------+--------+---------- | -------+-------+--------+---------- | -------+-------+--------+----------
0 0 green 0 | 16 2 green 16 | 32 4 green 32 | 48 6 green 48
1 0 green 1 | 17 2 green 17 | 33 4 green 33 | 49 6 green 49
2 0 green 2 | 18 2 green 18 | 34 4 green 34 | 50 6 green 50
3 0 green 3 | 19 2 green 19 | 35 4 green 35 | 51 6 green 51
4 0 green 4 | 20 4 yellow 20 | 36 4 yellow 36 | 52 6 green 52
5 0 green 5 | 21 2 green 21 | 37 4 green 37 | 53 6 green 53
6 0 green 6 | 22 2 yellow 22 | 38 4 yellow 38 | 54 6 green 54
7 0 green 7 | 23 2 green 23 | 39 4 green 39 | 55 6 green 55
8 1 green 8 | 24 3 green 24 | 40 5 green 40 | 56 7 green 56
9 1 green 9 | 25 3 green 25 | 41 5 green 41 | 57 7 green 57
10 1 green 10 | 26 3 green 26 | 42 5 green 42 | 58 7 green 58
11 1 green 11 | 27 3 green 27 | 43 5 green 43 | 59 7 green 59
12 1 yellow 12 | 28 3 yellow 28 | 44 5 green 44 | 60 7 green 60
13 1 green 13 | 29 3 green 29 | 45 5 green 45 | 61 7 green 61
14 1 yellow 14 | 30 3 yellow 30 | 46 5 green 46 | 62 7 green 62
15 1 green 15 | 31 3 green 31 | 47 5 green 47 | 63 7 green 63
VTEP5#show qos-profile DSCP_ENCAP
profile name: DSCP_ENCAP
profile type: dscp-encap
profile attached to 1 instances
configured mapping:
l3 dscp 20 dscpEncap 56
Detailed mapping:
L3 DSCP to DSCP-ENCAP
-------+-------- | -------+-------- | -------+-------- | -------+--------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
-------|-------- | -------|-------- | -------|-------- | -------|--------
DSCP | DSCP | DSCP | DSCP | DSCP | DSCP | DSCP | DSCP
-------+-------- | -------+-------- | -------+-------- | -------+--------
0 0 | 16 16 | 32 32 | 48 48
1 1 | 17 17 | 33 33 | 49 49
2 2 | 18 18 | 34 34 | 50 50
3 3 | 19 19 | 35 35 | 51 51
4 4 | 20 56 | 36 36 | 52 52
5 5 | 21 21 | 37 37 | 53 53
6 6 | 22 22 | 38 38 | 54 54
7 7 | 23 23 | 39 39 | 55 55
8 8 | 24 24 | 40 40 | 56 56
9 9 | 25 25 | 41 41 | 57 57
10 10 | 26 26 | 42 42 | 58 58
11 11 | 27 27 | 43 43 | 59 59
12 12 | 28 28 | 44 44 | 60 60
13 13 | 29 29 | 45 45 | 61 61
14 14 | 30 30 | 46 46 | 62 62
15 15 | 31 31 | 47 47 | 63 63
L2 Queue + Color to DSCP-ENCAP
---------------+--------| ----------------+--------| --------------+--------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
---------------+--------| ----------------+--------| --------------+--------
Queue | Color | DSCP | Queue | Color | DSCP | Queue | Color | DSCP
-------+-------+--------| -------+--------+--------| ------+-------+--------
0 green 0 | 0 yellow 0 | 0 red 0
1 green 8 | 1 yellow 8 | 1 red 8
2 green 16 | 2 yellow 16 | 2 red 16
3 green 24 | 3 yellow 24 | 3 red 24
4 green 32 | 4 yellow 32 | 4 red 32
5 green 40 | 5 yellow 40 | 5 red 40
6 green 48 | 6 yellow 48 | 6 red 48
7 green 56 | 7 yellow 56 | 7 red 56
VTEP5#show qos-profile interface irb1001
profile name: DSCP_QUEUE
profile type: dscp-to-queue (Ingress)
mapping:
-------+--------------------------- | -------+--------------------------- | -------+--------------------------- | -------+---------------------------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
-------+--------------------------- | -------+--------------------------- | -------+--------------------------- | -------+---------------------------
DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP
-------+-------+--------+---------- | -------+-------+--------+---------- | -------+-------+--------+---------- | -------+-------+--------+----------
0 0 green 0 | 16 2 green 16 | 32 4 green 32 | 48 6 green 48
1 0 green 1 | 17 2 green 17 | 33 4 green 33 | 49 6 green 49
2 0 green 2 | 18 2 green 18 | 34 4 green 34 | 50 6 green 50
3 0 green 3 | 19 2 green 19 | 35 4 green 35 | 51 6 green 51
4 0 green 4 | 20 4 yellow 20 | 36 4 yellow 36 | 52 6 green 52
5 0 green 5 | 21 2 green 21 | 37 4 green 37 | 53 6 green 53
6 0 green 6 | 22 2 yellow 22 | 38 4 yellow 38 | 54 6 green 54
7 0 green 7 | 23 2 green 23 | 39 4 green 39 | 55 6 green 55
8 1 green 8 | 24 3 green 24 | 40 5 green 40 | 56 7 green 56
9 1 green 9 | 25 3 green 25 | 41 5 green 41 | 57 7 green 57
10 1 green 10 | 26 3 green 26 | 42 5 green 42 | 58 7 green 58
11 1 green 11 | 27 3 green 27 | 43 5 green 43 | 59 7 green 59
12 1 yellow 12 | 28 3 yellow 28 | 44 5 green 44 | 60 7 green 60
13 1 green 13 | 29 3 green 29 | 45 5 green 45 | 61 7 green 61
14 1 yellow 14 | 30 3 yellow 30 | 46 5 green 46 | 62 7 green 62
15 1 green 15 | 31 3 green 31 | 47 5 green 47 | 63 7 green 63
profile name: default
profile type: dscp-to-dscp (Egress)
Status: Inactive
mapping:
----------------+---------- | ----------------+---------- | ----------------+----------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
----------------+---------- | ----------------+---------- | ----------------+----------
DSCP | Color | Out DSCP | DSCP | Color | Out DSCP | DSCP | Color | Out DSCP
------+---------+---------- | ------+---------+---------- | ------+---------+----------
0 green 0 | 0 yellow 0 | 0 red 0
1 green 1 | 1 yellow 1 | 1 red 1
2 green 2 | 2 yellow 2 | 2 red 2
3 green 3 | 3 yellow 3 | 3 red 3
4 green 4 | 4 yellow 4 | 4 red 4
5 green 5 | 5 yellow 5 | 5 red 5
6 green 6 | 6 yellow 6 | 6 red 6
7 green 7 | 7 yellow 7 | 7 red 7
8 green 8 | 8 yellow 8 | 8 red 8
9 green 9 | 9 yellow 9 | 9 red 9
10 green 10 | 10 yellow 12 | 10 red 14
11 green 11 | 11 yellow 11 | 11 red 11
12 green 12 | 12 yellow 12 | 12 red 14
13 green 13 | 13 yellow 13 | 13 red 13
14 green 14 | 14 yellow 14 | 14 red 14
15 green 15 | 15 yellow 15 | 15 red 15
16 green 16 | 16 yellow 16 | 16 red 16
17 green 17 | 17 yellow 17 | 17 red 17
18 green 18 | 18 yellow 20 | 18 red 22
19 green 19 | 19 yellow 19 | 19 red 19
20 green 20 | 20 yellow 20 | 20 red 22
21 green 21 | 21 yellow 21 | 21 red 21
22 green 22 | 22 yellow 22 | 22 red 22
23 green 23 | 23 yellow 23 | 23 red 23
24 green 24 | 24 yellow 24 | 24 red 24
25 green 25 | 25 yellow 25 | 25 red 25
26 green 26 | 26 yellow 28 | 26 red 30
27 green 27 | 27 yellow 27 | 27 red 27
28 green 28 | 28 yellow 28 | 28 red 30
29 green 29 | 29 yellow 29 | 29 red 29
30 green 30 | 30 yellow 30 | 30 red 30
31 green 31 | 31 yellow 31 | 31 red 31
32 green 32 | 32 yellow 32 | 32 red 32
33 green 33 | 33 yellow 33 | 33 red 33
34 green 34 | 34 yellow 36 | 34 red 38
35 green 35 | 35 yellow 35 | 35 red 35
36 green 36 | 36 yellow 36 | 36 red 38
37 green 37 | 37 yellow 37 | 37 red 37
38 green 38 | 38 yellow 38 | 38 red 38
39 green 39 | 39 yellow 39 | 39 red 39
40 green 40 | 40 yellow 40 | 40 red 40
41 green 41 | 41 yellow 41 | 41 red 41
42 green 42 | 42 yellow 42 | 42 red 42
43 green 43 | 43 yellow 43 | 43 red 43
44 green 44 | 44 yellow 44 | 44 red 44
45 green 45 | 45 yellow 45 | 45 red 45
46 green 46 | 46 yellow 46 | 46 red 46
47 green 47 | 47 yellow 47 | 47 red 47
48 green 48 | 48 yellow 48 | 48 red 48
49 green 49 | 49 yellow 49 | 49 red 49
50 green 50 | 50 yellow 50 | 50 red 50
51 green 51 | 51 yellow 51 | 51 red 51
52 green 52 | 52 yellow 52 | 52 red 52
53 green 53 | 53 yellow 53 | 53 red 53
54 green 54 | 54 yellow 54 | 54 red 54
55 green 55 | 55 yellow 55 | 55 red 55
56 green 56 | 56 yellow 56 | 56 red 56
57 green 57 | 57 yellow 57 | 57 red 57
58 green 58 | 58 yellow 58 | 58 red 58
59 green 59 | 59 yellow 59 | 59 red 59
60 green 60 | 60 yellow 60 | 60 red 60
61 green 61 | 61 yellow 61 | 61 red 61
62 green 62 | 62 yellow 62 | 62 red 62
63 green 63 | 63 yellow 63 | 63 red 63
VTEP4
VTEP4#show run qos
qos enable
qos remark dscp
!
!
qos profile queue-color-to-cos QUEUE_COS
queue 6 color all cos 2
!
qos profile dscp-to-queue DSCP_QUEUE
dscp 56 queue 6
!
qos profile dscp-to-dscp DSCP_DSCP
dscp 20 color all dscp 32
!
!
!
!
interface irb2001
qos map-profile dscp-to-dscp DSCP_DSCP
!
VTEP4#show run nvo vxlan
!
nvo vxlan enable
!
nvo vxlan irb
!
evpn vxlan multihoming enable
!
nvo vxlan vtep-ip-global 4.4.4.4
!
nvo vxlan tunnel qos-map-mode cos-dscp ingress DSCP_QUEUE
!
nvo vxlan id 101 ingress-replication inner-vid-disabled
vxlan host-reachability-protocol evpn-bgp L2VRF1
vni-name VNI-101
!
nvo vxlan id 201 ingress-replication inner-vid-disabled
vxlan host-reachability-protocol evpn-bgp L2VRF2
evpn irb2001
vni-name VNI-201
!
nvo vxlan access-if port-vlan xe3 20
map vnid 201
mac 0000.5555.1020 ip 21.21.21.101
map qos-profile queue-color-to-cos QUEUE_COS
!
!
VTEP4#show nvo vxlan tunnel
VXLAN Network tunnel Entries
Source Destination Status Up/Down Update
============================================================================
4.4.4.4 2.2.2.2 Installed 00:28:32 00:28:32
4.4.4.4 1.1.1.1 Installed 00:28:38 00:28:38
4.4.4.4 5.5.5.5 Installed 00:28:31 00:28:31
Total number of entries are 3
VTEP4#show nvo vxlan
VXLAN Information
=================
Codes: NW - Network Port
AC - Access Port
(u) - Untagged
VNID VNI-Name VNI-Type Type Interface ESI VLAN DF-Status Src-Addr Dst-Addr
_______________________________________________________________________________________________________________________________
101 VNI-101 L2 NW ---- ---- ---- ---- 4.4.4.4 2.2.2.2
101 VNI-101 L2 NW ---- ---- ---- ---- 4.4.4.4 1.1.1.1
101 VNI-101 L2 NW ---- ---- ---- ---- 4.4.4.4 5.5.5.5
201 VNI-201 L2 NW ---- ---- ---- ---- 4.4.4.4 2.2.2.2
201 VNI-201 L2 NW ---- ---- ---- ---- 4.4.4.4 1.1.1.1
201 VNI-201 L2 NW ---- ---- ---- ---- 4.4.4.4 5.5.5.5
201 VNI-201 -- AC xe3 --- Single Homed Port --- 20 ---- ---- ----
1000 ---- L3 NW ---- ---- ---- ---- 4.4.4.4 5.5.5.5
Total number of entries are 8
VTEP4#show nvo vxlan arp-cache
VXLAN ARP-CACHE Information
===========================
VNID Ip-Addr Mac-Addr Type Age-Out Retries-Left
____________________________________________________________________________
201 21.21.21.51 0000.2222.1020 Static Remote ----
201 21.21.21.1 3c2c.991a.da7a Static Local ----
201 21.21.21.101 0000.5555.1020 Static Local ----
101 11.11.11.51 0000.2222.1010 Static Remote ----
101 11.11.11.1 04f8.f82f.8eee Static Remote ----
101 11.11.11.201 0000.4444.1010 Static Remote ----
Total number of entries are 6
VTEP4#show nvo vxlan nd-cache
VXLAN ND-CACHE Information
===========================
VNID Ip-Addr Mac-Addr Type Age-Out Retries-Left
____________________________________________________________________________________________________
201 2121::1 3c2c.991a.da7a Static Local ----
101 1111::1 04f8.f82f.8eee Static Remote ----
Total number of entries are 2
VTEP4#show ipv4 route vrf L3VRF1
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2,
ia - IS-IS inter area, E - EVPN,
v - vrf leaked
* - candidate default
IP Route Table for VRF "L3VRF1"
B 5.5.5.5/32 [0/0] is directly connected, tunvxlan1000, 00:01:03
B 11.11.11.0/24 [200/0] via 5.5.5.5 (recursive is directly connected, tunvxlan1000), 00:01:03
C 21.21.21.0/24 is directly connected, irb2001, 00:03:31
C 127.0.0.0/8 is directly connected, lo.L3VRF1, 00:15:13
Gateway of last resort is not set
VTEP4#show ipv6 route vrf L3VRF1
IPv6 Routing Table
Codes: K - kernel route, C - connected, S - static, D- DHCP, R - RIP,
O - OSPF, IA - OSPF inter area, E1 - OSPF external type 1,
E2 - OSPF external type 2, E - EVPN N1 - OSPF NSSA external type 1,
N2 - OSPF NSSA external type 2, i - IS-IS, B - BGP,
v - vrf leaked
Timers: Uptime
IP Route Table for VRF "L3VRF1"
C ::1/128 via ::, lo.L3VRF1, 00:15:18
B ::ffff:505:505/128 [0/0] via ::, tunvxlan1000, 00:01:08
B 1111::/64 [200/0] via ::ffff:505:505 (recursive via ::, unknown), 00:01:08
C 2121::/64 via ::, irb2001, 00:03:36
C fe80::/64 via ::, irb2001, 00:03:36
VTEP4#show nvo vxlan l3vni-map
L3VNI L2VNI IRB-interface
===================================
1000 201 irb2001
VTEP4#show interface xe3 counter queue-stats
E - Egress, I - Ingress, Q-Size is in bytes
+--------------------+--------+-----------------+-------------------+-----------------+-------------------+
| Queue/Class-map | Q-Size | Tx pkts | Tx bytes | Dropped pkts | Dropped bytes |
+--------------------+--------+-----------------+-------------------+-----------------+-------------------+
q0 (E) 12517376 0 0 0 0
q1 (E) 12517376 0 0 0 0
q2 (E) 12517376 0 0 0 0
q3 (E) 12517376 0 0 0 0
q4 (E) 12517376 0 0 0 0
q5 (E) 12517376 0 0 0 0
q6 (E) 12517376 2219303 3209162748 0 0
q7 (E) 12517376 0 0 0 0
VTEP4#show qos-profile QUEUE_COS
profile name: QUEUE_COS
profile type: queue-color-to-cos
profile attached to 1 instances
configured mapping:
queue 6 color all cos 2
Detailed mapping:
----------------+-------- | ----------------+-------- | ----------------+--------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
----------------+-------- | ----------------+-------- | ----------------+--------
Queue | Color | COS | Queue | Color | COS | Queue | Color | COS
-------+--------+-------- | -------+--------+-------- | -------+--------+--------
0 green 0 | 0 yellow 0 | 0 red 0
1 green 1 | 1 yellow 1 | 1 red 1
2 green 2 | 2 yellow 2 | 2 red 2
3 green 3 | 3 yellow 3 | 3 red 3
4 green 4 | 4 yellow 4 | 4 red 4
5 green 5 | 5 yellow 5 | 5 red 5
6 green 2 | 6 yellow 2 | 6 red 2
7 green 7 | 7 yellow 7 | 7 red 7
VTEP4#show qos-profile DSCP_QUEUE
profile name: DSCP_QUEUE
profile type: dscp-to-queue
profile attached to 1 instances
configured mapping:
dscp 56 queue 6
Detailed mapping:
-------+--------------------------- | -------+--------------------------- | -------+--------------------------- | -------+---------------------------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
-------+--------------------------- | -------+--------------------------- | -------+--------------------------- | -------+---------------------------
DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP
-------+-------+--------+---------- | -------+-------+--------+---------- | -------+-------+--------+---------- | -------+-------+--------+----------
0 0 green 0 | 16 2 green 16 | 32 4 green 32 | 48 6 green 48
1 0 green 1 | 17 2 green 17 | 33 4 green 33 | 49 6 green 49
2 0 green 2 | 18 2 green 18 | 34 4 green 34 | 50 6 green 50
3 0 green 3 | 19 2 green 19 | 35 4 green 35 | 51 6 green 51
4 0 green 4 | 20 2 yellow 20 | 36 4 yellow 36 | 52 6 green 52
5 0 green 5 | 21 2 green 21 | 37 4 green 37 | 53 6 green 53
6 0 green 6 | 22 2 yellow 22 | 38 4 yellow 38 | 54 6 green 54
7 0 green 7 | 23 2 green 23 | 39 4 green 39 | 55 6 green 55
8 1 green 8 | 24 3 green 24 | 40 5 green 40 | 56 6 green 56
9 1 green 9 | 25 3 green 25 | 41 5 green 41 | 57 7 green 57
10 1 green 10 | 26 3 green 26 | 42 5 green 42 | 58 7 green 58
11 1 green 11 | 27 3 green 27 | 43 5 green 43 | 59 7 green 59
12 1 yellow 12 | 28 3 yellow 28 | 44 5 green 44 | 60 7 green 60
13 1 green 13 | 29 3 green 29 | 45 5 green 45 | 61 7 green 61
14 1 yellow 14 | 30 3 yellow 30 | 46 5 green 46 | 62 7 green 62
15 1 green 15 | 31 3 green 31 | 47 5 green 47 | 63 7 green 63
VTEP4#show qos-profile DSCP_DSCP
profile name: DSCP_DSCP
profile type: dscp-to-dscp
profile attached to 1 instances
configured mapping:
dscp 20 color all dscp 32
Detailed mapping:
----------------+---------- | ----------------+---------- | ----------------+----------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
----------------+---------- | ----------------+---------- | ----------------+----------
DSCP | Color | Out DSCP | DSCP | Color | Out DSCP | DSCP | Color | Out DSCP
------+---------+---------- | ------+---------+---------- | ------+---------+----------
0 green 0 | 0 yellow 0 | 0 red 0
1 green 1 | 1 yellow 1 | 1 red 1
2 green 2 | 2 yellow 2 | 2 red 2
3 green 3 | 3 yellow 3 | 3 red 3
4 green 4 | 4 yellow 4 | 4 red 4
5 green 5 | 5 yellow 5 | 5 red 5
6 green 6 | 6 yellow 6 | 6 red 6
7 green 7 | 7 yellow 7 | 7 red 7
8 green 8 | 8 yellow 8 | 8 red 8
9 green 9 | 9 yellow 9 | 9 red 9
10 green 10 | 10 yellow 12 | 10 red 14
11 green 11 | 11 yellow 11 | 11 red 11
12 green 12 | 12 yellow 12 | 12 red 14
13 green 13 | 13 yellow 13 | 13 red 13
14 green 14 | 14 yellow 14 | 14 red 14
15 green 15 | 15 yellow 15 | 15 red 15
16 green 16 | 16 yellow 16 | 16 red 16
17 green 17 | 17 yellow 17 | 17 red 17
18 green 18 | 18 yellow 20 | 18 red 22
19 green 19 | 19 yellow 19 | 19 red 19
20 green 32 | 20 yellow 32 | 20 red 32
21 green 21 | 21 yellow 21 | 21 red 21
22 green 22 | 22 yellow 22 | 22 red 22
23 green 23 | 23 yellow 23 | 23 red 23
24 green 24 | 24 yellow 24 | 24 red 24
25 green 25 | 25 yellow 25 | 25 red 25
26 green 26 | 26 yellow 28 | 26 red 30
27 green 27 | 27 yellow 27 | 27 red 27
28 green 28 | 28 yellow 28 | 28 red 30
29 green 29 | 29 yellow 29 | 29 red 29
30 green 30 | 30 yellow 30 | 30 red 30
31 green 31 | 31 yellow 31 | 31 red 31
32 green 32 | 32 yellow 32 | 32 red 32
33 green 33 | 33 yellow 33 | 33 red 33
34 green 34 | 34 yellow 36 | 34 red 38
35 green 35 | 35 yellow 35 | 35 red 35
36 green 36 | 36 yellow 36 | 36 red 38
37 green 37 | 37 yellow 37 | 37 red 37
38 green 38 | 38 yellow 38 | 38 red 38
39 green 39 | 39 yellow 39 | 39 red 39
40 green 40 | 40 yellow 40 | 40 red 40
41 green 41 | 41 yellow 41 | 41 red 41
42 green 42 | 42 yellow 42 | 42 red 42
43 green 43 | 43 yellow 43 | 43 red 43
44 green 44 | 44 yellow 44 | 44 red 44
45 green 45 | 45 yellow 45 | 45 red 45
46 green 46 | 46 yellow 46 | 46 red 46
47 green 47 | 47 yellow 47 | 47 red 47
48 green 48 | 48 yellow 48 | 48 red 48
49 green 49 | 49 yellow 49 | 49 red 49
50 green 50 | 50 yellow 50 | 50 red 50
51 green 51 | 51 yellow 51 | 51 red 51
52 green 52 | 52 yellow 52 | 52 red 52
53 green 53 | 53 yellow 53 | 53 red 53
54 green 54 | 54 yellow 54 | 54 red 54
55 green 55 | 55 yellow 55 | 55 red 55
56 green 56 | 56 yellow 56 | 56 red 56
57 green 57 | 57 yellow 57 | 57 red 57
58 green 58 | 58 yellow 58 | 58 red 58
59 green 59 | 59 yellow 59 | 59 red 59
60 green 60 | 60 yellow 60 | 60 red 60
61 green 61 | 61 yellow 61 | 61 red 61
62 green 62 | 62 yellow 62 | 62 red 62
63 green 63 | 63 yellow 63 | 63 red 63
VTEP4#show qos-profile interface irb2001
profile name: default
profile type: dscp-to-queue (Ingress)
mapping:
-------+--------------------------- | -------+--------------------------- | -------+--------------------------- | -------+---------------------------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
-------+--------------------------- | -------+--------------------------- | -------+--------------------------- | -------+---------------------------
DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP | DSCP | Queue | Color | Out DSCP
-------+-------+--------+---------- | -------+-------+--------+---------- | -------+-------+--------+---------- | -------+-------+--------+----------
0 0 green 0 | 16 2 green 16 | 32 4 green 32 | 48 6 green 48
1 0 green 1 | 17 2 green 17 | 33 4 green 33 | 49 6 green 49
2 0 green 2 | 18 2 green 18 | 34 4 green 34 | 50 6 green 50
3 0 green 3 | 19 2 green 19 | 35 4 green 35 | 51 6 green 51
4 0 green 4 | 20 2 yellow 20 | 36 4 yellow 36 | 52 6 green 52
5 0 green 5 | 21 2 green 21 | 37 4 green 37 | 53 6 green 53
6 0 green 6 | 22 2 yellow 22 | 38 4 yellow 38 | 54 6 green 54
7 0 green 7 | 23 2 green 23 | 39 4 green 39 | 55 6 green 55
8 1 green 8 | 24 3 green 24 | 40 5 green 40 | 56 7 green 56
9 1 green 9 | 25 3 green 25 | 41 5 green 41 | 57 7 green 57
10 1 green 10 | 26 3 green 26 | 42 5 green 42 | 58 7 green 58
11 1 green 11 | 27 3 green 27 | 43 5 green 43 | 59 7 green 59
12 1 yellow 12 | 28 3 yellow 28 | 44 5 green 44 | 60 7 green 60
13 1 green 13 | 29 3 green 29 | 45 5 green 45 | 61 7 green 61
14 1 yellow 14 | 30 3 yellow 30 | 46 5 green 46 | 62 7 green 62
15 1 green 15 | 31 3 green 31 | 47 5 green 47 | 63 7 green 63
profile name: DSCP_DSCP
profile type: dscp-to-dscp (Egress)
Status: Active
mapping:
----------------+---------- | ----------------+---------- | ----------------+----------
INPUT | OUTPUT | INPUT | OUTPUT | INPUT | OUTPUT
----------------+---------- | ----------------+---------- | ----------------+----------
DSCP | Color | Out DSCP | DSCP | Color | Out DSCP | DSCP | Color | Out DSCP
------+---------+---------- | ------+---------+---------- | ------+---------+----------
0 green 0 | 0 yellow 0 | 0 red 0
1 green 1 | 1 yellow 1 | 1 red 1
2 green 2 | 2 yellow 2 | 2 red 2
3 green 3 | 3 yellow 3 | 3 red 3
4 green 4 | 4 yellow 4 | 4 red 4
5 green 5 | 5 yellow 5 | 5 red 5
6 green 6 | 6 yellow 6 | 6 red 6
7 green 7 | 7 yellow 7 | 7 red 7
8 green 8 | 8 yellow 8 | 8 red 8
9 green 9 | 9 yellow 9 | 9 red 9
10 green 10 | 10 yellow 12 | 10 red 14
11 green 11 | 11 yellow 11 | 11 red 11
12 green 12 | 12 yellow 12 | 12 red 14
13 green 13 | 13 yellow 13 | 13 red 13
14 green 14 | 14 yellow 14 | 14 red 14
15 green 15 | 15 yellow 15 | 15 red 15
16 green 16 | 16 yellow 16 | 16 red 16
17 green 17 | 17 yellow 17 | 17 red 17
18 green 18 | 18 yellow 20 | 18 red 22
19 green 19 | 19 yellow 19 | 19 red 19
20 green 32 | 20 yellow 32 | 20 red 32
21 green 21 | 21 yellow 21 | 21 red 21
22 green 22 | 22 yellow 22 | 22 red 22
23 green 23 | 23 yellow 23 | 23 red 23
24 green 24 | 24 yellow 24 | 24 red 24
25 green 25 | 25 yellow 25 | 25 red 25
26 green 26 | 26 yellow 28 | 26 red 30
27 green 27 | 27 yellow 27 | 27 red 27
28 green 28 | 28 yellow 28 | 28 red 30
29 green 29 | 29 yellow 29 | 29 red 29
30 green 30 | 30 yellow 30 | 30 red 30
31 green 31 | 31 yellow 31 | 31 red 31
32 green 32 | 32 yellow 32 | 32 red 32
33 green 33 | 33 yellow 33 | 33 red 33
34 green 34 | 34 yellow 36 | 34 red 38
35 green 35 | 35 yellow 35 | 35 red 35
36 green 36 | 36 yellow 36 | 36 red 38
37 green 37 | 37 yellow 37 | 37 red 37
38 green 38 | 38 yellow 38 | 38 red 38
39 green 39 | 39 yellow 39 | 39 red 39
40 green 40 | 40 yellow 40 | 40 red 40
41 green 41 | 41 yellow 41 | 41 red 41
42 green 42 | 42 yellow 42 | 42 red 42
43 green 43 | 43 yellow 43 | 43 red 43
44 green 44 | 44 yellow 44 | 44 red 44
45 green 45 | 45 yellow 45 | 45 red 45
46 green 46 | 46 yellow 46 | 46 red 46
47 green 47 | 47 yellow 47 | 47 red 47
48 green 48 | 48 yellow 48 | 48 red 48
49 green 49 | 49 yellow 49 | 49 red 49
50 green 50 | 50 yellow 50 | 50 red 50
51 green 51 | 51 yellow 51 | 51 red 51
52 green 52 | 52 yellow 52 | 52 red 52
53 green 53 | 53 yellow 53 | 53 red 53
54 green 54 | 54 yellow 54 | 54 red 54
55 green 55 | 55 yellow 55 | 55 red 55
56 green 56 | 56 yellow 56 | 56 red 56
57 green 57 | 57 yellow 57 | 57 red 57
58 green 58 | 58 yellow 58 | 58 red 58
59 green 59 | 59 yellow 59 | 59 red 59
60 green 60 | 60 yellow 60 | 60 red 60
61 green 61 | 61 yellow 61 | 61 red 61
62 green 62 | 62 yellow 62 | 62 red 62
63 green 63 | 63 yellow 63 | 63 red 63