EVE-NG – SONiC switch configuration via CLI

This post will look at the configuration steps to get a functioning SONiC switch lab using the CLI. The topology will be a non-redundant pair of clusters each containing two leaf switches connected to a spine switch. The spine switches are connected to an agregation switch. The switches are all running at Layer3 with eBGP used for prefix redistibution.

First thing to do is change the default password, personally I find the default SONiC camelcase credentials a pain to type in (admin/YourPaSsWoRd), this can be changed with the linux passwd command.

The default running config comes with a lot of additional items which are not required. The first step is to replace this ‘t1’ config with a ‘l3’ config:

sudo cp /etc/sonic/config_db.json ~/OLD-config_db.json
sudo sonic-cfggen -H --preset l3 -k Force10-S6000 -p /usr/share/sonic/device/x86_64-kvm_x86_64-r0/Force10-S6000/port_config.ini > ~/tmp.json
sudo mv ~/tmp.json /etc/sonic/config_db.json
sudo config reload -y

Next create the management VRF which will automatically attach the EVE-NG Ethernet0 interface to it:

sudo config vrf add mgmt
sudo config save

The switchports between the switches will be routed using /31 prefixes. It is possible to use the sonic-cli utility but I have found that it has a reduce set of configuration commands it it tends to throw up errors when applying them. Instead we will use the shell commands. It is worth pointing out that the Ethernet interface IDs go up in multiples of four. At this point EVE-NG Ethernet0 is bound to eth0 on the SONiC switch, so EVE-NG Ethernet1 will be bound to Ethernet0 (fortyGigE0/0) on the SONiC switch.

sudo config interface ip add Ethernet4 169.254.2.3/31
sudo config save -y

You should also take the time to configure the default loopback0 IP address to something unique fitting your own addressing scheme:

sudo config interface ip remove Loopback0 10.1.0.1/32
sudo config interface ip add Loopback0 10.1.1.1/32

BGP configuration

To confuse things further, routing protocol configuration is not performed via the bash console or sonic-cli but instead via the FRR utility which is accessed with the command vtysh . Before you go ahead and begin configuration with FRR you must update /etc/sonic/config_db.json with the following line, then reload and save:

    "DEVICE_METADATA": {
        "localhost": {
            "docker_routing_config_mode": "split"
        }
    },

This will stop your configuration from being overrwirrten with a builtin template each time the FFR docker container is restarted. The docker_init.sh lists other options unified and seperated, both of which overwrite any FRR routing protocol configuration. Not sure why you would want that!?

admin@C02-T1-01:~$ vtysh

Hello, this is FRRouting (version 7.5.1-sonic).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

C02-T1-01# configure
C02-T1-01(config)# router bgp 64802
C02-T1-01(config-router)# neighbor 169.254.2.3 remote-as 64611
C02-T1-01(config-router)# end
C02-T1-01# wr mem
Note: this version of vtysh never writes vtysh.conf
Building Configuration...
Configuration saved to /etc/frr/zebra.conf
Configuration saved to /etc/frr/bgpd.conf
Configuration saved to /etc/frr/staticd.conf
C02-T1-01#

You will notice that when you excute the show bgp summary command from within vtysh, each neighbour reports ‘(policy) (policy)‘ indicating that no route-map has been applied for those peers.

C02-T1-01# show ip bgp summ

IPv4 Unicast Summary:
BGP router identifier 10.1.1.1, local AS number 64802 vrf-id 0
BGP table version 0
RIB entries 0, using 0 bytes of memory
Peers 3, using 64 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
169.254.2.1     4      64610         7         8        0    0    0 00:00:02     (Policy) (Policy)
169.254.2.3     4      64611        22        23        0    0    0 00:00:02     (Policy) (Policy)
169.254.3.2     4      64901        51        46        0    0    0 00:00:02     (Policy) (Policy)

Total number of neighbors 3

FRR will discard all BGP updates to peers if a policy is not applied to the peer.

C02-T1-01# show ip bgp neighbors 169.254.2.1
BGP neighbor is 169.254.2.1, remote AS 64610, local AS 64802, external link
 Description: C02-T0-01
Hostname: C02-T0-01
  BGP version 4, remote router ID 10.1.0.1, local router ID 10.1.1.1
  BGP state = Established, up for 00:00:47
  Last read 00:00:46, Last write 00:00:46
  Hold time is 180, keepalive interval is 60 seconds
  Neighbor capabilities:
    4 Byte AS: advertised and received
    AddPath:
      IPv4 Unicast: RX advertised IPv4 Unicast and received
    Route refresh: advertised and received(old & new)
    Address Family IPv4 Unicast: advertised and received
    Hostname Capability: advertised (name: C02-T1-01,domain name: n/a) received (name: C02-T0-01,domain name: n/a)
    Graceful Restart Capability: advertised and received
      Remote Restart timer is 120 seconds
      Address families by peer:
        none
  Graceful restart information:
    End-of-RIB send: IPv4 Unicast
    End-of-RIB received: IPv4 Unicast
    Local GR Mode: Helper*
    Remote GR Mode: Helper
    R bit: True
    Timers:
      Configured Restart Time(sec): 120
      Received Restart Time(sec): 120
    IPv4 Unicast:
      F bit: False
      End-of-RIB sent: Yes
      End-of-RIB sent after update: Yes
      End-of-RIB received: Yes
      Timers:
        Configured Stale Path Time(sec): 360
  Message statistics:
    Inq depth is 0
    Outq depth is 0
                         Sent       Rcvd
    Opens:                  2          2
    Notifications:          2          0
    Updates:                2          2
    Keepalives:             2          2
    Route Refresh:          0          1
    Capability:             0          0
    Total:                  8          7
  Minimum time between advertisement runs is 0 seconds

 For address family: IPv4 Unicast
  Update group 3, subgroup 4
  Packet Queue length 0
  Community attribute sent to this neighbor(all)
  Inbound updates discarded due to missing policy
  Outbound updates discarded due to missing policy
  0 accepted prefixes

  Connections established 2; dropped 1
  Last reset 00:00:49,  No AFI/SAFI activated for peer
Local host: 169.254.2.0, Local port: 179
Foreign host: 169.254.2.1, Foreign port: 43866
Nexthop: 169.254.2.0
Nexthop global: fe80::5200:ff:fe0a:0
Nexthop local: fe80::5200:ff:fe0a:0
BGP connection: shared network
BGP Connect Retry Timer in Seconds: 120
Read thread: on  Write thread: on  FD used: 30

You can disable warnings (no bgp ebgp-requires-policy), but it is best practice to apply both an inbound and outbound for BGP peers. The route-map below will only accept a default route from the T2 switch and advertise it towards the T0 switches. It will only accept prefixes from the T0 switches with an AS PATH which only includes the peers ASN. It will advertise all prefixes towards the T2 switch:

!
router bgp 64802
 neighbor 169.254.2.1 remote-as 64610
 neighbor 169.254.2.1 description C02-T0-01
 neighbor 169.254.2.3 remote-as 64611
 neighbor 169.254.2.3 description C02-T0-02
 neighbor 169.254.3.2 remote-as 64901
 neighbor 169.254.3.2 description T2-01
 !
 address-family ipv4 unicast
  neighbor 169.254.2.1 route-map C02-T0-01 in
  neighbor 169.254.2.1 route-map DEFAULT out
  neighbor 169.254.2.3 route-map C02-T0-02 in
  neighbor 169.254.2.3 route-map DEFAULT out
  neighbor 169.254.3.2 route-map DEFAULT in
  neighbor 169.254.3.2 route-map ALLOW-ALL out
 exit-address-family
!
access-list ALLOW-ALL seq 5 permit any
!
ip prefix-list DEFAULT seq 5 permit 0.0.0.0/0
!
bgp as-path access-list C02-T0-01 permit ^64610$
bgp as-path access-list C02-T0-02 permit ^64611$
!
route-map C02-T0-01 permit 5
 match as-path C02-T0-01
!
route-map C02-T0-02 permit 5
 match as-path C02-T0-02
!
route-map DEFAULT permit 5
 match ip address DEFAULT
!
route-map ALLOW-ALL permit 5
 match ip address ALLOW-ALL
!

Once the route-maps have been applied show bgp summary will no longer have the policy warnings:

C02-T1-01# show bgp summary

IPv4 Unicast Summary:
BGP router identifier 10.1.1.1, local AS number 64802 vrf-id 0
BGP table version 7
RIB entries 1, using 192 bytes of memory
Peers 3, using 64 KiB of memory

Neighbor        V         AS   MsgRcvd   MsgSent   TblVer  InQ OutQ  Up/Down State/PfxRcd   PfxSnt
169.254.2.1     4      64610        15        15        0    0    0 00:00:01            1        1
169.254.2.3     4      64611         7         9        0    0    0 00:00:01            0        1
169.254.3.2     4      64901        45        41        0    0    0 00:00:01            1        1

The BGP process can then install routes:

C02-T1-01# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

B>* 0.0.0.0/0 [20/0] via 169.254.3.2, Ethernet8, weight 1, 00:01:56
C>* 10.1.1.1/32 is directly connected, Loopback0, 00:33:32
C>* 169.254.2.0/31 is directly connected, Ethernet0, 00:33:07
C>* 169.254.2.2/31 is directly connected, Ethernet4, 00:33:07
C>* 169.254.3.2/31 is directly connected, Ethernet8, 00:33:07
B>* 192.168.100.1/32 [20/0] via 169.254.2.1, Ethernet0, weight 1, 00:01:55

In the next post we’ll look at configuring the other half of the topology in a more programatic way.