Open Defects

Execute EPG create, delete and re-create CLI in quick succession as below:

1. Create EPG/Networks with user-provided VNI parameter.

2. Delete EPG.

3. Create EPG again with the same parameters as in step-1.

1. Create BD-LIF (without any associated VLAN) configuration manually on the switch followed by EFA inventory update.

2. Create EPG which results in the same BD-LIF on the switch.

1. Delete the EPG which is in cfg-refresh-err state

2. Clean up the out-of-baned created LIFs from the switch.

3. EFA inventory update to reflect the modified config from the switch.

4. Create the EPG again.

The second EPG will get created successfully but the anycast-ipv6 provided in the second EPG will not get configured on the switch.

The second EPG create command should have failed with an error mentioning a unique anycast-ipv6 needs to be provided for a given BD name.

1. Create BD based L3 EPG with some BD label and an anycast-ipv6.

2. Create another BD based L3 EPG with the same BD label (as used in 1) but with a different anycast-ipv6.

EFA configures both "suppress-arp" (needed for ipv4) and "suppress-nd" (needed for ipv6) for the particular network (VLAN/BD).

EFA shouldn't have configured "suppress-nd" on the network.

When VRF is added and deleted to/from and Endpoint Group, in quick succession, multiple times, events received from inventory service can get interleaved with the commands.

This can cause EFA command execution path to find database entries that are yet to be deleted due to previous command run.

(efa:root)root@ubuntu:~/efa/efa_40# efa inventory rma execute --ip 10.24.95.157 --co<tab>

- cli gets stuck and needs ^C^C to exit, as it is waiting on user input on backend

(efa:extreme)extreme@tpvm2:~$ efa fabric create --name fab4 – <tab>

- this creates a fabric

Execute EPG update "vrf-add", "vrf-delete" and "vrf-add" operation CLI in quick succession as below:

1. Update EPG for operation vrf-add.

2. Update EPG for operation vrf-delete.

3. Update the same EPG again with operation vrf-add for the same vrf which was deleted in step 2.

Allow firmware download process to complete. Status of the same can be checked using command

efa inventory device firmware-download show --fabric {fabric name}

Incrementally add networks to the EPG.

Example as below:

efa tenant epg create --tenant <> --name <> --ctag-range 2-256

efa tenant epg update --tenant <> --name <> --operation ctag-range-add --ctag-range 257-512

Incrementally delete networks from the EPG and finally delete EPG

Example as below:

efa tenant epg update --tenant <> --name <> --operation ctag-range-delete --ctag-range 1001-1256

efa tenant epg delete --tenant <> --name <>

Incrementally add networks to the EPG.

Example as below:

efa tenant epg create --tenant <> --name <> --ctag-range 2-1000

efa tenant epg update --tenant <> --name <> --operation ctag-range-add --ctag-range 1001-2000

Allow firmware download process to complete. Status of the same can be checked using command

efa inventory device firmware-download show --fabric {fabric name}

When the switch configurations drift from the intended configurations in EFA due to scenarios as follows:

1. L3 Epg is created with large ctag-range (e.g. 2-2000)

2. EFA configured VLANs and PO configurations are manually removed from the switch.

3. Switch is reloaded in maintenance mode

Remove the devices from fabric and re-add

efa fabric device remove --name <fabric-name> --ip <device-ips>

efa inventory device delete --ip <device-ips>

efa fabric device add-bulk --name <fabric-name> --rack <rack-name> --username <username> --password <password> --ip <device-ips>

This issue observed with below steps,

1)Created CLOS Fabric with multiple MCT-Pairs and a Spine device

2)When checking drift for all devices part of fabric, Cluster Config can be seen as refreshed, but no impact can be seen on fabric show. Also drift status for fabric states false in drift o/p.

With below steps issue can be observed.

- Create VRF/EPG having route target, static route and bgp configuration.

- Introduce drift in VRF route target or static route or bgp configuration on SLX switch.

- Update device from efa command "efa inventory device update --ip <device ip>"

- Check device drift using efa command as "efa inventory drift-reconcile execute --ip <device ip>"

- VRF shows as "cfg-in-sync" though its child configuration was drifted.

There are 2 recovery options:

1. Fix the reason for failure on the device and re-run the same CLI which will complete the operation and EPG/PO will back to the there expected state.

2. Use --force operation if EPG/PO needs to be completely deleted from EFA.

Tenant commands can take more time (> the timeout =1000 seconds) to complete and hence the failure.

The status of timed out fail command can be seen using the relevant "show" command.

The issue can happen in the below scenario:

1. EFA has a scaled tenant configuration.

2. Significant configuration drift introduced between SLX and EFA.

3. Inventory device update is performed resulting in configuration drift derivation of the tenant entities.

4. Tenant commands are executed when (3) is in progress.

Drift is shown for "pseudowire-profile" configuration in command,

"efa inventory drift-reconcile execute --ip <device ip>"

This is observed with below steps

- Create fabric with back routing enable and configure the fabric.

- Create epg and delete the same epg.

- Make one of cluster pair device as admin down.

- Create same epg which was created and deleted above.

- Make the admin down device as admin up.

- "pseudowire-profile" still remain as drifted.

To recover please follow below steps,

- Delete "pseudowire-profile" from SLX switch.

- Execute drift reconcile as "efa inventory drift-reconcile execute --ip <SLX switch IP> --reconcile"

Any delete nature CLI of "service BGP peer" has failed because any of the below-listed reasons will lead the BGP service to this state.

1. Device connectivity issues

2. Failed to deconfigure the peer on device

3. Targeted topology falls into the APS category.

There are no side effects of this issue except "update" operations on that BGP service will be allowed which is not expected behavior after the "service bgp peer delete" operation has failed.

There are 2 recovery options:

1. Fix the reason for failure and again try to delete BGP service.

2. Delete BGP service using "--force" option. In this case, configs will remain on admin down the device.

- Remove the VLAN configuration from the device

- Reload the device with "maintenance mode on reboot enabled"

- Description configuration is missing for reconciled VLANs

EFA CLI completion can be disabled. This does not affect EFA CLI operation or limit the commands available; it prevents convenient discovery of extra sub-commands and flags by using the tab key from within the bash shell. To disable, do the following:

- remove /etc/bash_completion.d/efa

- comment out or remove the line "source /etc/bash_completion.d/efa" from /etc/bash.bashrc

- start a new shell for EFA CLI use

The Drift and Reconcile can be run manually using the below command:

'efa inventory drift-reconcile execute '

Executing the command “efa-sync execute” when one of the SLX switches in cluster is unreachable, could take several minutes to complete.

This happens only when multiple neutron entities have to be synced with EFA.

The command takes longer time to complete only when:

a)Out of sync configuration is large. That is, many neutron entities are out of sync on EFA.

b)One of the SLX nodes is not reachable.

Workaround:

1. After the device is reloaded and switch is taken out of maintenance mode, issue a "no shutdown" on the MCT PO interface on the device either through CLI or EFA exec mode.

2. Issue a manual drift and reconcile on the devices where this MCT PO is configured.

SR/SR-BFD delete operation in vrf update has failed because of bellow conditions

1. device connectivity issues

2. Failed to deconfigure the peer on device

3. Targeted topology falls into the APS category.

To reconcile the device having 300+ L3 Networks, first replay the last saved good configuration and execute reconcile command for the device.

1. Replay Config:

efa inventory config-replay execute --ip <device-ip> --uuid <saved-config>

2. Reconcile:

efa inventory drift-reconcile execute --ip <device-ip>

This is observed with the below steps

1. Put SLXs in maintenance mode

2. Remove the router bgp configuration, few VRFs from the device.

3. Reboots both the MCT nodes where TPVMs containing EFA are hosted.

4. Verify after 30 mins, the switches are out of maintenance mode.

5. After a few iterations most of the BGP configs are missing.

EFA system backup is generated using efa system backup command.

Undeply EFA and install fresh EFA

Restore EFA backup generated earlier using "efa system restore" command

The restore process preserves the config backup interval and count. However the auto config backup process is not taking backups as configured.

Below are the steps when the configuration drift created does not get reconciled by EFA:

- Configuration of fabric/tenant/po/vrf/epg

- 'maintenance mode enable-on-reboot ' enabled on the switch

- Drift in configuration created by deleting port channel

- Reloading of the switch

- As part of reconcilation, port channel was created but Eth <port> was not aggregated to channel-group.

Execute manual Drift and Reconcile using this command:

" efa inventory drift-reconcile execute --ip <IP of device> --reconcile "

This issue observed with below steps,

- Configured fabric/tenant/po/vrf/epg/bgp peer/peer-group

- Enabled maintenance mode enable-on-reboot on SLX switch.

- Reload SLX switch.

- Drift and Reconcile process shows drift for interface used in EPG which was not drifted.

Device setting parameters like health check enable, heartbeat miss threshold etc. shows as default values.

Drift-reconcile get triggered after default heartbeat miss threshold.

Update device setting parameters once again using CLI.

$ efa inventory device setting update --ip <Device-ip> --maint-mode-enable-on-reboot <yes> --health-check-enable <yes> --health-check-interval <7m> --health-check-heartbeat-miss-threshold <3> --config-backup-periodic-enable <yes> --config-backup-interval <4m> --number-of-config-backups <10>

Recommended way to perform an SLX firmware upgrade from the EFA HA cluster being hosted by the same SLX devices:

Upgrade the standby EFA node SLX device.

Perform HA failover by stop/start active TPVM

Upgrade the old active or new standby EFA node SLX device.

workaround to allow the firmware download status details to be read from the device and updated in inventory DB. This can be triggered by the command -

"efa inventory device update'

1) VRF is created with Invalid Target Network in earlier version of EFA (before EFA 2.3.0)

2) EndpointGroup is created with same VRF in earlier version of EFA (before EFA 2.3.0)

3) Drift and Reconcile operation is performed

1) Delete EndpointGroup using VRF having invalid Target Network

2) Delete VRF

3) Recreate VRF with valid Target Network

4) Recreate EndpointGroup with new VRF

1. Create EPG with 100 networks.

2. Update EPG with 20 ports/port-channels from one switch using "port-group-add" operation.

1) Delete EndpointGroup(s) using same/common ctag/network(s) on the device where drift generation has failed

2) Let Drift-Reconcile operation complete for Tenant

3) Recreate deleted EndpointGroup(s)

i) Delete device from fabric

ii) Re-add same device into fabric right away.

i) Wait for 4-5 min.

ii) Re-add devices into fabric by re-execute same fabric bulk add CLI.

[efa fabric device add-bulk --name small-fabric --border-leaf <device-ip> --username <name> --password <pass>]

i) Wait for 4-5 min.

ii) Re-add devices into fabric by re-execute same fabric bulk add CLI.

[efa fabric device add-bulk --name small-fabric --border-leaf <device-ip> --username <name> --password <pass>]

i) Wait for few 5-10 min.

ii) Update devices into Inventory.

[$ efa inventory device update --ip <IP address of devices separated by ','>]

iii) Complete firmware download for remaining device into fabric.

iv) Reconfigure the fabric to recover from "cfg resfreshed" state

[ $ efa fabric configure --name <Name>]

1. Log in to the OVA.
2. Confirm that the k3 service is running.
   

   sudo service k3s status
   
   k3s.service - Lightweight Kubernetes ... 
   Active: active (running) since Sat 2020-10-31 00:08:24 
   UTC; 2 days ago

3. Run the following script manually: /opt/godcapp/efa/adjust_single_node_ip_change.sh
4. Reboot the EFA server.
5. Log in to EFA.
6. Verify that EFA services are running.
   

   efactl status