In-flight Transaction Recovery

XCO can recover in-flight (in-progress) transactions after a service restart or high-availability failover.

Overview

In-flight transactions are those that are outstanding in the execution log after a restart or a failover. After a service restart or high-availability failover, XCO recovers in-flight transactions by rolling them backward or rolling them forward.
  • When transactions are rolled backward, the requested action is incomplete.
  • When transactions are rolled forward, the requested action is completed.
By default, the in-flight transaction recovery feature enables the automatic recovery of Day-1 through Day-N operations for tenant-related configurations. You can use the efa system feature update --inflight-transaction-auto-recovery disable command to disable the feature. The following table describes the recovery strategy when the feature is enabled:
Table 1. Recovery strategy
Operation type Commands Strategy
Create operations
  • efa tenant create
  • efa tenant epg create
  • efa tenant po create
  • efa tenant service bgp peer create
  • efa tenant service bgp peer-group create
  • efa tenant vrf create
Roll back
Delete operations
  • efa tenant delete
  • efa tenant epg delete
  • efa tenant po delete
  • efa tenant service bgp peer delete
  • efa tenant service bgp peer-group delete
  • efa tenant vrf delete
Roll forward
Update with add operations, such as port-add, ctag-range-add, and vrf-add
  • efa tenant update
  • efa tenant epg update
  • efa tenant po update
  • efa tenant service bgp peer update
  • efa tenant service bgp peer-group update
  • efa tenant vrf update
Roll back
Update with delete operations, such as port-delete, vrf-delete, and ctag-range-delete
  • efa tenant update
  • efa tenant epg update
  • efa tenant po update
  • efa tenant service bgp peer update
  • efa tenant service bgp peer-group update
  • efa tenant vrf update
Roll forward
Consider the following expected behaviors for in-flight transaction recovery:
  • During operations that take a long time, such as drift and reconcile and firmware downloads, tenant operations and recovery operations are blocked.
  • When multiple transactions are pending in the execution log after a restart or a failover, recovery occurs in the order in which the operations appear in the execution log.
  • If a service restart or high availability failover occurs during transaction recovery, then the status of those recovery operations is changed to a normal state. For example, if a restart occurs during the rollback of an endpoint group (EPG), the status changes to delete-pending. There is no automatic recovery of interrupted recovery transactions. You must manually verify and address the status of such operations.
Important

Important

Day-0 and administrative operations (those for the Inventory Services and Fabric Services) are not recovered automatically. If these operations are interrupted by a service restart or a failover, you must manually redo the operations.

Examples

The following example enables automatic in-flight transaction recovery:

efa system feature update --inflight-transaction-auto-recovery enable
Feature Setting Updated Successful
--- Time Elapsed: 634.557118ms ---

The following example disables automatic in-flight transaction recovery:

efa system feature update --inflight-transaction-auto-recovery disable
Feature Setting Updated Successful
--- Time Elapsed: 634.557125ms ---