ExtremeCloud™ Orchestrator CLI Administration Guide Version 3.5.0.
> Unified Health and Fault Management
Published May 01, 2024
Search this document
Print this page
Email this page
View PDF
Previous
Next
Preface
Text Conventions
Documentation and Training
Help and Support
Send Feedback
What's New in this Document
Introduction to ExtremeCloud Orchestrator
Evolution of EFA and XVM into XCO
Fabric Automation and Orchestration
Visibility Solution
XCO Microservices
REST API Documentation for XCO
XCO System Management
Verify the Running System and Services
Log in to XCO
XCO Certificate Management
Device Certificates
Syslog CA
OAuth Certificate
HTTPS Certificates
GRPC Certificates
Manual Installation of Certificates on Devices
XCO Certificates
XCO Root CA
XCO Intermediate CA
XCO Server Certificate
JWT Certificate
K3s CA
K3s Server Certificate
Host Authentication Certificate
Galera Certificate
External Certificates
Certificate Troubleshooting
Monitoring XCO Status
Verifying XCO System Health
XCO System Backup and Restoration
Back up and Restore the XCO System
Change the Host Name or IP Address
Display XCO Running Configurations
Audit Trail Logging
Transfer of Audit Trail Data
Logging and Log Files
Logging Customization
Configure Logging
Unconfigure Logging
Data Consistency
Periodic Device Discovery
Persistent Configuration
Drift and Reconcile
Network Elements
Idempotent Operations
Rollback Scenarios for Data Consistency
XCO High Availability Failover Scenarios
Multiple Management IP Networks
Add and Delete Management Routes
Configuration Supporting Multiple Management IP Networks
Add and Delete Management Sub Interfaces
Configure Static IP Addresses for Management Sub Interfaces
Change the Default Gateway of a TPVM
Configure DNS Nameserver Access
Change Password of efainternal User
Linux Exit Codes
Linux Error Exit Code
Fabric Infrastructure Provisioning
Fabric Service Overview
IP Fabric and Clos Orchestration Overview
SLX Device Prerequisites for Fabric Service
Clos Overview
Configure a 3-Stage Clos Fabric
Configure a 5-Stage Clos Fabric
Provisioning Model to Migrate a 3-Stage Clos to 5-Stage Clos Fabric
Migrate a 3-Stage Clos to 5-Stage Clos Fabric
Create a 3-Stage Clos Fabric
Migrate a 3-Stage Clos to 5-Stage Clos Fabric
Disconnect Border-leafs from Spine and Connect to Super-spine
Addition of Super-spine Devices to the Migrated 5-stage Clos Fabric
Configure Migrated 5-stage Clos Fabric
Traffic Disruption during Fabric Configure
Verification of Fabric Underlay Configuration on the Migrated 5-stage Clos Fabric
Verification of Fabric Physical Underlay and Overlay Topology on the Migrated 5-stage Clos Fabric
Operations Allowed on a Fabric in Migrate-failed State
Operations Allowed on a Fabric in Migrate-success State
Operations not Allowed on a Fabric in Migrate-success and Migrate-failed State
Conditions Supporting Fabric Migration
Conditions Not Supporting Fabric Migration
Supported Topology
Non-Clos Small Data Center Overview
Supported Small Data Center Topologies
Configure a Small Data Center Fabric
Dynamic ICL in Small Data Center
Static ICL in Small Data Center
Configure Static and Dynamic ICL
Static ICL to Dynamic ICL Conversion
Dynamic ICL to Static ICL Conversion
Overview of Day-0 Operations for a Small Data Center Fabric
View Device Error in Clos and Non-Clos Fabric
Router ID and VTEP Loopback IP Allocation in Clos and Non-Clos Fabric
Allocate IP using Uniform Loopback Scheme
Allocate IP using Granular Scheme
Configure Local Bias for Handling the LVTEP BUM Traffic
IP Multicast Fabric Provisioning
IP Multicast Fabric Overview
Bidirectional Forwarding Detection
Fabric Settings to Update BGP MD5 Password, BGP Dynamic Peer Listen Limit, and Single Rack Deployment
Configure an IP Multicast Fabric
Device Configuration
Configure Drift and Reconcile on Multicast Fabric
View Fabric Details
Edit Fabric Settings
Update md5-password on an Active Fabric
Update bgp-multihop on an Active Fabric
Fabric Configuration using Force
Fabric DRC using Force
Fabric Event Handling
Import a Fabric Database
Pre-validation of Configuration
Global Device Configuration
Interface Configuration
MCT Configuration
Overlay Gateway Configuration
EVPN Configuration
BGP Configuration
BGP Tables
BGP Events
Configure SLX Password Expiry Notification
SLX Password Expiry Alerts
Tenant Service Provisioning
Tenant Services Provisioning Overview
Clos Fabric with Non-auto VNI Maps
Clos Fabric with Auto VNI Map
Provision a Tenant Entity
Create a Tenant
Update a Tenant
Show a Tenant
Delete a Tenant
Configure a Tenant
Create a Private Tenant
Create a Shared Tenant
Scalability
VLAN-based Tenant
Bridge domain-based Tenant
Provision a Port Channel
Create a Port Channel
Update a Port Channel
Delete a Port Channel
Delete Pending Port Channel Configuration
Show a Port Channel
Configure a Port Channel
Configure Description on Port Channel
Configure Minimum Link Count on Port Channel
Configure MTU on Port Channel
Configure LACP-timeout on Port Channel
Configure Port on a Port Channel
Shared and Private Port Channel Configuration
Provision a VRF
Create a Tenant VRF
Update a Tenant VRF
Show a Tenant VRF
Delete a Tenant VRF
Delete Pending VRF Configuration
Shows a Tenant VRF Error
Configure a Tenant VRF
Configure Local ASN on Tenant VRF
Update Local ASN on VRF
Enable Graceful Restart on Tenant VRF
Configure MaxPaths on Tenant VRF
Configure Resilient Hashing on Tenant VRF
Configure Redistribute Attribute on Tenant VRF
Configure Advertise Network and Static Network on Tenant VRF
Configure Aggregate Address on Tenant VRF
Configure EVPN IRB VE Cluster Gateway on a Tenant VRF
Route Distinguisher (RD) Allocation Independent of Route Target (RT)
Configure Static VRF Route
Configure BFD on Static VRF Route
Configure Backup Routing on Tenant VRF
Distributed and Centralized Routing
Prepare Clos Fabric for Centralized Routing
Prepare Small Data Center Fabric for Centralized Routing
Enable Centralized Routing on Tenant VRF
Configure Physical Router for Centralized Routing on Tenant VRF
Configure Anycast IP on Tenant Endpoint Group
Configure Local IP on Tenant Endpoint Group
Configure Static Route on Tenant VRF
Configure Static Route BFD on Tenant VRF
Configure Peer Group on Tenant BGP
Configure Static Peer on Tenant BGP
Configure Dynamic Peer on Tenant BGP
Centralized Routing on Single Rack Small Data Center Leaf Pair (not Border Leaf Pair)
BFD Timers for Router BGP BFD and Static Route BFD Sessions
Configure Next Hop Recursion
Configure ECMP Paths
Enable Default Information Originate
Provision a Tenant Endpoint Group
Create a Tenant Endpoint Group
Update a Tenant Endpoint Group
Show a Tenant Endpoint Group
Delete a Tenant Endpoint Group
Delete Pending EPG Configuration
Force Delete an EPG
Configure Network Property Description on Tenant EPG
Configure Network Property on Tenant EPG
IP DHCP Relay on Tenant EPG
XCO Provisioning of DHCP Relay Server and Gateway
Enable or Disable ARP Suppression and Neighbor Discovery (ND)
Configure Port Property on Tenant EPG
Enable or Disable ICMP Redirect on Tenant EPG Networks
Update Anycast IP on an Existing Tenant Network
Configure Multiple Anycast IP
Configure IPv6 Neighbor Discovery (ND) on a Tenant Network
Configure BFD Session Type for an Endpoint Group
Configure Cluster Edge Port (CEP) Cluster Tracking for Endpoint Groups
Enable Cluster Tracking on CEP Interfaces
Configure Suppress Address Resolution Protocol and Neighbor Discovery on VLAN or Bridge Domain
Configure Local IP for Endpoint Group
EPG: Network Property: IP MTU
Software BFD Session Support on CEP
Provision a BGP Peer
Create BGP Static Peer
Add Path on Tenant BGP Peer
Create BGP Dynamic Peer
Configure Listen Limit on BGP Dynamic Peer
Force Delete the Associate Dynamic Peers on a Tenant BGP Peer Group
Delete Pending BGP Peer Configuration
Getting the Operational State of the BGP Peers
Configure Route Map Attribute
Configure remove-private-as on BGP Peer
Configure default-originate to Advertise Default Route on BGP Peer
Configure Backup Routing Neighbors on BGP Peer
Configure Send-Community on Tenant BGP Peer
Configure Out-of-band for a Tenant BGP Peer or Peer Group
Multi Protocol BGP
Configure Multi Protocol BGP on Tenant Static BGP Peer
Configure Multi Protocol BGP on Tenant Dynamic BGP Peer
Enable or Disable MP BGP Capability for IPv6 Prefix Exchange over IPv4 Peer
Provision a BGP Peer Group
Create a BGP Peer Group
Configure IP Prefix List and Route Map on Tenant BGP Peer Group
Configure Send-Community on Tenant BGP Peer Group
Add Path on Tenant BGP Peer Group
Configure remove-private-as on BGP Peer Group
Activate Peer Group on Tenant BGP
Delete Pending BGP Peer Group Configuration
Configure IPv6 Address as Update Source
Share Resources Across Tenants using Shared Tenant
Shared VRF and Router
Configure Shared Tenant, Shared VRF, and Private EPG using Shared VRF
Configure L3-Hand-Off EPG and BGP Peer under Ownership of Shared Tenant
Shared VRF and Router Usecase with Examples
Sharing Multiple VRFs with the Same RT (route-target)
Configure Tenant Admin Access to Shared Tenant Resources or Entities
Administered Partial Success
Administratively Manage a Device State
APS Behavior of Tenant Configuration
Existing behavior
APS: Pre-provisioning Support by Modifying the Target Device List to Include the MCT Neighbor
APS: Deletion Support for Pre-provisioned Configurations
Traffic Mirroring Overview
In-band Traffic Mirroring
Out-of-band Traffic Mirroring
Support Matrix
Provision a BGP Peer Group
Create a BGP Peer Group
Configure IP Prefix List and Route Map on Tenant BGP Peer Group
Configure Send-Community on Tenant BGP Peer Group
Add Path on Tenant BGP Peer Group
Configure remove-private-as on BGP Peer Group
Activate Peer Group on Tenant BGP
Delete Pending BGP Peer Group Configuration
Provision a Traffic Mirror Session
Configure Port-Based Mirroring in a Multi-Tenant Architecture
Configure Flow-Based Mirroring in a Multi-Tenant Architecture
Access Control List and Data Consistency Support
Configure VLAN-Based Mirroring in a Multi-Tenant Architecture
Configure ICL Port Mirroring in a Multi-Tenant Architecture
Configure Fabric Non-ICL Ports as Mirror Source
Delete Pending Mirror Session Configuration
Exclusion of VLANs and Bridge from Cluster Instance
In-flight Transaction Recovery
Scalability
Policy Service Provisioning
Policy Service Provisioning Overview
Prefix List
Configure IP Prefix List on Devices
Drift and Reconcile (DRC) and Idempotency for IP Prefix List Configuration
Drift and Reconcile (DRC) for IPv6 Prefix List
Route Map
Configure Route Map on devices
Drift and Reconcile (DRC) and Idempotency for Route Map Configuration
Event Handling for IP Prefix List
Community List
Configure Standard Community List
Rollback Support
Configure Extended Community List
Configure Large Community List
Drift and Reconcile (DRC), Idempotency for Standard and Extended Community-list Configuration
Route Map Match and Set of Community List
Configure Route Map Match and Set of Community List
Drift and Reconcile (DRC) and Idempotency for Route Map Match and Set Configuration
Route Map Match and Set of Large Community List
Configure Route Map Match
Configure Route Map Set
Drift and Reconcile (DRC), Idempotency for Route Map Configuration for Large Community List
Force Delete OOB Entries from Policy Configuration
Policy Configuration Rollback
Policy Incremental Updates
Policy Device Membership Updates
Provisioning Dependencies
Policy Service QoS Support
Configure QoS Map
Configure QoS Profile
Application of QoS Profile
Tenant and Port Level Binding
Apply QoS Profile
QoS Profile Binding Enhancement
Configure QoS Service Policy Map
XCO Device Management
Device Image Management
Hitless Firmware Upgrade
Super-Spine Firmware Upgrade in Clos
Spine Firmware Upgrade in Clos
Firmware Upgrade of an MCT Leaf Pair with Dual-Homed Servers in Clos
Firmware Upgrade of a Three-Rack Centralized MCT Pair in Small Data Center
Firmware Upgrade of a Three-Rack Ring MCT Pair in Small Data Center
Firmware Download
Firmware Upgrade with Minimal Traffic Loss
Firmware Download Restart on HA Failover or Inventory Service Restart
Firmware Download Implicit Fullinstall Support
XCO Command Blocking during Firmware Download
Failures During Group-based Firmware Download Execution
Group-based Firmware Download Restore
Fabric-wide Firmware Download
Group-based Firmware Download Preparation
Fabric-based Firmware Download Preparation
Group-based Firmware Download Execution
Firmware Upgrade Status
Get User Action Fabric-Wide Firmware-Download (Fwdl) History
Get Runtime (Operational) Status-based History
Roll Back Device Firmware
Traffic Loss Scenarios
Device Health Management
Monitor Device Health
Device Configuration Backup and Replay
Configure Backup and Replay
Return Material Authorization
Replace a Faulty Device
SLX Device Configuration
Enable Maintenance Mode on SLX Devices
Display Inventory Device Interface
Display LLDP Inventory Device
Configure Physical Port Speed
Device Running Config Persist
Device Execute CLI
Configure Breakout Ports
Configure MTU at Interface or System Level
Change the Admin Status of an Interface
Configure Hardware Profile to Limit IPv6 Prefix to 64
Configure NTP at Device and Fabric Levels
Configure Port Dampening on Interface
Configure RME on SLX Interface
Interface FEC
Device Configuration Synchronization
XCO Native Support for SLX Threshold Monitor Settings
Set Threshold Monitor Options
Remove Settings for Threshold Monitor Options
Display Threshold Monitor Settings
Additional Threshold Monitor Types
Re-worked Threshold Monitor Parameters
CLI Migration
Disable IP Option
SLX Configuration Backup
CLI Commands for Backups
Enable or Disable Flooding for IP DHCP Relay
Show Device Adapter Connection Status
Show Device Certificate Expiry Time
Configure Device Certificate Expiry Time
XCO Event Management
RASlog Service
RASlog Operations
Notification Service
Notification Types
Notification Sub-Filtering
Terminology
Additional Notification Filtering
Sub-Filter CLI
Sub-Filter Options during XCO Upgrade
Webhooks Payload
Syslog Subscribers Message Format
App Events RFC-5424 Format
Device Events RFC-5424 Format
XCO as SNMP Proxy
Configure SNMP View and Destination UDP Port
Drift and Reconcile (DRC) and Idempotency for SNMP
Configure Device SNMP Use-VRF
Configure Device SNMP Group
Unified Health and Fault Management
Unified Health and Fault Management Overview
Hierarchical Representation of Resources
Unified View of Health and Fault Updates
Fault Management - Alerts
Common Alert Payload to be Published via Syslog
Common Alert Payload to be published via Webhook
Alert Commands
Inventory of Alerts
Alert Details
Alarm Alerts
Backup and Restore Alerts
Certificate Alerts
Device Connectivity Alerts
Device Link Alerts
Fabric Health Alerts
High Availability Alerts
License Alerts
License Alerts
LDAP Alerts
Login Alerts
Storage Alerts
Upgrade Alerts
Missed Alerts
Alert Order
Fault Management - Alarms
Alarm Severity
Alarm Types
Alarm Inventory
Alarm Status Change Notifications
Alarm Commands
Health Management
Bubbling of Health Status
Health Commands
Health APIs
Fabric Health
Fabric Health Calculation
Fabric Status
Fabric Level Physical Topology Health
Device Health
Config State Health
Operational State Health
Sample Output of 3-stage Clos Fabric Creation
Create 3-Stage Clos Fabric
Add Two Spine Devices 10,20.246.1 and 2
Add Two MCT Pairs 10.20.246.5,6 and 10.20.20.246.3 and 4
Configure the Fabric
Shut Down the MCT Port Channel on Leaf Devices
Choose Not to Shut Down the MCT Leaf Nodes
Remove Cluster Config from one MCT Leaf
Trigger DRC to Reconcile Cluster Config
Remove Leaf to Spine Links between Devices
Add Incorrect Links between Two Spines
Remove Incorrect Links between Two Spines
Post Configure Fabric or DRC
Delete Router BGP Configuration in a Spine Device
Trigger DRC to Reconcile Router BGP Config on Switch
Delete BGP Neighbors from Leaf Device
Configure Neighbor again in Switch
Reload a Leaf Device
Verify that Spine Devices Have Been Deleted from the Fabric
Remove all Devices from Fabric
XCO License Service Management
XCO Licensing Overview
XCO-based Licensing Overview
How XCO Licensing Works
XCO License Type
XCO License Terminology
XCO Licensing Configuration Tasks
Generate a License
Query a License
XCO Licensing Tasks
Install a License
Configure a License
Display a License
Delete a License
Licensed Features and Part Numbers
Licensing for XCO Systems in Air Gap Mode
Licensing Supportsave Details
License Backup and Restore
Troubleshooting Licensing Issues
License is Not Properly Installed
License Error Handling
License Expiry Alert
Known Limitations
Known Limitations in Fabric Skill
Unified Health and Fault Management
Learn about managing alerts, alarms, and health status of managed entities in XCO.
