Fabric Health Alerts

Use the information in the following tables to learn about all possible fabric health alerts in detail that are raised by Fault Management.

Table 1. Port Flap Alert
31600 Port Flap Alert
Description Send an alert when there is continuous port flap on a registered fabric device port.
Preconditions The SFP is faulty or there are other hardware issues.

The number of admin up is greater than configured threshold within a specified time interval.
Requirements

Alert shows the following data:

  • Device IP
  • Port Name
The following is an example of a port flap alert: 
<114>1 2003-10-11T22:14:15.003Z xco.machine.com FaultManager - -
   [meta sequenceId=”47”]   
   [origin ip=”10.x.x.x” enterpriseId=”1916” software=”XCO” swVersion=”3.3.0”]   
   [alert@1916   
   resource=””
   alertId=”31600” 
   cause=”excessiveErrorRate”
   type=”communicationsAlarm”
   severity=”warning”]
   [alertData@1916
   device_ip=”10.x.x.x”
   port=”Ethernet 0/1”]   
   BOM The Port flap detected for device 10.x.x.x, port ethernet 0/1
Health Response
Response
{
    Resource: /App/Component/Asset/Device?device_ip=10.x.x.x&port=Ethernet0/1
    HQI {
        Color: Red
        Value: 0
    }
    Status Text: Port Ethernet0/1 on device 10.x.x.x is flapping continuously.
}
Table 2. Managed Fabric Physical Topology Degraded Notice
31706 Managed Fabric Physical Topology Degraded Notice
Description Send an alert when the fabric level physical topology health is changed from to Green to Red.
Preconditions Fabric is created and devices are added in XCO.

The severity for physical topology errors is Major.

The fabric services generate the alerts when the following conditions are not met:

  • Fabric level physical topology validations for non-Clos fabric:
    1. Each rack must contain two devices.
  • Fabric level physical topology validations for Clos fabric:
    1. Stage 3 fabric must contain at least one leaf or border leaf device and spine device.
    2. Stage 5 fabric must contain at least one leaf or border leaf device and super-spine devices
Requirements
Alert shows the following data:
  • Fabric Name
  • Fabric Health Info
The following example shows an alert when a fabric level physical topology health is degraded:
<114>1 2003-10-11T22:14:15.003Z xco.machine.com FaultManager - -     
   [meta sequenceId=”47”]    
   [origin ip=”10.x.x.x” enterpriseId=”1916” software=”XCO” swVersion=”3.3.0”]    
   [alert@1916    
   resource=”/App/Component/Fabric/Topology/Physical?fabric_name=fb” 
   alertId=”31706”  	    
   cause=”missingSpines”  
   type=”fabricService”    
   severity=”major”]     
   [alertData@1916    
   fabric_name=”fb”  
   fabric_health_info=”{ 
Missing_spines: true} 
}”  
   ] 
   BOM The fabric “fb” is missing spines.
Health Response
Response
{ 
    Resource: /App/Component/Fabric/Topology/Physical?fabric_name =fb 
    HQI { 
        Color: Red 
        Value: 3 
    } 
    StatusText: The fabric “fb” is missing spines
}
Table 3. Managed Fabric Physical Topology Healthy Notice
31707 Managed Fabric Physical Topology Healthy Notice
Description Send an alert when the fabric level physical topology health is changed from Red to Green.
Preconditions Fabric is created and devices are added in XCO.

The fabric services generate the alerts when the following conditions are met:

  • Fabric level physical topology validations for non-Clos fabric:
    1. Each rack must contain two devices.
  • Fabric level physical topology validations for Clos fabric:
    1. Stage 3 fabric must contain at least one leaf or border leaf device and spine device.
    2. Stage 5 fabric must contain at least one leaf or border leaf device and super-spine devices
Requirements
Alert shows the following data:
  • Fabric Name
  • Fabric Health Info
The following example shows an alert when a fabric level physical topology is in healthy condition:
<118>1 2003-10-11T22:14:15.003Z xco.machine.com FaultManager - -     
   [meta sequenceId=”47”]    
   [origin ip=”10.x.x.x” enterpriseId=”1916” software=”XCO” swVersion=”3.3.0”]    
   [alert@1916    resource=”/App/Component/Fabric/Topology/Physical?fabric_name=fb” 
   alertId=”31707”  	    
   cause=”fabricPhysicalTopologyHealthRestored”  
   type=”fabricService”    
   severity=”info”]     
   [alertData@1916    
   fabric_name=”fb”   
   fabric_health_info=”” 
”  
   ] 
   BOM The fabric “fb” has fabric physical topology in healthy state
Health Response
Response
{ 
    Resource: /App/Component/Fabric/Topology/Physical?fabric_name =fb 
    HQI { 
        Color: Green 
        Value: 0 
    } 
    StatusText: The fabric “fb” has fabric physical topology in healthy state.
}
Table 4. Managed Fabric Device Appstate Degraded Notice
31708 Managed Fabric Device Appstate Degraded Notice
Description Send an alert when fabric device app state health is changed from Green to Red or Black.
Preconditions

Fabric is created and devices are added in XCO. Application state changes based on adding devices, configuring success or failure, drift in configurations.

The alert has the following app states and the corresponding severity:
State Severity Description
cfg-ready Major Device configurations are ready to be pushed to device.
cfg-refreshed Major There is drift in configurations between switch and XCO intended configurations.
cfg-error Critical Configuration errors come before pushing configurations to the switch (for example, adding three leaf devices with links between all of them).
cfg-refresh-error Major Configuration failure like device reload state or missing links between device.
device-remove-failed Critical Remove device from fabric failed
Requirements
Alert shows the following data:
  • Fabric Name
  • Device IP
  • Fabric Health Info
The following example shows an alert when a fabric device app state health is degraded:
<114>1 2003-10-11T22:14:15.003Z xco.machine.com FaultManager - -     
   [meta sequenceId=”47”]    
   [origin ip=”10.x.x.x” enterpriseId=”1916” software=”XCO” swVersion=”3.3.0”]    
   [alert@1916    
   resource=”/App/Component/Fabric/Device/Configuration/AppState?fabric_name=fb&device_ip=10.x.x.x” 
   alertId=”31708”  	    
   cause=”configReady”  
   type=”fabricService”    
   severity=”major”]     
   [alertData@1916    
   fabric_name=”fb”  
   device_ip=”10.x.x.x” 
   fabric_health_info=”{AppState: cfg ready}”  
   ] 
   BOM The device “10.x.x.x” of fabric “fb” has app state set to cfg ready.
Health Response
Response
{ 
    Resource: /App/Component/Fabric/Device/Configuration/AppState?fabric_name=fb&device_ip=10.x.x.x 
    HQI { 
        Color: Red 
        Value: 3 
    } 
    StatusText: The device “10.x.x.x” of fabric “fb” has app state set to cfg ready
}
Table 5. Managed Fabric Device Appstate Healthy Notice
31709 Managed Fabric Device Appstate Healthy Notice
Description Send an alert when fabric device app state health is changed from Red or Black to Green.
Preconditions

Fabric is created, and devices are added and configured in XCO so that the App state changes to cfg-in-sync.

The alert has the following app states and the corresponding severity:
State Severity Description
cfg-in-sync Info Device configurations are pushed to switch and it is in sync with XCO .
Requirements
Alert shows the following data:
  • Fabric Name
  • Device IP
  • Fabric Health Info
The following example shows an alert when a fabric device app state is healthy:
<118>1 2003-10-11T22:14:15.003Z xco.machine.com FaultManager - -     
   [meta sequenceId=”47”]    
   [origin ip=”10.x.x.x” enterpriseId=”1916” software=”XCO” swVersion=”3.3.0”]    
   [alert@1916    
   resource=”/App/Component/Fabric/Device/Configuration/AppState?fabric_name=fb&device_ip=10.x.x.x” 
   alertId=”31709”  	    
   cause=”appStateHealthRestored”  
   type=”fabricService”    
   severity=”info”]     
   [alertData@1916    
   fabric_name=”fb”  
   device_ip=”10.x.x.x” 
   fabric_health_info=””  
   ] 
   BOM The device “10.x.x.x” of fabric “fb” has app state set to healthy.
Health Response
Response
{ 
    Resource: /App/Component/Fabric/Device/Configuration/AppState?fabric_name=fb&device_ip=10.x.x.x 
    HQI { 
        Color: Green 
        Value: 0 
    } 
    StatusText: The device “10.x.x.x” of fabric “fb” has app state set to healthy
}
Table 6. Managed Fabric Device Provisioning State Degraded Notice
31710 Managed Fabric Device Provisioning State Degraded Notice
Description Send an alert when fabric device provisioning state health is changed from Green to Black or Red.
Preconditions

Fabric is created, and devices are added in XCO.

The alert has the following app states and the corresponding severity:
State Severity
Not Provisioned Major
Provisioning Failed Critical
Requirements
Alert shows the following data:
  • Fabric Name
  • Device IP
  • Fabric Health Info
The following example shows an alert when a fabric device provisioning state health is degraded:
<114>1 2003-10-11T22:14:15.003Z xco.machine.com FaultManager - -     
   [meta sequenceId=”47”]    
   [origin ip=”10.x.x.x” enterpriseId=”1916” software=”XCO” swVersion=”3.3.0”]    
   [alert@1916    
   resource=”/App/Component/Fabric/Device/Configuration/DevState?fabric_name=fb&device_ip=10.x.x.x” 
   alertId=”31710”  	    
   cause=”notProvisioned”  
   type=”fabricService”    
   severity=”major”]     
   [alertData@1916    
   fabric_name=”fb”  
   device_ip=”10.x.x.x” 
   fabric_health_info=”{  DevState: not provisioned}”  
   ] 
   BOM The device “10.x.x.x” of fabric “fb” has dev state set to not provisioned.
Health Response
Response
{ 
    Resource: /App/Component/Fabric/Device/Configuration/DevState?fabric_name=fb&device_ip=10.x.x.x 
    HQI { 
        Color: Red 
        Value: 3 
    } 
    StatusText: The device “10.x.x.x” of fabric “fb” has dev state set to not provisioned.
}
Table 7. Managed Fabric Device Provisioning State Healthy Notice
31711 Managed Fabric Device Provisioning State Healthy Notice
Description Send an alert when fabric device provisioning state health is changed from Black or Red to Green.
Preconditions

Fabric is created, and devices are added and configured in XCO so that the devices move to provisioned state (Severity - Info).

Requirements
Alert shows the following data:
  • Fabric Name
  • Device IP
  • Fabric Health Info
The following example shows an alert when a fabric device provisioning state is healthy:
<118>1 2003-10-11T22:14:15.003Z xco.machine.com FaultManager - -     
   [meta sequenceId=”47”]    
   [origin ip=”10.20.30.40” enterpriseId=”1916” software=”XCO” swVersion=”3.3.0”]    
   [alert@1916    
   resource=”/App/Component/Fabric/Device/Configuration/DevState?fabric_name=fb&device_ip=10.x.x.x” 
   alertId=”31711”  	    
   cause=”devStateHealthRestored”  
   type=”fabricService”    
   severity=”info”]     
   [alertData@1916    
   fabric_name=”fb”  
   device_ip=”10.x.x.x”  
   fabric_health_info=””  
   ] 
   BOM The device “10.x.x.x” of fabric “fb” has dev state set to healthy.
Health Response
Response
{ 
    Resource: /App/Component/Fabric/Device/Configuration/DevState?fabric_name=fb&device_ip=10.x.x.x 
    HQI { 
        Color: Green 
        Value: 0 
    } 
    StatusText: The device “10.x.x.x” of fabric “fb” has dev state set to healthy.
}

Managed Fabric Device MCT Cluster Degraded Notice

31712 Managed Fabric Device MCT Cluster Degraded Notice
Description  Send an alert when fabric device MCT cluster health is changed from Green to Red or Black.
Preconditions 

Fabric is created and MCT devices are added and configured in XCO.

An alert is raised if any of the following cluster operational states are down:

  • PeerState: false
  • PeerKeepAliveState: false
  • ClusterState: false
Peer State Cluster State Peer Keep-alive State Severity
Up Up Up Info (Raised by alert ID 31519)
Up Up Down Major
Up Down Up Major
Up Down Down Major
Down Up Up Critical
Down Up Down Critical
Down Down Up Critical
Down Down Down Critical
Requirements 

Alert shows the following data:

  1. Fabric Name
  2. Device IP
  3. Fabric Health Info
The following example shows an alert when a fabric device MCT cluster state is degraded:
<114>1 2003-10-11T22:14:15.003Z xco.machine.com FaultManager - -     
   [meta sequenceId=”47”]    
   [origin ip=”10.x.x.x” enterpriseId=”1916” software=”XCO” swVersion=”3.3.0”]    
   [alert@1916    
resource=”/App/Component/Fabric/Device/Operational/Mct?fabric_name=fb&device_ip=10.x.x.x” 
   alertId=”31712”  	    
   cause=”mctClusterStateAndPeerKeepAliveStateDown”  
   type=”fabricService”    
   severity=”major”]     
   [alertData@1916    
   fabric_name=”fb”  
   device_ip=”10.x.x.x” 
   fabric_health_info=”{ 
PeerKeepAliveState: false 
ClusterState: false 
}” 
 ] 
   BOM The device “10.x.x.x” of fabric “fb” has MCT cluster state and peer keep alive state down.
Health Response
Response
{ 
    Resource: /App/Component/Fabric/Device/Operational/Mct?fabric_name=fb&device_ip=10.x.x.x 
    HQI { 
        Color: Red 
        Value: 3 
    } 
    StatusText The device “10.x.x.x” of fabric “fb” has MCT cluster state and peer keep alive state down
}

Managed Fabric Device MCT Cluster Healthy Notice

31713 Managed Fabric Device MCT Cluster Healthy Notice
Description  Send an alert when fabric device MCT cluster health is changed from Red or Black to Green.
Preconditions 

Fabric is created and MCT devices are added and configured in XCO.

An alert is raised if all the cluster operational states are up (Severity: Info):

  • PeerState: true
  • PeerKeepAliveState: true
  • ClusterState: true
Requirements 

Alert shows the following data:

  1. Fabric Name
  2. Device IP
  3. Fabric Health Info
The following example shows an alert when a fabric device MCT cluster state is healthy:
<118>1 2003-10-11T22:14:15.003Z xco.machine.com FaultManager - -     
   [meta sequenceId=”47”]    
   [origin ip=”10.x.x.x” enterpriseId=”1916” software=”XCO” swVersion=”3.3.0”]    
   [alert@1916    
   resource=”/App/Component/Fabric/Device/Operational/Mct?fabric_name=fb&device_ip=10.x.x.x” 
   alertId=”31713”  	    
   cause=”deviceMctClusterHealthRestored”  
   type=”fabricService”    
   severity=”info”]     
   [alertData@1916    
   name=”fb”  
   device_ip=”10.x.x.x” 
   fabric_health_info=””  
   ] 
   BOM The device “10.x.x.x” of fabric “fb” has MCT cluster in healthy state.
Health Response
Response
{ 
    Resource: /App/Component/Fabric/Device/Operational/Mct?fabric_name=fb&device_ip=10.x.x.x 
    HQI { 
        Color: Green 
        Value: 0 
    } 
    StatusText The device “10.x.x.x” of fabric “fb” has MCT cluster in healthy state
}

Managed Fabric Device Physical Topology Degraded Notice

31714 Managed Fabric Device Physical Topology Degraded Notice
Description  Send an alert when fabric device physical topology health is changed from Green to Red.
Preconditions 

Fabric is created and devices are added and configured in XCO.

Alerts are raised if the physical topology validation fails.

The severity for physical topology errors is Major.

Alerts are raised if the following conditions are not met:

  1. Device level physical topology validations for non-Clos fabric:
    • Two devices in rack must have link between them.
    • Each rack must be connected to at least another rack.
  2. Device level physical topology validations for Clos fabric:
    • Leaf node must be connected to all the Spine nodes.
    • Spine node must be connected to all the Leaf nodes.
    • Border Leaf node must be connected to all the Spine nodes or Super-spine nodes but not both.
    • Spine node must be connected to all the Border Leaf nodes.
    • More than two Leaf nodes must not be connected to each other.
    • More than two Border Leaf nodes must not be connected to each other.
    • Border leaf node and leaf node must not be connected.
    • Spine nodes must not be connected to each other.
    • Super Spine nodes must not be connected to each other.
    • If a Leaf node is "multi-homed", then the node must have an MCT neighbor.
    • If a Leaf node is "single-homed", then the node must not be connected to other Leaf nodes.
    • If a Border Leaf node is "multi-homed", then the node must have an MCT neighbor.
    • If a Border Leaf node is "single-homed", then the node must not be connected to other Border Leaf nodes.
Requirements 

Alert shows the following data:

  1. Fabric Name
  2. Device IP
  3. Fabric Health Info
The following example shows an alert when a fabric device physical topology health is degraded:
<114>1 2003-10-11T22:14:15.003Z xco.machine.com FaultManager - -     
   [meta sequenceId=”47”]    
   [origin ip=”10.x.x.x” enterpriseId=”1916” software=”XCO” swVersion=”3.3.0”]    
   [alert@1916    
resource=”/App/Component/Fabric/Device/Operational/Topology/Physical?fabric_name=fb&device_ip=10.x.x.x” 
   alertId=”31714”  	    
   cause=”missingLinks”  
   type=”fabricService”    
   severity=”major”]     
   [alertData@1916    
   fabric_name=”fb”  
   device_ip=”10.x.x.x”  
   fabric_health_info=”:{ 
 
Errors:[ 
{ 
Destination_ip: 10.x.x.a 
Destination_Role: leaf 
Device_links:[ 
{Error: missing_links} 
] 
}, 
Destination_ip: 10.x.x.b 
Destination_Role: leaf 
Device_links:[ 
{Error: missing_links} 
] 
] 
” 
 ] 
   BOM The device “10.x.x.x” of fabric “fb” has missing-links with devices [10.x.x.a,10.x.x.b].
Health Response
Response
{ 
    Resource: /App/Component/Fabric/Device/Operational/Topology/Physical?fabric_name=fb&device_ip=10.x.x.x 
    HQI { 
        Color: Red 
        Value: 3 
    } 
    StatusText The device “10.x.x.x” of fabric “fb” has missing-links with devices [10.x.x.a,10.x.x.b]. 
}

Managed Fabric Device Physical Healthy Notice

31715 Managed Fabric Device Physical Healthy Notice
Description  Send an alert when fabric device physical health is changed from Red to Green.
Preconditions 

Fabric is created and devices are added and configured in XCO.

Alerts are raised if the device physical topology validation is successful.

The severity for device physical topology errors is Info.

Alerts are raised if the following conditions are met:

  1. Device level physical topology validations for non-Clos fabric:
    • Two devices in rack must have link between them.
    • Each rack must be connected to at least another rack.
  2. Device level physical topology validations for Clos fabric:
    • Leaf node must be connected to all the Spine nodes.
    • Spine node must be connected to all the Leaf nodes.
    • Border Leaf node must be connected to all the Spine nodes or Super-spine nodes but not both.
    • Spine node must be connected to all the Border Leaf nodes.
    • More than two Leaf nodes must not be connected to each other.
    • More than two Border Leaf nodes must not be connected to each other.
    • Border leaf node and leaf node must not be connected.
    • Spine nodes must not be connected to each other.
    • Super Spine nodes must not be connected to each other.
    • If a Leaf node is "multi-homed", then the node must have an MCT neighbor.
    • If a Leaf node is "single-homed", then the node must not be connected to other Leaf nodes.
    • If a Border Leaf node is "multi-homed", then the node must have an MCT neighbor.
    • If a Border Leaf node is "single-homed", then the node must not be connected to other Border Leaf nodes.
Requirements 

Alert shows the following data:

  1. Fabric Name
  2. Device IP
  3. Fabric Health Info
The following example shows an alert when a fabric device physical health is Green (healthy):
<118>1 2003-10-11T22:14:15.003Z xco.machine.com FaultManager - -     
   [meta sequenceId=”47”]    
   [origin ip=”10.x.x.x” enterpriseId=”1916” software=”XCO” swVersion=”3.3.0”]    
   [alert@1916    
   resource=”/App/Component/Fabric/Device/Operational/Topology/Physical?fabric_name=fb&device_ip=10.x.x.x” 
   alertId=”31715”  	    
   cause=”devicePhysicalTopologyHealthRestored”  
   type=”fabricService”    
   severity=”info”]     
   [alertData@1916    
   fabric_name=”fb”  
   device_ip=”10.x.x.x” 
   fabric_health_info=””  
   ] 
   BOM The device “10.x.x.x” of fabric “fb” has physical topology in healthy state.
Health Response
Response
{ 
    Resource: /App/Component/Fabric/Device/Operational/Topology/Physical?fabric_name=fb&device_ip=10.x.x.x.a 
    HQI { 
        Color: Green 
        Value: 0 
    } 
    StatusText The device “10.x.x.x” of fabric “fb” has physical topology in healthy state. 
}

Managed Fabric Device Underlay Degraded Notice

31716 Managed Fabric Device Underlay Degraded Notice
Description  Send an alert when fabric device underlay health is changed from Green to Red or Black.
Preconditions 

Fabric is created and devices are added and configured in XCO.

The alerts are raised if the session state of any of the BGP neighbors is not established.

  • If BFP neighbors are not configured and devices are not in the provisioned state, then the severity is Major.
  • If some of the session state of BGP neighbor is down and some of the session state is up between devices, then the severity is Major.
  • If all the BGP neighbor sessions are down between devices, then the severity is Critical.
  • If the BGP neighbors are not configured and devices are in provisioned state, then the severity is Critical.
Requirements 

Alert shows the following data:

  1. Fabric Name
  2. Device IP
  3. Fabric Health Info
The following example shows an alert when a fabric device underlay health is degraded:
<114>1 2003-10-11T22:14:15.003Z xco.machine.com FaultManager - -     
   [meta sequenceId=”47”]    
   [origin ip=”10.x.x.x” enterpriseId=”1916” software=”XCO” swVersion=“3.3.0”]    
   [alert@1916    
resource=”/App/Component/Fabric/Device/Operational/Topology/Underlay?fabric_name=fb&device_ip=10.x.x.x” 
   alertId=”31716”  	    
   cause=”underlayNeighborsNotConfigured”  
   type=“fabricService”    
   severity=”major”]     
   [alertData@1916    
   fabric_name=”fb”  
   device_ip=”10.x.x.x” 
   fabric_health_info=”:{Errors:[ 
{ 
Destination_ip: 10.x.x.a 
Destination_Role: leaf 
NeighborIP:10.x.x.2 
SourceAsn: 6512, 
DeestinationAsn: 6500 
Error: neighbor_not_configured 
},{ 
Destination_ip: 10.x.x.b 
Destination_Role: leaf 
NeighborIP:10.x,x.3 
SourceAsn: 6512, 
DeestinationAsn: 6500 
Error: neighbor_not_configured 
}],” 
 ] 
   BOM The device “10.x.x.x” of fabric “fb” does not  have bgp neighbors  configured with [10.x.x.a,10.x.x.b].
Health Response
Response
{ 
    Resource: /App/Component/Fabric/Device/Operational/Topology/Underlay?fabric_name=fb&device_ip=10.x.x.x 
    HQI { 
        Color: Red 
        Value: 3 
    } 
    StatusText The device “10.x.x.x” of fabric “fb” does not  have bgp neighbors  configured with [10.x.x.a,10.x.x.b]
}

Managed Fabric Device Underlay Healthy Notice

31717 Managed Fabric Device Underlay Healthy Notice
Description  Send an alert when fabric device underlay health is changed from Red or Black to Green.
Preconditions 

Fabric is created and devices are added and configured in XCO.

The alerts are raised if the BGP neighbors session state are in established state. The severity is Info.
Requirements 

Alert shows the following data:

  1. Fabric Name
  2. Device IP
  3. Fabric Health Info
The following example shows an alert when a fabric device underlay health is Green (healthy):
<118>1 2003-10-11T22:14:15.003Z xco.machine.com FaultManager - -     
   [meta sequenceId=”47”]    
   [origin ip=”10.x.x.x” enterpriseId=”1916” software=”XCO” swVersion=“3.3.0”]    
   [alert@1916    
   resource=”/App/Component/Fabric/Device/Operational/Topology/Underlay?fabric_name=fb&device_ip=10.x.x.x” 
   alertId=”31717”  	    
   cause=” deviceUnderlayTopologyHealthRestored”  
   type=“fabricService”    
   severity=”info”]     
   [alertData@1916    
   fabric_name=”fb”  
   device_ip=”10.x.x.x” 
   fabric_health_info=””  
   ] 
   BOM The device 10.x.x.x of fabric fb has underlay topology healthy.
Health Response
Response
 
    Resource: /App/Component/Fabric/Device/Operational/Topology/Underlay?fabric_name=fb&device_ip=10.x.x.x 
    HQI { 
        Color: Green 
        Value: 0 
    } 
    StatusText The device “10.x.x.x” of fabric “fb” has underlay topology in healthy state
}

Managed Fabric Device Overlay Degraded Notice

31718 Managed Fabric Device Overlay Degraded Notice
Description  Send an alert when fabric device overlay health is changed from Green to Black.
Preconditions 

Fabric is created, devices are added and configured, and tenant L2 services are configured with common ctag range in XCO.

The alerts (Severity: Critical) are raised if the operational or admin status is down.
Requirements 

Alert shows the following data:

  1. Fabric Name
  2. Device IP
  3. Fabric Health Info
The following example shows an alert when a fabric device overlay health is degraded:
<114>1 2003-10-11T22:14:15.003Z xco.machine.com FaultManager - -     
   [meta sequenceId=”47”]    
   [origin ip=”10.20.30.40” enterpriseId=”1916” software=”XCO” swVersion=“3.3.0”]    
   [alert@1916    
resource=”/App/Component/Fabric/Device/Operational/Topology/Overlay?fabric_name=fb&device_ip=10.x.x.x” 
   alertId=”31718”  	    
   cause=”ovelayTunnelOperDown”  
   type=“fabricService”    
   severity=”critical”]     
   [alertData@1916    
   fabric_name=”fb”  
   device_ip=”10.x.x.x” 
   fabric_health_info=”:{ Errors:[{ 
Destination_ip: 10.x.x.a 
Destination_Role: leaf 
NeighborIP:10.10.10.2 
SourceVTEPIP:10.10.10.3, 
DestinationVTEPIP:10.10.10.6 
Admin_state:up, 
Oper_state: down 
Error: tunnel_oper_down 
},{ 
Destination_ip: 10.x.x.b 
Destination_Role: leaf 
SourceVTEPIP:10.x.x.3, 
DestinationVTEPIP:10.x.x.5 
Admin_state:up, 
Oper_state: down 
Error: tunnel_oper_down 
}]” 
 ] 
   BOM The device “10.x.x.x” of fabric “fb” does not have tunnels operationally up with [10.x.x.a,10.x.x.b]
Health Response
Response
{ 
    Resource: /App/Component/Fabric/Device/Operational/Topology/Overlay?fabric_name=fb&device_ip=10.x.x.x 
    HQI { 
        Color: Black 
        Value: 4 
    } 
    StatusText The device “10.x.x.x” of fabric “fb” does not have tunnels operationally up with [10.x.x.a,10.x.x.b]
}

Managed Fabric Device Overlay Healthy Notice

31719 Managed Fabric Device Overlay Healthy Notice
Description  Send an alert when fabric device overlay health is changed from Black to Green.
Preconditions 

Fabric is created, devices are added and configured, and tenant L2 services are configured with common ctag range in XCO.

The alerts (Severity: Info) are raised if the tunnel operational and admin status are up.
Requirements 

Alert shows the following data:

  1. Fabric Name
  2. Device IP
  3. Fabric Health Info
The following example shows an alert when a fabric device overlay health is Green (healthy):
<118>1 2003-10-11T22:14:15.003Z xco.machine.com FaultManager - -     
   [meta sequenceId=”47”]    
   [origin ip=”10.20.30.40” enterpriseId=”1916” software=”XCO” swVersion=“3.3.0”]    
   [alert@1916    
   resource=”/App/Component/Fabric/Device/Operational/Topology/Overlay?fabric_name=fb&device_ip=10.x.x.x” 
   alertId=”31719”  	    
   cause=” deviceOverlayTopologyHealthRestored”  
   type=“fabricService”    
   severity=”info”]     
   [alertData@1916    
   fabric_name=”fb”  
   device_ip=”10.x.x.x” 
   fabric_health_info=””  
   ] 
   BOM The device “10.x.x.x” of fabric “fb” has overlay topology in healthy state.
Health Response
Response
{ 
    Resource: /App/Component/Fabric/Device/Operational/Topology/Overlay?fabric_name=fb&device_ip=10.x.x.x 
    HQI { 
        Color: Green 
        Value: 0 
    } 
    StatusText The device “10.x.x.x” of fabric “fb” has overlay topology in healthy state.
 }

Managed Fabric Deleted Health Notice

31701 Managed Fabric Deleted Health Notice
Description  Send an alert when fabric is deleted.
Preconditions  Deletion of an existing fabric.
Requirements 

Alert shows the following data:

  • Fabric Name
The following example shows an alert when a fabric is deleted:
<118>1 2003-10-11T22:14:15.003Z xco.machine.com AppFaultManager - -     
   [meta sequenceId=”47”]    
   [origin ip=”10.20.30.40” enterpriseId=”1916” software=”XCO” swVersion=”3.3.0”]    
   [alert@1916    
   resource=”/App/Component/Fabric?fabric_name=fb” 
   alertId=”31701”  	    
   cause=”configRemoved”  
   type=”fabricService”    
   severity=”info”]     
   [alertData@1916    
   fabric_name=”fb”  
   ] 
   BOM : The fabric “fb” is deleted.
Health Response
Response
{ 
    Resource: /App/Component/Fabric?fabric_name=fb 
    HQI { 
        Color: Green 
        Value: 0 
    } 
    StatusText : The fabric “fb” is deleted. 
}

Managed Fabric Device Added Health Notice

31702 Managed Fabric Device Added Health Notice
Description  Send an alert when a fabric device is added.
Preconditions  Fabric is created and devices are added.
Requirements 

Alert shows the following data:

  • Fabric Name
  • Device IP
The following example shows an alert when a device is added to the fabric:
<118>1 2003-10-11T22:14:15.003Z xco.machine.com FaultManager - -     
   [meta sequenceId=”47”]    
   [origin ip=”10.x.x.x” enterpriseId=”1916” software=”XCO” swVersion=”3.3.0”]    
   [alert@1916    
   resource=”/App/Component/Fabric/Device?fabric_name=fb&device_ip=10.x.x.x” 
   alertId=”31702”  	    
   cause=”configCreated”  
   type=”fabricService”    
   severity=”info”]     
   [alertData@1916    
   fabric_name=”fb”  
   device_ip=”10.x.x.x” 
   ] 
   BOM The device “10.x.x.x” is added to the fabric “fb”
Health Response
Response
{ 
     Resource: /App/Component/Fabric/Device?fabric_name=fb&device_ip=10.x.x.x 
    HQI { 
        Color: Green 
        Value: 0 
    } 
    StatusText : The device “10.x.x.x” is added to the fabric “fb”
}

Managed Fabric Device Removed Health Notice

31703 Managed Fabric Device Removed Health Notice
Description  Send an alert when a fabric device is deleted.
Preconditions  Fabric is created. Devices are added and then deleted.
Requirements 

Alert shows the following data:

  • Fabric Name
  • Device IP
The following example shows an alert when an existing fabric is deleted:
<118>1 2003-10-11T22:14:15.003Z xco.machine.com FaultManager - -     
   [meta sequenceId=”47”]    
   [origin ip=”10.x.x.x” enterpriseId=”1916” software=”XCO” swVersion=”3.3.0”]    
   [alert@1916    
   resource=”/App/Component/Fabric/Device?fabric_name=fb&device_ip=10.x.x.x” 
   alertId=”31703”  	    
   cause=”configRemoved”  
   type=”fabricService”    
   severity=”info”]     
   [alertData@1916    
   fabric_name=”fb”  
   device_ip=”10.x.x.x” 
   ] 
   BOM The device “10.x.x.x” is removed from the fabric “fb”
Health Response
Response
{ 
    Resource: /App/Component/Fabric/Device?fabric_name=fb&device_ip=10.x.x.x 
    HQI { 
        Color: Green 
        Value: 0 
    } 
    StatusText : The device “10.x.x.x” is removed from fabric “fb”.
}

Fabric State Degraded Notice

31704 Fabric State Degraded Notice
Description  Send an alert when a fabric state health is changed from Green to Red or Black.
Preconditions  Fabric is created and devices are added in XCO.
The following are the states and the severities of an alert:
  • Configure-failed: Fabric configure fails (Critical)
  • Migrate-success: fabric is migrated (Major)
  • Migrate-failed: fabric migrate failed (Major)
  • Settings-updated: fabric settings is updated (Major)
Requirements 

Alert shows the following data:

  • Fabric Name
  • Fabric Health Info
The following example shows an alert when an existing fabric is deleted:
<114> 2003-10-11T22:14:15.003Z xco.machine.com FaultManager - -     
   [meta sequenceId=”47”]    
   [origin ip=”10.x.x.x” enterpriseId=”1916” software=”XCO” swVersion=”3.3.0”]    
   [alert@1916    
   resource=”/App/Component/Fabric/State?fabric_name=fb” 
   alertId=”31704”  	    
   cause=”fabricSettingsUpdated”  
   type=”fabricService”    
   severity=”major”]     
   [alertData@1916  
   fabric_name=”fb” 
   fabric_health_info=”{Fabric_status: setting updated}”  
   ] 
   BOM The fabric “fb” has status set to settings-updated due to change in fabric settings [BGP-MD5].
Health Response
Response
{ 
    Resource: /App/Component/Fabric/State?fabric_name=fb 
    HQI { 
        Color: Red 
        Value: 3 
    } 
    StatusText: The fabric “fb” has status set to settings-updated due to change in fabric settings [BGP-MD5]
}

Managed Fabric State Healthy Notice

31705 Managed Fabric State Healthy Notice
Description  Send an alert when fabric state health is changed from Red or Black to Green.
Preconditions  Fabric is created and devices are added in XCO. The alert has the following state changes:
  • Configure-success: Fabric configure success (Info)
Requirements 

Alert shows the following data:

  • Fabric Name
  • Fabric Health Info
The following example shows an alert when a fabric is created:
<118>1 2003-10-11T22:14:15.003Z xco.machine.com FaultManager - -     
   [meta sequenceId=”47”]    
   [origin ip=”10.x.x.x” enterpriseId=”1916” software=”XCO” swVersion=”3.3.0”]    
   [alert@1916    
   resource=”/App/Component/Fabric/State?fabric_name=fb” 
   alertId=”31705”  	    
  cause=”fabricStateHealthRestored”  
   type=”fabricService”    
   severity=”info”]     
   [alertData@1916    
   fabric_name=”fb”  
   fabric_health_info=””  
   ] 
   BOM The fabric “fb” has status set to healthy.
Health Response
Response
{ 
    Resource: /App/Component/Fabric/State?fabric_name=fb 
    HQI { 
        Color:Green 
        Value: 0 
    } 
    StatusText:The fabric “fb” has status set to healthy
}

Managed Fabric Created Notice

31700 Managed Fabric Created Notice
Description  Send an alert when a fabric is created.
Preconditions  None
Requirements 

Alert shows the following data:

  • Fabric Name
The following example shows an alert when a fabric is created:
<118>1 2003-10-11T22:14:15.003Z xco.machine.com FaultManager - -     
   [meta sequenceId=”47”]    
   [origin ip=”10.x.x.x” enterpriseId=”1916” software=”XCO” swVersion=“3.3.0”]    
   [alert@1916    
   resource=”/App/Component/Fabric?fabric_name=fb” 
   alertId=”31700”  	    
   cause=”configCreated”  
   type=”fabricService”    
   severity=”info”]     
   [alertData@1916    
   fabric_name=”fb”  
   ] 
   BOM : The fabric “fb” is created.
Health Response
Response
{ 
    Resource: /App/Component/Fabric?fabric_name=fb 
    HQI { 
        Color: Green 
        Value: 0 
    } 
    StatusText : The fabric "fb" is created. 
}

Managed Fabric Health Degraded Notice

31799 Managed Fabric Health Degraded Notice
Description  Send an alert there is a change in fabric health or its contributors.
Preconditions  Fabric is created and devices are added and configured in the fabric.

Severity is based on the following fabric health color value:

  • Red: Major
  • Black: Critical
  • Green: Info (Raised by the alert ID 31800)
Requirements 

Alert shows the following data:

  • Fabric Name
  • Fabric Health Info
The following example shows an alert when a fabric health is degraded:
<114>1 2003-10-11T22:14:15.003Z xco.machine.com FaultManager - -     
   [meta sequenceId=”47”]    
   [origin ip=”10.x.x.x” enterpriseId=”1916” software=”XCO” swVersion=“3.3.0”]    
   [alert@1916    
   resource=”/App/Component/Fabric?fabric_name=fb” 
   alertId=”31799”  	    
   cause=”fabricHealthCritical”  
   type=“fabricService”    
   severity=”critical”]     
   [alertData@1916    
   fabric_name=”fb”  
fabric_health_info=” Name : fb  
Health: Black 
Devices:[ 
 { 
  Ip: 10.x.x.1, 
  Device_health: Black 
  Oper_state_health:{ 
     Underlay_health:{ 
        	Health: Black 
Errors:[ 
   Source_ip: 10.x.x.1  
   Source_asn: 6500 
    Destination_asn: 6512 
    Destination_ip: 10.x.x.2 
    Neighbor_ip: 10.x.x.y 
    Underlay_state: CONN 
    Error: session_not_established 
     	 
] 
      }, 
     
                                   } 
} 
 
] 
” 
   ] 
   BOM : Fabric "fb" health is in critical state because the underlay topology has errors for the device(s) [10.x.x.1,10.x.x.2]
Health Response
Response
{ 
    Resource: /App/Component/Fabric?fabric_name=fb 
    HQI { 
        Color: Black 
        Value: 4 
    } 
    StatusText : Fabric "fb" health is in critical state because the underlay topology has errors for the device(s) [10.x.x.1,10.x.x.2].

Managed Fabric Health Restored Alert Notice

31800 Managed Fabric Health Restored Alert Notice
Description  Send an alert when the fabric health is changed to Green.
Preconditions  Fabric is created and devices are added and configured in the fabric.
Requirements 

Alert shows the following data:

  • Fabric Name
  • Fabric Health Info
The following example shows an alert when a fabric health is restored:
<118>1 2003-10-11T22:14:15.003Z xco.machine.com FaultManager - -     
   [meta sequenceId=”47”]    
   [origin ip=”10.x.x.x” enterpriseId=”1916” software=”XCO” swVersion=”3.3.0”]    
   [alert@1916    
   resource=”/App/Component/Fabric?fabric_name=fb” 
   alertId=”31800”  	    
   cause=”fabricHealthRestored”  
   type=”fabricService”    
   severity=”info”]     
   [alertData@1916    
   fabric_name=”fb”  
  fabric_health_info=”” 
   ] 
   BOM : The fabric “fb” health is restored.
Health Response
Response
{ 
    Resource: /App/Component/Fabric?fabric_name=fb 
    HQI { 
        Color: Green 
        Value: 0 
    } 
    StatusText : The fabric “fb” health is restored.
9037858-01 RevAA