ECMP Hardware Table

IP route sharing and the ECMP (Equal Cost Multi Paths) hardware table are introduced in IP Route Sharing and ECMP. The following sections provide guidelines for managing the ECMP hardware table:

ECMP Table Configuration Guidelines
Troubleshooting: ECMP Table-Full Messages

ECMP Table Configuration Guidelines

The ECMP table contains gateway sets, and each gateway set defines the equal-cost gateways that lead to a destination subnet. When IP route sharing is enabled, subnet entries in the LPM table can be mapped to gateway set entries in the ECMP table, instead of to a single gateway within the LPM table.

For improved ECMP scaling, each LPM table entry points to a gateway set entry in the ECMP table. Each gateway set entry is unique and appears only once in the ECMP table.

Each gateway set entry for the platforms listed above is unique and appears only once in the ECMP table. Multiple LPM table entries can point to the same gateway set entry. This efficient use of the ECMP table creates more room in the ECMP table for additional gateway set entries. It also makes IP route sharing available to every entry in the LPM table.

The following command allows you to configure the maximum number of next-hop gateways for gateway sets in the ECMP table:

configure iproute sharing max-gateways max_gateways

Each gateway entry in a gateway set consumes ECMP table space. As the max_gateways value decreases, the ECMP table supports more gateway sets. If you configure the max_gateways value to 32, the switch supports route sharing through up to 32 gateways per subnet, but supports the smallest number of gateway sets. If you do not need to support up to 32 different gateways for any subnet, you can decrease the max_gateways value to support more gateway sets.

To determine which gateways might be added to the ECMP table, consider how many local gateways are connected to the switch and can be used for ECMP, and consider the max_gateways value.

For example, suppose that you have four ECMP gateway candidates connected to the switch (labeled A, B, C, and D for this example) and the max_gateways option is set to 4. For platforms that allow a gateway set entry to support multiple subnets, this configuration could result in up to 11 gateway sets in the ECMP table: ABCD, ABC, ABD, ACD, BCD, AB, AC, AD, BC, BD, and CD.

If there are 4 gateways and you set max-gateways to 4, you can use the choose mathematical function to calculate the total number of gateway set possibilities as follows:

(4 choose 4) + (4 choose 3) + (4 choose 2) = 11

To calculate the number of gateway set possibilities for a given number of total gateways and a specific max-gateways value, use the choose function in the following formula:

(TGW choose MGW) + (TGW choose MGW-1) + ... + (TGW choose 2) = TGWsets

In the formula above, TGW represents the total local gateways, MGW represents the max_gateways value, and TGWsets represents the total gateway sets needed to support all possible shared paths.

To see if your platform supports the total gateway sets needed, do the following:

Calculate the total ECMP gateway sets possible as described above.
Compare your result to the IP route sharing (total combinations of gateway sets) capacities listed in the ExtremeXOS Release Notes to verify that the switch can support the desired number of gateway sets.

Troubleshooting: ECMP Table-Full Messages

If the ECMP table is full, no new gateway sets can be added, and IP forwarding is still done in hardware through one of the following:

For platforms that allow a gateway set entry to support multiple subnets, forwarding can be done using an existing gateway set that is a partial subset of the unavailable gateway set. If the unavailable gateway set consists of N gateways, the subset used could include a range of gateways from N-1 gateways down to a single gateway. For example, if the ECMP table does not have room for a new gateway set using gateways E, F, G, and H, a partial subset such as EFG, EF, or E will be used.
For platforms that require one gateway set entry per subnet, forwarding is done through a single gateway.

On Summit family switches, an ECMP table-full condition produces the following message:

<Info:Kern.IPv4FIB.Info> Slot-1: IPv4 route can not use sharing on all its gateways.  Hardware ECMP Table full.  Packets are HW forwarded across a subset of gateways.   (Logged at most once per hour.)

If the ECMP table-full message appears, consider the following remedies:

Reduce the number of gateways adjacent to the switch used for IP route sharing.
Monitor the switch to see if the condition is transient. For example, if the number of ECMP table entries temporarily increases due to a network event and then returns to within the supported range, a permanent change might not be required.
Determine if IP route sharing to all gateways is required. Since traffic is still being forwarded in hardware using one or more gateways, the condition may be acceptable.

Published March 2020prev | next

Email this topic

Print this page Print this page

Leave feedback Feedback