13.4.11 Graceful restart

<Structure of this section>

(1) Overview

Graceful restart is a function that reduces communication stop time by removing routes from the network when the master switch is switched or the routing program is restarted by an operation command during stack configuration.

In BGP4, the router on which BGP4 is restarted by a graceful restart is known as a restarting router. A neighboring router assisting the graceful restart is known as a receiving router.

The Switch supports the restart router function and the receive router function. The following figure shows an example of a graceful restart with the Switch.

Figure 13-28: Example of graceful restart

The external peer uses the address of the directly connected interface as the peer address. The internal peer uses the address set for the loopback interface as the peer address. In AS100, IGP exchanges routes addressed to the loopback interface.

Suppose that Switch B and Switch C on AS100 establish an internal-peer BGP connection whose peer address is the address set for the loopback interface. In addition, Switch B establishes a BGP connection between Switch A on AS200, and Switch C establishes an external peer whose peer address is the address of AS300 router. For each BGP connection, a graceful restart negotiation is established. When Switch B performs a graceful restart, Switch A and Switch C, which have BGP connections with the corresponding switch, operate as a receive router and continue forwarding packets through Switch B without stopping. This enables the end-to-end communication to be maintained through Switch B.

The conditions for a successful BGP4 graceful restart are as follows. If these conditions are not satisfied, a normal restart is performed and communication halts.

To operate the Switch as a restart router, restart or both parameters of the configuration command bgp graceful-restart mode must be set.
When the Switch operates as a receiving router, receive or both parameters of the configuration command bgp graceful-restart mode must be set.
A graceful restart has been negotiated between the device performing the graceful restart and the device performing the receiving router's roles in BGP connection.

(2) Graceful restart operation procedure

The following figure shows the sequence of steps for performing a graceful restart.

Figure 13-29: Sequence of graceful restart

Negotiates a graceful restart when establishing a BGP connection between a graceful restart router and its neighboring router, and prepares to perform a graceful restart.
The graceful restart commences when the router performing the graceful restart begins operating as the restarting router.
When the BGP connection closes, the neighboring router begins operating as a receiving router.It keeps the routing information it has already learned from the restarting router and continues forwarding the received packets.
When the BGP connection with the neighboring router is re-established, routing information is first distributed back from the receiving router to the restarting router.
When the restarting router has completed learning all the routing information for the protocols involved in the graceful restart, it distributes its routing information to the receiving router.
This step is the same as step 5.
Lastly, the receiving router discards the stale routing information that it previously learned from the restarting router and did not receive again after the BGP connection was re-established.

(3) Restart Router Functions

(a) Trigger

The following table describes the times when BGP restart router functionality operates on the Switch:

When the master switch is switched in the stack configuration
When the unicast routing program restarts.

(b) Restart Router Functions

After a graceful restart starts, the maximum wait time for BGP connection to be reestablished is monitored according to the specification in the configuration command bgp graceful-restart restart-time (see (a) in Figure 13-29: Sequence of graceful restart). If BGP connection is not reestablished within this period, the restart router does not wait for routing information to be distributed from the corresponding receiver router, but starts routing information distribution from the local router. This prevents the corresponding receiver router, which appears to be in an unstable state, from affecting path convergence.

The maximum length of time that the restart router waits for the completion of routing information reception and starts routing distribution depends on the specified configuration command routing options graceful-restart time-limit (see (c) in Figure 13-29: Sequence of graceful restart).

(c) Reference value of timer parameter

The table below shows the standard values for setting each timer parameter used in the restart router function.

Table 13-13: Reference values for timer parameters
Parameter	Reference value
restart-time^#1	Time required for switching the master switch + 30 seconds
time-limit^#2	Restart-time+60 seconds

Note #1: For an estimate of the time required for switching the master switch, see 8.11.3 Notes on using graceful restart. The approximate duration of 30 seconds is the length of time it takes to establish a BGP peering connection. May vary depending on the configuration.
Note #2: The approximate length of 60 seconds is the number of seconds after a BGP peer connection is reestablished until all routes have been learned and the forwarding table has been updated. Depends on the number of routes and the number of neighboring peers.

(d) When graceful restart fails

The graceful restart of BGP fails in the following cases:

If a BGP connection to a neighboring device is not reestablished after restart-time has elapsed, communication through the peer device stops.
If the peer unit that performs the receive router function restarts during graceful restart of the Switch, communication through the relevant peer unit stops.
If the peer device that performs the receive router function cannot hold the routing information learned from the Switch before the graceful restart starts, communication through the peer device stops.
If BGP connection is disconnected again after the graceful restart has started, and the routing data has not been distributed to all the receiver routers, communication through the relevant peer device is stopped.
If the number of routes learned from the restarting router exceeds the maximum number of routes learned by BGP4 learned route limit function after a graceful restart, and BGP connection is disconnected again, communication through the restarting router stops.

(4) Receive router features

(a) Trigger

The BGP4 receiving router functionality is activated in the Switch when:

The switch detects that the TCP session being used by the BGP connection has terminated, although no final NOTIFICATION message was received from the connected peer.
An OPEN message is received from a connected peer and a new TCP session is established on the BGP connection.

(b) Receive router features

After a graceful restart starts, the maximum wait time for BGP connection to be reestablished is monitored according to the specification in the configuration command bgp graceful-restart restart-time (see (b) in Figure 13-29: Sequence of graceful restart). If the BGP connection is not re-established within this time, the receiving router discards the routing information learned from the restarting router and suspends packet forwarding via the restarting router.

Restart-time is reported to the peer when the graceful restart is negotiated. If the value received from the peer by the Switch is less than its own setting, the BGP connection is monitored for the restart-time value reported to the switch.

The maximum length of time that the receiver router retains learned routing information prior to restarting the restart router is specified in the configuration command bgp graceful-restart stalepath-time (see (d) in Figure 13-29: Sequence of graceful restart).

(c) Reference value of timer parameter

The table below shows the standard values for setting each timer parameter used in the receive router function.

Table 13-14: Reference values for timer parameters
Parameter	Reference value
restart-time^#1	Time required to switch the master switch of the restart router + 30 seconds
stalepath-time^#2	Time-limit+120 seconds for the restart router

Note #1: For an estimate of the time required for switching the master switch, see 8.11.3 Notes on using graceful restart. The approximate duration of 30 seconds is the length of time it takes to establish a BGP peering connection. May vary depending on the configuration.
Note #2: The approximate time is 120 seconds after the restart router has completed route learning for all protocols, until it learns the routes distributed by the restart router and updates the forwarding table. This depends on the number of routes and the number of neighboring peers on the restart router.

(d) Case where the receiving router function fails

The BGP4 graceful restart functionality fails if communication via the restarting router stops. This can happen for the following reasons:

A BGP connection with the restarting router was not established within the restart time after commencement of a graceful restart.
The receiving router was restarted during operation as a receiving router.
The restarting router could not retain the routing information it learned before commencing a graceful restart.
The BGP connection that was established after commencement of a graceful restart was lost before distribution of routing information from the restarting router was complete.
The BGP connection was terminated again after commencement of a graceful restart because the number of BGP4 routes learned from the restarted router exceeded the limit.

(5) Notes on using graceful restart

Using TCP MD5 with a graceful restart

Under the BGP4 protocol rules, if the peer requests a new connection after establishing a BGP connection that supports the graceful restart functionality, the established BGP connection is dropped and the new BGP connection is used. To prevent security issues due to this behavior, use TCP MD5 authentication together with the graceful restart functionality.
Graceful restart in an IGP environment

If a IGP is used to resolve the NEXT_HOP attribute of BGP route information by using a IGP route, such as when routing information destined for a peer address is exchanged through an internal peer connection that is not directly connected, or when route reflection is used, also set graceful restart for IGP route. At this time, set BGP restart time to a time longer than IGP restart time + connection establishment time of BGP peer.
About graceful restart when using an extranet

If a Switch that functions as a restart router uses an extranet that exchanges routes between VRF, it takes 30 seconds longer than normal to distribute routes to the receiving router when graceful restart is performed. For this reason, set the upper limit of the routing information retention time (configuration command bgp graceful-restart stalepath-time) on the receiving router for 30 seconds longer than normal.