]> NTT Corporation

3-9-11 Midori-cho Musashino-shi, Tokyo 180-8585 Japan ogawa.kentaro@lab.ntt.co.jp

Zhejiang Gongshang University

149 Jiaogong Road Hangzhou 310035 P.R.China +86-571-88057712 wmwang@mail.zjgsu.edu.cn

University of Patras

Patras Greece ehalep@ece.upatras.gr

Mojatatu Networks

Ottawa, Ontario Canada hadi@mojatatu.com

Routing RFC Request for Comments I-D Internet-Draft ForCES HA This document discusses CE High Availability within a ForCES NE.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119. The following definitions are taken from and : Logical Functional Block (LFB) -- A template that represents a fine-grained, logically separate aspects of FE processing. ForCES Protocol -- The protocol used at the Fp reference point in the ForCES Framework in . ForCES Protocol Layer (ForCES PL) -- A layer in the ForCES architecture that embodies the ForCES protocol and the state transfer mechanisms as defined in . ForCES Protocol Transport Mapping Layer (ForCES TML) -- A layer in ForCES protocol architecture that specifically addresses the protocol message transportation issues, such as how the protocol messages are mapped to different transport media (like SCTP, IP, TCP, UDP, ATM, Ethernet, etc), and how to achieve and implement reliability, security, etc.

illustrates a ForCES NE controlled by a set of redundant CEs with CE1 being active and CE2 and CEn-1 being a backup.

The ForCES architecture allows FEs to be aware of multiple CEs but enforces that only one CE be the master controller. This is known in the industry as 1+N redundancy. The master CE controls the FEs via the ForCES protocol operating in the Fp interface. If the master CE becomes faulty, a backup CE takes over and NE operation continues. By definition, the current documented setup is known as cold-standby. The CE set is static and is passed to the FE by the FE Manager (FEM) via the Ff interface and to each CE by the CE Manager (CEM) in the Fc interface during the pre-association phase. From an FE perspective, the knobs of control for a CE set are defined by the FEPO LFB in , Appendix B. of this document details these knobs further.

It is assumed that the reader is aware of the ForCES architecture to make sense of the changes made here. This document provides minimal background to set the context of the discussion in . By current definition, the Fr interface is out of scope for the ForCES architecture. However, it is expected that organizations implementing a set of CEs will need to have the CEs communicate to each other via the Fr interface in order to achieve the synchronization necessary for controlling the FEs. The problem scope addressed by this document falls into 2 areas: To describe with more clarity (than ) how current cold-standby approach operates within the NE cluster. To describe how to evolve the cold-standby setup to a hot-standby redundancy setup so as to improve the failover time and NE availability.

The NE recovery and availability is dependent on several time-sensitive metrics: How fast the CE plane failure is detected the FE. How fast a backup CE becomes operational. How fast the FEs associate with the new master CE. How fast the FEs recover their state and become operational. The design goals of the current choices to meet the above goals are driven by desire for simplicity. To quantify the above criteria with the current prescribed ForCES CE setup in : How fast the CE side detects a CE failure is left undefined. To illustrate an extreme scenario, we could have a human operator acting as the monitoring entity to detect faulty CEs. How fast such detection happens could be in the range of seconds to days. A more active monitor on the Fr interface could improve this detection. How fast the backup CE becomes operational is also currently out of scope. In the current setup, a backup CE need not be operational at all (for example, to save power) and therefore it is feasible for a monitoring entity to boot up a backup CE after it detects the failure of the master CE. In this document we suggest that at least one backup CE be online so as to improve this metric. How fast an FE associates with new master CE is also currently undefined. The cost of an FE connecting and associating adds to the recovery overhead. As mentioned above we suggest having at least one backup CE online. In we propose to zero out the connection and association cost on failover by having each FE associate with all online backup CEs after associating to the active CE. Note that if an FE pre-associates with backup CEs, then the system will be technically operating in hot-standby mode. And last: How fast an FE recovers its state depends on how much NE state exists. By ForCES current definition, the new master CE assumes zero state on the FE and starts from scratch to update the FE. So the larger the state, the longer the recovery.

To achieve CE High Availabilty, FEs and CEs MUST inter-operate per definition which is repeated for contextual reasons in . It should be noted that in this default setup, which MUST be implemented by CEs and FEs needing HA, the Fr plane is out of scope (and if available is proprietary to an implementation).

As mentioned earlier, although there can be multiple redundant CEs, only one CE actively controls FEs in a ForCES NE. In practice there may be only one backup CE. At any moment in time only one master CE can control the FEs. In addition, the FE connects and associates to only the master CE. The FE and the CE PL are aware of the primary and one or more secondary CEs. This information (primary, secondary CEs) is configured on the FE and the CE PLs during pre-association by the FEM and the CEM respectively. below illustrates the Forces message sequences that the FE uses to recover the connection in current defined cold-standby scheme.

| | | | | | state update | | 2 |<--------------------->| | | | | | | | | FAILURE | | | | Asso Estb,Caps exchange | 3 |<------------------------------------------>| | | | Event Report (pri CE down) | 4 |------------------------------------------->| | | | state update from scratch | 5 |<------------------------------------------>| ]]>

High Availability parameterization in an FE is driven by configuring the FE Protocol Object (FEPO) LFB. The FEPO CEID component identifies the current master CE and the component table BackupCEs identifies the backup CEs. The FEPO FE Heartbeat Interval, CE Heartbeat Dead Interval, and CE Heartbeat policy help in detecting connectivity problems between an FE and CE. The CE Failover policy defines how the FE should react on a detected failure. illustrates the defined state machine that facilitates connection recovery. The FE connects to the CE specified on FEPO CEID component. If it fails to connect to the defined CE, it moves it to the bottom of table BackupCEs and sets its CEID component to be the first CE retrieved from table BackupCEs. The FE then attempts to associate with the CE designated as the new primary CE. The FE continues through this procedure until it successfully connects to one of the CEs.

-----+ | | (CE issues Teardown || +---+--------v----+ Lost association) && | Pre-Association | CE failover policy = 0 | (Association | +------------>-->-->| in +<----+ | | progress) | | | CE Issues +--------+--------+ | | Association | | CEFTI | Response V | timer | ___________________+ | expires | | ^ | V | +-+-----------+ +------+-----+ | | | Not | | | (CE issues Teardown || | Associated | | | Lost association) && | +->---+ | Associated | CE Failover Policy = 1 |(May | FE | | | | Continue |try v | |-------->------->------>| Forwarding)|assn | | | | |-<---+ | | | | +-------------+ +-------+-----+ ^ | | CE Issues v | Association | | Setup | +_________________________________________+ ]]>

When communication fails between the FE and CE (which can be caused by either the CE or link failure but not FE related), either the TML on the FE will trigger the FE PL regarding this failure or it will be detected using the HB messages between FEs and CEs. The communication failure, regardless of how it is detected, MUST be considered as a loss of association between the CE and corresponding FE. If the FE's FEPO CE Failover Policy is configured to mode 0 (the default), it will immediately transition to the pre-association phase. This means that if association is again established, all FE state will need to be re-established. If the FE's FEPO CE Failover Policy is configured to mode 1, it indicates that the FE is capable of HA restart recovery. In such a case, the FE transitions to the Not Associated state and the CEFTI timer[RFC 5810] is started. The FE MAY continue to forward packets during this state. It MAY also recycle through any configured backup CEs in a round-robin fashion. It first adds its primary CE to the bottom of table BackupCEs and sets its CEID component to be the first secondary retrieved from table BackupCEs. The FE then attempts to associate with the CE designated as the new primary CE. If it fails to re-associate with any CE and the CEFTI expires, the FE then transitions to the pre-association state. If the FE, while in the not associated state, manages to reconnect to a new primary CE before CEFTI expires it transitions to the Associated state. Once re-associated, the CE tries to synchronize any state that the FE may have lost during the not associated state. How the CE re-synchronizes such state is out of scope for the current ForCES architecture but would include issuing new configs and queries. An explicit message (a Config message setting Primary CE component in ForCES Protocol object) from the primary CE, can also be used to change the Primary CE for an FE during normal protocol operation. In this case, the FE transitions to the Not Associated State and attempts to Associate with the new CE.

TML Level: The TML controls logical connection availability and failover. The TML also controls peer HA management. At this level, control of all lower layers, for example transport level (such as IP addresses, MAC addresses etc) and associated links going down are the role of the TML. PL Level: All other functionality, including configuring the HA behavior during setup, the CE IDs used to identify primary and secondary CEs, protocol messages used to report CE failure (Event Report), Heartbeat messages used to detect association failure, messages to change the primary CE (Config), and other HA related operations described in , are the PL's responsibility. To put the two together, if a path to a primary CE is down, the TML would take care of failing over to a backup path, if one is available. If the CE is totally unreachable then the PL would be informed and it would take the appropriate actions described before.

In this section we describe small extensions to the existing scheme to enable hot standby HA. To achieve hot standby HA, we target specific goals defined in , namely: How fast a backup CE becomes operational. How fast the FEs associate with the new master CE. As described in , in the pre-association phase the FEM configures the FE to make it aware of all the CEs in the NE. The FEM MUST configure the FE to make it aware of which CE is the master and MAY specify any backup CE(s).

In order for the above to be achievable there is a need to make a few changes in the FEPO model. contains the xml definition of the new version 2 of the FEPO LFB. Changes from the version 1 of FEPO are: Addition of a new datatype, status (unsigned char) with special values 0 (Disconnected), 1 (Connected), 2 (Associated), 3 (Lost_Connection) and 4 (Unreachable). Change Component BackupCEs (9) to AllCEs and instead of an Array of unsigned integers(CEID), it MUST be an Array of unsigned integers (CEID) and unsigned char (status) for each CE. Add two special values to the CEFailoverPolicyValues. 2 (High availability without Graceful restart) and 3 (High availability with Graceful restart). Added one additional Event, the HAPrimaryCEDown event which reports last known CEID and tentative new master CEID. As the FEPO component 9 is not backwards compatible with the previous version there is the issue of interoperability between CE and FE. However this is a pre-association version mismatch and the managers have to identify the issue and not allow an association that would fail or cause problems.

The FE's FEPO LFB version 2 AllCEs table (previously BackupCEs) contains all the CEIDs that the FE may connect and associate with. The ordering of the CE IDs in this table defines the priority order in which an FE will connect to the CEs. In the pre-association phase, the first CE ID (lowest table index) in the AllCEs table MUST be the first CE ID that the FE will attempt to connect and associate with. If the FE fails to connect and associate with the first CE ID, it will attempt to connect to the second CE ID and so forth, and cycles back to the beggining of the list until there is a connection and an association. The FE MUST associate with at least one CE. Upon a successful association, the FEPO's CEID component identifies the current associated master CE. For the sake of simplicity, the FE MUST respond to messages issued only by the master CE. This simplifies the synchronization and avoids the concept of locking FE state. i.e the FE MUST drop any messages from backup CEs. However, asynchronous events that the master CE has subscribed to, as well as heartbeats are sent to all associated-to CEs. Packet redirects continue to be sent only to the master CE. The Heartbeat Interval, the CEHB Policy and the FEHB Policy MUST be the same for all CEs. illustrates the state machine that facilitates connection recovery with High Availability enabled.

-----+ | | ^ v (CE issues Teardown || +----+--------+---+ Lost association) && | Pre-Association | CE failover policy = 0 | (Association +<-------------------+ +------------>-->-->| in +<-----+ | | | progress) | | | | CE Issues +--------+--------+ | | | Association | | | | Response V Not Found || CEFTI | | ___________________+ timer expires | | | | | | V ^ | +-+-----------+ +------+------+ | | | | Not | | | | (CE issues Teardown || | Associated | | | | Lost association) && | | CEFTI | Associated | (CE Failover Policy=2|| | (May | timer | | CE Failover Policy=3) | Continue | expires | +---------->------->----->| Forwarding)| | | | | | | | | | Search for | | | | +--------->| next | | | | | | associated | | | | | | CE | | +-------------+ | +-------------+ | ^ | V | | | | | | | Found CE | | CEHDI Expires Send Event of | | | New CE ID. | | | | | | | V | | | +------+------+ | | ^---------+ Confirm +-------^ | | State | | Received +---->| | | different | | Wait for CE | | CE ID. ^ | to confirm | | Resend Event | | new CE ID | | +----<| | | +-----+-------+ | Received same CE ID | +_______________________________________+ ]]>

Once the FE has associated with a master CE it moves to the post-association phase (Associated state). In this state, the master CE MAY update the list of backup CEs. It MAY also instruct the FE to use a different master CE. It is assumed that the master CE will communicate with other CEs within the NE for the purpose of synchronization via the CE-CE interface. The CE-CE interface is out of scope for this document.

| | | | | | | | state update | | | 2 |<-------------------->| | | | | | | | Asso Estb,Caps exchg | | 3I|<--------------------------------->| | ... ... ... ... | Asso Estb,Caps exchg | 3N|<------------------------------------------>| | | | | 4 |<-------------------->| | | . . . . 4x|<-------------------->| | | | FAILURE | | | | | | | Event Report (CE#2 is new master) | | 5 |---------------------------------->|------->| | | | | Config (Set CEID to CEID of CE#2) | | 6 |<----------------------------------| | 7 |<--------------------------------->| | . . . . 7x|<--------------------------------->| | . . . . ]]>

While in the post-association phase, if the CE Failover Policy is set to 2 (High Availability without Graceful Restart) or 3 (High Availability with Graceful Restart) then the FE, after succesfully associating with the master CE, MUST attempt to connect and associate with all the CEs that is aware of. steps #1 and #2 illustrates the FE associating with CE#1 as the master and then proceeding to steps #3I to #3N the association with backup CE's CE#2 to CE#N. If the FE fails to connect or associate with some CEs, the FE MAY flag them as unreachable to avoid continuous attempts to connect. The FE MAY retry to reassociate with unreachable CEs when possible. When the master CE for any reason is considered to be down, then the FE will try to find the first associated CE from the list of all CEs in a round-robin fashion. If the FE is unable to find an associated FE in its list of CEs, then it will attempt to connect and associate with the first from the list of all CEs and continue in a round-robin fashion until it connects and associates with a CE. Once the FE selects the associated CE to use as the new master, the FE then sends a High Availability Primary CE Changed Event Notification to all associated CEs to notifying them that the primary CE is down as well as which CE the reporting FE considers to be the new master. The new master CE MUST configure the CEID component of the FE within the time limit defined in the CEHDI Failover Timeout as a confirmation that the FE made the right choice.

| | | | | | | | state update | | | 2 |<-------------------->| | | | | | | | Asso Estb,Caps exchg | | 3I|<--------------------------------->| | | | | | ... ... ... ... | Asso Estb,Caps exchg | 3N|<------------------------------------------>| | | | | 4 |<-------------------->| | | . . . . 4x|<-------------------->| | | | FAILURE | | | | | | | Event Report (CE#2 is new master) | | 5 |---------------------------------->|------->| | | | | | CEHDI Failover Timeout | | | | | | | Event Report (CE#N is new master) | | 6 |---------------------------------->|------->| | | | | | Config (Set CEID to CEID of CE#N) | 7 |<-------------------------------------------| 8a|<------------------------------------------>| . . . . 8x|<------------------------------------------>| ]]>

If the FE does not get confirmation within the CEHDI Failover Timeout, it picks the next CE on its list and advertises it as the new master. illustrates in step #5 selecting CE#2 as its new master. In step #6, the timeout occurs and it picks CE#N as its new master. The FE receives confirmation that CE#N is the new master in step #7. If the CE the FE assumed to be the master discovers that it should not be the new master CE, then it will configure the CEID with the ID of the proper master CE. How the CE decides who the new master CE is, is also out of scope of this document and is assumed to be done via a CE-CE communication protocol. The FE must then associate with then new CE. If the CEFTI timer expires at either the not-associated or confirm states without a new master CE confirmed, then the FE MUST revert to the pre-association stage. In most High Availability architectures there exists the possibility of split-brain. However, since in our setup the FE will never accept any configuration messages from any other than the master CE, we consider the FE as fenced against data corruption from the other CEs that consider themselves as the master. The split-brain issue becomes mostly a CE-CE communication problem which is considered to be out of scope. By virtue of having multiple CE connections, the FE switchover to a new master CE will be relatively much faster. The overall effect is improving the NE recovery time in case of communication failure or faults of the master CE. This satisfies the requirement we set to achieve.

TBA

TBA

&rfc5810; &rfc3654; &rfc3746; &rfc5812;

XXX: Describe this to conform to LFB extensions as prescribed in the model

CEHBPolicyValues The possible values of CE heartbeat policy uchar CEHBPolicy0 The CE heartbeat policy 0 CEHBPolicy1 The CE heartbeat policy 1 FEHBPolicyValues The possible values of FE heartbeat policy uchar FEHBPolicy0 The FE heartbeat policy 0 FEHBPolicy1 The FE heartbeat policy 1 FERestartPolicyValues The possible values of FE restart policy uchar FERestartPolicy0 The FE restart policy 0 CEFailoverPolicyValues The possible values of CE failover policy uchar CEFailoverPolicy0 The CE failover policy 0 No High Availability or Graceful Restart. CEFailoverPolicy1 Graceful Restart CEFailoverPolicy2 High Availability without Graceful Restart CEFailoverPolicy3 High Availability with Graceful Restart FEHACapab The supported HA features uchar GracefullRestart The FE supports Graceful Restart HA The FE supports HA CEStatusType Status values. Status for each CE. uchar Disconnected No connection attempt with the CE yet. Connected The FE has connected with the CE. Associated The FE has associated with the CE. Lost_Connection The FE was associated with the CE but lost the connection. Unreachable The CE is deemed as unreachable by the FE. AllCEType Table Type for AllCE component. CEID ID of the CE uint32 CEStatus Status of the CE CEStatusType FEPO The FE Protocol Object 2.0 CurrentRunningVersion Currently running ForCES version u8 FEID Unicast FEID uint32 MulticastFEIDs the table of all multicast IDs uint32 CEHBPolicy The CE Heartbeat Policy CEHBPolicyValues CEHDI The CE Heartbeat Dead Interval in millisecs uint32 FEHBPolicy The FE Heartbeat Policy FEHBPolicyValues FEHI The FE Heartbeat Interval in millisecs uint32 CEID The Primary CE this FE is associated with uint32 AllCEs The table of all CEs. AllCEType CEFailoverPolicy The CE Failover Policy CEFailoverPolicyValues CEFTI The CE Failover Timeout Interval in millisecs uint32 FERestartPolicy The FE Restart Policy FERestartPolicyValues LastCEID The Primary CE this FE was last associated with uint32 SupportableVersions the table of ForCES versions that FE supports u8 HACapabilities the table of HA capabilities the FE supports FEHACapab PrimaryCEDown The pimary CE has changed LastCEID LastCEID HAPrimaryCEDown The primary CE has changed LastCEID CEID LastCEID ]]>