Home Support Blog

Failover Testing: Proving a Redundant Link Actually Fails Over

Release date:2026-11-19

You prove a redundant link really fails over with failover testing: deliberately cut the active path during a planned window, then time how long live traffic takes to return. No amount of "the backup is configured" survives a real cut. Until you pull a fiber or drop a power feed and watch the packets, link redundancy is a diagram, not a guarantee. A working test plan records what broke, how long recovery took, and whether service came back.

Rayin (Shenzhen Rayin Technology) is a manufacturer of GPON OLTs and industrial Ethernet switches.

KEY TAKEAWAYS
  • Define "failure" and "service restored" before you cut anything: a failover test plan that lacks an acceptance threshold cannot pass or fail.

  • Test all four break types — pulled fiber, power loss, uplink loss and a hung device — because each exercises a different failure domain and a different convergence time.

  • Sub-50 ms ITU-T G.8032 ring protection is an industry-typical design target; measure your own recovery and label it as a reference, never as a product spec.

What counts as a failure before you run a failover test?

Failover testing needs two written definitions before anyone touches a connector. "Failure" is the single event you induce — one fiber pulled, one supply dropped, one uplink lost, one device held in reset. "Service restored" ends the clock: a stream playing without freezing, a PLC session that stays up. A Rayin industrial Ethernet switch logs a port-down and a topology-change event for each cut, the raw evidence. Write the threshold down; a test without one produces an argument, not a result. Under the ITU-T G.984 series a GPON port registers up to 128 ONUs, so decide whether the test covers one ONU or the whole port. Keep both definitions in the failover test plan and have the owner sign them before the window opens.

Rayin products

Why does every failover test need a maintenance window and a rollback plan?

A failover test interrupts live traffic by design: you remove a working path to watch the network repair itself, and while it does, real packets drop. There is no honest way to prove switchover without provoking it, so do it inside an approved maintenance window, with users told and a rollback ready. Book a window long enough for the full matrix plus a re-test, freeze unrelated changes, and save the previous configuration so you can restore the topology in minutes if the backup fails to carry load. A rollback plan is not a formality: if the secondary path is misconfigured, the test has just demonstrated an outage. Record the window ID, the owner and the restore command set in the plan before the first fiber is touched. RFC 2544 and ITU-T Y.1564 make those interruptions measurable as frame loss.

What does each of the four link-break scenarios actually verify?

Four cuts, four questions. Pulling a fiber tests path diversity and the optic's loss-of-signal reaction. Dropping a power feed tests whether dual supplies and the far-end device stay reachable. Losing an uplink tests the aggregation or routing layer; holding a device in reset tests whether a neighbour takes over the control role. When you map the uplink-loss case, comparing uplink redundancy options when you size an OLT uplink port helps you decide whether the backup is a second SFP or a separate fiber route. True link redundancy means the paths share no common conduit, supply or card. LACP, standardized as IEEE 802.3ad and later IEEE 802.1AX, reacts to a member-link loss, while ITU-T G.8032 handles ring protection — different failure domains, so one cut cannot stand in for all four.

Break typeWhat it exercisesSignalPass criterion
Fiber pulledPath diversity, LOS reactionPort down, LOS alarmTraffic returns within target
Power feed droppedDual supplies, far endPower alarmNo supply loss isolates the site
Uplink lostAggregation or routing layerLACP member downUpper-layer path reconverges
Device in resetControl-role takeoverNeighbour state changeStandby takes over cleanly

How do you measure convergence time and packet loss during the switch?

Measure three things at once: the recovery interval, lost frames, and whether the application came back. The recovery interval runs from the induced cut to the first frame on the new path; count dropped packets with a fixed-rate test flow, comparing sent versus received sequence numbers, a method from RFC 2544 benchmarking. Confirm the application recovered too — a PLC poll that resumes — because a low packet count means little if the session had to log in again. Industry-typical protection targets sit near the sub-50 ms figure associated with ITU-T G.8032 ring protection, but treat that as a design reference; the convergence time you measure depends on topology, load and the protocol in use. RFC 2544 uses 64-byte frames for a worst-case check, so repeat each cut at least three times at that size. A Rayin industrial Ethernet switch exposes the port counters and topology-change log you need, so capture those throughout each cut. Log the numbers below so the next audit has evidence.

MetricHow to capturePass criterionMethod source
Recovery intervalCut to first frame back timestampBelow the agreed thresholdRFC 2544 and ITU-T Y.1564 timing
Lost framesFixed-rate flow, sent minus receivedWithin the loss budgetRFC 2544 frame-loss test
Application recoverySession poll or stream checkSession resumes in timeITU-T Y.1564 activation
Flap countRepeat each cut three timesNo repeated re-convergenceOperational practice

Why does a protocol showing "up" not prove service survived?

A protocol state and a working service are different layers. RSTP, standardized as IEEE 802.1w and later folded into IEEE 802.1D, can report a port forwarding while the MAC table is still relearning and frames are dropped; ERPS under ITU-T G.8032 can signal a protected ring while the far-end host has not re-ARPed. An OLT that performs link-loss detection for ONU power loss and fiber cuts raises an alarm in seconds, but that is a control-plane event, not proof that subscriber traffic was restored. This is the gap that link redundancy is supposed to close: the backup path is only proven when real frames cross it. Watch for silent failures — a missing VLAN on the standby trunk, a full MAC table. One cut that shows "up" everywhere but drops a call is a failed test a paper review never catches.

What the device saysWhat can still be brokenHow to prove it
Port "up", RSTP forwardingMAC relearning, flooded framesFixed-rate flow, sent versus received
Ring "protected" (ERPS)Gateway not refreshedActive session poll during recovery
LACP bundle "up"One member hashing to a dead pathPer-member counter check
OLT alarm clearedSubscriber session not re-establishedApplication-level reachability check

What should a clean failover test report contain?

A report someone else can re-run. List the topology, the exact break induced, the window and owner, and the three recorded numbers — recovery interval, lost frames and application result — for each of the four cuts, repeated three times at the RFC 2544 64-byte frame size. Attach the logs and captures, then state whether each case met the threshold written in the failover test plan. Note anything that behaved differently under load, because a ring that recovers in a quiet lab may stall at full 1 Gbps line rate. Close with the rollback steps used and the sign-off. That document, not a green tick in a dashboard, is the proof the redundant design works on this site.

FAQs

Does a failover test have to interrupt live traffic?

Yes. A real switchover test removes a working path, so frames are dropped while the network reconverges; that is why it belongs in a booked maintenance window, with users notified and a rollback ready. A check that never disturbs traffic has verified configuration only, not recovery. Plan the cut, measure it, then restore.

Which switch counters should I record during the test?

Port state, CRC errors, lost-frame counts and the topology-change log, captured before, during and after each cut. On a Rayin managed switch, those counters and the syslog stream are the evidence you keep with the report. Timestamps need one common NTP source, or you cannot line up the cut with the recovery.

How many times should I repeat each cut?

At least three times per break type. One clean result can hide a flapping backup path or a MAC table that converges slowly under load; repeating exposes the intermittent case. Keep the same fixed-rate flow and frame size each run so the numbers compare, and log every run, including the failures.

What is the difference between a link-down alarm and a failed service?

A link-down alarm is a control-plane event: a port changed state. The service is the traffic users depend on, and it can stay broken after the alarm clears because the host has not re-ARPed or a session must log in again. Always pair the alarm with an application check such as a PLC poll.

Can I run the test without a dedicated tester?

Yes, for the basic cases. A laptop sending a fixed-rate ping or UDP flow, plus the switch's own port counters and syslog, measures lost frames and the recovery interval for a pulled fiber or a dropped feed, as RFC 2544 describes. You get repeatable evidence that the backup path carried traffic, without a dedicated instrument.

Related Reading


How this article was made

The statements above draw on public industry sources: RFC 2544 for benchmarking methodology and frame sizes, ITU-T Y.1564 for Ethernet service activation testing, ITU-T G.8032 for ring protection switching, the IEEE 802.1D and 802.1w spanning tree series, IEEE 802.3ad and 802.1AX for link aggregation, and the ITU-T G.984 series for the GPON registration ceiling. Standard numbers are cited by series name only where a specific clause was not verified. The four break types, the record tables and the wording were written by Sara, who checked every figure against those sources before publication. Anything presented as an industry-typical value is labelled as such; nothing here is a measured figure from a Rayin product or a claim of any certification.

About the author: Sara — Sara is a Customer Manager at Rayin with over 10 years of experience in the communications field. In her free time, she enjoys badminton and swimming.

Connect with Sara on LinkedIn


About Rayin: Shenzhen Rayin Technology Co., Ltd. — Company Profile

Get A Quote

You have agreed to this website’s《Privacy Policy》