For the last couple of weeks, my WAN connection here has been failing over from the Cable Modem to the backup DSL connection pretty frequently and the root cause has left me baffled.  Until tonight.

The failover is handled by my trusty old Sonicwall TZ-170 firewall.  It handles both the cable and dsl connections and can handle load balancing and failover.  For me, this combination makes sense since my public facing servers are all located at hosting centers so there's no need for a commercial network connection here. The total cost to have two different consumer grade connections isn't bad, and the reliability I get with the fail-over is worth it.

The problem has been that for the last couple of weeks, the faster cable connection has been periodically failing for no visible reason. Tests show me that the link is good, and if reboot the firewall it goes back to being happy...for a while. This configuration has been stable and reliable for a long time so it really had me scratching my head. It turns out that the answer was right there in the words "Trusty Old Sonicwall". The system had been reliable and stable, so I'd had no reason to look into updated firmware and it completely slipped my mind for the last several years.

You may remember that in 2007 we changed the date on which we switch the clocks for Daylight Savings Time. Well, I'd been lax in updating that trusty old Sonicwall and it was using the old dates. For the last few weeks, it's been off by one hour. When the system boots, it makes a brand new DHCP request and gets an address, then every few hours, it issues a DHCP "RENEW" transaction so that the address remains valid. When the DHCP RENEW transaction hit the provider's DHCP server, it had a timestamp that was off by an hour and was ignored. The firewall decided that since it couldn't renew its address, it had to invalidate that network port. Failover occurred.

I was able to test this theory by turning off the NTP (Network Time Protocol) settings and manually set the time on the firewall. Once I validated the fix, I went out and got an updated firmware for the TZ-170 and all is well.