NTP Glitch Reverts Clocks Back To 2000

An anonymous reader writes "It seems a glitch of some sort wreaked havoc on some NTP servers yesterday, causing many machines to revert to the year 2000. It seems the Y2K bug that never happened is finally catching up with us in 2012."

Not an NTP glitch

[Anonymous Coward]

It was a problem with the USNO servers (I.e. tick.usno.navy.mil, tock.usno.navy.mil etc.) being rebooted and starting to hand out the wrong time. Very few downstream startum 2 NTP servers should have accepted such a large skew, although they may have lost accuracy.

Amusingly I happen to work with an ex. USNO NTP admin, so I'll be sure to take the piss for the rest of the week.

Properly configured hosts not impacted

[jaredmauch]

If you saw this problem, your NTP time sources were not properly configured and diverse.

Consider using the NTP pool and not relying on so few sources to properly sync your time. Read 5.3.3 and 5.3.4 from http://support.ntp.org/bin/view/Support/SelectingOffsiteNTPServers [ntp.org] for help to correct your NTP setup.

Re:Not an NTP glitch

[Anonymous Coward]

It seems that VMware ESXi servers grabbed the configuration with little issue.

Re:Not an NTP glitch

[Anonymous Coward]

Yes and no.
This article is interesting: http://support.microsoft.com/kb/884776
Summary: Windows can do it, but before Server 2008 it defaulted to not doing any sanity checks.
Since 2008 it still is quite generous, allowing 48 hour jumps.
If you don't like it you have to adjust the value in the registry.
I guess it still shows that the Internet was an afterthought for Microsoft...

Re:The Y2K bug was REAL

[brentrad]

I beg to differ about not experiencing significant problems on 1/1/2000. We had significant issues that caused all our approximately 2000 store servers to repeatedly shut down until we unloaded the offending software.

I was working for Hollywood Video in the Tech Support department (support for the rental stores and all their computer equipment) leading up to and after Y2K, in Wilsonville, Oregon at the corporate office. (Of course this was before their two bankruptcies.) The software development department performed extensive (and probably expensive) testing on every facet of our current in-store software and hardware setup (custom COBOL software running on DOS 5.0 on NetWare 3.1 if you can believe it.) They were even going to scrap NetWare in favor of a brand spanking new Windows NT Remote Desktop-type setup, but we were highly disappointed when NetWare came up with a patch for NetWare 3.X series to make it Y2K compatible, so they scrapped the NT plans. But I digress...

Came in to work on 1/1/2000 a couple hours after midnight (yep they pretty much forced us to come in, and for very little extra pay - I may have been a bit drunk still.) Everything was already chaos: Almost every single store's NetWare server shut itself down at midnight, thinking there was a power outage. And since our stores' computers ran as dumb network-booted terminals to the main server, that means all the computers were down and rentals couldn't be performed except by writing the rentals on paper.

Problem was, in the test lab someone had commented out the UPS backup auto-shutdown software line in the servers' autoexec (or its NetWare equivalent, might have been autoexec.ncf or something.) And yes, I do know who that someone was (wasn't me.) :) So I guess no one thought to test that particular software. So all the servers would boot up, immediately think there was a power outage, and immediately shut themselves off. We did have a manager's station computer in each store that had its own hard drive and could be used in emergencies, and had pcAnywhere and a modem, so we manually dialed into each of our approximately 2000 stores (at 14.4 kbps.) Then we walked a bunch of clueless managers and minimum wage kids through taking the new autoexec we had copied to a floppy on their manager's station (and a bunch of the stores had to run out and buy a box of floppies on New Year's Day) and booting up their servers using the floppy.

I think we got the last few stores up and working by 2 or 3 pm Pacific. And before you say "who rents movies on New Year's Day?" - EVERYONE did. New Year's Day and Christmas Day were two of our biggest movie rental days of the year. People are home with their families, the festivities are over, everyone wants something to do and streaming from the internet didn't really exist yet. What did everyone do? Rent a video or go to a theater. I'm not sure how many tens of thousands of dollars in rentals we lost that day, but I'm sure it was significant.

TL;DR: Just because you didn't hear about any significant losses due to Y2K bugs, doesn't mean they didn't happen. It's not like businesses were eager to admit they screwed up and forgot to test something.