Become a fan of Slashdot on Facebook

 



Forgot your password?
typodupeerror
×
Bug Software News

NTP Glitch Reverts Clocks Back To 2000 179

An anonymous reader writes "It seems a glitch of some sort wreaked havoc on some NTP servers yesterday, causing many machines to revert to the year 2000. It seems the Y2K bug that never happened is finally catching up with us in 2012."
This discussion has been archived. No new comments can be posted.

NTP Glitch Reverts Clocks Back To 2000

Comments Filter:
  • by Anonymous Coward on Tuesday November 20, 2012 @04:09PM (#42046605)

    Oh sorry. My clock's off.

  • Not an NTP glitch (Score:5, Informative)

    by Anonymous Coward on Tuesday November 20, 2012 @04:13PM (#42046657)
    It was a problem with the USNO servers (I.e. tick.usno.navy.mil, tock.usno.navy.mil etc.) being rebooted and starting to hand out the wrong time. Very few downstream startum 2 NTP servers should have accepted such a large skew, although they may have lost accuracy.

    Amusingly I happen to work with an ex. USNO NTP admin, so I'll be sure to take the piss for the rest of the week.
    • Re:Not an NTP glitch (Score:4, Interesting)

      by Guspaz ( 556486 ) on Tuesday November 20, 2012 @04:18PM (#42046727)

      And end-user systems certainly don't accept that large a skew. ntpd on end-user systems would just have been unable to sync their time while the servers were affected.

      • end-user UNIX systems at least. TFA talks about Windows AD servers though, maybe they aren't doing the basic sanity checks on the upstream NTP data coming in?

        • Re:Not an NTP glitch (Score:4, Informative)

          by Anonymous Coward on Tuesday November 20, 2012 @05:02PM (#42047407)

          Yes and no.
          This article is interesting: http://support.microsoft.com/kb/884776
          Summary: Windows can do it, but before Server 2008 it defaulted to not doing any sanity checks.
          Since 2008 it still is quite generous, allowing 48 hour jumps.
          If you don't like it you have to adjust the value in the registry.
          I guess it still shows that the Internet was an afterthought for Microsoft...

          • by tlhIngan ( 30335 )

            Windows implements basically Daytime using NTP. What this means is instead of NTPd trying to adjust the clock to be in sync in small increments, Windows does what you do with Daytime instead - query the Daytime server, and set the clock. Except that since very few servers provide Daytime, Microsoft uses NTP to fetch the time and date.

            Even the latest Windows still does it - though if it drifts too far, it stops syncing. Very annoying as it doesn't stay synced and quietly fails.

        • by egamma ( 572162 )
          Most windows systems sync to time.microsoft.com.
      • by jamesh ( 87723 )

        And end-user systems certainly don't accept that large a skew. ntpd on end-user systems would just have been unable to sync their time while the servers were affected.

        Had your home router sync'd with such a server and been rebooted, it would have booted with the wrong time, and then wouldn't have accepted the corrected time when the error was fixed...

        • Had your home router sync'd with such a server and been rebooted, it would have booted with the wrong time, and then wouldn't have accepted the corrected time when the error was fixed...

          Why would it have accepted the big skew the first time, but not the second?

          • by jamesh ( 87723 )

            Had your home router sync'd with such a server and been rebooted, it would have booted with the wrong time, and then wouldn't have accepted the corrected time when the error was fixed...

            Why would it have accepted the big skew the first time, but not the second?

            A home router would typically not have a battery backed clock, and so on boot would simply accept whatever time it was given by the ntp server. Once it's up and running though it shouldn't accept a 12 year skew. My home router (openwrt) boots to something like 1/1/1970 I think.

            • by AmiMoJo ( 196126 ) *

              As it happens I recently implemented the RTC in an embedded product and used a simple trick to reject obviously wrong times. I compare the date with the firmware build date and reject it if it is earlier. In the case of a router it could easily try other NTP servers in that case.

              That simple test rejects 99% of incorrect dates where something has been reset and started from zero again (which in this case would seem to be 01/01/2000).

      • Re: (Score:2, Interesting)

        by Anonymous Coward

        Well... My "out of the box" (ie. no special tweaking) Linux on my laptop did accept it...
        I had to remove tick & tock.usno.navy.mil from my NTP config.

    • Re: (Score:2, Informative)

      by Anonymous Coward

      It seems that VMware ESXi servers grabbed the configuration with little issue.

  • I don't know like a check that says if you are going to change the year (which is rare) that you first verify with three more more devices and seek a consensus?
    • Re: (Score:3, Interesting)

      by Anonymous Coward

      NTP -- at least, the formal Unix client -- already has checks in place to reject huge clock skews like this.

      The failure, of course, was dipshit Windows clients talking directly to low-strata servers. And then blindly accepting such a massive skew. Good jorb there, Microsoft, as always.

      • by Richy_T ( 111409 )

        Linux fan here. But my Windows 7 boxes will refuse to NTP sync unless the skew is 1 day. It's a bit of a PITA sometimes to be honest.

      • How exactly would that happen? Wouldn't it be the administrator's fault? I set all my domain controllers to talk to us.pool.ntp.org
        • Honestly not sure... it did hit us on a couple of old boxes, but fixed itself pretty quickly, and we definitely weren't using the affected navy servers. tick and tock are supposed to be used only by gov, and stratum 2 servers. Maybe some servers forwarded the wrong information?

    • by DamonHD ( 794830 ) <d@hd.org> on Tuesday November 20, 2012 @04:22PM (#42046773) Homepage

      Generally xntpd and its ilk will not step the time more than a small amount, but will rather give up and quit instead. One machine such as (say) the RPi with no RTC that use ntpdate to *set* rather than *adjust* the clock, this is harder to avoid, but servers should not be automatically running ntpdate when configured properly.

      So there may be *two* bugs here to get the problem: one in an NTP implementation and one in use of ntpdate for anything other than manual updates on servers. But I haven't read TFA yet.

      (On the little bit of work I put into the ARCRON driver I inserted extra code to look out for year rolls, etc, on top of whatever xntpd would do. In part because ARCRON only sends 2-year dates IIRC.)

      Rgds

      Damon

  • by Anonymous Coward on Tuesday November 20, 2012 @04:14PM (#42046667)

    Why do people keep pretending that it wasn't? It was a real issue, that required real work to fix. If none of that work had happened, it would've hit and it would've hit hard. Celebrate success on occasion, sheesh.

    • by asdf7890 ( 1518587 ) on Tuesday November 20, 2012 @04:22PM (#42046775)
      Unfortunately most of the general public think that because nothing really went wrong there was not a problem in the first place, and that it was all hyped up by the media. Some of this is the simple truth that it was over-hyped by the media who over-hype everything so people are growing desensitised, some of it is people not bothering to research their opinions or properly engage their critical thinking abilities.
    • by mellon ( 7048 ) on Tuesday November 20, 2012 @04:27PM (#42046825) Homepage

      We have a built-in bias against successful disaster planning: since the planning was successful, the disaster didn't happen, and hence to the average observer, it looks like there wouldn't have been a disaster. The reasoning is flawed, of course, but apparently very hard to resist. This is why governments are only ever harmful—if they do any good, things would have gone well anyway, so they didn't need to spend all that money and go to all that effort. This cognitive flaw is why people who do diving catches are respected, and people who plan for the future and avoid problems are ignored, and why blithering idiots keep getting control of the reins and breaking things.

      • This is why governments are only ever harmful&#226;&#8364;"if they do any good, things would have gone well anyway, so they didn't need to spend all that money and go to all that effort

        That's funny, because I always see it in reverse with the US government: if we hadn't had the stimulus, unemployment would have been much worse, or if we didn't have federal loans, no one could afford college, or if we hadn't passed the Patriot Act, we would have had lots of terrorist attacks...

        Ignoring, of course

        • by mellon ( 7048 )

          Yeah, it cuts both ways. The fact that nothing happened doesn't mean that the preventative worked. But the fact that nothing happened also doesn't mean that the preventative wasn't needed. It's important to understand why the reasoning is flawed, and not just that it is flawed, or else you wind up falling into different flawed reasoning.

          E.g., the reason people believe the stimulus worked is not that things might have been worse without it, but rather that things _are_ demonstrably worse in places wher

      • by mcrbids ( 148650 )

        To be fair, it makes a bit of sense. I know, it sounds dumb, but the horrible truth is that being prepared can be rather expensive and the cost of preparing for every possibility is utterly infeasible.

        To be evolutionarily successful, you need to be prepared enough to survive long enough to reproduce, and perhaps to see that your offspring reproduce. Being prepared to live 100 years when you won't breed much past 35 means that there's up to 65 years of time where your cost/benefit to the gene pool drops to n

        • by rk ( 6314 )

          Just because you're not punching kids out THIS VERY MINUTE doesn't mean your cost/benefit to the gene pool drops to near zero. Humans in modern societies take 2 decades or more to become self-sustaining, to say nothing of the aid (financial, emotional, or educational) provided by grandparents and then the very act of going to work in the morning, living somewhere and paying property taxes for schools (ideally) educating kids still adds benefit, regardless of whether you have kids of your own or not. And f

    • Re: (Score:2, Insightful)

      by c0lo ( 1497653 )

      Why do people keep pretending that it wasn't? It was a real issue, that required real work to fix

      "the Y2K bug that never happened" != "the Y2K bug that never existed".

      So, what's your point again?

    • by Cinder6 ( 894572 )

      Probably because of sensationalist news outlets that claimed all sorts of things would go wrong, such as cars not starting, missiles being launched spontaneously, etc.

    • by ShanghaiBill ( 739463 ) on Tuesday November 20, 2012 @04:34PM (#42046957)

      Why do people keep pretending that it wasn't? It was a real issue, that required real work to fix. If none of that work had happened, it would've hit and it would've hit hard.

      Lots of organizations worked hard to prepare for Y2K. Lots of other organizations did absolutely nothing to prepare. Neither had any significant problems on 1/1/2000. The reason is there were very few problems to begin with. The myth was that back when memory was precious, programmers stored the year in two bytes instead of four. But those of us that actually programmed in the days when 4K was a LOT of RAM, know that we never used two ASCII chars to store a year. We used a single byte to store the offset from 1900 in binary. So there will be no overflow until 2156.

      • Re: (Score:2, Interesting)

        by Anonymous Coward

        Lots of organizations worked hard to prepare for Y2K. Lots of other organizations did absolutely nothing to prepare. Neither had any significant problems on 1/1/2000. The reason is there were very few problems to begin with. The myth was that back when memory was precious, programmers stored the year in two bytes instead of four.

        While I'm sure plenty of organizations did nothing and were fine, I know for a fact that storing years as two bytes was not a myth. At my organization I fixed such a problem in critical systems. We would have had people collecting GPS data in the field with no way to verify it's quality if I hadn't. Not doom for our organization, but not a "myth" as you call it.

        • by 0123456 ( 636235 )

          I lost several days of simulation work that had been running over Christmas. Again, not a big deal because I restarted it when I got back in January, but still a real Y2K bug that we hadn't found or fixed.

      • by BobNET ( 119675 ) on Tuesday November 20, 2012 @04:57PM (#42047329)

        We used a single byte to store the offset from 1900 in binary.

        Most TRS-80 operating systems figured 3 bits was enough [trs-80.org] to store an offset from 1980, so we've already lived through the Y1.988k bug.

        • by girlinatrainingbra ( 2738457 ) on Tuesday November 20, 2012 @06:20PM (#42048513)
          Actually, most TRS-80 operating systems didn't even keep TRACK of time. It's the Disk Operating System (LDOS) that you're talking about. The TRS-80 didn't even have a built in clock / timer board / dedicated bits or bytes for knowing the time. And Level I and Level II Basic on it don't ask about the time or save it on the cassette tape system. And neither did the Apple ][+. You had to buy add-in clock boards to keep track of the time.
          Even the IBM PC did not have a real-time clock [wikipedia.org] until the IBM-AT version [wikipedia.org] in 1984. [Thank you wikipedia for prior info]. If you boot up an old IBM machine, one of the first things that the system asks after boot-up is to enter the time and date. [actual info from turning the old computers on in the garage].
          • by Mal-2 ( 675116 )

            The PC and PC/XT were very commonly retrofitted with an AST Six Pack [artofhacking.com] or generic equivalent. It was expensive, but still cheaper than buying each function on a separate card (and using six slots, which the PC didn't even have, it had just five). Even then it required running a program at startup to read from the RTC and load it into the motherboard clock. Setting the time on the Six Pack also took an executable, as simply setting the time the normal way didn't write it back to the board.

      • It was a problem in COBOL and some databases. If you had no COBOL you would usually be fine. Many banks however have very old code, and they needed to have this problem fixed in 2000, and they got it fixed. Checking for Y2K problems was checking for COBOL code and particular patterns of programming and database-design where 2 chars were used for years.

        • It was a problem in many places. Y2K bugs did crop up. Some cropped up as early as 1996, a lot more showed up in '98 and '99. By then most of the major stuff had been figured out.

          COBOL is only significant here because it is the most common language used in some financial operations. However the problem existed in C, assembler, Fortran, Java, databases, etc. Yes, I hear someone say "but my favorite language has a Date type!", which really only matters if the original programmer used the date type correc

      • I know of at least one organization which had a significant Y2K problem, even after making preparations.

        Sadly, the preparations were "Hire someone to take the fall when the shit hits the fan so we can continue with business as usual. Er... hire someone to ensure Y2K preparedness."

        The fatal glitch in the plan was that the person who got hired made friends with an exec in the parent company before the ball dropped. So, when things went south the hire got a silver parachute while the rest of the company folded

      • by brentrad ( 1013501 ) on Tuesday November 20, 2012 @06:17PM (#42048485)
        I beg to differ about not experiencing significant problems on 1/1/2000. We had significant issues that caused all our approximately 2000 store servers to repeatedly shut down until we unloaded the offending software.

        I was working for Hollywood Video in the Tech Support department (support for the rental stores and all their computer equipment) leading up to and after Y2K, in Wilsonville, Oregon at the corporate office. (Of course this was before their two bankruptcies.) The software development department performed extensive (and probably expensive) testing on every facet of our current in-store software and hardware setup (custom COBOL software running on DOS 5.0 on NetWare 3.1 if you can believe it.) They were even going to scrap NetWare in favor of a brand spanking new Windows NT Remote Desktop-type setup, but we were highly disappointed when NetWare came up with a patch for NetWare 3.X series to make it Y2K compatible, so they scrapped the NT plans. But I digress...

        Came in to work on 1/1/2000 a couple hours after midnight (yep they pretty much forced us to come in, and for very little extra pay - I may have been a bit drunk still.) Everything was already chaos: Almost every single store's NetWare server shut itself down at midnight, thinking there was a power outage. And since our stores' computers ran as dumb network-booted terminals to the main server, that means all the computers were down and rentals couldn't be performed except by writing the rentals on paper.

        Problem was, in the test lab someone had commented out the UPS backup auto-shutdown software line in the servers' autoexec (or its NetWare equivalent, might have been autoexec.ncf or something.) And yes, I do know who that someone was (wasn't me.) :) So I guess no one thought to test that particular software. So all the servers would boot up, immediately think there was a power outage, and immediately shut themselves off. We did have a manager's station computer in each store that had its own hard drive and could be used in emergencies, and had pcAnywhere and a modem, so we manually dialed into each of our approximately 2000 stores (at 14.4 kbps.) Then we walked a bunch of clueless managers and minimum wage kids through taking the new autoexec we had copied to a floppy on their manager's station (and a bunch of the stores had to run out and buy a box of floppies on New Year's Day) and booting up their servers using the floppy.

        I think we got the last few stores up and working by 2 or 3 pm Pacific. And before you say "who rents movies on New Year's Day?" - EVERYONE did. New Year's Day and Christmas Day were two of our biggest movie rental days of the year. People are home with their families, the festivities are over, everyone wants something to do and streaming from the internet didn't really exist yet. What did everyone do? Rent a video or go to a theater. I'm not sure how many tens of thousands of dollars in rentals we lost that day, but I'm sure it was significant.

        TL;DR: Just because you didn't hear about any significant losses due to Y2K bugs, doesn't mean they didn't happen. It's not like businesses were eager to admit they screwed up and forgot to test something.
      • So there will be no overflow until 2156.

        Or 19256, as some of those systems would display that year as.

      • We used a single byte to store the offset from 1900 in binary. So there will be no overflow until 2156.

        The vast, vast majority of software that ran infrastructure was COBOL, and it almost always uses PICture statements, not single bytes, to define dates. Those PIC statements most commonly defined dates using 2-digit years. Stupid and short sighted, yes. But also a very real issue that took gajillions of man hours to fix.

        Y2K disasters would have been very real (not as real as the Press would have had us believe, but very painful) if all that code hadn't been hacked. Note that the problem wasn't actually f

      • I worked on a system processing medical records that used a number of distinct methods of storing dates

        * British 2-digit years : dd-MM-yy
        * British 4-digit years : dd-MM-yyyy
        * American 2-digit years : MM-dd-yy
        * ISO 8601 style : yyyy-MM-dd (this is the correct form for all cases where you have to store the date as text)
        * An integer offset in minutes from a custom epoch value

        The rule I tried to adopt was to always store the date in an unambiguous format, and to present it to the user in whatever format was ap

      • The fact is, when we put all our data on 80-column punch cards, we'd use only two columns for the year. It would have been possible to encode it in one EBCDIC column as you say, but the cards were keypunched by women* who wouldn't generally know how to do that, and sometimes spot-verified by office staff who could read a two-digit number much easier than a format putting 256 values into one column. (Not to mention that, if you did too many fancy things, you could conceivably wind up with a lace card that

    • and the 2038 bug as well as the Year 2042 bug are still to come.

      Also maybe a Day 32,768 and 65,536 bug as well.

      • Crap. I never even thought of that.

        Hah! I'll be eligible for retirement just before then! I don't care anymore.

    • I never had my own internal Y2K bug fixed. I keep writing "19112" on paperwork.

    • Why do people keep pretending that it wasn't? It was a real issue, that required real work to fix. If none of that work had happened, it would've hit and it would've hit hard. Celebrate success on occasion, sheesh.

      Because if you accept that the problem was real and the only reason life doesn't suck as badly as predicted is because the fix was real means you are going to have a LOT more trouble sticking your fingers in your ears and chanting LALALALA I CAN'T HEAR YOU whenever people discuss Global Climate Change.

  • The Mayan's are up to something sinister what did they know that we don't?
    • by mcgrew ( 92797 ) *

      The Mayan's are up to something sinister

      The Mayan's WHAT are up to something similar? Their goats? Their sheep? Their calendars? Don't keep us in suspense, man!

      • by Cinder6 ( 894572 )

        The Mayan's are. It's a lot like an alot [blogspot.com].

        • by mcgrew ( 92797 ) *

          It's a lot like an alot

          Indeed it is, it's simply ignorance and aliteracy. "Alot" for a lot is more like "loose" instead of "lose" in that it can completely change the meaning of what the writer was trying to convey.

          It shows one's ignorance and lack of reading and lack of education. When the uneducated aliterate comes to a site for nerds, they should be made fun of.

  • RTFA (Score:2, Funny)

    by Anonymous Coward

    For those of you who didn't read the article; US Naval Observatory rebooted one of their NTP servers (tock.usno.navy.mil I believe, though not sure), and the server's time reverted to 2000.. This time change got pushed out, and thus people freaked out claiming that NTP was broken.

    A lesson can be learned from this: change your BIOS battery when it's dead (alternatively, never reboot your servers).

    • alternatively, never reboot your servers

      Or, better. Reboot your servers one at a time, a couple of times a year. That way, when a problem happens (may it be the RTC battery failing, bad init scripts or watever else) you'll catch it in a much more safe way.

  • The Y2K bug (Score:5, Insightful)

    by geekoid ( 135745 ) <{moc.oohay} {ta} {dnaltropnidad}> on Tuesday November 20, 2012 @04:23PM (#42046779) Homepage Journal

    did happen. The Y2K disaster did not thanks to a lot of money and a lot of people working to fix the bug.

    • by PPH ( 736903 ) on Tuesday November 20, 2012 @04:27PM (#42046831)

      Trust the computer industry to shorten the term "Year 2000" to Y2K.

      It was this kind of thinking that got them in trouble in the first place.

      • by Cinder6 ( 894572 )

        I recently saw "2k12" written somewhere. *sigh*

      • by cdrudge ( 68377 )

        Trust the computer industry to shorten the term "Year 2000" to Y2K.

        It was this kind of thinking that got them in trouble in the first place.

        What's wrong with Y2K? 2K = 2000. The problem with Y2K was that they shortened 2000 to 00, dropping the most significant digits. Not just substituting 3 digits (000) with a letter (K).

      • by MobyDisk ( 75490 )

        Wait, isn't Y2K going to happen in 2048?

      • by Rich0 ( 548339 )

        Believe it or not I still see applications being developed with reliance on two-digit years.

        People say that Y2K was the result of computers not having enough memory to store 4 digits. That might have been true once upon a time for a few systems, but for the most part it is due to laziness. Ugh, can't you shorten that field? And so on...

    • by c0lo ( 1497653 )

      did happen. The Y2K disaster did not thanks to a lot of money and a lot of people working to fix the bug.

      Strange terminology you have here.
      Yes, the Y2K bugs was real and yes, a lot of people worked to fix it. Making the bug to... hold on... not happen (because by that time it was removed/fixed).

      • Just as a single word can have multiple meanings, a single phrase can have multiple meanings as well. The Y2K bug that a lot of people (including me!) helped fix, yes, that was real. However, the Y2K bug that some nutcases hyped, where everything that had a clock chip in it would go haywire at 00:00:01 Jan 1, 2000, did not exist.

        It's like person A saying "dragons don't exist", and person B replying, "Yes, they do! The komodo dragon is a real animal!" The komodo dragon is a dragon, but it's not the kin

    • did happen. The Y2K disaster did not thanks to a lot of money and a lot of people working to fix the bug.

      Yet lots of other people did absolutely nothing, and the disaster didn't happen for them either. I worked on Y2K. When we were testing, we tried booting early versions of MSDOS, Windows, OS/2, Linux, FreeBSD, MacOS. The older the better. None of these had a problem when the date transitioned. Or at least (in the case of Windows 3.0), nothing that a reboot wouldn't fix. The situation was similar with applications. We found a few applications that needed to be restarted after the date transitioned, but

      • by jamesh ( 87723 )

        did happen. The Y2K disaster did not thanks to a lot of money and a lot of people working to fix the bug.

        Yet lots of other people did absolutely nothing, and the disaster didn't happen for them either. I worked on Y2K. When we were testing, we tried booting early versions of MSDOS, Windows, OS/2, Linux, FreeBSD, MacOS. The older the better. None of these had a problem when the date transitioned. Or at least (in the case of Windows 3.0), nothing that a reboot wouldn't fix. The situation was similar with applications. We found a few applications that needed to be restarted after the date transitioned, but there were no show stoppers.

        Y2K was hyped up by the press and by software vendors peddling "critical" upgrades.

        the realtime clock in your PC bios in that era stored the date in BCD form (4 bits per digit) and often either had a problem at transition, or just didn't work. Easily fixable with a boot-time patch or something, but still a problem that needed to be addressed.

        The satellite images at www.bom.vic.gov.au showed the year as 19100 for a while. Again, not a show stopper but still an indication that someone didn't fix a problem that should have been fixed.

  • by jaredmauch ( 633928 ) <jared@puck.nether.net> on Tuesday November 20, 2012 @04:28PM (#42046843) Homepage

    If you saw this problem, your NTP time sources were not properly configured and diverse.

    Consider using the NTP pool and not relying on so few sources to properly sync your time. Read 5.3.3 and 5.3.4 from http://support.ntp.org/bin/view/Support/SelectingOffsiteNTPServers [ntp.org] for help to correct your NTP setup.

  • by Ol Biscuitbarrel ( 1859702 ) on Tuesday November 20, 2012 @04:28PM (#42046849)

    Poor Elián González! At least I was able to get ILOVEYOU off of my Gateway. Have you checked out Dora the Explorer?

  • pool.ntp.org is our one and only external resource that we need .

  • Last night AD went to year 2000. Linux clients joined to AD freaked out right away with scary security warnings. A reboot fixed them right away after AD time was set right. Or setting time manually.

    Windows had authentication trouble so Shares, Printers, etc stop working for clients. For some reason I had to rebooting them many times fixed them, no idea why one wasn't enough. Had trouble reported all today even though it was only off for 2 hours last night.

    Hard to believe Windows doesn't do sanity checking o

    • Hard to believe Windows doesn't do sanity checking on this stuff.

      Are you new here? Microsoft has a long history of not sanity checking, damn near anything until long after it breaks.

  • by organgtool ( 966989 ) on Tuesday November 20, 2012 @05:07PM (#42047485)
    Why didn't you warn us about this time-based bug, John Titor?!
  • by MichaelSmith ( 789609 ) on Tuesday November 20, 2012 @05:14PM (#42047607) Homepage Journal

    Our NTP sync came from a remote site and they were doing Y2K testing. Clocks jumped forward to March 2000 and screwed up the timestamps on our binaries. In the end we had to touch backwards to get our build system working again.

  • by Crypto Gnome ( 651401 ) on Tuesday November 20, 2012 @06:29PM (#42048677) Homepage Journal
    Was not the bug. Nor was it The Fix(ing/es/etc).

    It was THE MONEY.

    A large part of "LALALALA I CAN'T HEAR YOU" came from arguments which essentially boil down to "but if that's actually true, then fixing it would cost LOTS OF MONEY".

    Whoever said HISTORY NEVER REPEATS is a complete moron.

    These days s/Y2K bug/Global Warming/ and the situation is EXACTLY THE SAME.

    Lots of industries pushing for FLAT OUT DENIAL mainly because of attitudes which amount to not much more than "but if that's actually true, then fixing it would cost LOTS OF MONEY".

    Insert here well documented evidence that THE TOBACCO INDUSTRY knew, many many may ears ago that smoking tobacco was (a) bad for you and (b) addictive.

    Big Business KNOWS perfectly well that we're pretty much up Fitz Creek without a paddle (or even a canoe) but THEY DON'T CARE as long as they make lots of money TODAY.

    Show me one CEO where their bonuses/etc are indexed against ANYTHING more than the next 1-5 years of profit.

    An environmental (and therefore economic) CATASTROPHE that will occur no sooner than 10 years from now is akin to the heat-death-of-the-universe, we ALL know it's coming - but NOBODY CARES.
  • I distinctly remember on multiple occassions feeling irked by what I assumed were tinfoil hat wearing fools who required you to get the time kind of sort of right first before clocks would sync up from an external time source...especially within windows.

    I could swear windows and the standard unix ntp systems explicitly check for and do not allow such shenanigans...yet this happened and as I hear from a few sources real systems were effected... Maybe there is some nuance with stratums or something which come

    • by Lehk228 ( 705449 )
      other people who, like yourself, were irked about having to get the time close before clocks would synch up, overriding the sanity check
  • My phone, for no apparent reason, reset the date to (forgot the day) of March 2011 on Sunday. It's never done anything like that before.

  • At my last employer, they had the brilliant idea to use 111111 as infinity for time in their database. When 2011-11-11 happened, it was very ugly. Some of the software had been fixed AFTER to use 2911-11-11 as the new infinity date, but not all of it. I was still fixing software when I left in October.

    My suggestion to make the field NULL didn't go over well. They couldn't figure out how to represent NULL in their C programs that accessed Ingres. :)

  • by jthill ( 303417 ) on Wednesday November 21, 2012 @12:05AM (#42051467)

    A large part of NTP's sophistication lies in its ability to identify falsetickers/truechimers among multiple candidate sources. NTP is built to survive failures like this, but Microsoft apparently doesn't bother telling its Active Directory customers [microsoft.com] -- it actually instructs them to single-source their clocks.

  • If you really care about accuracy in your data center, use a GPS or CDMA source for time sync. If you're paranoid, get an atomic clock as well for comparison.

If all else fails, lower your standards.

Working...