Catch up on stories from the past week (and beyond) at the Slashdot story archive

 



Forgot your password?
typodupeerror
×
United States Network Security

US Homeland Security Employees Locked Out of Computer Networks (reuters.com) 133

Dustin Volz, reporting for Reuters: Some U.S. Department of Homeland Security employees in the Washington area and Philadelphia were unable to access some agency computer networks on Tuesday, according to three sources familiar with the matter. It was not clear how widespread the issue was or how significantly it affected daily functions at DHS, a large government agency whose responsibilities include immigration services, border security and cyber defense. In a statement, a DHS official confirmed a network outage that temporarily affected four U.S. Citizenship and Immigration Services (USCIS) facilities in the Washington area due to an "expired DHS certificate." Reuters first reported the incident earlier Tuesday, which a source familiar with the matter said also affected a USCIS facility in Philadelphia. Employees began experiencing problems logging into networks Tuesday morning due to a problem related to domain controllers, or servers that process authentication requests, which could not validate personal identity verification (PIV) cards used by federal workers and contractors to access certain information systems, according to the source.
This discussion has been archived. No new comments can be posted.

US Homeland Security Employees Locked Out of Computer Networks

Comments Filter:
  • Security focused (Score:5, Insightful)

    by Fire_Wraith ( 1460385 ) on Wednesday February 22, 2017 @12:06PM (#53911569)
    DHS is the primary government agency responsible for protecting the country's civilian infrastructure, including the internet and computer networks. I feel so much better knowing that they're so good at keeping their own systems secure, that even their own workers can't access them.
    • by Anonymous Coward on Wednesday February 22, 2017 @12:26PM (#53911691)

      What's so insecure about denying access due to an expired certificate? Isn't that an example of security measures working as expected?

      • Just because the security measures are EXPECTED to work that way, doesn't mean that they're good.
        • Exactly. Certificate expiry is a CA billing mechanism to make sure you pay your dues every year. Claiming that a certificate that's fully secure at 11:59:59 is totally insecure at 12:00:01 just because the clock ticked is nonsense.
      • by Anonymous Coward

        Availability is a third of the information security triad (Confidentiality, Integrity, Availability). They should have had a plan in place and an alert being sent to a person or three to ensure the cert gets renewed or replaced.

        • Replying to fix a fat-fingered mod.

          Much like the most secure computer ever made: no drives, no network, no HMI, and no power supply.
        • On the systems I administer, we have an alert that checks the certificate expiration once a day, and alerts it plenty of time to get it renewed.

          But a lot of people don't do that, they just mark it on a calendar somewhere, or expect the certificate issuer to notify them. For the latter, often the contact email is to a person no longer with the organization, or in a different role, so it is ignored. That's why my current $Employer insists that certificate emails go to an email list for a group, rather than ju

          • by ebvwfbw ( 864834 )

            How can you? Sure, if you have just one machine or two it's no big deal. Suppose you have a modern government agency, a business of any real size, etc? You have the web site - no big deal, they just get a warning message. Then there are the Unix based systems that run ldap, san, well most everything. Blade centers for VMs and such. Then the lightweight stuff that feed the dumb people like the Windows domain controllers and such. Things that people don't use much. It's getting to be a real PIA to find all o

      • Isn't that an example of security measures working as expected?

        Security measures working properly is only good in the proper context.

        Car analogy: Your alarm goes off and locks both your steering and brakes. This is good for anti-theft. This is bad if it happens as you're cruising down the freeway.

    • by sycodon ( 149926 )

      I work for one of the largest Defense companies in the nation. In the last year we have had two major network outages. One related to provider issues and the other related to firewall changes gone bad.

      This shit happens. Creating/Managing/Upgrading huge networks like this a very complicated and delicate task.

      • I work for one of the largest Defense companies in the nation. In the last year we have had two major network outages. One related to provider issues and the other related to firewall changes gone bad.

        This shit happens. Creating/Managing/Upgrading huge networks like this a very complicated and delicate task.

        Certificate management is not a complicated task. Expired certificates is an example of incompetence, not an example of "complicated shit that just happens". It should be somebody's job to manage those expiration dates, period.

        • by dbIII ( 701233 )
          It's an example of mismanagement and no proper procedures to keep the certificates up to date and an incredibly common failure. Even Microsoft had a very high profile failure of that kind not long ago.

          Incompetence is someone not doing their allotted tasks.
          Mismanagement is forgetting to allocate a required task to someone.
    • Sounds like the ultimate in security - pull the plug.
    • Re: (Score:2, Troll)

      by hambone142 ( 2551854 )

      We're talking about government "workers". Perhaps they were upgrading their LGBTQRSTV skills or brushing up on break taking.

    • by dbIII ( 701233 )

      I feel so much better knowing that they're so good at keeping their own systems secure, that even their own workers can't access them

      Given some of the people working for them (eg. the TSA gropers) I actually would feel good about it, but not as good as if they slash that massive mall cop welfare program and have some sort of professional airport security instead. While they are at it they can get rid of the DHS guys who go around to toy shops and check if rubicks cubes are legit instead of knockoffs (now t

  • by lionchild ( 581331 ) on Wednesday February 22, 2017 @12:15PM (#53911619) Journal

    I think I'd like to take this opportunity to point out that this is what happens as we do more and more with IT on less and less staff. While I understand sometimes we think of IT as a cost-center and not a revenue generator, it probably needs to be thought of as more like a utility; because without the lights, water, phones...and internet, you can't do business very effectively these days.

    That being said, this happens more and more. Someone is responsible for renewing certificates, but as we renew them for longer and longer periods, that means we simply start to forget about them. Then with the certificate issuer sends out an notification to that IT staffer who used to do that, but was 'right sized' a year and a half ago...no one gets the email. So, the certificate expires and this happens. Same song, different, louder verse, apparently when it happens to DHS, and likely more embarrassing.

    Bottom line: Doing more with less, isn't always in everyone's best interest.

    • by TechyImmigrant ( 175943 ) on Wednesday February 22, 2017 @12:33PM (#53911731) Homepage Journal

      That being said, this happens more and more. Someone is responsible for renewing certificates, but as we renew them for longer and longer periods, that means we simply start to forget about them.

      An alternative viewpoint is that this is one of the ludicrously bad failings of PKI. Requiring someone to remember to do an infrequent and short task at a point 1 or 2 years in the future, or the whole system collapses when they forget or leave or get booted. We could fix (I.E. delete and replace) PKI and this specific failure would not happen, so the overworked IT staff can go back to deploying Windows NT patches.

      • by swb ( 14022 )

        I think you're basically right, PKI implementations are horribly complex in practice and doubly (or more!) so with Windows.

        It seems to get worse as certificate-based security gets added into products as defaults installations. As an example, Exchange 2016 installs a self-signed certificate by default which gets assigned to SMTP and IIS. The normal (spanning back several releases) process of adding and assigning a public certificate to services doesn't change the self-signed certificate assignment and use

      • by EndlessNameless ( 673105 ) on Wednesday February 22, 2017 @01:31PM (#53912081)

        Requiring someone to remember to do an infrequent and short task at a point 1 or 2 years in the future

        Bullshit.

        I could write a PowerShell script in maybe 10 minutes that will list all of the computers in the domain, connect to them, and check for expiring certificates. I can get a reminder in advance---90 days, 30 days, a week, whatever I want. All I have to do is one thing: understand my job.

        Alternatively, some tools (like Nessus, which is FOSS) have audits which automatically check for expiring certificates. They can be configured to email a report, and you can notified every day/week/month if you have expiring certs.

        This is a stupid, incompetent failure. You can build or buy a tool to avoid this problem very easily. Compared to using passwords, the only reasonable complaint is that you require decent sys admins.

        • by swb ( 14022 )

          Why aren't these tools built in, though?

          IMHO, PKI on Windows is problematic less because PKI is complex but more because the in-built tools suck or are non-existent.

          Most IT admins are oversubscribed enough that writing that Powershell script or putting together the third party tools for certificate expiration won't happen, especially when you consider for most organizations the number of certificates that matter is relatively small.

          I will grant an exception for Homeland Security, though, as any organization

          • Why aren't these tools built in, though?

            PowerShell is a very powerful tool, and it is built in. But that's not what you meant.

            There are two ways to get it from the vendor. You pay in cash or labor.

            Microsoft is happy to sell you SCOM, which is their network management dashboard (among other things). Very useful in a Windows-dominated environment, but there are better third-party options for shops with Linux and Mac systems.

            Unless you're talking about the lowest tier of admins, scripting is part of the job. I cannot understand how people function w

            • by swb ( 14022 )

              I like the idea, but I find it clashes with reality too often.

              Management wants everything for free, SCOM they won't pay for and scripting is seen variously as a kind of technological masturbation and time wasting or the creation of unmanageable spaghetti.

        • OK, so you have written such scripts to notify you. Now the company decides they do not need you any more. Are you going to rewrite those scripts to notify someone else? Or even bother to mention to someone that they should do so?

          Or the other scenario where you get another job midway between renewals (when you have not had a notification in several months). Will you remember to change who gets notified? Will you remember to tell someone? When you remember 3-6 months later that you would be getting notific
          • by dbIII ( 701233 )

            OK, so you have written such scripts to notify you. Now the company decides they do not need you any more. Are you going to rewrite those scripts to notify someone else?

            In the *nix world you send it to root, postmaster or whatever - a role not a person, so that the next person in the role gets the notifications. That's assuming a real mail server someone and not an enormous flaky suite that tells you to Exchange it for something more reliable.

          • OK, so you have written such scripts to notify you. Now the company decides they do not need you any more. Are you going to rewrite those scripts to notify someone else? Or even bother to mention to someone that they should do so?

            Shouldn't be a problem.

            1. The script and its purpose should be documented. Another admin should be able to update it as needed.

            2. The output can be emailed or dumped to a file share. Virtually every mail servers supports lists, so the list (or the file share ACL) would just need to be updated.

            In a lot of companies, certificate renewal becomes someone's job because they are in the right place at the right time to handle it and everyone else forgets that it even happens until something goes wrong.

            First, this "problem" does nothing to change the fact that 2FA is far more secure than passwords.

            Second, this is the result of poor management. Any process can become failure-prone in the face of poor management. You n

        • by Rastl ( 955935 )

          Requiring someone to remember to do an infrequent and short task at a point 1 or 2 years in the future

          Bullshit.

          I could write a PowerShell script in maybe 10 minutes that will list all of the computers in the domain, connect to them, and check for expiring certificates. I can get a reminder in advance---90 days, 30 days, a week, whatever I want. All I have to do is one thing: understand my job.

          Alternatively, some tools (like Nessus, which is FOSS) have audits which automatically check for expiring certificat

          • by lgw ( 121541 )

            Terrible management if that happens. No doubt that's the case here.

            Any big network has a dedicated monitoring system with all sorts of plug-ins. Certificate monitoring is just another plug-in. You (if competent) write the plug-in once, and the notification is just the normal for the whole system. You (if good) write a system to auto-renew all your certs based on these scans and notifications, and alarm if the auto-renew fails for long enough..

            We had a team that did that where I work. It was particularl

            • Terrible management if that happens. No doubt that's the case here.

              Any big network has a dedicated monitoring system with all sorts of plug-ins. Certificate monitoring is just another plug-in. You (if competent) write the plug-in once, and the notification is just the normal for the whole system. You (if good) write a system to auto-renew all your certs based on these scans and notifications, and alarm if the auto-renew fails for long enough..

              We had a team that did that where I work. It was particularly amusing when that team's certs all expired - they had chosen to leave themselves out of their own system, for some reason.

              I've written plugins like that.

              What gets bad is the alert goes off, and says you have 90 days to renew. Having no power to spend money, you dutifully route a request for a renewal to be paid for. It goes back and forth to accounting for a couple months asking for justifications for the (trivial) expense because no one will give the operations people a p-card or budget. Finally, if you are lucky, a P.O. is issued (for a trivial amount), and you can buy a new certificate before the old one expires. If not, it

              • Having no power to spend money, you dutifully route a request for a renewal to be paid for. It goes back and forth to accounting for a couple months asking for justifications for the (trivial) expense because no one will give the operations people a p-card or budget.

                While it's usually bad to exaggerate, you don't need to.

                The justification should mention that the entire corporate network will become unstable or unavailable if this procurement is not completed by the deadline, which should be at least a few days ahead of the actual expiration date.

                Ideally, the IT management hierarchy will understand and push it through. If not, they should at least be capable of understanding the necessity when their experts start barking about the importance of such a minor purchase.

                And

        • by dbIII ( 701233 )
          Yes the task is not difficult.
          However very frequently nobody has been assigned to do the task.

          I'm in a small place and can feel smug due to stuff like certwatch notifiying multiple people, but in large places with poor management tasks fall between the cracks. "I thought X was going to do it" is a frequent cry in large barely functional shambolic orgs where execs spend more time golfing than managing, hence the DHS getting hit with this.
      • Requiring someone to remember

        Requiring someone to remember to do something is not a bad failing of PKI.

        It's a bad failing of organisational systems that are supposed to catch this.
        It's a bad failing of automation systems which could remove the task.
        It's a bad failing of management systems that ensure the task is complete before it becomes an issue and flag it for appropriate response.

        • Nope it's a bad failing of PKI. Writing a spec that takes something computers do well and humans to badly and handing it over to the humans.

          Not everybody has an IT department. Do you think they should not benefit from communication security because they don't fit the PKI model well.

          • Writing a spec that takes something computers do well and humans to badly and handing it over to the humans.

            The spec has done no such thing.

            • Writing a spec that takes something computers do well and humans to badly and handing it over to the humans.

              The spec has done no such thing.

              Show me where in any X.509/PKI/Application auth related spec it solves the automated continuity problem.
                 

    • There are several issues with most systems that require certificates to work correctly.

      1) Certificates Expire, on a regular basis, have a plan to update them (Auto renew)
      2) Notifications should be sent to a "group" email address, not an individual.
      3) We have these things called "Calendars" use them
      4) Documentation is key, even if 1-3 fail you should have a searchable document that has they dates listed for key events.

      The problem is, nobody ever documents shit like this, because actual documentation process

      • Additionally, all of the above isn't overly helpful (except maybe the group email address), if you start outsourcing whole departments. Even if if you document things, the chances are there will be some things, like this, lost in translation.

        However, you're right, they're generally effective steps to mitigate this issue. Especially number 1, if your credit card info never expires. :-)

  • No big worry if it is merely an expired certificate. Merely incompetence. An ordinary thing that is to be expected.

    It would have been a bigger concern if, for security reasons, the president had ordered all passwords changed to the same code used on the president's luggage.
  • Another Trump IT nominee on his first day in the job?

  • That's how expired certificates are supposed to work!

  • by ErichTheRed ( 39327 ) on Wednesday February 22, 2017 @12:54PM (#53911851)

    The interesting part of the article isn't about who is affected, but the "certificate expiration" aspect. I've recently started doing the legwork necessary to learn about public key infrastructure (for our company's internal consumption) and have found that there are 3 prevalent camps out there:
    - Developers who just say "here's my credit card, VeriSign, make my customers' browser address bars turn green."
    - Admins who get just enough of a PKI background to make the certificate errors go away, then run away screaming -- or worse yet, had it implemented a decade ago by a consultant and have NO CLUE how it works or how to fix it
    - Auditors who just say "lock icon, green browser windows, check. Congrats, you're PCI compliant."

    For something so critical like certificates, there really is a dearth of resources out there that isn't aimed at hardcore security programmers or one of these three groups. Cert expirations have figured prominently in many outages -- Azure had a partial outage a few years ago because of that very reason. I'm seriously considering writing a "PKI for non-dummies" series of blog posts or something because the amount of misinformation out there is scary!

    • I'm seriously considering writing a "PKI for non-dummies" series of blog posts or something because the amount of misinformation out there is scary!

      Please do. I'm going to have to start learning about this pretty soon for a project I'm working on. I've avoided it up to this point by Googling and clicking boxes and trying and knowing JUST ENOUGH to scrape by and expand existing infrastructure...

    • There really is plenty of good documentation out there, and it isn't that hard to manage. The real problem is category 2 and its permutations-- oh, they expire!, what is a CRL!, or where is the offline root certificate stored!?

      The main issue I have is effectively planning, compartmentalization, and execution to ensure a multi-level PKI system is effective and maintainable. It stops me each time I go to set up PKI for our VPN or phone system or ...anything else. A poorly planned system can make things wo
  • Just call Sandeep in the IT department and have him fix .......

    Uh, oh.

  • Some apps need to have the certs installed into them even with LDAP stuff each app may need the LDAP keys installed to it's own key store for it to be able to ldap login's.

  • I would be very funny to check Trumps laptop to see something like "Your files have been encryped. Send 2 million bitcoins if you ever want to see them again" It's always the boss that does this.
  • Giuliani was just converting all the servers to a five-year-old version of Joomla.

  • by IWantMoreSpamPlease ( 571972 ) on Wednesday February 22, 2017 @03:05PM (#53912879) Homepage Journal

    Always install a backdoor.

    For times like this.... ...and for "other" times, as needed.

"When the going gets tough, the tough get empirical." -- Jon Carroll

Working...