Fedora Infrastructure Compromised 115
Trailrunner7 writes "The infrastructure of the Fedora Project was compromised over the weekend and an account belonging to a Fedora contributor was taken over by an attacker. However, Fedora officials said they don't believe that the attacker was able to push any changes to the Fedora package system or make any actual changes to the infrastructure. The attack appears to have targeted one specific user account, which had some high-value privileges. The attacker was able to compromise the account externally, and then had the ability to connect remotely to some Fedora systems. The attacker also changed the account's SSH key, Fedora officials said."
Believe? (Score:3)
However, Fedora officials said they don't believe that the attacker was able to push any changes to the Fedora package system or make any actual changes to the infrastructure.
What do you mean you "don't believe"? You don't have logs?
Re:Believe? (Score:5, Insightful)
Logs can be faked. How about a bitwise comparison to the known-good package system?
Re: (Score:2)
Logs can be faked. How about a bitwise comparison to the known-good package system?
Fake information can be even more useful, if you detect it's fake of course.
Re: (Score:3)
The critical piece as I see it is the distribution of the checksums. If package maintainers and end users agree on the checksums (and neither of their systems is initially compromised), then everything should be fine. Or am I overlooking something?
Re: (Score:2)
If you're using Nulfs2 for the filesystem the logs are on, then you can determine if data within the logs have been altered.
Alternatively, if the logging daemon writes events to a logger on another machine, then logs could only ever be appended to and never altered.
In this day and age, it seems pitiful that anyone would use a setup where the logs could be faked.
Re: (Score:3)
Alternatively, if the logging daemon writes events to a logger on another machine, then logs could only ever be appended to and never altered.
This is why there's still a market for dot matrix printers, especially those with a dip switch that disables reverse paper feed.
Good luck erasing or modifying that audit trail remotely.
Re: (Score:2)
I accidentally set fire to a dot matrix once - the paper somehow wrapped itself from the output back in, causing the motor to jam and ignite.
Re: (Score:2)
Did you get the printer on fire error?
http://en.wikipedia.org/wiki/Lp0_on_fire [wikipedia.org]
Re: (Score:2)
nicely done.
Re: (Score:1)
Re: (Score:1)
Re: (Score:1)
man ps [ed.ac.uk]
I_WANT_A_BROKEN_PS
Force obsolete command line interpretation.
Re: (Score:2)
There's no need to get superstitious about it. A dot matrix printer is expensive and requires maintenance and consumable. it may be particularly difficult to erase the logs, but you're no better off than hooking up a standalone PC via null modem cable, or even an ethernet cable with a resistor on the TX line so it registers as live , but can't send packets, not even a basic handshake... Make that your syslog server and you're set. Your eats won't bleed during peak load either...
Re: (Score:2)
Re: (Score:1)
Is there a 'known-good' Fedora system?
Re:Believe? (Score:5, Informative)
Logs can be faked. How about a bitwise comparison to the known-good package system?
As a fedora dev account holder, I got the notification email. The filesystem was compared with a previous 'good' snapshot to determine what changes were made.
Re: (Score:1)
Logs can be faked. How about a bitwise comparison to the known-good package system?
Sorry, I'm busy remodeling my house..
Re: (Score:1)
Re: (Score:3)
Re: (Score:2)
Re: (Score:3)
would you trust the logs if you had them?
Re: (Score:2)
Well, I would update my local repository, and then check the change logs to see what's different. So yeah, I would trust that.
Re: (Score:2)
That was my first thought as well: "Uh, didn't they do a diff and look for the parts that changed?"
Re:Believe? (Score:5, Informative)
That's why any secure system should be sending logs to a remote machine as well as /var/log.
Re: (Score:3)
Excluding logs before and in the exact time of break-in, while attacker hasn't put his stealth instruments to the victim machine yet.
Re: (Score:2)
The logs sent to the other machine wouldn't be trustworthy either...
Only after the attacker has rooted the box and been able to block further logging of his activities. You can easily edit the log files on a machine and remove old entries showing you doing bad things, but you can't do that wih a remote machine that you have no access to.
Developing story (Score:2)
Re: (Score:1)
Yep... Make the devs change all passwords, wipe the affected system, and re-install. Or, they can do that thing were you put important data on a non-volatile media and put it somewhere in case you lose a system...
Fedora infrastructure intrusion (Score:2)
"The Infrastructure Team took the following actions after being notified of the issue:
1. Lock down access to the compromised account
2. Take filesystem snapshots of all systems the account had access to
(pkgs.fedoraproject.org, fedorapeople.org)
3. Audit SSH, FAS, Git, and Koji logs from the time of compromise to the present
Here, we found that the attacker did:
* Change the account's SSH key in FAS
* Login to fedorapeople.org
Re:Believe? (Score:5, Funny)
However, Fedora officials said they don't believe that the attacker was able to push any changes to the Fedora package system or make any actual changes to the infrastructure.
What do you mean you "don't believe"? You don't have logs?
Thankfully, I am on Windows, so I don't have to wonder whether hackers are conducting malicious activity.
Re: (Score:1)
Re: (Score:1)
However, Fedora officials said they don't believe that the attacker was able to push any changes to the Fedora package system or make any actual changes to the infrastructure.
What do you mean you "don't believe"? You don't have logs?
Yeah No spin doctors, we're probably all administrators or more. What exactly has happened?
I need to hand in my geek card... (Score:1)
...because I thought this was about shoddy hats.
Re: (Score:2)
Re: (Score:2)
You encountered a red fedora?
CC.
Re: (Score:2)
I only recently discovered what the hell does "fedora" mean apart from a Linux distro.
Red Hat Fedora
Apple MacIntosh (McIntosh Apples)
Sun SOLARis
There are a lot of plays on words out there in the tech field.
Re: (Score:2)
Apple MacIntosh (McIntosh Apples)
Bonus points for noticing the difference in the spelling!
Re: (Score:2)
You can't have my Fancy Fedora [teamfortress.com]. :|
Re: (Score:2)
Yeah, turns out the mercury used was driving the hatters mad.
Re: (Score:2)
Then I remembered Bogart was dead.
Then I remembered Fedora is a Linux distro.
again? (Score:2)
Re: (Score:2)
I think last year it was CentOS that got hit, not Fedora. Also, the nature of the attack was different and I believe some packages were compromised, or at least the repo signing keys.
Re: (Score:2)
I think last year it was CentOS that got hit, not Fedora. Also, the nature of the attack was different and I believe some packages were compromised, or at least the repo signing keys.
It was Red Hat, wasn't it?
http://www.redhat.com/security/data/openssh-blacklist.html [redhat.com]
Re: (Score:2)
That little gem can be blamed on Debian [debian.org] actually.
I love Debian so this one came as a real blow; but they did a good job disclosing it and worked with any and all who wanted to participate to develop the blacklist package.
Not a professional job (Score:2)
The first action the intruder took, changing the SSH password, set off an automatic email notification, which is how the compromise was detected. Pretty stupid.
A pity that the clueless black hats eventually learn, tho. Not that this means that open-source is totally helpless. In the past, malevolent software updates have been caught. If this becomes widespread, it just means that the development is slowed by the necessity for peer review.
Re: (Score:2)
slowed by the necessity for peer review.
What credible software organization feels that peer review isn't necessary? Automated testing only gets you so far...
Re: (Score:1)
> it is not necessary as it is assumed that software package people will not
> be introducing security holes into software
And we've seen how one can be bitten by this assumption, badly (Debian SSH patch-of-entropy-death).
Re: (Score:1)
I'd guess that most open-source projects are one- or two-developer deals, max (actually, if you look at SourceForge, you'd end up saying that most projects are zero-developer deals!). However, the most-used projects are much better "staffed", which might mean that there is more of a chance that the people in charge of vetting the commits have some specialized training to catch malevolent changes (and also that more than one set of eyeballs might be looking at every commit).
In the end, it comes down to a mat
Article and headline are completely wrong (Score:5, Informative)
The infrastructure was not compromised. One user's password appears to have been compromised and changed. That account did not have "high value privileges".
Re: (Score:3)
Or digging further:
# Push access to packages in the Fedora SCM.
# Ability to perform builds and make updates to Fedora packages.
Which I would qualify as high-value.
Re: (Score:2)
As far as I know, everyone gets those rights, subject to ACLs on each package that govern who can make changes.
Re: (Score:2)
Too big for this case.
No, they're not. The infrastructure itself was not compromised. One account's password was, and that account did not have privileges to affect the infrastructure.
Re: (Score:3)
Being able to change source code and that code getting pushed into builds which the RE group releases would suggest that it's a bit of a high value account.
I argued for some time at a previous company that they were out of compliance with Sarbanes-Oxley and segregation of duty rules. The reason was the network admins had access to source code repositories (VSS, StarTeam, and TFS). Since network admins did pushes to QAS and PRD, they could feasably alter source code and push it to production.
The reality, t
Re: (Score:2)
This never would have happened if they were running Lin...
Most likely it's a weak SSH password: SSH and VNC with crappy passwords seem to be the two most common ways to get into Linux machines these days... just open port 22 and watch a million Chinese hackers testing out a bazillion ssh passwords on your machine.
If Windows actually supported SSH then it would be just as vulnerable.
Re:This never would have happened... (Score:4, Insightful)
P.S. Of course if they were serious about security in the first place they wouldn't even allow logins with passwords and would require public key authentication instead.
Re: (Score:2)
Exactly. For example, any machines I have which have to have an Internet facing ssh port are definitely not going to be accepting passwords. Tools like ssh-guard are nice, but it isn't hard for a determined attacker to just keep coming from different IP ranges. To add a little bit of security, port knocking is a nice ability to have, just so an attacker doesn't see an open port to start having fun with.
What would be ideal is if OATH support would advance to the point where I can just enter my username, t
Re: (Score:2)
Weakest Link (Score:2)
The web UI where one uploads SSH public keys, however, uses a password. It was that password that the attacker changed.
See also: weakest link.
Some history (Score:2)
Re: (Score:1)
Re: (Score:1)
The actual email in case anyone wants the facts (Score:5, Informative)
http://lists.fedoraproject.org/pipermail/devel-announce/2011-January/000746.html [fedoraproject.org]
Summary: Fedora infrastructure intrusion but no impact on product integrity
On January 22, 2011 a Fedora contributor received an email from the Fedora Accounts System indicating that his account details had been changed. He contacted the Fedora Infrastructure Team indicating that he had received the email, but had not made changes to his FAS account. The Infrastructure Team immediately began investigating, and confirmed that the account had indeed been compromised.
At this time, the Infrastructure Team has evidence that indicates the account credentials were compromised externally, and that the Fedora Infrastructure was not subject to any code vulnerability or exploit.
The account in question was not a member of any sysadmin or Release Engineering groups. The following is a complete list of privileges on the account:
The Infrastructure Team took the following actions after being notified of the issue:
The attacker did not:
Based on the results of our investigation so far, we do not believe that any Fedora packages or other Fedora contributor accounts were affected by this compromise.
While the user in question had the ability to commit to Fedora SCM, the Infrastructure Team does not believe that the compromised account was used to do this, or cause any builds or updates in the Fedora build system. The Infrastructure Team believes that Fedora users are in no way threatened by this security breach and we have found no evidence that the compromise extended beyond this single account.
As always, Fedora packagers are recommended to regularly review commits to their packages and report any suspicious activity that they notice.
Fedora contributors are strongly encouraged to choose a strong FAS password. Contributors should *NOT* use their FAS password on any other websites or user accounts. If you receive an email from FAS notifying you of changes to your account that you did not make, please contact the Fedora Infrastructure team immediately via admin@fedoraproject.org.
We are still performing a more in-depth investigation and security audit and we will post again if there are any material changes to our understanding.
--
Jared Smith
Fedora Project Leader
The good news is... (Score:1)
I will be releasing Fedora 15 Desktop Edition next week. Standby for download links.
Why would he change the key? (Score:2)
Re: (Score:1)
Because, as they say, "you can't get there from here". Changing the key is the only way to get any further into the system, unless you already have the private key and passphrase.
Re:Yay, Open Source! (Score:4, Insightful)
Re: (Score:2, Interesting)
Actually, even Microsoft employees working remotely have to jump through so many hoops for a flaky VPN connection that there is no way anyone could get in for long enough to do significant damages. A Microsoft employee recruiting in colleges on the East coast showed me their system. You can't use a standard VPN client - even the built-in Windows one. It uses SmartCards and multiple passwords for authentication and disconnects if the card even shifts a bit in the card reader.
It would probably be easier to
Re: (Score:3, Insightful)
Re: (Score:1)
Re: (Score:3)
Re:Yay, Open Source! (Score:4, Funny)
IIRC, something like that has happened before. The attacker managed to get RDP access to one of Microsoft's servers where they keep source code. However, when authorities were able to trace the connection back to his house, they entered to find he had died of a simultaneous heart attack, aneurysm, and stroke, with the Windows kernel source code open on his screen.
Re: (Score:2)
to find he had died of a simultaneous heart attack, aneurysm, and stroke, with the Windows kernel source code open on his screen.
And people say that it's selfish of MS to keep it to themselves...
Re: (Score:2)
Re: (Score:1)
Are you sure he didn't die laughing?
That's an excellent alternative ending.
Nah, Security by obscure VC software (Score:2)
Re: (Score:1)
really low
Re: (Score:3)
Re: (Score:2)
Re: (Score:2)
What are the odds that you are just trolling?
Re: (Score:2)
Nah, if that was true, he would have installed Ubuntu on the system he compromised.