When Sysadmins Go Bad 522
An anonymous reader writes "Here is a story about what can happen when you think you're being oh so clever. This sysadmin planted so-called logic bombs on the systems he was responsible for and then quit. He also tried to game the stock market, buying put options on his former company, hoping to cash in when the disaster he engineered struck. Who can companies trust if they're afraid that this kind of thing can happen? How can they prevent it?"
Someone's been reading a bit too much BOFH... (Score:3, Funny)
Everyone died today? Large lack of posts!
Sheesh! (Score:5, Insightful)
Seriously though, security is a very delicate matter which is entirely built on trust.
Ways to improve security is to limit access to only what you actually need to use. In the case of system administrators and the like, it's not quite as easy as they obviously need a high level of access.
One solution would be to have third party audits of the systems, perhaps with read-only access in order to prevent tampering, but even then you need to trust the integrity and skill of the auditors.
Another thing to remember is to have a solid disaster recovery plan, but that's only good AFTER something happens and the person designing and implementing this plan will likely be the person that has the most access.
There's no universal answer to this problem. If I knew of one, I'd be rich as heck from selling it to companies.
Re:Sheesh! (Score:4, Interesting)
1: The sys-admin had enough access to the systems that he could change the configuration and clean up and prevent the changes from being detected.
2:
The company didn't have proper procedures inplace to stop 1 happening.
Examples of good procedures could be.
*Systems provide automated roll back.
*Changes can only be applied through a script that is run by xyz and required GOD access (say knowlage of a password that changes daily)
*System should be configured to audit any changes that take place
*A review process, where by any changes are reviewed by another member of staff
etc.......
the sysadmin was bad the company was useless, I'm not supprised he quit and tried to take the company down.
Re:Sheesh! (Score:2, Insightful)
1: The sys-admin had enough access to the systems that he could change the configuration and clean up and prevent the changes from being detected.
Right on the money. This situation is yet another good reason why you should have a large enough IT staff.
I also couldn't help noticing that only *nix is capable of meeting your system change policy with any degree of reliability. Fancy that.
Re:Sheesh! (Score:5, Insightful)
One comment on the sysadmin - cretinous moron. If he wanted make money on the options he should have been much more subtle. A sudden surge of damage makes everyone go to the backup tape rack. Everything is restored to pristine state in a day or so and the perpetrator is easily caught.
Compared to this slow corruption and small logical errors in the nth sign after the decimal are much harder to pinpoint and deal with. A similar case in germanyt a while ago operated for more then 5 years before negotiating a settlement. He did not even get caught.
Overall - what a greedy cretinous idiot. They should have fired him earlier for stupidity.
Re:Sheesh! (Score:3, Insightful)
* treat employees with respect and dignity and they won't want to fuck you over
Oh no - that'll never happen.
Re:Sheesh! (Score:5, Insightful)
Examples of good procedures could be. *Systems provide automated roll back.
This isn't a procedure. This is a potential feature of the system itself. When I was a unix admin, I versioned config files, because unix doesn't provide automatic versioning of files, allowing rollback of config changes. However, as the person who set up the versioning system, if I had gone bad I would have been able to sabotage the files under revision control as well. Unless the system itself enforces this (i.e, the system keeps all versions of all files and does not allow an admin to change, in any manner, old versions), this sort of precaution can be bypassed.
*Changes can only be applied through a script that is run by xyz and required GOD access (say knowlage of a password that changes daily)
This, also, sounds good. However, on some Unix systems, at least, there have been issues with setuid scripts related to how the system loads and executes them, allowing race conditions that can lead to root access. Note that the issue I'm talking about is -not- a bug in the script, but rather a side effect of how #! loading is handled by some systems. A large percentage of the Unix S.A.s I know rightly disallow the use of setuid scripts for this reason, and the fact that it's easy to write a script that allows things like
*System should be configured to audit any changes that take place.
Again, not a procedure, but a potential feature of the system. If the system doesn't allow this directly, how do you propose to implement it?
*A review process, where by any changes are reviewed by another member of staff
"Hey Dave, I'm sabotaging the system -- Can you review my change for me? Thanks!" - Do you really think someone's going to let a change like that get into the queue for a review process? Are you advocating a line-by-line code/config review of -everything- every single time a change is made, and do you realize how impractical that is, especially if the deployed system is complex or the number of deployed machines is large? Do you understand that it is possible to make a change that cannot be reviewed [bell-labs.com]?
You can do things to attempt to prevent this sort of thing, but you have to understand that there is no procedural solution for this problem -> the best you can do is reduce the odds that someone can do this and not get caught. This is a laudable goal, but, while in pursuit of this goal, the practical limitations need to be kept in sight.
The moral of the story is, it's very easy to post on Slashdot saying 'x, y, and z would have prevented this', with x, y, and z being impractical/impossible to implement, and through some twist of logic, come to a conclusion such as:
the sysadmin was bad the company was useless, I'm not supprised he quit and tried to take the company down.
Re:Sheesh! (Score:4, Insightful)
Departments do this all the time, with much more complex code. Those departments are collectively called "Software Engineering". It may be impossible to grasp by IT departments, but it is possible, and desired, to review every line of code making its way into the system.
To be fair though, IT has different requirements. When the system is down, you don't have time for a review. But that's no reason not to do a post-fix review.
Re:Not possible... (Score:3, Interesting)
As a volunteer department, it takes us between two and ten minutes to get to the scene. When we get there, we have to appraise the situation, even before parking apparatus. (What good is an engine if powerlines detach from a home and fall on it?)
We don't make split-second decisions. If you rush, you make mistakes. Even if the mistakes seem minor, people can die. Including you.
You follow every procedure you're taught.
Right down to feeling doors with the back of your hand before opening them. If you forget, you're going to get hit with a backdraft.
Forget to wear latex gloves before treating a bloody accident victim? You better hope they're not HIV positive.
Did you remember to put the spanners back in their mounts? (A spanner is a firefighter's wrench.) If not, how are the people running the engine going to know where to get the spanners to tighten the leaky coupling between the hose and the engine itself?
Did you remember to turn the coupling between that 200psi hose in the right direction, to tighten it? No? I wouldn't want to be in your shoes when it whips around like a possesed snake. (For reference, a 2 1/2" uncapped hoseline expels enough force to accelerate a 50' charged section of hose at 12 m/s^2.)
The bottom line is, you don't come up with a solution to the problem halfway through, you need to spend some time coming up with a plan. For large public locations, like a Best Buy or a Sears, the fire department responsible for the area will usually work out a plan ahead of time for handling anticipatable situations.
Re:Not possible... (Score:3, Interesting)
They are never 100%
Re:Sheesh! (Score:5, Insightful)
Sad but true all too often.
I had a friend who after being with a company for three years was the victim of a whole lot of drummed up charges, it was clear that the real motive was cut backs, I guess HR and many others didnt like the fact that he earned more than all of the rest of the administrators combined. So one day he was escorted out of the building, after which they literally unplugged the network, the whole Australian network (3000+ users) was offline for three days while the rest of the admins rebuilt every server!
Did it do any good? No, of course not. A typical simple minded HR view, after spending probably many thousands of dollars in time (and consultants) rebuilding the network not only was he still able to gain access, but he won a big unfair dismissal payout!
Sounded cruel at the time. (Score:5, Insightful)
Re:Sounded cruel at the time. (Score:5, Insightful)
Re:Sounded cruel at the time. (Score:5, Interesting)
Gets my vote. I saw this blow up at my current workplace when a former IT drone's account was deleted (not suspended) as soon as she left the building, without anyone realising it was used as the service account for many things, including the backup server. It took many hours to track down all the things it was used for and to furnish them with saner accounts. I think this probably counts as an accidental logic bomb.
The really sad part of this is tale that it took over a fortnight for anyone to notice in the first place. Weep.
(I'm not part of the local IT department, so I'm blameless with respect to this particular fuck-up. I commit enough fuck-ups of my own without claiming responsibility for anyone else's!)
Re:Sounded cruel at the time. (Score:4, Interesting)
This absolutely screams of bad process design and the blame must go to inept management.
Some suggestions I'd pass along (having learned the hard way the first time, as well have having played on both tech and manager side of the fence):
- use role accounts/contacts, not personal ones: Domain registration, administrative accounts on servers, contact email addresses for company stuff, etc. should all point to a generic role contact or account. It's easy to map these to the appropriate individual accounts, but avoids the hell of deleting accounts when someone leaves. I've had to personally intervene with countless companies that have had their Internet domains registered in an employee's name (individual, not role) and experienced all sorts of nonsense when the employee left.
- require documentation (and if you're a tech, provide it and maintain even if you're not asked): Too many tech folks act as if knowing and not sharing process information, passwords, etc. is job security. It's not - it only ensures that when you go, they'll get rid of you like ripping off a bandaid, rather than offer obligatory goodies (severance, consulting contracts, etc.). I've been an advisor to many of these episodes where some tech had attained too much system control and refused to share it. The slightest demand for special treatment from these techs usually creates a knee-jerk reaction, but in the end, the tech always loses (so what if he downs the company's server for a few days - he just ensures bad references will spread and he'll be unemployable at any real job). Share your information! Document your password. Give copies to your boss. Being open like this creates trust and you'll be rewarded by knowing more things not usually shared, or in the even of a downturn, you'll probably get favorable treatment or even be retained (because they can trust you).
*scoove*
Re:Sounded cruel at the time. (Score:5, Funny)
Please, please, please take his advice!
I would be extremely disappointed if my cron jobs that sabotage the company did not run after I left!
Re:Sounded cruel at the time. (Score:5, Informative)
Re:Sounded cruel at the time. (Score:5, Informative)
C//
Re:Sounded cruel at the time. (Score:4, Informative)
Yep. Standard practice at several places I've worked is for me to be asked to watch for a certain person to walk into the HR department. As soon as they're through the door, disable the account. That way, by the time they know they're being made redundant, they've already lost their access to the system. At a bank I worked at, that was followed by the unlucky victim being frogmarched to their desk by security, allowed to collect their personal artifacts, and then being escorted from the building...
Re:Sounded cruel at the time. (Score:2)
Re:Sounded cruel at the time. (Score:3, Insightful)
As others have mentioned, disabling accounts is significantly better than deleting them. A very good paper on the process of dealing with the termination of a system administrator is Matthew Ringel and Tom Limoncelli's Adverse Termination Procedures [lumeta.com].
--Phil (I highly recommend Limoncelli's other papers, too, especially "Deconstructing User Requests".)
Don't delete, disable... (Score:3, Insightful)
Nuking an account kills part of your auditing trail and/or proper file associations when you do it. Besdies, if you need to check something as a specific user it can be a bear to undo the dammage. Temporarily suspending access can happen just as often depending on the environment, so why not simplify it to one process?
Besides the practical option of re-enabling the account if the person comes back, disabling accounts is a good habit preventing nasty problems fixing mistakes (John Smith vs. Johan Smith).
Re:Sounded cruel at the time. (Score:3, Informative)
You didn't happen to work at a biotech production lab in Cincinnati, did you?
Re:Sounded cruel at the time. (Score:3, Funny)
Dilbert Comic:
'Ted the Generic Guy' walks into the office and complains to the boss: Jeez, my security card access wouldn't work so I had to tailgate into the building. Then my phone mail refuses to let me in and my network password was refused! Is it possible for anything *else* to stop working today?
The Boss: Tee hee hee ...
When /. Sysadmins Go Bad? (Score:3, Funny)
I go to post a comment and I get a page full of ads. I think someone set up
/. caught the clap from k5 (Score:2)
How can they prevent it? (Score:3, Funny)
Have everyone running WINDOWS XP! That doesn't need any system admistration at all, it has perfect uptime and is fully transparent for even the dumbest user!
[/sarcasm]
perhaps a little late but... (Score:5, Informative)
http://sageweb.sage.org/resources/publications/
google cache:
http://216.239.53.100/search?q=cache:tKsX
Re:perhaps a little late but... (Score:2, Informative)
What the gentleman above meant was:
Every sysadmin should read this...
Code of Ethics [sage.org] (Google cache) [216.239.53.100]
Re:perhaps a little late but... (Score:2)
Re:perhaps a little late but... (URL Update) (Score:2)
http://sageweb.sage.org/resources/publications/co
For some reason, the Slash page is putting a space in the original URL? It did the same here, but I added the HTML for the link and it works.
Re:perhaps a little late but... (Score:4, Interesting)
Secondly - I am reminded of a time when I worked for this little VOIP company in San Francisco.
No naming names - but this company was ran by a very nepatistic group. Husband=CEO Wife=COO Best Friend=CTO Other Friend=CFO etc etc...
anyway - when we came on board to build out their VOIP systems they didnt want any of the admins to have the root passwords to any of the machines. any of them.
I had to explain to them that at this stage (build out) that was impossible. They were really really concerned that they couldnt trust any of us.
It was sad - as they really thought that the only people who should have the root passwd werethe COO and the CEO. The two people in the company who couldnt tell a login prompt from their ass. They knew the least about any system in the building - yet were adamant that the admins cannot be trusted and shouldnt have root.
I basically told them that if we didnt have root - we may as well quite because we wont be able to do anything they needed. I explained that after the system was actually built out and running - that we could reign back in control - and only a few people would have the root passwd and we would run things through SUDO.
They were still skeptical. I then explained to them that there is just an operational ethics that IT admins have. I told them that there are things that admins just do not do. I told them that we have access to every file on the network, access to all email. Anything. But that it just is not acceptable to look at any of these. That we just *dont* look at these things.
I thought this would make it better - but it was worse. These idiots didnt realize that having root actually meant that we had access to every single file on the network. This made it really bad - now they really really didnt trust us. They didnt understand the fact that there had to be a certain level of TRUST in their admins.
We later realized the reason why they couldnt trust any of us was because they were just a bunch of slimy non-trustworthy lying execs themselves - and didnt see how anyone else could possibly be any different.
Thank god I dont work there any more. Too bad they are still in business too... I understand that they are still late with employee paychecks. Although I am sure the exec salaries are always on time.
Re: Sysadmins never go bad (Score:4, Funny)
Re: Sysadmins never go bad (Score:2, Interesting)
Only
Not too bright (Score:2)
You *could*... (Score:4, Insightful)
Obviously, you could get two bad apples and have the same thing happen, but odds are slim.
Problem is, it tough to find ONE good admin, much less two, esp. with tough times for business... having to dole out twice the budget to protect yourself "just in case". Then again, it would double the job market =)
OR mabye CVS everything, and look through all changes an employee made after they quit... then again, the clever get around this, etc.....
*sigh* People just suck sometimes.
Re:You *could*... (Score:3, Insightful)
Even assuming the absence of an all-powerful superuser, there are problems. Someone has to be responsible for installing, maintaining and perhaps upgrading the application that manages the dual-approver system, so there's at least one person who doesn't need any confirmation before setting you up the bomb.
And even if you solve that problem, there's the problem with untrustworthy hardware. Someone somewhere has physical access to the box, which would provide them with the ability to, say, take the disk drive "for maintenance", mount it in their own box, diddle whatever code they want, and return the "fixed" drive to service.
And that brings up the problem of... and then the problem of... not to mention the problem of... it just keeps going. With our current technology, it's literally impossible to eliminate the issue of trust in our computing environments. They say everyone has their price. Scary thought, isn't it?
Re:You *could*... (Score:3, Informative)
Add additional safeguards as you see fit - for instance you could have two people who know one half of the password and two different people knowing the other half, or three people each knowing a third of the password, and so on. It might be inconvenient on occassion, but hey, since when has decent security not caused a little inconvenience?
Re:You *could*... (Score:4, Insightful)
No one who has ever worked in the real world would come up with such a thing! I'm just a lowly tech and I need root on the workstations I work on on a several time per day basis. If every time I wanted to do something I had to track down another person and have them be in the same physical place as me it would be insane. Now think of the sysadmins out there who get paged at 3am when something blows up. Now not only do they have to get up but so does someone else and they both have to believe that the other person will show up. The reality is you screen applicants, make sure you have some kind of regular contact with your employees, and finally have some system for angry people to vent without fear of reprisal. On my team I established an email list for bitching and complaining and made sure that no managers were on the list but also made sure management was aware of the lists existance.
Re:You *could*... (Score:5, Interesting)
Re:You *could*... (Score:3, Informative)
Don't get me wrong, I use sudo every day, and it's a great tool. But you have the following problems anyway:
These occasions occur on a regular basis. If I had to track down a frickin' envelope and get an Act of Congress to let me open it each time, I'd just quit.
Re:You *could*... (Score:3, Insightful)
Staff your IT department (Score:5, Insightful)
First of all you'd have less disgruntled employees.
Second, you'd have less disgruntled employees.
Third, you wouldn't need to trust anyone 100%. Most egos of sysadmins wouldn't let them let someone else compromise their system. If you have 2 or more admins 100% responsible for the integrity of a system, and each performing checks on each other, you would reduce the occurences of these types of attacks.
Re:Staff your IT department (Score:2)
PHB: I have one sysadmin on salary, and he seems to be over worked, I could:
- Hire another -- NO, HR would want me to fill out a whole pile of forms to justify this, and I wouldn't have time for golf.
- Just tell the existing one we will hire another -- YES, but really do nothing, and keep working him into the ground. Then only hire another when this one quits.... eeeexcellent.... when was that tee time again.
It really comes down to "perceived" value, since I.T. does nothing but "suck" money out of the company, them all must be a bunch of slackers who deserve nothing.PHB at review time: The sysadmin seem to work lots of overtime, but since he's on salary this doesn't cost me anything, he must lack in organizational skills, we'll have to cut his bonus for that (and add it to mine for pointing out this flaw in the employee, I'll be able to get that new driver for my golf game now).
Remember in the PHB's eyes salary == slavery
Damn (Score:5, Funny)
What can be done? (Score:5, Insightful)
> this kind of thing can happen?
Nobody.
> How can they prevent it?
They can't.
Employee misbehavior spans an entire spectrum of seriousness, from stealing paper clips to embezzling billions. You can't prevent a determined and dishonest sysadmin from sabotaging a system any more than you can prevent an accountant from diverting funds or an after-hours custodian from taking things off peoples' desks.
There is no panacea, technological or otherwise.
Preventing employee misbehavior has several parallels with Copy Protection. No affordable and practical scheme is bulletproof if the person is determined enough, so the best method is to remove the motivation. The same rules apply to all employees: treat and compensate people fairly and they will be less likely to want to hurt you.
But even that doesn't work in all cases. If your staff is large enough there will always be people who feel that you are mistreating them, or underpaying them, and who will feel compelled to get what is "rightfully theirs" in other ways, large and small. And many people steal/etc. without regard to the harm it causes the company or other employees; their motivation is purely selfish, so it doesn't matter how well they are treated and paid.
So even if you treat and compensate people fairly, and trust everybody you hire, you must monitor people's activity, investigate suspicious behavior, and, when necessary, prosecute wrongdoers to the fullest extent of the law.
I probably sound cynical, but I speak from experience.
Re:What can be done? (Score:2)
Ah, but you can make it harder, by having servers administered rotate among admins. That way, you cross train, and if something looks fishy, it can be.. fixed.
Re:What can be done? (Score:4, Insightful)
For some reason technical people tend to ignore many years of experience of similar problems in other domains. Quite simply, there are several effective mechanisms for preventing this type of abuse, but very few people which sufficient know-how to implement them.
The business rules for prevention of white collar crime are division of responsibilities, and cross checking (or auditing). The rules do not change just because you are working with computers.
The first thing to realise is that on most "enterprise" operating systems other than standard unix, the system administrator is NOT god. On NT, 2000, Novell and Trusted Solaris (amongst others) there is provision for delegating administrative privlidges and locking out the original administrator in an irrevocable manner. On most other Unix systems you can use "sudo" (or an equivalent) to selectively grant privlidges, and lock down root logon or "su" to the console only. Coupled with dual-key physical access control, this prevents any single person from becoming god ((s)he can't even modify hardware or reinstall because of physical controls). This scenario presumes procedures/rules (never leave just one admin in the room, watch and verify all operations, etc).
Many admins baulk at this idea, but if you're serious about security, there has to be a physical barrier preventing complete power over the system. In the absence of computer systems designed for dual authentication for privledged operations, physical controls (and associated procedures) must be used.
When responsibilities are divided, there needs to be an analysis of which privledges can interoperate, and which should not (because they could cause a security risk). The privledge of clearing log files should be limited to "god" - i.e. physical access to the console, which requires two people. Backups should be encrypted, if possible in such a manner that the key for recovery is split between two people (there is software to handle this sort of thing).
Auditing is also essential. Every so often, external experts should be brought in and allowed to inspect the system, under the supervision of one or more of the administrators. It is likewise important that administrators be forced to take time off (instead of infinitely accuring annual leave) -- this is when fraudulent activity is usually stumbled upon.
Does this offer complete protection? No. It won't work in organisations where there is only one admin (unless another technically savvy person can hold the second key for physical access), and it breaks down when two admins cooperate in the fraud. But it provides a whole lot more protection than the current practices, and in time can be improved (by drawing on other business and accounting practices).
What can be done? (Score:4, Insightful)
> They can't.
They can at least reduce the chance a lot with redundency.
If you have a team of sys-admins, you have a good chance that the other might catch the bad one before it's too late. And if they feel treated well by the company and don't share the sentiment of the saboteur, the damage is usually contained.
Another policy I've seen in some banks is that all employees have to take 2 continuous weeks paid vacation each year (the rest of the paid vacation time can be distributed at will). This promotes cross-training and redundancy.
Re:What can be done? (Score:4, Insightful)
you hit it on the head.... A "bad" sysadmin is far less dangerous than your "bad" accountant..
many MANY companies were robbed blind by a bad accountant embezzling money yes you dont hear this sensationalized like this article. it doesnt matter, from the janitor to the CEO EVERY EMPLOYEE has the ability to completely ruin your company.. anyone that is paranoid about it means they know they are screwing their employees and are sure they are disgruntled and TRYING to get back at them.
if you want to reduce the risk of having disgruntled employees screwing your company there are 2 things you need...
1 - Pay them fairly and treat them well. this is the MOST important thing. they will NOT respect you or your company if you don't respect them.
2 - critical parts of your company need redundancy.. if you have 15 computers and 1 sysadmin... HIRE AN ASSISTANT FOR THE SYSADMIN. less sneaky stuff happens when someone has a shadow. same as Accounting... have your books audited by someone else on a regular basis.. wow now is a good time to actually LEARN how to run your business instead of playing golf or having your Mercedes detailed.
99% of all bad things that happen in a business is the managemet's fault. their inattentiveness or apathy coupled with ignorance and sometimes just being a plain old asshole to their employees.
similar story (Score:5, Interesting)
A couple weeks after he left, the system started crashing and losing data. Apparently he used a rather well-known bomb because the company they used for support was able to dial in and found it rather quickly. He was charged, arrested, tried, and found guilty. It was a big deal because the state (South Carolina) had just passed some really though computer crime laws at the time, and the Attorney General wanted a "test case" for the law.
My Dad and his partner's requested that the guy not get any jail time since he had a wife and some kids, but he got major probation and a huge fine (something like $60,000, which was a lot back then). Plus he now has a felony charge on his record. Last I heard, he had gotten out of the computer biz and was working in a family business.
Anyway, the short lesson is: if you're a company firing someone with privileges, pay them the two weeks or whatever but don't let them back on site. And if you're the guy getting sacked, don't try to get revenge through sabotage; it's just not worth it.
As an aside: every place I've worked had a policy that whenever someone was fired they were led to their desk with a cardboard box, then escorted out of the building that very moment.
Re:similar story (Score:2, Insightful)
> was a lot back then).
Wow. I must not be making enough money, because I think that is still a lot.
Re:similar story (Score:2, Informative)
For example, right now UBS stock is about $50 and for $0.40 (last trade) you can by a put option with a strike price of $45 that expires in about a month. So for $0.40, you can by the right to sell the stock at $45.
If the UBS were to drop to $40, the payoff would be $4.60 (45-40-0.40). A $21K investment would pay $241K (less commissions).
...so? (Score:3, Insightful)
How is this different from any other kind of sabotage by employees or ex-employees? As long as there have been accountants, there has been embezzlement. A short-order cook could forget to wash his hands. A construction contractor can use sub-standard building materials.
You gotta trust somebody; just make sure it's somebody worthy of trust.
As for preventing this particular kind of sabotage, use the same principles as everywhere else: supervision, audits, bonds, insurance, and the threat of jail time if the rest fails. Oh--a good disaster recovery plan sure doesn't hurt, either.
Cheers,
b&
A novel way to pay for retirement... (Score:5, Interesting)
Configuration Control (Score:5, Informative)
Re:Configuration Control (Score:2)
Tech plan = Good; Financial plan = Bad (Score:4, Funny)
"Hmmm, there's the guy who had access to the company's computers and made all those put options, but I don't know if there's any way we can prove motive or opportunity."
Re:Tech plan = Good; Financial plan = Bad (Score:2)
Who can you trust? (Score:4, Interesting)
Trust in God; Everybody else pays cash
Who can you trust? -- Nobody. As our master said:
Machievelli, The Prince Ch 17. [bartleby.com]The answer to the question is no one, not even your mother. If you are not secure against being hacked by an insider, you are not secure. And that means everybody, Newspapers are full of headlines about CEO's ripping off their companies. Stories about long-time trusted employees who embezzle a few hundred thousand dollars are so common that they usually wind up on page 7 of the Metro section.
they can never prevent this happen (Score:5, Insightful)
SysAdmin, as the word says, it's the Administrator of the System.
there's no technical way to restrict their actions, or we should restrict the computer's capacity.
people do bad things for money, that's all, how could we prevent this happen? how could we prevent crime? how could we prevent people shoot each other? these are analog.
it's political or human issue. not technical.
How to avoid this problem (Score:5, Insightful)
This means:
1) Help work to keep employees happily employed (not with bribes - with real career paths, personal interest, etc.). If you keep wage-slaves, expect mutiny.
2) Actively replace employees who can't be kept happily employed. Get others who are competent and glad to have the spot (which shouldn't be too hard in this economy). Keeping people around who don't want the position isn't doing them any favors. If no one who would be qualified would also be glad to have the spot, rethink the position.
"Management" should be helping manage situations like this. If this guy had been disgruntled for a long time, it seems to be their fault for keeping him (and keeping him unhappy and ultimately vengeful). Sounds like someone did a bad job at people-management . . . sounds like the type of willfull neglect that is inexcusable but all too common. Many people think that "management" is watching the bottom line -- that is a lazy, oversimplified way of looking at an important job.
Re:How to avoid this problem (Score:3, Interesting)
I second that motion. Money is only one means of rewarding/compensating your staff. Respect is another one, and one which often is ignored.
I once did a gig as a conslutant for $COMPANY. When the $PHB who hired me introduced me to the SysAdmin, the $SA was visibly displeased. I suspect that
- $PHB had failed to mentioned to $SA that this hire was taking place
- the $SA didn't have a say in the hiring process (he certainly didn't interview me)
- the $PHB may not have mentioned to the $SA that $PROJECT was taking place.
So, when $PHB mentioned to $SA that he needed to set me up with a computer and network account, $SA gave me the list of all of the admin passwords on all of their servers and said I could set up my computer and account myself. $SA quit within a week after I was hired.Needless to say, that was an interesting experience.
Sysadmins? (Score:5, Funny)
C'mon -- this is really small potatoes ...
Re:Sysadmins? (Score:5, Insightful)
yeah, but the difference is, the sysadmin is a criminal, a CEO that's stealing is just unethical...
Re:Sysadmins? (Score:2)
My guess is the Paine-Webber guy will be reamed, and that's justice. But what about the people whose salarary+bonuses alone cost more than the damage this guy caused, and whose shenanigans drive a billion-dollar company into the ground, destroying people's pensions? That's a heck of a lot worse than forcing a Windows reinstall.
I can't believe (Score:3, Funny)
There's hope for me yet.
don't put all your security eggs in one basket (Score:3, Insightful)
BOFH alert! (Score:2)
That guy annoyed the hell out of me one too many times.
Forensic hacking (Score:2, Interesting)
Ethics aside, I have to admire this guys balls!
I'll put my ethics back on and fix the sendmail f'up I made this morning now :-)
This article isn't very good. Neat story though. (Score:2)
So-called logic bombs are pieces of software code buried within another program and are designed to disrupt computer systems. They are often delivered by e-mail.
Ok boys and girls, would someone like to explain how this is different than a virus/Trojan?
Keep in-mind that I am not a financial expert, nor the general public that I can assume are reading this article. With that in mind... the following statement is even more mind boggling:
He allegedly bought more than $21,000 of put options, which grants an investor the right to sell a certain amount of underlying stock at a certain price. By giving the investor the right to sell underlying stock at a given price, put options increase in value when the stock value falls.
Christ.... wtf does that mean
All in all this article goes into no detail in regards to how he was caught, and how they in intend to prove it's him.
--Noodles
Re:This article isn't very good. Neat story though (Score:5, Informative)
Suppose that the stock of company FooBar is worth $80 today.
I buy the *option* of selling that stock at $80 in one weeks time (this of course cost me something since there is a risk involved for the entity that I buy this option from).
Let's say that priviledge costs me $1 (since everybody considers company FooBars stock prices to be quite stable).
Now, one week later the "bomb" has blown up their computer system and the stock has plunged to $40.
The option of selling one stock at $80 is now worth $40 since the stock is currently priced at 40$. I don't even have to own the stock since someone who does can buy the option from me instead.
In total I've made 39$ on an investment of 1$ in one weeks time.
Re:This article isn't very good. Neat story though (Score:2)
The article is just not clear about the definition of a put option at all.
(Yes I know there is another definition linked in the slashdot post.)
Response (Score:2)
Management: "We don't need a sysadmin, everything is working just great!"
Insider Threat (Score:3, Insightful)
TANSTAAFL.
what the... (Score:2)
"how do people prevent people with privledge from fucking them over?"
i think employee onto employer a mockery of the likelihood of employer unto employee.
i dont have much more of flame bait suggestions for answers, just more or less refined questions.
Easy answer... (Score:5, Funny)
Who can you trust?
Microsoft. Trustworthy computing.
At Microsoft, we make operating systems that administer themselves, so you don't have to hire those untrustworthy and expensive system administrators. Nearly any high-school graduate, or poo-flinging monkey, with the proper brainwa^H^H^H^H^H^H^H training can become a Microsoft-Only Operations Certified Omnipotent Worker. Get your own MOOCOW today, and let us handle your security problems. You shouldn't have to worry about these computer dealies - that's our job.
Microsoft. Trusted Computing since 2002.
Sysadmins hell, I want to kill the execs; story (Score:4, Interesting)
At a big EDA firm I worked at the sysadmin got into big trouble (I think he was fooling around on his old lady and was trying to run away with some other chick). He decided to hose the backups by placing a small magnet on the read/write head (IIRC). Then he did real backups, which he hid in the drop-down ceiling. His stupidity led him to try to blackmail the company (gold coins). The episode ended badly--high speed chase, crash, prison. Now that I think about it, yeah, a Fox mini-series!
doug
For large businesses - multiple admins (Score:2)
In the schools where I work, I can walk into another admin's school and be fairly comfortable with making fixes/changes to their system - since everything runs similarly. This is convenient if one of us gets sick, or has a holiday, etc, and a server goes kaput somewhere.
Some of us are more well-versed than others, and one of the other admins has a much better knowledge of most of the systems than me - in particular our main user repository.
I can get by fairly well the "armadillo book" (0'Reilly) when there's something I don't understand, but sometimes I still need to call him when things go awry. For those that need to catch up with other admins, I do recommend the O'Reilly books though. I've only been here a few months, and I expect that after time (and reading) I'll be much more confortable with some of the systems I'm not currently as fluent in as others.
"Logic bomb"? (Score:2)
It just sounds like to me the guy set up a nice little crontab entry that no one bothered to check that did a rm -rf /* on their systems. But, then again, the article did say...
Duronio, a computer systems administrator, resigned from PaineWebber on Feb. 22 after complaining about his salary and bonuses. The logic bomb he allegedly constructed from November 2001 until February of this year was activated on March 4, U.S. Attorney Christopher Christie said in a statement.
So this guy was clearly dumb, executing something like this only two weeks after he left. I could see how it would take him from November to February to figure out how to work cron.
sounds like... (Score:2)
Prevention is not all that hard (Score:5, Insightful)
All of this costs money, but think of it as cheap insurance, compared to the cost of rogue sysadmin. Is it worth penny-pinching on salaries and benefits, while maxing out the workload if that results in disgruntled employees who timebomb your systems as they head for a new job?
If you paid the sysadmins $1 million per year, there would be zero theft, zero funny business, and zero turnover. Of course, nobody can do that and stay in business. At some level less than $1 million and higher than fast-food wages, you can retain decent people and discourage malicious tactics. The key to avoiding a technological meltdown is to treat people well enough so that your recruiting process lets you avoid the marginal candidates. Once hired, a properly compensated person should feel as if the "have something to lose", and therefore you can expect such a person to act as a professional. Paying hamburger wages and putting a person in the sysadmin seat would be like staffing a nuclear power plant control room with random selections from the phone book.
This is a very interesting topic, especially right now. We are in a down market, and there is an irresistable temptation for some employers to make lowball offers to currently-unemployed candidates. This allows the employer to cheaply refill vacancies (or exert leverage against current employees). Those employers who are gung-ho about bottom-feeding are setting the stage for big trouble later. Employee turnover is just the tip of the iceberg.
Change control + tripwire (Score:2)
- Take daily md5 snap shots of systems
- Always keep off site duplicates of your monthly full back ups. It's not just for DR; it's also for versioning.
- Sue him out of existence and make sure EVERY employer in the area knows about it - not just for vengence, but also as a heads up to other rouge sysadmins.
In other words, follow best practices and procedures.
On a related thought... (Score:4, Interesting)
My take on it would simply be that your employer did not pay enough attention to your activities abd subsequently due to their mismanagement you would not be at fault. Comments?
How can you prevent it? (Score:3, Insightful)
Oil Strike in Venezuela (Score:5, Interesting)
Time bombs (Score:4, Interesting)
1985: A travel company with several offices (local big group) had only one sysadmin for their computerized booking system. He was this nasty guy who was related to one of the founders, and no one wanted to fire the guy because only he knew how to run the damn things. Not that he did a good job. He was lazy, rude, and demanding. Well, one day, new management got sick of him, and tried to get an "assistant" for him (read "learn his job so we can fire him"). Sysadmin was wise to that, and basically they went through several employees in as few months. Finally, they decided to fire the guy, and hire a contractor to replace the systems. The firing was ugly, they ex-admin had to get dragged out by the police in the end. Days later, the whole system went down. Guess what? No backups. No one knew how it ran, and years of data was lost, chaos among their customers ensued, and six months later the company went out of business.
1996: Our company bought out a competetor. They guy in charge of the call center was the only one we didn't lay off right after the merger because he was the only one who knew what went where, and he used this knowledge to leverage his job security. He was impossible to work with, never did anything on time, never answered his pages, and did just enough work not get fired, but it was really, really hard to get him to do anything else. Finally, we gathered a team of experts (our staff plus vendors) to go as a group, figure out what he was doing, then fire him. His response? He deleted all the call center tables, databases, and destroyed all paperwork... then quit. We had him arrested, but he posted bail, and we never found him again. It took half a month to get everything working right, which meant we had to tell 300 call center employees they couldn't come to work or get paid until we called them back. Boy, was that a clusterfuck.
I saw this button once, "Now that I have changed the master password for the database, it is time to discuss my salary." Heh.
1997: The head of our HR department was fired due to some political bullshit. Standard procedure was to take an ex-employee's computer, wipe it, and give it back to the tech department. Guess what we lost because no one thought about it? All employee records for the department. Backup was on a single floppy that wouldn't load, and she hadn't done backup since the first of the year anyway. We had to have every employee resubmit 1099s and W4s, plus tell us honestly what vacation and sick they already took.
1999: Same company, same situation, but this time it was the guy who kept the entire tech department hardware inventory records. It took a year to recount what we had, and re-enter serial numbers and license keys into a new database. The stupid thing was, this guy made regular backups on the network drive... which was on a server they wiped by accident. Doh!
2001: After a round of layoffs, one of our more brilliant and inspired programmers had "expiration dates" on all his compiled software. He wrote most of the tools we still use today. Months after he was laid off, all of them stopped working on September 17th, 2001 at 12:00 midnight. The only way we got saved was that no one wiped his original desktop box (which had the source code on it, which is how we found out about the "expiration date"). So we recompiled without the date, and everything worked again. Due to WHEN it happened, our whole company thought we'd been attacked by terrorists (the clever generic error only said there was a "network failure") until the truth was revealed. Later we found 9/17 was his birthday, and it was just coincidence it happened so close to 9/11; the layoffs were in March, and they were unexpected and sudden. I doubt this guy had Al-Queda (sp?) connections, so he must have been planning this "job security" (as the comment in the code labeled it) way in advance.
Jurassic Park Lesson (Score:4, Funny)
Remember the lesson of "Jurassic Park":
If you don't pay your programmers enough money, a tyrannosaurus rex will eat your lawyer.
Had this myself (Score:3, Interesting)
Payback is a bitch (Score:3, Insightful)
Remember Columbine? Think about it... For every PHB... For every dumb joke about "geeks"... For every clueless idiot employee that businesses give computers to who eventually DEMAND that the Sysadmin practically do their job for them... If you run a business, consider yourself LUCKY that they used a "LOGIC" bomb instead of getting MEDEVIL ON YOUR SORRY ASS.
Here's how you avoid all this (and consequently save your wretched, lost corporate soul):
o Pay them what they're really worth
o Let them charge your company internally for the services they provide. That way you'll see exactly how they are NOT JUST OVERHEAD.
o Run your business ethically. Don't force them to do your dirty work.
o Hire more IT staff. I GUARANTEE you that you don't have enough right now.
o FORCE your people to get trained or don't let them touch a computer.
o Hire the best IT guys you can find (not just paper M.C.S.E's) and outsource your Help Desk so innane questions from lame lazy-ass employees can drive someone else insane.
o Actually spend the money it takes to do IT correctly. It actually does cost lots of money. You're IT budget is probably off by a factor of five.
o Stop trying to figure out how to cover your corporate ass in the event of a Sysadmin "going bad" (Translation: "that you drove insane") and follow the steps above.
Remember, Sysadmins don't "go bad". They eventually become INFECTED and INFESTED with the EVIL that corporations have learned to LOVE while doing business.
Answer: Sysadmin diversification! (Score:3, Insightful)
Not only will having more sysadmins reduce the ability of one bad seed to FUBAR things, but paying and training them decently will also make them all happier, and less likely to *want* to.
But this is all purely leftist drivel.
Some comments on this... (Score:3, Interesting)
I've heard some other random stories about coders and sysadmins setting up time bombs that will go off after they are gone. Some of it is very minor (such as a small prank) to very dustructive (such as deleting years of records, etc.).
I also know that many coders do not document or comment very well on purpose, as they percieve this as a way to preserve job security (if I'm the only one who knows how to use it...) as well. This is not quite as bad... but still...
Anyway, there are a couple things that might protect from this happening, though there are probably no ways to prevent this 100%.
#1. Have multiple admins, and audit everything!
Having multiple administrators who know how to
run lots of machines will minimize damage, or
prevent it. Having audits of everything, and
requiring good documentation/comment also
will help to dissipate the risks.
#2. Don't treat your employees like crap!
While employer mistreatment doesn't justify
sabotoge, it does encourage it. If management treats employees like crap, retaliation is more likely.
On my honeymoon (Score:3, Interesting)
Turns out some fool had modified a record without using the proper indexes (ancient FoxPro for DOS). Because the indexes were no longer synchronized, the software's "do while order == opened" loop hit a closed record that was indexed as open, and exited prematurely.
I went in, fixed it in five minutes, and left. They were bankrupt within 4 months, and I was thankfully on to a new employer (that didn't trust employees any further, but that's another story).
Unfortunately (Score:2, Insightful)
Re:How does this profit? (Score:4, Informative)
With a "short sale" you can borrow stock that you don't own, sell it, then later on, after the value has fallen, buy it, and give it back to its owner. Think of borrowing your neighbour's lawnmower in April when lawnmowers are expensive. Sell it for $200. Then in November when lawnmowers are cheap, buy a lawnmower on clearance for $100 and give it back to your neighbour.
Options (a put option is one of two kinds of option) are a bit different in that you don't actually buy any stock. You only buy *the right* to buy (call options) or sell (put options) the stock at a given price.
What's the difference?
Well, for options, you have a limited risk (it's impossible to lose more money than you put in -- the worst that can happen is that your options become worthless and you throw them away). But with a short sale, the risk is potentially limitless, since it's possible for the stock price to be infinitely high when you have to buy them back and repay the lender.
Re:Good story until... (Score:3, Insightful)
Re:20 years (Score:3, Funny)
I think it was in SF, and they got called for a survey about crime. They got asked "how do you feel about crime rates?" They asnwered "I think they're going up, Enron is stealing millions WorldCom is stealing millions, so is Xerox and a bunch of other guys." The survey taker was taken somewhat by surprise by this. "Um, no I mean street crime". "Oh you mean some guy who's going to steal $6 from my wallet instead of a couple thousand dollars from my grandma's pension fund?" The survey taker sid "um, yes." "Oh, I think that's getting better, though crime overall is bad."
Some junkie jonesing for a fix steals some car parts, goes to prison. WorldCom execs lie and still get millions from bankrupt companies. [theregister.co.uk]
Re:On a somewhat related note... (Score:5, Interesting)
He can't. I've had this happen to me one or two times. I've been pushed in to sysadmining (dammit, Jim, I'm a programmer, not a sysadm!) in this small association (about 60 employees, about 60000 members), and initially just assumed the system I took over was OK. After a year or so I discover, quite by accident, the first horrible thing... Every user PC has a small script on it, that contains the root password to the main server in plaintext.
Apparently, no-one knew. I was responsible, even if it was my predecessor (or his) that had written that script. What to do? Go up to the boss and say "Hey Joe! Funny thing, any employee may have had root access to the DB in the last five years! Ain't that funny?". No. Fix it. Shut up.
There were a few almost as horrible things I fixed quietly over the next few months.
I also have to confess that I have did a horrible blunder myself, that has gone undetected. What do you do when you find that a bug in an old program you wrote has lead (over the last six months) to >4% of your members mailing addresses beeing slowly mangled? When membership dues are mostly collected by mail? Which has lead to large losses for the association, and great unhappiness among the members?
Fix the bug, correct the adresses as much as possible, delete the evidence, lie when confronted. That's what you do.
Re:On a somewhat related note... (Score:4, Informative)
If you took it upon yourself to "audit" the system without specificly getting permission, then you probably violated a policy and potentially broken the law. The real answer is "don't do that."
Obviously "good" is tied to "doing what you're authorized to do," NOT "finding things that could potentially be held over someone's head but not yet taking advantage of them.
The company is repsonsible for ensuring its shareholder value is protected from people who violate policies and laws.
Randall Schwartz got a felony conviction- I don't believe anyone argued that he was going to maliciously use the information he gathered, but he violated policy and the current law in that jurisdiction. Exceeding your authority accessing computer systems is wrong. If you want to look around *get written permission* from someone who's authorized to grant it.
I do computer forensics relatively often on behalf of corporate clients. If something ominous happened to a machine you'd just probed that evidence wouldn't do you any good- even if you weren't linked to the orginal problem.
If the work environment is right, go in and admit improper access, explain why it won't happen again without permisson and explain the findings. Otherwise, an unrelated event could put a bad spin on it that could do you real damage.
Paul
Re:Logic Bomb? (Score:4, Informative)