Slashdot's Setup, Part 2- Software 151
The software side of Slashdot takes over at the point where our load balancers -- described in Friday's hardware story -- hand off your incoming HTTP request to our pound servers.
Pound is a reverse proxy, which means it doesn't service the request itself, it just chooses which web server to hand it off to. We run 6 pounds, one for HTTPS traffic and the other 5 for regular HTTP. (Didn't know we support HTTPS, did ya? It's one of the perks for subscribers: you get to read Slashdot on the same webhead that admins use, which is always going to be responsive even during a crush of traffic -- because if it isn't, Rob's going to breathe down our necks!)
The pounds send traffic to one of the 16 apaches on our 16 webheads -- 15 regular, and the 1 HTTPS. Now, pound itself is so undemanding that we run it side-by-side with the apaches. The HTTPS pound handles SSL itself, handing off a plaintext HTTP request to its machine's apache, so the apache it redirects traffic to doesn't need mod_ssl compiled in. One less headache! Of our other 15 webheads, 5 also run a pound, not to distribute load but just for redundancy.
(Trivia: pound normally adds an X-Forwarded-For header, which Slash::Apache substitutes for the (internal) IP of pound itself. But sometimes if you use a proxy on the internet to do something bad, it will send us an X-Forwarded-For header too, which we use to try to track abuse. So we patched pound to insert a special X-Forward-Pound header, so it doesn't overwrite what may come from an abuser's proxy.)
The other 15 webheads are segregated by type. This segregation is mostly what pound is for. We have 2 webheads for static (.shtml) requests, 4 for the dynamic homepage, 6 for dynamic comment-delivery pages (comments, article, pollBooth.pl), and 3 for all other dynamic scripts (ajax, tags, bookmarks, firehose). We segregate partly so that if there's a performance problem or a DDoS on a specific page, the rest of the site will remain functional. We're constantly changing the code and this sets up "performance firewalls" for when us silly coders decide to write infinite loops.
But we also segregate for efficiency reasons like httpd-level caching, and MaxClients tuning. Our webhead bottleneck is CPU, not RAM. We run MaxClients that might seem absurdly low (5-15 for dynamic webheads, 25 for static) but our philosophy is if we're not turning over requests quickly anyway, something's wrong, and stacking up more requests won't help the CPU chew through them any faster.
All the webheads run the same software, which they mount from a /usr/local exported by a read-only NFS machine. Everyone I've ever met outside of this company gives an involuntary shudder when NFS is mentioned, and yet we haven't had any problems since shortly after it was set up (2002-ish). I attribute this to a combination of our brilliant sysadmins and the fact that we only export read-only. The backend task that writes to /usr/local (to update index.shtml every minute, for example) runs on the NFS server itself.
The apaches are versions 1.3, because there's never been a reason for us to switch to 2.0. We compile in mod_perl, and lingerd to free up RAM during delivery, but the only other nonstandard module we use is mod_auth_useragent to keep unfriendly bots away. Slash does make extensive use of each phase of the request loop (largely so we can send our 403's to out-of-control bots using a minimum of resources, and so your page is fully on its way while we write to the logging DB).
Slash, of course, is the open-source perl code that runs Slashdot. If you're thinking of playing around with it, grab a recent copy from CVS: it's been years since we got around to a tarball release. The various scripts that handle web requests access the database through Slash's SQL API, implemented on top of DBD::mysql (now maintained, incidentally, by one of the original Slash 1.0 coders) and of course DBI.pm. The most interesting parts of this layer might be:
(a) We don't use Apache::DBI. We use connect_cached, but actually our main connection cache is the global objects that hold the connections. Some small chunks of data are so frequently used that we keep them around in those objects.
(b) We almost never use statement handles. We have eleven ways of doing a SELECT and the differences are mostly how we massage the results into the perl data structure they return.
(c) We don't use placeholders. Originally because DBD::mysql didn't take advantage of them, and now because we think any speed increase in a reasonably-optimized web app should be a trivial payoff for non-self-documenting argument order. Discuss!
(d) We built in replication support. A database object requested as a reader picks a random slave to read from for the duration of your HTTP request (or the backend task). We can weight them manually, and we have a task that reweights them automatically. (If we do something stupid and wedge a slave's replication thread, every Slash process, across 17 machines, starts throttling back its connections to that machine within 10 seconds. This was originally written to handle slave DBs getting bogged down by load, but with our new faster DBs, that just never happens, so if a slave falls behind, one of us probably typed something dumb at the mysql> prompt.)
(e) We bolted on memcached support. Why bolted-on? Because back when we first tried memcached, we got a huge performance boost by caching our three big data types (users, stories, comment text) and we're pretty sure additional caching would provide minimal benefit at this point. Memcached's main use is to get and set data objects, and Slash doesn't really bottleneck that way.
Slash 1.0 was written way back in early 2000 with decent support for get and set methods to abstract objects out of a database (getDescriptions, subclassed _wheresql) -- but over the years we've only used them a few times. Most data types that are candidates to be objectified either are processed in large numbers (like tags and comments), in ways that would be difficult to do efficiently by subclassing, or have complicated table structures and pre- and post-processing (like users) that would make any generic objectification code pretty complicated. So most data access is done through get and set methods written custom for each data type, or, just as often, through methods that perform one specific update or select.
Overall, we're pretty happy with the database side of things. Most tables are fairly well normalized, not fully but mostly, and we've found this improves performance in most cases. Even on a fairly large site like Slashdot, with modern hardware and a little thinking ahead, we're able to push code and schema changes live quickly. Thanks to running multiple-master replication, we can keep the site fully live even during blocking queries like ALTER TABLE. After changes go live, we can find performance problem spots and optimize (which usually means caching, caching, caching, and occasionally multi-pass log processing for things like detecting abuse and picking users out of a hat who get mod points).
In fact, I'll go further than "pretty happy." Writing a database-backed web site has changed dramatically over the past seven years. The database used to be the bottleneck: centralized, hard to expand, slow. Now even a cheap DB server can run a pretty big site if you code defensively, and thanks to Moore's Law, memcached, and improvements in open-source database software, that part of the scaling issue isn't really a problem until you're practically the size of eBay. It's an exciting time to be coding web applications.
Lacks certain details (Score:5, Funny)
Re: (Score:2, Interesting)
Re:Lacks certain details (Score:4, Informative)
Re: (Score:2, Insightful)
Nothing for you to see here. Please move along. (Score:5, Interesting)
Re:Nothing for you to see here. Please move along. (Score:5, Informative)
Heh. Actually that's a longstanding bug because of the way we write out .shtml files. We don't pick a timestamp and use it consistently on both index.shtml and the articles' .shtml files. Our backend task grabs a list of which stories are "live" and then chugs through all of them writing that list, then when it's done, writes the index.shtml file. But when it's done, a minute boundary may have been crossed, and index.shtml may be pointing to an article .shtml that wasn't written.
For example, when this story went live, the task wrote, in order:
so for those 52 seconds index.shtml pointed to a 145209.shtml that hadn't been written.
I should probably get around to fixing this. But at this point it's kinda become one of those Slashdot things. It's barely a bug, it's almost like an easter egg to find a "nothing to see here." OK, I'm rationalizing. I should get around to fixing this. The index.pl and article.pl scripts need to accept a timestamp on the command line that mean "pretend it's this time."
The workaround is to create an account and log in [slashdot.org] so you get dynamic article.pl pages :)
Re:Nothing for you to see here. Please move along. (Score:5, Insightful)
Re:Nothing for you to see here. Please move along. (Score:5, Informative)
Often like apart from this morning? Because (ironically) we had database replication problems this morning. First time in months though.
If so please shoot me a private email and we can try to figure out what's going on.
Re:Nothing for you to see here. Please move along. (Score:5, Informative)
Re: (Score:2)
Re: (Score:1)
I see it very often immediately after a story goes leaves the ~mysterious future~ and hits the present.
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
No placeholders? (Score:1)
Re:No placeholders? (Score:5, Informative)
Please do download our code (and email us at security@slashcode.com if you find any bugs). We quote arguments in the approved fashion [sourceforge.net] before using them in a query string, and additionally we do regex whitelist-style filtering [sourceforge.net] on many commonly-used params (e.g. $form->{cid} is guaranteed to be numeric). Generally we're pretty good at this stuff. Which is not to say we never make mistakes [slashcode.com]...
Re:No placeholders? (Score:5, Insightful)
Re: (Score:1)
Re:No placeholders? (Score:5, Interesting)
Not that we are perfect, but we have a pretty good track record that, I think, speaks for itself. So I'll say that saying it is "wrong" is rather, uh, wrong.
Re: (Score:2)
If you don't use placeholders on ASCII text files, easy to sanitize anyway. But if you don't use placeholders on multi-byte encoded characters, then it is SQL injection and a security hole.
http://www.postgresql.org/docs/techdocs.50 [postgresql.org]
PostgreSQL
Re: (Score:2)
Re: (Score:2)
Re: (Score:3, Funny)
Re: (Score:2)
False. It's only a hole if it is a hole.
The problem is that you've increased the probability of a SQL injection attack from zero (which is what you'd get if you were using placeholders) to nonzero.
That's not true, unfortunately.
This talk of diligence is weird to me. This only comes up when you are writing a new call to the DB, or modifying one; so if you don't properly handle data that needs to be handled, how is that significantly different from, say, not using placeholders? Writing a function that bypasses them? That's possible too. It's not a zero probability at all. Both methods work, if the methods are used. If they are not used, they don't work.
Mechanisms only work if you use them.
In this case, you have a defense against it but it's not a bulletproof defense.
In fa
Re: (Score:2)
In fact, when we use the defenses, yes, they are as bulletproof as placeholders. And when your bulletproof defense is to use placeholders, but then you don't use them, well, you are wide open.
Would not the use of stored procedures make it bulletproof? Assuming your programmers don't have admin access to the database, you could provide them with a database user who has no access other than to make calls to certain stored procedures. This would enforce the security model as they otherwise couldn't do anything with the DB.
Re: (Score:2)
The best security model would be requiring writing everything in assembly, and then not having any developers to risk our security.
Re: (Score:2)
This talk of diligence is weird to me. This only comes up when you are writing a new call to the DB, or modifying one; so if you don't properly handle data that needs to be handled, how is that significantly different from, say, not using placeholders? Writing a function that bypasses them? That's possible too. It's not a zero probability at all. Both methods work, if the methods are used. If they are not used, they don't work.
Mechanisms only work if you use them.
The use of placeholders can be enforced in code, if incompletely.
Well that's the thing, it is easy to bypass them.
You can't prevent the programmer from bypassing the mechanisms you put in place
Exactly.
but that's not why they're there -- they're there to prevent mistakes.
They are there to prevent security holes. Which is no different from our mechanisms.
In a way, this "quoting versus placeholders" debate is similar to the debate on weak typing versus strong typing. The former is quicker and easier to use, but the latter reduces the chances of a mistake. And years of experience with many languages have shown me that the latter is more desirable and less costly in the long run.
And yet, we use Perl, which has no typing at all.
The bottom line here is that you are correct, except in that you think I am incorrect. We're both correct. There is no Right Way, except as defined by the people running the project. If we have an actual hole, obviously, that's Wrong. But that's not the case here: you just think we're preventing holes "wrongly,"
Re: (Score:2)
We all see the merits of each side, and each of you is defending a position that is entirely seperate from the other person's position.
Yeah, it's great to use the libs to do stuff, because the libs have been committee-vetted
Yeah, it's great to roll your own, because you don't have to worry about removing features
So give it up, the blood pressure increases are not worth the hassle of proving "I'm right"
Re: (Score:2)
but that's not why they're there -- they're there to prevent mistakes.
They are there to prevent security holes. Which is no different from our mechanisms.
No, I meant mechanisms which would enforce the use of placeholders to keep developers from inadvertently using unsafe queries. Those mechanisms could be bypassed intentionally by the programmer, and if they are then there's an increased risk of security holes because the security mechanism you want the programmer to use (placeholders) isn't being used.
Exactly. That's what I meant. If the mechanisms are not used, they don't work. Same as our mechanisms.
And yet, we use Perl, which has no typing at all.
Yes. And how many bugs have you encountered at runtime that occurred because of this, when a strongly-typed language would have caught them at compile time? My bet is that it's significantly more than zero.
Who cares? Our development time is much quicker, our lines of code much smaller, than a strongly typed language. I'll take a very rare problem that would have been caught by strong typing, as I am way ahead even still.
The Right Way is defined by the set of goals and their relative priority.
The Right Way is determined by whoever is doing the project. There is no obejctive "right way." That's something CS people often don't get.
Saying that there's no Right Way can be interpreted to mean that all solutions are equivalent
Yes, I cannot prevent incorrect interpreta
Re: (Score:2)
Re: (Score:2)
And who cares about how big or small your lines of code are? What matters is how easy the code is to write and to understand (you want both). Perl is generally horrible in the latter regard (it's possible to write easily-read Perl but that requires inhuman amounts of discipline to do consistently)
Ah, here we go. A language bigot. Someone who lacks proficiency in a language sufficient to judge it with any rationality, and yet condemns it despite his ignorance.
What is true, likely, is that YOU do not understand written Perl. Written Perl is very easy for me to understand. I write, and read, Perl every day. It doesn't take anything "inhuman." It only takes competent programmers who know Perl. You are not one of those, more likely because
Re: (Score:2)
This is pretty laughable considering that I've used Perl for various things for some 17 years, with about 10 of them involving intense development at times. I suppose, then, that you might regard me as incompetent, but I assure you it won't be for lack of familiarity with the language (that said, I've not done any serious Perl for about the last 5 years).
I do not believe you for a second. No one truly competent in Perl believes it is hard to write readable Perl.
I am not bothering to read the rest of your post.
Re: (Score:2)
I said it requires a great deal of discipline to do consistently.
And that is a clearly false statement. I, and many other people, do it all the time, without any significant exertion.
This is twice that you've refused to read the rest of what I've said, apparently because I don't happen to hold your favorite language with the same high regard that you do.
False. It is because you were presenting your (uneducated) opinion about Perl as fact. Saying you don't prefer Perl is fine. Making categorical statements about Perl that simply aren't true in the real world -- for people who are software engineers rather than computer scientists -- is boring.
From where I sit, that means it is you who is showing disregard for reason, because by doing so you implicitly assume that, because I fundamentally disagree with you about this one particular thing, the other things I have to say must have no merit.
False. I implicitly assume no such thing. I merely consider someone who presents their (une
Re: (Score:2)
I was really talking to the other poster, who is trying to slam ya'll repeatedly, for no apparent purpose, as he doesn't want to see a different viewpoint.
Cheerio
*or not, as appropriate to smoking preferences
Re: (Score:2)
Re: (Score:2)
Oh, and placeholders give you one more advantage: they're database independent. If a database supports placeholders at all, you're done. If you instead do quoting, you have to tailor your quoting function to the database engine you're using, because they're not all alike in that regard.
Actually, no, DBI.pm (and Slash's DB layer) handle that for you.
Of course there would be a zillion other things to do if we wanted Slash to be portable to other RDBMS software at this point...
you've increased the probability of a SQL injection attack from zero (which is what you'd get if you were using placeholders) to nonzero... you're relying on programmer diligence to avoid security holes when a mechanism exists to avoid those very same holes.
There's some places where you can't use placeholders or it's just silly or inefficient to. An IN clause, for example (we have some with a thousand or more IDs in an IN). Or if you want to compare a column's equality against a variable if the variable has a value but compare IS NULL if it's undef.
Whitelisting in
Re: (Score:2)
the CPU improvement you'd get from caching compiled statements isn't worth it?
Well, I haven't rewritten the whole application to benchmark it :) but I can say pretty definitively, not even close.
but I've seen one or two badly-written apps peg their DB servers' CPUs just compiling bazillions of queries, where placeholders would solve the problem
I think the badly-written part is the bazillions of queries, not the lack of placeholders. Except when you need to insert a bazillion rows, I can't think of any other case where code should have to issue the same query a large number of times in a row.
Slash doesn't really have anyplace where we do that kind of insert, that I can think of offhand. Actually there's a bazillion-row REPLACE
Re: (Score:2)
Instead of debating which one should be used, why don't just use both? More security, all the better.
Using placeholders is a form of dilligence too. What if some programmers (like the new guys) forget, some of the time, to use them for newly-written SQL statements? On the other hand, sanitizing request parameters e.g. uid, sid, *id will work on all new SQL statements that involve these parameters.
Re: (Score:2)
A programmer using $form->{uid} directly in SQL will never allow SQL injection, because the programmer will never get $form without $form->{uid} having already been sanitized.
And what if the programmer wants to use $form->{uid}* in some other place than an SQL query? Does he have to unescape it himself, or will he catch the PHP disease of spewing backslashes all over the page?
It is not escaped. So, no. :-)
Or some other form value that can legitimately contain arbitrary text, if uid isn't one of those.
Yeah, we don't escape it, we sanitize it. Most of such data is integer data, or simple character data. Arbitrary text is handled by an API that escapes it on insertion/updating, and in the case of selects, we handle it case-by-case. It works for us.
Re: (Score:2)
$form->{sid} is sanitized, not escaped. It's guaranteed to match \d{2}/\d{2}/\d{2}/\d{3,8}|\d{1,8} but its value in perl is its actual value. When you pass it to the DB you have to escape it.
Re: (Score:2)
The major benefit of placeholders is not speed
Oh, there are speed benefits, but you have to use a statement handle (which they also do not use):
Note: you should always check for DBI errors but I did not for the sake of simplicity.
What that code essentially does is reuse the same statement handle t
Re:No placeholders? (Score:5, Interesting)
Nah. I've submitted bugs to slashcode at sourceforge in the past. Actual bugs. They are always closed with snarky remarks about how it's not a bug. You should add a status "Will Not Fix" or "Cannot Fix". At least admit where your site is broken and maybe give a little explanation as to why.
The most obvious of these is how nested mode is horribly broken in stories with a lot of comments or a comment that receives a large number of replies such that it is beyond the comments per page threshold. If you try to go to the next page, you get the same page of comments. Same with the next page and the next. A side effect is that many comments are completely lost in this mode. This has been a bug for many years.
Code defensively? What's that? (Score:1)
Sadly, where I work, they just hope for the best and throw more hardware at the problem. We're running a new site, hosted off-site, which is killing our network bandwidth. Not my choice. I just shake my head.
Re: (Score:2)
- It's cheaper because your datacenter buys energy and connectivity wholesale
- The connectivity gives you lots more headroom when the going gets tough
So... what traffic is passing between the datacenter and your company?Re: (Score:2)
Error descriptions (Score:5, Interesting)
Re:Error descriptions (Score:5, Funny)
Re:Error descriptions (Score:5, Interesting)
I mentioned this phenomenon here [slashdot.org] -- but ironically, this morning the databases have been hallucinating (sigh)
Placeholders (Score:5, Interesting)
I guess speed might be one consideration. Generally I like to use place holders because it adds simplicity when passing some types of queries. If there's one thing I've seen a problem with, it's failing to properly sanitize incoming information that is passed to the database. A LOT of php code out there is rather easy to blow a hole through due to this. It also simplifies a lot of junk I'd rather not deal with like quoting and such. In either case you're an idiot if you don't sanitize everything first anyway, but my mantra is safety first. I actually loath doing many applications where I can't use them (like the ruby database libraries).
Well that's my take anyway. There's some rather nice code in slash that taut me some better methods in perl, and I'd say you guys are way above my level.
Re:Placeholders (Score:5, Insightful)
Personally, I much prefer placeholders / bind variables. They do help with sanitizing the data, but also ( for databases which have the feature ) can really reduce CPU utilization ( since the "same" statement isn't reparsed over and over again just because the variables changed ).
OS? (Score:5, Funny)
They don't use Windows Server, duh (Score:1)
Re: (Score:2)
Personally, I can't wait for that. It will really improve the looks of the site (+1 for the eye-candy). I mean it is Areo for websites (took them long enough). I just can't wait till M$ releases SQL-FS so they can ditch MySQL. Probably shouldn't be saying this, but its been rumored that they are going to do a total re-write in F# so that everyone (even Joe Sixpack
https and login (Score:4, Interesting)
Re:https and login (Score:5, Informative)
Re: (Score:2)
Again, thanks!
Re: (Score:2)
Re:https and login (Score:5, Insightful)
Re:https and login (Score:5, Insightful)
i've experimented with a way to fix this by overloading document.write, regexing for script tags and replacing the source HTTP with HTTPS, but i'm hesitant to make it live.
Re:https and login - Try using relative URLs (Score:3, Informative)
From the article:
If this reference appears in an HTTPS page, the mixed content warning will appear. How to craft a reference that works for both? The answer is again relative URLs, but using a more obscure syntax:
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
My number one complaint (Score:5, Insightful)
Yes, it's a nice perk, but a even nicer perk would be to let everybody at least login through HTTPS. Weren't we bashing companies earlier for not using SSL by default for logins?
Is this the place for complaints/suggestions? (Score:5, Insightful)
- separating +1 Funny into "+1 Funny-Raise Karma" and "+1 Funny-Karma Unaffected"
- combining -1 Flamebait with -1 Troll
- adding -1 Wrong (or -1 Misinformed), the opposite of "+1 Informative"
- combining +1 Interesting with +1 Insightful
(In fact, we can have diametrically opposed mods, like Informative/Misinformed, Underrated/Overrated, Insightful/Flamebait, Funny/Rude, etc.
And, yes, put in my vote for logging in via HTTPS.
Re:Is this the place for complaints/suggestions? (Score:5, Interesting)
- combining +1 Interesting with +1 Insightful
Asking for that change means that you don't understand what the word insightful means. Don't worry, you're not alone: quite a few moderators don't. But please allow those who do understand to use the words properly. There are enough of us around to still make the distinction work.
Re: (Score:3, Insightful)
Re: (Score:2)
Re: (Score:2)
Maybe "+1 Interesting" should be renamed "+1 Interesting, but not insightful". But if it's interesting and not insightful, then it's really merely entertaining. However, if we add a "+1 Entertaining", people would confuse it with "+1 Funny". To resolve that, we'd have to add "+1 Entertaining, but not funny".
Otherwise, people might just have to think abo
Re: (Score:2)
So if a comment has been marked as insightful by many, and interesting by fewer, then the output should be "(Score:5; Insightful, Interesting)".
Yes, indeed. Or if only a single world can be displayed, the one that was used most should be chosen. Right now, a true +5 insightful can be sort of "destroyed" by a single subsequent malicious "-1 troll". OK, the resulting "+4 troll" still is a sign that the post might be worth something, but the annotation looses the reason why. Malda, are you reading this?
your post deserves a (Score:4, Funny)
Re: (Score:1)
I find the posts marked "Troll" are the most interesting/entertaining ones. Anyone else do that?
Re: (Score:2)
Re: (Score:2)
I've found a few times that I've been moderated overrated for posting a contentious comment: in other words, some moderators misuse their points to give their opinion on the conclusion that is being stated. Re
Re: (Score:2)
Re: (Score:2)
It's much better for the discussion if you post a reply explaining WHY they are wrong or misinformed. This *is* a discussion site, after all, and how else do you expect the person to understand why you think they are wrong?
Re: (Score:2)
Re: (Score:2)
Moderate those replies up?
Moderation should ideally be minimally subjective.
Re: (Score:2)
Logging in via HTTPS, and using cookies for authentication persistence won't keep anyone with a sniffer from getting access to your account. From getting your password, yes, but not your account.
Slash update? (Score:1)
Also is there an up-to-date list of other sites running the code? The Slashcode sites list is sadly empty these days (which I hope doesn't mean there aren't any other sites that run Slash)
Re: (Score:3, Informative)
When did you drop the FrontPage Extensions? (Score:2, Interesting)
Bookmarked (Score:1)
Is This Some Kind of Joke? (Score:5, Funny)
Redefining .shtml? (Score:2, Interesting)
Wouldn't it have been better to choose an extension/term not already used, such as
Re:Redefining .shtml? (Score:5, Informative)
Wouldn't it have been better to choose an extension/term not already used, such as
X-FORWARDED-FOR headers (Score:2, Informative)
From wikipedia:
X-Forwarded-For: client1, proxy1, proxy2
Re: (Score:2)
Isn't the standard thing to do to append the source IP you (pound or whatever) see to the existing contents of the header (if the header exists) separated by commas? There should be no need for a separate header.
Technically speaking, the proxy should add another X-Forwarded-For header after existing ones, and it is recommended that it merges them all by separating them with commas, though it's not strictly required. It's up to the server to read all values in all X-Forwarded-For headers and use the last one for instance.
The comma is just used in HTTP to compact multiple headers into one, and has no special meaning for this particular header. There are headers which are not allowed to be repeated multiple times (eg
Mine eyes (Score:2)
When WE silly coders. It sounds goofy if you say when us decide, doesn't it?
ALTER TABLE blocking (Score:2)
Re: (Score:2)
NFS isn't scary. (Score:2)
What annoys me in particular about the latter is that sometimes these implementations evolve from fragile to usable or good enough or whatever. But some admins seem to be unable to comprehend that things improve and stick to some home grown process without looking to see what ha
Re: (Score:2)
Re: (Score:2)
Re: (Score:3, Funny)