Forgot your password?
typodupeerror
Slashdot.org News

Slashdot's Setup, Part 2- Software 151

Posted by CmdrTaco
from the its-all-just-ones-and-zeros dept.
Today we have Part 2 in our exciting 2 part series about the infrastructure that powers Slashdot. Last week Uriah told us all about the hardware powering the system. This week, Jamie McCarthy picks up the story and tells us about the software... from pound to memcached to mysql and more. Hit that link and read on.

The software side of Slashdot takes over at the point where our load balancers -- described in Friday's hardware story -- hand off your incoming HTTP request to our pound servers.

Pound is a reverse proxy, which means it doesn't service the request itself, it just chooses which web server to hand it off to. We run 6 pounds, one for HTTPS traffic and the other 5 for regular HTTP. (Didn't know we support HTTPS, did ya? It's one of the perks for subscribers: you get to read Slashdot on the same webhead that admins use, which is always going to be responsive even during a crush of traffic -- because if it isn't, Rob's going to breathe down our necks!)

The pounds send traffic to one of the 16 apaches on our 16 webheads -- 15 regular, and the 1 HTTPS. Now, pound itself is so undemanding that we run it side-by-side with the apaches. The HTTPS pound handles SSL itself, handing off a plaintext HTTP request to its machine's apache, so the apache it redirects traffic to doesn't need mod_ssl compiled in. One less headache! Of our other 15 webheads, 5 also run a pound, not to distribute load but just for redundancy.

(Trivia: pound normally adds an X-Forwarded-For header, which Slash::Apache substitutes for the (internal) IP of pound itself. But sometimes if you use a proxy on the internet to do something bad, it will send us an X-Forwarded-For header too, which we use to try to track abuse. So we patched pound to insert a special X-Forward-Pound header, so it doesn't overwrite what may come from an abuser's proxy.)

The other 15 webheads are segregated by type. This segregation is mostly what pound is for. We have 2 webheads for static (.shtml) requests, 4 for the dynamic homepage, 6 for dynamic comment-delivery pages (comments, article, pollBooth.pl), and 3 for all other dynamic scripts (ajax, tags, bookmarks, firehose). We segregate partly so that if there's a performance problem or a DDoS on a specific page, the rest of the site will remain functional. We're constantly changing the code and this sets up "performance firewalls" for when us silly coders decide to write infinite loops.

But we also segregate for efficiency reasons like httpd-level caching, and MaxClients tuning. Our webhead bottleneck is CPU, not RAM. We run MaxClients that might seem absurdly low (5-15 for dynamic webheads, 25 for static) but our philosophy is if we're not turning over requests quickly anyway, something's wrong, and stacking up more requests won't help the CPU chew through them any faster.

All the webheads run the same software, which they mount from a /usr/local exported by a read-only NFS machine. Everyone I've ever met outside of this company gives an involuntary shudder when NFS is mentioned, and yet we haven't had any problems since shortly after it was set up (2002-ish). I attribute this to a combination of our brilliant sysadmins and the fact that we only export read-only. The backend task that writes to /usr/local (to update index.shtml every minute, for example) runs on the NFS server itself.

The apaches are versions 1.3, because there's never been a reason for us to switch to 2.0. We compile in mod_perl, and lingerd to free up RAM during delivery, but the only other nonstandard module we use is mod_auth_useragent to keep unfriendly bots away. Slash does make extensive use of each phase of the request loop (largely so we can send our 403's to out-of-control bots using a minimum of resources, and so your page is fully on its way while we write to the logging DB).

Slash, of course, is the open-source perl code that runs Slashdot. If you're thinking of playing around with it, grab a recent copy from CVS: it's been years since we got around to a tarball release. The various scripts that handle web requests access the database through Slash's SQL API, implemented on top of DBD::mysql (now maintained, incidentally, by one of the original Slash 1.0 coders) and of course DBI.pm. The most interesting parts of this layer might be:

(a) We don't use Apache::DBI. We use connect_cached, but actually our main connection cache is the global objects that hold the connections. Some small chunks of data are so frequently used that we keep them around in those objects.

(b) We almost never use statement handles. We have eleven ways of doing a SELECT and the differences are mostly how we massage the results into the perl data structure they return.

(c) We don't use placeholders. Originally because DBD::mysql didn't take advantage of them, and now because we think any speed increase in a reasonably-optimized web app should be a trivial payoff for non-self-documenting argument order. Discuss!

(d) We built in replication support. A database object requested as a reader picks a random slave to read from for the duration of your HTTP request (or the backend task). We can weight them manually, and we have a task that reweights them automatically. (If we do something stupid and wedge a slave's replication thread, every Slash process, across 17 machines, starts throttling back its connections to that machine within 10 seconds. This was originally written to handle slave DBs getting bogged down by load, but with our new faster DBs, that just never happens, so if a slave falls behind, one of us probably typed something dumb at the mysql> prompt.)

(e) We bolted on memcached support. Why bolted-on? Because back when we first tried memcached, we got a huge performance boost by caching our three big data types (users, stories, comment text) and we're pretty sure additional caching would provide minimal benefit at this point. Memcached's main use is to get and set data objects, and Slash doesn't really bottleneck that way.

Slash 1.0 was written way back in early 2000 with decent support for get and set methods to abstract objects out of a database (getDescriptions, subclassed _wheresql) -- but over the years we've only used them a few times. Most data types that are candidates to be objectified either are processed in large numbers (like tags and comments), in ways that would be difficult to do efficiently by subclassing, or have complicated table structures and pre- and post-processing (like users) that would make any generic objectification code pretty complicated. So most data access is done through get and set methods written custom for each data type, or, just as often, through methods that perform one specific update or select.

Overall, we're pretty happy with the database side of things. Most tables are fairly well normalized, not fully but mostly, and we've found this improves performance in most cases. Even on a fairly large site like Slashdot, with modern hardware and a little thinking ahead, we're able to push code and schema changes live quickly. Thanks to running multiple-master replication, we can keep the site fully live even during blocking queries like ALTER TABLE. After changes go live, we can find performance problem spots and optimize (which usually means caching, caching, caching, and occasionally multi-pass log processing for things like detecting abuse and picking users out of a hat who get mod points).

In fact, I'll go further than "pretty happy." Writing a database-backed web site has changed dramatically over the past seven years. The database used to be the bottleneck: centralized, hard to expand, slow. Now even a cheap DB server can run a pretty big site if you code defensively, and thanks to Moore's Law, memcached, and improvements in open-source database software, that part of the scaling issue isn't really a problem until you're practically the size of eBay. It's an exciting time to be coding web applications.

This discussion has been archived. No new comments can be posted.

Slashdot's Setup, Part 2- Software

Comments Filter:
  • by Anonymous Coward on Friday October 26, 2007 @12:53PM (#21130633)
    Could we have a better run-down of what unpatched software is running on the server? It would really help. Thanks.
    • Re: (Score:2, Interesting)

      by Anonymous Coward
      How do you select using DBI without using statement handles? Do you mean you use a convenience method like selectall_arrayref on the database handle instead?
  • by corsec67 (627446) on Friday October 26, 2007 @12:54PM (#21130663) Homepage Journal
    Yay for segregated caching, where one machine gets data before the others do...
    • by jamie (78724) * Works for Slashdot <jamie@slashdot.org> on Friday October 26, 2007 @01:12PM (#21130985) Journal

      Heh. Actually that's a longstanding bug because of the way we write out .shtml files. We don't pick a timestamp and use it consistently on both index.shtml and the articles' .shtml files. Our backend task grabs a list of which stories are "live" and then chugs through all of them writing that list, then when it's done, writes the index.shtml file. But when it's done, a minute boundary may have been crossed, and index.shtml may be pointing to an article .shtml that wasn't written.

      For example, when this story went live, the task wrote, in order:

      Fri Oct 26 16:52:12 2007 [freshenup.pl] index.pl virtual_user=slashdot ssi=yes section='idle' bytes=19132
      Fri Oct 26 16:53:04 2007 [freshenup.pl] updated xxxxx meta:07/10/22/145209 (Slashdot's Setup, Part 2- Software)

      so for those 52 seconds index.shtml pointed to a 145209.shtml that hadn't been written.

      I should probably get around to fixing this. But at this point it's kinda become one of those Slashdot things. It's barely a bug, it's almost like an easter egg to find a "nothing to see here." OK, I'm rationalizing. I should get around to fixing this. The index.pl and article.pl scripts need to accept a timestamp on the command line that mean "pretend it's this time."

      The workaround is to create an account and log in [slashdot.org] so you get dynamic article.pl pages :)

  • by Anonymous Coward
    Seems like only a matter of time before someone could nail you with an sql injection attack. Maybe I will download your code...
    • Re:No placeholders? (Score:5, Informative)

      by jamie (78724) * Works for Slashdot <jamie@slashdot.org> on Friday October 26, 2007 @01:18PM (#21131105) Journal

      Please do download our code (and email us at security@slashcode.com if you find any bugs). We quote arguments in the approved fashion [sourceforge.net] before using them in a query string, and additionally we do regex whitelist-style filtering [sourceforge.net] on many commonly-used params (e.g. $form->{cid} is guaranteed to be numeric). Generally we're pretty good at this stuff. Which is not to say we never make mistakes [slashcode.com]...

      • by Spy Hunter (317220) on Friday October 26, 2007 @02:47PM (#21132441) Journal
        I think your justification for not using placeholders is rather, uh, wrong. I agree that the argument order thing can be an issue, but that's what *named* placeholders are for. The major benefit of placeholders is not speed, it's absolute resistance to SQL injection. You may be diligent in quoting, but standard software development wisdom is that it's always better to eliminate the possibility of a bug than rely on programmer checking all the time, due to Murphy's Law of course. Also, the admittedly small speed benefit of placeholders when used with precompiled statements is going to be on the DB side, not the web side, which can make a bigger difference (as the DB is harder to scale).
        • Does MySQL have named placeholders? I've only ever used it through PHP, and you need to use question marks if you wanna do that. I've never really bothered (not actually written anything I'd think to release yet in this arena) myself. A few days ago I and a friend had a go at writing a wrapper to handle named placeholders, but it was a pretty dirty hack (using preg_replace_callback and global variables ... uurrrgghh).
        • Re:No placeholders? (Score:5, Interesting)

          by pudge (3605) * Works for Slashdot <slashdot@NOspam.pudge.net> on Friday October 26, 2007 @04:17PM (#21133629) Homepage Journal

          I think your justification for not using placeholders is rather, uh, wrong. ... The major benefit of placeholders is ... absolute resistance to SQL injection.
          Yes, but there is more than one way to do it.

          You may be diligent in quoting, but standard software development wisdom is that it's always better to eliminate the possibility of a bug than rely on programmer checking all the time
          Sure. We do not rely on programmers to check all the time. Indeed, what jamie didn't mention (I think) is that most of the time, we sanitize user input before it ever gets to the programmer. A programmer using $form->{uid} directly in SQL will never allow SQL injection, because the programmer will never get $form without $form->{uid} having already been sanitized.

          Not that we are perfect, but we have a pretty good track record that, I think, speaks for itself. So I'll say that saying it is "wrong" is rather, uh, wrong. ;-)
          • by gnuman99 (746007)
            Security is about how things are done not how they turn out to be. This means you use placeholders such that then the database layer doesn't have to parse the data. If the database driver is stupid such that placeholders are just cosmetic, then well, no security gained.

            If you don't use placeholders on ASCII text files, easy to sanitize anyway. But if you don't use placeholders on multi-byte encoded characters, then it is SQL injection and a security hole.

            http://www.postgresql.org/docs/techdocs.50 [postgresql.org]

            PostgreSQL
            • by pudge (3605) * Works for Slashdot

              Security is about how things are done not how they turn out to be.
              Sure. And they way we do it works.

              This means you use placeholders ...
              Well, no. That is ONE way to do it.

              But if you don't use placeholders on multi-byte encoded characters, then it is SQL injection and a security hole.
              No. If you do not use placeholders on data that is potentially dangerous, THAT is a security hole.
        • by tknd (979052)

          The major benefit of placeholders is not speed

          Oh, there are speed benefits, but you have to use a statement handle (which they also do not use):

          my $sth = $dbh->prepare("insert into SOME_TABLE values (?, ?)");

          foreach $something (@bunch_of_somethings) {
          $sth->execute($something->{'key'}, $something->{'value'});
          }

          $dbh->commit;

          Note: you should always check for DBI errors but I did not for the sake of simplicity.

          What that code essentially does is reuse the same statement handle t

      • Re:No placeholders? (Score:5, Interesting)

        by Anonymous Crowhead (577505) on Friday October 26, 2007 @06:22PM (#21134985)
        Please do download our code (and email us at security@slashcode.com if you find any bugs)

        Nah. I've submitted bugs to slashcode at sourceforge in the past. Actual bugs. They are always closed with snarky remarks about how it's not a bug. You should add a status "Will Not Fix" or "Cannot Fix". At least admit where your site is broken and maybe give a little explanation as to why.

        The most obvious of these is how nested mode is horribly broken in stories with a lot of comments or a comment that receives a large number of replies such that it is beyond the comments per page threshold. If you try to go to the next page, you get the same page of comments. Same with the next page and the next. A side effect is that many comments are completely lost in this mode. This has been a bug for many years.
  • Sadly, where I work, they just hope for the best and throw more hardware at the problem. We're running a new site, hosted off-site, which is killing our network bandwidth. Not my choice. I just shake my head.

    • If it's hosted off-site, why is it killing your company's network bandwidth? The point of hosting off-site is that:
      • It's cheaper because your datacenter buys energy and connectivity wholesale
      • The connectivity gives you lots more headroom when the going gets tough
      So... what traffic is passing between the datacenter and your company?
  • Error descriptions (Score:5, Interesting)

    by 192939495969798999 (58312) <.info. .at. .devinmoore.com.> on Friday October 26, 2007 @01:06PM (#21130893) Homepage Journal
    I've always wondered, what's happening system-wise when we see "nothing for you to see here" vs. "page not found"?
  • Placeholders (Score:5, Interesting)

    by archen (447353) on Friday October 26, 2007 @01:09PM (#21130917)
    c) We don't use placeholders. Originally because DBD::mysql didn't take advantage of them, and now because we think any speed increase in a reasonably-optimized web app should be a trivial payoff for non-self-documenting argument order. Discuss!

    I guess speed might be one consideration. Generally I like to use place holders because it adds simplicity when passing some types of queries. If there's one thing I've seen a problem with, it's failing to properly sanitize incoming information that is passed to the database. A LOT of php code out there is rather easy to blow a hole through due to this. It also simplifies a lot of junk I'd rather not deal with like quoting and such. In either case you're an idiot if you don't sanitize everything first anyway, but my mantra is safety first. I actually loath doing many applications where I can't use them (like the ruby database libraries).

    Well that's my take anyway. There's some rather nice code in slash that taut me some better methods in perl, and I'd say you guys are way above my level.
    • Re:Placeholders (Score:5, Insightful)

      by grassy_knoll (412409) on Friday October 26, 2007 @02:07PM (#21131867) Homepage
      That caught my eye as well.

      Personally, I much prefer placeholders / bind variables. They do help with sanitizing the data, but also ( for databases which have the feature ) can really reduce CPU utilization ( since the "same" statement isn't reparsed over and over again just because the variables changed ).

  • OS? (Score:5, Funny)

    by mmullings (1142559) on Friday October 26, 2007 @01:15PM (#21131027)
    But what version of Windows Server are you running all this on? *duck*
    • OS/2. To The Maxxxxx.
    • by suggsjc (726146)
      They just got upgraded to Server 2003 - Web SP2 about 2 years ago. But from what I heard, they are evaluating Windows Server Vista Edition.

      Personally, I can't wait for that. It will really improve the looks of the site (+1 for the eye-candy). I mean it is Areo for websites (took them long enough). I just can't wait till M$ releases SQL-FS so they can ditch MySQL. Probably shouldn't be saying this, but its been rumored that they are going to do a total re-write in F# so that everyone (even Joe Sixpack
  • https and login (Score:4, Interesting)

    by Spy der Mann (805235) <spydermann DOT slashdot AT gmail DOT com> on Friday October 26, 2007 @01:15PM (#21131037) Homepage Journal
    One question. When someone logs in, is there a way to login through https? Because it doesn't matter if you get to https AFTER login - the login procedure can be sniffed anyway :-/
    • Re:https and login (Score:5, Informative)

      by jamie (78724) * Works for Slashdot <jamie@slashdot.org> on Friday October 26, 2007 @01:23PM (#21131187) Journal
      You have to be a subscriber, and go directly to https://slashdot.org/login.pl [slashdot.org].
    • by Shakrai (717556)

      Because it doesn't matter if you get to https AFTER login - the login procedure can be sniffed anyway :-/

      I'm all about security and paranoia, but do you really think there is somebody sitting on a router or network between you and /. with nothing better to do then steal your /. pw?

      Granted, I'll be using https the next time I get some spare cash to subscribe, now that I know it exists.... but not because I'm worried about somebody sniffing my pw.

      • Damn straight - if I hack somebody's slashpass, what are the chances I also have their email password or bank site password?
  • by OverlordQ (264228) on Friday October 26, 2007 @01:57PM (#21131699) Journal
    (Didn't know we support HTTPS, did ya? It's one of the perks for subscribers: you get to read Slashdot on the same webhead that admins use, which is always going to be responsive even during a crush of traffic -- because if it isn't, Rob's going to breathe down our necks!)

    Yes, it's a nice perk, but a even nicer perk would be to let everybody at least login through HTTPS. Weren't we bashing companies earlier for not using SSL by default for logins?
    • by KWTm (808824) on Friday October 26, 2007 @02:19PM (#21132043) Journal
      If this is the place for suggestions and complaints for Slashdot, may I put in my two cents? Sounds like people have been suggesting a number of new mods and changes to the old ones. Could this be revised? For example:

      - separating +1 Funny into "+1 Funny-Raise Karma" and "+1 Funny-Karma Unaffected"
      - combining -1 Flamebait with -1 Troll
      - adding -1 Wrong (or -1 Misinformed), the opposite of "+1 Informative"
      - combining +1 Interesting with +1 Insightful

      (In fact, we can have diametrically opposed mods, like Informative/Misinformed, Underrated/Overrated, Insightful/Flamebait, Funny/Rude, etc. ... but maybe we'll take it one step at a time.)

      And, yes, put in my vote for logging in via HTTPS.
      • by mce (509) * on Friday October 26, 2007 @02:31PM (#21132211) Homepage Journal

        - combining +1 Interesting with +1 Insightful

        Asking for that change means that you don't understand what the word insightful means. Don't worry, you're not alone: quite a few moderators don't. But please allow those who do understand to use the words properly. There are enough of us around to still make the distinction work.

        • Re: (Score:3, Insightful)

          by bennomatic (691188)
          But why can't we combine "+1 Crow" with "+1 Black Bird"? Every time I see a crow, it's a black bird. I don't see any problem with combining the two!

          • by mce (509) *
            Because as a reader I tend to value "+X insightful" over "+X interesting". Yes, the really insightful ones will likely also be interesting, but not the other way around. Look at it this way: if I'm a crow looking for a mate, I'm not interested in just any black bird, but I sure am interested in other crows (even those of the wrong sex, as they are unwanted competition :-) ).
            • :) That is exactly the point I was making. I understand the (not-so) subtle difference and was giving an illustration of where the confusion may lay.

              Maybe "+1 Interesting" should be renamed "+1 Interesting, but not insightful". But if it's interesting and not insightful, then it's really merely entertaining. However, if we add a "+1 Entertaining", people would confuse it with "+1 Funny". To resolve that, we'd have to add "+1 Entertaining, but not funny".

              Otherwise, people might just have to think abo

      • +0.5 Somewhat Useful Kvetching
      • by finnw (415539)

        combining -1 Flamebait with -1 Troll
        Please don't do that. It will spoil my +2 troll modifier.
        I find the posts marked "Troll" are the most interesting/entertaining ones. Anyone else do that?
        • by smash (1351)
          Yeah, i have troll/flamebait set to +2. it makes the site much more amusing :)
      • The idea behind moderation is that it's meant to be positive, so that the truly informative, insightful, interesting or funny posts rise out of the general hubub. The negative moderation, as far as I can see, is intended to increase the signal to noise ratio without unduly affecting genuine posters.

        I've found a few times that I've been moderated overrated for posting a contentious comment: in other words, some moderators misuse their points to give their opinion on the conclusion that is being stated. Re

      • And while you're at it, how about making it possible to undo mods done via the ajaxy-dropdown thing. I've accidentally modded a few things the wrong way (doh) and the only way to reverse it is to post a pointless comment to the article...
      • by Leebert (1694)

        - adding -1 Wrong (or -1 Misinformed), the opposite of "+1 Informative"


        It's much better for the discussion if you post a reply explaining WHY they are wrong or misinformed. This *is* a discussion site, after all, and how else do you expect the person to understand why you think they are wrong?
        • by beuges (613130)
          Because there are loads of +5 Insightful posts here which are quite simply misinformed. After the first 5 people post replies pointing out why the item is misinformed, what is left to do? The only other option really is to mod the offending post Redundant, which isn't entirely true. The other problem is that unless a post is explicitly marked as Misinformed, the uneducated masses who believe every positively rated comment is accurate will go on repeating that misinformed content as if it were the gospel tru
          • by Leebert (1694)

            After the first 5 people post replies pointing out why the item is misinformed, what is left to do?


            Moderate those replies up?

            Moderation should ideally be minimally subjective.
    • by gatekeep (122108)
      Yes, it's a nice perk, but a even nicer perk would be to let everybody at least login through HTTPS. Weren't we bashing companies earlier for not using SSL by default for logins?

      Logging in via HTTPS, and using cookies for authentication persistence won't keep anyone with a sniffer from getting access to your account. From getting your password, yes, but not your account.
  • Just curious if there is another release of Slash coming anytime soon?

    Also is there an up-to-date list of other sites running the code? The Slashcode sites list is sadly empty these days (which I hope doesn't mean there aren't any other sites that run Slash)
    • Re: (Score:3, Informative)

      by pudge (3605) * Works for Slashdot
      No, I doubt we will ever do another release of the code. We make it available on CVS for anyone to grab it, and use CVS tags to mark "releases." And we don't have an up-to-date list of Slash sites, but there are bunches.
  • It seemed an oddity to me that the old /. code base used FrontPage Extensions. Why did you use it and what did you replace it with when you dropped it?
  • Honestly, I never bookmark anything on slashdot because it's news (and so is in passing) - but I bookmarked this. This is a resource. I do perl application coding for the web, and while I don't pretend to be an expert in any form, this could be really useful if I ever need to scale up performance - thanks for posting this!
  • by Black-Man (198831) on Friday October 26, 2007 @03:54PM (#21133323)
    'cause I see no reference to IIS, SQL Server, .NET and I just read a Microsoft press release that says high-volume web portals can't provide high-availability service w/o them??

  • Redefining .shtml? (Score:2, Interesting)

    by VGPowerlord (621254)
    Last time I checked (which was like 10 years ago), .shtml stood for Server Side Includes [apache.org] (SSI) HTML, which are definitely not static.

    Wouldn't it have been better to choose an extension/term not already used, such as .htmls?
  • Isn't the standard thing to do to append the source IP you (pound or whatever) see to the existing contents of the header (if the header exists) separated by commas? There should be no need for a separate header. This works fine using F5 Load Balancers certainly, although I haven't used Pound myself.

    From wikipedia:

    X-Forwarded-For: client1, proxy1, proxy2
    • by wtarreau (324106)

      Isn't the standard thing to do to append the source IP you (pound or whatever) see to the existing contents of the header (if the header exists) separated by commas? There should be no need for a separate header.

      Technically speaking, the proxy should add another X-Forwarded-For header after existing ones, and it is recommended that it merges them all by separating them with commas, though it's not strictly required. It's up to the server to read all values in all X-Forwarded-For headers and use the last one for instance.

      The comma is just used in HTTP to compact multiple headers into one, and has no special meaning for this particular header. There are headers which are not allowed to be repeated multiple times (eg

  • for when us silly coders deci
    When WE silly coders. It sounds goofy if you say when us decide, doesn't it?
  • even during blocking queries like ALTER TABLE
    You must have something seriously borked up if ALTER TABLE is blocking your reads/writes.
  • Like you, I've seen a lot of people freak out when one mentions using NFS. I've come to the conclusion that either a) these people didn't know what they were doing or b) were using an incredibly broken/fragile NFS implementation.

    What annoys me in particular about the latter is that sometimes these implementations evolve from fragile to usable or good enough or whatever. But some admins seem to be unable to comprehend that things improve and stick to some home grown process without looking to see what ha

Kleeneness is next to Godelness.

Working...