Surviving Slashdotting with a Small Server 307
S.BartFarst writes "Our little departmental server has been slashdotted twice in the last year and survived! Implementation of a two-headed redundant hardware scheme using linux virtual server and backup and failover capabilities enhanced by the linux high-availability tools has produced a nifty low-cost solution. Gotta love those little white boxes!
(also having a university-supplied BIG PIPE doesn't hurt). More interesting is the documentation of the apparent exponentially decaying attention span of slashdotters. Anybody else observed similar phenomena?"
Third Time's a Charm? (Score:5, Funny)
Wait... is this a challenge?
Mike
Re:Third Time's a Charm? (Score:4, Funny)
Re:Third Time's a Charm? (Score:5, Interesting)
OT: It was interesting reading the logs. There are quite a few Linux users on here (but even more Windows users), and I saw lots of people using Mozilla, Opera, Safari, etc. Compare that to sites aimed at the average user where 95% of visitors are using IE or AOL and don't know that there's anything better out there.
Re:Third Time's a Charm? (Score:3, Funny)
Wow. Impressive. However, I would have hoped that by now the Slashbots would have learned not to click on image links [198.247.175.96] occuring in a random Slashdot comment...
Re:Third Time's a Charm? (Score:5, Insightful)
Oh, come on. Even my little old G3 iMac [utah.edu] is capable of handling quite a load from Slashdot and this site is serving up graphics intensive stuff. What you need to prevent a good Slashdotting is bandwidth that universities provide. T3 backbone connections are a wonderful thing.
Go ahead click all you want.
Re:Third Time's a Charm? (Score:3, Insightful)
Most slashdottings come from limited bandwidth or php/perl/asp/mysql/ etc being unable to handle it. Most dynamic content could be static content. (Most slashdot pages are static caches).
Re:Third Time's a Charm? (Score:5, Informative)
In 1998 I was working at a startup and we served Olympics news to Excite.com, which was one of the very largest sites on the web back then. Excite linked to pages on our server which were at some URL like olympics.excite.com that pointed to us.
What they didn't know was that we were serving all this off of ONE Sun Ultra 1 workstation on my desk (which was all we could afford at the time). I had it set up with Squid so everything was coming out of memory.
It worked fine, even at peak times everything popped up about as fast as any other page on Excite.
Moral: if you need speed and the page doesn't really have to be dynamic make it static or cache it.
Re:OT (Score:3, Interesting)
Re:Well its sorta dead (Score:5, Informative)
Pinging geology.heroy.smu.edu [129.119.223.84] with 32 bytes of data:
Reply from 129.119.223.84: bytes=32 time=58ms TTL=232
Reply from 129.119.223.84: bytes=32 time=61ms TTL=232
Reply from 129.119.223.84: bytes=32 time=60ms TTL=232
Reply from 129.119.223.84: bytes=32 time=56ms TTL=232
Reply from 129.119.223.84: bytes=32 time=58ms TTL=232
Reply from 129.119.223.84: bytes=32 time=74ms TTL=232
Reply from 129.119.223.84: bytes=32 time=67ms TTL=232
Ping statistics for 129.119.223.84:
Packets: Sent = 16, Received = 16, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 56ms, Maximum = 74ms, Average = 60ms
I mean, that's pretty damn smooth for a 30 minute old story. That's probably peak
Re:Well its sorta dead (Score:3, Funny)
Apparently not
Apparently... (Score:5, Funny)
Here are the testing materials (Score:5, Informative)
Re:Here are the testing materials (Score:4, Funny)
And download those MPEGs...
Weeee.... (Score:3, Funny)
Heh, well they're actually still kicking, oh well, it would've been so apropos.
Thou shall not (Score:5, Funny)
Re:Thou shall not (Score:3, Funny)
[Arthur, King of the Britons] One, Two, Five!
[Knight] Three, Sir!
[Arthur, King of the Britons] Three!
**HALLELUIAH**
*BooM*
Clever, clever (Score:4, Insightful)
Notice this comment was posted on a slow Sunday afternoon (EST). Very clever, because they know that /.'ers can't resist a challenge like that. Feel sorry for them on Monday morning though...
Re:Clever, clever (Score:5, Funny)
Monday morning!!?? You're kidding, right?
They have already noticed the "exponentially decaying attention span" [smu.edu] of Slashdotters.
By Monday morning this story and the site will be relegated to un-clicked graveyard of "Older Stuff"
Re:Clever, clever (Score:4, Funny)
Wouldn't you need an expensive switch? (Score:5, Interesting)
I was under the impression that a 20k fiber or 100mbs one that can dynamically shift traffic would be needed.
Re:Wouldn't you need an expensive switch? (Score:3, Insightful)
Each machine has its own independent 100 Mbit connection to the Gigabit SMU internet service,
I really didn't see 'cheap dual-channel switch' in there anywhere . . I DID see independent 100 Mbit connection though.
Re:Wouldn't you need an expensive switch? (Score:5, Interesting)
Very inexpensively, I might add. Beats having to buy the hardware for an, I dunno, . . .
Beowulf cluster?
Re:Wouldn't you need an expensive switch? (Score:4, Insightful)
On the stats link provided for a January slashdotting, the most bandwidth used in a day was a little under 7.4 GB. Assume that it was posted on /. at noon, so there were 12 hours to spread the big hit over: about 617 MB/hr. That's a little over 10 MB/Min, or 170 KB/sec. Even if we assume that the initial hit was double that, I can easily stream a 1000 KB/sec Divx movie over my 100 Mbps switched home LAN. The limiting factors here are the servers, routers and bandwidth to the Internet, not the local network connecting the servers.
well golly gee... (Score:5, Insightful)
well there you go... having a massive amount of bandwidth will allow you to survive a slashdotting. In most cases of slashdotting, I dont think the server was the bottleneck... its no problem for a server to dish out static pages... its the bandwidth, especially for serving pictures or videos....
Re:well golly gee... (Score:5, Informative)
simply use thttpd.
it has throttling built in so that the slashdotting wont take down the server. It simply stops serving images at anything but a really slow rate and eliminates multiple requests from same hosts... (A bitch for NAT'ed companies)
There are httpd servers out there that are much better than apache for handling insane loads.
Re:well golly gee... (Score:5, Interesting)
What's the point? Either way you're slashdotted.
Besides, I think in the case of server overload (as opposed to network overload), throttling will only exacerbate the problem by increasing the number of slow clients you have to deal with. This is the #1 bottleneck in web servers, the more clients you have, the longer it takes to deal with each one of them. Losts of processes to switch between, long arrays in an out of select(), etc.
Also, when a user doesn't get a page in his browser, what does he do? He clicks the link again and again.... even more connections to handle.
Really the only way to cure an overloaded server is to drop incoming SYNs. Any other measure is just pouring gasoline on the fire.
Asymmetric load balancing can help too (Score:3, Insightful)
Re:well golly gee... (Score:5, Insightful)
The problem here, is that the bandwidth bottleneck will make your server either (a) run out of processes/threads, (b) run out of ports/sockets, or (c) run out of memory from spawning all of the processes/threads to handle all of the stalled connections.
Once this happens, people no longer can connect to you, and you're toast. The crazy thing here is that this can happen even at 10mbit/sec if you're machine is configured well enough, and if the content you're serving is large enough (IE: image/media serving).
So cheers for these guys at keeping their bandwidth/server ratio high, I actually really like their architecture. But note that the greatest architecture in the world won't save you from a slashdotting if your server(s) are on a business dsl line.
Re:well golly gee... (Score:3, Interesting)
That's not true at all.
Configure your maximum number of users to something your server/bandwidth can handle, and then send everyone else an error page that says they need to wait. That error page can automatically try to re-load the main page every 1-5 minutes, until it is successful.
Even on your dial-up connection you could survive a slashdotting.
Re:well golly gee... (Score:3, Insightful)
Re:well golly gee... (Score:5, Insightful)
What we are talking about is sending a different, low-bandwidth page, which will reload the main page after a certain ammount of time.
Sure, you aren't immediately serving up the page to everyone who is requesting it, but people will only see a delay of a minute or so, rather than the server not serving up anyone, and crashing and burning for a day before anyone resets the thing. Or maybe just saturating the line, so nobody gets anything faster than 0.00001kbps, which is far far worse than having to leave the error page open in the background a couple minutes before it has a chance to load the main page.
Re:well golly gee... (Score:5, Interesting)
With static web pages, server power is rarely a problem, it's all about the pipes. However, if the pages are dynamically generated, and don't have a lot of caching, then you've got yourself a big problem.
So take, for example, loading a forum page in UltimateBB. AntiOnline handily tells you how many db requests it takes to create a page, and how long it took. This one over here says 61 requests and .3 seconds. Now, the poster claims to be peaking at ~37000 page views/hour, which is 10 hits per second. Now in that .3 seconds, where 61 database connections were established, there were another 3 requests coming in, making it an average of 240 database connections every .3 seconds. That's not an unreasonable number of connections, but what if your DB server can't keep up? What if, due to the load, the queries take 10x longer than usual? At that point, over .3 seconds, you get 240 connections, but only service 24 of them. Over the next .3 seconds, you get another 240 requests, but service only another 24, leaving you with 436 pending. After 30 seconds, you've serviced 2400 requests, but have another 21,600 pending. before too long, you're out of possible TCP ports.
There are ways to keep your servers from crapping out under heavy load. One is to buy a studly, fire-breathing DB server that can process requests faster than your web servers can send them. Another (cheaper) solution would be to pool and marshall your DB requests, being sure to remove requests from the queue when the remote user times out (either by clicking the stop button or running up against a built-in limit of their browser). This way, your site may get slow, but nothing will crash. A final method is to use enough caching on the web server that you pages are, essentially, static. This is, for instance, what Vignette does, which is why all the major news sites use it. This method combines the flexibility of database-backed CMS systems with the database load of static web pages.
So, essentially, there are many ways to let your database-backed web site survive a slashdoting, but embedding a bunch of PHP SQL queries against a locally-running installation of MySQL is not one of them. Unless you have a big honkin' cluster.
Re:well golly gee... (Score:5, Informative)
As soon as Vignette licensed it, they were able to claim C|Net as a "Big Customer", which of course was a fib. This let them sell loads of software to organizations that didn't know any better.
Moral of the story? Static pages are a great idea. If you can't do static easily, put a cache in front of your dynamic pages and decide how dynamic you really need them (Is 15 minutes delayed OK for you or is each page totally dynamic?). You don't need to spend $250K for Vignette to get this to work. In fact, you can do it with free software quite easily.
Web caching and optimization is a big topic but in general the more clever you can be the less money you have to spend on big iron.
Is it decaying attention span? (Score:5, Insightful)
Or is it where the article is at any given time? Top of front page gives lots of hits. As it drifts down, the hits slow as fewer read; to the sidebar, fewer but still substantial hits; then off to the specialty pages such as Science or Games, then only a few will read.
Of course, the only test would be to repost the article and see if there's the same number of hits... Nah, slashdot would never go for duplicate stories.
Re:Is it decaying attention span? (Score:2)
Also in the news: Carrot Top; Still Not Funny (Score:2, Insightful)
When the article is new, the rush is on for insightful comments that deal with commenting on elements of the referenced links. (Might as well, there aren't any comments beyond ascii pictures, and troll expiditionary forces.) They have their responses, which then triggers the volley of RTFA's, and now there are a number of posts, people don't have to RTFA so much and the thre
Re:Is it decaying attention span? (Score:2)
That hits graph (Score:4, Insightful)
Re:That hits graph (Score:2)
there is probably more of an explanation than just a short attention span. Wait, that is what you were talking about, right?
Re:That hits graph (Score:2)
Re:That hits graph (Score:3, Funny)
Re:That hits graph (Score:3, Informative)
However I don't know how to explain this phenomenon (Slashdot is good for making research and writing some thesis).
well... (Score:3, Funny)
and some exponentially growing forms of life in some beer cans...
does that count?
Need.... more.... band.... width..... (Score:2)
2Mb? It's nothing for a nice server.
A.D.D. crowd (Score:5, Funny)
Well, I was gonna reply, but I forgot what the post was about.
Re:A.D.D. crowd (Score:4, Funny)
Oh, oh, the dupes, the dupes. (Score:2, Funny)
I hope the dupe jokes that get posted are better than what I had here before I decided to just post this in hopes of drawing better jokes later in the thread. They weren't good! But when are they?
zerg (Score:3, Funny)
These SMU guys are obviously begging for it, I say we give them hell!
A slashdotting? (Score:3, Funny)
Re:A slashdotting? (Score:2)
while true
do
wget --delete-after -m -p http://www.geology.smu.edu
done
or something like that.
Re:A slashdotting? (Score:2)
If open a few more shells or vt's, and get like 4 or 12 or 50 wget's going with -m and -p, its a nice way to make the server thrash as well, assuming the site has considerably more content than ram (which it probably does, with all the media files i saw). I went from pulling 80k to 50k currently, so they have bandwidth and cpu to spare. need t
Nope... (Score:5, Funny)
Nope. In our jobs they make us do work.
Yeah but it is a small home page... (Score:3, Informative)
Found a big one one guys 113 Mbytes (Score:2)
Try a hand at downloading the above file from the site (113MB) and see what transfer rate you get?
Re:Found a big one one guys 113 Mbytes (Score:2)
Nice but your big pipes the real secret (Score:3, Informative)
While your setup may make you real safe from machine outages, the effects of a slashdotting are to flood your resources rather than break them. So your configuration gives you at best the performance of two machines instead of one - which you could also have achieved by just ramping up the CPU or memory.
[GASP!] You mean LOAD BALANCING HELPS! Stunning! (Score:3, Insightful)
Every medium to large website out there will be pleased to know that what they have been doing for the last 8 years is actually VALID, thanks guys!
I think the only reason this made it to the front page is the slashdot self-reference.
Eleventh Post (Score:3, Insightful)
I'll bet if you chart the data hour-by-hour, you'll see a sudden dropoff at the very moment the story scrolls off.
Bandwidth? (Score:2)
Don't let me crap all over your cutesy linux saves the day story, though.
Re:Bandwidth? (Score:2)
That and intelligent site design. Most of the pages on ticalc.org (most =~ all) are pregenerated.
--
lds
decay pattern (Score:3, Interesting)
It's also interesting that there was a second little bump about a week later. Anyone have any ideas why?
Re:decay pattern (Score:2)
Re:decay pattern (Score:2)
Re:decay pattern (Score:2)
You didn't get Slashdotted (Score:5, Insightful)
umm.. whaaa? (Score:2)
I think this is why Slashdot makes you wait 20 seconds before you can submit a comment. They figure this will weed out 80% of the responses as the people get bored of waiting for the 20 seconds to be up and they just leave.
How much bandwidth do you need to survive? (Score:3, Interesting)
Re:How much bandwidth do you need to survive? (Score:2)
They asked for it (Score:2)
Re:They asked for it (Score:2)
Pretty simple... (Score:5, Interesting)
The first time I learned my lesson. The server was on a T1 line that was 2/3 full already, and slashdot linked to a page full of large photos. That'll kill your link pretty quickly. Low-budget solution: sign up for a burstable web hosting account somewhere and just put all your large images there.
Later when we got some actual office space for the business [slimdevices.com], I moved the main server up to a colo [he.net] facility in fremont. All slahdottable content is hosted there on a fast server with a 100mbps ethernet link. Other oddball services that need their own machine are hosted from the other end of a point-to-point T1 line going directly back to the office from the colo.
So depending on your budget it's really not hard to set up your site to survive a slashdotting. If you don't have a lot of dough to spend but you want to run your own server for configurability/security reasons, just host the static stuff somewhere else. Or if you're serving enough to make it economical, get a colo account with a burstable link.
There's a widespread misconception here that slashdotting is caused by server overload. In reality this is almost never the case. It's caused by insufficient bandwith. This in turn may cause server overload because of too many slow clients being connected, but that is purely a secondary effect.
Re:Pretty simple... (Score:3, Funny)
It's more likely to be server overload here than anywhere else, don't you think?
Dear Slashdot,
I just installed a webserver on my Gameboy. Would you please turn it into a smoldering hole in the ground?
Thanks!
Also, with a little smarts from the server, you can prevent bandwidth overload. All you really need is to limit the number of concurrent users that can be connected to
Advice: add trailing slash to URLs if needed (Score:2, Informative)
BTW, using static pages also helps too. What is more, the "how to not survive" includes "generate content dynamically every time".
Serve up some dynamic pages (Score:2)
Slashdot effect analysis (Score:3, Interesting)
Also, there was an article on a hardware review site, if I remember correctly, where their approach to handling extreme load was discussed after their site was linked on Slashdot. Unfortunately, I can't find the article right now. Anyone around here who remembers?
Testing, testing... (Score:5, Funny)
ar405eng.exe [smu.edu] (5.41 MB)
from their webserver
5.41MB per slashdot reader should provide a test worth of such a fat pipe
Re:Testing, testing... (Score:2)
I know... (Score:3, Funny)
Huh? What?
Exponentially decaying attention span? (Score:3, Informative)
Attention Span (Score:5, Interesting)
When articles appear on the first page, they get attention, as they scroll to the bottom they get less, as they move to background pages they get significant;y less.
While I often look beyond the front page, I am less likely to delve into the articles or discussions there, since almost everything that needs to be said HAS been said by then.
I've carried on conversations with people regarding Slashdot articles long after the article appears. This can take place in journal entries or via e-mail where the discussion material can be easily kept as opposed to Slashdot comments which ultimately disappear anyway.
The fact that people don't continue to click on the original source URLs doesn't mean anything.
Mirrors (Score:3, Insightful)
Somebody's... (Score:3, Funny)
I bet these guys are glad... (Score:2)
More detail, please (Score:5, Interesting)
Congratulations on surviving /.ing. I have a few questions.
How were LVS and HA configured? With two systems, I can only guess that each was a real server (using the LVS terminology). Also both would be load balancers, with one being selected as active using HA.
How did using HA or LVS help surivive a /.ing? Were there failovers? How many? When? Why? If surviving /.ing consisted of a high rate of failovers then the hardware wasn't up to the job.
What is the "automated backup system?" Are you rsyncing the contents? From each other? From another system? Or does it refer to regular "tar" backups to tape?
Having separate UPSs is overkill, unless the one UPS could not handle the load of both systems.
Is there any dynamic content on the servers? Databases? How was keeping these synchronized handled?
Subscriber Slashdottings (Score:5, Interesting)
Attention span could be useful (Score:3, Interesting)
1) RSS feeds would get the update -last- or in some form of randomness.
2) Anonymous (no cookie) clients get the same treatment
3) People logged in get the article sooner but are also stretched out. An example:
a) If your UID is in the 25% of the oldest active users you get the article as soon as it is published (after going out to subscribers, who always get it first, another very mild reason to subscribe especially if you like to FP)
b) If your UID is in the 26-50% of the oldest active users you get the article 30 minutes after it is published.
c) If your UID is in the 51%-75% you get it 1 hour after it is published
d) If your UID is in the last block you get it 90 minutes after publishing.
e) If you are pulling from RSS or anonymously you get it 120 minutes after publishing.
This also gives a little treat to the folks who have been around the longest while not removing the benfit of subscribing.
Another example could work like the above but randomly change which order each block of UIDs will get the article (with RSS and Anon getting it last) if you wanted to not show preferrence to older users.
Increments could be adjusted
The only people this would affect negatively are FPers, SPAMboarders and people who have a cow-orker walk by and go "hey d00d, seen that new article yet?". No one else would probably even be aware of it unless they find it from another site that found it on
Re:Attention span could be useful (Score:3, Insightful)
Beware of side effects when you try to implement simple solutions like that...
Exponential decay (Score:5, Insightful)
Anything which follows a steady fractional diminishment will have a curve of y = ke**-ax, where k and a are constants. You see this basic equasion pop up all the time in physics, economics, statistics... etc. Why should server slashdotting be any different?
exponentially decaying attention span? (Score:5, Funny)
You get an exponentially decaying number of hits, yes, but how many of those are people doing exactly what I did and not staying, as opposed to those who stay a while because they find geology interesting?
The last time you were
Bring Up The Holy Slashdot! (Score:3, Funny)
AC's chanting: Pie Iesu domine, dona eis requiem.
Brother Neal: Armaments, chapter two, verse nine to twenty one.
Brother Neal: And Saint Cowboy Neal raised the Slashdot up high, saying, 'O Lord, bless this Thy Slashdot that, with it, Thou mayest slashdot Thine enemies to tiny bits in Thy mercy'. And the Lord did grin, and the AC's did feast upon first posts, trolls, GNAA posts, and...
Taco: Skip it a bit, Brother.
Brother Neal: And the Lord spake, saying, 'First shalt thou click on the holy link called Slashdot. Then, shalt thou count to three. No more. No less. Three shalt be the number thou shalt count, and the the number of the counting shall be three. Four shalt thou not count, nor neither count thou two, excepting that thou then proceed to three. Five is right out. Once the number three, being the third number, be reached, then, clickest thou holy Slashdot of OSDL towards thy server, who being naughty in My sight, shall snuff it.'
Taco: Amen
exponentially decaying attention (Score:4, Funny)
It is called Geek A.D.D.
I just don't understand it. Really. (Score:3, Interesting)
The guy is talking about bursts of 42,000 hits per day, and talking about it "bringing their system down". Now I could see that on Windows, but not on Linux.
Now, before you think I'm talking out of my posterior orifice, when my company was young and bright, we had a server built on a single 450 MHz Pentium 2, and 256 megs of RAM. It ran both Apache and PostgreSQL. Many of our pages were database-driven, which of course is a much larger load on the server than simple static pages.
That little machine would peak out at around 60,000 hits per day. At that point, it was slow enough to be self-limitting, but there was never any fear (or realization) of having the machine "brought down".
So, still "back in the day", I replaced it with a dual P3/650. That machine would peak out at around 100,000 hits/day (database driven!), without much problem at all. Also, as time goes on, and we develop new apps that make further use of our data, we tend to need more power to generate every page. Even still, we could crank out 40,000 hits per day on what would now be a relatively anemic server.
Now, with 7 front-end web servers and a dedicated DB machine, we crank out 5 million hits/day without problem. And even when our systems have been IMMENSELY overloaded from both legitimate and illegitimate traffic, the systems have still responded, and never once have I ever worried about a machine "going down" from the load. Failed hardware, perhaps, but not the load.
steve
Probably a dumb question (Score:3, Funny)
Re:Probably a dumb question (Score:4, Funny)
The server configuration is designed to handle the load, with multiple servers, load-balanced arrays, that sort of thing, whereas the people they link to are typically running on shared servers, or have only a single server.
Slashdot uses cached pages to avoid hitting the DB on every page load (mostly for the front pages), whereas smaller sites can get away with making a direct connection and doing more processor-intensive queries. Until they get linked by a site like Slashdot, anyway.
Slashdot's DB server is most likely of the 'fire-breathing god' variety, able to handle standard Slashdot traffic without too much difficulty. Smaller sites typically have the database server on the same machine as the webserver, and sometimes both are shared.
In general, it's all a matter of configuration. When you run a moderately successful small site, you're generally prepared for the amount of traffic you have, plus or minus 50%. Traffic generally grows slowly, so you have time to make adjustments when things start to get tight.
When Slashdot links your site, you get a huge influx of traffic to a site that is designed to handle a tiny fraction of that traffic. It leads to badness.
It's like trying to put an elephant into your freezer. If you're prepared for it, you have a big walk-in freezer. But most sites only need a small half-height fridge to keep their beer cold.
ha! (Score:3, Interesting)
The problem was, as it is for most people who get slashdotted, I didn't have a big enough pipe. Nothing to be done about that. I can't afford an OC3.
Bah (Score:3, Insightful)
1. Not enough bandwidth
2. Poorly configured web server
3. 300k images & 20MB mpeg/avi downloads (see #1)
4. Not enough RAM (1GB is generally enough)
I host about 50 domains for friends on my webserver (an Athlon 1.2Ghz w/ 1GB RAM) and have survived a simultaneous Slashdot and Fark link.
--Brent
Anti SCO T-Shirt [antitshirts.com]
The Slashdot worm. (Score:5, Funny)
while
Don't forget to fix the space in the URL.
Re:The Slashdot worm. (Score:5, Funny)
while true
do
wget --delete-after -m -p http://www.geology.smu.edu/
done
which has the advantage of not only hammering their bandwidth, but since it requests everything on the entire site, including images and many multiple mb files, then the server cant cache it all, and will have to read from the disk alot. also, since it deletes after downloading, it doesn't take up your precious drive space, lol.
but nice to know another sick fucker had the same basic idea.
I survived, with such a pipe. (Score:2)
Re:Exponential? (Score:3, Insightful)
Er... no. Logarithms increase.
Exponential decay is of the form:
y = e^-a x
Where a is positive. It's exponential in that a fixed increase in x multiplies y by a fixed number; it's just that this number is less than one.