Slashdot Log In
Surviving Slashdotting with a Small Server
Posted by
michael
on Sun Aug 10, 2003 04:41 PM
from the soon-to-be-an-oreilly-book dept.
from the soon-to-be-an-oreilly-book dept.
S.BartFarst writes "Our little departmental server has been slashdotted twice in the last year and survived! Implementation of a two-headed redundant hardware scheme using linux virtual server and backup and failover capabilities enhanced by the linux high-availability tools has produced a nifty low-cost solution. Gotta love those little white boxes!
(also having a university-supplied BIG PIPE doesn't hurt). More interesting is the documentation of the apparent exponentially decaying attention span of slashdotters. Anybody else observed similar phenomena?"
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Third Time's a Charm? (Score:5, Funny)
Wait... is this a challenge?
Mike
Re:Third Time's a Charm? (Score:4, Funny)
Parent
Re:Third Time's a Charm? (Score:5, Interesting)
OT: It was interesting reading the logs. There are quite a few Linux users on here (but even more Windows users), and I saw lots of people using Mozilla, Opera, Safari, etc. Compare that to sites aimed at the average user where 95% of visitors are using IE or AOL and don't know that there's anything better out there.
Parent
Re:Third Time's a Charm? (Score:5, Insightful)
Oh, come on. Even my little old G3 iMac [utah.edu] is capable of handling quite a load from Slashdot and this site is serving up graphics intensive stuff. What you need to prevent a good Slashdotting is bandwidth that universities provide. T3 backbone connections are a wonderful thing.
Go ahead click all you want.
Parent
Re:Third Time's a Charm? (Score:5, Informative)
In 1998 I was working at a startup and we served Olympics news to Excite.com, which was one of the very largest sites on the web back then. Excite linked to pages on our server which were at some URL like olympics.excite.com that pointed to us.
What they didn't know was that we were serving all this off of ONE Sun Ultra 1 workstation on my desk (which was all we could afford at the time). I had it set up with Squid so everything was coming out of memory.
It worked fine, even at peak times everything popped up about as fast as any other page on Excite.
Moral: if you need speed and the page doesn't really have to be dynamic make it static or cache it.
Parent
Re:Well its sorta dead (Score:5, Informative)
Pinging geology.heroy.smu.edu [129.119.223.84] with 32 bytes of data:
Reply from 129.119.223.84: bytes=32 time=58ms TTL=232
Reply from 129.119.223.84: bytes=32 time=61ms TTL=232
Reply from 129.119.223.84: bytes=32 time=60ms TTL=232
Reply from 129.119.223.84: bytes=32 time=56ms TTL=232
Reply from 129.119.223.84: bytes=32 time=58ms TTL=232
Reply from 129.119.223.84: bytes=32 time=74ms TTL=232
Reply from 129.119.223.84: bytes=32 time=67ms TTL=232
Ping statistics for 129.119.223.84:
Packets: Sent = 16, Received = 16, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 56ms, Maximum = 74ms, Average = 60ms
I mean, that's pretty damn smooth for a 30 minute old story. That's probably peak
Parent
Apparently... (Score:5, Funny)
Here are the testing materials (Score:5, Informative)
Parent
Re:Here are the testing materials (Score:4, Funny)
And download those MPEGs...
Parent
Weeee.... (Score:3, Funny)
Heh, well they're actually still kicking, oh well, it would've been so apropos.
Thou shall not (Score:5, Funny)
Clever, clever (Score:4, Insightful)
Notice this comment was posted on a slow Sunday afternoon (EST). Very clever, because they know that /.'ers can't resist a challenge like that. Feel sorry for them on Monday morning though...
Re:Clever, clever (Score:5, Funny)
Monday morning!!?? You're kidding, right?
They have already noticed the "exponentially decaying attention span" [smu.edu] of Slashdotters.
By Monday morning this story and the site will be relegated to un-clicked graveyard of "Older Stuff"
Parent
Re:Clever, clever (Score:4, Funny)
Parent
Wouldn't you need an expensive switch? (Score:5, Interesting)
I was under the impression that a 20k fiber or 100mbs one that can dynamically shift traffic would be needed.
Re:Wouldn't you need an expensive switch? (Score:4, Insightful)
On the stats link provided for a January slashdotting, the most bandwidth used in a day was a little under 7.4 GB. Assume that it was posted on /. at noon, so there were 12 hours to spread the big hit over: about 617 MB/hr. That's a little over 10 MB/Min, or 170 KB/sec. Even if we assume that the initial hit was double that, I can easily stream a 1000 KB/sec Divx movie over my 100 Mbps switched home LAN. The limiting factors here are the servers, routers and bandwidth to the Internet, not the local network connecting the servers.
Parent
Re:Wouldn't you need an expensive switch? (Score:5, Interesting)
Very inexpensively, I might add. Beats having to buy the hardware for an, I dunno, . . .
Beowulf cluster?
Parent
well golly gee... (Score:5, Insightful)
well there you go... having a massive amount of bandwidth will allow you to survive a slashdotting. In most cases of slashdotting, I dont think the server was the bottleneck... its no problem for a server to dish out static pages... its the bandwidth, especially for serving pictures or videos....
Re:well golly gee... (Score:5, Informative)
simply use thttpd.
it has throttling built in so that the slashdotting wont take down the server. It simply stops serving images at anything but a really slow rate and eliminates multiple requests from same hosts... (A bitch for NAT'ed companies)
There are httpd servers out there that are much better than apache for handling insane loads.
Parent
Re:well golly gee... (Score:5, Interesting)
What's the point? Either way you're slashdotted.
Besides, I think in the case of server overload (as opposed to network overload), throttling will only exacerbate the problem by increasing the number of slow clients you have to deal with. This is the #1 bottleneck in web servers, the more clients you have, the longer it takes to deal with each one of them. Losts of processes to switch between, long arrays in an out of select(), etc.
Also, when a user doesn't get a page in his browser, what does he do? He clicks the link again and again.... even more connections to handle.
Really the only way to cure an overloaded server is to drop incoming SYNs. Any other measure is just pouring gasoline on the fire.
Parent
Re:well golly gee... (Score:5, Insightful)
The problem here, is that the bandwidth bottleneck will make your server either (a) run out of processes/threads, (b) run out of ports/sockets, or (c) run out of memory from spawning all of the processes/threads to handle all of the stalled connections.
Once this happens, people no longer can connect to you, and you're toast. The crazy thing here is that this can happen even at 10mbit/sec if you're machine is configured well enough, and if the content you're serving is large enough (IE: image/media serving).
So cheers for these guys at keeping their bandwidth/server ratio high, I actually really like their architecture. But note that the greatest architecture in the world won't save you from a slashdotting if your server(s) are on a business dsl line.
Parent
Re:well golly gee... (Score:5, Insightful)
What we are talking about is sending a different, low-bandwidth page, which will reload the main page after a certain ammount of time.
Sure, you aren't immediately serving up the page to everyone who is requesting it, but people will only see a delay of a minute or so, rather than the server not serving up anyone, and crashing and burning for a day before anyone resets the thing. Or maybe just saturating the line, so nobody gets anything faster than 0.00001kbps, which is far far worse than having to leave the error page open in the background a couple minutes before it has a chance to load the main page.
Parent
Re:well golly gee... (Score:5, Interesting)
With static web pages, server power is rarely a problem, it's all about the pipes. However, if the pages are dynamically generated, and don't have a lot of caching, then you've got yourself a big problem.
So take, for example, loading a forum page in UltimateBB. AntiOnline handily tells you how many db requests it takes to create a page, and how long it took. This one over here says 61 requests and .3 seconds. Now, the poster claims to be peaking at ~37000 page views/hour, which is 10 hits per second. Now in that .3 seconds, where 61 database connections were established, there were another 3 requests coming in, making it an average of 240 database connections every .3 seconds. That's not an unreasonable number of connections, but what if your DB server can't keep up? What if, due to the load, the queries take 10x longer than usual? At that point, over .3 seconds, you get 240 connections, but only service 24 of them. Over the next .3 seconds, you get another 240 requests, but service only another 24, leaving you with 436 pending. After 30 seconds, you've serviced 2400 requests, but have another 21,600 pending. before too long, you're out of possible TCP ports.
There are ways to keep your servers from crapping out under heavy load. One is to buy a studly, fire-breathing DB server that can process requests faster than your web servers can send them. Another (cheaper) solution would be to pool and marshall your DB requests, being sure to remove requests from the queue when the remote user times out (either by clicking the stop button or running up against a built-in limit of their browser). This way, your site may get slow, but nothing will crash. A final method is to use enough caching on the web server that you pages are, essentially, static. This is, for instance, what Vignette does, which is why all the major news sites use it. This method combines the flexibility of database-backed CMS systems with the database load of static web pages.
So, essentially, there are many ways to let your database-backed web site survive a slashdoting, but embedding a bunch of PHP SQL queries against a locally-running installation of MySQL is not one of them. Unless you have a big honkin' cluster.
Parent
Re:well golly gee... (Score:5, Informative)
As soon as Vignette licensed it, they were able to claim C|Net as a "Big Customer", which of course was a fib. This let them sell loads of software to organizations that didn't know any better.
Moral of the story? Static pages are a great idea. If you can't do static easily, put a cache in front of your dynamic pages and decide how dynamic you really need them (Is 15 minutes delayed OK for you or is each page totally dynamic?). You don't need to spend $250K for Vignette to get this to work. In fact, you can do it with free software quite easily.
Web caching and optimization is a big topic but in general the more clever you can be the less money you have to spend on big iron.
Parent
Is it decaying attention span? (Score:5, Insightful)
Or is it where the article is at any given time? Top of front page gives lots of hits. As it drifts down, the hits slow as fewer read; to the sidebar, fewer but still substantial hits; then off to the specialty pages such as Science or Games, then only a few will read.
Of course, the only test would be to repost the article and see if there's the same number of hits... Nah, slashdot would never go for duplicate stories.
That hits graph (Score:4, Insightful)
well... (Score:3, Funny)
and some exponentially growing forms of life in some beer cans...
does that count?
A.D.D. crowd (Score:5, Funny)
Well, I was gonna reply, but I forgot what the post was about.
Re:A.D.D. crowd (Score:4, Funny)
Parent
zerg (Score:3, Funny)
These SMU guys are obviously begging for it, I say we give them hell!
A slashdotting? (Score:3, Funny)
Nope... (Score:5, Funny)
Nope. In our jobs they make us do work.
Yeah but it is a small home page... (Score:3, Informative)
Nice but your big pipes the real secret (Score:3, Informative)
While your setup may make you real safe from machine outages, the effects of a slashdotting are to flood your resources rather than break them. So your configuration gives you at best the performance of two machines instead of one - which you could also have achieved by just ramping up the CPU or memory.
[GASP!] You mean LOAD BALANCING HELPS! Stunning! (Score:3, Insightful)
Every medium to large website out there will be pleased to know that what they have been doing for the last 8 years is actually VALID, thanks guys!
I think the only reason this made it to the front page is the slashdot self-reference.
Eleventh Post (Score:3, Insightful)
I'll bet if you chart the data hour-by-hour, you'll see a sudden dropoff at the very moment the story scrolls off.
decay pattern (Score:3, Interesting)
It's also interesting that there was a second little bump about a week later. Anyone have any ideas why?
You didn't get Slashdotted (Score:5, Insightful)
How much bandwidth do you need to survive? (Score:3, Interesting)
Pretty simple... (Score:5, Interesting)
The first time I learned my lesson. The server was on a T1 line that was 2/3 full already, and slashdot linked to a page full of large photos. That'll kill your link pretty quickly. Low-budget solution: sign up for a burstable web hosting account somewhere and just put all your large images there.
Later when we got some actual office space for the business [slimdevices.com], I moved the main server up to a colo [he.net] facility in fremont. All slahdottable content is hosted there on a fast server with a 100mbps ethernet link. Other oddball services that need their own machine are hosted from the other end of a point-to-point T1 line going directly back to the office from the colo.
So depending on your budget it's really not hard to set up your site to survive a slashdotting. If you don't have a lot of dough to spend but you want to run your own server for configurability/security reasons, just host the static stuff somewhere else. Or if you're serving enough to make it economical, get a colo account with a burstable link.
There's a widespread misconception here that slashdotting is caused by server overload. In reality this is almost never the case. It's caused by insufficient bandwith. This in turn may cause server overload because of too many slow clients being connected, but that is purely a secondary effect.
Testing, testing... (Score:5, Funny)
ar405eng.exe [smu.edu] (5.41 MB)
from their webserver
5.41MB per slashdot reader should provide a test worth of such a fat pipe
Attention Span (Score:5, Interesting)
When articles appear on the first page, they get attention, as they scroll to the bottom they get less, as they move to background pages they get significant;y less.
While I often look beyond the front page, I am less likely to delve into the articles or discussions there, since almost everything that needs to be said HAS been said by then.
I've carried on conversations with people regarding Slashdot articles long after the article appears. This can take place in journal entries or via e-mail where the discussion material can be easily kept as opposed to Slashdot comments which ultimately disappear anyway.
The fact that people don't continue to click on the original source URLs doesn't mean anything.
More detail, please (Score:5, Interesting)
Congratulations on surviving /.ing. I have a few questions.
How were LVS and HA configured? With two systems, I can only guess that each was a real server (using the LVS terminology). Also both would be load balancers, with one being selected as active using HA.
How did using HA or LVS help surivive a /.ing? Were there failovers? How many? When? Why? If surviving /.ing consisted of a high rate of failovers then the hardware wasn't up to the job.
What is the "automated backup system?" Are you rsyncing the contents? From each other? From another system? Or does it refer to regular "tar" backups to tape?
Having separate UPSs is overkill, unless the one UPS could not handle the load of both systems.
Is there any dynamic content on the servers? Databases? How was keeping these synchronized handled?
Subscriber Slashdottings (Score:5, Interesting)
Exponential decay (Score:5, Insightful)
Anything which follows a steady fractional diminishment will have a curve of y = ke**-ax, where k and a are constants. You see this basic equasion pop up all the time in physics, economics, statistics... etc. Why should server slashdotting be any different?
exponentially decaying attention span? (Score:5, Funny)
You get an exponentially decaying number of hits, yes, but how many of those are people doing exactly what I did and not staying, as opposed to those who stay a while because they find geology interesting?
The last time you were
exponentially decaying attention (Score:4, Funny)
It is called Geek A.D.D.
The Slashdot worm. (Score:5, Funny)
while
Don't forget to fix the space in the URL.
Parent
Re:The Slashdot worm. (Score:5, Funny)
while true
do
wget --delete-after -m -p http://www.geology.smu.edu/
done
which has the advantage of not only hammering their bandwidth, but since it requests everything on the entire site, including images and many multiple mb files, then the server cant cache it all, and will have to read from the disk alot. also, since it deletes after downloading, it doesn't take up your precious drive space, lol.
but nice to know another sick fucker had the same basic idea.
Parent
Re:Probably a dumb question (Score:4, Funny)
The server configuration is designed to handle the load, with multiple servers, load-balanced arrays, that sort of thing, whereas the people they link to are typically running on shared servers, or have only a single server.
Slashdot uses cached pages to avoid hitting the DB on every page load (mostly for the front pages), whereas smaller sites can get away with making a direct connection and doing more processor-intensive queries. Until they get linked by a site like Slashdot, anyway.
Slashdot's DB server is most likely of the 'fire-breathing god' variety, able to handle standard Slashdot traffic without too much difficulty. Smaller sites typically have the database server on the same machine as the webserver, and sometimes both are shared.
In general, it's all a matter of configuration. When you run a moderately successful small site, you're generally prepared for the amount of traffic you have, plus or minus 50%. Traffic generally grows slowly, so you have time to make adjustments when things start to get tight.
When Slashdot links your site, you get a huge influx of traffic to a site that is designed to handle a tiny fraction of that traffic. It leads to badness.
It's like trying to put an elephant into your freezer. If you're prepared for it, you have a big walk-in freezer. But most sites only need a small half-height fridge to keep their beer cold.
Parent