Handling the Loads 890
I woke up and it seemed like a normal day. Around 8:30 I got to the office and made a pot of coffee. I hopped on IRC, started rummaging through the submissions bin, and of course, began reading my mail. Within minutes someone told me on IRC what had happened just moments after the impact of the first plane. Just a minute or 2 later, submissions started streaming into the bin. And at 9:12 a.m. Eastern Time, I made the decision to cancel Slashdot's normal daily coverage of "News for Nerds, Stuff that Matters," and instead focus on something more important then anything we had ever covered.
I couldn't get to CNN, and MSBNC loaded only enough to show me my first picture of the tragedy. I posted whatever facts we had: these were coming from random links over the net, and from Howard Stern who syndicates live from NY, even to my town. Over the next hour I updated the story as events happened. I updated when the towers collapsed. And the number of comments exploded as readers expressed their outrage, sadness, and confusion following the tragedy.
Not surprisingly, the load on Slashdot began to swell dramatically. Normally at 9:30 a.m., Slashdot is serving 18-20 pages a second. By 10 we were up to 30 and spiking to 40. This is when we started having problems.
At this point Jamie and Pudge were online and we started trying to sort out what we could do. The database crashed and Jamie went into action bringing it back up. I called Krow: he's on Western time, but he knows the DB best, and I had to wake him up. But worst of all, I had to tell him what had happened in New York. It was one of the strangest things I've ever done: it still hadn't settled in. I had seen a few grainy photos but I don't have a TV in my office and hadn't yet seen any of the footage. After I hung up the phone I almost broke down. It was the first time, but not the last.
The DB problem was a known bug and the decision was made to switch to the backup box. This machine was a replicated mirror of Slashdot, but running a newer version of MySQL. We hadn't switched the live box simply because it meant taking the site down for a few minutes. Well we were down anyway, and the box was a complete replica of the live DB, so we quickly moved.
At this point the DB stopped being a bottleneck, and we started to notice new rate limits on the performance of the 6 web servers themselves. Recently we fixed a glitch with Apache::SizeLimit: Functionally, it kills httpd processes that use more then a certain amount of memory, but the size limit was to low and processes were dying after serving just a few requests. This was complicated by the fact that the first story quickly swelled to more than a thousand comments ... we've tuned our caching to Slashdot's normal traffic: 5000-6000 comments a day, with stories having 200-500 comments. And this was definitely not the normal story. Our cache simply wasn't ready to handle this.
Our httpd processes cache a lot of data: this reduces hits to the database and just generally makes everything better. We turned down the number of httpd processes (From 60 on each machine, to 40) and increased the RAM that each process could use up (From 30 to 40 and later 45 megs) We also turned off reverse hostname lookups which we use for geotargetting ads: The time required to do the rdns is fine under normal load, but under huge loads we need that extra second to keep up with the primary job: spitting out pages as fast as possible.
This was around noon or so. I was keeping a close eye on the DB and we noticed a few queries that were taking a little too long. Jamie went in and switched our search from our own internal search, to hitting Google: Search is a somewhat expensive call on our end right now, and this was necessary just to make sure that we could keep up. We were serving 40-50 pages/second ... twice our usual peak loads of around "Just" 25 pages a second. I drove the 10 minutes to get home so I could watch CNN and keep up better with what was happening.
We trimmed a few minor functions out temporarily just to reduce the number of updates going to frequently read tables. But it was just not enough: The database was now beginning to be overworked and page views were slowing down. The homepage was full of discussions that were 3-4x the average size. The solution was to drop a few boxes from generating dynamic pages to serving static ones.
Let me explain: most people (around 60-70%) view the same content. They read the homepage and the 15 or so stories on the homepage. And they never mess with thresholds and filters and logins. In fact, when we have technical problems, we serve static pages. They don't require any database load, and the apache processes use very little memory. So for the next few hours, we ran with 4 of our boxes serving dynamic pages, and 2 serving static. This meant that 60-70% of people would never notice, and the others would only be affected when they tried to save something ... and then they would only notice if they hit a static box, which would happen only one in 3 times. It's not the ideal solution, but at this point we were serving 60-70 pages a second: 3x our usual traffic, and twice what we designed the system for. We got a lot of good data and found a lot of bottlenecks, so next time something that causes our traffic to triple, we'll be much more prepared.
At the end of the day we had served nearly 3 million pages -- almost twice our previous record of 1.6M, and far more then our daily average of 1.4M. During the peak hours, average page serving time slowed by just 2 seconds per page ... and over 8000 comments were posted in about 12 hours, and 15,000 in 48 hours.
On Wed. we started to put additional web servers into the pool, but that ended up not being necessary. We stayed dynamic and had no real problems on all 6 boxes all day. We peaked at around 35-40 pages/second. We served about 2 million pages. Thursday traffic loads were high, but relatively normal.
Summary So here is what we learned from the experience.
- We have great readers. I had only one single flame emailed to me in 24 hours, and countless notes of thanks and appreciation. We were all frazzled over here and your words of encouragement meant so much. You'll never know.
- Slashteam kicks butt. Jamie, Pudge, Krow, Yazz, Cliff, Michael, Jamie, Timothy, CowboyNeal, you guys all rocked. From collecting links to monitoring servers, to fixing bits of code in real time. It was good seeing the team function together so well ... I can't begin to describe the strangess of seeing 2 seperate discussions in our channel: one about keeping servers working, and another about bombs, terrorists, and war. But through it all these guys each did their part.
- Slash is getting really excellent. With tweaks that we learned from this, I think that our setup will soon be able to handle a quarter million pages an hour. In other words, it should handle 3x Slashdot's usual load, without any additional hardware. And with a more monstrous database, who knows how far it could scale.
- Watch out for Apache::SizeLimit if you are doing Caching.
- Writing and reading to the same innodb MySQL tables can be done since it does row-level locking. But as load increases, it can start being less then desirable.
- A layer of proxy is desirable so we could send static requests to a box tuned for static pages. For a long time now we've known that this was important, but its a tricky task. But it is super necessary for us to increase the size of caches in order to ease DB load and speed up page generation time ... but along with that we need to make sure that pages that don't use those caches don't hog precious apache forks that have them. Currently only images are served seperately, but anonymous homepages, xml, rdf, and many other pages could easily be handled by a stripped down process.
What happened on Tuesday was a terrible tragedy. I'm not a very emotional person but I still keep getting choked up when I see some new heart breaking photo, or a new camera angle, learn some new bit of heart breaking information, or read about something wonderful that somebody has done. This whole thing has shook me like nothing I can remember. But I'm proud of everyone involved with Slashdot for working together to keep a line of communication open for a lot of people during a crisis. I'm not kidding myself by thinking that what we did is as important as participating in the rescue effort, but I think our contribution was still important. And thanks to the countless readers who have written me over the last few days to thank us for providing them with what, for many, was their only source of news during this whole thing. And thanks to the whole team who made it happen. I'm proud of all of you.
The Community Was Served. (Score:5, Interesting)
Take into consideration that during the day at some point all major media web sites died.
Many people found Slashdot as their only source of updated information that was staying up.
This sentiment was echoed in pieces by Salon and Wired writers that mentioned Slashdot specifically as a site that had what people were looking for.
You should be proud and satisfied that what you have created did provide a needed service. Thanks, again.
did you forward this ... (Score:3, Interesting)
This is a great writeup. It covers all the things you could have done on your end to make /. fly. I guess the only prerequisite that most of us have trouble with are the Phat Pipes you folks can afford.
Cheers,
- RLJ
other sites... (Score:3, Interesting)
Thought y'all might be interested (the sites are generally eCommerce sites in Europe)
CNN's problems (Score:5, Interesting)
While the load was heavy, it wasn't anything Akamai wasn't prepared to handle.
Unfortunately, Akamai's co-founder was one of the passengers flying out of Boston on a hijacked flight Tuesday. I have friends who work at Akamai for whom he was not just a boss, but a friend.
Primary source (Score:2, Interesting)
Kudos to the slashdot team for their tireless efforts here...while work came to a halt everywhere, you guys managed to troubleshoot problems that would have given ordinary people fits on an average day. I am amazed at how quickly you adapted and improved, even though you no doubt would have preferred just to watch TV in saddened silence like the rest of us.
Once again ... (Score:2, Interesting)
I heard that the other sites weren't available, but I wouldn't have known this empirically since there were already so many quality posts, comments, and bits of information to sort through in the Slashdot forum.
Any long term Usenet denizen will know what I speak of when I refer to that rare, but addictive, experience that seems to be only able to be brought by these forums of such spatially disconnected people, joined only by common interests. When, seemingly all of a sudden, someone writes something so perfect, so funny, so outrageous, so wonderful, so _different_, or so incredibly informative, that you all of a sudden feel justified for all of the times that you wondered why you just kept coming back.
Slashdot did it again.
BTW, perhaps the moderators have been out in force, or maybe I'm just getting old and much more interested in politics than before, but it seems that the quality of posts in the last few days have been much more thoughtful and interesting.
Kudos to both the Slashteam and community.
Emergency Mode (Score:3, Interesting)
I'd also like to throw in my public note of thanks to everyone who kept the site up on Tuesday. We thirsted for answers, and you were there to provide, as always. Your work and dedication are wholly appreciated.
Re:A request (Score:4, Interesting)
I almost wonder if this is a subconscious attempt from people to rally support for white Christian America. It's easy for those who fit comfortably into the majority to see this as an Us vs. Them battle.
Those people, however, are hideous hypocrites; They are "Christian" in the same way that those flying the suicide planes were "Muslim"; i.e., only in that they happen to use that term to describe themselves. Any white Christian who starts seeing those of other ethnicities or religions as "Them" is not only a poor excuse for a Christian, but ought to be considered as bad as the terrorists themselves.
Re:The Community Was Served. (Score:3, Interesting)
Not looking for a db war (Score:4, Interesting)
Thank you, Slashdot (Score:2, Interesting)
This next part is slightly offtopic; I'd like to not be modded down because of it, but by the same token I'd like to not be modded up either. Not that that will sway anyone. :)
The feds found evidence linked to the hijackers that strongly suggests they used MS Flight Simulator to practice dry runs on buildings. What I want to know is, where are all those hypocrites who were pointing fingers and suing game makers for games like Doom and Quake after the Columbine incident? They seem strangely silent on this point. Could it be because MS Flight Sim is (all together now) JUST A GAME?
I know it's offtopic, but I felt it had to be said.
-Legion
Re:A request (Score:5, Interesting)
Speaking of religious fanatics, we have our own here in the US:
God Gave U.S. 'What We Deserve,' Falwell Says [washingtonpost.com]
Jerry Falwell and Pat Robertson blaming the events on liberals, feminists, etc. etc. etc.
Sick."Substantially less than major news sites?" (Score:2, Interesting)
"That is substantially less than major news sites. The Lycos news network -- of which Wired News is a part -- receives about 115 page views per second each day."
I can see how the entire Lycos news network can get that much traffic, but did any one site get hit that much? I haven't heard any other news site statistics.
Sorry, folks, but I have to say it... (Score:0, Interesting)
Folks, I don't give a RATS ASS how the back-end of a web site stayed up and running during the crush of viewers. I don't give a RATS ASS if you know someone who got out and saw bodies on the ground outside of the WTC. I don't give a RATS ASS if you saw stage X of the WTC collapse from 10 miles away.
STOP TRYING TO STEAL ATTENTION FROM THE REAL THING! In a time of crisis, I can always count on folks on the Internet trying to steal some of the thunder by trying to highlight how they did X over the other people, or how they know more people who were affected than others, or how they saw X happen live on TV. I DON'T CARE. You are affected just like the rest of us, but YOU ARE NOT DIRECTLY INVOLVED AND I DON'T CARE ABOUT YOUR "HOW IT ALL WENT DOWN" STORY!!
All I care about are the following:
1) The victims in NY, D.C., and Pennsylvania
2) The survivors
3) The people who lost loved ones
4) Getting the bastards who did this
5) Moving on with life
If I see another CT diatribe about the high-action excitement of keeping a news site live on the Internet, or another Jon Katz "feel sorry for me; I saw it all from miles away" story, I'm going to fscking go off! You people are LOW for trying to focus the spotlight on yourselves when there are so many people who are suffering who actually WERE part of it all.
You're sad... Just go away and let people focus on real issues.
Net traffic in general (Score:3, Interesting)
What this means is that people already knew where to go for the information, people know to look to certain sites for information. When MSNBC, NPR, CNN, and all the other sites failed I came here, and from here found a link to shoutcast which in turn led me to people who put up audio feeds from CNN - my only way to get the information while at work.
Thanks guys,
Travis
Re:A request (Score:5, Interesting)
Speaking as a white Christian...bingo. You just hit the nail on the head. In fact, it's that very attitude that allowed these terrorists to believe that what they were doing was somehow God's Will.
Christianity, Judaism, and Islam are all filled with references to people who, though they weren't Official Churchgoing Believers, represented God's will better than the average Believer. And an ongoing theme in both the Talmud and the Bible (I can't speak for the Koran, although I've been told Mohammed's teachings are very tolerant of other religions) is the failure of church leaders.
It's ironic. All of these religions which these misguided fundamentalist-whacko "leaders" (such as Osama Bin Laden and Jerry Falwell) supposedly follow condemn the most the Bin Ladens and Falwells of the world, who use God's Name to mislead people, or cause people to commit terrible atrocities.
This nearly happened before (Score:4, Interesting)
this is a little offtopic but It has something to do with the WTC towers event.
First I wish to say that I sympathize with all americans and families of those that died in the terrorist attack.
I'm a french citizen and wanted to say that this kind of event nearly took place in Paris in 1994. At the time a large airplane (can't remember if it was a boeing or airbus) was hijaacked by a group of algerian islamists.That plane had departed from Algeria. The terrorists requests on board the plane were very vague and unclear. The french air force was hesitating to shoot the plane down, which would have been a very difficult decision to take.
Luckily the plane which wasn't meant to fly to Paris didn't have enough fuel on board and had to land in Marseille (south of France) where the GIGN (~SWAT) took control of the plane. At the time the government didn't explain the reasons for this hijaacking and we never got any explanations from the terrorists themselves.
Later the interior minister Charles Pasqua admitted the plane was headed to Paris to crash on the Eiffel tower. Maybe he was wrong but this does seem suspiciously comparable to what has happened in New York 3 days ago.
Hey Slashdot Team.... (Score:3, Interesting)
Slashdot was my ONLY news source (Score:2, Interesting)
So I was at the airport in Seattle at 6:00am. on Tuesday Sept 11. 2001
I had just checked in and as usual logged on to the 802.11 network and started surfing for news.
I normally hit Yahoo first, but for some reason I could not get to yahoo.com. My first reaction was, "Oh maybe the nameserver on this WayPort thingy is messed up". So then I tried
The top article was the first one to be posted about a jet crashing into the WTC. By reading the comments, I was able to find an active link to the picture of the smoke pouring out of the first building.
By this time they are starting to board the plane, and NOONE else knows what has just happened. (The CNN channel on the monitors was playing pre-recorded stuff). Since I was not able to find out much more, I just boarded the plane. However I can still get a good signal from the plane, so as soon as I got to my seat I started reading
That is when I saw the second plane had hit the second tower. At this point I call my wife on the cell phone and she is crying and I can't hear very well (her best friend worked in the Amex building and is safe).
So I feel VERY awkward that I am the only one on this plane that knows about this. So I lean over to the guy sitting next to me and show him the pictures. He kinda looks dazed, and then asks, "So how are you surfing the web on a PLANE?" It just didn't sink in, he was much more interested in learning about the wireless technology than the incidents that were happening in NY.
(Oh they had long ago told us to shut off all electronics, but I wan't about to). Then the captian comes in over the intercom and tells us that they are going to reopen the doors and unload us because all US flights across the country recieved groundstop orders. At this point people start asking questions and word quickly spreads through the plane about info that I was able to gather from
I just want to thank EVERYONE at
BRAVO!
--Wayne
Re: hits or pages? (Score:3, Interesting)
- mark
Re:/.ers: Don't get too cocky... (Score:5, Interesting)
Props to Taco and team, not only for their hard work keeping the site up, but for this behind-the-scenes look at what it took to do so.
CNN was peaking at about an estimated 50,000 hits per second.
I also noticed after CNN came back up that they seemed to be in a sort of stripped-down static-only "combat mode". A talk with the guys behind CNN's site during the height of Tuesdays events would make for a great slashdot interview.
The moderation system made the difference (Score:2, Interesting)
Among all the useful slashdot features the moderation system seems to be the most important feature, and even if it is not balanced to deal with 2000+ articles, it still behaved pretty well, minor glitches aside.
Overall, slashdot and it's moderastion system helped me a least threefold to sustain this disaster:
It provided a huge low noise discussion forum by encouraging people to think what other readers might like, and by discouraging people to dump garbage into the system
It is very rewarding to post a comment and see it beeing moderated up; it was especially helpful in this case to see other people reading what I wrote, reducing the feeling of fear and powerlessness in this situation
The slashdot community, often critized for beeing somewhat narrow minded, and especially the moderators have clearly shown that they cover a large political and social spektrum, and few extreme lunatics write here, compared to other places on the internet.
Thank you all, the slashdot team, and everybody else who contributed to make the last days a bit easier to endure.
p.
Re:/.ers: Don't get too cocky... (Score:2, Interesting)
Our network of local news sites (http://www.ibsys.com [ibsys.com]) peaked at above 1.5M pages per hour - about 425 pages per second.
We got slow in the morning, but recovered by about noon - where we were serving pretty quickly.
Interestingly, our biggest problem is that we use Akamai extensively for image serving, and Akamai fell apart early in the day and took a while to recover.
Between pages and video, and our local bandwidth and Akamai/IBEAM video serving, we peaked at about 450Mb/s...
Has Slashdot's own search been removed for good? (Score:5, Interesting)
and this is what I got:
Slashdot | Pentium 4 Under Linux
... Under Linux, I would not buy a P4
out there selling G4 motherboards with standard form factors and
www.slashdot.org/articles/01/07/15/209215.shtml - 69k - Cached - Similar pages
Slashdot | Linux Intel Chipset Comparison
... it in march, and i run linux on it, and it performs
by Diabolus (troy
www.slashdot.org/articles/00/12/18/056248.shtml - 46k - Cached - Similar pages
Slashdot | AMD Athlon Multi-Processor Under Linux
... on several single-CPU motherboards; check your favourite vendor's
under linux on the following boards
www.slashdot.org/articles/01/07/12/1838238.shtm
Slashdot | Intel To Drop Rambus Exclusivity, Support SDRAM
... problems with the newest linux kernels - but widespread - well
motherboards and chipsets, but
www.slashdot.org/articles/01/07/26/1153225.shtm
That is just the first few but I looked through a number of them and I couldn't find the story I was looking for.
Re:Pray Or Meditate Or Whatever For President Bush (Score:2, Interesting)
Christianity is at its worst... (Score:5, Interesting)
I won't go into a long diatribe at the offensiveness of this mindset or this behavior, but rather reference it in order to point out that, as a genre of religion which is bent on conversion, i.e. selling their viewpoint to others, Christian sects tend to be obsessed with appearance as much as substance. Whether it is cloaked as "setting a good example to others," "representing your faith/church to others," or "demonstrating through actions what it is to be a good Christian," none of which are as blatent as the Mormon adage of "avoid the appearance of evil," the underlying message is clear: appearances are at least as important as substance. With a mindset like that, reinforced every sunday from one's spiritual leaders, is it any suprise that people who look even a little non-mainstream garner the reactions like you describe?
We should kick ass and eradicate our enemies. Not in the name of God, not in the name of some religion, but in the name our our country and our people, which have been attacked and shall be avenged. Keep church and state where they belong, separate, and obliterate the bastards who committed these atrocities last Tuesday in the name of our secular, democratic instutions, leaving each of us to pray, and to grieve, in our own fashion, according to our own beliefs. And never make the mistake that just because someone doesn't share your beliefs, ethnic background, or skin color that they are in any respect less capable of grieving than you.
no one will probably read this, but.... (Score:2, Interesting)
Re:Time for some highly unpopular opinion... (Score:4, Interesting)
I barely know where to begin when I read crap like this. The simple truth is that people hate us because we're the biggest kid on the block.
That is not entirely true. I simply don't believe it's possible to convince people to die terrorizing a country because they're the "biggest kid on the block." The hatred runs far far deeper than this and it must be resolved if there is any hope of this not happenning in the future.
Did you read the Guardian article? Have you looked into your country's foreign policies over the past 50 years? There are people who have very legitimate reasons to be upset at the U.S. because of the way it handles itself in foreign lands.
Well fuck you all very much, planet earth. We didn't ask to be the only superpower. We're not itching to feed your hungry or shelter your homeless or finance your economic devastations, but we're the ones you call on first when you need those things done.
Please go do some research before you post so you don't come off so totally ignorant of the reality.
Actually yes the U.S. does strive to be the only superpower. What did America do to Russia? What is it doing with China? I can't believe you actually think that nations don't strive to be the biggest and best. I believe that the U.S. is the only nation not paying its UN membership fees. Or behaving poorly in UN. Or trying to be the police of the world. The U.S. government does flaunt its power; that is what I believe got the United States into this mess in the first place!
You complain because american hegemony is destroying your cultures, then you go out and buy coca-cola and watch Friends on TV. You complain about our imperialism while ignoring the fact that Germany and Japan are our biggest competitors exactly BECAUSE we rebuilt them at OUR EXPENSE after we could have conquered them.
Actually I don't drink Coke and dislike Friends, but thanks for generalizing.
Your comments on WW2 are laughable at best. The American government sold arms (to both sides? I don't remember) when WW2 broke out. America didn't come to the aid of the other nations; it waited until it was hit before getting involved, instead profiting from the war. Whether this reserved attitude is good or bad I reserve judgement -- but don't think that for a second that America saved the world; if you read through your history you'll see that the Soviets and British played key roles in ending the war, as did America. It was a group effort.
We're damned if we do and damned if we don't, so don't give me any shit that we had it coming because of our policies. NO FUCKING POLICY WILL MAKE EVERYONE HAPPY!
You're right, America can't make everyone happy. Instead of meddling in the affairs of foreign nations by selling arms and supporting conflict, it should do more peacekeeping and put its own military into action. The "hands off" kind of "support" that the U.S. has done in the near past reminds me of a parent shouting at fighting kids to stop instead of getting their hands dirty and stopping the conflict themselves.
Or, better yet, let the nations battle amongst themselves without making profit from it! Maybe even by providing relief to the civilians!
As a citizen, you can stand up, take notice and get involved in your government and the politics of your government! What was voter turnout this past election, something like 46%? While Canada isn't much better in this regard, we don't have the problems that America has and our foreign policies seem to be less convoluted and more focussed on peace than profit.
I hope you realize that I did not say the American citizens had it coming. Killing thousands of civilians is horrible and will be met with vengeance. Terrorist attacks like this are beyond words. No war was declared, no "plain-to-see" warnings... this is terrorism of the worst sort.
But to think for a second that America is right and innocent and that they didn't have an attack coming is just plain stupid, ignorant and totally unbelievable. I am sorry it happened and I do mourn the loss of civilian life but I am also not blinded by my patriotism like I believe you are.
Re:A request (Score:4, Interesting)
Re:A request (Score:2, Interesting)
As someone who doesn't pray in the religious sense, (as I guess you could say I'm agnostic, though I never liked the term because it implies I'm not sure, when rather, I figure if there is something there, fine, if there isn't, fine - I'll keep living the way I always have) I've always taken "pray" to mean "think and hope for the best," rather than asking a diety for the best. Because praying varies among different religions, I feel that hoping for the best and thinking positive is similiar to praying for us non-religous types.
Slashdot kicks ass: (Score:2, Interesting)
Well, everyone where I work (well, now used to work [uga.edu] as of today) was feeling so out of it because there is literally no cable access in the entire building(s).
People wanted *information* and the tradegy also opened a few eyes with us being asked (the techs).
My response was "slashdot.org is the best place to go...and don't forget to read the comments section which will have more valuable information by support from other techs and people out there".
Heck, even today one of the Directors asked where to go for even more info...and once again, I recommended slashdot.org with the "down and dirty" explanation 'this is a community of "nerds/techs" that kept the information flowing by giving up their time and resources to keep the rest of the world up to date.'
Some of my coworkers asked why, to which I would smile and say "what does the "I" in IT stand for, but "Information".
Moose.
Oh, and over at arstechnica.com's "lounge" section there was a comment that deserves to be on
"We seem to project to the rest of the world that we are 'fat, lazy, and kind of goofy', but what most of those that dislike us forget is: when you piss us off we tend to fight like cornered badgers (Tooth and Nail or the movie "tombstone" the "I'm coming and I'm bringing hell with me for those that don't get the badger reference).
I *am* Interesting and I *am* Insightful and IF ppl would read my FSCKing comments they'd see that. (me ranting after several post, esp after losing a +4 to database corruption...now..who the hell cares? It ain't that important in the grand scheme of things.
Amazing what happens when you actually get on board the "cluetrain".
Heh, I've babbled enuf, thanks to everyone + world.
Possible project? (Score:3, Interesting)
Wouldn't that be a nice open content project for the open source software community? Everybody contributes, many can profit from this, everybody's happy (except for the guys selling this).
Any comments on how one could set this up?