Load Testing the New Server (Take 2) 41
Load testing the new boxes is going well: I've actually flushed out a pair of really bizarre bugs. If you're up for more load testing,
feel free to visit the new box but let me remind you that we're trying to simulate normal activity- just lots of it, so writing a script that tries to submit the same comment a few thousand times in a few minutes just isn't gonna help (for that matter, just reloading the same page doesn't help either really) And thanks to the guy from 209.80.X.X (I'm sure you know who you are) who must have been running something really smart 'cuz (except for the fact that it was all coming from one IP) it was pulling down like 20 different pages a second! Anyway, the new setup was handling ~3x the usual Slashdot load for awhile there. I wanna try to get to 4x and see how that goes. Then we just gotta change the DNS, and there shall be faster Slashdot for everyone!
new webserver configuration (Score:1)
webserver configuration along with some load
stats(maybe an mrtg traffic graph)?
I am curious to see what kind of load you are getting that is prompting you to goto a cluster
system.
Slashdot slashdotted? (Score:1)
I can't even reach the new site [209.207.224.40]! It responds to ping though.
Is Slashdot being slashdotted? :-)
BAD speed. (Score:1)
Hopefully this is just a result of over-agressive stress testing, but I haven't seen any speed benefits with the new site yet.
GO LIVE NOW!!!! (Score:1)
"I have no respect for a man who can only spell a word one way." - Mark Twain
good speed. (Score:1)
Gorfin
209.80.X.X (Score:1)
Bugs? (Score:1)
Yee-har! (Score:1)
Rob obviously ain't no cowboy!
He plans to do everything possible to do accurate load testing. That's why he said scripts posting the same shit over and over are no good.
I applaud this approach. There's no way you can accurately simulate a REAL production load, but you can come pretty damn close. That's what he's aiming for.
What he really needs to do is get a lively flamewar happening over there. Maybe he should post a coupla vi/Open Source/Gnome/RedHat/sendmail vs. emacs/Free Software/KDE/Debian/Qmail flamebait stories over there.
Now that's load testing!
Re:I'm first (Score:1)
:Will the new Slashdot automatically kill "First comment" postings?
No, just the posters.
s/poster/poseur/g
Oh, and btw...You're not.
A Question for the guy at 209.80.X.X (Score:1)
I'd REALLY appreciate hearing about it.
Really.
Not Sure, but.. (Score:1)
Re:209.80.X.X (Score:1)
simple script on a fat pipe to load pages from all
over the new site.
#!/bin/sh
While $I
do
lynx -dump http://
lynx -dump http://
lynx -dump http://
etc.. etc..
done
Of course you wouldnt use the same address over and over. You want to pick different articles and
links and such throughout the new site.
I am adding at least a page or two a second
by doing this..
Mike
Didn't Slashdot upgrade about 9 months ago? (Score:1)
see a plot of number of pages served per day as a function of time. If the old server only
lasted this long before getting beaten to a pulp, I can hardly wait to see what you'll need to upgrade to a year from now
Re:209.80.X.X (Score:1)
will@servername:/home/will/tmp/slash_test > ls
st_1 st_11 st_13 st_15 st_17 st_19 st_20 st_4 st_6 st_8
st_10 st_12 st_14 st_16 st_18 st_2 st_3 st_5 st_7 st_9
will@servername:/home/will/tmp/slash_test > ps ax | grep wget | wc -l
20
will@servername:/home/will/tmp/slash_test > foreach file ( st_* )
foreach? cd $file
foreach? wget -b -r -l 4 http://209.207.224.40
foreach? cd
foreach? end
Continuing in background.
Output will be written to `wget-log.2'.
Continuing in background.
So where are the stats? (Score:1)
It's about damn time!
Floris
How does the load balancing work? (Score:1)
4x Here we come. (Score:1)
Re:GO LIVE NOW!!!! (Score:1)
Re:I'm first (Score:2)
victim = isFirstPoster(getPoster());
hang(victim);
draw(victim);
quarter(victim);
little perl script to do do testing... (Score:1)
#!/usr/bin/perl
#$site = "209.207.224.40";
$site = "http://209.207.224.40/";
$num_of_forks = 0;
$MAXFORKS = 20;
$SIG{CHLD} = sub { wait; $num_of_forks--; };
$me = `basename $0`;
$me =~ s/\n//;
if ($ARGV[0] eq "-k") {
print "Killing all $me processes\n";
system "kill -9 `ps axuw | grep $me | grep -v -e -k | grep -v grep | aw
k '{print \$2}'` >
exit 0;
} elsif ($ARGV[0] eq "-h" || $ARGV[0] eq "-help" || $ARGV[0] eq "--help") {
print "usage: $0 -k|-h/-help/--help|\n";
print "\t-k kills a running version of this script\n";
print "\t-h|-help|--help gives this duh\n";
print "\tanything else should be a url\n";
exit 0;
} elsif ($ARGV[0] ne "") {
if ($ARGV[0] =~
print "Invalid option: $ARGV[0]\n";
print "Try $ARGV -h\n";
exit 1;
}
$site = $ARGV[0];
}
$getslash = "lynx -dump $site | grep -e $site | grep http |awk -Fhttp '{print \
"http\"\$2}'";
$iterations = 5;
### HERE IS WHERE WE REALLY GET GOING
print "getting urls from $site\n";
@urls = `$getslash`;
print "starting the good stuff\n";
print "Running $iterations iterations\n";
print "Making $MAXFORKS max connections at a time\n";
while ($iterations) {
$iterations--;
}
sub ProcessURLs
{
foreach $foo (@urls) {
unless ($child = fork) {
$parentpid = getppid();
&GetUrl($foo);
exit 0;
}
$num_of_forks++;
while ($num_of_forks > $MAXFORKS) { } #waiting
}
}
sub GetUrl
{
local($foo) = @_;
$foo =~ s/\n//;
($foo) = split(/
if ($foo ne "$site" || $foo ne "http://$site/") {
system "lynx -dump \"$foo\" >
print "retrieved $foo\n";
}
}
What kind of load balancer is being used? (Score:1)
There are boxes that perform this kind of function (we use 'em), like the Cisco Local Director (works like a modified bridge) and various layer 4 switches too. That's why I was asking. What is Slashdot using?
Re:209.80.X.X (Score:1)
angelos
Have you considered Akamai? (Score:1)
little perl script to do testing (Score:1)
#!/usr/bin/perl
#$site = "209.207.224.40";
$site = "http://209.207.224.40/";
$num_of_forks = 0;
$MAXFORKS = 20;
$SIG{CHLD} = sub { wait; $num_of_forks--; };
$me = `basename $0`;
$me =~ s/\n//;
if ($ARGV[0] eq "-k") {
print "Killing all $me processes\n";
system "kill -9 `ps axuw | grep $me | grep -v -e -k | grep -v grep | aw
k '{print \$2}'` >
exit 0;
} elsif ($ARGV[0] eq "-h" || $ARGV[0] eq "-help" || $ARGV[0] eq "--help") {
print "usage: $0 -k|-h/-help/--help|\n";
print "\t-k kills a running version of this script\n";
print "\t-h|-help|--help gives this duh\n";
print "\tanything else should be a url\n";
exit 0;
} elsif ($ARGV[0] ne "") {
if ($ARGV[0] =~
print "Invalid option: $ARGV[0]\n";
print "Try $ARGV -h\n";
exit 1;
}
$site = $ARGV[0];
}
$getslash = "lynx -dump $site | grep -e $site | grep http |awk -Fhttp '{print \
"http\"\$2}'";
$iterations = 5;
### HERE IS WHERE WE REALLY GET GOING
print "getting urls from $site\n";
@urls = `$getslash`;
print "starting the good stuff\n";
print "Running $iterations iterations\n";
print "Making $MAXFORKS max connections at a time\n";
while ($iterations) {
$iterations--;
}
sub ProcessURLs
{
foreach $foo (@urls) {
unless ($child = fork) {
$parentpid = getppid();
&GetUrl($foo);
exit 0;
}
$num_of_forks++;
while ($num_of_forks > $MAXFORKS) { } #waiting
}
}
sub GetUrl
{
local($foo) = @_;
$foo =~ s/\n//;
($foo) = split(/
if ($foo ne "$site" || $foo ne "http://$site/") {
system "lynx -dump \"$foo\" >
print "retrieved $foo\n";
}
}
Logging in to new server (Score:1)
Re:Bugs? It's Open Source... (Score:1)
Re:How does the load balancing work? (Score:1)
log log log (Score:1)
just wanted to let you know, and wonder if anyone else was having probs like that...
Flame Bait (Score:1)
Aside from that, the best an individual can do is use the new system in the same manner as the previous one and hope other people do likewise
Re:How does the load balancing work? (Score:1)
I, personally, use it for a hi-av proxy cluster. If one goes down, my users never know.
Comics:
Sluggy.com [sluggy.com] - Poing!
Re:Logging in to new server (Score:1)
Re:209.80.X.X (Score:1)
while true; do wget -r 209.207.224.40; done
and the other window, same dir
while true; do rm -rf 209.207.224.40; sleep 5; done
Simply to keep from filling my
/. is highly compressable: needs optimizing? (Score:1)
Re:Bugs? It's Open Source... (Score:1)
.Laz
--
My car is orange, my sig is not.