Third Largest Supercomputer... at Weta Digital

Designadrug writes "This story at the BBC details how the worlds third largest supercomputer (conditions apply) lives at Weta Digital - the company that provided CGI effects for The Lord of the Rings movies. The article also goes on to discuss the 500 TeraBytes of data generated for the films and how the epic Battle of Pelennor Fields almost defeated the film itself."

Re:What platform?

[OS24Ever]

It was the IBM eServer BladeCenter [ibm.com]

Looking forward ...

[arhar]

I'm really looking forward to Weta doing more amazing special effects work in future projects.. without them, LOTR movies would be, of course, still good, but probably wouldn't have the amazing success among the mainstream, non-geek audience.

Actually, I'm drooling right now thinking what could they do with "The Lion, the Witch and the Wardrobe" - depending on how much their stuff would be used in the movie.

Re:File system ?

[paitre]

Ummm, I've had DB files -much- larger than 2 GB on 32bit linux systems...3 years ago.
I don't find it at all implausible that they've got that much data lying around, at all.
I know of at -least- one research center with more data on disk than that. :)

Re:File system ?

[BiggestPOS]

I can have 9 gig files on NTFS in Windows XP routinely. All the linuxs and BSD are easily capable of this as well.

Top500

[FU_Fish]

*cough* Top500 List [top500.org] *cough*

Re:Post-project emotional crash

[LiquidCoooled]

Indeed he does.

He sounds like he's shitting himself at the prospect.

Sure, they CAN do hair - Monsters Inc was the proving ground for that, but when it comes down to it, the rendering difference between geometric wig units, and actual dynamic flowing furr is immense.
It is similar to the steps from real time game graphics to full on ray-tracing.

(Unless of course they have done some pretty damned nifty optimisations in the last few years)

Re:File system ?

[neuroklinik]

HFS+ has no theoretical limit on file size. It is limited only by the size of the volume (Max 16TB in Panther).

See Apple Knowledgebase article 25557 for more.

Re:3rd Largest?

[IWannaBeAnAC]

Correction, they currently come in 44th [top500.org]. This presumably doesn't include the extra 1000 processors mentioned in the article, doubling the performance would place it as 16 - although in practice for a distributed calculation you wouldn't get anything like double the performance by doubling the number of boxes.

Look at the difference in R_max (maximum achieved performance in a real LINPACK benchmark) and R_peak (theoretical maximum) - you can see that it is not a machine built for distributed calculations.

Re:This is just the beginning...

[dioscaido]

Yeah, famous made up quote.

http://www.wired.com/news/print/0,1294,1484,00.htm l [wired.com]

Still, your point about storage stands.

--

QUESTION: "I read in a newspaper that in l981 you said '640K of memory should be enough for anybody.' What did you mean when you said this?"

ANSWER: "I've said some stupid things and some wrong things, but not that. No one involved in computers would ever say that a certain amount of memory is enough for all time."

Gates goes on a bit about 16-bit computers and megabytes of logical address space, but the kid's question (will this boy never work at Microsoft?) clearly rankled the billionaire visionary.

"Meanwhile, I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again."

Silly quotations do have a way of floating like rumors.

Well, the truth starts here.

He never said it. No free software.

--

Not quite true.

[anzha]

Saying the WETA render farm is the third largest machine in the world based on the number of processors is wrong. Just check the latest top 500 list [top500.org] and a quick skim points out that Lawrence Livermore National Lab's ASCI White (8192) and ASCI Blue Pacific (5808), Lawrence Berkeley National Lab/NERSC's seaborg (6656), Sandia National Lab's ASCI Red (9632), and Los Alamos National Lab's ASCI Blue Mountain (6144) all have more processors as well as the two already listed.

Also interesting that WETA Digital is listed as #44 on the list too, huh? They only listed a Xeon cluster though with 1080 processors. (prolly not be the same machine, but...).

I love technical articles from the popular press about technical subjects. They do soooo much in depth research. I hope that they don't hurt themselves.

*Disgusted look*

Re:3rd Largest?

[galen]

The system is called Massive. During one of the early runs they noticed the guys in the back (on both sides) were wandering off. The problem was that they couldn't 'see' the action so they wandered around randomly looking for opponents. The effect was that it looked like they were running away. The problem was solved by giving the agents something similar to the ability to hear. Thus they could sense the action over greater distances and act accordingly.

I've seen this misrepresented so many times it's begun to bug me. I believe all the information I just dropped is available on the Extended Edition DVDs special features (probably FotR). If not, then I'm sure a Google search will turn it up.

Re:What platform?

[Chewie]

Damn, if I had just gone to Top500, I would have seen that it's running on IA32. Not sure whether they're 2-ways or 4-ways, but they're definitely Xeons.

Not Third Largest

[turm]

"...third largest supercomputer on the planet if you count the number of processors, 3300, it can call on..."

From the Top500 List for November 2003 [top500.org]:

Earth Simulator - 5120

LANL / ASCI Q - 8192

LLNL / ASCI White - 8192

NERSC / LBNL / Seaborg - 6656

Nice research, BBC.

It's not a supercomputer

[EnglishTim]

It's not a supercomputer, it's a render farm - there's a big difference. I'm sure there there are several VFX companies that have that number of processors in their renderfarm. The one I work for has somewhere between 2000 and 2500 processors in it, and I'm sure companies like PDI, ILM and Escape have more.

The way the render farms tend to work is this: you have a bunch of jobs, which get sent to the farm. You'll have a bit of software which allocates each job to a computer, or set of computers. Your job might be something like:

for frames 1 - 100
generate .rib file from scene for frame #
munge .rib
send .rib to PRMan

This would generate 300 jobs that will get allocated on the farm - obviously some of them are dependent on others being completed. However, each section of the job is a standard program on a fairly standard (normally) Linux install.

If all the computers were being combined into a supercomputer, they wouldn't all be running their own programs, they'd be combined into a huge 'virtual' computer, presumably with each processor running a virtual thread on that computer.

Re:Not just graphics

[Coryoth]

They also enlisted the help of a sole Kiwi inventor, who'd made a handheld laser modeling gun. Held in the hand, with no external supports or machinery (like a barcode scanner at the shops), it can quickly scan a model into the computer with great detail. You "paint" it with the laser, and it draws it on screen.

It's a small company in New Zealand. They do a lot of interestign work, including some interesting non-mesh based methods for 3D models. Their website is here: http://www.aranz.co.nz [aranz.co.nz]

Jedidiah.

Re:It's not a supercomputer

[malducin]

ILM had about 2500 processors on the renderfarm as recent as last year, could ne more now with the work for Ep. 3. Pixar also recently upgraded. At Rhythm and Hues (and many other places) they incorporte the workstations at night for the render jobs so it also depends how you count "render processors".

Re:Post-project emotional crash

[malducin]

It wasn't really an independent animatronic, but a costumed performer. The practical gorilla was made by Rick Baker, though the face was animatronic (radio controlled). Lots of those shots were done either with perspective tricks or just composited in.

Both Dream Quest Images and ILM did CG version of Joe with digital hair. ILM did several shots when Joe is running evading capture, when he crosses the freeway at night with the heli on top and the final shot of Joe running, among others. DQI's Cg Joes was featured in the ferris wheel sequence at the end, when it's in Hollywood (the fun mirror and sitting on the car) and several others.

There are even previous examples than that. One of the first movie CG hair examples is from Jumanji for which they won a Sci-Tech Academy Award later. Other early examples were some commercials by PDI (I believe at least a few months earlier than Jumanji), and Island of Dr. Moreau by Digital Domain.

Re:What processors?

[protohiro1]

IBM dual xeon blades and dell dual xeon racks. Very off the shelf.