600 PowerMacs Make One DVD

vaporland writes "NYTimes.com has this story about using a network of 600 PowerMac G5's to scan original movie negatives at 4000 lines per inch and create high-resolution digital recreations of classic movies."

Re:What is the point of scanning at such a high re

[Anonymous Coward]

It makes it easier to work with when they are cleaning up and removing artifacts later on.

google link (no registration

[lith2k]

click! [nytimes.com]

Re:How much visual difference will there be...

[Snuffub]

As the article clearly points out the big difference isnt on DVDs but rather the ability to archive a digital master in such a high quality format. So 500 years down the road when we're all watching movies at 4000p instead of 480i they dont have to go back to the original film which will undoubtedly be nearly destroyed.

Re:What is the point of scanning at such a high re

[Morgahastu]

the poster got it right wrong. The film isn't scanned 4000 times per square inch, the entire film is scanned at 4000 LINES of resolution.

Current HDTV displays 1080 lines interlaced.

Re:What is the point of scanning at such a high re

[saddino]

at 4000 dpi, the film grains will be dozens of pixels in diameter

Doubtful, given that a standard 35mm print is only 24 mm tall (barely an inch).

Common misconception

[Digitus1337]

Most people are confusing 4000 DPI (dots per inch) with 4000 Lines Per Inch. A line could be any length, as the inch is only a measurement one way; this is one of those techniques for making something seem bigger and/or better than it really is (think weight loss commercials).

Re:What is the point of scanning at such a high re

[Anonymous Coward]

Film grain represents the physical resolution of the film, it's not dust or something which can be removed by duplicating adjacent pixels. Moreover film grain is aestethically much nicer than any rounding and blurring the kind of filter you are proposing would produce.

Re:What are the Macs for?

[way2trivial]

cleanup-- "He then processed the images with his film-restoration software, which he'd programmed onto some Macintosh G4 computers. (The effort took months, as the faster G5's weren't out yet.) The processed picture was clearer, sharper and more detailed still. He could see every divot on the turf. What had once looked like a smudge in the background was now recognizable as a boat on the lake."

Re:What are the Macs for?

[ToddML]

If you RTFA, then how did you miss this?

Thirty-five years ago, Mr. Lowry, who is now 71, patented a method of cleaning up NASA's live televised transmissions from the moon. Six years ago, as the DVD took off, he set up Lowry Digital -- then a two-man R & D shop -- to apply his techniques to digital restoration.

He hired a photographer to make a short 35-millimeter film clip of some children playing soccer on a lakeshore. He paid a local lab to transfer the film to digital video, using a 4K scanner. The picture was clear, sharp, detailed. He then processed the images with his film-restoration software, which he'd programmed onto some Macintosh G4 computers. (The effort took months, as the faster G5's weren't out yet.)The processed picture was clearer, sharper and more detailed still. He could see every divot on the turf. What had once looked like a smudge in the background was now recognizable as a boat on the lake.

In January 2000, some executives from Warner Brothers saw his demo. They were so impressed, they faxed him an order the same day to restore the masters for three DVD's: "Gone With the Wind," "Now Voyager" and "North by Northwest." With the advance money, he bought the computers he needed to do the job.

Re:What are the Macs for?

[mikeophile]

You Only Live Twice is 117 minutes long.

At 24 frames per second, it contains 168480 frames.

The article says there are a pair of Imager XE-Advanced scanners.

Each scanner takes four minutes per frame.

Using these numbers, You Only Live Twice will take about 25 days to scan.

To answer your question, I have no fucking idea why so many Macs are being used, except maybe for their hard drives.

The point of all of this....

[Sancho]

How you sample analog material plays a big part in the overal quality of the finished product. For music, you typically think of samples per second (CDs play at 44.1khz). But typically for the initial digitization of analog material, you oversample (perhaps sampling the analog music at 88.2khz, or even higher). This gives you something that's much closer to the original work than normal, and allows you to work with a higher quality, well, sample. Performing digital transformations, including cleaning up the video, removing scratches, etc. always works better if you have more samples to work from. So a higher resolution picture will make it easier to get rid of any scratches or imperfections in the original film.

Eventually, of course, you have to downsample to fit the format that you will be distributing. For CDs, you downsample to 44.1khz. For DVDs, you downsample (the resolution) to 720x480 NTSC or 720x576 PAL. Note that that's somewhere around 1/8th the resolution that they're scanning.
The idea is simple. With this one scan, they can be prepared for format changes. Once high definition DVDs come out, they can downsample to whatever that resolution will be. If they want to broadcast a movie on an HD television channel, they can downsample to 1080i or whatever HD format they wish.

This seems to be about making a high-resolution copy now for archival purposes, so that if the film itself degrades (as it is prone to do) there will still be something really close to the original to work from. Not a bad idea, I think.

Re:careful...

[Peale]

but if its so secret, they better watch out... those pesky copyright lawyers might come after them....

If you'd read the article, you would have found that this was an official project. It's MGM that wants this done.

Re:But what about the sound?

[flux]

It would hardly make any sense to be cheat with storage space, as one second of the original movie could take 2 gigabytes of storage. If you just waste one 1/1000 of that to sound, you've already got 32 bit 300kHz sound..

Re:Common misconception

[Anonymous Coward]

This is neither 4000dpi nor 4000lpi. Its 4000 lines per frame of film. Think 4000p vs 720p in HDTV or 480p in DVD.

Re:Future-proof -- until your storage array dies

[ptrangerv8]

They store them on huge multi-terabyte disk arrays....

Here in the shop where I work, we have 1.5 TB of storage space, sitting in 2' of 19" rack space...

Disk storage is NOT an issue for something like this...

And for all you people who are asking what the macs are used for, it's to process the scanned frames.....

Re:What is the point of scanning at such a high re

[Andy_R]

Actually, it's about 2-3 film grains per pixel.

I used to make 35mm slides from computer files with my Agfa QCR-Z slide writer (and I still do from time to time for the few places that still use 35mms for projection).

It has the same resolution of 4k (4000 lpi) that these films are being scanned at. The pixels are significantly bigger than film grains, but are just about too small to bring into focus with a really good 35mm projector.

Later on, they made 8k and 16k resolution versions, which were mostly used for larger format than 35mm output because of the film grain issue (and the fact that the damn device used an RS-232 connection and therefore took 4-5 minutes to image a 4k line file)

Re:Compression and color dynamic range

[xtrochu]

The CINEON file format, which is the standard for digital movie pictures, use a density-linear bitdepth of 10 (or 12, can't remember) bits per channel. Due to the fact that it is density linear (and not light linear), you get much more precise information of what the film stock captured.

One has to understand that the density of a negative film stock is not linear to the intensity of light it received, but linear to i^some_gamma_value, where i is the intensity and some_gamma_value is roughly a constant that is dependent on the type of the filmstock, the process used for development of the film stock, and the archiving environment.

Another standard format is DPX, which supports 16 bits density linear bitdepth. But AFAIK, it is not as much used as CINEON.

Re:What is the point of scanning at such a high re

[Jeff DeMaagd]

Film grain represents the physical resolution of the film

One thing to keep in mind is that there are varying sizes of film grain, and having multiple grain sizes is a good thing, larger grains are good for low light image capture, smaller grains are good for capturing detail. Thus, one would want to make sure that the scanning resolution is higher than the finest grain in the image.

Also, there are good filter available in much more sophisticated means than simple blurring. If you ever get a chance to see the last two Matrix movies in IMAX, or any other film not shot on IMAX systems transferred to IMAX. Their system is IMO fantastic, had they just been 35mm projections or direct unprocessed transfers, it would have looked horrible.

Yes

[artemis67]

From what I gathered, from this article and the profile of Lowry on Apple's website, the software doesn't just remove dust and scratches but also film grain, by comparing each frame in the context of the surrounding frames and then softening or even removing irregularities. Yes, the difference will be huge.

Even on DVD; a 4th generation copy is like a movie that has had compression added 4 times, and each copy is progressively worse. Ideally, you want the cleanest print possible before you add lossy compression.

Re:actually, its you who is wrong.

[Overzeetop]

Agreed. I always read LPI (lines per inch) as "line PAIRS per inch", as it is a black/white pair which must be resolved. Of course, I think in mm, as that's how most kodak 35mm films are spec'd.

FWIW, most color negs run 60-80 line pairs per millimeter (1500-2000 lpi). Ektar 25 color neg claimed 125 line pairs per mm, comparable to kodak's Tmax b&w negative films. Kodak TechPan - a high contrast technical film - can be shot at 25-40ASA and developed in a special low contrast developer to yield (IMHO beautiful) continuous tone B&W negatives which come in at about 200 line pairs per mm. Enlarging TechPan by hand can be tricky, as you need a high power grain focuser to get the image in perfect focus, but it can provide the amatuer photog print quality which rivals medium format (2.25" negative) images with "normal" films.

Re:What is the point of scanning at such a high re

[drinkypoo]

That's the whole point of scanning at a higher resolution. Since the pixels and the location of the grains in the film are never going to match up 1:1, currently the best solution is to scan at vastly higher resolution than the source media can provide, to provide the truest possible digital representation of the appearance of that particular frame.

It may also be possible to construct a virtual frame in memory at a much higher resolution, then use positional manipulation of the frame (I.E. move it) while imaging it. Just as the handheld "scanner" technology for cellphones etc will allow you to wave a camera over a printed page and build a high resolution scan based on multiple passes, correlation, and interpolation, so we could do with movies. The problem with digital scans is of course that your scan quality is limited by the CCD pixel element size, the film grain size, the difference in their sizes, and the correlation (or lack thereof) of their positions.

As for duplicating adjacent pixels, no one uses that for a scaling algorithm any more unless they are a complete nincompoop, since so many other algorithms are readily available, but you're correct (obviously) in that data is always lost when using digital enhancement, which makes it useful for things like trying to decipher what license plate is on the back of a car, but not so useful for improving the quality of digital media.

Re:How much visual difference will there be...

[downix]

I would disagree there about the film being nearly destroyed. In a test, Kodak ran film shot by Edison in 1898, and it was as clear as the day it was developed. Using electron microscopes, kodak has estimated that the film will be viewable well into the 24th century. One area where degredation might occur would be with color-stocks. But, using the same process on early kodachrome, they've found a life expectancy in the hundreds of years. With technicolor, about the same. With the newer stocks however, the aging is occuring faster, so only 150-200 years for an original stock before some loss occurs.

Re:Wow, whats up with the NY Times?

[Anonymous Coward]

The article was also unclear why such horsepower is needed for such a mundane process as scanning and storing film.

From the article:

Thirty-five years ago, Mr. Lowry, who is now 71, patented a method of cleaning up NASA's live televised transmissions from the moon.

He hired a photographer to make a short 35-millimeter film clip of some children playing soccer on a lakeshore. He paid a local lab to transfer the film to digital video, using a 4K scanner. The picture was clear, sharp, detailed. He then processed the images with his film-restoration software, which he'd programmed onto some Macintosh G4 computers. (The effort took months, as the faster G5's weren't out yet.) The processed picture was clearer, sharper and more detailed still. He could see every divot on the turf. What had once looked like a smudge in the background was now recognizable as a boat on the lake.

That's why he needs such horsepower, so he can use his film-restoration software to cleanup the films before they're archived or converted to DVD.

Shady reporting...

[linkthese]

That's an interesting article, they don't even speak of what the 600 G5's are used for. Mr. Lowery's process is for dirt and scratch removal, that's what the G5's are for. The 4k scan, is to aid in the removal of dirt and scratches, but doesn't make it look much better since it's being downresed to HD or SD. NYTimes should fire this reporter, and should be ashamed for printing this article. It reports nothing.

Old News...

[jollygreengiantlikes]

This story was posted in more and less confusing detail on Apple's own pro-user webspace months ago. The article written by Joe Cellini is much better at explaining why the high resolution of scans, etc. The primary purpose of this studio is to remaster degraded and degrading films.

Here's the link:
http://www.apple.com/pro/film/lowry/ [apple.com]

Warner Brothers is doing something even better

[Anonymous Coward]

Many of their cartoons were filmed separately in R, G and B. That allows them to scan each color frame separately and use the multiple copies to find and eliminate scratches, etc. They also eliminate the registration errors in the final combined prints.

The net result is a version that is vastly superior to the originals.

(Posting as AC so they don't have me killed.)

some tech details...

[cwg_at_opc]

1 - scanning: the imagica XE can scan at a resolution of 4096x3112(1.31:1 aspect ratio), the just-announced
xe-advanced uses a 10k capture device that allows overscanning and subsequent downsampling from
8k(8192) to 4k. the 4096 pixels is the horizontal res. from perf-to-perf, and is nothing new(i've been
doing 4k since ~1995). the reason for 4k at the moment, is that 4096 pixels across is just _below_ the
grain of commonly used Oneg/intermediate stocks. using higher resolutions is a waste of processing
time and disk-space when your scanned resolution is higher than the source(this applies to t-grained
(tabular)films as well.)
anyway, you shouldn't see any pixels unless the color calibration is sub-optimal, you're looking at a digital
projection or there were hardware probs.
kodak(cinesite) has had "dust-busting" on their menu for quite a while now, although it was originally
done by hand, by artists using high-res paint programs(photoshop/matador, etc).
as correctly noted by another poster, the scanner is run by a linux based machine. the previous version
of their scanner used an SGI o2 running IRIX. see: www.imagica.com
Kodak used to make a commercial scanner(the cineon genesis scanner) that i believe is no longer avalable.
another scanner to look at is the Oxberry Cinescan.
this is the week to look for info as it's NAB time; new products and updates are typically announced there.

2 - color: the dynamic range of film is described in logarithmic terms(due to the sensitivity function of the
emulsion-processing chemistry) so it is appropriate to record/store using a log-based imaging format.
in this case, a 14bit DAC is used to generate 10bit log/pixel color data stored in the industry standard
Cineon format(created by Glenn Kennel @kodak and subsequently adopted industry-wide. see FIDO, Cineon)
10bits log is equivalent to 14 bits linear and covers approximately a 10-stop range or a density
range from zero(or film base) to somewhere around 2.048D to as much as 3.0D depending and the
scanner and recorder.

3 - lowry and warner: lowry and warner are both working on restoration systems. warner has a large library of
SE(sequential exposure) shows that will need duplicate archives and cleaning for DVD releases. SE is a method
for recording the RGB channels on individual-sequential frames. this process retains color integrity by
maintaining channel separation as long as possible avoiding channel bleed/crossover. lowry is using
the Macs to do the image processing; a feature-length film can be very, very large(90min x 24fps x @4k)
since each image can be ~50MB each - lots of disk space and processing time. as previously mentioned,
warner has a system which resizes/aligns each channel in a logical frame, resulting in a very clean image
with no(virtually no) fringing or edge artifacts due to sep misalignment. this is normally not an
issue with SE as each sep is on a single piece of film. for three-strip technicolor, the alignment is
more critical as there are three individual pieces of film that were run through a special camera(the
Technicolor camera) which i believe has a patent... for an interesting site with info on SE(w/pictures) goto:
thedigitalbits.com/articles/robertharris/harris072 303.html

4 - some resolutions:
HDTV - 1280x720 or 1920x1080
NTSC - 640x480(4:3)
PAL - 720x486
film - 2048x1536(1.33:1 AR)
4096x6144(vista-vision 8-perf)


i can expound more if additional details/info is needed.

Re:The size of the original

[cwg_at_opc]

If you're rendering at 1920x1080, almost all current film recorders will place the 1920 in the academy area; so unless you've got lots of money, time, money and more money, uprezing from HD to 2k isn't worth it. 4k maybe... Getting a Spirit Datacine(4k) would be a bit expensive, although not a whole lot more than film-scanning all your oneg at 4k. You benefit from the datacine by the ability to go straight to a color correction suite(DaVinci) and speed, whereas a film-scan forces you deal with your shots as 'frames', not terrible, but you have to manage your production differently.

As for automated uprezing, you might want to investigate something like Genuine Fractals which does a good job of uprezing 1024 all the way to big poster size(something like 6k across, last i heard, but the limit is 600%) so in your case, you could batch-render the whole lot from 1920 to 11520. Going to 4k academy would give you a nice 'velvety' look. That's not to say 1920 in academy sucks; it doesn't, it's just that the uprez works on a diminishing-returns function and you only realize true quality if you originate at high-ish resolutions.

I've personally worked on 2k and 4k shows and the difference is so subtle that the average movie-goer will not see a difference. We've even used 2k VistaVision scans with good results on film(2k acad.) What you really need the higher resolutions and bit-depth is for effects. Any processing, compositing, filtering, etc. should be done at a higher rez and then resized to your final output. Higher rez(dimensions) reduces edge artifacts by having more pixels to work with. Using higher bit depth allows color-correction/artistic-bending with little or no quantization artifacts(aka - banding.)