Stories
Slash Boxes
Comments
Search

News for nerds, stuff that matters

Slashdot Log In

Log In

Create Account  |  Retrieve Password

MXF+JPEG-2000+HDD = Future of Video Preservation?

Posted by CmdrTaco on Sun Feb 06, 2005 11:03 AM
from the archivists-know-how-to-party dept.
Media Television
Anonymous Archivist writes "Media Matters, a technical consultancy specializing in archival audio and video material, recently completed a Mellon Foundation funded Digital Video Reformatting Preservation Project for the Dance Heritage Coalition. They conclude that MXF is the recommended container format, JPEG-2000 is the recommended encoding format and HDD is the recommended storage media. It's a very valuable series of experiments and offers a strong indication of where the archival preservation of analogue video is heading."
+ -
story
This discussion has been archived. No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.

MXF+JPEG-2000+HDD = Future of Video Preservation? 50 Comments More /

 Full
 Abbreviated
 Hidden
More
Loading... please wait.
  • Enlighten me. What's MXF?

    • Re:MXF? (Score:5, Informative)

      by Raul654 (453029) on Sunday February 06 2005, @11:08AM (#11589854) Homepage
      " The Material eXchange Format (MXF) is an open file format targeted at the interchange of audio-visual material with associated data and metadata. It has been designed and implemented with the aim of improving file based interoperability between servers, workstations and other content creation devices. These improvements should result in improved workflows and result in more efficient working than is possible with today's mixed and proprietary file formats." -- What is MXF [broadcastpapers.com]

    • Re:MXF? (Score:4, Funny)

      by The Ultimate Fartkno (756456) on Sunday February 06 2005, @12:22PM (#11590353)
      MXF is the new, proprietary video compression method jointly sponsored by Microsoft and MTV. The new Most eXtreme Format is the video compression of choice for today's most hard-core, edgy, in-your-face artists with an attitude!

      Ashlee Simpson says "When I'm performing for a half-time show of 10,000 screaming fans, I want to make sure that every bit of the live energy is caught perfectly! I give 100% for my fans and want to make sure they get every bit of my performance!"

      MXF... in your FACE, Quicktime! This isn't your father's archive-quality lossless video compression algorithm!

      (and keep an eye out for Ogg Vorbis 2 - by Mountain Dew!)

  • JPEG-2000 (Score:3, Interesting)

    by Anonymous Coward on Sunday February 06 2005, @11:08AM (#11589858)
    Why would they go with a compression format that doesn't do inter-frame compression?
    It might be nice for editing, but you could get more quality in the same space with something like h264, or even h263 if they have to do this right now (i.e. before h264 is quite ready for prime time).

  • Recommended Storage Media (Score:5, Funny)

    by Antonymous Flower (848759) on Sunday February 06 2005, @11:09AM (#11589864) Homepage
    Recommended Storage Media: Peer to Peer network.
    • This sounds funny, but it is absolutely true. It is like a giant distributed RAID and as long as two or more people want to keep the video, it will be there for a long time. If nobody wants the file, it is obviously not that important and probably shouldn't be saved anyway.

    • Re:Recommended Storage Media (Score:5, Insightful)

      by remahl (698283) on Sunday February 06 2005, @11:54AM (#11590169)
      Definitely not!

      Most if not all peer to peer networks require a certain level of interest in an item for it to be retained. Popular items are always easy to find while obscure / old items gradually disappear from the network.

      Try finding a movie that's a few years old. You'll have more trouble finding the original Jurassic Park than Jurassic Park III.

      Peer to peer is not a great way to reliably and systematically preserve cultural heritage.

      • Re:Recommended Storage Media (Score:4, Insightful)

        by Catbeller (118204) on Sunday February 06 2005, @02:10PM (#11591084) Homepage
        Well, that's more a function of the cost and size of data storage. Give me a Petabyte of soldid state nonvolatile storage, and I'll toss Jurrassic Park I in there for giggles, along with 20's silent films, clips from "Bozo's Circus" on WGN on 1969's Chicago TV, the collected books of mankind, complete 3D terrain maps of Mars and every old time radio recording in existence. Gimme a $200 unit that does this, and I'll preserve anything I can get my hands on!
    • Along with the recommended exchange formant:

      Multi RAR'ed 14mb archives of .BIN and .CUE files. Including the ever necessary .nfo files.

  • Why HDD? (Score:5, Interesting)

    by Orinthe (680210) on Sunday February 06 2005, @11:13AM (#11589899) Homepage
    The HDD recommendation doesn't seem to make much sense. The article talks about cost-per-gigabyte, but obviously it is much cheaper to use CDRs or DVDRs. This is video preservation, after all, not storing indefinitely for video /editing/, which would require a more malleable storage medium. And before someone points out that there are studies showing that the longevity of CDR/DVDR discs is questionable, surely proper storage of discs (and not buying the Best Buy free-after-rebate special) would be sufficient. HDD, after all, is susceptible to head crashes, and being a magnetic medium can be more easily overwritten.
    • If you just mirror it on two hard drives and then put them into storage, they will last for a very long time. HDDs only die when run via wearing out and not just sitting on the shelf.
      • If you just mirror it on two hard drives and then put them into storage, they will last for a very long time.

        If you mirror across two disks and put the into storage, and one develops some minor errors, it is not possible to tell which one has the errors unless the data itself stores error checking and correction information. This is why God RAID-5 was invented. Using 3 drives you can identify and repair any errors that develop on any one drive.

        If you just mirror it on two hard drives and then put them

        • Re:Why HDD? (Score:5, Insightful)

          by drinkypoo (153816) <martin.espinoza@gmail.com> on Sunday February 06 2005, @11:37AM (#11590063) Homepage Journal
          It would be smarter to use PAR2 (or similar) on a filesystem basis, than to use a RAID filesystem. It's easier to deal with user space programs for reconstructing data.

        • Re:Why HDD? (Score:5, Informative)

          by Jah-Wren Ryel (80510) on Sunday February 06 2005, @12:50PM (#11590571)
          If you mirror across two disks and put the into storage, and one develops some minor errors, it is not possible to tell which one has the errors

          Exceptionally incorrect, prepare for smackdown.

          All data on a hard disk is protected by very sophisticated error detection and correction elgorithms. The chance of getting "some minor errors" is effectively nil - either they are corrected by the disc's controller, or the controller returns a "sector unreadble" error - which is what keys any effective mirroring system to go get the data from the second disk. You just don't get bad data from modern hard disks.

          This is why God RAID-5 was invented.

          No, raid-5 was invented to maintain the I in RAID. Mirroring doubles your costs, RAID-5 only increases them by one disk out of the N disks in the parity group, where N is usually but not limited to 4-5 drives.
      • HDDs only die when run via wearing out and not just sitting on the shelf.

        But how long do they live after being knocked off of the shelf?

    • Re:Why HDD? (Score:3, Interesting)

      by FireBug (83228)
      I didn't read the article (or the rest of the /. comments), but hard drives make much more sense than any optical storage medium in certain cases.

      Media will always wear out, regardless of what type it is. When you have huge amounts of data to back up, it's much nicer to be able to copy it to the latest greatest storage medium quickly and efficiently. Thousands of CDs/DVDs even with an automated "disc changer" would take a hell of a lot longer to transfer than a bunch of servers with hard drives.

      With a h
    • That'why you make backups, and transfer the data to new hard drives from time to time -- a process much easier with hard disks than with DVDs, if you can afford all the hard disks.

    • The HDD recommendation doesn't seem to make much sense. The article talks about cost-per-gigabyte, but obviously it is much cheaper to use CDRs or DVDRs.

      Wrong. Here in Finland, a new 160 GB hard disk ( Maxtor DiamondMax 10) costs 89 euros. An empty 700 MB cd costs 1 euro. Assuming 1 GB = 1000 MB, it would take 160 GB / 0.7 GB = 229 CD's to get the same capacity as that one HDD. So, if you use CD's, you pay 229 euros, if you use HDD's, you pay 89 euros.

      The cost per gigabyte in my example HDD is about 0

  • first step (Score:3, Funny)

    by same_old_story (833424) on Sunday February 06 2005, @11:21AM (#11589950)
    make their report available on a format other than a '.doc' file. it is known to change a lot and therefore not suitable for long term storage.

  • Nonsense (Score:5, Insightful)

    by sql*kitten (1359) * on Sunday February 06 2005, @11:25AM (#11589970)
    OK, let's talk archiveability. Let's talk about a medium that you can leave in a shoebox for a hundred years and read just by shining a light through it. I'm not talking hypothetical here - this technology is proven by the fact that people used it a hundred years ago and it worked. And the technology is even better now, even more stable.

    I am of course talking about film. It is very very easy now to write digital images onto film, not very much more difficult than it is to scan film. There's no need to worry about whether the file format will be supported in the future, as I've already said. You don't need to shovel money into vendor's pockets every few years just to copy it to the latest trendiest type of disc. You can build a machine to project film out of junk if you need to, or you can scan it if you want a digital image and when you have a better scanner (e.g. a higher DMax), you can just scan it again.

    The dude who wrote this report is just blowing smoke. He's trying to sell snake oil.
    • Do you have any information on the overall cost of film storage as opposed to hard disk drives? Specifically, one must account for the initial cost of equipment, the cost of storage, the cost of recovery of damaged data, and the cost of paying the people involved to implement everything. Since film and hard drives have [vaguely] similar storage requirements, I'd say the cost of storage is basically a function of density, another thing I have no idea about. I understand that film has been used for archival o
    • I am of course talking about film. It is very very easy now to write digital images onto film, not very much more difficult than it is to scan film.

      And it is even EASIER to burn film. Yeah. Great preservation, indeed.

    • Simplistic (Score:3, Interesting)

      by fm6 (162816)
      I suspect that your picture of the survivability of film stock is a little optimistic. But I'll leave that issue to somebody who actually knows the technology. What really bothers me about your argument is your focus on a single factor: keeping the data available as long as possible with an absolute minimum of maintenance. If that were the only consideration, then film is actually a bad choice. Many more archival techniques are obviously more survivable. You could, for example, etch the data on platinum pla
      • also "you can scan it if you want a digital image and when you have a better scanner (e.g. a higher DMax), you can just scan it again" everytime you convert from analog to digital or the other way you loose quality.

        Every kind of video goes through an ADC at one point, whether or not it's inside the camera/recording device or done through scanning later doesn't really matter. Archiving in film is still the absolute highest quality you can achieve. Scanning it once does not deteriorate the analog copy for

      • Re:Nonsense (Score:4, Informative)

        by Jeff DeMaagd (2015) on Sunday February 06 2005, @12:13PM (#11590300) Homepage Journal
        and no you cannot " build a machine to project film out of junk". do you know how much film projectors cost? (hint a good lens alone, is over $5000,00.

        I think it is possible. Such an expensive lense isn't necessary. The best film projectors cost a lot, but I don't see it as that difficult to fabricate a basic one from scratch. It won't be the best but it could actually be watchable on a small scale.

        It might look hard to the monkeys that assembles ATX computers but I think a decent one could be made from scratch as a small senior engineering project for college, and probably could be adjustable with different sprockets and such. A little more complex than just shining a light through it. It may be hard to imagine, but there was a time when people had portable film cameras for home videos. It wasn't fancy and didn't need to be.

        Kodak announcing they'll stop producing film has little to do with anything, IMO. Five years is a lot of time but thus far, the drive to push digital projection is going much slower than people expected. Lucas wanted his Episode III to be exclusively projected in digital video, but it's not going to happen unless he wants to drastically cut the number of screens, I'm thinking a tenth of the screens is not an unrealistic figure.

        Of course, part of that is political and economic, because it saves the film distributors from major costs, but they refuse to pass on the savings to the theater companies that must invest as much as a quarter million dollars just to get started.

  • JPEG 2000 for video? Huh? (Score:3, Interesting)

    by Zarhan (415465) on Sunday February 06 2005, @11:28AM (#11590000)
    Okay, so JPEG 2000 uses wavelets and is therefore quite advanced, but as I have understood, it's still geared for still images (ok, there is probably some form of motion jpeg 2000?).

    I would think that most optimal method would be to use something like DIRAC [bbc.co.uk] instead (or Ogg Theora). DIRAC uses wavelets and adaptive arithmetic coding, so it should be "on par" with JPEG 2000 - and should also be free of patent encumberance.

    JPEG 2000 has one feature that might make it better in "archival" purposes - there is a lossless mode which still achieves higher compression ratios than PNG.

    • Intraframe vs. interframe (Score:5, Interesting)

      by Wesley Felter (138342) <wesley@felter.org> on Sunday February 06 2005, @03:12PM (#11591532) Homepage
      For whatever reason (I'm not a video expert) many people prefer intraframe codecs for archival. As you probably guessed, Motion JPEG 2000 just treats each video frame as a still image and compresses it with JPEG 2000.

      Dirac will give much better compression that JPEG 2000, but it also introduces the possibility of interframe artifacts.
      • Dear kind AC,

        MPEG-2 has nothing to do with wavelets, MPEG-2 is based on DCT. In general, there are four methods for compression, discrete cosine transform (DCT), vector quantization (VQ), fractal compression, and discrete wavelet transform (DWT).

        MPEG codecs (1, 2, 4, H.26x) all use DCT. Have a nice day.

  • OK, so when are we going to have support for it? (Score:4, Informative)

    by bersl2 (689221) on Sunday February 06 2005, @11:30AM (#11590014) Journal
    https://bugzilla.mozilla.org/show_bug.cgi?id=36351 (no link for obvious reasons) is the bug report, which has been around since April 2000 but has not progressed much due to licensing issues (copyright ones fixed, patent ones not?).

  • Turn it up! (Score:4, Interesting)

    by belg4mit (152620) on Sunday February 06 2005, @11:30AM (#11590019) Homepage
    Ummm what about the sound?!
  • Storing digital information on paper is feasible and lots of research efforts have been put into it.

    Storing data on anything magnetic or optical is a bit worrysome. But then, it's not critical data so I guess it doesn't really matter.

    • Paper isn't an optical medium? Punchcards perhaps... I assume you are talking about better tech than my laser printer + scanner which I've estimated could hold about 1MB (which I believe is what the Paper Disk software can do) on a 8.5"x11" page. So what kind of data density can you achieve on paper, and what sort of printer and scanner are you using?

      But, it's an optical format... so why not design materials that can achieve much greater data densities? Like DVDs? Surely it's possible to design something


      • >>it's not critical data so I guess it doesn't really matter

        >Ouch, burn. There is nothing quite like the feeling of being told that your culture and history isn't important, and doesn't matter.

        Oh, c'mon.. I mean, culture.. history.. it's hardly porn. Who cares if a few decades of historical records get wiped? Heck, just make 'em up again. Losing part of your porn collection though.. now that's a disaster.

  • HDs (Score:3, Interesting)

    by eno2001 (527078) on Sunday February 06 2005, @11:38AM (#11590076) Homepage Journal
    I could have told people this as they've replaced video tape, and audio tape for me for the past decade. I find them much more convenient, portable and cross platform. I have SCSI drives from 1994 that will still work in a PC (Linux or Windows) or Mac today. They are easy to backup to and restore from. The HD is about as close to perfection as you can get in a storage medium. At least until you get flash drives that can store 1 terabyte at minimum, and have an infinite number of writes. At least a 100 year lifespan.

  • Ars Technica... (Score:4, Interesting)

    by EMIce (30092) on Sunday February 06 2005, @11:57AM (#11590189) Homepage
    ...had a guide on capturing analog video [arstechnica.com], said to be the part of a 3 part series, going over each capturing, cleaning, and compressing. Only part I ever came out - Ars do you read slashdot? - I am waiting on the last guides for some advice on how to preserve these rotting home VHS tapes.

    Meanwhile, does anyone else have advice on capturing and cleaning video since we are already talking about compression? What settings are good for capturing and what sort of software exists to clean up VHS and give it the appearance of more clarity? I am using a WinTV card as Ars recommended it.
    • I wouldn't bother with ArsTechnica. For the definitive guide to capturing analog video and digitally archiving it, you would want to read this guide [doom9.org] on Doom9. Plus, they have many other video-related guides on that site and a forum that is second to none in terms of the sheer amount of expertise exhibited by the users there.

  • The move to disk backup continues (Score:4, Insightful)

    by RonBurk (543988) on Sunday February 06 2005, @12:20PM (#11590339) Homepage Journal
    People objecting to the use of hard drives for backup miss several points.

    • Yes, they are (somewhat, not excessively) vulnerable to magnetism. But optical discs are vulnerable to light, fingerprints, chemicals, etc.
    • Optical discs continue to lag far behind in capacity. So, in the land of audio/video backup, the choice is between a single hard disk, or dozens of optical discs. The risk of failure of multiple optical discs is amplified by the increasing number of discs.
    • Bandwidth is another issue. Though hard disk bandwidth lags behind the growth of hard disk storage, optical disc bandwidth lags even further behind.
    • Restore time is another issue. You can line up a bunch of optical disc drives and try to make all your data available at once, but you're probably never going to get the restore speed of solutions like Massive Arrays of Inactive Disks [backupcritic.com].


    There will always be multiple backup solutions, but the biggest trend continues to be towards using hard disks for backup. When your data files are enormous (such as with audio/visual data), HDD backup is even more attractive.

  • Yawn, another technology of the day... (Score:4, Insightful)

    by HockeyPuck (141947) on Sunday February 06 2005, @01:11PM (#11590714)
    All this is is a method to line some guy's pockets. I'm sure the tape guys are gonna say, use XYZ type of tape. The disk guys are gonna say disk.

    What makes this guy think that the interface to the HDD is going to be around in X years?

    PC's have only had two dead (non-(e)IDE/ATA) interfaces, the ESDI and the ST506/ST-412 interfaces.

    But what if you were trying to find a computer with IPI (1960s mainframe) interface.

    The Fed gov't has this problem with trying to find parts for their old 8/9track tape drives..

    Here's a good list of all the HDD interfaces over the years: http://www.i-t-s.com/corporate/terms.html [i-t-s.com]

    Stick with microfiche, film, that way we don't have to pay some vendor $$$/yr to keep alive a dead technology or pay some other vendor $$$/media to move them from old to new media.

  • .doc? (Score:3, Funny)

    by b1t r0t (216468) on Sunday February 06 2005, @04:23PM (#11592004)
    For people concerned with the preservation of "data", they've sure picked an interesting format to write about it in.
  • indeed, lossless for archival preservation is the
    only way, as it fits the basic rule of art restoration
    technology -- never apply "improvements" which
    cannot be reversibly undone to take advantage
    of future science.

    ironically then, the lossless format doesn't matter.

    however, at least for the instant case of dance video,
    the likely input (a myriad of digital tape formats)
    is hopelessly neanderthal -- anything having to do with DV,
    or MPEG, or even ATSC HDTV already tosses away much
    color information. (4:1:1, 4:2:0, and 4:2:2 colorspace is embarrassing
    to preserve "losslessly".) ditto for temporal
    info, with interlacing being the culprit. even film at
    24fps just will not cut it for motion such as dance.

    so here's to better camera technology, whether it's
    10- or 12-bit 4:4:4 RGB, or something like
    carver mead's foveon made swift.

    • Re:Lossy file formats... (Score:5, Informative)

      by iezhy (623955) on Sunday February 06 2005, @11:07AM (#11589848) Homepage
      JPEG standart defines several encoding formats, which include lossless compression as well
      • You absolutely can have TIFF video.

        Source: Macintosh HD:Users:Leo:Movies:Reagan.mov
        Format: Integer (big endian), Stereo, 48000 Hz, 16 bits
        TIFF, 720 x 480, Millions
        Movie FPS: 29.97
        Playing FPS: (Available when playing)
        Data Size: 1617.9 MB
        Data Rate: 19.9 MB/sec
        Current Time: 00:00:00.00
        Duration: 00:01:21.04
        Normal Size: 720 x 480 pixels
        Current Size: 720 x 480 pixels (Normal)

        TIFF would be a much better choice for archiving, because it's a much simpler format and is much easier to decode.

          • Because when you're archiving digital data, recoverability is paramount. You have to ask yourself, "What if all I had was a piece of this data, say, a hundred gigabytes from the middle of the disk? Could I turn that data into useful information?"

            If you're dealing with a run-length-encoded array of packed pixels, the answer is obviously yes. That's among the simplest forms of encoding known. (If you don't RLE the data it's even simpler, but a trade-off between simplicity and storage requirements is okay as

All trademarks and copyrights on this page are owned by their respective owners. Comments are owned by the Poster. The Rest © 1997-2009 Geeknet, Inc.