Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×
Television Media

MXF+JPEG-2000+HDD = Future of Video Preservation? 214

Posted by CmdrTaco from the archivists-know-how-to-party dept.
Anonymous Archivist writes "Media Matters, a technical consultancy specializing in archival audio and video material, recently completed a Mellon Foundation funded Digital Video Reformatting Preservation Project for the Dance Heritage Coalition. They conclude that MXF is the recommended container format, JPEG-2000 is the recommended encoding format and HDD is the recommended storage media. It's a very valuable series of experiments and offers a strong indication of where the archival preservation of analogue video is heading."
This discussion has been archived. No new comments can be posted.

MXF+JPEG-2000+HDD = Future of Video Preservation? More

MXF+JPEG-2000+HDD = Future of Video Preservation?

Comments Filter:
  • Enlighten me. What's MXF?

    • Re:MXF? (Score:5, Informative)

      by Raul654 ( 453029 ) on Sunday February 06, 2005 @11:08AM (#11589854) Homepage
      " The Material eXchange Format (MXF) is an open file format targeted at the interchange of audio-visual material with associated data and metadata. It has been designed and implemented with the aim of improving file based interoperability between servers, workstations and other content creation devices. These improvements should result in improved workflows and result in more efficient working than is possible with today's mixed and proprietary file formats." -- What is MXF [broadcastpapers.com]

    • Re:MXF? (Score:4, Funny)

      by The Ultimate Fartkno ( 756456 ) on Sunday February 06, 2005 @12:22PM (#11590353)
      MXF is the new, proprietary video compression method jointly sponsored by Microsoft and MTV. The new Most eXtreme Format is the video compression of choice for today's most hard-core, edgy, in-your-face artists with an attitude!

      Ashlee Simpson says "When I'm performing for a half-time show of 10,000 screaming fans, I want to make sure that every bit of the live energy is caught perfectly! I give 100% for my fans and want to make sure they get every bit of my performance!"

      MXF... in your FACE, Quicktime! This isn't your father's archive-quality lossless video compression algorithm!

      (and keep an eye out for Ogg Vorbis 2 - by Mountain Dew!)

  • JPEG-2000 (Score:3, Interesting)

    by Anonymous Coward on Sunday February 06, 2005 @11:08AM (#11589858)
    Why would they go with a compression format that doesn't do inter-frame compression?
    It might be nice for editing, but you could get more quality in the same space with something like h264, or even h263 if they have to do this right now (i.e. before h264 is quite ready for prime time).

  • Recommended Storage Media (Score:5, Funny)

    by Antonymous Flower ( 848759 ) on Sunday February 06, 2005 @11:09AM (#11589864) Homepage
    Recommended Storage Media: Peer to Peer network.

  • Why HDD? (Score:5, Interesting)

    by Orinthe ( 680210 ) on Sunday February 06, 2005 @11:13AM (#11589899) Homepage
    The HDD recommendation doesn't seem to make much sense. The article talks about cost-per-gigabyte, but obviously it is much cheaper to use CDRs or DVDRs. This is video preservation, after all, not storing indefinitely for video /editing/, which would require a more malleable storage medium. And before someone points out that there are studies showing that the longevity of CDR/DVDR discs is questionable, surely proper storage of discs (and not buying the Best Buy free-after-rebate special) would be sufficient. HDD, after all, is susceptible to head crashes, and being a magnetic medium can be more easily overwritten.

    • Re:Why HDD? (Score:3, Informative)

      by chotchki ( 856592 )
      If you just mirror it on two hard drives and then put them into storage, they will last for a very long time. HDDs only die when run via wearing out and not just sitting on the shelf.

      • Re:Why HDD? (Score:3, Insightful)

        by TheRaven64 ( 641858 )
        If you just mirror it on two hard drives and then put them into storage, they will last for a very long time.

        If you mirror across two disks and put the into storage, and one develops some minor errors, it is not possible to tell which one has the errors unless the data itself stores error checking and correction information. This is why God RAID-5 was invented. Using 3 drives you can identify and repair any errors that develop on any one drive.

        If you just mirror it on two hard drives and then put them

        • Re:Why HDD? (Score:5, Insightful)

          by drinkypoo ( 153816 ) <drink@hyperlogos.org> on Sunday February 06, 2005 @11:37AM (#11590063) Homepage Journal
          It would be smarter to use PAR2 (or similar) on a filesystem basis, than to use a RAID filesystem. It's easier to deal with user space programs for reconstructing data.
          • Dunno. I presume you would want to keep the files on one place on the disk. You could spread them out, but that would be like RAID 5. If you keep them at one spot, and you loose the disk, then PAR/PAR2 is not going to save you. If you do PAR over multiple files ... well ... is possible, but yuk. RAID has been proven technology as well.

            Personally I think this whole hard disk storage thing will work out fine for my personal collection, but for something that does not have to be online all the time I don't th
            • For something that has to be online all the time, hard disks are pretty much the only solution. For things that have to be near-online, we have optical disc, but AFAIK the largest capacity shipping right now is DVD9. The densest jukeboxes I've seen (in the audio world, but the design would be easy enough to adapt) put 100 discs in a space about 14 inches on a side. That's 8.4 * 100 = 840 GB, still not even 1 TB. So I'd say that's got absolutely horrible density. Plus, DVD9 disks are still quite expensive. F

        • Re:Why HDD? (Score:5, Informative)

          by Jah-Wren Ryel ( 80510 ) on Sunday February 06, 2005 @12:50PM (#11590571)
          If you mirror across two disks and put the into storage, and one develops some minor errors, it is not possible to tell which one has the errors

          Exceptionally incorrect, prepare for smackdown.

          All data on a hard disk is protected by very sophisticated error detection and correction elgorithms. The chance of getting "some minor errors" is effectively nil - either they are corrected by the disc's controller, or the controller returns a "sector unreadble" error - which is what keys any effective mirroring system to go get the data from the second disk. You just don't get bad data from modern hard disks.

          This is why God RAID-5 was invented.

          No, raid-5 was invented to maintain the I in RAID. Mirroring doubles your costs, RAID-5 only increases them by one disk out of the N disks in the parity group, where N is usually but not limited to 4-5 drives.
        • I'd be happy to be corrected, but my understanding is that RAID 5 is no more capable of telling you which drive is "right" about a block than RAID 1, at least its most common default configuration.

          RAID 5 with one parity disk essentially stores the XOR of all the other disks' data on the partity disk. (It's not that simple because it stripes it across disks, but in terms of the logic that doesn't matter).

          If I have three values, a, b, and p where:

          p = a XOR b

          and I find out that *one* of them is wrong becau
        • If you mirror across two disks and put the into storage, and one develops some minor errors, it is not possible to tell which one has the errors unless the data itself stores error checking and correction information.

          Actually, all hard drives store a CRC or other error detection method with each and every sector, and have been doing this for decades. Generally it should be easily possible and reliable to tell whether any given data is correct or not. It's the error recovery that's always been the tricky

      • HDDs only die when run via wearing out and not just sitting on the shelf.

        But how long do they live after being knocked off of the shelf?

        • A friend of mine has a friend who was running lots of host machines (web servers, irc, file sharing, etc.) in a kind of ghetto set-up in Calgary. One day, everyone sitting in irc saw the connection drop, all the stuff being served was gone. No web site, no streaming audio, no irc.

          The servers were all sitting on a shelf... you know, the kind you use those brackets that screw into the wall, and put some board on top? I will leave it to your imagination to figure out what the technical problem was that day.

          T
      • HDDs can also die when bearing lubricants harden, or for some other reason the spindle freezes. In some cases, this is more likely to happen when sitting on the shelf than when running.

    • Re:Why HDD? (Score:3, Interesting)

      by FireBug ( 83228 )
      I didn't read the article (or the rest of the /. comments), but hard drives make much more sense than any optical storage medium in certain cases.

      Media will always wear out, regardless of what type it is. When you have huge amounts of data to back up, it's much nicer to be able to copy it to the latest greatest storage medium quickly and efficiently. Thousands of CDs/DVDs even with an automated "disc changer" would take a hell of a lot longer to transfer than a bunch of servers with hard drives.

      With a h
    • That'why you make backups, and transfer the data to new hard drives from time to time -- a process much easier with hard disks than with DVDs, if you can afford all the hard disks.

    • Re:Why HDD? (Score:3, Informative)

      by ultranova ( 717540 )

      The HDD recommendation doesn't seem to make much sense. The article talks about cost-per-gigabyte, but obviously it is much cheaper to use CDRs or DVDRs.

      Wrong. Here in Finland, a new 160 GB hard disk ( Maxtor DiamondMax 10) costs 89 euros. An empty 700 MB cd costs 1 euro. Assuming 1 GB = 1000 MB, it would take 160 GB / 0.7 GB = 229 CD's to get the same capacity as that one HDD. So, if you use CD's, you pay 229 euros, if you use HDD's, you pay 89 euros.

      The cost per gigabyte in my example HDD is about 0

      • For what it's worth, 1 Euro per writable CD is extremely expensive. Here in Germany, it's more like 0.30 Euro and less, depending, of course, on how many you buy. That said, DVDs have been far cheaper per megabyte for a long time now, so that's what you'd probably use. 1 Euro per DVD seems about right, if and only if you buy just one - if you buy a spindle of 100 DVDs, it's more like 0.35 apiece. If CDs and DVDs are really that expensive in Finland, I'd seriously consider importing them...
      • Wrong. Here in Finland, a new 160 GB hard disk ( Maxtor DiamondMax 10) costs 89 euros. An empty 700 MB cd costs 1 euro.

        WOW do blank media prices suck in Europe! I don't mean that as a taunt or as nationalistic gloating, but just... Wow.

        Here in the US, I can get a 100ct spool of blank name-brand 8x DVD+R for $35 (for noname 4x media, I found them as low as $19/100, but wouldn't really trust those for anything but a throwaway means of getting some large but replaceable files between two places). Compari

  • first step (Score:3, Funny)

    by same_old_story ( 833424 ) on Sunday February 06, 2005 @11:21AM (#11589950)
    make their report available on a format other than a '.doc' file. it is known to change a lot and therefore not suitable for long term storage.
    • Really? I thought it hadn't changed substantially (if at all) since Office 97, which was probably released in 1996, so 9 years ago. In computer terms that's a long lasting format.

  • Nonsense (Score:5, Insightful)

    by sql*kitten ( 1359 ) * on Sunday February 06, 2005 @11:25AM (#11589970)
    OK, let's talk archiveability. Let's talk about a medium that you can leave in a shoebox for a hundred years and read just by shining a light through it. I'm not talking hypothetical here - this technology is proven by the fact that people used it a hundred years ago and it worked. And the technology is even better now, even more stable.

    I am of course talking about film. It is very very easy now to write digital images onto film, not very much more difficult than it is to scan film. There's no need to worry about whether the file format will be supported in the future, as I've already said. You don't need to shovel money into vendor's pockets every few years just to copy it to the latest trendiest type of disc. You can build a machine to project film out of junk if you need to, or you can scan it if you want a digital image and when you have a better scanner (e.g. a higher DMax), you can just scan it again.

    The dude who wrote this report is just blowing smoke. He's trying to sell snake oil.
    • Do you have any information on the overall cost of film storage as opposed to hard disk drives? Specifically, one must account for the initial cost of equipment, the cost of storage, the cost of recovery of damaged data, and the cost of paying the people involved to implement everything. Since film and hard drives have [vaguely] similar storage requirements, I'd say the cost of storage is basically a function of density, another thing I have no idea about. I understand that film has been used for archival o
    • I am of course talking about film. It is very very easy now to write digital images onto film, not very much more difficult than it is to scan film.

      And it is even EASIER to burn film. Yeah. Great preservation, indeed.

    • Simplistic (Score:3, Interesting)

      by fm6 ( 162816 )
      I suspect that your picture of the survivability of film stock is a little optimistic. But I'll leave that issue to somebody who actually knows the technology. What really bothers me about your argument is your focus on a single factor: keeping the data available as long as possible with an absolute minimum of maintenance. If that were the only consideration, then film is actually a bad choice. Many more archival techniques are obviously more survivable. You could, for example, etch the data on platinum pla
      • The 5.25" floppy was a pervasive consumer device.

        Try finding anything that can read them today.
        • You can still buy 5.25" drives. But even if you were right, this format was never as popular as the CD is today. Not by many orders of magnitude.

          I'm not arguing that optical formats will be around forever. But they'll be around for a lot longer than floppy drives.

          Any format you could pick is a tradeoff between long term availability and the other factors I mentioned. No format will be around forever.

    • Re:Nonsense (Score:2, Insightful)

      by Waffle Iron ( 339739 )
      OK, let's talk archiveability. Let's talk about a medium that you can leave in a shoebox for a hundred years and read just by shining a light through it.

      Most if not all film from 100 years ago was made from nitrocellulose. If you left that in a shoebox for 100 years, you would probably end up with a box of dust.

      Most of the color film from 50 years ago was made with unstable dyes. If you kept that for 100 years, you'd have a box of transparent plastic.

      Now they think they have film that's more stable. C

      • Kodachrome from 50 years ago looks as good today as when it was shot. And that's film that hasn't been stored optimally for preservation. It doesn't use unstable dyes; it instead uses silver, much like black and white film.
        • Kodachrome uses dyes, they are incorporated in the processing instead of being part of the film as manufactured. Although Kodachrome's dyes are stable with temperature and reasonable humidity, they are not stable with exposure to light, and in fact are poorer than most other chromes in that regard.
    • The nice thing about digital media is that you can leave it in an unrefrigerated shoe box for a decade or two, then come back and make a perfect copy of it with absolutely no degradation.

      You can also make a perfect copy and stick it in numerous locations, making them harder to lose in a fire/terrorist attack/rampaging llama incident. They don't require refrigeration, and they take up a lot less room.

      But it's not perfect: there's a analog-to-digital step, and you lose information there. Even a print of t
    • U r 100% right that paper and film have a proven track record, which is more then u can say for anyting digial (proven = > 100 years) Analog is also inherently superior to dig for archive in that partial loss is only partial loss.
      The required equipment could be built in volume at a reasonable price (and even custom glass lens are not that $$, as you can see from say photonics magazine), but as u c every day in cameras, you can mold pretty good lens out of polycarbonate or coc cheaply). The cost of 35 or
      • I'm sorry, I stopped reading after " U r "

        Geeze... if you have anything worthwhile to post, could you try an approximation to english?

  • JPEG 2000 for video? Huh? (Score:3, Interesting)

    by Zarhan ( 415465 ) on Sunday February 06, 2005 @11:28AM (#11590000)
    Okay, so JPEG 2000 uses wavelets and is therefore quite advanced, but as I have understood, it's still geared for still images (ok, there is probably some form of motion jpeg 2000?).

    I would think that most optimal method would be to use something like DIRAC [bbc.co.uk] instead (or Ogg Theora). DIRAC uses wavelets and adaptive arithmetic coding, so it should be "on par" with JPEG 2000 - and should also be free of patent encumberance.

    JPEG 2000 has one feature that might make it better in "archival" purposes - there is a lossless mode which still achieves higher compression ratios than PNG.
    • As I understand it, matching wavelets with motion compensation is very computationally intensive.

      Snow [doom9.org] (cached [64.233.161.104]) is the most promising attempt.

      Of traditional block-based MPEG codecs, Nero Recode's H.264 implementation is by far the best [doom9.org] for low bitrates.
      For high bitrate archival purposes, XviD [doom9.org] might be better.

    • JPEG 2000 has one feature that might make it better in "archival" purposes - there is a lossless mode which still achieves higher compression ratios than PNG.

      Yes, lossless JPEG 2000 is a reasonable option. I'm not sure any lossy video codec counts as 'archival' storage. Might as well just put published DVDs in a preservation vault. The wide release of movies of movies on DVD has done more for the preservation of movies themselves than anything else in history.

      Still, for a digital archive of the film m

      • And if you can spare the space, a directory with a wav file and a stack of uncompressed TIFF images is even better. Compression formats are complicated to reverse engineer.

        Store .mng + .flac + source code for libmng and libflac, and you don't need to worry about any sort of complicated gnireenigne.

    • Intraframe vs. interframe (Score:5, Interesting)

      by Wesley Felter ( 138342 ) <wesley@felter.org> on Sunday February 06, 2005 @03:12PM (#11591532) Homepage
      For whatever reason (I'm not a video expert) many people prefer intraframe codecs for archival. As you probably guessed, Motion JPEG 2000 just treats each video frame as a still image and compresses it with JPEG 2000.

      Dirac will give much better compression that JPEG 2000, but it also introduces the possibility of interframe artifacts.

  • OK, so when are we going to have support for it? (Score:4, Informative)

    by bersl2 ( 689221 ) on Sunday February 06, 2005 @11:30AM (#11590014) Journal
    https://bugzilla.mozilla.org/show_bug.cgi?id=36351 (no link for obvious reasons) is the bug report, which has been around since April 2000 but has not progressed much due to licensing issues (copyright ones fixed, patent ones not?).

  • Turn it up! (Score:4, Interesting)

    by belg4mit ( 152620 ) on Sunday February 06, 2005 @11:30AM (#11590019) Homepage
    Ummm what about the sound?!
  • Storing digital information on paper is feasible and lots of research efforts have been put into it.

    Storing data on anything magnetic or optical is a bit worrysome. But then, it's not critical data so I guess it doesn't really matter.

    • Paper isn't an optical medium? Punchcards perhaps... I assume you are talking about better tech than my laser printer + scanner which I've estimated could hold about 1MB (which I believe is what the Paper Disk software can do) on a 8.5"x11" page. So what kind of data density can you achieve on paper, and what sort of printer and scanner are you using?

      But, it's an optical format... so why not design materials that can achieve much greater data densities? Like DVDs? Surely it's possible to design something

  • HDs (Score:3, Interesting)

    by eno2001 ( 527078 ) on Sunday February 06, 2005 @11:38AM (#11590076) Homepage Journal
    I could have told people this as they've replaced video tape, and audio tape for me for the past decade. I find them much more convenient, portable and cross platform. I have SCSI drives from 1994 that will still work in a PC (Linux or Windows) or Mac today. They are easy to backup to and restore from. The HD is about as close to perfection as you can get in a storage medium. At least until you get flash drives that can store 1 terabyte at minimum, and have an infinite number of writes. At least a 100 year lifespan.
    • Just out of curiosity, why do you list infinite writes as a requirement for the ideal medium for archival?

      Dan East
      • Actually, I wasn't really thinking archival here. I was thinking more in terms of day to day use, which is how I use them. I have a backup system for the home server that completely wipes the entire backup drive system, reformats it and then copies everything from the live data drive system to it on a daily basis. So I write about 250 gigs per day to the backup system. It's not perfect though. I'm hoping to eventually set something up that works similar to an HP VA7400 series storage array for home use

  • Ars Technica... (Score:4, Interesting)

    by EMIce ( 30092 ) on Sunday February 06, 2005 @11:57AM (#11590189) Homepage
    ...had a guide on capturing analog video [arstechnica.com], said to be the part of a 3 part series, going over each capturing, cleaning, and compressing. Only part I ever came out - Ars do you read slashdot? - I am waiting on the last guides for some advice on how to preserve these rotting home VHS tapes.

    Meanwhile, does anyone else have advice on capturing and cleaning video since we are already talking about compression? What settings are good for capturing and what sort of software exists to clean up VHS and give it the appearance of more clarity? I am using a WinTV card as Ars recommended it.
    • I wouldn't bother with ArsTechnica. For the definitive guide to capturing analog video and digitally archiving it, you would want to read this guide [doom9.org] on Doom9. Plus, they have many other video-related guides on that site and a forum that is second to none in terms of the sheer amount of expertise exhibited by the users there.

  • The move to disk backup continues (Score:4, Insightful)

    by RonBurk ( 543988 ) on Sunday February 06, 2005 @12:20PM (#11590339) Homepage Journal
    People objecting to the use of hard drives for backup miss several points.

    • Yes, they are (somewhat, not excessively) vulnerable to magnetism. But optical discs are vulnerable to light, fingerprints, chemicals, etc.
    • Optical discs continue to lag far behind in capacity. So, in the land of audio/video backup, the choice is between a single hard disk, or dozens of optical discs. The risk of failure of multiple optical discs is amplified by the increasing number of discs.
    • Bandwidth is another issue. Though hard disk bandwidth lags behind the growth of hard disk storage, optical disc bandwidth lags even further behind.
    • Restore time is another issue. You can line up a bunch of optical disc drives and try to make all your data available at once, but you're probably never going to get the restore speed of solutions like Massive Arrays of Inactive Disks [backupcritic.com].


    There will always be multiple backup solutions, but the biggest trend continues to be towards using hard disks for backup. When your data files are enormous (such as with audio/visual data), HDD backup is even more attractive.

  • More JPEG-2000 stuff (Score:2, Informative)

    by fnord_uk ( 842775 )
    There is more to jpeg2000 than a compression scheme offering scaleable quality and resolution within a single losslessly compressed file. There is also the interactive delivery mechanism offered by the JPIP protocol. Now there is something really useful...

  • Yawn, another technology of the day... (Score:4, Insightful)

    by HockeyPuck ( 141947 ) on Sunday February 06, 2005 @01:11PM (#11590714)
    All this is is a method to line some guy's pockets. I'm sure the tape guys are gonna say, use XYZ type of tape. The disk guys are gonna say disk.

    What makes this guy think that the interface to the HDD is going to be around in X years?

    PC's have only had two dead (non-(e)IDE/ATA) interfaces, the ESDI and the ST506/ST-412 interfaces.

    But what if you were trying to find a computer with IPI (1960s mainframe) interface.

    The Fed gov't has this problem with trying to find parts for their old 8/9track tape drives..

    Here's a good list of all the HDD interfaces over the years: http://www.i-t-s.com/corporate/terms.html [i-t-s.com]

    Stick with microfiche, film, that way we don't have to pay some vendor $$$/yr to keep alive a dead technology or pay some other vendor $$$/media to move them from old to new media.

    • This is not an issue because you never remove the hard disk from its computer. When the computer becomes obsolete, you buy a new one (with new disks), and copy the data over.
  • TSIA. It would probably be a bad idea to start publishing material with it until the patent expires.

  • .doc? (Score:3, Funny)

    by b1t r0t ( 216468 ) on Sunday February 06, 2005 @04:23PM (#11592004)
    For people concerned with the preservation of "data", they've sure picked an interesting format to write about it in.

  • GIGO and the "born digital" problem (Score:3, Insightful)

    by retiarius ( 72746 ) on Sunday February 06, 2005 @07:50PM (#11593129)
    indeed, lossless for archival preservation is the
    only way, as it fits the basic rule of art restoration
    technology -- never apply "improvements" which
    cannot be reversibly undone to take advantage
    of future science.

    ironically then, the lossless format doesn't matter.

    however, at least for the instant case of dance video,
    the likely input (a myriad of digital tape formats)
    is hopelessly neanderthal -- anything having to do with DV,
    or MPEG, or even ATSC HDTV already tosses away much
    color information. (4:1:1, 4:2:0, and 4:2:2 colorspace is embarrassing
    to preserve "losslessly".) ditto for temporal
    info, with interlacing being the culprit. even film at
    24fps just will not cut it for motion such as dance.

    so here's to better camera technology, whether it's
    10- or 12-bit 4:4:4 RGB, or something like
    carver mead's foveon made swift.

Slashdot Top Deals

Children begin by loving their parents. After a time they judge them. Rarely, if ever, do they forgive them. - Oscar Wilde

Close