Forgot your password?
typodupeerror
Media Music

Analyzing YouTube's Audio Fingerprinter 116

Posted by timothy
from the streeeeeehtch-thiiiiingsss-ouuuuuut dept.
Al Benedetto writes "I stumbled across this article which analyzes the YouTube audio content identification system in-depth. Apparently, since YouTube's system has no transparency, the behaviors had to be determined based on dozens of trial-and-error video uploads. The author tries things like speed/pitch adjustment, the addition of background noise, as well as other audio tweaks to determine exactly what you'd need to adjust before the fingerprinter started mis-identifying material. From the article: 'When I muted the beginning of the song up until 0:30 (leaving the rest to play) the fingerprinter missed it. When I kept the beginning up until 0:30 and muted everything from 0:30 to the end, the fingerprinter caught it. That indicates that the content database only knows about something in the first 30 seconds of the song. As long as you cut that part off, you can theoretically use the remainder of the song without being detected. I don't know if all samples in the content database suffer from similar weaknesses, but it's something that merits further research.'"
This discussion has been archived. No new comments can be posted.

Analyzing YouTube's Audio Fingerprinter

Comments Filter:
  • music ip? (Score:5, Informative)

    by FredFredrickson (1177871) * on Wednesday April 22, 2009 @02:13PM (#27677041) Homepage Journal
    There's the open-source library - libOFA - developed by Music IP (http://code.google.com/p/musicip-libofa/) which happens to create PUIDs on the first 135 seconds of audio in a track. It's used in the music-IP mixer (for mood mixes) but is also used by music database projects such as MusicBrainz.

    From what I've seen, it's pretty decent audio fingerprinting, but I'm sure would be subject to the same limitations- if you remove the first 30 seconds of a clip- it would produce a very different fingerprint.

    There's no reason to believe youtube isn't using this library or a derivative. There's also no reason to believe this result isn't intended. If the first 30 seconds of a song are missing- maybe that makes youtube confident that it could be considered fairuse.

    Either way, I could imagine creating a fingerprint based on different sections of a song has the same problems doing an MD5 hash would- each fingerprint would be entirely different. If you don't just compare bit-to-bit, it'll be impossible to catch ALL permutations. And the fact is, that's a lot of computing power anyhow.
    • by Jurily (900488)

      If the first 30 seconds of a song are missing- maybe that makes youtube confident that it could be considered fairuse.

      Nope. The principle of CYA says that if there's any possibility of a lawsuit, nuke it from orbit.

    • Re: (Score:3, Interesting)

      by Joe Snipe (224958)

      If the first 30 seconds of a song are missing- maybe that makes youtube confident that it could be considered fairuse

      Or if 30 seconds of additional blank footage were tacked on to the beginning?

      And FWIW, there is a very valid reason for assumming they aren't using this fingerprint system: They already had their own in-house created system that they based off of their thumbnail maker program. It is also limited to within 30 sec of a clip if I recall.

    • by Nova77 (613150)

      There's also last.fm fingerprint library which is open source: svn://svn.audioscrobbler.net/recommendation/MusicID/lastfm_fplib

    • Either way, I could imagine creating a fingerprint based on different sections of a song has the same problems doing an MD5 hash would- each fingerprint would be entirely different. If you don't just compare bit-to-bit, it'll be impossible to catch ALL permutations. And the fact is, that's a lot of computing power anyhow.

      To be honest, I'd be fairly surprised if they used a method that boiled down to a hash for exactly the problem you point out. I would make a bet they either currently use, or will use, a

  • by Rayeth (1335201) on Wednesday April 22, 2009 @02:16PM (#27677083)
    I thought the purpose (however misguided it may be) was to prevent people from uploading copyrighted songs/music videos and re-mixing them. So if I only use portions of the song that aren't in the first 30s I'm home free? That seems silly, the system must still be under refinement or is only there to stop the most blatant offenders.
    • by tepples (727027)

      So if I only use portions of the song that aren't in the first 30s I'm home free? That seems silly, the system must still be under refinement or is only there to stop the most blatant offenders.

      I'm inclined to believe the latter. If a video doesn't use more than about 30 seconds of a recording at a time, it's likely that the video's author attempted to use the work fairly [wikipedia.org]. I guess my video [google.com] got flagged because I opened with a vocal-cut version of one of the songs on which I was commenting.

  • Whew! (Score:5, Funny)

    by Serenissima (1210562) on Wednesday April 22, 2009 @02:17PM (#27677093)
    It's a good thing no one at Youtube reads Slashdot. Otherwise they might come up with a fix! So, everyone keep this a secret! SHHHH!
    • Re: (Score:3, Interesting)

      by Tsunayoshi (789351)

      There-in lies the rub with the "all information should be free" mindset...EVERYONE gets to look at it.

      I think that is a feature, not a bug.

      • Re: (Score:1, Funny)

        by Anonymous Coward

        Which leads to the metaphysical question bugging us all: does a WHOOSH count as information?!

  • by eclectro (227083) on Wednesday April 22, 2009 @02:21PM (#27677129)

    Here's an idea. Start out the video with a useless narrative for the first thirty seconds "blah blah blah skip until :30 and ignore this intro blah blah" then start the music. That way everybody is happy. All google employees are too elitist to read slashdot, right?

    • by spydabyte (1032538) on Wednesday April 22, 2009 @03:08PM (#27677615)
      Sounds like packaging copyright material between thousands of papers and delivering it in PDF format to my university printing service to print out all my textbooks for free... except with less wasted paper.
    • by DriedClexler (814907) on Wednesday April 22, 2009 @03:20PM (#27677763)

      Heh. I think people have already tried that.

      Hi, I'm am amateur movie critic. Today I'm going to show you an example of poor film-making. blah blah blah ...

      *Plays entire Star Wars: Episode I*

      So, as you can see by the [cinematography jargon] and [screen writing jargon], this movie sucked and I hope you learn from it in making your own movies.

      One week later:

      "No! You can't take down my video. This is CLEARLY fair use, since I have OBVIOUSLY used it for educational commentary, and the entire clip was VITAL for showing how much Episode I sucked."

    • by Locklin (1074657) on Wednesday April 22, 2009 @03:23PM (#27677791) Homepage

      Here's an idea. Start out the video with a useless narrative for the first thirty seconds "blah blah blah skip until :30 and ignore this intro blah blah" then start the music.

      Like on the radio?

    • Re: (Score:3, Insightful)

      by Hurricane78 (562437)

      ...and then you realize, that YouTube changed the algorithm, and that its compression makes your song sound so shitty anyway, that you actually want all the uploads to be taken down. ^^

    • Re: (Score:3, Insightful)

      by Anonymous Coward

      I see where you're going with this: put an advertisement at the start :-)

    • Re: (Score:3, Insightful)

      Even better - upload the video backwards.
      Someone should make backward youtube plugin for firefox. It even might autodetect backward songs and play them properly.

    • Why not encrypt the audio and have users download a special player that will decrypt at runtime. End user hears the intended audio and youtube only hears noise.
  • pHash (Score:5, Interesting)

    by b1ng0 (7449) on Wednesday April 22, 2009 @02:21PM (#27677135)
    This seems like a good time to pump my own open source project: pHash [phash.org]. pHash is a perceptual hashing library that computes hashes for audio, video and image files, with text and PDF hashing coming soon. We use an algorithm similar to YouTube's audio fingerprinting method but we do not only take into account the first 30 seconds. Although, it's impossible to tell from this basic test whether their algorithm truly only looks at the first 30 seconds, or if the algorithm considers them to be different audio files. If the song is only 1 minute in duration, and 30 seconds is blank, is that really the same audio file as the full 1 minute version? At some point the audio files are not really the same anymore, although the perceptual hashes should be somewhat close to each other. Please give pHash a try. We could use some feedback from the OSS community and would appreciate it greatly.
    • Re:pHash (Score:4, Interesting)

      by FredFredrickson (1177871) * on Wednesday April 22, 2009 @02:26PM (#27677191) Homepage Journal
      Out of curiosity, how well could pHash be used to find similar songs from a list of songs? Maybe not actually similar, but similar sounding (or same mood)...?

      Any ideas how one would go about doing this sort of thing?
      • Yeah (Score:4, Interesting)

        by Brain-Fu (1274756) on Wednesday April 22, 2009 @03:26PM (#27677827) Homepage Journal

        Music Genome [pandora.com]

        • Re:Yeah (Score:4, Informative)

          by Anonymous Coward on Wednesday April 22, 2009 @04:42PM (#27678853)

          Dear Pandora Visitor,

          We are deeply, deeply sorry to say that due to licensing constraints, we can no longer allow access to Pandora for listeners located outside of the U.S. We will continue to work diligently to realize the vision of a truly global Pandora, but for the time being we are required to restrict its use. We are very sad to have to do this, but there is no other alternative.

          If you believe we have made a mistake, we apologize and ask that you please contact us at pandora-support@pandora.com

          If you are a paid subscriber, please contact us at pandora-support@pandora.com and we will issue a pro-rated refund to the credit card you used to sign up. If you have been using Pandora, we will keep a record of your existing stations and bookmarked artists and songs, so that when we are able to launch in your country, they will be waiting for you.

          We will be notifying listeners as licensing agreements are established in individual countries. If you would like to be notified by email when Pandora is available in your country, please enter your email address below. The pace of global licensing is hard to predict, but we have the ultimate goal of being able to offer our service everywhere.

          We share your disappointment and greatly appreciate your understanding.

        • Re: (Score:2, Informative)

          by ausekilis (1513635)
          Perhaps a better link for information: Music Genome Project [wikipedia.org]. A little more detail from Pandora's blog [google.com].
      • Re:pHash (Score:4, Interesting)

        by ash211 (1177227) on Wednesday April 22, 2009 @03:44PM (#27678049) Homepage

        The problem you're describing is known in the Music Information Retrieval (MIR) world as content-based recommendation (CBR). There are a number of ways to do it, but they're all based on measuring similarity.

        The idea is that people perceive songs as similar based on the characteristics they have, which are termed features. By representing a song's features in a model you can compare the models to see how "distant" they are, and then choose songs from a set that are least-distant. The work that my research group is pursuing represents songs based on timbral features (MFCCs) and rhythmic features (bpm, pulse clarity, syncopation, etc).

        If you're interested in the approach, see http://paragchordia.com/research/cbr.html [paragchordia.com]

    • Re:pHash (Score:5, Funny)

      by Anonymous Coward on Wednesday April 22, 2009 @02:30PM (#27677229)

      fucking pHashist...

    • by bcrowell (177657)

      This seems like a good time to pump my own open source project: pHash. pHash is a perceptual hashing library that computes hashes for audio, video and image files, with text and PDF hashing coming soon.

      Cool! The history of these algorithms, and of databases like CDDB, is kind of depressing to an open-source guy like me. It's great to see someone doing this as an open-source project. How stable is the algorithm? If I compute a pHash today, will it still be compatible with pHashes computed next year? Any pla

  • Tragic (Score:2, Insightful)

    by dedazo (737510)

    That cool tech like this is being used to prevent "piracy" instead of something more useful.

  • by American Terrorist (1494195) on Wednesday April 22, 2009 @02:28PM (#27677205)
    The big issue here is what Lessig talked about years ago: Free Culture

    Then a car commercial parody I made (arguably one of my better videos) was taken down because I used an unlicensed song. That pissed me off. I couldn't easily go back and re-edit the video to remove the song, as the source media had long since been archived in a shoebox somewhere. And I couldn't simply re-upload the video, as it got identified and taken down every time. I needed to find a way to outsmart the fingerprinter. I was angry and I had a lot of free time. Not a good combination.

    The guy who wrote TFA is upset that his largely unviewed videos didn't pass an automated test.

    My beef with the system is that when culurally significant videos such as the Chinese "Caonima [youtube.com]" get taken down because the song violates some copyright of a company I've never heard of on a song I've never in a million years think of buying.

    Hope that link works, I had to copy it from Google since I can't even access Youtube anymore here in China.

    • It's 2:30 AM, srry typos
    • Re: (Score:2, Insightful)

      by Anonymous Coward

      My beef with the system is that when culurally significant videos such as the Chinese "Caonima" get taken down because the song violates some copyright of a company I've never heard of on a song I've never in a million years think of buying.

      So copyrights only apply to companies you've personally heard of and it's a song you'd buy? That's pretty stupid.

      • I'm drunk and tired waiting for the English Premier League games to come on, no claims to being super insightful right now. I just meant that it pisses me off even more than it already does. If that makes any sense.
      • So copyrights only apply to companies you've personally heard of and it's a song you'd buy? That's pretty stupid.

        I believe the point wasn't the significance of the company filing the complaint, but the content of the video being removed. If a video has a significant contribution, a larger company might be more willing to let a small infringement slide on the basis of good will, since they have other sources of income. A small company would be more likely to be zealous, since even small infringements represent a significant portion of potential income. Of course, that paradigm hardly applies in all situations, probably

  • Research? (Score:2, Insightful)

    by mi (197448)

    but it's something that merits further research.

    Why exactly does it merit any research? This is not riddle posed by Nature — people devised this device (ha-ha), and know all the answers perfectly already, they just don't want to tell you. You are not advancing scientific progress by figuring out somebody's scheme.

    You may be advancing your own knowledge and skills, but calling it "research" has no more merit, than paparazzis' "research" into celebrities' lives...

    • So, anthropology is not research?

    • Re:Research? (Score:5, Insightful)

      by radtea (464814) on Wednesday April 22, 2009 @03:33PM (#27677921)

      This is not riddle posed by Nature

      This is one of the wonderful things about science: it doesn't matter where the puzzle comes from, the same techniques work to solve it.

      Reverse engineering of this kind is one of the most useful areas of applied science, and it is as much research as any other area of scientific enquiry. It is frequently the case that there are many ways to find the answer to a puzzle, and this guy has chosen one of them based on the resources he has available. More power to him for demonstrating how good science can be used to discover what others want to keep secret.

    • Why exactly does it merit any research? This is not riddle posed by Nature... You are not advancing scientific progress by figuring out somebody's scheme.

      Good grief. If you're trying to find out something that you can't just go look up at the library, and you're forming and testing hypotheses to do so, that could reasonably be called "research". Don't be so pedantic. :-)

      (Anyway, it may well turn out that Nature is just stuff that someone already knows all the answers perfectly to and just doesn't want to tell you.)

    • by billcopc (196330)

      The "research" only leads to working around the content filters, posting material the site operators explicitly do not want.

      It would be more interesting if there was a productive application for this knowledge. Putting Rihanna songs on Youtube does not fit my idea of "productive".

      • It would be more interesting if there was a productive application for this knowledge. Putting Rihanna songs on Youtube does not fit my idea of "productive".

        Why is it that so many people are so ready to condemn others because of their own lack of imagination?

        My niece is a working print model and aspiring actress who has starred in a few very high profile music videos (the kind that get nominated for MTV's annual awards) and been featured in a few national commercials. She has put together a youtube channel to promote her career - the goal was to include a copy of very video work she has been in and title it so that her name was explicitly associated with the v

        • by mi (197448)

          Turns out that most of the blocked ones are already posted on youtube via official channels with very poor video quality (letterboxed and pillarboxed for example).

          So, the copyright belongs to someone else, and they chose to use a lower-quality version. I don't see, how this gives your niece — although she appears in the video, she is not the owner of it — the right to post her own version...

          Back to the point about advancing human progress, I don't think, a particular fashion model's success or

          • You are clearly unfamiliar with how hollywood works and the concepts of reels and portfolios.

            Back to the point about advancing human progress, I don't think, a particular fashion model's success or failure have any effect on it...

            Did I say she was a fashion model? Or are you just projecting your own pejorative attitude about the entertainment industry? Ironic you are so dismissive of the industry and yet so quick to defend an entirely bogus preconceived notion about some of their rights.

            • Did your niece receive ownership of the various content as compensation or put up the money for the production or something along those lines? I doubt it but there's always that chance.

              What I find most likely, however, is that she was hired to perform a service and compensated according to a contract she agreed to. It seems fairly unrealistic that distribution rights were part of that compensation, and if that is indeed the case, she shouldn't be distributing no matter how personally beneficial she would

              • I would certainly find it lucrative if I could resell some of the software I've written under work-for-hire contracts, but I agreed to the conditions involving my employers retaining those rights in exchange for compensation.

                Damn! Yet another poster without a clue as to how portfolios and reels are used within the entertainment and advertising business. Why do you guys think you know so much about something you clearly have zero knowledge of? And why are you all so fucking high and mighty about it too? Repugnant? Jesus!

                Here's your clue - it is standard procedure to send copies of your reel - i.e. a dvd with significant examples of your prior work for hire to agents (when you don't yet have an agent) and casting directors.

    • by Eil (82413)

      Why exactly does it merit any research? This is not riddle posed by Nature -- people devised this device (ha-ha), and know all the answers perfectly already, they just don't want to tell you. You are not advancing scientific progress by figuring out somebody's scheme.

      So as long as somebody knows exactly how the system works, that's good enough for you? Fine, but that's not how all of us are wired [catb.org]. Google's knowledge of their audio fingerprinting scheme is useless to me if I want to know how it works and th

  • by thomasdz (178114) on Wednesday April 22, 2009 @02:31PM (#27677253)

    And who fingerprints the analyzers who analyze the analyzers?

    • Simple. Make it a recursive loop.

      In Haskell you would do it like this:

      import Helpers (loadAudio,Analyzer)

      mkAnalyzer a =
      let aa = Analyzer a
      in aa : mkAnalizer aa

      analyze (audio:[]) = audio
      analyze (analyzee:as) = (analyze as) analyzee

      main = do
      audio <- loadAudio "someAudio.wav"
      let safeAnalyzersChain = mkAnalyzer audio
      analyze safeAnalyzersChain

      Of course this is verbose and not optimized, because I am still a Haskell amateur. And of course this

  • Now we are going to have a ton of 30sec Ops of soundless Text about how cool the author is now.
    • You misunderstand. It's not the video. It's the song. So cut in after the first verse, even right at the start of the song, and it's likely to pass inspection.
  • There is also an option to claim fair use (although I think it uses different words) after it identifies a song. I did this for an artwork of mine on Youtube that included the first 30 seconds of a Cure song, and the video stayed. I really do think that in my case, it was fair use. But if you're just trying to upload an old Pearl Jam video, then this probably won't help.
  • by Knave75 (894961) on Wednesday April 22, 2009 @02:46PM (#27677403)

    An unfortunate result. The last 30 seconds of most songs are not usually as interesting as the first 30 seconds.

    I wonder if he tried mangling the first 30 seconds at all. For example, keep the first 5 seconds, mess up the 6th and 7th seconds, and then continue on. Or perhaps adding in a base line that would be hard to hear. Or something at the high end of the audio frequency spectrum, to annoy all those teenagers while I listen to my free music in peace.

    • And to make your animals go crazy? And I mean pathologically crazy, as in If it were a human, it would need strong medication an a padded room.".

    • ...just to pass some filter, then you must think the song's very well-worth listening to.. which, to me, implies it ought to be worth buying.

      If it's just some background music piece - I dunno, try another song.. plenty of royalty-free and even completely free ones (nope, they're probably not in the billboard top 100 right now - so sorry).

      I'm more curious about the cases where there really IS fair use involved.. what happens in those cases.. do you get to hit a checkbox saying "I believe this is fair use, pl

    • by syousef (465911)

      I wonder if he tried mangling the first 30 seconds at all. For example, keep the first 5 seconds, mess up the 6th and 7th seconds, and then continue on.

      If you're going to do that, you're going to ruin the song. If that's what you want to do it would be much more effective to sing it at the top of your lungs off key in the shower and put that in as your soundtrack instead. As a bonus, you're still violating copyright.

  • This hole doesn't really indentify a hole in the technology itself, just in the implementation. I'm more interested in hearing some audio that sounds the same while defeating the fingerprinting scheme. Much more interesting.

    • by maxume (22995)

      Hearing audio that sounds the same doesn't seem that interesting to me.

      Being able to generate it, on the other hand...

  • I've noticed that is picks up vocals very well. There are some songs on Youtube that have had their individual parts lifted from guitar hero and uploaded for people to learn each part. Instrumental parts are fine, as are most full instrumental songs. However, vocal parts ARE picked up by the fingerprinter. (Search Muse acapella for examples)
  • MusicBrainz works exactly as described.

  • by bipbop (1144919) on Wednesday April 22, 2009 @04:05PM (#27678335)
    Youtube uses Audible Magic's audio fingerprinting technology, which is based on this patent by MuscleFish: http://patft.uspto.gov/netacgi/nph-Parser?Sect1=PTO1&Sect2=HITOFF&d=PALL&p=1&u=%2Fnetahtml%2FPTO%2Fsrchnum.htm&r=1&f=G&l=50&s1=5918223.PN.&OS=PN/5918223&RS=PN/5918223 [uspto.gov]
    • Re: (Score:2, Informative)

      by BabyDuckHat (1503839)
      This is actually very useful information for someone looking for ways to defeat the filter, in that is lists the features of the audio that are used for generating the fingerprint. A successful work-around would most likely require modifications to several aspects of the signal.

      From the patent:

      The feature vector thus consists of the mean and standard deviation of each of the trajectories (amplitude, pitch, brightness, bass, bandwidth, and MFCCs, plus the first derivative of each of these). These number
  • I met the founder of Auditude.com, a competitor to the company that supplies audio fingerprinting for YouTube. Fascinating guy, but even more fascinating technology. They claim they can identify any clip as short as 5 seconds from any portion of the original recording.

    You can test them out on myspace, I'd be interested to see how well they stack up in real world tests to YouTube's provider.
  • The real tragedy is that a rickroll is no longer as easy. You never know how long a link might work.
  • Don't submit videos with music by the big labels. Using creative commons music or music from labels who approve of the free advertising will simultaneously keep you from having your videos taken down and provide more visibility to non-RIAA-label artists, helping to make their cartel useless.

    • I once uploaded a vid with music by Ulrich Schnauss, published by Domino Recording Co. I don't know if they're a big label or not but they were cool with me using the soundtrack because they get to post an iTunes 'buy now' button and an amazon.com 'buy now' button on my movie's page.

      On the plus side, an enlightened record company can use this as a means of getting free advertising and driving sales as long as they don't find the video objectionable.

      On the negative side, who's to say that all record compani

  • by illectro (697914) on Wednesday April 22, 2009 @05:54PM (#27679919)

    imeem have been doing this for the last few years, and they don't use audible magic, they used the Snocap fingerprint system which apparently was good enough for them to buy Snocap. Their business model has always been built around using the content identification system to make sure the right people get paid for audio played on the site.

    imeem is primarily used by people uploading and sharing audio, so using an audio fingerprinting system seems more appropriate than youtube relying on an audio fingerprinter for video content.

  • If they start using whatever Shazaam uses, we're screwed. In any case I'm sure this is the start of an arms race in which the fingerprinter keeps getting more and more elaborate to counter the effects of people trying to fool it.

  • <RIAA>
    Any use that doesn't result in me getting obsene amounts of money is NOT FAIR!
    </RIAA>
  • People .. stop using youtube and build your own video streaming server. All you need is 10mbit / 20mbit upload, some space and http://www.longtailvideo.com/players/jw-flv-player/ [longtailvideo.com] that can be downloaded here: http://netsky.org/SpcVideo/flvplayer.zip [netsky.org]. I've created my own test server and it does the job better than youtube. Check here: http://netsky.org/SpcVideo/sea1.htm [netsky.org] This is a concert i recorded from a TV over a satelite. If you create your own server then no one will delete your videos. :)
  • http://foosic.org/ [foosic.org] this is the most accurate and reliable fingerprinting algorithm I've used.
  • From TFA:

    It's downright dumb: Wrap your heads around this. When I muted the beginning of the song up until 0:30 (leaving the rest to play) the fingerprinter missed it. When I kept the beginning up until 0:30 and muted everything from 0:30 to the end, the fingerprinter caught it. That indicates that the content database only knows about something in the first 30 seconds of the song. As long as you cut that part off, you can theoretically use the remainder of the song without being detected. I don't know if a

  • you rock. anything to bring back the excitement that was once youtube. fuck wmg.

"Just Say No." - Nancy Reagan "No." - Ronald Reagan

Working...