Video Inpainting Software Deletes People From HD Video Footage 124
cylonlover writes "In a development sure to send conspiracy theorists into a tizzy, researchers at the Max Planck Institute for Informatics (MPII) have developed video inpainting software that can effectively delete people or objects from high-definition footage. The software analyzes each video frame and calculates what pixels should replace a moving area that has been marked for removal. In a world first, the software can compensate for multiple people overlapped by the unwanted element, even if they are walking towards (or away from) the camera."
Summary Fail (Score:2, Informative)
Background has to be static for it to work.
Nevertheless, an interesting accomplishment.
Re: (Score:2)
Background has to be static for it to work.
Nevertheless, an interesting accomplishment.
Surely the person has to move far enough across the static background to reveal at last in one frame what's behind the person? I mean, if I'm standing in front of a dwarf for the whole duration of the video, how is the software to know about the dwarf?
Re: (Score:3)
Re: (Score:1)
It doesn't know, but if this works like Photoshop's content aware fill, it can convincingly fake the rest of the wall. That being said, it's my experience that older, manual methods, usually work better.
Yeah, figured as much but, it's early here :)
Re: (Score:2, Insightful)
That being said, it's my experience that older, manual methods, usually work better.
Yes, but how fast can you manually remove someone from 30 seconds of video? You're talking about manually touching up over 700 pictures. How many can you do in a day? This software would probably do that 30 seconds in 30 seconds.
Re:Summary Fail (Score:5, Informative)
If you look at the video, though, the background doesn't have to be static. Objects moving over other moving objects can be removed as well. But, yeah, they have to be visible at some point.
Re: (Score:1)
But, then, how is anyone else to know about the dwarf. From the viewer's perspective the dwarf doesn't exist.
True, unless the viewer at some point can examine the scene. (Photos and videos from a multitude of other sources, especially earlier ones. Granted, this works better with backgrounds less ambulatory than dwarfs)
For that matter, dwarfs might not even exist at all!
Might. All you know is that you haven't observed one :p
Comment removed (Score:5, Funny)
Re: (Score:1)
Did his name happened to be Hank?
Re:Summary Fail (Score:4, Funny)
Re: (Score:1)
Yes, well, I altered the reference. Pray I don't alter it any further...
Re: (Score:2)
Zoom and inhance.
Re:Summary Fail (Score:5, Informative)
Background has to be static for it to work.
Nope [youtube.com]
Re: (Score:2, Interesting)
Re:Summary Fail (Score:5, Informative)
Re: (Score:2)
Why do you claim it's impossible when that's the very thing that's demonstrated in the first few seconds of the video link you responded to?
I know people don't RTFA, but not to look at the link of the post you're responding to?
Re: (Score:2)
Using the free Windows software from Microsoft, it is trivial for me to stitch many photos together to make a stunning panorama photo. For example I once sat and made one while a boat pulled in to dock alongside me, to pick up a few passengers and then it pulled away. I probably shot about a dozen photos during this short time, all the while the boat was directly within my panorama. The resulting photo looks amazing, like there's 3-4 boats, (with the same people!). In fact I think I had to shoot quickly to
Re: (Score:2)
Background has to be static for it to work.
Nope [youtube.com]
That's still a static background, just the camera position and field of view isn't static. The view of the background isn't static, but the background is. If there were moving people in the background, and you wanted to remove someone in between, you've got a problem (because you don't know what the people were doing when they were blocked).
Re: (Score:2)
How about the swinging arm of the guy behind the bench? That is background that is not static, yet it is redrawn with the arm motion neatly interpolated.
Ofcourse a lot of the background should be static, but it nearly always is unless you're looking at movie SFX or sea footage.
A boon which is sure to send Starwars fans into... (Score:5, Funny)
Researchers have developed video inpainting to remove the character Jar Jar Binks from the Star Wars Prequals.
Re: (Score:2)
This would be a cool usecase. I dont mind JarJar so much though, theres lots of stranger things in starwars, but i'd make a good technology demo.
Re: (Score:2)
They'd have to develop a similar algorithm for audio though.
I think I'd find a mute Jar Jar far easier to tolerate than an invisible one.
Re: (Score:1)
Almost like the Samsung Galaxy S4? (Score:2, Interesting)
Don't the new Galaxy S4 have a similar feature, if I read correctly? Although only for photos.
Re: (Score:2)
I'd really like it to then allow users to chose a specific thing that is allowed to move, to make cinematic gifs.
Re: (Score:1)
Successive approximation dejitter followed by temporal mode filter. I could throw that together in twenty minutes. Making it run on mobile phone hardware is rather more difficult.
Re: (Score:2)
Re: (Score:2)
To argue there there is no 'magic algorithm' to what the S4 does. I'm no programming genius, just a dabbler who can knock out a few functions in C. If I can do it, it isn't hard.
Reflections (Score:5, Interesting)
I liked the fact that you could still see the pedestrians in the reflections of the display window in the video of the musicians, even though they had been erased from the front end. Like the vampire test, but the other way around.
Re:Reflections (Score:5, Funny)
Re:Reflections (Score:5, Informative)
Have these people never seen Rising Sun? This was a plot point 20 years ago.
Re:Reflections (Score:5, Funny)
Re: (Score:3)
Have these people never seen Rising Sun?
I've not only seen that book, I also bothered to read it. As usual, it was better than the adaptation.
oh, great (Score:5, Funny)
So combine this tech with Google Glass and identify people you just don't want to see ever again, and you may end up walking right into them without even knowing.
Re: (Score:2, Interesting)
Rainbows End, by Vernor Vinge. What you described is just the beginning.
Re: (Score:1)
Re: (Score:2)
Already done in a Charlie Stross novel... people are walking the streets but appear as pixellated blurs... Anonymity in the crowd taken to the extreme... ..
Sounds like the scramble suits in Philip K Dick's A Scanner Darkly.
Re:oh, great (Score:5, Funny)
So combine this tech with Google Glass and identify people you just don't want to see ever again, and you may end up walking right into them without even knowing.
"Hey dude, what's u...Ow! Dude, what the hell, you just ran into me!"
"Oh hey, sorry about that. I had you flagged as spam. Didn't even see you there."
Re: (Score:1)
I would have to flag some as ham, some as bull, some as chicken.
If it's possible to replace them with avatars, they would all look like supermodels. All of a sudden it is just me and supermodels around me. Actually I begin to like the idea. But they would have to be dressed or I'd be crushed by incoming traffic.
For programmers in offices they could replace colleagues with naked supermodels. I wonder what that would do to productivity.
Re: (Score:2)
Do you fleece the sheep, or do you butcher them up for dinner?
Re: (Score:2)
Imagine the spam restaurant, a visual spam filter, and Google Glass. Enter the restaurant, order food, eat everything you see. "But you've barely touched the food!" "What do you mean, I've eaten everything!" Epic.
You could also get rid of the Vikings, if you're not that much into the Nordic stuff. (Just don't do that when they turn violent: You'll never see it coming.)
Re: (Score:2, Funny)
Oh when you're eyes get hacked and you're unable to see people who don't want to be seen.
Re: (Score:2)
Re: (Score:2)
So combine this tech with Google Glass and identify people you just don't want to see ever again, and you may end up walking right into them without even knowing.
We all know it would not removing them but likely replacing them with some attractive person. For /. probably w/o clothes, or if for Reddit, they'd be turned into cats.
Re: (Score:1)
I wonder what that would do to programmers at work. [slashdot.org]
What's next? (Score:5, Funny)
Ten years ago, I predicted a "nudie button," which, instead of removing people from live video, would simply remove their clothing (through interpolation). I do not endorse the use of such a button on your TV's remote control, I merely predict its future existence.
Re: (Score:2)
I endorse this!
Re: (Score:2)
There is an pseudo-app for photo's: Nudifier
https://itunes.apple.com/us/app/nudifier/id554023264?mt=8 [apple.com]
Someone just needs to take it to the next step. :-)
Re: (Score:1)
Someone just needs to take it to the next step. :-)
Chatroulette?
Re:What's next? (Score:5, Interesting)
What you proposed isn't that far-fetched, as I ended up having to contrive and implement the equivalent of this, i.e., passive, automated estimation of body shape under clothing, either from a single image or from multiple video frames, for some work I did in action recognition that required a fairly accurate representation of the person's proportions. Others, e.g., A. O. Balan and M. J. Black, "The naked truth: Estimating body shape under clothing," in Proceedings of the European Conference on Computer Vision (ECCV), 2008, pp. 15–29, have come up with solutions too.
Re: (Score:2)
Make sure always to add inserts under your clothes in the most unexpected (and some obvious) places. Estimate this, mofo.
Re: (Score:1)
Sounds like the best research project ever! Tell me, when you set up video experiments, what do you use as a control group? Do you just tape a bunch of test subjects walking around naked?
Wait a second - do you pick your own test subjects, or do you have to take anyone who volunteers?
JK.
Re: (Score:2)
As you noted, the project was fun to undertake, even though it was only a sub-component to a much larger endeavor. I may yet go back, visit it, and submit an extension as a nice stand-alone article.
To answer your questions, though, I relied on a pool of around seventy subjects, equally distributed across genders and with a tri-modal distribution for age, many of whom were nudists that had heard about the data collection through some friends of mine. I also had a couple of adventurous fellow students and pe
Walmart (Score:1)
...make sure you turn it off before going into a Walmart or KFC etc.
If that happens, I predict some nerds with Google Glasses gouging their eyes out...
Re: (Score:3)
Re: (Score:1)
Re:Conspiracy Theorists? (Score:5, Insightful)
Re: (Score:2)
You got that wrong. Now they will say see, we told you this existed. But if this is what they are willing to show us, think how more powerful the government version is that they won't show us? They can remove you from one video walking down the street, and put you in the same scene last week (ATM time stamp) robbing someone!
That isn't doing the conspiracy theorists justice!
These days there is no need to go through all the work of framing you for a crime and then inventing evidence of it. Oh no.
Instead, they will simply say you broke a secret law, but can't tell you what that law is (for national security after all), but they DO have evidence, but can't show what that is for more national security reasons.
A judge will agree that is acceptable, and without seeing the evidence will say it is enough to prove your guilt, and thus
Re:Conspiracy Theorists? (Score:5, Insightful)
Not sure if you're making fun of conspiracy theorists or making fun of people who call others conspiracy theorists?
What you describe was put into law by the 2012 NDAA. There isn't even a "judge" required.
Government says: "You're a terrorist!" The evidence is secret so you have no right to examine or challenge it. You have no right to legal counsel, no right to ever see a judge or jury, and it's off to prison you go for as long as the government says. Or maybe they just kill you, which they also claim the power to do.
And hilarity ensues (Score:5, Funny)
Re: (Score:2)
Heheh, I want to see the video with a swarm of hang-gliders removed. Or Indy-car racing sans cars "And as we enter the final lap the head of Al Unser Junior pulls into the lead..."
so, an end to surveillance cameras? (Score:3, Interesting)
If we extrapolate this, perhaps we won't be able to trust video as evidence any longer, so there's no reason to have all these surveillance cameras around.
Re:so, an end to surveillance cameras? (Score:4, Insightful)
If we extrapolate this, perhaps we won't be able to trust video as evidence any longer, so there's no reason to have all these surveillance cameras around.
The corrupt legal system will never allow that amount of common sense to creep in.
And we're slowly being made irrelevant to do anything about it.
Re: (Score:3)
It's already well established that eye witness testimony is highly unreliable [schneier.com] and it's still treated as the most important evidence in any criminal case.
A course on that subject (Score:1)
This course [coursera.org] on Coursera describes the basics of that technology. It's in its final week but I'm sure it will be reissued later on. They have other courses about computer vision announced for the next months.
The course was pretty interesting but you don't really have to do any programming to get a grade (programming assignments are optional). Lucky for me, because I have a job and no time to spend on lengthy programming assignments, but one can't become an expert of that subject just by listening at the les
Re: (Score:1)
or Star Wars without Jar Jar Binks
Re: (Score:2)
Think outside the box - just delete episodes 1-3 inclusive.
Or nuke it from orbit, it's the only way to be sure...
Stalin (Score:3)
Very ancient technology (Score:5, Informative)
People have been doing this for far more than the last 5 years. It is a trivial application of so-called 'optical flow' where motion vectors are used to identify independently moving objects within a scene.
One interesting application (seen, for instance, in the Will Smith film "I am legend") takes video footage of a real environment, and converts the footage into a virtual static 'texture' for the background elements. Artists can then repaint over this 'texture' to add damage to buildings etc. The new texture can now be reapplied to the original footage, so the moving shot appears to show the artistic changes in visual context. Clearly this method will not stand up to the same scrutiny as remodelling buildings in CGI, and inserting them into a virtual set, but it works well for backgrounds.
Films today frequently use a so-called skybox- a 360 panorama stitched from multiple still photos shot on location. This skybox allows a virtual background to be 'projected' behind the actors (say when they are pretending to be on top of a tall building or mountain) that can track the rotational movement of the camera.
The idea of element extraction forms the basis of various camera enhanced video games found on the current consoles. Usually, the technique is the reverse of the example in the article, where it is the background that is removed so that the player may be isolated and inserted into a virtual scene.
Slashdot needs editors that know something about technology, but that isn't going to happen while the owners of Slashdot use the tech stories to draw readers to the constant anti-Iranian warmongering propaganda that appears here almost daily.
Re:Very ancient technology (Score:4, Interesting)
...The idea of element extraction forms the basis of various camera enhanced video games found on the current consoles. Usually, the technique is the reverse of the example in the article, where it is the background that is removed so that the player may be isolated and inserted into a virtual scene.
Uh, yeah, and now that we've revealed that removing someone is "ancient technology" is is exactly this reversed scenario that most should fear today and is ripe for abuse in a corrupt world.
One day, you think it's cool that you've been "painted" into a video game...until you realize that same technology can "paint" you right into Exhibit A: The murder scene.
How long before innocent people are framed? Judges can't even understand how the internet works. You think they're going to grasp this and give you a fair trial?
Re: (Score:1)
easy, just paint the judge into the same scene as evidence it can be faked
Re:Very ancient technology (Score:4, Informative)
One day, you think it's cool that you've been "painted" into a video game...until you realize that same technology can "paint" you right into Exhibit A: The murder scene.
How long before innocent people are framed? Judges can't even understand how the internet works. You think they're going to grasp this and give you a fair trial?
Already, people are routinely convicted based on bullshit forensic pseudoscience: PBS Frontline: The Real CSI [pbs.org] [torrent] [kat.ph]
Re: (Score:3)
Re: (Score:2)
The photobooth trick is called a 'difference matte' or 'difference key.' It's found in most high-end video editing software, though not always by the same name. I wrote one myself too - http://birds-are-nice.me/video/bluescreen.shtml [birds-are-nice.me]
As you can see from my effort, it isn't as easy as you'd think to get good results.
Re: (Score:2)
Re: (Score:2)
Very noisy camera. I was trying to best replicate the conditions typical of a zero-budget production: No proper backdrop, a too-small studio (thus shadows cast behind the actor), no professional lighting, a cheap consumer camera. I ran the video through virtualdub's temporal smooth, that took the worst off. The only places the filter really struggles are the very dark regions - those near-black folds in the trousers, the shadows under the arms - where noise overwhelms the color component. There's a lot of f
Re: (Score:2)
Dark areas, and motion blur - you can see that clearly in the final seconds. It's a binary mask. The solution to this, I feel, is in physics rather than any change to the algorithm: Just set up a couple of those cheap 100W DIY worklights. No more near-0,0,0 blacks, and the motion blur reduced as the camera automatically reduces shutter'* speed.
* That term is going to be with us long after the physical shutter is gone. It's easier than saying 'CCD sampling interval duration.'
Re: (Score:1)
I love how you turned your whole personal vibe around there at the last second.
Common software (Score:3)
Re: (Score:2)
It's interesting you mention Boujou - the same company (2d3) announced basically this software, with some mighty impressive demo videos, at SIGGRAPH about 11 years ago.
As far as I know, though, it never saw the light of day.
Re: (Score:2)
I'm just interested if there is any open-source versions of this available. Mainly, I want to remove TV logos and ads from shows I want to save on my PC.
I thought what I'd do was... (Score:5, Insightful)
... I'd pretend I was one of those deaf-mutes.
Meh... (Score:2)
I have seen this done in After Effects 5 years ago. In fact I remember a script/template that was floating around that tried to automate it quite well.
what about competing TV advertising? (Score:1)
The Laughing Man (Score:5, Interesting)
Anyone reminded of the Laughing Man from Ghost in the Shell: Stand Alone Complex?
A hacker who was able to hack the cybernetic vision of others in real-time to make himself invisible...
Re: (Score:2)
God dammit got here too late to quote Catcher in the rie!
This is doubleplus ungood for future unpersons. (Score:2)
Prior art. (Score:2)
I believe this is just an extension of the research done by this [garfieldmi...rfield.net] guy.
Life After People! Remove the speech, too. (Score:1)
Now I have installed an Active People Filter to remove people from all those Internet videos and movies and DVDs I watch. Along with speech and vocal removal filters for music, speech removal.
The filters have learned what people look like and do a fair job at stamping them out of still images too.
Now my life is like a series of paintings in still-life. Sitcoms are rooms of silent furniture and no stupid laughter, the Olympics and football a breathtaking vista of grand spaces and odd sporting equipment lyin
why mention HD? (Score:1)