Slashdot Log In
Using Photographs To Enhance Videos
Posted by
timothy
on Thu Aug 14, 2008 05:40 PM
from the ready-for-my-closeup dept.
from the ready-for-my-closeup dept.
seussman71 writes with a link to some very interesting research out of the University of Washington that employs "a method of using high quality photographs to enhance a video taken of the same subject. The project page gives a good overview of what they are doing and the video on the page gives some really nice examples of how their technology works. Hopefully someone can take the technology and run with it, but one thing's for sure: this could make amateur video-making look even better than it does now." And if adding mustaches would improve your opinion of the people in amateur videos, check out the unwrap-mosaics technique from Microsoft Research.
Related Stories
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Is anyone else sick of demos? (Score:4, Insightful)
Why is UW not releasing their source code? If they intend to spin off commercial products, why are they releasing demos? Hell, even *Microsoft* is releasing demos of this stuff.. is Apple and Google the only companies that can ship product these days (even if it is "beta" you can at least freakin' use it).
No more demos. We know you're smart, now make something useful please.
Re:Is anyone else sick of demos? (Score:5, Insightful)
Simple reason:
They _say_ that it does this automatically.
Translation: We put some phd student on it who spend some months optimizing the settings for the 2 selected scenes so we can make a nice publication and maybe get more money.
If you just look at their steps of the workflow, the way they discribe it just isnt possible (like the way they "stereoscopically" create a depth-map from a _single_ still photograph..).
Not to mention that the first scene looks like a bad video game level after their "improvement".
Parent
Find Video Frame - Apply (Score:4, Insightful)
The easy way would be to use the already calculated depth field from the frame in the video that best matches the photo.
Parent
Re: (Score:2)
True. But still, notice how they use a reference movie thats basically a side-scroller to create this depth-map?
Basically, stuff like this is released for _every_ siggraph. The first few times i was still blown away, but now that i am used to it (and in academia myself, so i know the paper game), i simply cannot get excited for semething that will never see realisation.
Re:Is anyone else sick of demos? (Score:5, Funny)
like the way they "stereoscopically" create a depth-map from a _single_ still photograph.
No no no, read the fine print
Stereocycloptically, not stereoscopically!
Parent
Re:Is anyone else sick of demos? (Score:5, Informative)
like the way they "stereoscopically" create a depth-map from a _single_ still photograph
TFV said they were using video frames to do stereoscopic depth-mapping. Since the source footage changed perspective, they can build a depth map based on the relative shift of each object in the video, and then project the high-quality photograph on top of the derived 3D structure
Parent
Re: (Score:2)
In the video they say they use structure from motion to create the depth map.
Re: (Score:3, Informative)
Pics or it didn't happen. Or in this case, apps or it happened only in photoshop/whatever.
Re: (Score:2)
Pics or it didn't happen. Or in this case, apps or it happened only in photoshop/whatever.
Perhaps you're right. However, apps like SynthEyes are already like 90% of the way there. Their demos aren't a huge leap away from what is already on the market.
Re:Is anyone else sick of demos? (Score:4, Informative)
The publication is supposed to contain enough information to recreate the results.
Question 4 on the SIGGRAPH review form -
"4. Could the work be reproduced by one or more skilled graduate students? Are all important algorithmic or system details discussed adequately? Are the limitations and drawbacks of the work clear?"
If you or a company wants it bad enough, the information is there, unless the review process failed (which does happen).
This wasn't a SIGGRAPH paper but the ability to reproduce results is none-the-less a standard prerequisite for academic publication.
It's certainly not as convenient as releasing source code, but that's sometimes a big challenge for an academic researcher because the last thing they want is to have to support buggy, poorly documented research code for random people on the internet.
Parent
CREEPY! (Score:2, Interesting)
Re: (Score:2)
If *that* creeped you out please don't print out/assemble one of these: http://www.youtube.com/watch?v=iBri21uXF-k [youtube.com]
3D models from videos (Score:5, Interesting)
Re:3D models from videos (Score:5, Informative)
Takeo Kanade's lab [cmu.edu] at Carnegie Mellon's Robotics Institute did this in the mid 90's [cmu.edu]...
Parent
Patent Encumbered? (Score:4, Insightful)
I always get this feeling when I see a university-styled promotional release that the *software* patents are already pending.
I haven't the time to search just now but I'll bet there's at least one application pertaining to this method which encompasses a hell of a lot more.
Of the two, I find the Microsoft one to be better (Score:5, Interesting)
Really, the ability for their software to 'unwrap' a 3D object and allow you to fiddle with it as you wish is very cool.
And not limited to a 'static' scene.
And, really, if you're going to go to the effort of videoing a scene, then photographing the scene, then passing the video and the photos through their software. All to get better exposure and resolution.
Um.
Wouldn't it be a far better cost/effort equation to just buy a better HD camera in the damn first place?
Re:Of the two, I find the Microsoft one to be bett (Score:3, Interesting)
Re:Of the two, I find the Microsoft one to be bett (Score:4, Informative)
That would greatly lower the cost of doing special effects, if you didn't have to do them frame by frame.
Parent
Re: (Score:2)
I assure you that in all but the most insane cases, doing that frame-by-frame is already not done anymore (high quality rotoscoping, on the other hand.. yipe). You model a quick mock-up in a 3D application, project your painted texture onto that, and composite that with the original footage.
What it does do is remove that whole 'You model a quick mock-up' part in many (not all) cases. Now to see who gets the patents, how much they are to license, and who get(s) to toss it into their editing suite.
Re: (Score:2)
All to get better exposure and resolution.
Clearly it's intended for pr0n!
Re: (Score:2)
Wouldn't it be a far better cost/effort equation to just buy a better HD camera in the damn first place?
What about you do buy a HD camera and combine it with a 12 megapixel still camera? Besides just a HD camera doesn't fix the issues you can fix by then adding HDR shots to the mix.
Re: (Score:2, Insightful)
Then just have the still camera do continuous shooting @ ~1fps while you video. Match them up in this software when you're done and you're good to go...now if I could just get a hold of their software.
Build this into the video camera? (Score:2)
Still cameras can pretty much always take higher res pictures than video cameras no matter what price range you're looking at. I wonder if they could combine a still camera into a video camera and have it take high res still frames as the video is shot and then use this software to improve the video. It seems that'd be a cost effective way to squeeze more out of any level of camera.
Re:Of the two, I find the Microsoft one to be bett (Score:2)
Wouldn't it be a far better cost/effort equation to just buy a better HD camera in the damn first place?
I hate to be captain obvious here, but historical footage strikes me as the #1 reason (historical meaning everything up to yesterday). I mean you can't go back and reshoot the millions of hours of footage the world already has, but there's a lot of high resolution photos of the some of the same subjects.
Secondly, the still resolution on most point and shoot cameras is a lot higher than the video reso
Re: (Score:2, Insightful)
Wouldn't it be a far better cost/effort equation to just buy a better HD camera in the damn first place?
Hint: years old amateur/family/etc video meets modern high-res camera.
Re: (Score:2, Insightful)
"you..videoing..." isn't the only application. This could be used to enhance other videos. Let's say someone else made a great video (captured some really great scenes, focused on some details) and you want to publish it but even if they focused on cool details, they're not enough. You take a few pictures and enhance their video.
Also, this is just the start. They are currently enhancing static videos but I'm sure in the near future, if this is worke
A better use? (Score:5, Interesting)
All of these techniques are pretty awesome and will certainly be a boon to home video enthusiasts the world over (plus plenty of commercial places that are on a tight budget), but I've got another idea.
You see it on TV all the time, CCTV footage of robberies and the like, couldn't this technology be used to effectively map out a 3D image of the purpetrator?
I know it wont be perfect and most CCTV is probably too low quality to be used, but it would certainly be pretty cool (and useful) to have a vaugely accurate 3D model of the guy, giving you height, build, etc. and with the help of supplementary images, a really easy way to adjust it's appearance.
Re: (Score:2, Informative)
You see it on TV all the time, CCTV footage of robberies and the like, couldn't this technology be used to effectively map out a 3D image of the purpetrator? I know it wont be perfect and most CCTV is probably too low quality to be used, but it would certainly be pretty cool (and useful) to have a vaugely accurate 3D model of the guy, giving you height, build, etc. and with the help of supplementary images, a really easy way to adjust it's appearance.
Yes, like, you could adjust the appearance to look exactly like someone else! Not saying that one would or should do this, but now that they can, they probably will.
Re: (Score:3, Interesting)
I would say that mounting a high res still camera in parallel with the CCTV camera and taking, say 1 picture every 10 seconds after the CCTV montion sensors are tripped, which would have quality comparable to a high-end consumer camera (i.e. 7-8 Mpixels), then use that data to enhance the video taken to aid in identifying suspects
That said, I don't think these 'enhanced' videos should be admissible as evidence, since the videos have been effectively tampered with and given the possibility of altering identi
Re: (Score:2)
No, but if it helps me find the guy with the QuickStop drop bag in his back seat stuffed with bills and a .38 snub-nose, that would help. :)
Re: (Score:2)
Re: (Score:2)
Except it doesn't. Check 0ut the Microsoft link at the bottom of the summary (I know, it's a slightly different thing, but it's incredibly similar) and you'll see they do almost exactly what I describe - with moving objects.
Re: (Score:2)
A few years ago that would be entirely valid, but now you can pick up a 1Tb HDD for around $100, storage really isn't as big an issue these days.
It's going to be big... (Score:3, Insightful)
When the ability to deconstruct a video into a 3-d model & skin (the opposite of what a video card does now) is placed into an open-source API, the possibilities are going to be HUGE (and a little frightening).
Anyone want to post a few ideas? I'll give you a few topics to kick things off:
Change detection (Finding lost objects in a room, seeing boxes left in a government office, where's my remote) ...)
Change observation (plant growth, things that change too gradually for us to notice)
Creating 3-d models from humans (extracted from old films, walking down the street)
Weapon systems (Undetectable lasers blinding targets, Unmanned guns with perfect accuracy)
Home interaction (Make a sign with your hand, computer changes the channel, lighting, heat,
Office monitoring (Exactly where each person is any time just by typing "Where's bill" into your PC)
All things that could be done by any hobbiest/hacker with the right API.
(I assume that to get real-time you could use the massively parallel abilities of a video card, making this stuff run on any hardware...)
Also, just storing models and skins is extremely efficient--You could film a room for years in extremely high resolution and use virtually no storage (almost none except when something or someone new enters the room, then just one new high-def skin)
Other ideas?
Re: (Score:2)
Well there goes using video as evidence in court (Score:2, Insightful)
evidence fabrication? (Score:3, Insightful)
This software, if it actually works as described, could also be used to easily fabricate video "evidence". An average viewer would not be able to tell the difference.
Kinda scary...
Re: (Score:2)
evidence (Score:2)
automatic or not - that was a huge eye-opener if that technology is available at the grad-student level. Available in commercial/comsumer products in 3-5 years.
So much for "video evidence". So much for reality. "Your honor, I have a video of what happened and I wasn't there! ... see?"
3d track, projection, basic compositing. (Score:2, Informative)
This is a 3d track of the shot (which generates a point cloud of 3d points, which can then be used to generate an automatic 3d model of the scene). They then project (a method of texturing that paints a model based on points of projection.. what happens when you stand in front of a projector- you get projected onto) the still photos onto the 3d model, recreating all aspects of the texture and geometry, but instead of SD resolution, you now have gigapixel resolution built into the model.
The reason it looks l
Re: (Score:2)
Dealing with copyright problems (Score:2, Interesting)
Getting close... (Score:2)
This is getting really close to being something that every straight male porn viewer (which means every straight male on Slashdot...) would pay a lot for.
Combine the ability to remove items from a video, like they showed with the lamppost and sign in the flower shop, with the ability to insert new things into the video, and you could turn a boring man on woman porn movie into a lesbian twin incest movie.
Image manipulation and Investigation (Score:2)
Maybe those new tech might be used to produce that kind of useful result and not only better pops and moms holiday pictures..
Old article: http://www.guardian.co.uk/uk/2007/oct/19/ukcrime.internationalcrime [guardian.co.uk]
More intresting technology than photosynth (Score:2)
These two (article) technologies IMHO are more important than Photosynth
http://www.youtube.com/watch?v=556FvXHLtAo [youtube.com]
Re: (Score:2)
I'm afraid it's going to be used in music videos to suddenly make make flowers or tentacles spread on walls, or other such stupid uses.
By the way, considered how you can modify an object in a scene by replacing a frame of it, or adding a picture to the mix, does it mean we can make Clint Eastwood look like he's 30 again by using a picture of him when he was young on some recent footage of him, or even do entire "head transplants" on videos?
Re: (Score:2)
So my by reckoning, it'll only be a few years before the stuff featured in CSI and similar shows isn't *total* bollocks. A lot of people see photos and stuff being magically enhanced to perfect resolution, revealing the identity of some suspect, and think that can be done in real life.
No not at all. You can get a good resolution out of a video if and only if you have a good picture to go with it, and it doesn't bring out any more data than you have in both your picture and video, it only combines what they have. Infinite zoom with a mere security camera as shown on CSI is theoretically impossible, well, with the exception of aliasing-based super-resolution techniques, but that sort of processing is done on many frames, takes the necessary aliasing and probably a few other conditions like
Re: (Score:2)
Actually two bad quality images can be used to make a better quality image.
Re: (Score:2)
Re: (Score:2)