PhotoSketch: Internet Image Montage

It’s making a huge buzz on the internet right now and if you’re heading to SIGGRAPH ASIA then you’ll get a chance to hear alot more about it.  What is it? PhotoSketch.

We present a system that composes a realistic picture from a simple freehand sketch annotated with text labels. The composed picture is generated by seamlessly stitching several photographs in agreement with the sketch and text labels; these are found by searching the Internet. Although online image search generates many inappropriate results, our system is able to automatically select suitable photographs to generate a high quality composition, using a filtering scheme to exclude undesirable images. We also provide a novel image blending algorithm to allow seamless image composition.
Each blending result is given a numeric score, allowing us to find an optimal combination of discovered images. Experimental results show the method is very successful; we also evaluate our system using the results from two user studies.

Cliff notes version: Draw a basic sketch of what you want and label the components (as shown above), and the system automatically extracts the necessary components from a huge database and composites them together.  The research consists of 2 main parts:  A new blending algorithm to make the images fit in their new environments much better, and a new filtering and classification system for images.

You can read the full PDF of their work here (with many more examples than the sailboat/kiss floating around everywhere), and see a great demonstration video after the break.

via PhotoSketch: Internet Image Montage on Vimeo.

PhotoSketch: Internet Image Montage from tao chen on Vimeo.

About the Author: Randall Hand

Randall Hand is a visualization scientist working for a federal research lab, aiding researchers to discover the insights buried within their terabyte datasets generated on some of the most powerful supercomputers in the world. He also runs VizWorld.com .

Science algorithm, image processing, siggraph