SketchyScenes: Understanding Scene Sketches

Being natural to everyone, language-based inputs have demonstrated effective for various tasks such as object detection and image generation. This paper for the first time presents a language-based system for interactive colorization of scene sketches, based on their semantic comprehension. Compared with prior scribble-based interfaces, which require a minimum level of professional skills, our language-based interface is more natural for novice users. The proposed system is built upon deep neural networks trained on a large-scale repository of scene sketches and cartoon-style color images with text descriptions. Given a scene sketch, our system allows users, via language-based instructions, to interactively localize and colorize specific object instances to meet various colorization requirements in a progressive way. We demonstrate the effectiveness of our approach via comprehensive experimental results including alternative studies, comparison with the state-of-the-art, and generalization user studies.

Related Publications

teaser image of Language-Based Colorization of Scene Sketches

Language-Based Colorization of Scene Sketches

ACM Transactions on Graphics (SIGGRAPH Asia), 2019.
Keywords: deep neural networks; image segmentation; language-based editing; scene sketch; sketch colorization
teaser image of SketchyScene: Richly-Annotated Scene Sketches

SketchyScene: Richly-Annotated Scene Sketches

European Conference on Computer Vision (ECCV), 2018.
Keywords: sketch dataset, scene sketch, sketch segmentation

Related Talks

Related Videos

Language-Based Colorization of Scene Sketches


Stay In Touch