Semi-supervised reference-based sketch extraction using a contrastive learning framework

Chang Wook Seo, Amirsaman Ashtari, Junyong Noh

KAIST, Visual Media Lab
SIGGRAPH 2023 Journal track

Input color images and sketches extracted by our method. Without requiring repetitive training of the network to make pre-trained weights for each style, our model produces various style sketches by imitating the input reference sketches.

Abstract

Sketches reflect the drawing style of individual artists; therefore, it is important to consider their unique styles when extracting sketches from color images for various applications. Unfortunately, most existing sketch extraction methods are designed to extract sketches of a single style. Although there have been some attempts to generate various style sketches, the methods generally suffer from two limitations: low quality results and difficulty in training the model due to the requirement of a paired dataset. In this paper, we propose a novel multi-modal sketch extraction method that can imitate the style of a given reference sketch with unpaired data training in a semi-supervised manner. Our method outperforms state-of-the-art sketch extraction methods and unpaired image translation methods in both quantitative and qualitative evaluations.

Overview of our network design.

Illustration of contrastive learning used in our method, which groups smiliar sketch styles in embedded space regardless of the image shape.

Examples generated by our method and baselines.

Examples generated by our method.

Contributions

• A novel multi-modal sketch extraction method that can imitate the drawing style of the input reference sketch. Also trainable with unpaired sketch and color images in a semi-supervised manner.
• Generated sketches can be utilized for related studies such as auto-colorization and sketch style transfer.
• Provide a new authentic sketch dataset prepared by a professional artist. Check this dataset (4SKST) in the LINK

Our method can generate various styles of sketch dataset, and this can enhance the performance of learning-based auto-colorization methods.

Sketches of different styles can be placed easily in one scene after the style transfer by our method.

Rough sketches can be simplified by transferring the style with our method.

BibTeX


    @article{seo2023semi,
      title={Semi-supervised reference-based sketch extraction using a contrastive learning framework},
      author={Seo, Chang Wook and Ashtari, Amirsaman and Noh, Junyong},
      journal={ACM Transactions on Graphics (TOG)},
      volume={42},
      number={4},
      pages={1--12},
      year={2023},
      publisher={ACM New York, NY, USA}
    }