Adjustable Real-Time Style Transfer: Control Augmentation, Output Refinement, and Model Compression

Abstract

The objective of style transfer is to render a new image that preserves the content of one image and the style of another. While existing texture transfer techniques could be utilized for this task, more recent work showed that representations from deep neural networks could aid in the generation of high quality style transfers. This finding led to renewed interest in the field, and subsequent work showed that neural networks could be trained to perform neural-based stylizations directly (instead of using an optimization routine for each transfer). 

A downside of training a neural network to perform style transfer is that certain settings must be decided at model training time. To address this, it was shown that these hyperparameters could be converted to model inputs, deferring their selection to the time of stylization. Such models serve as the foundation for our research.

The aforementioned adjustable real-time approach allows for controlling style transfer output by effectively modifying the contribution of different terms in the style transfer loss function. We extend the model to allow for adjustment of style scale in addition to the existing controls.

Style transfer preferences differ across users and also depend on the pairing of style and content images. Adjustable models permit users to modify the weightings of loss terms to their liking, but the results may be considered of lesser quality than traditional optimization-based neural style transfer. We investigate the impact of using the traditional technique as a post-processing step—to refine the output of fast controllable models (which are well-suited to quickly find desirable stylization settings). Experiments are conducted to evaluate the effectiveness.

Lastly, the adjustable models that we consider are notably larger than their uncontrollable counterparts. Our research investigates various compression techniques that can be used to reduce model size; we analyze the consequences.