A LEARNED REPRESENTATION FOR ARTISTIC STYLE

A LEARNED REPRESENTATION FOR ARTISTIC STYLE

Vincent Dumoulin, Jonathon Shlens, Manjunath Kudlur
ICLR 2017

Abstract

construct a single, scalable deep network that can parsimoniously capture the artistic style of a diversity of paintings by reducing a painting to a point in an embedding space

Introduction

pastiche: an artistic work that imitates the style of another one
automate pastiche/style transfer: render an image in the style of another one
traditional methods: “grow” textures one pixel at a time using non-parametric sampling of pixels in an examplar image →$\to$ “growing” textures one patch at a time →$\to$ …
machine learning methods: neural style(expensive) →$\to$ feedforward style transfer network (the style transfer network is tied to a single style)
solution: conditional instance normalization(reduces each style image into a point in an embedding space)

STYLE TRANSFER WITH DEEP NETWORKS

style transfer: finding a pastiche image p$p$ whose content is similar to that of a content image c$c$ but whose style is similar to that of a style image s$s$ (high-level features in classifiers tend to correspond to higher levels of abstractions for visualizations)
content similarity: distance between high-level features extracted by a trained classifier
style similarity: distance between Gram matrices G$G$ of low-level features as extracted by a trained classifier (the artistic style of a painting may be interpreted as a visual texture)
neural style:

minpL(s,c,p)=λsLs(p)+λcLc(p)$$\underset{p}{min}L(s,c,p)={\lambda }_s{L}_s(p)+{\lambda }_c{L}_c(p)$$

feed-forward method: style transfer network T:c↦p$T:c\mapsto p$

the network T is tied to one specific painting style

N$N$-STYLES FEEDFORWARD STYLE TRANSFER NETWORKS

intuition: many styles probably share some degree of computation
train a single conditional style transfer network T(c,s)$T(c,s)$ for N$N$ styles
to model a style, it is sufficient to specialize scaling and shifting parameters after normalization to each specific style
all convolutional weights of a style transfer network can be shared across many styles
it is sufficient to tune parameters for an affine transformation after normalization for each style

conditional instance normalization: transform a layer’s activations x$x$ into a normalized activation z$z$ specific to painting style s$s$

z=γsx−μσ+βs$$z={\gamma }_s\frac{x-\mu }{\sigma }+{\beta }_s$$

μ,σ$\mu ,\sigma$: x$x$’s mean and standard deviation taken across spatial axes
γs,βs${\gamma }_s,{\beta }_s$: obtained by selecting the row corresponding to s$s$ in the γ$\gamma$ and β$\beta$ matrices

integrating an N+1$N+1$-th style to the network
原理很简单

EXPERIMENTAL RESULTS

METHODOLOGY

the same network architecture as in “Perceptual losses for real-time style transfer and super-resolution”
train the N$N$-style network with stochastic gradient descent using the Adam optimizer

Discussion

in the case of art stylization when posed as a feedforward network, it could be that the specific network architecture is unable to take full advantage of its capacity: pruning the architecture leads to qualitatively similar results;
the convolutional weights of the style transfer network encode transformations that represent “elements of style”