Notebooks and code (GitHub)
The explanation of the code is in the GitHub, for the most part in the docstrings and comments of the utils folder.
Models
2 objective discriminator GAN model.zip
Table of Contents
Image-to-Image translation (I2I) strives to identify the mapping between two domains. A prominent challenge for I2I is the difficulty to obtain aligned and paired images or the lack thereof (painting ↔ real photo). One of the most common methods used to solve this is to create a model, which is competent in identifying the association amongst given unpaired image datasets and makes way for unpaired image-to-image translation. In essence, we were able to map from the ‘Monet’/’Van Gogh dataset to (and from) the ‘real photo’ dataset in regard to the artistic style and content.
Sample of the photo and Monet datasets
In this project, we applied 3 different architectures of GANs as our models to convert a given image into a Monet-style image or Van Gogh style. This method leads to more available datasets (as opposed to the use of aligned datasets) by having the ability to maneuver unpaired images through cycle-consistency loss. To assess these models, we used the MiFID metric used automatically by Kaggle based on the frechet inception distance.
Within the Kaggle challenge, we were given different paintings dataset of 300 images and real images dataset with 7028 images, all in the size of 256x256. We dealt with this challenge with three different models (Cycle-Gan, Two objective discriminators, and Gan with differentiable augmentation). In this project report, we will explain the problem and the processes that these different GAN structures are based on and the comparison of their results.
The environments used were either on MacOS or Linux. On MacOS, we used Python 3.8, as well as Tensorflow, Keras, Numpy, and Matplotlib. We chose to deal with TFRecords instead of jpg images, to familiarize ourselve with a new Tensorflow object.
GAN stands for Generative Adversarial Network. Their goal is to generate synthetic data resembling existing datasets. GANs are composed of two models, a generator and a discriminator.
The generator creates new data instances from an existing data point and a random noise vector. The discriminator tries to detect if an image is authentic or created by the generator. By working together, both models train and improve their accuracy, thus generating more convincing pictures and detecting “fakes” better. For a given set of datapoints $X$ and labels $Y$, the generator will capture $P(X, Y)$, and the discriminator will capture $P(X|Y)$. For example, say we have a set of emails, and the model’s job is to determine if they are spam or not. The generator would ask itself, “What does a spam email look like? What are its features?”, and the discriminator would ask itself “Given these features, is this email spam or not?”.
The diagram below depicts the basic functioning of a GAN.
Schematic Diagram of a GAN
The generator model $G$, takes random noise vectors $\{z_1, ..., z_m\}$ and create fake data samples, $\{G(z_1), ..., G(z_m)\}$. This set, as well as the initial data $\{x_1, ..., x_m\}$ go through the discriminator model $D$, which will decide if the sample was fake or real.
The cost function of the vanilla GAN is defined as:
$$ \underset{G}{min}\ \underset{D}{max}\ V(D, G)= \mathbb{E}{x\sim p{data}(x)}[logD(x)] + \mathbb{E}_{z\sim p_z(z)}[log(1-D(G(z)))] $$