This project contains most of the code used to derive the results of my bachelor thesis, written at TU Dortmund University at the Chair for Aritificial Intelligence lead by Prof. Stefan Harmeling. Without the professional guidance and access to compute resources this results would have not been realisable for me.
Goal of the thesis was to train an Generative Image-to-Text Transformer (Wang, J., Yang, Z., Hu, X., Li, L., Lin, K., Gan, Z., ... & Wang, L. (2022), https://arxiv.org/abs/2205.14100) on a small dataset and measure its performance.
In a first step, as as proof of concept, a Vision Transformer (Dosovitskiy, A. "An image is worth 16x16 word: Transformers for image recognition at scale" (2021)) is implemented in PyTorch from scratch and trained on the CIFAR-10 dataset for a classification task. This ensures the correct implementation of the Transformer Architecture (Vaswani, Ashish, et al. "Attention is all you need." (2017)).
For training coordination and hyperparameter sweeping Weights&Biases (https://wandb.ai) was used throughout the whole project.
In the second step, the full Generative Image-to-Text Transformer is implemented in PyTorch (code found in folder "captioning").
For training and evaluation the MS-COCO dataset is used in the split of Andrej Karpathy (https://cs.stanford.edu/people/karpathy/deepimagesent/). Additionally I use the Conceptual Captions dataset from google (https://ai.google.com/research/ConceptualCaptions/download).
During the first implementation phase the models suffered from extremely bad performance. To investigate the issue and exclude an error in the implementation of token handling, I conducted overfitting experiments, in which the model was required to learn the captions of a small set of samples by heart. The results of these experiments are found in this folder.
To evaluate the model's performance, the CIDEr-D score (Vedantam, R. et al. "Cider: Consensus-based image description evaluation." (2015)) is used, as it is explicitly developed for the rating of generated image captions. The algorithm is implemented on my own in the metrics folder. To speed up computation of the score in mid training, the document frequencies of all n-grams are precomputed and stored for the used dataset.
As the datasets and training procedure used by me are tiny in comparison to the implementation by the authors, the models cannot reach comparable performance, so augmentation of the dataset was a promising option.
Similar to the authors we used a heavily pretrained VisionTransformer as image encoder. Therefore we can identify the text decoder as the component of our model, that needs more training to boost performance.
In order to achieve this, I used a Llama 3 model (https://www.llama.com/) from Facebook to generate additional captions for the MS-COCO dataset from the existing ones. It is important to mention that for this step, the acutal images were never used, but a LLM was used solely. I created two versions of augemented COCO datasets using
-
a zero-shot prompt that requests more captions for the same situation described by the other ones
-
a more sophasticated one-shot prompt, that also included the instructions that were given the Amazon Turk workers, who created the captions of MS-COCO.
This folder contains a jupyter notebook, used to generate all graphics, that I used in my thesis. It queries data from the Weights&Biases online service and plots them appropriately.