June 17, 2020
Image GPT
Illustration: Ben Barry
We find that, just as a large transformer model trained on language can generate coherent text, the same exact model trained on pixel sequences can generate coherent image completions and samples . By establishing a correlation between sample quality and image classification accuracy, we show that our best generative model also contains features competitive with top convolutional nets in the unsupervised setting.
Introduction
Unsupervised and self-supervised learning, 1 or learning without human-labeled data, is a longstanding challenge of machine learning. Recently, it has seen incredible success in language, as transformer 2 models like BERT, 3 GPT‑2, 4 RoBERTa, 5 T5, 6 and other variants 7 , 8 , 9 , 10 have achieved top performance on a wide array of language tasks. However, the same broad class of models has not been successful in producing strong features for image classification. 11 Our work aims to understand and bridge this gap.
Transformer models like BERT and GPT‑2 are domain agnostic, meaning that they can be directly applied to 1-D sequences of any form. When we train GPT‑2 on images unrolled into long sequences of pixels, which we call iGPT, we find that the model appears to understand 2-D image characteristics such as object appearance and category. This is evidenced by the diverse range of coherent image samples it generates, even without the guidance of human provided labels. As further proof, features from the model achieve state-of-the-art performance on a number of classification datasets and near state-of-the-art unsupervised accuracy A on ImageNet.
Evaluation
Dataset
Our Result
Best non-iGPT Result
Logistic regression on learned features (linear probe)
CIFAR-10
96.3 iGPT‑L 32x32 w/ 1536 features
95.3 SimCLR 12 w/ 8192 features
CIFAR-100
82.8 iGPT‑L 32x32 w/ 1536 features
80.2 SimCLR w/ 8192 features
STL-10
95.5 iGPT‑L 32x32 w/ 1536 features
94.2 AMDIM 13 w/ 8192 features
ImageNet
72.0 iGPT‑XL a 64x64 w/ 15360 features
76.5 SimCLR w/ 8192 features
Full fine-tune
CIFAR-10
99.0 iGPT‑L 32x32, trained on ImageNet
99.0 b GPipe, 14 trained on ImageNet
ImageNet 32x32
66.3 iGPT‑L 32x32
70.2 Isometric Nets 15
- We only show ImageNet linear probe accuracy for iGPT‑XL since other experiments did not finish before we needed to transition to different supercomputing facilities.
- Bit-L, trained on JFT (300M images with 18K classes), achieved a result of 99.3.
To highlight the potential of generative 17 , 18 sequence modeling 19 , 20 , 21 , 22 as a general purpose unsupervised learning algorithm, we deliberately use the same transformer architecture as GPT‑2 in language. As a consequence, we require significantly more compute in order to produce features competitive with those from top unsupervised convolutional nets. 13 , 23 , 24 , 25 , 12 However, our results suggest that when faced with a new domain where the correct model priors are unknown, a large GPT‑2 can learn excellent features without the need for domain-specific 26 , 27 , 28 architectural design choices.
From language GPT to image GPT
In language, unsupervised learning algorithms that rely on word prediction (like GPT‑2 and BERT) have been extremely successful, achieving top performance on a wide array of language tasks. One possible reason for this success is that instances of downstream language tasks appear naturally in text: questions are often followed by answers (which could help with question-answering) and passages are often followed by summaries (which could help with summarization). In contrast, sequences of pixels do not clearly contain labels for the images they belong to.
Even without this explicit supervision, there is still a reason why GPT‑2 on images might work: a sufficiently large transformer trained on next pixel prediction might eventually learn to generate diverse B samples with clearly recognizable objects. Once it learns to do so, an idea known as “Analysis by Synthesis” 29 , 30 , C suggests that the model will also know about object categories. Many early generative models 31 , 32 , 33 , 34 , 35 , 36 were motivated by this idea, and more recently, BigBiGAN 37 was an example which produced encouraging samples and features. In our work, we first show that better generative models achieve stronger classification performance. Then, through optimizing GPT‑2 for generative capabilities, we achieve top-level classification performance in many settings, providing further evidence for analysis by synthesis.
Towards general unsupervised learning
Generative sequence modeling is a universal unsupervised learning algorithm: since all data types can be represented as sequences of bytes, a transformer can be directly applied to any data type without additional engineering. Our work tests the power of this generality by directly applying the architecture used to train GPT‑2 on natural language to image generation. We deliberately chose to forgo hand coding any image specific knowledge in the form of convolutions 38 or techniques like relative attention, 39 sparse attention, 40 and 2-D position embeddings. 27
As a consequence of its generality, our method requires significantly more compute to achieve competitive performance in the unsupervised setting. Indeed, contrastive methods 41 , 42 , 43 , 44 , 45 , 13 , 23 , 24 , 25 , 12 are still the most computationally efficient methods for producing high quality features from images. However, in showing that an unsupervised transformer model is competitive with the best unsupervised convolutional nets, 24 , 25 , 12 we provide evidence that it is possible to trade off hand coded domain knowledge for compute. In new domains, 46 , 47 where there isn’t much knowledge to hand code, scaling compute seems an appropriate technique to test.
Approach
We train iGPT‑S, iGPT‑M, and iGPT‑L, transformers containing 76M, 455M, and 1.4B parameters respectively, on ImageNet. We also train iGPT‑XL D , a 6.8 billion parameter transformer, on a mix of ImageNet and images from the web. Due to the large computational cost of modeling long sequences with dense attention, we train at the low resolutions of 32x32, 48x48, and 64x64.
While it is tempting to work at even lower resolutions to further reduce compute cost, prior work has demonstrated that human performance on image classification begins to drop rapidly below these sizes. 48 Instead, motivated by early color display palettes, 49 we create our own 9-bit color palette to represent pixels. Using this palette yields an input sequence length 3 times shorter than the standard (R, G, B) palette, while still encoding color faithfully.
Experimental results
There are two methods we use to assess model performance, both of which involve a downstream classification task. The first, which we refer to as a linear probe, uses the trained model to extract features E from the images in the downstream dataset, and then fits a logistic regression to the labels. The second method fine-tunes F the entire model on the downstream dataset.
Since next pixel prediction is not obviously relevant to image classification, features from the final layer may not be the most predictive of the object category. Our first result shows that feature quality is a sharply increasing, then mildly decreasing function of depth. This behavior suggests that a transformer generative model operates in two phases: in the first phase, each position gathers information from its surrounding context in order to build a contextualized image feature. In the second phase, this contextualized feature is used to solve the conditional next pixel prediction task. The observed two stage performance of our linear probes is reminiscent of another unsupervised neural net, the bottleneck autoencoder, which is manually designed so that features in the middle are used.
Our next result establishes the link between generative performance and feature quality. We find that both increasing the scale of our models and training for more iterations result in better generative performance, which directly translates into better feature quality.
When we evaluate our features using linear probes on CIFAR-10, CIFAR-100, and STL-10, we outperform features from all supervised and unsupervised transfer algorithms. Our results are also compelling in the full fine-tuning setting.
Pre-trained on ImageNet
Evaluation
Model
Accuracy
w/o labels
w/ labels
CIFAR-10
Linear Probe
ResNet-152 50
94.0
✔
SimCLR 12
95.3
✔
iGPT‑L 32x32
96.3
✔
✔
CIFAR-100
Linear Probe
ResNet-152
78.0
✔
SimCLR
80.2
✔
iGPT‑L 32x32
82.8
✔
STL-10
Linear Probe
AMDIM-L
94.2
✔
iGPT‑L 32x32
95.5
✔
CIFAR-10
Fine-tune
AutoAugment
98.5
SimCLR
98.6
✔
GPipe
99.0
✔
iGPT‑L
99.0
✔
CIFAR-100
Fine-tune
iGPT‑L
88.5
✔
SimCLR
89.0
✔
AutoAugment
89.3
EfficientNet 52
91.7
✔
A comparison of linear probe and fine-tune accuracies between our models and top performing models which utilize either unsupervised or supervised ImageNet transfer. We also include AutoAugment, the best performing model trained end-to-end on CIFAR.
Given the resurgence of interest in unsupervised and self-supervised learning on ImageNet, we also evaluate the performance of our models using linear probes on ImageNet. This is an especially difficult setting, as we do not train at the standard ImageNet input resolution. Nevertheless, a linear probe on the 1536 features from the best layer of iGPT‑L trained on 48x48 images yields 65.2% top-1 accuracy, outperforming AlexNet.
Contrastive methods typically report their best results on 8192 features, so we would ideally evaluate iGPT with an embedding dimension of 8192 for comparison. However, training such a model is prohibitively expensive, so we instead concatenate features from multiple layers as an approximation. Unfortunately, our features tend to be correlated across layers, so we need more of them to be competitive. Taking 15360 features from 5 layers in iGPT‑XL yields 72.0% top-1 accuracy, outperforming AMDIM, MoCo, and CPC v2, but still underperforming SimCLR by a decent margin.
Method
Input Resolution
Features
Parameters
Accuracy
Rotation 53
original
8192
86M
55.4
iGPT‑L
32x32
1536
1362M
60.3
BigBiGAN 37
original
16384
86M
61.3
iGPT‑L
48x48
1536
1362M
65.2
AMDIM 13