Towards Super Resolution
Resolution is a very important quality metric in the domain of video and image data. Resolution refers to the size of the image in terms of height x width x color channels, (color channels usually being 3, RGB). Resolution transcends size however, since it is usually implies that an HD image with 1024 x 780 resolution would look better than a low-resolution 32x32 image that is expanded to be of size 1024 x 780.
The challenge with super-resolution is to take these low-resolution images and increase their size such that the image is not blurry or of poor quality.
The standard way of blowing up a low-resolution image into a higher size is to use bicubic interpolation. This can be implemented in OpenCV using the cv2.resize function.
One interesting way of doing this is by using a deep convolutional network to learn a mapping from low resolution to high resolution images. Here are the steps required to do this:
- Assemble a dataset of high-resolution images
- Copy them into another array, (this will be the x train data)
- Downsample these x input images to the desired low-resolution size ex: 600 x 600 → 200 x 200
- Use bicubic interpolation to upsample the images back to 600 x 600
- Now train the Convolutional Network given inputs of upsampled low-resolution images and outputs of the original high-resolution images.
This network will learn the mapping from low to high resolution. We can use this network to take a low-resolution image of size 200 x 200 and map it into 600 x 600.
This is the fundamental concept of using Deep Convolutional Networks to achieve image super-resolution. However, this can be expanded upon within the framework of generative adversarial networks.
The best way of understanding GANs is with the cop and the counterfeiter anecdote. The counterfeiter, (generator network), is trying to produce money such that the cop, (discriminator network), cannot tell if it is real money or money produced by the counterfeiter. The discriminator and generator networks optimize each other over time, thus the discriminator learns an adaptive loss function that would be impossible to hand-craft.
Back to the task of super-resolution..
GANs typically work by taking in a vector of random values, however, they can also input images. In this case, we take the image upsampled via Deep Convolutional Neural Network and give it as input to the generator. The generator now learns how to transform this input such that it is acceptable to the discriminator, (meaning that it looks like the other HD images in the y set).
Achieving image resolution is an interesting application of convolutional networks and GANs. It is very interesting to see how the representations sequentially learned by neural networks can be constructed in tasks other than classification and regression. Hopefully this short article helped you gain interest in the task of super-resolution. Please leave any questions or suggestions on errors I might have made in this description, Thank you!
CShorten
Connor Shorten is a Computer Science student at Florida Atlantic University. Research interests in deep learning and software engineering.
