Cloud backed on-demand thumbnails of any size
A solution of how to provide on-demand thumbnails of any size, based on user uploaded images, stored on Google Cloud Storage with Nginx and a Java restful web service. Cache generated thumbnails and make them disposable keeping only originals. Save storage space by deleting old or all thumbnails.
Problem – How to provide thumbnails of any size?
In web systems where users can create content part of which are images, sooner or later the same image has to be displayed as a thumbnail in different size, depending on where or when the thumbnail is displayed. For example display a properly resized thumbnail for each resolution of responsive web application. Another example is a social post with multiple two images that is edited and now contains 5 images that need to be displayed in the same box. Another example is where the same data needs to be presented to multiple web apps with different look and feel.
There’s an option to scale images with CSS but that doesn’t take into account aspect ratio and results are usually not satisfactory.
Another aspect is that once users upload and they preview an image, they usually upload a few more before they finally decide which one to keep. That piles up files which are not used and have to be cleaned up every now and then to preserve storage space.
If we assume there is a predefined set of thumbnail sizes there are two options:
- generated upon upload of the original
- generated upon request (on-demand)
But then if the set of sizes changes – all existing images have to get their thumbnail set filled up – not nice, as may be not all of them are used and that would just consume space. Also they have to be fed somehow to the image processing services for additional thumbnail generation only for the new sizes. That may lead to other issues and we can go on and on stating issues coming from that.
If we go with on-demand then for every image every time we have to generate a thumbnail in the requested size. That looks much more attractive but it requires a processing resources every time and if there are frequent reads that may be slow and ineffective.
So if we take the best of those two worlds, that may be a very good solutions to our problem. Let’s see how this can be done.
Proposed solution
If we combine on-demand generation with storing originals and thumbnails on a cloud storage, then thumbnails can be served through a reverse proxy if they exist and failover to a thumbnail generating service if they don’t. This way the service can generate any size on-demand and put(cache) it in the cloud for future reuse. If all images and thumbnails are served through the same domain then cloud storage will stay transparent and can be changed at anytime as long as files structure is the same (or easy to map). On the other hand all thumbnails can be deleted at any time to preserve storage space/reduce cost and only the ones that are still in use will be re-generated upon their next request. Another benefit is that all user uploads can go through the same image processing service and store in the cloud for easy management.
Let’s see how you can do that for your web system.
Components
Here are the main components needed for that solution
- Reverse proxy – i.e. Nginx
- RESTful web service for image processing – i.e. Java
- Cloud Storage for image and thumbnails – i.e. Google Cloud
You can use any technology for all of them.
Diagram
As a picture speaks a thousand words, here’s a tiny diagram representing the setup.
Flow chart
Actions
Let’s look at that solution from an API perspective. Thumbnail generation is actually an image processing service. Uploaded images and their thumbnails are resources and they can be managed. In RESTful terms this a collection and CRUD operations shall be performed against it. The standard of implementing CRUD over HTTP is using the main verbs:
- GET – retrieve an image/thumbnail
- PUT – upload a new image
- POST – confirm/save an uploaded image
- DELETE – delete an uploaded image
All must be configured in the reverse proxy so that PUT, POST and DELETE go directly to the Image service while GET will first try to serve the request from cloud storage and as a fallback to forward to the Image service.
Let’s go through them one by one to see how they will help create a full user flow.
User uploads a new image
As initially there will be no images in the system, there needs to be a way for uploading a new Image. That will happen through calling PUT and passing the image as multipart-form-data. As users upload usually more than one image before they Save, that one must be saved in a temporary storage (local or cloud) until confirmed, as it may never be confirmed. At that point the service must return a URL to the new resource, so that user can preview it.
request/response example
User previews an uploaded image
Once an image is uploaded but before it is confirmed, the user shall receive a resource URL which can be used for preview by making a GET request. That will go through the proxy which will extract the path and try to get it from Cloud Storage. This will fail because no one has requested that thumbnail before and the proxy must fallback to Image service passing the same path. Original image will be found, a thumbnail will be generated and stored on the cloud storage and returned to the user.
request/response example
User confirms an image
If the user is not happy with the image, previous steps are repeated. Else the user sends a POST request passing the ImageID to confirm it. Image service will move that image to cloud storage and return a permanent resource URL.
request/response example
User request a thumbnail
A thumbnail of any size needs to be presented to the user. They send a GET request and add the size at the needed size at the end of the original URL. Our proxy will try to serve that from Cloud storage and again fallback to Image processing which will find the original, generate the thumbnail, store it for re-use and return it.
request/response example
Conclusion
This solution is a balanced but effective approach between storage and processing. It can be implemented with any proxy, service and storage while still serving the main purpose of flexibly delivering thumbnails of any size with added caching for faster response times and less cpu resources.
Hi Salomon,
I’m not very good at following up on emails, so please use any of the social networks in the footer and send me a private message.
Thanks in advance!