How Yipit Scales Thumbnailing With Thumbor and Cloudfront
Yipit, like many other sites, displays collected images, and like many other sites, Yipit needs to display these images in different sizes in different places.
Up until recently, we were using django-imagekit, which works pretty well but has presented some issues as we’ve grown.
Dynamic Generation Issues
Imagekit supports dynamic thumbnail generation, but it checks for and creates the image while rendering the final url for where the image will be accessed. This means, in order to take advanatge of this feature, rendering a page with 10 thumbnails needs to, best case, make 10 network calls to check for the images existence, and worst case scenario, retrieve, process, and upload 10 images.
The other option, which is how Yipit used Imagekit, is to pre-generate all thumbnails and assume they’ve been created properly when rendering a pages. Of course, they haven’t always been generated properly, so having a system to find and re-generate those images is a pain to maintain. Also, adding a new image size for a new design requires going back and creating new thumbnails for all of the old images.
We wanted a thumbnailing solution that would:
- Dynamically create images when they’re needed
- Serve those images quickly
- Not slow down server response times
Thumbor is a service written in python that allows you to pass the url of an image as well as thumbnailing options in a URI and then dynamically creates the images. There are libraries for URI generation in Python, Node.js, Ruby, and Java.
Thumbor is installable via pip. We run it in a virtualenv at
/var/www/thumbor-env so our entire installation is essentially this:
1 2 3 4
Configuring the thumbor server and the available options are very well documented with a sample configuration file.
We then run it with supervisor behind nginx with these configurations:
1 2 3 4 5 6 7 8 9 10 11
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
This setup runs 4 tornado processes load balanced behind nginx.
Setting Up Cloudfront
Setting up Cloudfront is also very easy. If you want to setup a dedicated cloudfront distribution for Thumbor just create a new distribution through the AWS Web Console set your thumbor url (In this case
thumbor.yipit.com:8000) as your origin domain.
If you want to set up a namespace for thumbor on an existing distribution, first you’ll need to create a new origin pointing to your thumbor server and then a new behavior with the pattern
thumbor/* that points at that origin.
Using it in your application
Now your application server doesn’t need to worry about image generation or exstience. All you need to do is render thumbor URIs. Here’s the function we use at Yipit:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
This can also be easily wrapped from a template tag:
1 2 3
making adding thumbnails to your pages as as easy as:
Oh, by the way, we’re hiring.