Serving static with nginx and varnish

I used nginx as reverse-proxy in front of amazon s3.

A month ago I decided to try varnish. It is designed from the ground up as a reverse-proxy. Also, I though that nginx solution wasted a lot of resources when keeping lots of tiny images in separate files.

But, after month of experiments, I discovered high iowait values and severe load on the hard disk, causing service problems. I rolled back to the previous nginx static scheme. Iowait dropped from frightening 100-150 to acceptable 25.

I used varnish 2.0.4 running with 3GB file storage. It consumed 0.5-1 GB of memory. Does anyone have a clue why varnish performed so much worser than nginx?

How to save $60 a month with 20 lines of code

We use Amazon AWS S3 service to store users pictures. There is huge amount of small pictures.

As the traffic at out WEB service increased, the cost for serving the pictures from the S3 increased also. Last month we had about 250 GB of traffic for pictures and this was around $60.

While we could move all the pictures to a separate static server this would imply some downtime and complexity in further migrations of our servers. And also reduce reliability.

I modified few lines in Nginx configuration to make it work as a proxy to Amazon S3:

   location @s3 {
       internal;
       proxy_pass           http://your-bucket-name.s3.amazonaws.com;
       proxy_store          on;
       proxy_store_access   user:rw  group:rw  all:r;
       proxy_temp_path      /var/static_data/tmp;
       root                 /var/static_data;
   }

   location ~* \.(jpg|jpeg|gif|png|ico|css|bmp|js|swf|mp3)$ {
       access_log           off;
       error_log            /var/log/nginx/static_cache_miss.log;
       expires              max;
       root                 /var/static_data;
       error_page           404 = @s3;
   }

Now the traffic for S3 has reduced from ~10 GB a day to less than 400 MB.