9029 views 29 comments

Image Comparisons: PNG, WebP, and HIPIX

by on October 7, 2010

With so many new image formats coming out this week, it seems only prudent to put them through some paces.  In this short feature, find some results about how the various algorithms actually work in practice.  There are a few important questions I wanted to answer:

  • Which one offers the best compression?
  • Which one offers the most visually pleasing results?
  • Which one offers the most numerically accurate results?

So I set out to test all three of them and see which one fared the best.  Using first the CIE 1931 Chromaticity diagram, and then a real photograph, I ran several tests.  The results are all inside after the break.

My initial testing started with the CIE 1931 Chromaticity Diagram, a great way to stress the entire color spectrum with some very fine gradients, along with some fine details from the gridlines and axes.  I then performed three conversions:

  • Using Imagemagick, convert the image to JPG and then back to PNG.
  • Using the HIPIX software, convert the image to HPX, and then use the HIPIX Viewer to convert it back to PNG
  • Using the WebPConv software, convert the image to WebP and then back to PNG

I then had 4 images in PNG format, the original image and the 3 processed images.  You could very quickly see some artifacts of the conversion.  See some of the results below.

Of course, the compression artifacts are only part of the story.  What about the Filesizes?

  • Original PNG – 334,891 bytes
  • Compressed JPG – 115,868 bytes
  • WebP on default settings – 28,094 Bytes
  • WebP on high quality – 92,592 bytes
  • HIPIX on Normal Quality -29,942 bytes
  • HIPIX on High Quality – 71,931 bytes

So, the WebP offers the best compression, but at the default settings the quality is quite obviously degraded.  At “high Quality”, the HIPIX image looks great and beats WebP on high quality for filesize, but at lower qualities seems to have difficulties with finer details as evidenced by the grainy text edges.  So, is there a way to more accurately describe the quality?

Using some techniques I learned back in my old Image Processing classes, I decided to compute the Root Mean Square Error (RMSE) of each resulting PNG compared to the original starting image.  This should give us a numerical ‘error’ value indicating how closely they match. I used the ImageMagick ‘compare’ utility to compute the RMSE value of each channel. (Simply ‘compare -metric RMSE image1 image2 null:’).

Algorithm RMSE Red RMSE Green RMSE Blue RMSE All
JPG 582.034 413.529 983.099 701.488
WebP Normal 971.199 779.245 1270.66 1027.14
WebP High 540.978 355.74 957.172 667.18
HIPIX Normal 2671.07 2551.81 2343.14 2525.65
HIPIX High 680.746 490.151 670.427 619.983

So, the ragged edges on the text cause ridiculously high RMS error figures in the HIPIX Normal test, but the HIgh-Quality HIPIX test winds up being the most accurate to the reference image.  It seems to place less emphasis on the red channel and focuses on the Blue channel (as evidenced by the RMS error being higher there), but the WebP high is pretty decent as well.

So.. With that out of the way, let’s try a real world example.