With so many new image formats coming out this week, it seems only prudent to put them through some paces. In this short feature, find some results about how the various algorithms actually work in practice. There are a few important questions I wanted to answer:
- Which one offers the best compression?
- Which one offers the most visually pleasing results?
- Which one offers the most numerically accurate results?
So I set out to test all three of them and see which one fared the best. Using first the CIE 1931 Chromaticity diagram, and then a real photograph, I ran several tests. The results are all inside after the break.
My initial testing started with the CIE 1931 Chromaticity Diagram, a great way to stress the entire color spectrum with some very fine gradients, along with some fine details from the gridlines and axes. I then performed three conversions:
- Using Imagemagick, convert the image to JPG and then back to PNG.
- Using the HIPIX software, convert the image to HPX, and then use the HIPIX Viewer to convert it back to PNG
- Using the WebPConv software, convert the image to WebP and then back to PNG
I then had 4 images in PNG format, the original image and the 3 processed images. You could very quickly see some artifacts of the conversion. See some of the results below.
Of course, the compression artifacts are only part of the story. What about the Filesizes?
- Original PNG – 334,891 bytes
- Compressed JPG – 115,868 bytes
- WebP on default settings – 28,094 Bytes
- WebP on high quality – 92,592 bytes
- HIPIX on Normal Quality -29,942 bytes
- HIPIX on High Quality – 71,931 bytes
So, the WebP offers the best compression, but at the default settings the quality is quite obviously degraded. At “high Quality”, the HIPIX image looks great and beats WebP on high quality for filesize, but at lower qualities seems to have difficulties with finer details as evidenced by the grainy text edges. So, is there a way to more accurately describe the quality?
Using some techniques I learned back in my old Image Processing classes, I decided to compute the Root Mean Square Error (RMSE) of each resulting PNG compared to the original starting image. This should give us a numerical ‘error’ value indicating how closely they match. I used the ImageMagick ‘compare’ utility to compute the RMSE value of each channel. (Simply ‘compare -metric RMSE image1 image2 null:’).
| Algorithm | RMSE Red | RMSE Green | RMSE Blue | RMSE All |
|---|---|---|---|---|
| JPG | 582.034 | 413.529 | 983.099 | 701.488 |
| WebP Normal | 971.199 | 779.245 | 1270.66 | 1027.14 |
| WebP High | 540.978 | 355.74 | 957.172 | 667.18 |
| HIPIX Normal | 2671.07 | 2551.81 | 2343.14 | 2525.65 |
| HIPIX High | 680.746 | 490.151 | 670.427 | 619.983 |
So, the ragged edges on the text cause ridiculously high RMS error figures in the HIPIX Normal test, but the HIgh-Quality HIPIX test winds up being the most accurate to the reference image. It seems to place less emphasis on the red channel and focuses on the Blue channel (as evidenced by the RMS error being higher there), but the WebP high is pretty decent as well.
So.. With that out of the way, let’s try a real world example.
The WebP high quality photo compression was most certainly *not* higher quality than the JPEG. Look at any red-to-dark colour border areas, especially those with a curve. On the JPEG it looks nearly as smooth as the original; however on the WebP there is clear blocky artifacting.
PNGs are great for the screen only, however if you’re designing for both the screen and print, a PNG does not convert smoothly to CMYK.
@MaxSt is now happier with the WebP out. Thanks!
A nice decoder/driver update fixed this.
http://img833.imageshack.us/img833/4313/comparewebp.png
It’s so hard to convince you… OK, instead of “taste” and what “looks better”, let’s talk numbers here. But first tell me – should chroma interpolation affect the luma? Because that’s happening with Hipix! Chroma info is bleeding into Luma!
Dark red circle is rgb=(132,0,0), Y = 39.
Dark green backgr: rgb=(0,64,16), Y = 38.
Notice how Luma is basically the same? Now lets look at the individual pixel (150,196) which is right on the edge between the circle and background:
JPEG 4:2:2 at 100%: rgb=(49,40,9) Y = 39
Hipix at Perfect 1: rgb=(78,45,21) Y = 52
What’s going on here? How come Y = 52 after Hipix? That’s 1/3 brighter!
Obviously, JPEG is doing the better job here, it keeps the Luma constant. While after Hipix some pixels are much brighter and some are much darker. Correct me if I’m wrong, but I think Luma shouldn’t be affected by chroma interpolation.
P.S. I’m using this formula:
Y = 0.299 * R + 0.587 * G + 0.114 * B
First. the test screen does have many diagonals (not reds) and curves – just look at the letters and numbers.
But more important: “At such high quality settings they should be almost identical” – Yes, you are right, but who is THEY? The hipix circle looks far more like the PNG source. There are no gradual shades of brownish red around it like in the JPEG one.
Your test screen is full of color boxes, but you would need diagonal/curved edges to see the red pixelization artifacts. So here I added a red circle:
http://img220.imageshack.us/img220/3585/circlej.png
Then I converted it into “hipix perfect 1” and “jpeg 100% at 4:2:2”.
JPEG 422 : http://img835.imageshack.us/img835/1623/circle422.jpg
HIPIX Perfect 1 : you can do it yourself
At such high quality settings they should be almost identical. But they are not.
Right and left border of the circle are blurred in JPEG, but pixelated in Hipix. Overall it looks better in JPEG.
Dear MaxSt. Please download the zip file at: http://rcpt.yousendit.com/974259031/930f53bf8d6b50dd50e48f7365547988
You’ll find there 3 files: 1920×1080 SMPTE test screen, the same test screen coded with Irfabview (quality 15, 80KB) and a hipix file (Good3 preset, 50KB). As you’ll see, at 1:1 the quality of hipix is extremely better than that of the JPEG. Look at the charachters and at the color bleeding of the JPEG. You won’t see any of it in the hipix image. Since you have the source BMP, you can repeat this test by yourself.
I updated, but I still see the pixelization in red areas.
Based on users feedback we have modified our presets to include 14 levels instead of 5. There was a chroma bug, which was fixed (thanks MaxSt). Anybody who downloaded the SW, just click on the update (under (?). Enjoy!
I see red pixelation here. Compare the girl’s face and the red area next to it. They look different.
Read my posts #6 and #8 here for details.
Dear MaxSt,
Look at this png screen capture: http://rcpt.yousendit.com/968609567/808b8548a01736edb6393609e121af20
Coded by hipix (high preset) 23KB. No artifacts at 272% zoom compared to the original JPEG. BTW, I coded the image using PNG ending up with 187KB… So, for those who wondered, PNG is not the best choice for photos.
@ Ira Dvir It’s possible, but I did make a conscious effort to leave the thumbnail out. I might’ve botched it tho. I’ll doublecheck tonight when I have some time.
@MaxSt
MaxSt, could you please send me the original iamge?
[email protected]
Ira, please test Hipix on this:
http://img408.imageshack.us/img408/6503/santaorig.png
What’s up with red pixelation?
Dear Randall,
Thanks for the tests you ran. Correct me if I am wrong, but it seems like you ran the test with the thumbnail creating checkbox checked. The thumbnail we add is (becuase of lack of cautiousness) is of 320×240 and it usually consumes about 12-20KB. These KBs are negligible when dealing with a few hundred KBs. However, with small images and filesize of sub 30KB, it is very significant. Just for the sake of the trial, compare the “Good” hipix preset with those images you used (for equal to WebP default size) making sure you compress without the internal thumbnail addition.
Thanks, Ira
No one is saying JPEG is that good. The question is, of the solutions that are better than JPEG, which of those is better than the rest? Anyone come across any JPEG XR comparisons?
Look here! Hipix against Photoshop. x2 advantage for Hipix tested by a pro photographer.
http://cyleow.blogspot.com/2010/10/jpeg-hell.html
@ Tim
Agreed. It’s easy to quantify filesize differences. It is much harder to quantify the artifacts and how different viewers will respond to them.
This is a terrible comparison. You have to compare quality at the same file sizes, otherwise the comparison is almost entirely pointless.
HEVC says that “H.264 is outdated, obsolete technology. Get with the times.”, and then links to a Wikipedia article on High Efficiency Video Coding (HEVC). That Wikipedia article says:
“The current timeline calls for completing the drafting of a final standard for HEVC by approximately July 2012.”
Therefore, since HEVC is not available in 2010, right now, that means that H.264 is not out of date. In fact, you can say that currently HEVC is vaporware.
H.264 is outdated, obsolete technology. Get with the times.
http://en.wikipedia.org/wiki/High_Efficiency_Video_Coding
webp is made with quality 95, filesize 33579.
hipix is made with profile “good”, filesize 33714.
Here’s also 2 versions of JPEG:
http://img710.imageshack.us/img710/2196/santacompare2.png
One with 4:4:4 subsampling (best), another with 4:2:0 (worst)
I choose the compression so filesizes will be close to 33kb.
Switching to 4:2:0 in JPEG affects the same area, but its decoder interpolates the chroma info, instead of just doubling the pixels.
I send it to WebP folks, and promised to fix the decoder:
http://code.google.com/p/webp/issues/detail?id=14
@ MaxSt For a more accurate comparison, you need to bench it against JPEG. PNG is lossless afterall, you need a common “gold standard” lossy codec. Also, what compression parameters did you use for WebP & HIPIX?
Here’s the result of my test (cropped and magnified):
http://img822.imageshack.us/img822/125/santacrop.png
I think both WebP and Hipix use 4:2:0 subsampling here, which means the chroma info is saved at half the resolution. But the real problem is that both decoders can’t upscale it correctly. Double fail.
I guess we’ll have to wait and see which buggy decoder will be fixed first.
P.S. Original image: http://img408.imageshack.us/img408/6503/santaorig.png
Yeah, png is great. Can I have png in my mobile handset and play it on my picture frame and TV? All these support h.264, so it’s likely that hipix will catch up. It seems more efficient. The only question is when will facebook allow you to have hipix images. I’d like to upload these from my Xperia. I tried it and the presets of what hipix call good and high are mazingly effective.
How about putting JPEG XR into the mix? It offers very high quality and unlike these other formats is fine with high-depth images.
HIPIX is dead on arrival because no one would touch proprietary, patent encumbered formats, everyone learnt that lesson well from the GIF debacle.
http://burnallgifs.org/archives/
WebP is mostly dead on arrival since it doesn’t really reduce file sizes much and the blurring and loss of detail is as bad as the JPEG blocking artifacts.
http://x264dev.multimedia.cx/archives/541
PNG is the only format you ever need. None of the artifacts of lossy formats, widely supported in modern browsers except a certain crappy browser from Redmond and everywhere, patent free, open format. Bandwidth and internet speeds will keep on growing by huge amounts and it will make all these lossy formats obsolete and irrelevant in the future.
http://en.wikipedia.org/wiki/Portable_Network_Graphics
I agree that PNG is far superior, but there is and always will be a place for lossy image compression codecs. As bandwidth increases, you’re right that we’ll see more PNG’s. But there will always be places where space is at a premium, be it on the web, in games, or in hardware, and those places will want to squeeze out every byte they can but keep the highest quality possible. HIPIX does have an uphill battle due to the patent restrictions, but I can’t deny that the results were the best amongst the formats I tested.
Google needs to cut their VP8 losses. It’s quite embarrassing. They shouldn’t bought out MPEG LA and open sourced their stuff.