JPEG XL

https://www.gdcc.tech/rules/

dim 3 and 4 in general have much weaker prediction

<https://www.ebi.ac.uk/empiar/> <https://www.sdss.org/> <https://pds-atmospheres.nmsu.edu/data_and_services/atmospheres_data/catalog.htm>

Maybe something like JXL + internal Brotli (for data) might work

you can always stuff whatever you want in a JUMBF box of some sort and then brotlify it...

But brotli encoding takes more time. Encoding a image with brotli will be slow. Isn't?

Because it's not just images, it's images and specific data

🤔 https://bugzilla.mozilla.org/show_bug.cgi?id=1723555

Brotli has a speed setting, it can be very fast and still better than gzip

So brotli image format :p

Brotli compression mode is already available and supported in JXL

I thought it is only for metadata

We used to have Brotli itself as a separate Modular entropy coder backend, but we removed that from the spec

Instead we added lz77 to the ANS entropy coder

Which basically gives the same power as Brotli except for the initialization dictionary (which is useless for images anyway)

That means you don't need Brotli if you just want to decode pixels (which is quite good because otherwise libjxl_dec would be at least twice as large)

For metadata we do still have Brotli though, since 1) Brotli is useful for that, and 2) we assume decoder binary size is not a big deal in use cases where you care about metadata

And also Brotli lib is usually already available in browsers

What if you remove jpg decoder and separate brotli libs in browsers and use just djxl for those things

Or djxl uses brotli libs of browsers so doing it doesn't make any sense?

I think so, and for Jpeg it is possible, but the Jpeg encoder is still used and needed, so that is the problem

Is it possible to predict missing chroma by luma on 4:2:0 recompressed-jpg jxl? Or Something like this : https://gist.github.com/igv/a015fc885d5c22e6891820ad89555637

The thing with chroma from luma in jxl is that in case of 4:2:0 it will use the top left quadrant of luma to predict chroma, which is quite likely totally not useful

<https://encode.su/threads/3108-Google-s-compression-proje%D1%81ts?p=70534&viewfull=1#post70534>

As far as I remember it requires JPEG XT (although I have seen 10-bit modifications of regular JPEG, but they will not be compatible with most decoders)

well, DCT coefficients of JPEG 1 are 12-bit

you can gain a few bits using more precision in IDCT + ycbcr conversion

I'm not 100% sure how many more bits you get that way, but I suspect you gain one or two bits over "traditional" JPEG decoding

possibly yes

didn't test it yet

the simplest way to try it out is decompress to PNG with libjpeg and get version 1, and do lossless JPEG recompression and then decompress that to PNG for version 2, and then see if you can tell the difference 😛

yup

iirc libjpeg, libjpeg-turbo and mozjpeg all decode in the same way

I believe yes. My back-of-the-envelope (or actually back in the furthest neuron) hints that JPEGs can be decoded to about 10.5 bits

jpeg xl computes intermediate results with higher accuracy -- but even more accuracy is possible than jpeg xl is producing today (i.e., we can go to HDR10+ with traditional jpegs)

how exactly? floats are pretty precise as it is...

What's the PAE and MAE we can get on a jpeg with all-1s quant tables?

I suppose we need a jpeg encoder that does ycbcr and fdct in high precision (starting from 16-bit png, say)

today we would show traditional jpegs in an 8 bit frame buffer in the end

What about chrome? Wouldn't we give it half float buffers if the jpeg has a PQ or HLG icc profile?

But we need a jpeg encoder that takes more than 8-bit rgb as input

to be honest, I don't know -- if we do, that's great

iirc, libjpeg can optionally be built in 12-bit mode

that was one reason for using `JSAMPLE`/`MAXJSAMPLE`/`BITS_IN_JSAMPLE` in `codec_jpg`

but I don’t think we have ever actually tested it

Does 12-bit mode libjpeg produce jpegs that can be decoded with default libjpeg?

that’s something I wasn’t sure of and would like to try

it's going to be subtle for sure

... probably? not 100% sure

well, at least a 12-bit JPEG cannot be read by imagemagick nor cjxl

Yes, but so is arithmetic coding, lossless jpeg, and hierarchical jpeg, which are all in the 1992 spec but not supported by most jpeg decoders

oh, disappointing

was it one you generated yourself?

nah, random one from the libjpeg-turbo test suite

anyone use android 12. avif works on it?

android11, opens fine in chrome but not in files. i wonder if/when ill get update to 12

still in beta. not anytime soon i guess

Pixel phones have android 12 beta

Someone ported android 12 to my phone. But I'm lazy to install it.

https://medium.com/adobetech/image-optimisation-with-next-gen-image-formats-webp-and-avif-248c75afacc4

> How do we ensure if the image has same visual quality? Answer is PSNR (Peak Signal to Noise Ratio) 🤔

Ugh why do people keep claiming psnr says anything about visual quality...

I guess engineer love PSNR? > PSNR, PSNR-HVS, Y-PSNR, X-PSNR

PSNR correlates with human perception like the weight of a painting in kilograms correlates to its artistic value.

well, maybe to its economic value 😛

because it’s _objective_!

it answers the wrong question, but objectively

like frequentist statistics

Right. It reflects the amount of paint that went into the painting, but it doesn't care about where what color went. Pretty accurate analogy.

there are similar problems in medicine

tests that turn out not to tell at all what they are supposed to tell

but nevertheless they keep being used dogmatically because “it’s the best we have”

with appeals to consequences (https://www.logicallyfallacious.com/logicalfallacies/Appeal-to-Consequences) like “if we are not to use this test, it means that we have no viable test for this”

and, well, yes, it does mean that

(sorry for the digression)

basically, wishful thinking is rampant

it’s good to be on the lookout for it (including in oneself)

In case of psnr, better metrics do exist though

But even those are not perfect

So I guess many people have the attitude "no objective metric is a substitute for subjective eval, so we might as well use the simplest and oldest metric"

ah, so a nirvana fallacy (https://rationalwiki.org/wiki/Nirvana_fallacy) in response to new metrics?

Yes, I think so.

to give PSNR some credit here, it is one of the few (only?) metrics I know how to solve some optimization problems for in an exact way, and in some cases that's even useful

(in general, those cases are the ones where you manage to transform your image so that some weighted PSNR actually helps in that space :P)

It's also useful if your problem is not visual optimization but just "storing a 2D array of numbers in a lossy way".

https://twitter.com/jonsneyers/status/1425512124063813640?s=19

WebP2 has triangular previews, which is an interesting thing, but it is an additional image for the preview, unlike JXL, where it is part of the image <https://000566524.codepen.website/?preview=AzFhqCBwmt34gyNzfH7iVoErk7T3RW6hS6Xo4vEq2AT2TcphQkyZZpyDDyZxHoGflzF7crbOvhvvl26pVjYkNWgdBdprEeqUURSfwEwXiVuHmt2kuSl8lO5ThRSfluaOTDjc/P/EdTHUHFkia/wBTh73YNYS9zFrrbzGtk8tS9gg3dJ5bAjiwmM+gxXvIu5rm42ObIr0YMxMgeAtWjV3mH/ita9nKI9PrRD7qvuQGoDg5BPhAO+kPXhddDDziI8Ocb3MLImE46ILFAEJTDeV4PvG2srfDAWVp+h8Dntgm4FnTr4h+NR23ExOx8CLakoSHI598ZzMOVMW3g==>

So far, WebP2 differs from AVIF in more well designed as a format specifically for images and also better lossless compression, but if consider the combination of AVIF and Jpeg XL, it loses in almost everything

Personally I would like to see WebP2 instead of AVIF, but when AVIF already exists this is a big problem, now for its existence WebP2 should be much better than AVIF (currently it is either a little better or about the same or worse in some areas)

AVIF does seem like a bit of a bodged together thing that just came out of the side of AV1.

The encoding is pretty good, but it lacks a lot of the niceties of a dedicated image format.

AVIF is more like something made very quickly for images based on things already used or planned to use, like AV1 is already there as a decoder anyway (or an encoder somewhere), HEIF is also already used to support HEIC format on many smartphones (at least for decoding), so, like, let's take these ready and used technologies

That's exactly what it is. It's a great video codec with a mediocre and verbose image container around it.

LMAO, does he know that distance is based on a metric as well?

Drop Butteraugli and add PSNR <:Hypers:808826266060193874>

Well to be fair, there might be cases when the 'image' is not visual and then Butteraugli will not make sense, while psnr might.

I think the guy is having trouble understanding however... https://github.com/libjxl/libjxl/issues/436#issuecomment-898218016

If you do `-m -c 1 -Q 50,70` or something like that (no XYB, and 'higher' chroma quality than luma quality so they end up getting similar loss), it does basically psnr optimization

(Lossy modular is quite simplistic and always optimizes for PSNR, just with different amount of loss for luma and for chroma, and by default it works in XYB)

I think he needs tool a compare to compare other image formats using butteraugli

I'm not sure. I think he wanted one thing initially: 1. Have target PSNR/SSIM to objectively measure quality, not having to use target file size/target bpp Then it devolved to him wanting that, and an easy way to compare encode "quality" using PSNR/SSIM. 😛

Actually, how did he get to looking at the file size/bpp option without looking at distance in the 1st place?

SSIM is slightly better but almost as bad as psnr

Multi-scale ssim in a perceptual color space is getting a bit better

Doing aggregation of the heatmap with a higher norm than the usual 'just average it' and you start having a reasonable metric

That's what DSSIM and SSIMULACRA do

They're ssim-based, but 1) in a perceptual color space (not ycbcr), 2) multi-scale, 3) not just looking at avg error but also penalizing worst error

It's better, but SSIM isn't optimal either, for one reason it particular: it only takes into account luma values, even though it takes into consideration frequency data in its analysis, while only spitting out an average value.

For example, as an RD tune, SSIM is good for better detail retention, but as seen in aomenc and x265, using SSIM as an RD tune means colors get sacrificed in the process, and so do low variance spots in the image.

They use Y-only ssim? Or chroma just gets too low weight?

Chroma gets too low weight in that mode.

It's not really an issue where the color are vibrant and saturated, but any time colors gets slightly less saturated, you can easily tell the SSIM RD tune just makes colors look noticeably worse. It's not a large difference, but it's not really subtle either.

https://ai.googleblog.com/2021/08/soundstream-end-to-end-neural-audio.html

https://1.bp.blogspot.com/-tzWjqrUOrvk/YRRAc81_kJI/AAAAAAAAIBU/wVzK2cKGjakFls0JSvcE93G6hFMmhg_bgCLcBGAsYHQ/s1061/image1.png

>The quality of the reconstructed audio using either of these codecs is excellent at medium-to-low bitrates (12–20 kbps), but it degrades sharply when operating at very low bitrates (⪅3 kbps) why would anyone want 3kbps, whatever codec it'd be

aint networks getting faster, and things take advantage of that (eg higher quality VoLTE stuff)

For many things, for amateur radio, for data transmission from space, also the network may not always be stable even at high maximum speed and such a codec can scale well for different bandwidths

https://google-research.github.io/seanet/soundstream/examples/

Just send things as text instead of audio if bandwidth is such a problem, imo...

I prefer 1kb of text over 50kb of overcompressed audio that sounds ok but the AI might be subtly changing the words.

Mostly yes, but it's not always convenient or possible to type or translate voice to text, also this SoundStream is suitable for everything, not just voice, unlike the last Lyra implementation, also it can dynamically scale to unstable network and bandwidth, so it can be useful in many use cases, just like it is better to hear something at low bitrate, with unstable connection, than to drop connection or hear constantly laggy or unclear speech at higher bitrate Also after significant progress in video compression, audio compression has been a bit behind and with very limited bandwidth it's not uncommon for audio to take up more bandwidth than video (because otherwise the quality wasn't enough to be understandable)

I'm in awaiting Google SoundStream vs Microsoft Satin comparison 👀

btw, ms ai codec in production, google is half a year behind

For the times when my reception is absolutely awful and to use as little data as possible.

For reference, I only have a 250MB data cap on my mobile plan.

such low datacaps are absolutely vile in current year

https://twitter.com/Android/status/1366801139547660289

https://github.com/google/lyra > Earlier this year, Google released Lyra, a neural audio codec trained to compress low-bitrate speech. SoundStream extends this work with a system consisting of an encoder, decoder, and quantizer

Yep, it's basically an improved Lyra, for voice SoundStream also works better, according to the examples

I'm actually surprised by one thing: why didn't they compare to xHE-AAC?

Android 11 natively supports xHE-AAC decoding, so it would have been interesting to test it against the best of the best.

Probably because xHE-AAC is not for VoIP

I suspect it'd perform pretty bad

> Improved quality at stereo bit rates of 12 to 500 kbit/s or above so yes it's not for 3kbit/s

It's better than Opus between 12-32 kbit/s, but it can't be lower than 12 kbit/s

that's actually a good thing :^)

I wonder how it compares to opus at higher bitrates

so it turns out that with a good encoder and decoder pair (like in https://github.com/libjxl/libjxl/pull/427) you can actually make HDR JPEG 1 that are visually lossless (at least on my TV :P)

https://old.lucaversari.it/hdr/ files I used for testing

(needs a JXL-enabled chrome 'cause that's what I used)

(none of that code even tries to get decent filesizes yet, you've been warned :P)

Oh that's very bad.

for my macbook/retina they look identical, but jxl a little bit darker on each file

is it with Chrome with HDR support?

not sure, just chrome with jxl enabled flag

you can tell by how the .jpg look like

if the .jpg look the same as the .jxl, then it's no HDR

(just for completeness: this https://old.lucaversari.it/hdr/hbd_djxl.jxl is a full-high-bit-depth pipeline, https://old.lucaversari.it/hdr/mozjpeg.jxl is a 100% JPEG 1 pipeline, and https://old.lucaversari.it/hdr/orig.jxl is the original)

now i found the difference for mozjpeg.jxl. Clouds and windows on the building have a sharp structure. hbd_djxl.jxl and orig.jxl looks same

btw, these jpeg files do not work in safari

what do you mean they don't work in Safari?

hbd_djxl and orig looking the same is kinda the point yes 😛

Looks bad on chrome Android

HDR avif also look bad like this on Android chrome.

that's not visualized in HDR mode

at least not in the screenshot xD

but that's also some rather extreme banding...

(what file is that? that's worse than anything I've seen myself)

The original and jxl ones look nice <:Poggers:805392625934663710>

~~If you ignore the terrible chromatic aberration~~

what...

that's the photographer's fault 😛

Yeah <:cheems:720670067091570719>

At least it works properly for me in HDR mode

does it work in other software? I have to admit I don't have a lot of patience for whatever it is that Safari might be doing here

mac preview not working too

gimp works

well that's at least consistent

does `mozjpeg.jpg` work, btw?

yes

https://old.lucaversari.it/hdr/hbd_djxl.jxl this

ok, seems like something in hdr on Android is broken

sounds like fun... I assume I did something that Mac's JPEG decoder doesn't like

I think it's a chrome bug. avif HDR worked fine on Firefox but not on chrome.

ah, yes, I meant HDR in Chrome 🙂

ah I think I know what's wrong

missing APP0 marker... gimme a minute

can you try again? I should have updated the files

https://evilmartians.com/chronicles/decoding-avif-deep-dive-with-cats-and-imgproxy

So maybe that's what makes AVIF good at compressing art images

that article is pretty good

no, for now it's just a JPEG 1 *en*coder - I do decoding with cjxl to do lossless recompression and then djxl

it's kinda silly, but representative of the pixels you'd get if you implemented JPEG 1 decoding with libjxl

just without actually coding it 😛

seems like that

so far I only got ~5bpp HDR JPEGs xD but I don't see a reason why it shouldn't be possible to do much much better

did we try with `jpegtran -optimize` in the end?

yup, the number is after running that

oh

(also it's apparently needed for it to work on OSX... don't ask me...)

isn’t that chroma from luma?

it is 😛

CfL in JPEG recompression will increase the per-pixel error *slightly*

you mean with CfL for JPEG recompression?

the trick is that for JPEG recompression CfL is applied on integers 😄 (but not when decoding to pixels) - so the encoder can do the exact opposite and when you want to get the DCT coefficients back, you can

I don't think it affects quality much either way (likely CfL gives you slightly better quality than not doing it, as long as the encoder takes it into account), but a decoder that starts from a JPEG cannot assume that the encoder was aware of it

I had a go at making an encoder that would produce decent bitrates for HDR (1.5~3 bpp depending on the image), I think it looks pretty good (there are a couple of artefacts here and there but kinda hard to notice unless you put your nose on the screen :D) but (a) I didn't look for too long and (b) I literally wrote this in two hours, so it could be super suboptimal 😄 files are here if anybody is interested: https://old.lucaversari.it/hdr/smaller/

but I think it's about time for me to switch to "weekend mode" 😛

i get used to avif

the more i want 480p at 2,2 px fonts

like windows 11

the more i get used to avif

Ok. just use avif. Good for you.

i don't like avif

but the point i'm get used

1509x1080 explicit in webp 56,1 KB (57.454 byte)

I tried compressing jxl source code with dependencies with brotli but file size and encoding time is larger than xz. So, Brotli works good only for htmls?

Not only, Brotli as well as Zstd have the main advantage in decompression speed, but at high compression modes they will always have slower compression than LZMAx (with still fast decompression)

could be worth trying with `--large_window=30`

https://www.reddit.com/r/PleX/comments/6y9211/thinking_about_switching_your_library_to_hevch265/ already in the txt <#794206170445119489>

old 2017 reddit post

> Minimal image container: add convert tool https://chromium.googlesource.com/codecs/libwebp2/+/be4db2b35eb3940a21f0a40363afb60fc56484f7%5E%21/ Looks like AVIF with minimal container/headers? 🤔 But perhaps this is only for WebP2 or for AVIF-like without HEIF and most likely which is not compatible with the specs

AFAIU they want to extend the AVIF spec to have a more "decent" container

Makes sense, but also somewhat problematic to keep changing a spec. That makes it hard to know what "I support `image/avif`" actually means...

it works now

good, thanks!

`jpegtran` gives an error: `Premature end of JPEG file`. How can I fix such a file? Because `cjxl` doesn't want to read it.

Viewers open the file without any problems.

If I add `FF D9` to the end, I get another error `Corrupt JPEG data: premature end of data segment`.

But I don't see any visual corruption on the image. So I think it can be fixed.

It's likely a progressive jpeg that was truncated

Probably the bottom is a bit lower quality than the top

Silly question. `jpegtran` can do rotation and do it losslessly, right? Then I can just rotate the image twice and fix it. <:kekw:808717074305122316>

that doesn't give an error?

you can also hack jpegtran to not abort on that error, I guess

that's what most viewers do: just assume that the missing coefficients are all zeroes

It gives an error on the first rotation, but creates a new rotated file.

`jpegtran -copy none -optimize -outfile 1.jpg 0.jpg` gives an error and creates a new file, but it is just a copy of the original with the same problem. But with `-rotate` the file has `FF D9` at the end. So I think that would be the way to fix it for me.

There is a difference, after all. ``` PS > jpegtran -rotate 180 -outfile 1.jpg 0.jpg Premature end of JPEG file PS > jpegtran -rotate 180 -outfile 2.jpg 1.jpg PS > ssimulacra 0.jpg 2.jpg 0.05208653 PS > ls 0.jpg,2.jpg | Format-Table Name,Length Name Length ---- ------ 0.jpg 2961762 2.jpg 3665479 ```

Fixed it with `-trim` without rotation. I guess this is the best way to minimize damage to the original file.

https://news.ycombinator.com/item?id=28187895

https://twitter.com/jaffathecake/status/1427950852241870855

Also, even without lossy color reduction, Oxipng is not the most efficient optimizer, so other services can create smaller PNGs in lossless mode too

I wonder who came up with how `clap` (CleanAperture) is defined in heif/isobmff but it is seriously messed up

It's just a crop rectangle definition

In jxl we define it in a simple way: a frame can have a width and height that differs from the image dimensions, and it has x0, y0 offsets, and then it gets painted on the image canvas with its topleft corner at position (x0,y0).

In avif/heif, you use a `clap` box to do cropping, and it consists of 8 (!) signed 32-bit numbers

The numbers are numerators and denominators, so you can put rational numbers there (but they have to be integers in the case of width and height, so those denominators are completely silly)

The offset is not an x0,y0 of the topleft corner, oh no

It is the offset of the *center* of the crop compared to the *center* of the image

Which is why you need rationals: if you want to crop a 9x9 image from the topleft of a 10x10 image, then the center of the crop is -1/2 pixel above and to the left of the center of the image

Nevermind that you could avoid that by just using the topleft corner coords instead of center points, making those 16 bytes of denominators totally unnecessary.... The plot thickens!

One reason to use `clap` cropping is that hevc and av1 cannot represent images with odd dimensions when they use 4:2:0

So if you want to have an image that is 99x99 pixels and 4:2:0, you need to encode it as a 100x100 image and then crop it at the avif/heif level

But a recent amendment of the isobmff spec from which avif/heif derive the `clap` spec says that in case of 4:2:0, the crop dimensions have to be even!

That means that now a `clap` box that would use such a -1/2 center offset to crop a 100x100 image to 99x99 had become illegal (and viewers will either ignore it or refuse to show the image)

Meaning there is again no way to do a 99x99 image in 4:2:0 in heic/avif with the `clap` property (but there are other tricks, I think)

It's just so incredibly messy and unelegant, using 32 bytes just to specify a crop (something that will typically take 7 bytes or so in jxl), with way more restrictions than seem reasonable

E.g. you also cannot crop the central 100x100 pixels from a 102x102 image in a 4:2:0 avif/heif, because the coordinates of the topleft pixel also need to be even

(it kind of makes sense, if you want to stay in yuv420 then you would need that restriction, but still, that's a very video-oriented thing to restrict what you can do in an image codec by assuming you will never go to rgb)

TL;DR: the `clap` box wastes quite some bytes on a rational representation of crop dimensions/offsets, reflecting a typical "designed by committee" attitude of "let's make it as generic as possible", and then later on they restricted things to integers anyway, and even in case of 4:2:0 to _even_ integers only (but the bytes wasted in the bitstream syntax remain wasted)

Oh and all of that is as recent as 2021

ISO/IEC 23000-22:2019/DAM 2:2021

You can find older versions of that spec online but not the current version

Because of the ISO paywall

So if you think the avif spec is any better than the jxl spec in terms of open access, think again

Most of the avif spec consists of references to MIAF/HEIF and to recursively resolve those references to their most recent version, you do hit the ISO paywall

</end rant>

no it's not 🙂

yes

you lose half a bit or so with the colorspace conversion, and a bit more with the DCT

I think you can notice more easily in specific gradients and such

TN panels are not fully 8 bit IIRC as it is so...

dark areas and some peculiar blue-green shades can show it more easily

Banding artifacts are more visible on a true 8bit panel

I've experienced it

often 6 bits + FRC

I mean, most TN displays are just 6bit + frc. They do dithering

I'd guess it depends on the manufacturer/model 😄

you normally wouldnt notice it even side by side to an ips

that, I doubt 😛 at least for people that did a lot of image comparisons...

yes, if you first nearest-neighbor upsample the image 8x so only the DC coefficient is used, and then downscale again 😅

but no, let's just say jpeg is always lossy

*and* don't use YCbCr 😛

`convert foo.png -sample 800% upsampled.ppm; cjpeg -quality 100 -rgb -sample 1x1 upsampled.ppm > foo.jpg`

Then `convert foo.jpg -sample 12.5% decoded.png` might be lossless

(haven't tried though)

🤔 <https://chromium.googlesource.com/codecs/libwebp2/+/refs/heads/main/doc/container/#conversion-from_to-avif> > AVIF is a MIAF/ISOBMFF wrapper around an OBU bitstream, which is itself usually made of a sequence header, a frame header and an AV1 tile group. > > Stripping everything but the AV1 tile group and the necessary fields in the sequence and frame headers would produce lighter web image files: > - Easy to parse and write, > - No redundant field that could be contradictory and/or invalid (image dimensions stored in MIAF, sequence header and frame header), > - Fewer features means low vulnerability, > - Smaller files, especially for tiny images or icons.

Comparison between a minimal header (left) and a Sequence Header OBU (right):

JPEG's DC encoding is kinda really really bad 😛

lossless transcode could do a *lot* better than that, but doesn't try to do i.e. RCTs in that case because 99% of cases your image is YCbCr and RCTs are not that useful

Btw, even VP10 had a palette coding and it is not a separate mode for the entire image/frame, also as far as I understand this mode can apply all the filtering and other stuff, maybe some things can be applied and will be useful in JXL 🤔 https://youtu.be/MQhVGAUrWyk?t=1285

We have a "no transform" (not really no transform, but no dct) in VarDCT

And of course there's patches or multi-layer where things can be done in Modular mode

Not sure if we have good heuristics to select Hornuss (the arbitrary name we gave to the "no transform" option, since it still does something, so "Identity" was considered to be a bad name)

So it's kind of a patches, too 🤔

Intra block copy is like patches, except with patches you first encode the "sprite sheet" (hidden frame) and then the main frame, so decode of the main frame can still be done in parallel. Intra block copy makes decode inherently sequential.

So this is another thing for the usefulness of adding multithreaded decoding in browsers 🤔

<https://encode.su/threads/3397-JPEG-XL-vs-AVIF?p=70658&viewfull=1#post70658>

**SolidComp** > One challenge with hardware acceleration, including GPU, is that Firefox and Chrome seem to be throwing their users under a bus when it comes to modern codecs like HEVC. They're choosing not to use the built-in OS decoders, which are likely to be hardware-based or at least GPU accelerated. Windows 10, macOS, etc. have full support for formats like HEVC, and before that H.264, but those creeps at Firefox and Chrome refuse to use those built-in APIs. The whole drama of whether a browser "supports" these formats is deceptive, because they don't need to – it's already built into the OSes, so the browsers don't need to do anything but use the features. For a while Firefox was offering a form of support for H.264 that involved repeatedly downloading an open-source plug-in from Cisco, instead of just using the decoders built into Windows 10 or whatever. Maybe they still do that. I don't know what their exact arguments are, but it might some leftist ideological nonsense. The OS codecs are free, so it's not like they'd have to spend any money to support HEVC and so forth. > > In any case, there's not much transparency or rigor in the development of these new image formats. They don't seem to do the kind of full-effort deep thinking about hardware acceleration, GPU acceleration, or SIMD optimization in advance that I'd expect from billion dollar companies, before designing the codec. It seems like it's always just an informal effort by a handful of guys at Google and that photo hosting company, with unclear supervision or accountability, and no real concrete goals or benchmarks, especially regarding hardware, GPU, SIMD, etc. Or battery. Battery is huge. I wouldn't do anything at all without setting goals for real-world battery usage, and then design and deliver a codec that hit those goals. Hardware acceleration usually means big battery savings.

> One codec under development that specifically considers hardware optimization from the outset is AV2, a potential successor to AV1. Apparently AV1 sucks for hardware implementation (that they introduced a video codec at this point in history without optimizing it for hardware is just incredible, and illustrates how poorly supervised these Google teams are). So now some people are trying to follow up with AV2, with hardware in mind. I don't know if it's about hardware encoding or decoding or both.

based perspective

"the browsers don't need to do anything but use the features" <- that's an interesting perspective... (not particularly *correct*, but interesting)

having said that the amount of truth in the rest of the post feels kinda low, so there's that 😛

I guess the reason they don't is so they can work cross platform?

Yes, using OS-specific stuff is not exactly great for portability.

There is also the *small* detail that 99% of the code that a browser writes for a format has nothing to do with decoding the format, and everything to do with interacting with the decoder library for that format...

how to convert in libaom with ffmpeg media encoder on android

i'm at 26 frame after 3 minutes and 50

with a 640x800 video

30 fps

1% after 5 minutes

is running 1,1 fps

Just don't.

snapdragon 780

i bought best processor

why so bad

1:20 83 fps

Bruh

so is risky to force threads in a smartphone

smartphone don't work in threads

they aren't pc

how many threads use snapdragon 780 by default

i read 4

but at what frequency

Even Snapdragon 8xx series can't encode av1 at reasonable speeds.

is termux any faster

why lack of ARM

so rav1e works better?

Don't use mobile devices for av1 encode

instructions

assembly

yes but is encoding fairly great

can i wait until it ends

why not

at least for one video

Do you want to ruin battery?

00:53 seconds

is

16 % in 5:58

pog

i expected much better

but the video will be good quality

even 4 cores can ruin battery of entire phone

or even 1 core

is core a different concept than phone

like even 0 core can run battery of a phone

i will not force threads

i know is a phone

Just use a pc to encode av1.

it needs 40 minutes

don't know if is right

Stop it

i stopped

i thought it was a smartphone for gaming

the snapdragon 780

my mom said or samsung or xiaomi

because she don't like oppo

so smartphone for gaming don't exist?

i remember you compressing a jxl in lossless modular with a similar phone or maybe orum (user) at 8 mpx/s

Encoding jxl is okay but not av1.

Good mom

What has av1 so complex? Is It a video codec?

You're asking this question even after spending a year at av1 discord server?

3 years and half

Also, from AV1 discord, well optimized software vs. hardware decoding

I guess hw decode is still better for battery

But yes, that's a rather good illustration of why hw decode for still images doesn't really make sense

For video, if you're going to be decoding for several minutes, maybe hours, then hw decode is nice because battery impact of cpu decode is somewhat significant

For still, battery impact of cpu decode is kind of irrelevant

the tldr being software decoding is faster?

well, I suspect this is different on mobile

I wonder if one could make a gpu-backed sw decoder that would be faster...

yeah how do i know that isnt a 5995wx

nvidia also has jpeg decoding

and it's only meant for mass jpeg decoding for machine learning (for nvidia's case)

No idea, do you know what is using that hw decoder?

Should I keep renewing flif.info ?

i don't track visitors, it's just a domain name that points to FLIF-hub.github.io

cost is not a big issue, it's 14.5 EUR per year

just seems a bit unnecessary to keep it around since I consider it superseded

meh, I'll keep it for another year

that's pretty expensive for a domain 😛

yeah i'm probably not using the cheapest registrar, also there's the usual trick that the first year or first two years it's cheap and then it's more expensive if you want to keep the domain

if you want expensive, look at .io

(or worse, .tt, but that’s more niche)

yes, .io is around $40 / year

Because mindfreeze only has a 10900k <:kekw:808717074305122316>

can you tell us a bit more about the test setup? 😛

10900k with stock power limits(Skylake 10C/20T) with Git dav1d at the time. RTX 3090. 64GB of RAM at 3200MHz with CL16 XMP timings.

ok, so definitely not "cheap gpu on super powerful cpu"

(although I guess nvdec performance doesn't depend on the GPU performance?)

Indeed. NVDEC only depends on its HW decoder and some memory bandwidth.

I assume sw decoding was multi-threaded

Yes, definitely not single threaded 😛 I don't think a single Skylake CPU core can do over 1000MP/s on a single core of video decoding at 1080p.

it might be interesting to redo that with different core counts

I mean, it does make sense to use hw decoding if the alternative is basically halting the rest of the system because all the CPU cores are crunching pixels like crazy...

Yeah, and in this case, even with a 3090 that has some very high idle power draw, I'm sure it's more efficient than a monster 10900k decoding stuff at 4,5-5GHz 😛

But, NVDEC at the moment is also the fastest AV1 hardware decoder (something like 2-3 times faster than what AMD video cards have), although, I do not know about the Intel decoder speed

And current decoders in smartphones are probably even slower, but for power saving yes, they are useful

Actually, throughput on Samsung's AV1 HW decoders is excellent for a mobile device.

Throughput is very high, which is why it can do 8k30 10bpc.

However, this is the first generation of AV1 hardware decoders, judging by AVC and HEVC their speed with new generations can significantly increase (more than the increase in performance of new CPUs)

The annoying thing with hw is that it takes quite a lot of time and up-front investment to make it, and then it adds cost to devices even if there are no royalties to be paid (SoC area, amortized design costs)

I wonder if we shouldn't aim more for a modular hw design where the hw doesn't do specific bitstreams but it does generic things like dct transforms and convolutions - sufficiently standardized and ubiquitous that software can rely on it being there the way it now relies on floating point arithmetic to be there

I suspect (but do not know) that you can get 50% of the way there by doing GPU decoding

I don't think there are that many standardized components in video codecs though 😛

one annoying limitation is that IIRC they only do 420

Indeed. Profile 1 4:4:4 decoders will come later.

But at least all should support 10-bit, as far as I know

I mean more like extra SIMD that does things like say a 1D dct8/16 or other useful building blocks

4:4:4 12-bit hw decode when

av1 HW decoder also can decode full-range avif(ycocg)?

Is YCoCg even in the spec already?

I guess? I not sure this is a full implement. > https://github.com/AOMediaCodec/libavif/blob/master/CHANGELOG.md > 0.8.4 - 2020-11-23 > Added > YCgCo support (full-range only, wantehchang) > avifenc --cicp 2/2/8 or --cicp 1/13/8

I remember YCoCg define in h.273. > https://www.itu.int/rec/T-REC-H.273-201612-I/en

https://github.com/AOMediaCodec/libavif/blob/772b49e2c313181eb7cf15780ce30782b803996f/src/reformat.c#L286

That's not the reversible YCoCg

I wonder what the point is of doing YCoCg if it's for lossy only - it's not particularly better than YCbCr in terms of being perceptually relevant

https://github.com/AOMediaCodec/libavif/blob/772b49e2c313181eb7cf15780ce30782b803996f/src/reformat.c#L610

There's a reversible one there though

Anyway, a hw av1 decoder will probably have yuv output and consider it not its problem how to interpret it

Probably yuv420 only in most cases

And then the yuv420 will go to the gpu, which will probably support it natively (but I dunno if you can tell it to do something different from the usual ycbcr)

YCoCg is mostly useful imo because it can be done in a reversible way, so it is good for lossless

But that only makes sense if it's done in 444

one thing I don't understand is why "TV range" YUV is still a thing

I have three hypotheses:

1) inertia - not bothering to change existing stuff that does that from the analog days

2) some hw implementations can perhaps get away with fewer precision bits or simpler clamping logic if they can assume there's a bit of headroom and footroom in the ranges?

3) smaller ranges mean less information/entropy in the source signal, so you get "better compression" if you measure it in the naive way (since you just have a slightly less noisy / less precise signal to compress in the first place)

TV range is like encoding 8-bit sRGB images in a wider gamut space like P3 but still in 8-bit

You will get smaller files

And more banding

from what I recall, the ITU allows TV range for HLG (makes sense since HLG is made for TV production) but strongly discourages it for PQ

I wonder if I can find the reference again

No, FLIF is completely super seeded by JXL lossless.

There might be the occasional image where flif still does better, or has a better encode time vs density, but as a codec, I think it's superseded and the only thing left to do with it is to convert any .flif file that people might have to (png and then to) jxl.

Let me see

avif.io Domain was 40€ for first year, then 5€ per month. Same for Jpegxl

Basically the only reason there's ads on the blog post, so it becomes even. Besides the investment of time for content and marketing, plus the initial money on both converter functionalities. but since it's more of a hobby, that's fine. 🙂

It's a great idea to integrate libjxl to ffmpeg. Someone has try but i don't think that it's in FFMPEG https://ffaux-bg.ffmpeg.org/project/ffmpeg/patch/20200412113100.80562-1-varun.gupta140@gmail.com/

security problems

absolutely, just put a stub there

I think if you don't someone malicious could snap it up

ffmpeg is a multimedia swiss army knife

included with it it are a number of encoders and decoders and muxers and demuxers

it has support for many images either via internal or library support as well

It would not be *too* difficult to try to add a jpegxl decoder to libavcodec and libavformat, via libjxl

but jpeg lossless transcoding would be impossible

and the encoder would be more difficult as well

But, patch is old and seems to be abandoned

I think it's better to ask for it after browser support.

Because they will say some lame reasons for not adding support.

Jxl is new. So they have more reasons.

Yes, I think that FFMPEG use libwebp (to encode en decode) for Webp. So i think it's possible to make this with libjxl. That's give more visibility to JPEG-XL. I try to send a mail to the guy who begin the patch but a have no reponse

it would be straightforward to write it yourself wiw

I can only code "Hello world" 🙂

haha

I saw a project in 2020 with the ​Google Summer of Code 2020 but nothing in the result (only for FLIF Encoder & Decoder Project) https://trac.ffmpeg.org/wiki/SponsoringPrograms/GSoC/2020#JPEGXLDecoder

Does anyone know of any good breakdowns of how JPEG works?

This one is quite nice imo: https://parametric.press/issue-01/unraveling-the-jpeg/

Thanks!

That should probably help. It should arrive in repositories after the next FLIF update, but those that compiled FLIF themselves are screwed if they don't visit flif.info on their own... https://github.com/FLIF-hub/FLIF/pull/563

Oh, by the way <@!794205442175402004>: FLIF's README.md still points to GitLab and not GitHub, where all issues and PRs are. https://github.com/FLIF-hub/FLIF/pull/561

How does exactly "Hornuss" work? And where did that name come from?

How it works: <@179701849576833024> knows better, but the idea is that it's a transform to be used when DCT is not a good idea

Originally we called it the "Identity" transform

But an identity transform would just be a no-op, encoding raw pixel values

That's not quite what that transform does

So at some point one of the national bodies of ISO was reviewing the spec and said "you shouldn't call this an identity transform if it does something that is not the identity"

We had to come up with a new name

"Hornuss" is basically a random word that is not the word "Identity"

https://en.wikipedia.org/wiki/Hornussen

It's the name of the puck in some obscure sport I never heard about

ugh, I really have been neglecting the flif repo, sorry about that

https://github.com/godotengine/godot/issues/31568 does someone feel like opening an issue like that but for jxl?

Ehhh, I don't exactly remember - but it's roughly a "subtract the average from every pixel" kind of transform

I'm not sure who would be the relavent contact, but the same should probably be done with the link on jpeg.org

well strictly speaking, the gitlab repo is the one owned by JPEG

Ah, fair.

maybe we should add a self-reference to the github repo in the main README though, so you can easily find it from the gitlab repo

(since the gitlab repo mirrors the github one)

That sounds like a good idea. Discoverability of the Github from the Gitlab is pretty poor.

Also <https://github.com/cloudinary/fuif/pull/10>

Could do that, though it might be already too late

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=903600

Today I learned that flif was actually in Debian unstable at some point (I didn't know that)

Today I also learned that it was removed from Debian unstable 3 years ago 🙂

because of mostly silly things, it seems ("security issues" being feeding a broken/malicious input file to the _encoder_ which causes it to crash instead of gracefully printing an error)

the encoder crashing *is* a security issue, not everyone only uses decoders on user input 😉

Yeah it is

But the binary is just an example application

If you want to do it properly, you use the library, not the example app

But ok, the distro was shipping the example app so there's that

Any thoughts about WebP v2?

I discovered it with squoosh.app

I found it could compete a bit with jxl...

when it comes to natural photos, it looked more realistic than avif when i looked, but still less than jxl

btw, can it losslessly represent a WebP v1?

It has lossless compression, but I don't think it can use the WebP v1 codec to compress.

Ok, so the lifecycle of a typical image on the web will be lossy compressed JPEG, which was then recompressed to WebP, then recompressed again to AVIF and lastly back to WebP2...

WebP2 has nothing to do with WebP1 except the name and some of the people involved, afaik

I don't think it's still in active development atm, it's kind of unclear to me what its advantage would be over avif or jxl