JPEG XL

I prefer find’s ability to recursively find

Although I could probably pipe find into parallel

yep

Parallel’s —bar is kind of enticing for me tbh

https://cdn.discordapp.com/emojis/806636752952754186.gif?size=48&quality=lossless&name=CatYes

Fun! parallels brew formula on the Mac seems busted though.

Works fine for me

That's weird. Just tried again and it worked.

Internet issues?

[⠀](https://media.discordapp.net/stickers/1052898883971338291.png?size=160&name=Skill+Issue)

<:KekDog:805390049033191445>

cjxl -g 3 -e 10 -I 100 --patches 1 -q 100 <:JXL:805850130203934781> loseless bruteforce<:JPEG_XL:805860709039865937>

`q` hould be rational because medium range cameras are 10+ Mpixels...

my position on this should be obvious

I don't understand what that means

I don't understand what any of this means

try xl converter https://github.com/JacobDev1/xl-converter/releases

Your ffmpeg example looks like DOS syntax so you probably aren't using a bash shell

Also the script doesn't check if the output filename already exists before clobbering it (minor issue in this case) or if the input filename is a symlink or an actual file. Or if the output result is less bytes than the input.

Would be cool to keep a running total of how much space was saved too, for extra credit

All of this will be on the test

Ooo, I was hoping something like this existed. Otherwise I would have gone janky bash script

I love my janky bash scripts

I do too.

:)

What about build scripts that decode test cases and inject them into ssh keys?

My favourite ❤️

I want to losslessly recompress jpegs but strip any useless byte (including stripping the jbrd box), how can I remove all meta data ? I guess this command is not enough : `cjxl --quiet --lossless_jpeg=1 --allow_jpeg_reconstruction=0`

did I stripped everything if I add `-x strip=exif -x strip=xmp -x strip=jumbf` ?

I think it may be enough to specify --container 0?

No, wrong, --container 0 just leaves it up to the encoder

Some people will be very upset if you say metadata is useless 🙂

😅 let say, I just want undocumented and untraceable arrays of pixels

For photographers it's essential

With cjxl, is quality 80 mapped to distance = 2?

1.9 according to a table made by someone

For decreasing each 1 from q99 to q30, it increases each 0.9 from d0.19 to d6.4

Thanks, although I don't get the math, I don't need a precise answer anyway 🙂

Then 1.9 is the answer

It'll say what distance when you run the command

Do you mean it decreases by 0.09 per 1 ?

Oops that's 0.09

Below q30 the curve of distance becomes sharper

`-d 0 -e 11 --allow_expert_options`

haters need to get a faster computer

-e 12

Although, I did beat e11 last night just by using e10 with - P 0, so it seems like there's a few options missed since Monad's PR

`-e 13` time

v0.8 e10 beats Monad's e12 sometimes... <:kekw:808717074305122316>

Found it, zig is too powerful for e12 😂 https://discord.com/channels/794206087879852103/794206087879852106/1292927543904571432

It does try P0, but with additional configuration. Old e11 tried P0 and P15; new tries P0, P4, P15 or P0, P6, P14, P15 depending on the image.

Let's gooo

I think in this case it was `-P 0 -I 0`

I wonder why `cjpegli` doesn't have `--effort` option. JPEG encoding is so fast, there still must be room for improvement.

`cjpegli` already has good heuristics if you want better compression (lossless quality), check IJG's jpegtran, jpegultrascan and/or ect

`jpegultrascan` does further compress JPEGs encoded by `cjpegli`, I've tried that. However, I though adaptive quantization could be improved as well with more effort.

<#1301682361502531594>

Not necessarily.

How could we know if we don't try?

It depends on what you try. Time doesn't have inherent value if it's not used correctly.

Spending extra time on something doesn't always mean better results.

That's why jpegli is much faster than mozjpeg and much better looking at the same time. It uses time more effectively.

It doesn't necessarily need more time, and spending extra time is not necessarily productive

For example, performing more extensive search for the optimal quantization to avoid the unwanted smoothing.

I've also noticed jpegli seems to smooth chroma too much, resulting in a similar effect like chroma subsampling. Even at very low distances under 0.5

Hello everyone! I think I found something interesting. Could a libjxl developer please take a look at this issue? https://github.com/libjxl/libjxl/issues/3746#issuecomment-2567331492 This majorly reduces RAM usage and increases encoding speed for cjxl users. I'm not sure if this is officially supported for non-PPM inputs, though. It does seem to work great so far.

It should be relatively easy to implement on a row group basis for png input

Streaming encoding is currently only implemented for PPM inputs.

But more examples of how to use JxlChunkedFrameInputSource would be welcome.

The documentation could be wrong/outdated, I've also pointed it out here: https://github.com/libjxl/libjxl/issues/3517#issuecomment-2513479999

It seems like you can force streaming for formats other than just PPM, I've gotten it to work on lossless jxl and jpeg (`-j 0`) inputs. I don't know if this behavior is intentional or not.

Adding `--streaming_input` enables streaming encoding in all cases I tested (yes, PNG input, not PPM). The pixel output seems to be the same as if PPM proxy was used. My question is, is this ready to be used? Is this a result of an oversight or an undocumented feature? I could really use an opinion from a libjxl dev. I want to include this in XL Converter. Encoding an 8 k image (PNG, `-e 9`, VarDCT) requires 8 GB of RAM. With `--streaming_input`, that PNG now requires 16 times less RAM and processes multiple times faster.

I think this is the commit that introduced this behavior: https://github.com/libjxl/libjxl/commit/0e3e08f89305e821e8467df7e058fc7889fc913e

After reviewing the source code, the `--streaming_input` argument is most likely unintentionally triggering `JXL_ENC_FRAME_SETTING_BUFFERING` being appended to the `params.options` vector. ```c if (args.streaming_input) { params.options.emplace_back(JXL_ENC_FRAME_SETTING_BUFFERING, static_cast<int64_t>(3), 0); } ``` The speed / RAM usage improvements are incredible, but I'm unfamiliar with the code base, so I can't tell if this is safe to use all the time.

As far as I know that should be fine, the one thing that is not supported is not decoding the PNG into ram

Hi <@179701849576833024> thanks for the answer. I tried doing something like this. Do you think this is a sound idea? ```c diff --git a/tools/cjxl_main.cc b/tools/cjxl_main.cc index 4b01e18..7031a10 100644 --- a/tools/cjxl_main.cc +++ b/tools/cjxl_main.cc @@ -1163,10 +1163,7 @@ int main(int argc, char** argv) { params.runner = JxlThreadParallelRunner; params.runner_opaque = runner.get(); - if (args.streaming_input) { - params.options.emplace_back(JXL_ENC_FRAME_SETTING_BUFFERING, - static_cast<int64_t>(3), 0); - } + params.options.emplace_back(JXL_ENC_FRAME_SETTING_BUFFERING, static_cast<int64_t>(3), 0); jpegxl::tools::SpeedStats stats; jpegxl::tools::JxlOutputProcessor output_processor; ```

Answering that would require me remembering what the code does 🤣

I'll try to remember to look into it next week when we're back from the holidays

Hah, yes the code base is certainly extensive. If you do work on it, here is the associated libjxl issue: https://github.com/libjxl/libjxl/issues/3746

The header documentation says:``` /** Control what kind of buffering is used, when using chunked image frames. * -1 = default (let the encoder decide) * 0 = buffers everything, basically the same as non-streamed code path (mainly for testing) * 1 = buffers everything for images that are smaller than 2048 x 2048, and * uses streaming input and output for larger images * 2 = uses streaming input and output for all images that are larger than * one group, i.e. 256 x 256 pixels by default * 3 = currently same as 2 * * When using streaming input and output the encoder minimizes memory usage at * the cost of compression density. Also note that images produced with * streaming mode might not be progressively decodeable. */ JXL_ENC_FRAME_SETTING_BUFFERING = 34,```

IIRC libjxl uses the chunked API internally when you call `JxlEncoderAddImageFrame`.

I went digging through that code a while back while trying to figure out what the intended usage pattern is for JxlChunkedFrameInputSource.

Huh, oh yeah this has been known for a few month now. The difference is between streamed input and chunked encode. Streamed input implicitly enables chunked encoding, even if the input format doesn't support streaming. Thereby disabling any full frame tools that cause most of the memory consumption (effort 8 and 9 used to enable it, but I thought it was moved to 10 for images above 2048 width/height)

It's strange that when I increase the effort, the filesize is bigger (I guess it's because my images aren't "perceptual"). I thought I'd share though ! It's quite consistent. All inputs are 300MB 16bits TIFF files. Rows are: filename, distance, effort, output filesize (I know you'll say my distance makes no sense, but I'm not sure it should behave this way anyway)

I just tested with intensity_target=60000 and distance=0.5, the behavior stays the same, image quality is approx the same as before

for lossy, higher effort means consistent quality throughout image. it might not produce smaller file

so the encoder used more bits to preserve details

Interesting !

Okay, I see, then I might want to use a low effort then! Thank you 🙂

or maybe find good balance between distance and effort? I'm not an expert of encoding params though

Higher efforts are more accurate to the quality you set, but also *try* to be more dense too. So either use a low effort with less consistent results, or a higher effort with a lower quality

<:Agree:805404376658608138> ideally, you should adjust effort to moderate compute, distance to moderate quality. but it is possible some intersection of digital image content and encoding behavior reveals pathological cases.

I see... I'd like to have a consistent filesize but that might be hard to achieve

All my files will be 300MB in 16bits TIFF format, and I want them to be 20~30MB, so I'm trying to find parameters that do just that but of course the images are different and some compress to 15MB and some to 60MB for the same parameters :/

That's due to the perceptual nature of libjxl. To reach the same target quality, some files need more or less bits. There was a target file size option but that was depreciated quite sometime ago. Why do you want to target file size specifically? > Intensity_target=60000 and distance=0.5 I have a feeling your images aren't perceptual at all. Is it raw data or something like that?

Yes it is raw camera output, so no gamma, only linear, only processing is debayer. The goal is to transfer this data for preview/processing. The filesize must be relatively consistent as the transfer time is quite big, so I need to be in control of that to a degree

I assume lossless doesn't get close?

Input is 192.3MB PNG Using v0.11.1: ``` cjxl -e X -d 0 PNGs/inp.PNG out_lossless_X.jxl ``` With X the effort value ``` | Effort | Size (MB) | MP/s | |--------|-----------|----------| | 1 | 134.7 | 354.607 | | 2 | 130.8 | 72.732 | | 3 | 128.6 | 41.254 | | 4 | 128.4 | 19.268 | | 5 | 128.4 | 10.978 | | 6 | 128.7 | 6.916 | | 7 | 130.3 | 3.209 | | 8 | 131.6 | 1.097 | | 9 | 131.2 | 0.412 | | 10 | 134.7 | 0.050 | ``` And with the settings for lossy described above I get around 25MB

It's interesting that e=4 yields the smallest size So it's definitely not near enough :/

old problem, I make report about it

but doesn't remember solved or not

can you share file?

the PNG?

yes

https://we.tl/t-4Y8dzRlYYG

hmmm

quality should also be assessed

otherwise it's pointless

I have encountered preset 10 (fastest svt-av1 preset) producing the smallest videos given the same quantizer

what

oh this is lossless

😁

it's lossless...

this message confused me

my bad 😁

And my JXL lossless experience is 8 being the sweet spot

i use 9 for big and 10 for small/gray images

this is so interesting though. Effort 1 and 10 give the same size?

source image is 48 bit

i think that because jxl have problem with it

Hmm, that's probably why

webp 24 compress to 3.2 mb

For bayer DNG files effort 2 is usually best, so it really depends on the image content and bitdepth. I'll do a bit of testing too and see if there's a specific flag that's going wrong

i test that image, but the error did not occur

I looked at the image and well there's nothing left in terms of, well anything really

Yeah... Don't do that

Well, that's interesting ```cjxl M31-1280.tif.PNG M31-1280.jxl -d 0 -e 2 JPEG XL encoder v0.12.0 03e432d3 [_AVX2_,SSE4,SSE2] {Clang 20.1.8} Encoding [Modular, lossless, effort: 2] Compressed to 130839.8 kB (20.936 bpp). 7779 x 6427, 26.780 MP/s, 16 threads.```

I guess we did something right for v0.12, because now effort 2 beats all your results. 124MB

wdym

hmmm I'm not so sure, what I reported in the table is the size given by cjxl You have 130.89 which is what I got with e2

Ah right, MiB vs Mb

When you stretch it, it should look more like this (35MB jxl)

I appreciate you both taking the time and having the curiosity to try out my input data 🙂

Also... How does this file look?

I was thinking, would it work to compress a few 256x256 patches and quickly converge to a good distance parameter?

I have checked against the reference:

It's quite good, but it creates some "holes" some quite dark pixels and the blocks are very visible, with colors behaving a little strange

But what you have is consistent with my findings so far, for these noisy images, 25MB is just about enough to remove most of the egregious blockyness

What parameters did you use?

`-d 0.01 -m 1` It uses Modular encoding, which is normally for lossless, in a lossy way. It should perform better than VarDCT at such low distances, but it runs at a speed and memory cost similar to effort 10, so might not be worth it

Interesting

the result is indeed quite impressive for this filesize

I made a 7.2 MB jxl with -e3 -d 0.04

I'll compare

Oh wow ! the difference is huge, yours is much much better When I said the blocks are very visible, well, no they're quite discreet compared to that one

I think I know how it could be even more efficient, possibly for lossless too, but we'd need to plug some new wires into libjxl. Right now `-R` is a bool to enable the Squeeze transform, which recursively halves the resolution and stores the difference. If we changed it to be the levels of halving instead, then we could do something like `-R 2` to hopefully seperate the noise from the real image

What would that entail ?

effort 3 is functionally the same as a JPEG, so if you can I'd aim for effort 5 <https://github.com/libjxl/libjxl/blob/main/doc/encode_effort.md>

Ah that's very interesting

Work on our part mostly, it'd be an update to the library so unfortunately we can't try it yet

I mean what would be the expected result?

Ah, in an ideal world it would only compress the noise, and then we could use photon noise to add it back in

So, if I change the distance so that -e 3 and -e 5 give the same filesize, the -e 5 should be significantly better?

Hopefully 😅

I have to see if the added compute time is worth it

It is better indeed with equally sized outputs! But absolutely no match for the modular one you showed

Really, I should do more testing and just enable modular automatically at such low distances, but there's always a risk that something explodes

Especially since it goes singlethreaded

Something happens when I use my encoder vs cjxl... Maybe it's because of the intensity target Edit: yes it's the intensity target that makes the quality much higher for the same filesize

Makes sense. Intensity target effectively brightens the image internally, so it sees the result you want rather than the raw data, and can separate the image from the noise more effectively

Okay, e3 35MB vs e5 26MB with same distance (0.5 intensity target 60000) and the quality is really really close

I guess I'll go with e5, do a few passes of encoding on 256x256 patches to find the correct distance and aim for 25MB 🙂

Now that I see it, e5 is more than twice the time as e3...

Then why not stretch it to begin with? Since it's more perceptual libjxl will preserve more details but I guess that might not fix the consistent file size problem...

Is lossless at least somewhat consistent?

Yes that's a good suggestion, I have tried it before. Buf if I stretch it before export, I lose control over the stretching after export (which is the whole point), or I need to unstretch it. All of that costs time...

And in all cases, it won't change the issue that I need to find the relevant distance for each file so that it compresses to ~25MB

In the end, I also needed to find the correct stretch so that it looks good for jxl, but then it was hard to manage the quality of the dark regions, so this approach doesn't work that well for me

You basically want near-lossless, but lossy modular is veeeery slow

yes 🙂 near lossless, and as much "non lossless" as needed to get x10-x12 compression

I think that using jpeg2000 might be better here, as opj_compress can set the compression ratio, and you can just set it to 12, and it will hit 25MB perfectly.

I see... that was our initial choice <:AngryCry:805396146322145301>

the compression kind of sucks, though

Hi! I need some clarification on how the encoding parameters work. My understanding was that modular encoding meant lossless encoding and that distance was no longer relevant, but I observe (using cjxl) that `-m 1` is lossy while `-d 0` is lossless. So how does modular/varDCT and distance actually play together? I also read in the libjxl API docs that setting a distance of 0 will create a "mathematically lossless" image but that it is not enough to produce a "true lossless" image. Could someone explain whatever is going on there, since I observe lossless encoding using that in cjxl?

Math lossless is 1.0 ssim, visual - human eye lossless. d 0 is true lossless

the documentation says that when using the API, setting the distance to 0 is not the only requirement for lossless, but that doesn’t apply to `cjxl`, which does whatever is needed for lossless if it detects that it’s given a distance of 0

it doesn’t *just* set the distance

https://github.com/libjxl/libjxl/blob/1bb7f02535a006afbc753282f37a7ec348afe3d1/lib/extras/enc/jxl.cc#L249 https://github.com/libjxl/libjxl/blob/1bb7f02535a006afbc753282f37a7ec348afe3d1/lib/extras/enc/jxl.cc#L289-L293

Lossless has to be modular, but modular doesn't have to be lossless

Alright, thanks!

mathematically lossless means not doing the XYB color transform and storing the RGB values exactly. VarDCT mode cannot store values exactly (it's like the old JPEG in that regard) and in libjxl, it always uses the XYB color transform. Modular mode can store values exactly but it also has various mechanisms to do lossy — generally not as effective as VarDCT but sometimes relatively competitive in compression (though slower). When you do `-m 1` with a distance > 0 (like the default `-d 1`) it will use XYB and do something lossy. When you set the distance to 0, cjxl will both disable XYB and use modular in a lossless way (or if the input is a JPEG, it will apply lossless recompression of the JPEG data, which can be done since VarDCT can represent JPEG's quantized DCT coefficients directly).

Thanks for the breakdown! Now I understand better how the options work.