JPEG XL

wait, does this impact whether JXL can be adopted as a new format for DICOM?

at least that was at that time

dicom sounded excited about jxl in last week's podcast (s7 e2, first mention i recall around 29min in)

I read that as an effort parameter at first

“they’re excited about effort 2 specifically? odd, but why not”

the SIIMcast link Jon posted in <#822105409312653333> ?

hm, that's e3

It is useful for jxl-in-dng raws. All modern cameras have 14 bpp precision, CMOS based DB's and I believe some latest "small" cameras have 16 bpp. And at least in case of DB's these 2 extra bits are not only noise: tried zeroing 14 MSB, and the leftover had some image residuals.

Is there a way to convert (Linear) DNGs to JXL? Like the ones an iPhone output?

Like listlessly #

Losslessly

Adobe dng converter can do it, I believe, if you ask it correctly

Does it keep the 12bit data?

I’m asking because the iphones’s DNG uses Lossless jpeg

And I’m thinking lossless JXL would be higher compression efficiency

It does, however I did not get significant size reduction. Don't know why. But I tested on 16 bit raw, maybe it is the reason

My phone makes 20MP RAWs, uncompressed they're 24MB, default Lossless Jpeg is around 7MB, JXL was around 6MB, although at default e7 (I think, the converter is horrid to use)

Page 21 https://helpx.adobe.com/content/dam/help/en/photoshop/pdf/DNG_Spec_1_7_1_0.pdf

I got ~7 MB difference on 100 MP raws, like 122 MB vs 115

Problem I don’t have Photoshop

It is not about photoshop. PS currently does not support jxl or jxl-dng directly, only in ACR

Does cameras cost money?

Me neither, it's free https://helpx.adobe.com/uk/camera-raw/using/adobe-dng-converter.html

I've never used an Adobe product, apart from this

Is it being considered to reduce the default effort only in lossless mode to level 3, to be closer in performance with vardct effort 7 speed?

There isn't that big a difference in bitrate compared to the huge speed gain you get, when using lossless mode

I think it would give people a better first impression of the format and tools

I remember promoting effort 5 for lossless, since that's when patches is enabled to match lossy's e7. But I was unaware how inaccurate and slow patches is anyway. And I think it's disabled for the default streaming mode now anyway

After effort 3, the speed of lossless mode drops tenfold for marginal benefit

A faster default setting for lossless mode would be very useful especially for first impressions, which is important at this early stage

At least until a genius like Jyrki ports his lossless webp encoder to output jpegxl

But the speed is only one half of impression

In libjxl 0.10, e7 lossless will be significantly faster than in libjxl 0.9, at least for larger images

The speed/bitrate is at a very good balance at e3 in my experience (in lossless mode only)

By comparison e7 is much much slower in lossless mode than in lossy mode and this goes against what I would assume would be most people's expectations when getting a first impression.

e3 already easily beats other lossless formats in a very quick encode speed. The only thing competitive is lossless webp

For photos, e3 is very good

The speed of e7 lossless will be in the same ballpark as lossy. But maybe it would make sense to make e6 the default for both...

Hi guys, anyone know how to compile libjxl to wasm? I've tried this: ``` emcmake cmake .. \ -D CMAKE_BUILD_TYPE=Release \ -D BUILD_TESTING=OFF \ -D JPEGXL_ENABLE_DEVTOOLS=OFF \ -D JPEGXL_ENABLE_TOOLS=OFF \ -D JPEGXL_ENABLE_DOXYGEN=OFF \ -D JPEGXL_ENABLE_MANPAGES=OFF \ -D JPEGXL_ENABLE_BENCHMARK=OFF \ -D JPEGXL_ENABLE_EXAMPLES=OFF \ -D BROTLI_BUNDLED_MODE=ON make ``` No errors, but only some `.a` static libs, `brotli.wasm` and `hwy_list_targets.wasm` are generated. I'm not quite familar with cmake system. So anyone know how to solve this?

<@625718249262284800> did you followed : https://github.com/libjxl/libjxl/blob/main/doc/building_wasm.md ? my compile script looks very much like it :

```sh source ${JXL_root_DIR}/repo/emsdk/emsdk_env.sh pushd $JXL_root_DIR/build build_dir=$JXL_root_DIR/build/wasm32 source_dir=$JXL_root_DIR/repo/libjxl if [[ -d ${build_dir} ]] then rm -rf ${build_dir} fi pushd ${source_dir} BUILD_TARGET=wasm32 BUILD_DIR=$build_dir SKIP_TEST=1 ENABLE_WASM_SIMD=1 emconfigure ./ci.sh release popd popd ```

That's unexpected. They are like 1000x difference right now

Thank you, I'll try this.

I tried this, it said: `CMake Error: CMake was unable to find a build program corresponding to "Ninja". CMAKE_MAKE_PROGRAM is not set. You probably need to select a different build tool.`

if you use ubuntu : `sudo apt install ninja-build`

thank you, i'm not using linux, but i managed to install ninja

oki, what do you use ?

homebrew on macos

It's weird. With `-G Ninja`, it will cause `make: *** No targets specified and no makefile found. Stop.` to run `make`.

And I cannot see any `wasm` files generated

if using `-G Ninja`, you should run `ninja` instead of `make`

it’s its own command

which uses a `build.ninja` instead of a `Makefile`

indeed and ./ci.sh does it

I cannot run ci.sh, I don't have a lot of dependencies installed, and ci.sh cannot disable these features. So I choose to manually disable the flags.

Thank you. `ninja` works.

after `ninja`, there are still only `.a` files in build folder. I can merge they to `wasm` file by using `emcc`, but it will return JXL_ENC_ERROR when calling some functions like setting basic info.

anyway, thank you guys. I'll try to figure this out myself.

<@1051137277813870592> btw, how did you use the libjxl wasm? Did you write another c++ file and compile to another wasm file? Or call the libjxl wasm from js directly(i don't think it's possible to create a c struct basic info in javascript)?

ha ! I used only the demo... (https://github.com/libjxl/libjxl/blob/main/tools/wasm_demo/README.md) I planned to be more fluent in the integration of wasm into my own code but I'm not. I expected to write the glue code in javascript, I think it's what most people do

if I remember well, the integration of libjxl.wasm into a web page is not so easy, you want to setup workers to do the job in backends, and perhaps a caching system to save your converted file... I wanted to use the demo as a way to learn a bit how I could adapt the wasm to my needs and website but I couldn't spend more time on it yet

Thanks for the info you provided. I'll try these.

<@625718249262284800> what's your final goal ? as for me I wanted to build a simple gallery webapp. If you learn something interesting don't hesitate to document your journey 😄

encode image to jxl on browser

I manage to set basic info now. But another issue occurred during processing: ``` libjxl.js:1 Uncaught RuntimeError: Aborted(OOM). Build with -sASSERTIONS for more info. at abort (libjxl.js:1:5258) at abortOnCannotGrowMemory (libjxl.js:1:57926) at _emscripten_resize_heap (libjxl.js:1:58029) at libjxl.wasm:0x1eed at libjxl.wasm:0x2623 at libjxl.wasm:0x19ef at libjxl.wasm:0x1d7251 ``` during `JxlEncoderProcessOutput`

My code is just trying to encode a 16x16 size RGB data

The code runs no problem for the desktop target.

you could try to find the guy who made squoosh, have a look at this post https://discord.com/channels/794206087879852103/803645746661425173/1209853020259618848

Oh, i figured out the issue. I need to add `-sALLOW_MEMORY_GROWTH` to emcc to enable memory growth during running.

If you can, I'd try building with 0.10.0, for the same reason I mentioned in <#803645746661425173>

Well that was faster than expected

😎

That's a good summary of v0.10

Faster than expected

nobody expects the v0.10

should we be afraid of v0.10?

Shame we don't have names for versions

We already have weird names for the efforts, I think that's enough

what are they

is it bad or smth?

no, just unexpected so fast

is it fine if i stay on `ab47708d`

i looked through the Commit-logs and nothing pops out

Probably fine if using the release causes issues for you

just when reporting any bugs/problems is easier with fixed releases, ppl responding don't have to search for the particular commit

are all the aborts removed yet

still not, iirc

is -e 9 from previous versions now equal to -e 10? I'd like to use the global compression tools

no, -e 9 is still there

previous e10 is now e11, and there's now a new e10

yes, but -e 9 now uses streaming and does not use patches as far as I can tell. so it does less than before?

hm, then yea use e10 if you want nonstreaming with patches

thank you

TectonicPlate, Glacier, Tortoise, Kitten, Squirrel, Wombat, Hare, Cheetah, Falcon, Thunder, Lightning

Is tectonic plate e11

perhaps, I couldn't find exactly the correspondance as tectonic has a value of -1 as shown in `lib/jxl/common.h`

the associated comment says : ```cpp // Try multiple combinations of Glacier flags for modular mode. Otherwise // like kGlacier. ```

yup

encoder is being fuzzed though, so we should not be hitting them, at least

Hmm, did a quick test, and 0.10 -e 10 is slightly worse than 0.9 -e 9 when compressing jpg. Really slightly, but a bit unexpected. And funny enough 0.10 -e 9 gave slightly better result than 0.10 -e 10 with that jpg, but 0.9 -e 9 is still better

Also, just curious: JPEG XL encoder v0.9.2 41b8cda [AVX2,SSE4,SSSE3,SSE2] JPEG XL encoder v0.10.0 19bcd82 [AVX2,SSE2] Are SSE4 and SSSE3 disabled intentionally?

no

did you change compilers perhaps?

that happens also on my builds, using clang on arch linux

> encoder API: implemented new api functions for streaming encoding. does this mean cjxl v0.10.0 is not yet using streaming encoding, just that it is available? or is it already being used?

I just downloaded win build from github

huh, also for v0.9.2?

No, JPEG XL encoder v0.9.2 41b8cda [AVX2,SSE4,SSSE3,SSE2]

I don't understand this answer...

Well, then I did not get the question, sorry

did you download v0.9.2 also on github?

Ah, yes, sorry

... interesting

it might be the highway update

maybe SSE2 is there because it's static, and AVX2 because it's dynamically selected? seems a bit awkward

I suspect SSE4 is not built

maybe I can test the binary in machine with SSE4 but not with AVX2

``` ~/libjxl-v0.10.0 ❯ ./djxl -V djxl v0.10.0 19bcd82 [SSE4,SSE2] Copyright (c) the JPEG XL Project ```

so it seems like it just shows only the selected one

ah

The same with lossless encoding: 0.10 e 10 is little worse than 0.9 e 9 😦

It's already being used, and streaming is now default for both cjxl and libjxl until effort 10

weird, i'm getting the same [AVX2] vs [AVX2,SSE4,SSE2] thing

i built libjxl v0.10.0 exactly the same as v0.9.2

ah, if this is intentional, it should probably be communicated somewhere

Part of me wonders why those extensions are enabled/disabled, since you'd assume SSE3 is better than SSE2, although I could be misremembering what ones my chip has and it's a naive assumption

I'll check when I'm home

I think `uint32_t` here: <https://github.com/libjxl/libjxl/blob/v0.10.0/tools/codec_config.cc#L42> should be `int64_t`, see e.g. commit: <https://github.com/google/highway/commit/379457ceb1041d2e38c35af02b4ea9990143a070>. Not sure if that fixes the `[AVX2,SSE4,SSSE3,SSE2]` versus `[AVX2,SSE2]` thing.

(let me verify this and open a PR if required)

That doesn't fix it, but I think `SSSE3` should never be listed there after commit <https://github.com/libjxl/libjxl/commit/beb3dae3fb7219525c4278e9e775d681c0b381f7>. Not sure about `SSE4` though.

Indeed, I suspect that this may also be the cause. See for example: ```console $ djxl -V djxl v0.10.0 b38469de [AVX3_ZEN4,AVX3,AVX2,SSE4,SSE2] Copyright (c) the JPEG XL Project $ vips --targets builtin targets: AVX3_SPR AVX3_ZEN4 AVX3 AVX2 SSE4 SSE2 supported targets: AVX3_ZEN4 AVX3_DL AVX3 AVX2 SSE4 SSSE3 SSE2 ``` (libjxl was built with `-DJPEGXL_ENABLE_AVX512{,_ZEN4,_SPR}=TRUE`) So, the targets listed by `djxl -V` are the ones that are built and supported by the current CPU. So, if it isn't built (or not supported), it is not listed. `vips --targets` would display both, so the ones supported by the current CPU and the ones that are builtin.

SSSE3 I can understand

SSE4 is about as common and just better

I don't understand SSE4 disappearing though

I'm getting SSE4 on Intel Celeron so it's actually in prebuilt binary

uh well I'm getting AVX2 and SSE4 on another CPU with my build ``` ❯ djxl -V djxl v0.10.0 b38469de [AVX2,SSE4,SSE2] Copyright (c) the JPEG XL Project ```

I'll try prebuilt one

It might be compiler dependent

yeah I'm getting SSE4, this is the statically linked one ``` /tmp/libjxl-v0.10.0/tools ❯ ./djxl -V djxl v0.10.0 19bcd82 [AVX2,SSE4,SSE2] Copyright (c) the JPEG XL Project ```

Looks like it's a MSVC thing: <https://github.com/libjxl/libjxl/actions/runs/7993393111/job/21829120447#step:10:4008>

Found it: https://github.com/google/highway/commit/30a85e23fc820dacbeff72e5a666712c9982b305#diff-bfe5635b4f0b39e741ed4ce8f585674994a3434568c7caa471c0e7be9a7938c8R604

So, only `AVX2` and `SSE2` are attainable when building with MSVC.

Does vcpkg support building with `clang-cl`? AFAIK, that would cause `HWY_COMPILER_MSVC` to become `0`.

On the plus side, it means 0.10.0 could actually be even faster ;P

mhhh, I'd much rather still have SSE4 in MSVC builds

not if you support AVX2, no 😛

Ahh, I've not spent a weekend reading up on extensions yet, only half a dozen image formats and a few video codecs so far xD

are faster builds the only reason?

it seems to be? at least if one trusts the commit message

I used to get only AVX2 since I can remember on my custom builds on linux

https://github.com/libjxl/libjxl/issues/235#issue-931512361 2021

this is on arch linux btw, not sure why sse4 and sse2 would just disappear when i haven't changed any compiler options

What does `-E` do? I thought it described how many input image channels will be looked at at a time when encoding. But when `-E` is larger than the number of input image channels, it still increases encoding time. Should I open a bug report on GitHub?

Yeah maybe, it could be doing something silly like recording useless all-zeroes properties...

it probably does exactly that

the output file does not change if that matters

Makes sense to cap it to the number of prev channels that actually exist...

how many threads is libjxl parallelized in ffmpeg on -e 1 ?

Actually, when would ffmpeg get updated to libjxl 0.10?

someday... (messages 2 weeks ago)

<:PepeOK:805388754545934396> I'm sure they know

defaults to cpu count

can be updated in place, since there was no ABI break

on arch at least, ffmpeg links to a versioned .so name so it's a little problematic

but it just needs to be rebuilt

there's nothing that ffmpeg has to change

oh yeah, it just annoys me a little because I build libjxl from git as a system package and it breaks mpv and ffmpeg. I guess I'll start symlinking to the previous .so version as a workaround. I don't want to rebuild ffmpeg and mpv

yeah, libjxl i have in it's own chroot thingy, i don't globally run git since it breaks a lot of things too like kde stuff too

I guess since I'm already running a custom PKGBUILD, I might as well just modify it to not replace the original libjxl I definitely prefer to have it as a system package

this won't work

the reason the versions are marked as incompatible is libjxl has an ABI break between 0.8 and 0.9

so if you just symlink it you'll get incorrect behavior

arch has 0.9.2 now in repos. it does work with 0.10.0, I just tested it. at least, does not throw errors

jxl lossless jpeg grayscale reports num_color_channels 3 is this intended? because now I have to dump the gray icc

did you check the original jpeg?

djxl makes it the grayscale jpeg where libjpeg says it's jpeg_color_space JCS_GRAYSCALE

libjxl cms also skips the gray icc in the jpeg

Maybe update the 0.10 release to include this PR too? I know there's a system to update release packages already https://github.com/libjxl/libjxl/pull/3322

fair point

there's a few fixes we probably want to have in 0.10.1

When will 0.10.1 be released then? Is there an approximate date?

hopefully this week

doesn't effort 0-9 make more sense from a programmer's perspective than 1-10?

no

0 is generally thought to be "uncompressed"

I personally think both are not very good choices for that reason 😉

Why should 0 and 100 be confusing? they are just abstract names with a clear hierarchy.

that still indicates something that was not intended, so leaving them out does make sense.

what do they indicate? percentages on some hypothetical absolute scale? but that would make no sense and would be corrected promptly.

nothing, but human generally perceives round numbers as some sort of extreme values anyway.

FWIW, cjxl has a special code for d0 (mathematically lossless), and as a consequence it also converts any non-zero distance < 0.01 to d0.01.

while zero is singular and 100 is more or less arbitrary, roughly same argument would apply.

surprising how?

I would have prefered that you can type any name of an animal and libjxl fetches its speed on wikipedia a use that to set the effort level

It seems v0.10.0 remove the needs for `libskcms`, am i right?

Most of the time we count starting at zero. At least in the programmer world

Not 1

nah count at 1

Why? Then effort 10 is a whole extra digit to type

isnt effort 11 already a thing

That's too much effort to type a whole extra keypress

Only in insanity mode

skill issue

Effort 11 SHOULD be difficult to reach since it's almost certainly not what people looking for it actually want

require effort to be typed in unary

-e -e -e -e -e -e -e

https://www.youtube.com/watch?v=Qskm9MTz2V4

surely you only count from zero when the language makes that canonical. if you're counting from zero when you don't have to, that's weird.

Well, setting a slider for a software library, that's a context where it makes sense to count from zero...

The URL of this video is what activates `-e 11`

Oh, also because turning it up to 11 for insanity just makes sense

I dimmly remember somebody asked for rebasing the chrome patch to a specific version (122?) of chrome two days ago. I'm happy to do that, but cannot find the message here anymore...

I saw it too 😄 don't remember the version neither

Maybe whatever build issue they were having was resolved and they deleted the message

I found 2 animated pngs that produce "Getting pixel data failed." error from cjxl, but are played correctly by image viewers. Should I upload it somewhere?

possibly relevant: https://github.com/libjxl/libjxl/issues/756

Seems so, thanks

`FastLosslessEncode()` was changed to `JxlFastLosslessEncode()`. What is the correct way to call it single threaded?

if you pass `nullptr` as the last argument, it should be single threaded

nothing really changed in that code though, afaik. The biggest speedups for lossless are at e4+, for which you have to use the libjxl api (the fast lossless encoder is used by libjxl to implement e1).

Thanks. So JxlFastLosslessEncode() needs to be called for lossless E0, E1, E2, E3, right?

... and is there a recipe for FJxlParallelRunner, which is not optional?

I'd just call the main library API with effort 1/2/3

effort 1 calls what used to be fjxl - you can still call the "old" entry point, but it's mostly meant to be used in case you want a standalone encoder

FastLosslesEncode only implements lossless E1. Anything higher effort requires libjxl.

What actually changed for 0.10's massive speedups?

streaming mode

<@179701849576833024> Here's what I have today: https://github.com/sandflow/libench/blob/90d8fb3ca0f84b8355b91382ac9d7f394b2f86e4/src/main/cpp/jxl_codec.cpp#L32, can you suggest what it should be to be compatible with 0.10?

just renaming the function to JxlFastLosslessEncode and adding two `nullptr` parameters at the end should do the trick

so ``` this->cb_.size = JxlFastLosslessEncode(image.planes8[0], image.width, image.width * 3, image.height, 3, 8, E, &this->cb_.codestream, nullptr, nullptr); ```

What do the nullptrs signify ?

"no parallel runner" (i.e. run everything sequentially)

just renaming the function to

Does lossless encoding in libjxl support multithreading? For v0.10.0, it seems to uses only one cpu core. I was calling the C API with `JxlEncoderSetParallelRunner`. The VarDCT encoding mode has no problem to use this parallel processing.

how large images are you encoding?

theoretically it should thread over groups

but if your image is small enough to have only one group, thrn probably not

it's about `1000*1000`, is the smallest group `1024*1024`?

ah, there's some logic in lossless to make images up to that size a single group, iirc

ah no, it'd be half that size

ignore that

Is lossless e9 generating larger file size than e4 normal?

yes

there is no guarantee that higher effort will produce smaller output, but there is a tendency that it should be true

``` mean colors/px images mean MP | mean e3/e1 | share | share | | tag 1 100.00% 0.00 100.00% 0.00 0.55 DirectClass_sRGB_Alpha 1 100.00% 0.00 100.00% 0.00 0.55 all 1 100.00% 0.00 100.00% 0.00 0.55 icon 1 100.00% 0.00 100.00% 0.00 0.55 source_app mean Mpx/(user+system s) | mean Mpx/(real s) mean bpp | | best of 0.64843750 0.7 0.7 cjxl_0.9.0_d0e7 0.64843750 0.5 0.5 cjxl_0.9.0_d0e8 0.65429688 0.34 0.34 cjxl_0.9.0_d0e9 0.66992188 0.0004973 0.00931 cjxl_0.9.0_d0e10 0.69921875 0.7 1 cjxl_0.9.0_d0e5 0.69921875 0.7 1 cjxl_0.9.0_d0e6 0.71289062 0.7 1 cjxl_0.9.0_d0e4 0.74414062 0.7 1 cjxl_0.9.0_d0e3 0.75781250 0.8 1 cjxl_0.9.0_d0e2 1.36132812 1 1 cjxl_0.9.0_d0e1```

Ok. I get it. Thank you~

nah, -g3 is 1k

yes, but IIRC the heuristic triggers only for images below 512x512 or so

https://github.com/libjxl/libjxl/blob/main/lib/jxl/enc_frame.cc#L319

400x400

512x512 is default group size, trying other sizes is only at high efforts

is there a way to maximize or capping cpu usage for encodes beyond fiddling with effort/threads? curious as to wether dedicating more cpu cycles could result in even quicker encoding

not sure what you mean

in one session losslessly converting an animated gif into jxl consumed only 13% cpu cycles on an old processor (used 8 threads)

We should maybe add some frame parallelism mechanism, for things like a 200x200 animation with 100 frames it's a bit silly that we can only use one thread...

0.0

iirc default groupsize is 256

modular can be anywhere from 128 to 1024

but iirc the default was to use the same as VarDCT, which is only 256 and nothing else

more accurately ``` mean colors/px images mean MP | mean e3/e1 | share | share | | tag 1 100.00% 0.61 100.00% 0.00 0.82 DirectClass_sRGB 1 100.00% 0.61 100.00% 0.00 0.82 all 1 100.00% 0.61 100.00% 0.00 0.82 lossy 1 100.00% 0.61 100.00% 0.00 0.82 photo 1 100.00% 0.61 100.00% 0.00 0.82 source_imagemagick Pareto front for user+system time and size |Pareto front for real time and size || mean Mpx/(user+system s) || | mean Mpx/(real s) || mean bpp | | best of UR 2.95271812 3.16 5.78 cjxl_0.10.1_d0e4 UR 2.99067811 4.85 15 cjxl_0.10.1_d0e3 UR 3.43969780 8.3 17 cjxl_0.10.1_d0e2 ·· 3.64973696 0.1306 0.1359 cjxl_0.10.1_d0e10 UR 3.65891381 34 36 cjxl_0.10.1_d0e1 ·· 3.65962581 0.5602 0.692 cjxl_0.10.1_d0e9 ·· 3.69968224 0.852 1.09 cjxl_0.10.1_d0e8 ·· 3.71802276 1.63 2.54 cjxl_0.10.1_d0e7 ·· 3.76017563 2.28 3.70 cjxl_0.10.1_d0e6 ·· 3.82520470 2.33 4.60 cjxl_0.10.1_d0e5```

You're right, guess my hangover was worse than I thought :P

Is it possible to use a custom effort for VarDCT? Like `e4.5: e4 + gabor-like transform`

For some low end CPU devices, `e4` is quite fast but quality is much worst than `e5` which is too slow

So I'd like to have a balance between these two, don't know if it's possible.

--gaborish=1 to cjxl

That's cool. But how about C api in libjxl?

there's a frame setting for it

Thank you, I'll check it out

The release note of `0.10.1` wrote **fixing a significant speed regression present since 0.9.0**, but which situation could cause the speed regression from 0.9.0 to 0.10.0?

Oh, I updated my program to v0.10.1, which is way faster now with e5 (about 1/2 time saved) for VarDCT.

iirc it was an issue with multithreading in windows builds

where it would only use ~200% even on cpus with 8+ cores

(100% being of one core)

it was because of allocations

we did too many short-lived ones

we’ve consolidated some of them

It reminds me that, this issue also existed in macos/ios in previous versions. Now it's fixed. Many thanks to the dev team~

ugh, does anyone know how to solve this issue on macos? ``` a2x: ERROR: "xmllint" --nonet --noout --valid "/Users/jonsneyers/dev/libjxl/build/djxl.xml" returned non-zero exit status 4 ```

it's caused by the thing not finding the relevant xml thingie: ``` $ "xmllint" --nonet --noout --valid "/Users/jonsneyers/dev/libjxl/build/djxl.xml" I/O error : Attempt to load network entity http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd /Users/jonsneyers/dev/libjxl/build/djxl.xml:2: warning: failed to load external entity "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" D DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd" ^ /Users/jonsneyers/dev/libjxl/build/djxl.xml:5: validity error : Validation failed: no DTD found ! <refentry lang="en"> ^ ```

looks like it's network error

yes well I guess `--nonet` doesn't allow it to fetch it

I think it's supposed to have that particular definition locally, or something. except it doesn't.

Is there a way to convert a JPEG to PNG with it actually transcoding (I don’t need the reconstruction feature and I want the actual file size savings from VarDCT)

Currently I just do convert image.jpg image.png

The covert the png

You can do -j 0 to disable transcoding, or --allow_jpeg_reconstruction=0 to only remove the reconstruction data

what are you trying to? just decode the JPEG to pixels?

if so, you can always decode the JPEG to PNG using `djpegli input.jxl out.png`

if you meant JPEG to JXL, you can then do `cjxl out.png out.jxl`

I recommend using jpegli first, because while you *can* do `cjxl -j 0 input.jpg output.jxl`, cjxl won't use jpegli to decode the jpeg, it will use system libjpeg turbo

Is Jpegli better?

who knows

maybe more mathematically accurate

`export XML_CATALOG_FILES="/opt/homebrew/etc/xml/catalog"` did the trick for me, in case anyone else is running into this issue when building on mac

another annoying thing I'm getting now: ``` /Library/Developer/CommandLineTools/usr/bin/clang++ -DFJXL_ENABLE_AVX512=0 -DHWY_DISABLED_TARGETS="(HWY_SSSE3|HWY_AVX3|HWY_AVX3_SPR|HWY_AVX3_ZEN4|HWY_SVE|HWY_SVE2|HWY_SVE_256|HWY_SVE2_128|HWY_RVV)" -DJPEGXL_ENABLE_APNG=1 -DJPEGXL_ENABLE_EXR=1 -DJPEGXL_ENABLE_GIF=1 -DJPEGXL_ENABLE_JPEG=1 -DJPEGXL_ENABLE_SJPEG=1 -DJXL_EXPORT="" -D__DATE__=\"redacted\" -D__TIMESTAMP__=\"redacted\" -D__TIME__=\"redacted\" -I/opt/homebrew/include -I/Users/jonsneyers/dev/libjxl/lib/../third_party/sjpeg/src -I/opt/homebrew/Cellar/openexr/3.2.2/include -I/opt/homebrew/Cellar/openexr/3.2.2/include/OpenEXR -I/opt/homebrew/Cellar/imath/3.1.10/include -I/opt/homebrew/Cellar/imath/3.1.10/include/Imath -I/Users/jonsneyers/dev/libjxl -I/Users/jonsneyers/dev/libjxl/lib/include -I/Users/jonsneyers/dev/libjxl/build/lib/include -I/Users/jonsneyers/dev/libjxl/third_party/highway -DJXL_DEBUG_WARNING [...] In file included from /Users/jonsneyers/dev/libjxl/lib/jxl/frame_header.h:26: In file included from /Users/jonsneyers/dev/libjxl/lib/jxl/image_metadata.h:21: /Users/jonsneyers/dev/libjxl/lib/jxl/color_encoding_internal.h:348:12: error: use of undeclared identifier 'FROM_JXL_BOOL' return FROM_JXL_BOOL(cms_.run(cms_data_, thread, buf_src, buf_dst, xsize)); ^ 1 error generated. ```

I suppose the issue here is that the `-I/opt/homebrew/include` comes before the `-I/Users/jonsneyers/dev/libjxl/lib/include` so it is taking includes from the system libjxl headers (which are still a v0.9.1 here) and not from the local source tree

Build systems are such a mess, with scripts generating scripts generating scripts and all that — does anyone know where to change something so that order gets changed so outdated system headers do not get used?

you can specify whether project include directories come before or after system ones, this might help https://stackoverflow.com/a/34274943

Thanks! That did the trick.

hello, I have a dumb question! I'm trying to encode a pixel-interleaved CMYK image I'm setting num_color_channels = 3, num_extra_channels = 1, and using JxlEncoderSetExtraChannelInfo() to tag the extra channel as BLACK but with libjxl 0.10.1 I get "./lib/jxl/encode.cc:820: Extra channel 0 is not initialized" do I need to pull out the K channel and encode it separately? the docs seem to suggest I can just pass a 4 channel image to JxlEncoderAddImageFrame() I've probably misunderstood

the PR with the code fwiw https://github.com/libvips/libvips/pull/3883

you can pass it a 4-channel image, but only if it is RGBA

😦 ah that's a shame

would CMYKA be CMYA + K as a separate channel?

yep

that would work

Thanks! I think I'll quietly park CMYK support for now heh, it'll be too much work

hello

i have a compilation error with the last release on windows (msys2 + mingw64) :

C:/Documents/msys2/home/vincent.torri/ewpi_64/share/ewpi/packages/libjxl/libjxl-0.10.2/lib/jxl/color_encoding_internal.h:348:12: error: 'FROM_JXL_BOOL' was not declared in this scope; did you mean 'TO_JXL_BOOL'?

is it known ?

seems like this https://github.com/libjxl/libjxl/pull/3397

it happens when you already have libjxl installed

oh, ok

trying

good

thank you

It should be only a matter of designing the api (JxlPixelFormat etc) in a somewhat more flexible way; there's no real reason we couldn't also support things like CMYK and BGRA or even CMYKA

or even more silly: if you happen to have or need R, G, B in separate planar channels, then the API forces you to convert it to/from interleaved, while internally libjxl does things planar anyway

as a jpeg decoder yea, it produces higher quality results from the same jpeg

JPEG decoding isn't fully specified, unlike most other image codecs

so there's often more than one correct way to decode a jpeg that have slight differences

jpegli provides a higher-quality way to do this, libjpeg-turbo focuses on being fast and compliant

can we fix that? I feel like CMYK is a common use-case for wanting to do that

at least common enough

as far as I understand, isn't the reason for this that most CMS libraries take rgb interleaved? like lcms

so the first stage of encoding / final stage of decoding needs packed

yes, often it's like that.

Someone should make a way to use jpegli with cjxl directly like something like —use-jpegli 1

I agree, cjxl/djxl should use jpegli for jpeg decoding

and encoding, if you happen to output to jpeg

ye

ye, since --pixels_to_jpeg exists

It would be a very small patch to change cjxl to use jpegli

Honestly it is very perplexing that it doesn’t

What's even the point of having all that code there that isn't even being used for anything

So many missed opportunities and low hanging fruit in the libjxl source. And a lot of weirdness too like for example in the bayer matrix code, for some reason the matrix is a weird size for a weirdly explained reason

Like it's working around bugs and quirks in the code instead of actually fixing them and making the code clearer

Not meaning to offend, hopefully this criticism is constructive.

There's literally 3 jpeg encoders in the libjxl source tree

One code path uses sjpeg, one code path uses libjpeg library, and one uses jpegli, which is identical to libjpeg but with "li" added

That's two more than is necessary to exist

Is this a crypto currency related account?

Who doesn’t love spam

wheres the documentation on faster decoding flag

People who want to ban Fab <:YEP:808828808127971399>

Fab is an interesting person though, I still don't understand him

``` cjxl -v -v --help ```

under "Expert Options" third one down iirc

is that all?

is what all?

iirc there's no documentation on what exactly `--faster_decoding` does internally

just hwo to enable it

yeah thats what i meant

I find usually 1 and 2 do the same thing as eachother, and 3 and 4 do the same thing. 90% of the time I get matching hashes

faster decoding just disables various stuff that will cause decoding to be slower, sacrificing some compression for faster decoding. Should generally not be needed for most use cases.

what does it mean "at the expense of quality"

well for lossless it means at the expense of ratio

it behaves like a faster preset basically

What about lossy

same with lossy, behaves like a faster preset but in a way that makes decoding faster

in the sense that faster presets have higher file sizes for the same quality

So why does it say at the expense of quality

higher file sizes for the same quality translates to lower quality at the same file sizes

Alright thanks

`--keep_invisible` should work for this or it's just for some transparent pixels? https://github.com/libjxl/libjxl/issues/3415

the size difference is really small (147 bytes out of ~460 KB), so it's not a big deal IMO

small, but still there and it's only for this image such a setting would be useful or probably `--keep_invisible` should work for such cases as well

and for RGB+Alpha to 1-bit grayscale, perhaps takes the same behavior and transform from webp

--keep_invisible=0 in lossless seems to just zero out fully transparent pixels.

yeah, but basically a fully unused transparent channel is also useless as just some pixels (except for some specific data)

Sure but this particular flag does something different, it's intended for images that actually use the alpha.

for example something like `--keep_invisible=2` should be applied to unused alpha channels as well

However, such optimizations like reducing to RGB or grayscale would be certainly useful. This would require creating a new flag to avoid confusion.

I suppose optimizations should be turned off for lossless by default but they could be turned on for lossy.

at least some settings to be able to replicate the same behavior as lossless webp

It seems VP8L is working exclusively in ARGB but it's doing a great job. It has alpha_is_used flag in the header but that is only informative.

I guess it's debatable if converting an all-opaque RGBA image to RGB is really "lossless", just like converting all-gray RGB to Grayscale, or a 16-bit image to 8-bit if it only actually uses 256 different values per channel. I would say mostly yes, but you could also argue that this image metadata is part of the information that needs to be preserved.

From the libjxl API perspective, we have to decide whether to signal that there's an alpha channel _before_ we receive the actual image frame(s). So we could do such stripping of trivial alpha channels only if the input is fully sent and closed before the first output is requested, and we're not doing a streaming encode.

maybe not by default, but as an option, like for anything lossless as a visible image, but not as some data like optimizers and webp do

The easier way is to let cjxl do it (add an option to check if the input image can be reduced to RGB or to Grayscale), but then of course it's only cjxl that does it.

I have noticed cjxl reducing 24 bit images to 8 in lossless, but that's as far as it goes

I think using oxipng with fast settings to do the optimization beforehand is not that bad.

Not really useful for anything past a single person though

IMHO the application is responsible for using the appropriate pixel format.

in lossless? it shouldn't

24 bit? as in 8 bpc 3 channel or 24 bpc

a recent change may have changed this for greyscale images that used to be generated as rgb

You made me realise it is the same bitdepth, just getting mixed up between per pixel and per channel again

There might be another issue, converting a grayscale image encoded as RGB to the actual grayscale may invalidate the included ICC profile. Anyhow, JXL can encode 256 grays in RGB using palette.

yes, or apply an RCT that turns it into one grayscale channel and two all-zero channels

JXL is pretty good in compressing empty channels. PNG on the other hand not so much. Deflate also has a limit how many bytes can be compressed to a single byte, so large areas of single color result in strings of repeating bytes in the codestream.

Yes, in JXL a single-value channel basically has a constant size (regardless of image dimensions). Unlike most other formats. I should at some point make an update of this blogpost: https://cloudinary.com/blog/a_one_color_image_is_worth_two_thousand_words 🙂

So I suppose JXL is using similar strategy as FLIF.

Not quite, FLIF cheated a bit by encoding channel ranges explicitly and taking those into account in its symbol coder (which is a rather computationally expensive thing to do), so in that specific case it can do it with zero bits but it cannot quite do JXL art style stuff, once the channel range is not a singleton it still has a minimum cost per pixel.

While jxl doesn't have explicit channel ranges, but it does allow contexts to have singleton histograms which are really zero bits per symbol.

(in FLIF that's not an option since it uses a binary arithmetic coding with chances that are adapting dynamically at decode time, so the chance can never be an exact 0 or 1. In jxl the chances are static and signaled in advance)

JXL uses brotli prefix codes if you choose to use prefix codes, and one of the things you can do is you can set the alphabet size to 1

if so, symbols don't take up any bits

the JXL entropy decoder is set up so you request a symbol from the entropy stream, and then it provides it

you can do this as many times as you need. you request a symbol, and it is provided. sometimes the entropy stream needs to read more bits to provide that symbol, sometimes it does not.

in the case of a prefix code with one symbol, it never reads more bits. it just returns the only symbol, every time you ask

so you set up a prefix code where the empty string maps to "0" and that's it.

For the encoder to setup the JXL file this way, the encoder would have to know the contents of the entire image in advance, so it presumably wouldn't be possible to do automatically in streaming mode.

no, but each individual group could be done this way

so an all-zero channel in a group could be encoded in a separate context with essentially no overhead for that group

so while yes theoretically streaming mode makes it so the image size matters, you end up with like, an extremely small overhead for each 2048x2048 chunk, which is still fairly negligible in the grand scheme of things, and significantly lower than, say PNG

It's good to know that JXL can handle that pretty well by itself. I always convert images to the best fitting pixel format before encoding, so the encoder has easier job.

<:Poggers:805392625934663710> maybe also port some webp transforms, palette ordering? https://github.com/libjxl/libjxl/pull/3420

yeah there's still quite some room for improvement in the palette stuff

especially deciding how many colors to allow is something somewhat tricky — in png/webp the limit is 256 anyway, so it's less of an issue, but in jxl you can have thousands of palette colors but just because you can doesn't mean you should 🙂

or has everything useful from here already been taken? <https://github.com/webmproject/libwebp/blob/main/src/utils/palette.c>

no, I haven't implemented any fancy palette ordering yet, it's still basically just luma sorting (with the minor tweak of putting very rarely used colors at the end)

these improvements are just for e4 and slower?

yes, e2/e3 don't do palette and e1 has a separate code path for palette

<@794205442175402004> is this pr ready to merge or still need some changes?

ready to merge — though there's still a lot that can be improved in libjxl's handling of palettes, both regarding speed and compression

these were just some easy changes, that do seem to improve compression by ~1% for manga-type images, and by much more for small icons where the previous code was kind of messing up and using palettes too easily — if a 64x64 icon has 1000 distinct colors, it's unlikely to be a good idea to do palette 🙂

and something fast for e2/e3 would be nice too <:YEP:808828808127971399>

I think 256 colors would be a reasonable default. It can be overrided by the user anyway. Historically color palettes were limited to 256 colors as well. Some kind of RGB arrangement was usually used for more colors.

The current default is to have 1024 as a hard limit, but there are heuristics to make the effective limit smaller.

Also note that in jxl you can have a palette for the whole image, but also a palette for just one group (256x256 pixels, by default).

Also: the palette can involve full pixels, e.g. every palette entry is an RGB or RGBA color, but it can also involve only some of the channels, even just one channel. Single-channel palettes can be quite useful (they have their own heuristics/limits)

Was trying to encode some animations to jxl via apng, but keep getting `JXL_FAILURE: IDAT chunk after fcTL chunk` errors from cjxl. Neither of these produces acceptable by cjxl apng, is there any other way? * `apngasm *.png` * `magick convert *.png APNG:out.apng` * `ffmpeg -f concat -i ./concat_demuxer -plays 0 -c:v apng out.apng`

Maybe open an issue, could be we have a regression in the apng reading...

Looks like there is still an issue with alpha channel decoding performance in 10.0.2: trying 100 decoding cycles for the same image: jxl_lossy_alpha: 794.9941000ms jxl_lossless_a: 786.3887000ms jxl_removed_alpha: 489.7623000ms webp: 146.8004000ms So removing alpha channel altogether gives very high boost, while there is close to zero difference between lossy and lossless alpha decoding speed. the file size differs quite a lot though: lossy_alpha: 20kb lossless_alpha 34kb no_alpha 11kb webp 44kb

the image itself

interestingly, one of the most time-consuming functions is jxl::N_AVX2::MakeUnsigned when finalizing the frame

the difference between alpha/no alpha performance is all in DecodeModularChannelMAANS function and it is huge

DJXL stats: lossy alpha: 8.877 MP/s lossless alpha: 9.117 MP/s no alpha: 14.694 MP/s

that's likely because alpha is modular and modular is much slower than VarDCT

but that said if the alpha channel is all 1s then it shouldn't be that slow

Likely a stupid question - why alpha cannot be VarDCT? Or maybe there is a way to force specifically faster alpha decoding with the size trade off?

VarDCT is hardcoded for 3 channels in the spec. For extra channels, Modular is the only option.

Well, a simple solution. I split alpha and RGB, compress RGB with jxl and alpha with webp. The result is: The overall size is comparable to jxl with lossy alpha - 22kb for jxl without alpha+webp vs 20kb for jxl with lossy alpha. Speed wise - about 2x faster for jxl without alpha+webp

Maybe it is possible to incorporate some really simple solution similar to this split? Even 2x times improvement in decompression speed is essential and the size can be obviously smaller as there is no need for extra header and there is no need to store all three components for grayscale "alpha" image

Maybe there should be an "extra channels effort" option in addition to the main "effort" option, so that a faster setting can be chosen for the alpha. And maybe the effort scale should be changed so that modular and vardct more closely match in encoding speed.

This could be an indication that there is some room for improvement in the decode speed...

Most test images have no alpha so probably it hasn't been tested/optimized as much as it could have been.

oh, decompression speed, not compression speed, derp. I gotta read more carefully.

yes, compression speed is not really important - it is compressed once but decompressed thousand of times

why doesnt cjxl take webp input

mainly, there isn’t that much point

why doesnt cjxl take avif input 🧐

can cjxl convert to xyb like cjpegli?

cjxl always does that unless you ask for lossless

what if i want "lossless" (which it wont be anymore) but make it xyb

that’s also possible

how do i do that?

not sure `cjxl` allows it with “lossless”, but `-m 1 -q 99` should get you close-to-lossless xyb

sorry, `-m` takes a value now (fixed it)

What about heic? Phones use that so that would be valuable

There are tools other than cjxl to convert between formats too

what do you guys use

xl converter. handles resizing unlike convert-only cli, as well as convert to a predetermined filesize https://github.com/JacobDev1/xl-converter

used to jxl batch converter since its more configurable but lags in jxl library support (can still work with newer binaries)

what do you mean

cjxl is supposed to only to basic conversions, anything more advanced like supporting extra formats as source is the domain of specialized conversion/export utilities

i mean this

updated less frequently

but it doesnt have to?

its focused on jxl so youd expect it matching libjxl releases or have a way to fetch any newer binaries

well other than not getting the binaries for you, does it have any other issues

no but thats just me (these are straight frontends calling libjxl after all). check whichever suits your needs better but unless you need advanced options, xl converter is easier to handle

ive been using jxl batch converter so i was curious

how can I see `cjpegli` version ? there's no `--version` argument and `--help` doesn't print any version information

not sure there’s anything better than `cjxl --version`

Is this lossless webp?

lossless webp is overall better than libjxl. The only exceptions being at extreme speeds.

(either extremely fast or extremely slow)

that’s not what https://cloudinary.com/blog/jpeg-xl-and-the-pareto-front#the_pareto_front found

Tbh it's pretty dishonest for that to come from a first party

it's not dishonest to test your own things and report the results

it would be dishonest if the data were manipulated intentionally to get what you want

~~it~~ everything is manipulated intentionally

no?

you do realize it's fairly standard to publish results about your own inventions

not dishonest

yeah but this argument is more of a "not for me"

and when you present data, you always have to manipulate it to show your point

no you don't

you only have to manipulate it if it doesn't already show your point naturally

analyzing data is not the same thing as manipulating it

manipulating it is when you change it or fabricate it post-experiment

umm

what

are you saying the JXL folks are lying

or being intentionally misleading

no

then how is it dishonest

I don't understand the criticism here

for lossless it's harder, especially if using someone else's datasets, like these from avif and daala teams but still some, also for speed, like a specific arch or the number of cores is more favorable and for lossy there's a lot more options

and anyway lossless webp is not overall better, not even close for any speeds (with streaming mode), it may be better only for some specific images like apps or desktop screenshots, but even for screenshots it is very type dependent

<@1028567873007927297> Except on some cherry picked samples, as shown on this old benchmark, libjxl is better at lossless than webp https://docs.google.com/spreadsheets/u/0/d/1ju4q1WkaXT7WoxZINmQpf4ElgMD2VMlqeDN2DuZ6yJ8/htmlview?pli=1

But you are right webp lossless is very good. Good news is, most webp tricks for problematic images should be portable in the jxl format, but haven't been yet

Lossless WebP is the second best lossless format, and lossy AVIF is the second best lossy format, and both come quite close to JXL. If you pick specific images, you can make WebP / AVIF look better than JXL. When I select images to test on, I just try to find images that are relevant / representative for the use case. Ideally testing is done by independents, but in practice it seems to be mostly people working on codecs who are doing codec testing.

infamous p-values

Maybe something changed recently. I remember webp lossless dominating the middle and jxl at the extreme ends. But it looks like jxl got significant speedups in the last update