JPEG XL

It could be more simple and flexible, for a widely consumed codec library

Any unnecessary differences in complexity compared to other common codec libraries is an unnecessary barrier to adoption. Ideally it should be as straightforward as brotli, but that's probably too optimistic. At least libwebp, then, since that's a mature and established competitor.

Oh yeah, I opened a thread here for uploading image which JXL struggles to compress: https://discord.com/channels/794206087879852103/1278292301038227489

it should just be a matter of having the appropriate submodules present (if not using git, we have `deps.sh` to do that) and then running cmake as usual (which is also used by libpng, libjpeg-turbo and libwebp so arguably not that obscure)

what are differences in complexity that you have encountered?

Well for example it's not easy to compile multiple objects for libjpeg/jpegli for different abi versions. It would be nice if there was more granularity and control over what gets compiled, where it gets installed, and with which settings (like api version)

Also I have a feeling a small readable makefile with zero logic would cover 99% of people

JPEGXL_STATIC doesn't work when linking with system libraries

it attempts to link in /usr/lib/libpng.so statically

if you disable JPEGXL_STATIC it dynamically makes libjxl.so as well

~~time to port it to meson~~

bundling in all the dependencies as submodules is a very google thing to do and also not ideal

IMO

I personally prefer it so long as it's optional

it isn't optional because the thing will fail to build if you don't clone all the billion repos

it gives a very solid ecosystem of support that you don't have to worry about distros mucking it about.

this seems like a cmake config issue [Hmm](https://cdn.discordapp.com/emojis/1113499891314991275.webp?size=48&quality=lossless&name=Hmm)

no, it literally just says "please run ./deps.sh"

regardless of whether or not you actually need them

idk, look how much more approachable literally every other codec library source tree looks compared to the spaghetti that is libjxl

yeah this is a config issue, for instance png ```cmake # libpng if (JPEGXL_BUNDLE_LIBPNG AND EMSCRIPTEN) if (NOT EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/libpng/CMakeLists.txt") message(FATAL_ERROR "Please run ${PROJECT_SOURCE_DIR}/deps.sh to fetch the " "build dependencies.") endif() ... ```

this isn't inherently an issue with submodules, it's just that it's hard coded in cmake currently

It definitely sticks out compared to every other codec libraries I have seen, even it's predecessors PIK and FUIF source trees

It just got more messy and spaghetti like over time

it's a problem because they force you to clone them

I personally really like how the actual source is layed out myself, my only issue is the buld system

regardless of why it's a problem, the libjxl build system definitely could use improvement

[yep](https://cdn.discordapp.com/emojis/1075610585141628928.webp?size=48&quality=lossless&name=yep)

really? libjxl code is some of the most unreadable library code I've ever seen

It's not a matter of more code either, but how it's organized into manageable and logical parts

yeah, maybe it's mostly cause I just use vscode or a configured vim + grep for navigation, but I rather like it as far as C/C++ projects go

The different components are not clearly separated.

yea, figuring out where something is done is very hard

if you aren't alreayd familiar

There should be a more clear separation between the library code, and the utility code. And the devtools from the basic tools.

well in any case ~~meson when~~ the build system for sure is my biggest gripe with cross compiling and what not

And it should be simpler to build multiple versions of libjpeg for the different API levels, and have the build artifacts in separate locations, so it's easier for packagers to create packages for "libjxl-jpegli" or "libjpeg-jpegli" or whatever packagers end up naming the package.

idk what's so great about meson compared to cmake though. I am not an expert in either, but you know you can tell cmake to generate a ninja build system?

I believe for 99% of people a really dumb zero-logic makefile will do the job

none of them are really special, it's just meson is a lot easier to read and write IMO, and has some nice "constant" options for things like static compilation

In the library code itself, it would be good if we could better separate encoder, decoder, and unit tests. Even if only to make it easier to count lines of code of the decoder 🙂

xD

Probably it was already asked, but I did not see the answer. Is it possible to add png compression level switch to djxl, or at least switch to enable\disable png compression? For large files in a batch png compression uses more time than jxl decoding itself. I know that there are P*M formats for every usecase, but png is much more convenient for average user

It should be possible, but it's been mentioned a lot about changing to a different, faster PNG library too

I don't know anything about meson other than it requires python.

add a call to `png_set_compression_level()` to this file: https://github.com/libjxl/libjxl/blob/main/lib/extras/enc/apng.cc ideally send a pull request afterwards too. I recommend maybe setting it to 3.

or lower. The PNG encoder is extremely slow, orders of magnitude slower than decoding the image, so it would be great if people didn't get a bad impression of decoding speeds by accident because of this. It's kind of embarrassing.

also it would kinda make jxl look better

oh look how large this PNG is compared to the JXL

although that could be cheating idk

even oxipng is so much faster than libpng default settings ;_;

It's horrible

If I recall oxipng allows feeding direct image data to use as a library

what does `jxl-oxide` use?

*speaking of speed* https://discord.com/channels/794206087879852103/804324493420920833/1204883195926024212

literally someone just needs to add this line `png_set_compression_level(png_ptr, 3);` after this initialization in `apng.cc`: ```cc png_ptr = png_create_write_struct(PNG_LIBPNG_VER_STRING, nullptr, nullptr, nullptr); if (!png_ptr) return JXL_FAILURE("Could not init png encoder"); info_ptr = png_create_info_struct(png_ptr); if (!info_ptr) return JXL_FAILURE("Could not init png info struct"); ``` ez patch

make it even less than 3 if that's too slow.

Haven't done a speed test yet

"good first issue"?

```diff png_ptr = png_create_write_struct(PNG_LIBPNG_VER_STRING, nullptr, nullptr, nullptr); if (!png_ptr) return JXL_FAILURE("Could not init png encoder"); info_ptr = png_create_info_struct(png_ptr); if (!info_ptr) return JXL_FAILURE("Could not init png info struct"); + png_set_compression_level(png_ptr, 3); ```

i would but i can't work out the SLA's

CLA's ?

can't remember what they're called

and so I don't know what you mean <:KekDog:805390049033191445>

Crazy Lawyer Agreement

[Contributor Licence Agreement](https://code.google.com/legal/individual-cla-v1.0.html)

By signing this legally binding agreement, you hereby agree that Google is the best company ever and irrevokably forfeit your rights over your soul. Press Accept to continue.

Click Submit to submit.

i'll click submit once the chrome codec team gets their mind together

https://github.com/libjxl/libjxl/pull/3819

Well hopefully that's right

Wow, that's awsum.

Looks right but did you compile and test?

Measure the improvement?

Letting Github actions do half of that for me, but I did check zlib benchmarks and 3 does seem like a good value. Around 2x faster at a 2-3% density decrease

I'd be curious to know the result.

That's ludicrous and really makes me wonder why the zlib default is so awful.

they probably expect people to change the value for their use-case or something idk

2% for 200% is never a good tradeoff

``` Level Comp Comptime min/avg/max/stddev Decomptime min/avg/max/stddev Compressed size 1 44.805% 0.512/0.536/0.544/0.008 0.128/0.141/0.145/0.003 7,050,618 2 43.764% 0.563/0.586/0.593/0.006 0.129/0.138/0.142/0.002 6,886,856 3 42.712% 0.687/0.725/0.736/0.010 0.129/0.136/0.139/0.002 6,721,288 4 42.062% 0.716/0.778/0.788/0.010 0.123/0.134/0.138/0.003 6,619,035 5 41.235% 0.953/1.037/1.072/0.035 0.123/0.134/0.137/0.003 6,488,854 6 40.734% 1.287/1.428/1.499/0.065 0.123/0.132/0.135/0.003 6,409,974 7 40.628% 1.520/1.692/1.790/0.087 0.124/0.131/0.135/0.002 6,393,327 8 40.505% 2.289/2.527/2.642/0.102 0.119/0.132/0.135/0.003 6,374,005 9 40.490% 2.501/2.760/2.883/0.101 0.123/0.131/0.135/0.003 6,371,682```

Stolen from a github page comparing zlib to other forks <https://github.com/zlib-ng/zlib-ng/discussions/871>

Might actually be better to set it to 2 actually

I mean ideally we'd switch to a faster library entirely

The assumption is often that encode speed matters less than decode speed. Except here of course the encode is done just to store the result of a decode 🙂

Yep well that takes more time than 1 line of code

posting this again because it's relevant: https://discord.com/channels/794206087879852103/804324493420920833/1204883195926024212

is zlib-ng a dropin?

*Technically* setting it to 0 would mean minimal impact on JXL decode time

congrats, you invented bmp

I heard Luca was planning on porting pnge to libjxl

API and ABI wise yes if it's compiled with `ZLIB_COMPAT` although I'm not a build system person so idk how much effort it would be from that angle.

If fpnge happens then there's no need to worry about zlib or whatever

Maybe someone could take <@179701849576833024>'s super fast PNG encoder and convert the x86 intrinsics to generic highway simd so it is portable. And then we could use that to write PNG extremely fast. It would also be nice if we can get rid of the libpng dependency completely, so also for the decoding...

Luca wrote the fastest png encoder ever afaik

You could also use simde instead of highway to make it portable in a more lazy way

Yeah fpnge, but it's currently x64-only iirc. We need something that works on all platforms, even if it's somewhat less efficient.

It wouldn't be as optimized on non intel though

Simde then!

That someone is not me 😛

Simde provides platform specific vectors on other platforms in a sorta hacky way

But that's to be expected

Idk how it compares to highway in terms of stability. But it's definitely less work than porting to hwy

It's a drop-in, portable implementation of platform specific vectors

But we depend on hwy already anyway, seems more sensible to use it than to bring in a new dependency.

True... but, as long as you keep it under the fpnge folder, it should be fine...

It's a header-only library

and it would only be used to make fpnge portable in the laziest way possible

Just tested it with an 8192 x 8192 8 bit normal map texture ```Main 53.5 MB 32.45 Seconds PNG3 61.6 MB 12.84 Seconds PNG1 65.1 MB 8.55 Seconds PPM 192 MB 3.71 Seconds No Output ?.?? MB 3.16 Seconds``` 20% Filesize increase for a 73% Speed increase at level 1

So 3x speed increase, but it still takes around 3x longer than the actual JXL decoding

Gonna edit the PR to use 1, let GA run so I can test and see what it's like. Worst case I'll just change it back to 2 or 3

I'm sure I win a contest for laziest high effort coder, only editing on Github and making Actions compile so I don't have to download anything

Running a few tests, including 16 bit, it's consistently that ratio +/-5% on both density and speed. If everyone else thinks that's fine, then level 1 should do until a replacement library can be used

``` Main 3.63 MB 1.47 Seconds PNG3 3.95 MB 0.51 Seconds PNG1 4.10 MB 0.36 Seconds PPM 5.93 MB 0.09 Seconds No Output ?.?? MB 0.09 Seconds```

So still 2x slower than the decode itself, but that's the best we're getting without just re-making PPM

how fast is PPM?

Same as `--disable_output`, but I expect that could change with huge image sizes

what about the file size?

Added to the charts, running on the 8K file writing the PPM added around half a second, so I was right about that

btw https://github.com/libjxl/libjxl/releases/tag/v0.11.0 dropped

yay

now just waiting for Homebrew to update

a Friday-the-13th release, I wonder if we'll need a v0.11.1 soon 🙂

lol

good job scoop

unless they're always up to date with `HEAD`

oh Scoop is.... for Windows <:SadCheems:890866831047417898>

When I was ~~overtiming~~ working in Samsung, we had a sad joke, that we invented new end of working ~~night~~ day git push strategy: push-and-run (out of the office, before any conflicts appear). Just reminded me

`git push --force`

No, the idea is to be able to leave office before issues are noticed

never push new code on a friday

`sleep 12h; git push --force` 💯

macOS, not MacOS

Z_BEST_SPEED is a macro you can use instead of 1

Welp it's already merged so feel free to make another 1 line PR xP

Oh lmao

That was fast

is "avoiding abort in release build" indicative of progress towards not aborting on memory allocations too?

yep

https://files.catbox.moe/ebskml.png cjxl v0.11.0 is not able to encode this PNG, it does works with 0.10.3 though

"Getting pixel data failed"

It works after using oxipng on it `Removing sBIT chunk as it no longer matches the image data`

Can't find it, but I thought someone made an issue about that already. Maybe it was only mentioned here

od

sBIT is for signalling significant bits

maybe JXL is having issues truncating the values

Is there any way to get the size of the original JPEG from a transcoded JXL?

I want to write a FUSE file system that transparently compresses JPEGs to JXL and back

Like disk compression but specifically JXL

I don't think you can easily get that info without doing an actual reconstruction

How fast is reconstruction?

It's basically only entropy decode followed by fixed Huffman encode. So quite fast. Not fast enough to do it on the fly just to do `ls` though 🙂

Maybe I can cache the sizes

Just treat it like any non-jpeg file

https://repology.org/badge/vertical-allrepos/libjxl.svg?exclude_unsupported=1&columns=3&exclude_sources=modules,site&header=libjxl%20packaging%20status&minversion=0.10

maybe we should add this `&minversion=0.10` to that packaging overview on the github main page, i.e. if a distro is more than one minor version number behind, it becomes a darker red than the current red...

handful of distros missing (ie mx current, rosa, endeavour, cachyos, popos, nobara, ubuntu next/nightly) and other relevant package names not tracked (ie enlightenment desktop's imlib2-jxl, loader variants and packages integrating or requiring libjxl like imagick)

==> Upgrading jpeg-xl 0.10.3 -> 0.11.0 ==> Pouring jpeg-xl--0.11.0.sonoma.bottle.tar.gz 🍺 /usr/local/Cellar/jpeg-xl/0.11.0: 64 files, 45.6MB

Can you report this to repology.org?

I'm having problems with the libjxl debian package. It's not in the package manager (I almost accidentally installed `libjxr-tools`) so I tried the .deb file from the 0.11.0 release on github, which installed fine, but it doesn't seem to make any CLI tools available

`rehash`

what do you mean?

zsh doesn't automatically recognize newly installed programs in PATH

Running rehash didn't work, and it doesn't work from the bash shell either

I've never had this problem before

Have you tried whether it works in bash?

Yeh, doesn't work either

I think the correct package is libjxl-tools

Is that available for you?

No, it wasn't. So I used the release from github. I tried installing these two. The first one errors with code 100, and the second one is successfully installed but doesn't seem to make any commands available

Is cjxl in your /usr/bin/?

try `type cjxl` or do an ls

What distro are you using?

I'm using Pop!_OS

It's not :/

What error do you get when installing the first package?

I think the jxl package contains cjxl and djxl, and depends on the libjxl package

Oh, these apparently: ``` The following packages have unmet dependencies: jxl : Depends: libjpeg62-turbo (>= 1.3.1) but it is not installable Depends: libopenexr-3-1-30 (>= 3.1.5) but it is not installable Depends: libtcmalloc-minimal4 (>= 2.10) but it is not going to be installed E: Unable to correct problems, you have held broken packages. ```

You should use the ubuntu 22.04 libjxl release since popos is based around Ubuntu LTS https://github.com/libjxl/libjxl/releases/download/v0.11.0/jxl-debs-amd64-ubuntu-22.04-v0.11.0.tar.gz

Ohhh. Right. I'll try that, thanks

If this doesn't work either just download the binaries via https://github.com/libjxl/libjxl/releases/download/v0.11.0/jxl-linux-x86_64-static-v0.11.0.tar.gz and put them in your PATH lol

Also tell your distro to get jxl in it 🙂

This distro is actually in the process of dropping GNOME and using a DE + compositor they wrote from scratch, so I'm definitely going to see if I can get them to adopt JXL hehe

If they're being all new and fancy it doesn't hurt to get JXL in there too!

Their GUI toolkit is `libcosmic/iced`, which is a Rust library. I'm waiting for the Rust JPEG XL decoder from Google Research to see if I can add it as a library to be used by the image handling code in libcosmic

this worked, thanks 🙂

Perfect

You're welcome

I thought Mozilla is the one who will develop a Rusty JXL decoder?

Mozilla wants it, the team here will make it

I thought Google Research was going to make it

well a large amount of the JXL devs are from Google Research

Ah, gotcha

[Firefox will consider a Rust implementation of JPEG-XL #1064](<https://github.com/mozilla/standards-positions/pull/1064>) > the team at Google has agreed to apply their subject matter expertise to build a safe, performant, compact, and compatible JPEG-XL decoder in Rust, and integrate this decoder into Firefox.

I see a surprising amount of people write JPEG XL with a dash that's not supposed to be there

we are 😉

The Google Research has spoken!

**The** Google Research himself

That inaccuracy always makes me feel disgusted

JPEG----XL

JPEG+XL

Slightly different but after finalising it’s the same!

I bet everyone's also going to think the XL is for Extra Large

jpeg minus extra large

JPEG⸺XL

JPEG⸻XL

JPEG〜XL

JPEG〰〰〰〰〰〰XL

JPEG±XL JPEG⁺∕₋XL JPEG|XL

JPEG_XL

(which has the advantage of normalizing to space on Wikipedia and other MediaWiki-based sites)

JPEG‾XL

JPEG‾‾‾‾‾‾‾XL

JPEG™XL

```J P E G \/ | /\ |_```

doing ASCII art now, heh?

``` / ‾| \/ | | /\ |_ / ```

giggling at the thought of a manager walking in and just seeing the last like 12 messages

If you go with a hyphen you can sit back and enjoy watching wiki editors argue about which specific dash it is and whether they should use that or use the minus sign for simplicity

`_| >< |_`

The new logo on jpegxl.info

``` _ __ ___ __ _ _ _ | |_)|_ / __ \/ | \_| | |__ \_/ _/\_ |_/ ```

Looks like a new emoticon sequence

``` _ __ ___ _ _ _ _ | |_)|_ / __ \/ | (__| | |__\__) _/\_ |__) ```

like the ¯\_(ツ)_/¯

I like this one a lot ngl

JPEG¬XL

where ¬ is the logical not

but that gives regular JPEG

(I may be biased to this one)

JPEG<:pancakexl:1283670260209156128>XL

Should put your tile heuristics to work with a JXL logo made of the pancake emoji

Jonny

awsom3 Although in another server, the name is a bit more fitting

JPEG -v -v XL

`$ jpeg --xl`

``` J L J X X L J L```

honestly the new JXL logo from the community website is so freaking cool, I hope projects adapt it as the icon for JXL files. The official one <:JPEG_XL:805860709039865937> is much less appealing imo

yeah <@594623456415449088> and his team did a great job on that logo — they made a bunch of really cool logos, but that one was the best.

i like the old one

<:JPEG_XL:805860709039865937><:JXL:805850130203934781><:Xjxl:822166843313225758><:pancakexl:1283670260209156128>

Did someone actually make the JPEG XL pancake

Jon

https://discord.com/channels/794206087879852103/794206170445119489/1283392553613529138

That's so cute

actually that was not the first time

https://discord.com/channels/794206087879852103/806898911091753051/1036693263446458449

jpeg -v -v -v -v xl

I also like the rotationally symmetric <:JXL:805850130203934781>

yeah I also still like that one

i'd like to keep <:JXL:805850130203934781> as a logo for libjxl

axially symmetric or rotationally symmetric, take your pick

(I hesitated to make it “take your PIK”)

Really should've put a through hole pin on my badge instead of a safety pin so it could spin :P

when people here in Belgium — people who understand Dutch — tell me JPEG XL is a bad name (because the files are not extra large, ha ha), I tell them about the ancestors of jxl

PIK and FUIF are two Dutch words

then they usually agree with me that JPEG XL is not that bad, actually

I guess "PIK FUIF" is like how a French person might interpret the company name "ByteDance"

Pik fuif is a 10/10 name though <:KekDog:884736660376535040>

Bet if I were to tell younger family members about it, they would tink it's funny <:KekDog:884736660376535040>

high-fidelity d*ck pics thanks to the brand-new "penis party" codec

nah I'm happy with "JPEG XL"

even without speaking Dutch, http://wordsafety.com/ would have warned us

> Direct match, this is probably bad! > 21 million native speakers.

I have a feeling relatives would have like jxl if it meant "dick party" <:KekDog:884736660376535040>

and even if extra large, extra large might be not about file sizes

extra large resolution support, for example <:KekDog:805390049033191445>

Extra large amount of coding tools

As a Dutch person I'm never going to look at these image formats in the same way again, thanks

Freaking pik and fuif, you can't make that up

i guess you could say they worked hard on those formats

"JPEG XL images can on average compress down to 60% their original size" => "JPEG XL images can expand to more than twice their original size"

isn't it 60% of other formats

Alternatively, JPEGGING your pixels until less then half of them are left

so technically at a bpp of `2.4 bpp` and `24bpc` it expands over 10x

Squeese transform?

what's fuif

in dutch?

https://en.wiktionary.org/wiki/fuif

It's specifically Belgian Dutch

> Historically the word was also used in the Netherlands, but usage has become rare.

I thought it would make a funny acronym since English speakers would say fweef and have no clue that it means something

I thought the name pik was pretty smart not gonna lie. The shorthand of picture (pic) just sounds right in English.

Never would I have thought it something meant something more vulgar.😂

what version do I set for jpegli if I want libjpeg 8?

is it just 8? cause libjpeg6 is 62, is why I'm confused

(for jpegli)

They don't mess around with the Dutch sample sentences on Wiktionary

<:kekw:808717074305122316>

🫣

lmao

Reminded me of this (18+) https://www.youtube.com/watch?v=9wOUMEVd2XY

Maybe I should encode it with PIK...

Why did I misread "Netherlands" as "Neanderthals"... (no offense)

I'm quite prone to misreading though

maybe “Holland” _is_ better after all!

_hides_

are you referring to that Wiktionary sample sentence there?

"Holland" and "Netherlands" both refer to the fact that the country has some altitude challenges

but "hol" does have some particular other connotations in Dutch besides just "hollow"

as Wiktionary seems to be brutally aware of

oh wait, I always assumed that "Holland" comes from "hol land" (hollow land), but apparently it comes from "houtland" (woodland). TIL

I was referring to https://youtu.be/2Pcv2ySMi40?t=1m56s

Ah, I like watching that RobWords guy

it looks like libjpeg-jpegli 8.0 is missing some symbols

``` /usr/bin/ld: /usr/lib/libtiff.so.6: undefined reference to `jpeg12_write_scanlines@LIBJPEG_8.0' /usr/bin/ld: /usr/lib/libtiff.so.6: undefined reference to `jpeg12_read_scanlines@LIBJPEG_8.0' /usr/bin/ld: /usr/lib/libtiff.so.6: undefined reference to `jpeg12_read_raw_data@LIBJPEG_8.0' /usr/bin/ld: /usr/lib/libtiff.so.6: undefined reference to `jpeg12_write_raw_data@LIBJPEG_8.0' ```

did you link to libjpeg-turbo 3.0.0+ ?

same issue as here: https://github.com/mozilla/mozjpeg/issues/437

system libtiff so probably

oh, that new version does support lossless jpeg!

> Added support for 8-bit-per-component, 12-bit-per-component, and 16-bit-per-component lossless JPEG images. A new libjpeg API function (`jpeg_enable_lossless()`), TurboJPEG API parameters […], and a cjpeg/TJBench option (`-lossless`) can be used to create a lossless JPEG image. (Decompression of lossless JPEG images is handled automatically.)

So after only 31 years, the original JPEG spec is finally getting implemented in more places?

Wild, what's next? Arithmetic jpeg support?

that’s already there

including for 12-bit jpeg, starting with this release > Arithmetic entropy coding is now supported with 12-bit-per-component JPEG images.

Whaaa amazing the amount of energy they are using just to avoid jpeg xl

well libjpeg and libjpeg-turbo has been around forever and the spec for JPEG does have 12-bit stuff defined sooo

also it already supported 12-bit and arithmetic *separately* but now it supports doing both at the same time it seems

It is nice libjeg-turbo does it. I find the timing of this release suspicious

It's probably just that people are more interested in image formats recently, with HDR becoming more mainstream etc

Anyway, I think it's too late for JPEG 1 to become a real HDR-capable codec — even though 12-bit mode has been in the spec from the beginning, and even in only 8-bit mode you kind of can do HDR like jpegli does it, and with gain maps you can also do it — since there are just way too many deployments of JPEG implementations that just assume 8-bit buffers are OK.

Maybe make the new files have a different extension to tell them apart

Or am I just inventing JPEG XT

Just like JPEG 1 is never going to support alpha transparency even though technically the codec has no problems encoding 4-component images. Too many applications already exist that assume there will be no alpha.

And also JPEG 1’s performance is bested by most modern formats

So if you’re going to adopt something that most things won’t support, adopt AVIF or JPEG XL

You can guess which one of those two I would pick though

thing about JPEG XT is it's fully backwards compatible with pre-existing JPEG decoders while the 12-bit mode in JPEG is not

So if anything it would probably make more sense to use XT

So you get new features as well as backwards compatibility

Instead of using the format in a way that nothing can use

Annoying thing is the XT SPEC is split into 9 parts so you pay like 1500 for the whole thing

I don’t actually get why XT wasn’t adopted more

It seems like a decent idea

I guess it could be a chicken and egg

They can be opened in compatibility mode in any software so no software was bothered to the new features

But then since nothing supported it no one used the features

Yes, I think that's the problem with graceful degradation

Sometimes breaking things is better than gracefully degrading

Is there a recommended way to losslessly transcode a 287 MB JPEG that does not involve buying five time the RAM I have currently? :p

Try a lower effort setting?

`--streaming_output` doesn't affect RAM usage, and `--streaming_input` doesn't support JPEG (yet?)

Oo I'll try that

It seems this jpeg is too powerful even for `--streaming_output --lossless_jpeg=1 -e 1 --num_threads=0` unfortunately

I could increase disk swap, yeah

Effort 7 and lower are not very memory hungry, I believe. I do not have 287 MB jpeg, but for 81 MB -e7 used 4 GB of ram. Not that terrible

I have effectively 32 GB currently

I don't care that much anyway, was trying to convert picture from the JWST out of curiosity

Can you share it? I would be interested to try it with my PC

Absolutely

https://wormhole.app/A1L45#7L1z0dVN2UypjBe_m2pIPQ

<@167354761270525953>

Thank you

Wait this is the night watch, not the JWST picture

It is the one I was talking about though

It'd actually be kind of interesting to see how lossless super high resolution scans of old paintings compress down to lossy image formats on their default setting. As a benchmark for how appropriate different formats are for digitally preserving art

so without jpeg reconstruction?

this kind of thing is why a progress indicator would be helpful

maybe an optional callback function?

like `JxlRegisterProgressHandler(void (*callback)(JxlProgress progress))`

not sure what exactly would be in `JxlProgress` struct

ram usage pinned at 58 gb

maybe i shouldn't have used `-e 9`

you should use e4

I honestly have a hard time getting lossless images from scanners to begin with, usually they default to JPEG and then when I find a way to get them to output as a lossless format they still end up having JPEG artifacts

do only higher end ones allow you to get lossless scans?

Apparently there are scanners specifically for paintings, I imagine they might be able to. I can't imagine your average office scanner to be able to do this

https://www.youtube.com/watch?v=dkrGxjbU8wk

This might be what my image above was from

But it may also have been from a scrape of google's high resolution art viewer project

I know I mentioned it before that the transcode isn't chunked like lossy and lossless. I've hit 6 GB of memory by nuding it up to e8 on my 12 MP 4 MB phone photos, yet alone JPEGs hitting the dozens of megabytes

Oh wow, I never noticed it was using that much memory, that's interesting

Oh apparently this is a 3D scan as well, because Rembrandt used a lot of thick blobs of paint to represent different textured materials. Yet another use case JPEG XL

Main issue is the megapixels instead of the filesize. Since JPEGs are lossy you go "Oh it's only 20 MB" and then it expands to 2 GB in memory just to view

When jxl is recompressing jpeg does it 'save' 8x8 blocks just with better compression, or changes to VarDCT or something else?

Hmm, I doubt you could do that, and keep it lossless

It’s just 8x8 but better entropy

And using Modular for DC instead of whatever JPEG uses

In some certain cases you might be able to

Like you could merge 2 empty 8x8 blocks into an 8x16, but I don’t think JXL supports that

It would barely help and would significantly complicate reconstruction

Then it is not clear why libjxl uses SO much memory. Can it just go by one 8x8 at once?

1. Modular

2. It still has to decompress the 8x8 blocks into pretty much as much data as an uncompressed image

Each coeff is 1 byte approx

3. If you did one by one you would lose a lot of prediction and context modelling

Probably 3 is the key

Or... It just wasn't part of the chunked encoding PRs since it was too much effort or they forgor

Regarding point three, you could still use larger clusters of blocks, like divide the image into 16 chunks which can be done in parallel to improve speed or consecutively to use less memory

You mean basically just groups?

Yeah

i converted using -e8 but it fails to decode.?

``` JPEG XL decoder v0.11.0 0.11.0 [NEON] Failed to decode image Warning: could not decode losslessly to JPEG. Retrying with --pixels_to_jpeg... ```

Chunked encoding is currently only implemented when encoding from pixels, right <@179701849576833024> ?

yup

Perhaps for sequential jpegs we could also do streaming input, but it seems a bit tricky

what about jxl input? 🤔 It's already divided into groups with a table of contents that can be decoded in any order...

seems pretty convenient

typically yes, but in the general case things can get messy when there's squeeze involved, or delta palette, or dc frames, etc.

but I guess we could fall back to full-image when it gets too hard, and only do the chunked thing for the easy things like default lossless and default lossy

Is there a good place to read the api docs online aside from just browsing the header files?

Oops, didn't mean to reply to that message.

But it is kinda relevant.

There is a readthedocs

Can’t remember the url but it’s somewhere

libjxl supports progressive decoding, so I kind of assume it must have some kind of non-blocking, asynchronous polling api already? since that woudl be the most natural way to do that kind of thing

doing that synchronously would be very obtuse

I would imagine.

Maybe there's already a progress object

I wonder where the "official" api guide is.

I've been experimenting with JXL animation, especially in terms of optimizing the difference between animation frames using JXL's ability to store multiple reference frames

~~Unexpectedly it turned out that storing layers as crops rather than full frames, sometimes is less efficient both in terms of compression and decoding speed if layers is large enough~~ Edit: I've had a few times where full frame was more effective, but I cannot reproduce it anymore

Splitting an animation frame with large empty regions into several (2-3) smaller layers, without empty regions, using cropping, turned out to be even worse for decoding performance, so at sizes close to 4K frames did not have enough time to decode and were skipped on Ryzen 5 5600X

0.0

sounds like there could be a performance bug for this case of cropped layers — I suspect it's not something we have been testing a lot

I can't even open it despite having enough RAM. The image viewers I#ve tried just crash or show error messages

you don't have gimp?

Gimp just crashed

uh, okay

More interesting though: Gewnview and nomacs, both Qt-based showed the error message `JXL image (57813x48438) is bigger than security limit 256 megapixels` THis can probably be bypassed using an env var, though I'll have to look for it

probably capping at level5

that limit is general, not a jxl level thing

xnview nearly gets there for me, but has some display glitching

although my xnview has been behaving weird, i might just need to update

What is the best way to view large JXL files then? I could always decode, but that is a suboptimal workaround

once the cropped decode api is ready that can be used

Any idea when that'll be?

its on the road map

wait, that limit is built into the jxl plugin

i'm sure Qt generally fails on large images regardless

probably to prevent killing the ram

It does by default, but can be bypassed using the env var `QT_IMAGEIO_MAXALLOC=0`

Which is why I assumed there would be something similar for the JXL limit

<https://github.com/novomesk/qt-jpegxl-image-plugin/blob/main/src/qjpegxlhandler.cpp#L204>

Same thing in kimageformats https://github.com/KDE/kimageformats/blob/master/src/imageformats/jxl.cpp#L220

A fair limation, since images so large may lead to memory running out

well, while xnview and gimp both fully decoded the jpg (xnview using much less memory), nomacs just crashed my environment

If you have gwenview try `QT_IMAGEIO_MAXALLOC=0 gwenview The_Night_Watch_-_HD.jpg`

The fully loaded image with gwenview takes around 12.5GB memory

Oxide already has cropped decode, but naturally that only works for decoding to PNG currently, and last time I tried it didn't seem to make any difference in memory usage for some reason

can i use libjxl in a project if it's written in C++ but it's in C?

as in it's C but it is C++

Sure

as in is libjxl in C or C++?

The API is made in C so you can use it from both C and C++

ok good

thanks

Looks like this thing started moving again https://github.com/libjxl/jxl-rs I wonder how much can by just copy-pasted from jxl-oxide. For now I see container parser was used from that project.

Why not a fork of tirr's project?

long story short, the work to minimize memory usage in jxl-oxide would be quite significant (I still have minor trauma from doing it in libjxl) and a rewrite would be about as much work if not less (except for things that *can* be re-used, which <@206628065147748352> has already started contributing :-))

:)

``` cjxl dune.png dune.jxl -d 0 ``` ``` JPEG XL encoder v0.11.0 0.11.0 [AVX2,SSE4,SSE2] Getting pixel data failed. ``` Any clue why this might happen when trying to convert this file?

Using the verbose options didn't tell me anything else either

This image seems to use a simple palette and has pixel-aligned letters great for patches, so I was really curious how well this one would compress, but unfortunately it fails

With ver 0.10.3 all go ok

JPEG XL encoder v0.10.3 4a3b22d [AVX2,SSE2] Encoding [Modular, lossless, effort: 7] Compressed to 3178.5 kB (2.099 bpp). 2805 x 4318, 13.335 MP/s [13.34, 13.34], , 1 reps, 12 threads.

Oh, okay. I should update

v0.10.3 is older than your version (v0.11.0)

my god

I misread that as 0.13.0

it's late and I'm tired

I use XnView MP to make copy png file and this copy go ok in ver 0.11.0

cjxl dune_copy.png dune.jxl -d 0 JPEG XL encoder v0.11.0 4df1e9e [AVX2,SSE2] Encoding [Modular, lossless, effort: 7] Compressed to 3141.7 kB (2.075 bpp). 2805 x 4318, 10.044 MP/s [10.04, 10.04], , 1 reps, 12 threads.

BTW dune_copy.png is smaller 🙂

What You use to make this png file?

``` lib/extras/dec/apng.cc:895: JXL_FAILURE: Exuberant input after IEND chunk lib/extras/dec/decode.cc:151: JXL_FAILURE: Codecs failed to decode```

`vips jxlsave dune.png dune.jxl --lossless=true --effort=9`

Sounds like the png decoder is mad that there's extra stuff at the end of the file.

Exuberant :)

if you optimize the png with oxipng it also fixes the problem

exiftool: ``` Bit Depth : 8 Color Type : RGB Compression : Deflate/Inflate Filter : Adaptive Interlace : Noninterlaced Warning : [minor] Trailer data after PNG IEND chunk Image Size : 2805x4318 Megapixels : 12.1 ```

it just has some random data tacked onto the end

here it is compressed with 0.11.0 effort 9

<@179701849576833024> continuing here from <#848189884614705192> [link](<https://discord.com/channels/794206087879852103/848189884614705192/1287023247577186405>) what exactly is the "derive macro"?

i thought derive was for things like Debug and Clone

for like default `impl`s

is bitstream reading autogenerated using macros?

yup

well, for headers

is that permanent or just to get things done quickly?

I'd say permanent

there's no good reason to hand-craft those impls

just wondering is there any way to PR without a google account, as my google account is linked to my IRL identity

also am planning to delete my google account as i never use it

or should i make a seperate account just for JPEG XL

when you sign the CLA, you just enter your github username, and i *think* the link between your google account and github account isn't public

As in the thing with my GitHub username and those numbers?

> isn’t public Can someone confirm this?

you can use that github noreply email for the commit, and the bot looks up the signed CLA just from your github username

I didn't make it myself, I got it from a Dune forum. It caught my eye because all text was pixel aligned which seemed like the perfect situation for the patches feature

I plan on making a JPEG XL inspector tool to for example see where patches are used so this seemed like a great test image for that

Thanks, I'm able to use this file :)

Hi guys, I failed to compile libjxl v0.11.0 for iOS. ``` ... -- Check if compiler accepts -pthread -- Check if compiler accepts -pthread - no -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - not found CMake Error at /opt/homebrew/Cellar/cmake/3.30.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:233 (message): Could NOT find Threads (missing: Threads_FOUND) Call Stack (most recent call first): /opt/homebrew/Cellar/cmake/3.30.2/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:603 (_FPHSA_FAILURE_MESSAGE) /opt/homebrew/Cellar/cmake/3.30.2/share/cmake/Modules/FindThreads.cmake:226 (FIND_PACKAGE_HANDLE_STANDARD_ARGS) CMakeLists.txt:244 (find_package) ``` But the same compile command works for v0.10.3 ``` cmake ../ \ -D CMAKE_BUILD_TYPE=Release \ -D CMAKE_SYSTEM_NAME=iOS \ -D CMAKE_OSX_SYSROOT="/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS.sdk" \ -D CMAKE_OSX_DEPLOYMENT_TARGET="16.0" \ -D BUILD_TESTING=OFF \ -D BUILD_SHARED_LIBS=OFF \ -D JPEGXL_ENABLE_DEVTOOLS=OFF \ -D JPEGXL_ENABLE_TOOLS=OFF \ -D JPEGXL_ENABLE_DOXYGEN=OFF \ -D JPEGXL_ENABLE_MANPAGES=OFF \ -D JPEGXL_ENABLE_BENCHMARK=OFF \ -D JPEGXL_ENABLE_EXAMPLES=OFF \ -D BROTLI_BUNDLED_MODE=ON \ -D JPEGXL_STATIC=ON \ -D ENABLE_JPEGLI_DEFAULT=ON ```

hello, libvips dev here, I'm looking at large image support (again) these are eg. slide images and are typically 50,000 x 50,000 pixels to 300,000 x 300,000 pixels (!!)

libjxl won't work well for monsters like this, you really need a tiled mode, with perhaps 512 x 512 pixels in a tile

libtiff has a tag set aside for JXL compression, so tiled tiff with JXL tiles seems the best current solution

does anyone have any thoughts on this? what should I write for each tile? just a libjxl image with as little metadata as possible? (since metadata should be in the enclosing TIFF file)

what would a good tilesize be? somewhere between 512 x 512 and 2048 x 2048 I suppose?