JPEG XL

does djxl use rendering intent specified when converting from xyb to a given gamut? for instance say I encode a JXL image from a bt.2020 image and decode it using colorspace into srgb will `--color_space RGB_D65_SRG_Per_SRG` use a perceptual gamut mode and `--color_space RGB_D65_SRG_Rel_SRG` use a relative gamut mapping mode? or will it just apply the appropriate icc stuff?

I just took a look. Looks like it's just using regular libpng.

And whatever the default speed settings are.

A simple png_set_compression_level might help

I don't think so

but <@604964375924834314> knows better than I

(among other things, he probably even understands the question!)

for context, I want to look into the possibility on mastering for JXL directly, taking advantage of the built in gamut mapping JXL can support

djxl does a really decent job from what I can tell already for this

I believe libwebp has streaming decoding

hydrium uses spng. it's not a library but a single include

`--effort` in djxl for output formats is also not a bad idea, at least for switching between default, slow (best compression) and fastest modes but, it may be a bit complex for many formats and libraries

but, for a fully libpng replacement there is still no apng support, which is important for libjxl

to be honest, PNG decoding is so simple that I'm considering adding a home-rolled PNG decoder to libhydrium

It would, of course, require zlib but I don't see what's wrong with that

saves me the trouble of rolling my own deflate decoder

libpng requires setjmp for error handling which makes it somewhat not portable

pretty sure lodepng can do apng relatively easily

kinda want to write a fast Rust png decoder at some point of my life though xD

(and maybe port fpnge while I'm at it)

yeah, but lodepng doesn't support streaming

fair enough

fpnge with streaming (and potential multithreading for chunks, sadly there is no unified standard) and a fast png decoder with fast path for fpnge would be something unbeatable <:Stonks:806137886726553651>

pretty much all the variants boil down to "at this DAT chunk, the gzip stream is reset, prediction mode is 0, and a row starts", no?

(well, s/is reset/behaves as if it were reset/)

https://github.com/w3c/PNG-spec/issues/54#issuecomment-996883205

I recognize a bunch of names there 😛

although it's lovely to see that none of the proposals seem to do what I would do 😆 (I probably would record a list of # of DAT chunks and # of rows such that you can start parallel decoding at those boundaries)

It could help to see how Pigz does it

pigz is mark adler's parallel zlib/gzip/etc. encoder

generally you can flush the stream to make each fragment independent from others, and record the stream offset and compressed offset for seeking

hmm, it seems that pigz actually uses `Z_SYNC_FLUSH` which does not reset states (i.e. later fragments can refer to prior fragments for backreferences)

anyway pigz doesn't generally support parallel decompression, so that should be not very relevant to that proposal

ah fair

Silly question, but are there DLLs being hosted anywhere from builds? I was going to try replacing old libjpeg DLLs with jpegli, but very quickly realised the dev builds don't have DLLs

yeah, that would be nice, but as far as I know there is still no dll for jpegli to seamlessly replacing libjpeg

Huh, so I guess it's just placeholder here then? <https://github.com/libjxl/libjxl/tree/main/lib/jpegli#building>

only as .so libs for linux, but not as dlls for windows

It seems a decently common use case for the library is to request pixel data from libjxl in the tagged space, for XYB encoded images

Should there be a function to shortcircuit this process?

Ive seen code in a few places that calls getColorAsEncodedProfile(target original), followed by setPreferedColorProfile, followed by getColorAsEncodedProfile(target data)

I'm wondering if this dance can be short-circuited

<@532010383041363969> ^ is jpegli development finished or just other tasks for libjxl are more important right now? and anyway it would be nice to also have windows libraries for drop-in replacement

It was just now I realised they helped make Zopfli too, seems they're an expert at squeezing the most out of the old formats

We have all the data, everything lined up for the blog post lightly documenting jpegli's performance, but the team prioritized streaming encoding in JPEG XL before that. I believe technically it is ready, just the evidence that is actually awesome is a secret still... 😅

Jpegli is finished from technical viewpoint or at least in a milestone where it is great to implement

It might make sense to add more dequantization or filtering modes for the decoder later, but if it ever happens it is more likely for 2027 than 2024

Shortly put: it's finished 🌞🌞🌞

i see, that's great so it's just the compilation scripts? would be nice to fix this too, because currently there are only binary encoders and decoders for windows, but not libs for easy libjpeg replacement

it should probably be pretty simple to compile a dll for windows, just like it is on linux.

keep in mind though that it only implements the libjpeg 6b interface

not 7

Some software was made with v7 in mind

for some reason Arch Linux uses v7 system-wide

idk if a compatibility layer exists

most likely even for 8.0

Afaik nobody uses v8 or higher and everyone seems to agree libjpeg API changes suck and stupidly break compatibility

v6b and v7 seem to be what everyone uses

Jpegli gives 16 but frame buffer access through API extensions

Does it implement some sort of dithering for 8 bit output?

Arch for me is running on libjpeg 8

I sometimes inject jpegli when converting via imagemagick

Wow, you're right...

Apparently Arch provides v6 and v8

Not v7. Weird.

is v6 the actual libjpeg v6 or libjpeg-turbo or mozjpeg (both of which derive from v6)?

v9 is the first version of IJG libjpeg that most people don't use

v8 is where turbojpeg stopped following the ABI

https://libjpeg-turbo.org/About/Jpeg-9

What about software that uses v7? Is v8 compatible?

apparently not

v7 is quite rare to use iirc

Hi all, I'm a just a jxl noob and sorry for the long text. From the perspective of an end user, is Jpeg XL format ready for production yet? Example: I convert my photos to JPEG XL format using Libjxl 0.8 tools, but in the future libjxl will reach version 1.0 with some changes to the image format and could make those jxl photos incompatible. Is it highly recommended to wait for version 1.0 to be released? Another thing I would like to now is about the amazing work of jxl art. It looks to me that jxl has pseudo vectorial capabilities. For example: a jxl with splines for anime lineart decodes for a specific resolution. But with proper software those splines points can be re-calculated easily and the lineart upscaled without loss of quality like a vectorial program (not sure if original tree can be retrieved from jxl though). And I wonder doing this automated on the web as well. I suspect many tree elements won't be easily recalculated for upscaling and probably only a few selected functions can be used if intending to use jxl as a vectorial image format. But not only jxl could replace png, gif and jpeg but could replace vectorial formats in some situations. Am I right about this? Likewise, I wonder if it's possible to use older formats like png or jpeg similarly as "jxl art".

The format is finalized.

So the files should be future proof and future tools will be able to decode them no problem

Thanks. I couldn't get any answer when doing an internet search using keywords: is jxl complete, is jxl ready, is jxl finalized.

As Pashi said, any files made now will work indefinitely. Yes, splines can be used as basic replacements for some vector graphics (there's even a tool already made https://github.com/wwww-wwww/spline). The original tree can be retrieved from a file with some work, there was a Google CTF where it was required to solve <https://github.com/google/google-ctf/tree/master/2023/quals/rev-jxl/solution>. And not easily. JXL can have block sizes of 1024x1024 that are turing complete, so you can code in a way similar to shader art

<https://github.com/libjxl/libjxl/blob/main/CHANGELOG.md#added-6>

It used to be listed under the 0.2 release, but it seems the older ones have since been removed

Thank you for the answers

jxl is quite flexible but it's not turing complete; it would be decoder's nightmare if it were turing complete, because it wouldn't know when the decoding ends

Rather, it would be if not limited to the 1024x1024 blocks

The bitstream format is finalized, but IIRC the current encoder does not yet generate splines. So if you plan encoding lossy, and delete the originals, then maybe future versions of the encoder will achieve beter compression ratios due to more usage of more advanced techniques. If you plan to use losless mode this doesn't matter, as you'll still have the unchanged image. Anyway anything you'll encode will be opened by any future versions of libjxl.

Oh. I see _wb_ posted the status about current encoder today https://discord.com/channels/794206087879852103/848189884614705192/1181543024354938921

I’m usually wary of replacing originals with _any_ lossily-compressed version

even if it’s the ultimate lossy codec one could possibly ever want, as soon as you go in the other direction (need to make a lossy copy for an old device that doesn’t support it), you still incur generation loss

Earlier today I realised I had converted about 350 game screenshots into jpeg to save space about a year ago (Originally to copy onto my phone, but apparently I overwrote the original PNGs too). Now I'm wishing I hadn't so I could use all the new encoders I've learnt about since

Even if I had just known about OxiPNG back then I likely wouldn't have needed to jpeg everything

now you can at least loselessly-jxlize those jpegs 😁

Yeah, worst thing is I used a ffmpeg script to do it, only using `-q:v 1` for quality control. So I have no idea what actual quality they used, baseline or progressive, subsampling, ect

-q:v sets the qp to 1

Yeah, but I don't know what jpeg quality that translates to. I could check with trial and error, or spend 15 minutes reading sourcecode, but maybe later

"jpeg quality" isn't really a thing

there's "quality roughly matching a specific encoder's Q setting"

but that's about it

in other words quality?

right but it's not universal

since jpeg encoders aren't universal

For Jpeg is practically is

that's just pedantic

Oh, hmm

no? windows jpeg encoder notably is different from turbojpeg

which is notably different from mozjpeg (iirc gimp on windows uses mozjpeg, for example)

well encoder was never mentioned

just jpeg

Well yeah, every implementation is different, but they all try to use roughly the same scale

we all know what bad jpeg and good jpeg is

Point was, I rushed into compressing everything before I knew what I was doing

do you regret knowing what you are doing now

ffmpeg's jpeg encoder uses the same quality control as its mpegvideo encoder

which is based on qscale

which is floored at 3

I did wonder about that, which is why I also used -qmin 1

in either case, if you're looking for specific numbers

-q:v 1 is roughly on-par with libjpeg-turbo at ~87 or so

in terms of file size

About what I expected, thank you

Image compression results dependends on multiple factors, not just chosing an ideal image format and it's parameters. Examples: lowering resolution exactly to 1/2, 1/3, 1/4 and so on, to give better metric results, lowering colour depth, indexing colours (ex.: lineart, screenshots), approximating small images to disc cluster size, cropping image a certan amount of pixels so it's divisible by 2, 3, 4.., applying filters like selective gaussian blur and desaturating gently to facilitate the compression algorithms. Is there a "magical" program that intelligently does all this calculations automatically when the user wishes to lower a 3Mb photo to lets say 50Kb with the best quality possible? Ideally the program would give suggestions to the user when necessary, while still allowing to manually tweak some factors for better results? I'm interested in a program that tackles the most basic important factors easy to deal with, but that many times we don't think about it as they are so many. Advanced A.I. features are welcome but not necessary (ex.: face recognition and face mask). That program together with jxl would be amazing.

jpeg itself has no concept of quality level, only quantization tables

Traneptora’s point is that the mapping of q0..q100 to quantization tables varies widely by encoder, as therefore does the reverse

if the encoder even uses a q0..q100 scale (Photoshop doesn’t, it’s 0..12)

but everything else does that. does that mean nothing has quality?

and by everything I mean everything in the world since quality is subjective

everything else does what?

map quantity to some other quantity

right, but if you ask “which distance is this .jxl?”, most people will assume that it has been produced by libjxl, and that the question is which distance was used then

or perhaps, if it was not produced by libjxl, what libjxl distance would produce a similar file

for a jpeg produced by ffmpeg, what does the question mean?

could mean image quality

which libjpeg-turbo quality level would use similar quantization tables? which mozjpeg quality? which sjpeg quality?

the question here was “jpeg quality”

Traneptora described why there is essentially no such thing

you took issue with the fact that encoder was never mentioned, but they are arguably what gives “jpeg quality” any meaning at all in the first place

I initially interpreted it as the likes of image quality

encoding brainrot

I'm working on getting very early previews of images (at 1% of the file size) which as I understand should be possible with progressive_dc, is that possible with libjxl as it currently stands? I've done a few experiments but haven't been able to get it to decode with less than 30% of the image.

Tirr was doing some experiments on this, and was able to pull it off https://discord.com/channels/794206087879852103/1065165415598272582/1167351454110064660 https://discord.com/channels/794206087879852103/1065165415598272582/1167387552500678656 https://discord.com/channels/794206087879852103/1065165415598272582/1176213246437494935 https://discord.com/channels/794206087879852103/1065165415598272582/1176552787786612797 The decoder was jxl-oxide, and I'm not sure what encode parameters were.

maybe libjxl renders only when a progressive scan is complete

It used to have an LQIP decode mode but it's not in the current decoder https://youtu.be/uzYHXqjGgL4

when building on MacOS, after `cmake --install .` I get this: ``` $ cjxl dyld[80790]: Library not loaded: @rpath/libjxl_extras_codec.0.10.dylib Referenced from: <D820F31B-DF5D-33E1-AE1D-BB075178CF04> /usr/local/bin/cjxl Reason: no LC_RPATH's found ```

does anyone know how to fix that? I can run cjxl from the local `build/tools/cjxl` fine, but the installed one isn't finding the libraries

do the libraries actually get installed correctly?

they're in /usr/local/lib

a stopgap solution would be to rebuild/reinstall after `cmake -DBUILD_SHARED_LIBS=OFF .`

to make static binaries?

semi-static, so to speak

libjxl* would be built and linked statically

system libraries, still dynamically

oh I just realized I was working in x86 mode for some silly reason, trying again in arm64 mode

how is your dynamic loader configured?

Linux has ldconfig and ld.so.conf

im sure macOS has an equiv

if not, does macho have rpath?

What is LQIP decode mode & why was it removed?

You can see in the video, it allowed the image to be displayed after a single KB, as for where it went... I'm struggling to find any mention of it now

<:PepeOK:805388754545934396>

Just stumbled across this issue, thought it would be worth bringing up. Since being JXL compatible on release might help down the road ;P https://github.com/libjxl/libjxl/issues/2284

This has been fixed, I can close the issue

Ah, nice

When decompressing an XL to JPEG, non-reconstruction... does it use jpegli?

No, it uses libjpeg

``` leo@gauss ~/Pictures/George :) $ cjxl --lossless_jpeg=0 -d 0 input.jpg test.jxl JPEG XL encoder v0.10.0 739d9dab [AVX2] Encoding [Modular, lossless, effort: 7] Compressed to 928.7 kB (9.555 bpp). 1080 x 720, 0.654 MP/s [0.65, 0.65], 1 reps, 8 threads. leo@gauss ~/Pictures/George :) $ djpegli input.jpg jpegli.png Read 604428 compressed bytes. 1080 x 720, 26.362 MP/s [26.36, 26.36], 1 reps, 1 threads. leo@gauss ~/Pictures/George :) $ djxl test.jxl libjxl-default-jpeg.png JPEG XL decoder v0.10.0 739d9dab [AVX2] Decoded to pixels. 1080 x 720, 17.303 MP/s [17.30, 17.30], 1 reps, 8 threads. leo@gauss ~/Pictures/George :) $ /usr/bin/djpeg -targa input.jpg | magick convert tga:- libjpeg.png leo@gauss ~/Pictures/George :) $ magick compare -verbose -metric pae jpegli.png libjxl-default-jpeg.png null:- jpegli.png PNG 1080x720 1080x720+0+0 8-bit TrueColor sRGB 1.22903MiB 0.020u 0:00.023 libjxl-default-jpeg.png PNG 1080x720 1080x720+0+0 8-bit TrueColor sRGB 1.23833MiB 0.020u 0:00.023 Image: jpegli.png Channel distortion: PAE red: 1028 (0.0156863) green: 1028 (0.0156863) blue: 1799 (0.027451) all: 1799 (0.027451) jpegli.png=>- PNG 1080x720 8-bit sRGB 1.22903MiB 0.330u 0:00.049 leo@gauss ~/Pictures/George :( $ magick compare -verbose -metric pae libjpeg.png libjxl-default-jpeg.png null:- libjpeg.png PNG 1080x720 1080x720+0+0 8-bit TrueColor sRGB 1.23273MiB 0.020u 0:00.023 libjxl-default-jpeg.png PNG 1080x720 1080x720+0+0 8-bit TrueColor sRGB 1.23833MiB 0.020u 0:00.023 Image: libjpeg.png Channel distortion: PAE red: 0 (0) green: 0 (0) blue: 0 (0) all: 0 (0) libjpeg.png=>- PNG 1080x720 8-bit sRGB 1.23273MiB 0.300u 0:00.037 leo@gauss ~/Pictures/George :) $ ```

if you do `cjxl --lossless_jpeg=0 -d 0 input.jpg output.jxl`, you get identical pixel data to if you had done /usr/bin/djpeg

if you do djpegli you get something different

when I say libjpeg I mean whatever it linked to, which in my system is libjpeg-turbo

``` leo@gauss ~/Pictures/George :) $ ldd /usr/bin/djpeg linux-vdso.so.1 (0x00007fff7abf9000) libjpeg.so.8 => /usr/lib/libjpeg.so.8 (0x00007f66fd5d0000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f66fd3ee000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f66fd68c000) leo@gauss ~/Pictures/George :) $ pacman -Qo /usr/lib/libjpeg.so.8 /usr/lib/libjpeg.so.8 is owned by libjpeg-turbo 3.0.1-1 ```

Well it's kind of weird that libjxl depends on so many external libraries, but especially in the case where it already comes with its own, superior JPEG encoder and decoder library that it doesn't even use itself, for anything.

This is *compressing* JPEG to JXL, I think <@1028567873007927297> meant *decompressing* JXL to JPEG But it's probably the same in both cases?

Still, it’s using the system library instead of its own.

you are correct, but I believe it is not different

And considering jpegli is supposed to offer supreme quality improvements, it makes little sense not to capitalize on that

open an issue ig

Yep, it isn't. Using djxl straight to JPG is the same as djxl -> PNG -> magick -> JPG at q95 (default).

libjxl is poised to take over the planet if it could just capitalize on its strengths without getting in its own way. The libjpeg quality improvement is a huge deal, just like mozjpeg was a huge deal. But it’s not emphasized or advertised at all, or linked to, and it’s impossible for the layman to even know about the existence of it.

Eventually I think we should integrate jpegli into libjxl, make the decode API also handle jpeg input and have encode options to produce jpeg output and jpeg-reconstructible-jxl output. Not for version 1.0 though.

I think this has been talked about since last year, when we were discussing using libjxl as a drop-in replacement for libjpeg. It would be nice, but I agree that it's probably not a very high priority.

When I use libjxl to save an image with an ICC profile and then re-open the same image, why is the ICC profile read back using JXL_COLOR_PROFILE_TARGET_ORIGINAL sometimes the actual profile I provided when saving, and sometimes a functionally identical one but with the description changed to what looks like a formulaic description? Eg if I save a file with a Rec2020 ICC profile with the description tag Rec2020-elle-V4-g10.icc then that's still the ICC profile description when I read the image back in, but if I save it with a standard sRGB profile with description sRGB-elle-V4-srgbtrc.icc it is changed to RGB_D65_SRG_Per_SRG.

perhaps libjxl checks if the profile is known one

When doing lossless, it keeps the icc profile as is. When doing lossy, it analyzes the icc profile and if it can equivalently be represented as an Enum colorspace (which is more compact to signal), it does that. In that case, when you read it back, it will not match the original icc profile exactly, only functionally.

Ah, thank you.

Hmm, if that meant even ancient applications could be upgraded to JXL with a drag and drop, I'd be very impressed

I think it is a very big deal, that could be the turning point in libjxl popularity and adoption, if more people knew about its ability to produce regular JPEG at greatly improved quality/bitrate ratio

It’s the perfect “gateway drug” to bring it into the mainstream

But I guess I’m the only one who sees it that way, or it would be higher priority to put it front and center

It seems to me like libjxl is getting in the way of its own popularity and success, by not taking advantage of its unique abilities and strengths that it can offer to the largest groups of people.

The other two problems that I see, is... aside from the GIMP plugin, which is a nice start, there doesn't seem to be anyone from libjxl looking over the libjxl glue code for any other popular image libraries, to offer suggestions and improvements for how to use the libjxl API. For example: https://github.com/libvips/libvips/blob/master/libvips/foreign/jxlload.c https://github.com/libvips/libvips/blob/master/libvips/foreign/jxlsave.c https://sourceforge.net/p/graphicsmagick/code/ci/default/tree/coders/jxl.c Lastly, the only other problem I see getting in the way of broader adoption and spread and pull requests, is the very messy libjxl source tree with lots of external dependencies.

libvips devs were asking for help recently though.

cjxl detects ICC profiles and if it can be represented with the enums, then it will use those instead

so what's likely happening here is cjxl sees that the profile is actually just sRGB, and rather than save the sRGB profile, it's just tagging the image as sRGB

and then when you request sRGB, it synthesizes an sRGB profile

Yes, the libvips devs said they would really like someone familiar with the libjxl api and recommended usage to look over those two files and offer suggestions or comments. That sort of communication could potentially lead to improvements in libjxl’s API itself.

The most important thing about libjxl is how it will be used and adopted in practice and libvips is a very important library and a very big “gateway” to broader adoption and usage.

Same with Magick, since Magick has a lot of users as well, even though vips is imo a far better replacement. But most importantly, looking at the glue code and seeing how libjxl is being integrated in practice, will provide a lot of insight into the design and expectations of the public API for libjxl.

My biggest gripe is that there a appears to be a fairly common use case that should have a shortcut function

which is 1. Request the tagged/`TARGET_ORIGINAL` information. 2. Take that information and feed it to `SetPreferredColorEncoding`. 3. Then, request `TARGET_DATA`.

Basically `TARGET_DATA` *defaults* to sRGB, rather than defaulting to what is tagged. IMO, for an XYB-encoded image with enum values tagged, TARGET_DATA should default to equal TARGET_ORIGINAL. That way you don't have to do that dance.

Stuff like that is perfect, the exact sort of feedback and attention that libjxl needs at this early stage.

In fact I should link to the libjxl-ffmpeg glue here so more people look at it

or rather, short-circuit it with a function 'JxlDecoderRequestOriginalSpace'

etc.

the big question becomes "what if the image is XYB encoded by the original space is an ICC profile space."

then you need a CMS library set

https://git.ffmpeg.org/gitweb/ffmpeg.git/blob/HEAD:/libavcodec/libjxldec.c https://git.ffmpeg.org/gitweb/ffmpeg.git/blob/HEAD:/libavcodec/libjxlenc.c

Here’s also ImageMagick to compare with the GraphicsMagick implementation. https://github.com/ImageMagick/ImageMagick/blob/main/coders/jxl.c

another possible issue is that when decoding animated JXL, the FULL_IMAGE event outputs one presented frame, but there's no way to get the duration of that frame

you'd need to subscribe to the FRAME event as well, and use the most recent nonzero duration from the frame header, stash it somewhere, and then use that next time FULL_IMAGE is fired

this is counterintuitive and kinda weird

possibly relevant: https://patchwork.ffmpeg.org/project/ffmpeg/patch/20231208173106.165084-1-leo.izen@gmail.com/

Unfortunately I'm not sure who the best person to talk to is, when it comes to API suggestions or being an authority on the public API

I just ask in <#848189884614705192>

I don't think there's any authority, just the yellownames are the ones who make the decisions

btw, with regard to libjxl, do you have to call `JxlDecoderSetCms` before you can use a CMS on decode, and if so, how do you get the default CMS interface?

I believe that currently, we set it to ours by default

sorry, no, I had that mixed up with the encoder

is there a way to call `JxlDecoderSetDefaultCms()`

you get it through the new `libjxl_cms` which contains `JxlGetDefaultCms()`

I don't see this function in https://libjxl.readthedocs.io/en/latest/

it’s here: https://github.com/libjxl/libjxl/blob/main/lib/jxl/cms/jxl_cms.h

this looks like it requires <jxl/jxl_cms_export.h>

I don't see where that's located

we should be building that now, but maybe we forget to install it?

where is this installed?

this is what I have installed

``` leo@gauss ~ :) $ ll .local/include/jxl/ total 252K -rw-r--r-- 1 root root 1.9K May 22 2023 butteraugli_cxx.h -rw-r--r-- 1 root root 5.1K May 22 2023 butteraugli.h -rw-r--r-- 1 leo leo 10K Oct 5 23:02 cms_interface.h -rw-r--r-- 1 leo leo 15K Oct 5 23:02 codestream_header.h -rw-r--r-- 1 leo leo 5.6K Oct 5 23:02 color_encoding.h -rw-r--r-- 1 leo leo 1.9K Oct 5 23:02 decode_cxx.h -rw-r--r-- 1 leo leo 69K Nov 25 21:31 decode.h -rw-r--r-- 1 leo leo 1.9K Oct 5 23:02 encode_cxx.h -rw-r--r-- 1 leo leo 68K Dec 2 22:04 encode.h -rw-r--r-- 1 leo leo 1.1K Dec 8 09:45 jxl_cms_export.h -rw-r--r-- 1 leo leo 950 Oct 5 23:02 jxl_export.h -rw-r--r-- 1 leo leo 1.2K Oct 5 23:02 jxl_threads_export.h -rw-r--r-- 1 leo leo 2.1K Oct 5 23:02 memory_manager.h -rw-r--r-- 1 leo leo 6.5K Oct 5 23:02 parallel_runner.h -rw-r--r-- 1 leo leo 2.3K Sep 27 08:19 resizable_parallel_runner_cxx.h -rw-r--r-- 1 leo leo 2.7K Sep 27 08:19 resizable_parallel_runner.h -rw-r--r-- 1 leo leo 2.9K Oct 5 23:02 stats.h -rw-r--r-- 1 leo leo 2.3K Sep 27 08:19 thread_parallel_runner_cxx.h -rw-r--r-- 1 leo leo 2.5K Sep 27 08:19 thread_parallel_runner.h -rw-r--r-- 1 leo leo 5.0K Nov 25 21:31 types.h -rw-r--r-- 1 leo leo 1.1K Nov 25 21:31 version.h ```

oops, it seems we are not installing the headers

we install the .a/.dll/.so, as well as the .pc, but not the headers

``` $ grep -ERHnI -e 'JxlGetDefaultCms' .local/include/jxl/ || echo "Not Found" Not Found ```

it also appears that jxl_cms is now a required depedency of jxl-dec, which it shouldn't be

see: https://github.com/libjxl/libjxl/issues/3009

https://github.com/libjxl/libjxl/issues/3010 opened an issue for tracking purposes

I’ve just tried just not telling cmake to link jxl_dec to jxl_cms, to see what kind of dependency we have there

we have these: ``` C:/msys64/mingw64/bin/ld: lib/libjxl_dec.a(decode.cc.obj):decode.cc:(.text$_ZN3jxl13ColorEncoding12FromExternalERK16JxlColorEncoding[_ZN3jxl13ColorEncoding12FromExternalERK16JxlColorEncoding]+0x55): undefined reference to `JxlCmsCreateProfile' C:/msys64/mingw64/bin/ld: lib/libjxl_dec.a(color_encoding_internal.cc.obj):color_encoding_int:(.text+0x9b): undefined reference to `JxlCmsCreateProfile' C:/msys64/mingw64/bin/ld: lib/libjxl_dec.a(dec_xyb.cc.obj):dec_xyb.cc:(.text+0x253b): undefined reference to `JxlCmsPrimariesToXYZ' ```

as far as I remember (I’ll double check), those don’t actually depend on lcms2/skcms

perhaps we could move them

(to yet another library? 😵‍💫 )

what if these are in their own object files that can be linked into either jxl_cms or jxl_dec

yeah, probably something like that

since you typically won't be using both

at first glance, it does seem like the implementation of those functions does not depend on jxl_cms.cc

there is only some dependency the other way around

that makes sense

I am so annoyed with this refactor...

There should be an install script in the build system to install (or even use or link with) libjpeg from "jpegli"

and the libjpeg example programs cjpeg, djpeg, jpegtran could also be imported into the source tree, from libjpeg-turbo for example, since I think those programs merely interface with the public libjpeg API

jpegli only provides the v6 api atm

cjpeg djpeg etc. on a variety of systems interface with the v8 api

which makes libjpegli not a drop in replacement

and this isn't for changing version compatibility? https://github.com/libjxl/libjxl/pull/2473

I thought libjpeg-turbo supports api 6b

it does, if you build it that way

but some systems by default build libjpeg.so as libjpeg.so.8

Yes, like Arch. But either way, that means it should be possible to just import the cjpeg/djpeg/jpegtran tools from libjpeg-turbo and compile them with libjxl "jpegli" libjpeg

yes

Assuming they use the public API like I think they do

they do

I don't know. v6 and v8 are incompatible in both directions so I'm not sure how this would work

changing the so version number without changing the actual incompatibilities doesn't make sense

It would probably be a good idea to import those tools into the libjxl source tree

god no

what we definitely don't need is more external repositories imported into the source tree

they already shouldn't be doing that in the first place

No, not an external repo

that's even worse

just lifting code from another repo and dropping into your repo is a bad idea

Just the .c files with unneeded stuff removed

god no

that's how security vulnerabilities stay unfixed

Uh, god yes? It's necessary if libjxl is going to be a drop-in replacement for libjpeg library and binaries

why would you copy and paste code

Why wouldn't you?

because that's how security vulnerabilities and bugs stay unfixed

god created dynamic linking for a reason

It's just the standard libjpeg example tools

sure, and those work perfectly fine without copy/paste

there's literally no reason to include them

You don't want to have external dependencies on some other competing libjpeg library provider, that's stupid.

you just don't need to include them at all

there is already cjpegli/djpegli

good, keep it that way

Why?

because they're actually different pieces of code

if all you are doing is copying and pasting code you shouldn't be doing that

ideally it should be a drop-in replacement.

it already is

so they should be renamed and ideally provide jpegtran too

tbh you could rename cjpegli to cjpeg and djpegli to djpeg and you'd get the same user interface. there's no reason to literally copy and paste code

so that everything is in the expected place and works as expected

that's true, assuming that cjpegli/djpegli parse arguments the same way.

you could always make them

¯\_(ツ)_/¯

but there's generally no need for that

many things claiming to have "gzip-compatible" switches are not actually gzip-compatible

and are not literal drop in replacements with gzip

people won't expect that to work

but if you want to, you could do that

copying and pasting someone else's code into your repository is how bugs and vulnerabilities don't get fixed

if cjpegli and djpegli have the same argument syntax as cjpeg and djpeg, then they should just be renamed. And jpegtran should be imported (copy+paste, with non-applicable/dead code removed)

lolno

copying and pasting someone else's code into your repository is how bugs and vulnerabilities don't get fixed

this is not a hypothetical gripe btw, this is a real problem that actually happens IRL

copy and pasting is what you DO when you fork something

<:Weirdga:839363765559885844>

when you fork something it *becomes your code* that's not what you are suggesting

You're essentially making a fork of jpegtran when you copy jpegtran.c and remove things that don't apply to jpegli

you are literally not

but okay

What exactly would you "literally" call it then?

copying and pasting someone else's code into your repository

that's not forking

that's not what a fork is

If you already have a libjpeg but not a jpegtran, it makes no sense to start from scratch

Copying it from an existing implementation is what you do. You only copy what you need and what's applicable.

I hardly see a point in a jpegli-using `jpegtran`, to be honest

as far as I can tell, none of what `jpegtran` can do makes any use of jpegli’s strengths

its point is precisely not to decode and reencode image data

Don't know how, but it (sort of?) does. On Arch, which uses v8, I sometimes LD_PRELOAD jpegli when converting to JPG. Seems to work fine for me.

I know, but, for completeness sake, to be a complete replacement it would have to come with that utility… Like I said maybe a standalone jpegtran can just be copypasted from libjpeg-turbo

why is lcms2 more commonly used, compared to skcms, because it is installed by default in most distros?

it tends to support more stuff (e.g. more profile types, more rendering intents)

and it’s quite battle-tested

AFAIK, skcms is exercised mainly by Chrome, which doesn’t do that much with it (for example, it assumes that all monitor profiles have the sRGB tone curve)

Also skcms appears to have no useful API documentation except the code itself. I had a quick look, it feels very much like something Google uses internally and open sourced but don't really care about much. And knowing Google they'll get bored and randomly cancel it. Not much incentive to move away from lcms2.

it's been around for a long time and is what's used in chromium

Which is Google codebase. Just my opinion but it just doesn't look appealing to use unless you have to. Poor documentation, not widely adopted, doesn't appear to offer anything that lcms can't do.

just like jxl 🤓

(and I should add that's one of the things I've found best about libjxl, the documentation is thorough and accessible and the examples are really helpful. The only thing I think could be improved (unless I've missed it) is a note saying which API version functions appeared at.)

If skcms has a proper release or if libjxl bundle the skcms sources in released tar.gz, it would be easier for maintainers to use it. Maintainers have lot of work and when there is something not standard or complicated, they prefer to do something else. It is convenient to use lcms2 because it is properly packaged and available everywhere.

the main advantage skcms has is arguably execution speed

if feature-completeness, reliability or accuracy are more important, I would tend to favour lcms2

lcms has a gpl speed increasing module

GPL license means libjxl can’t bundle it or depend on it

so an unofficial GPL libjxl fork might use them?

IMO whether to use such a module is a decision best left to applications

Yeah, I don't think there's any reason why libjxl itself would need to depend on it

its not a separate dependency either afaik, its bundled w lcms2, just need to #include the relevant header & do a function call to register the plugin

are the `tps_numerator` and `tps_denominator` in `JxlAnimationHeader` supposed to be flipped? i'm implementing encoding animated jpeg xl's in ffmpeg using the library, but for some reason ```c info->animation.tps_numerator = frame->time_base.num; info->animation.tps_denominator = frame->time_base.den; ``` produces long files, but it works correctly when i flip them i'm using `0.8.1` from linux mint's repos

long files as decoded/displayed by what?

hey remember how my literal one libjxl contribution was opening a file in binary rather than image mode bc otherwise windows fucks it up

turns out stdin and stdout are also in text mode so windows cant read images from / write to there properly lol

& freopen doesnt seem to work for stdin/out on windows (at least not on vscode2022 clang). stackexchange suggested some windows specific api, _setmode, so i just did that for my local build for now but someone who actually knows c can hopefully do a better fix

are you talking about pipe | command? if so that likely powershell issue you are talking about shells like nu and cmd work fine, and piping in programming should also be fine

nope i'm just piping in bytes programatically

hmm...

ive checked reading from stdin and it is just that (at least anything using tools::ReadFile) interprets the input stream as text rather than binary

so it converts \r\n to \n (inside image data) and ends way before the actual data does, presumably because it encountered a null byte and read that as the end of the "text"

setting the stdin mode to rb fixes this and makes everything work how its supposed to

https://learn.microsoft.com/en-us/cpp/c-runtime-library/text-and-binary-mode-file-i-o

Sounds like Microsoft brain damage

Again I’m surprised how something so broken and shitty can even exist

more like why people failed to agree on newline sequences, which made MS had to adapt to C which was designed with Unix in mind and add a new mode

Could have added text translation stuff to windows.h or better yet, not be broken in the first place

well that dates long back to the DOS era, so...

it's not broken it's just different

Well if it's different in a way that pointlessly breaks existing portable C code then I'd say it's broken in the worst way.

If there's no reason, after reading the spec, for a developer to expect the implementation to mangle or "translate" the content of file descriptors, or that the implementation should assume that stdin and stdout are "text" and should have filtering and translation applied to them, and the implementation does that anyways, then it's utterly broken garbage

you can say the same in reverse

Nothing in the C programming language says developers should assume stdin and stdout or any other file descriptors should have their contents filtered or translated and changed without the developer explicitly asking for that

But on Windows the developer has to explicitly ask for it NOT to be changed/translated

There's no reason to assume stdin or stdout are text by default either.

I think it's reasonable when the only stdin and stdout is a console/printer

Nothing about the language specification would give anyone that idea

well it's use is entirely contextual

it often is, but that's just one case and it's common for stdin and stdout to be fed into other programs in pipelines

at the time it was the only case

hindsight 2020

I wasn't there and I don't know the history of _fmode and related braindead microsoft things

The article says nothing about the history

well you weren't there so you can't call it braindead

Well the fact that it's even still a thing that matters today is pretty braindead

I can't talk about back then since I know nothing about its history and I can't find any articles about its history

by that you can infer that at the time nobody could agree to a standard

well, everything is a product of its time

i imagine in 100 years people will be complaining about using C

and not like

G

or like why are we using english and not mandarin

I used to think like you until I started working on a 30 year old code base for work so I'm kind of passionate about this

Well it's pretty easy to recognize braindead stuff at first glance, isn't it? The stuff we complain about today is usually things people warned about back when it was first designed.

Also I still can't find any information about the history of the "file descriptor translation mode" idea but everything points to it being a Microsoft-exclusive idea, and they did not invent DOS.

it is how it is

The only information I can find on it, is relatively recent, and from Microsoft themselves... Pipelines were a pretty big deal on Unix, and C and Unix were made at the same time by the same team.

So it would make sense if you're implementing a C environment, not to assume the developer wants the data they read from file descriptors to be silently mangled without them asking for it

I don't think DOS even has a concept of file descriptors.

C11 says stdin and stdout are text streams, see §7.21.3, ¶7: > At program startup, three text streams are predefined and need not be opened explicitly — standard input (for reading conventional input), standard output (for writing conventional output), and standard error (for writing diagnostic output). (and the previous section says text streams may be mangled to hell and back)

And many different environments mangle text streams, not only Windows / DOS.

(actually, a good number of embedded systems mangle binary streams too!)

So the C11 spec says it's ok to mangle pipelines by default? Well if true, I guess I stand corrected...

Still seems pretty braindead without a specified standard way to request not to mangle

why i hate finger pointing

also gotta consider the mac

even more odd

Well what Lonnie says is news to me. Nothing I have ever read when reading about C has ever suggested to me that I should assume or expect file descriptors to lie to me about their contents

Like this for example... https://pubs.opengroup.org/onlinepubs/9699919799/functions/stdin.html

I don't remember anything in the K&R book telling me to assume file descriptors don't actually pass through their actual contents to me when I read or write to them.

I think the cause for exception is the terminal stdin

But maybe the K&R book actually did warn me about it and I don't remember.

like how do you differentiate EOF from new line when typing

ctrl+D usually sends the EOF signal

or how do you put a newline inside a single line of input

Newline just sends a newline character

You can read past the newline character, but a lot of standard functions in the C library are designed to stop at a newline.

oh but if it's a text file it might have different newlines so maybe it needs translation

<:trollface:834012731710505011>

https://pubs.opengroup.org/onlinepubs/9699919799/functions/getline.html

Well the C language doesn't really care what a newline is.

If a system sends CRLF as a newline then the getline function implementation in the C library can just use CRLF as the delimiter

getline is just a variant of getdelim

well the fopen stuff has been around since at least windows xp if but iirc windows 98

actually yeah it would have been windows 9x since it was at the very least availible in 1998

tps is in units of ticks per second, ffmpeg timebase units are in seconds

so you have to flip it

see avformat/jxl_animdec.c

actually the C programming language by K&R book does tell you this

Lemme pull out my copy, I have one in paper

with text streams, the expectation is that the OS newline will be translated to and from `\n` (which will be LF if you use ASCII, but if you don't use ASCII `\n` may be something else.)

What is the recommended approach to handling thumbnails for a jpeg XL file? The application I'm adding support to has a thumbnail getter which has a few bespoke functions for FITS, XISF etc but for most files it uses exiv to look for a thumbnail added to the file as metadata. I haven't been saving exiv2 thumbnails because adding a small copy of the image as metadata feels like it goes against the jpeg XL focus on good compression ratios. On the other hand I haven't found any specific libjxl thumbnail-related functions and I don't really want to have to do a full decode of every jxl image in a directory just to produce a thumbnail.

So if stdout wasn't a text stream, trying to write a new line to the console wouldn't work for normal C code that uses `\n` to indicate newlines.

If you’re dealing with strings and characters, but what about read() and write()?

what do you mean?

which app ? I heard that if you decode only the first block you get a 8:1 downsampled image (with var DCT)

but I don't know how to get the size of the first block

the idea is that, since the format is progressive, a truncated file can be a valid image file, where to truncate it to have consistent results is the main issue

I'm adding support to Siril, which is an astrophotography processing programme. Jpeg XL is nice for a couple of use cases there - both for high quality lossy versions for web display (once browser support improves) and for a more space efficient format for lossless archiving of final renders.

https://pubs.opengroup.org/onlinepubs/9699919799/functions/read.html

oh okay, then flipping them was the correct solution. thanks!

I mean reading data from a file descriptor…

Thanks, that was the clue I needed, it looks like getting the JXL_DEC_FRAME_PROGRESSION should indicate when there is enough data for the 8x downsampled version.

read(), write(), and file descriptors don't exist in standard C.

those are UNIX system calls

you should expect them to do what UNIX does

(later, retrofitted to POSIX)

Windows has file descriptors and fopen()

Windows is not a POSIX system so you shouldn't expect POSIX-compliance

cause it does not make that promise

POSIX != portable

even if it has some features that look similar to POSIX features

I thought we were talking about Windows mangling data when reading from file descriptors

don't you want to fight in <#806898911091753051> ?

no

it has to be here

nice ! I'll be curious to learn what percentage of the image is required for this 🙂 on average

generally you only need the LF groups for this

which is a small fraction

not sure I could give you a % based on average photographic content encoded with libjxl though

rough average

siril looks like a nice project, je bookmarke

Just based on some tests I did the other day, around 14% seemed to be reliable, but this was on maybe half a dozen images rather than *actual* benchmarking

Varies based on image and encoder/decoder settings

<:Poggers:805392625934663710> https://github.com/libjxl/libjxl/pull/3017

that was one fun review xD

the same is planned for lossless? maybe it will help in slowdowns i haven't compared decoding speeds, but smaller individual blocks may also be slower to decode for lossless?

Lossless -e1 already does that

I'm trying to speed lossless up right now 🙂

(and possibly to finish a few more things, but that will not happen today)

https://github.com/libjxl/libjxl/pull/3020 that was easier than I expected

Is this a bug in libjxl? https://github.com/ImageMagick/ImageMagick/issues/3397

I have the same issue, developer of ImageMagick think libjxl developers may help

at best you can call it a bug in the JPEG spec

JPEG decoders are not required to give you the same pixels

right, that the .jxl can be used to reconstruct the .jpg bit-exactly doesn’t imply that decoding the .jpg with one library (presumably libjpeg-turbo or mozjpeg) and the .jxl with another (libjxl) will return the same pixels

(and arguably, the pixels returned by libjxl are better)

that makes sense, thank you.

Can someone explain how to enable the steaming encoding mode in nightly? https://github.com/libjxl/libjxl/pull/3017 This has already been merged.I tried to have a test. But it still consumes a lot of memory.

probably `--streaming_input --streaming_output` and most likely works only for ppm

thanks i'll have a try

that is certainly true

Windows version done too

(not that I tested it...)

windows https://discord.com/channels/794206087879852103/803645746661425173/1184615338319548538

That seems pretty reasonable

except with these os-specific stuff now ProcProfile doesn't hook the file size <:YEP:808828808127971399> `0 -> 60,601,160: 0.00%. real 152 mb/s (0.379 sec) = 94%. ram 135896 KB, vmem 7980 KB`

that's not surprising, but it should still work on a machine with much less ram than image data

streaming mode is faster by internal jxl benchmark, but by external timers it seems to be somewhat slower, not sure which one is right

it's quite surprising that it would be faster

I'm trying to understand how `JxlEncoderAddChunkedFrame` works. I tried to encode a 6000 * 6000 image with streaming encoding. But as I run the `JxlEncoderAddChunkedFrame`, the `get_color_channel_data_at` runs only once and print `xpos: 0, ypos: 0, xsize: 6000, ysize: 6000, row_offset: 0`(shouldn't it be several times?).

should i set the width and height to 2048 in basic_info? but this will result in a 2048 * 2048 output.

The doc says it's a callback function and xsize and ysize will be at most 2048. Why can I get a 6000 * 6000 in the function? https://libjxl.readthedocs.io/en/latest/api_encoder.html#_CPPv4N26JxlChunkedFrameInputSource25get_color_channel_data_atE

After using the main branch nightly version, the `row_offset` shows up randomly. Sometimes it is `1` and sometimes it becomes very big like `5383389184` or `5525995520`. Is it a bug? Older version like `v0.9.x` doesn't have this issue.

that sounds suspicious

are you in a position to bisect this easily?

oh, in release mode, it will always be big int like `1538260754305061192` or `17104812128822824508`, it won't be `1` anymore. The branch `v0.9.x` has no problem in release mode.

I'm trying

oh, i got it wrong. The row_offset should be set by the callee.

no problems now

<@303475549517512704> are you by any chance the one I've been talking to on the FFmpeg mailing list about libjxl_anim encoder?

hello, that's right

noice, good work btw

this is something I've been meaning to do for a while and I just really haven't gotten around to it

glad that it's getting some love

thanks, i just wanted to see how it compares to gif and saw that there's not really a way to transcode into animated jxl

np. the issue with packetization earlier is that it's not clearly defined for JXL

and the container format 18181-2 makes things significantly more difficult to packetize it

i don't really know what 18181-2 specifies because i haven't read it, but i'm fine with only emitting a single packet if that's preferable

I was testing out cjpegli and djpegli just now, and when using the --xyb flag with cjpegli, the image appears to turn into vertical stripes. Even after using djpegli to create a decoded file.

post sample image that does this?

or is it every image

I can't reproduce this

It's every image. It is after I install libjxl, but with "install jpegli libjpeg" set to false

So the jpegli tools are installed but not libjpeg.

The tools are linked with the system libjpeg according to ldd