JPEG XL

Info

rules 57
github 35276
reddit 647

JPEG XL

tools 4225
website 1655
adoption 20712
image-compression-forum 0

General chat

welcome 3810
introduce-yourself 291
color 1414
photography 3435
other-codecs 23765
on-topic 24923
off-topic 22701

Voice Channels

General 2147

Archived

bot-spam 4380

libjxl

Demiurge
2024-07-19 12:17:16
Weird that jpegli got its own repo, but not jxl-tools like cjxl, which would have a much greater benefit (less dependencies for libjxl}
jonnyawsom3
2024-07-19 01:06:59
It was split so that libjxl updates wouldn't hold back jpegli updates. Namely all the fixes for failing decodes which won't officially be released until 0.11 of libjxl
Demiurge
2024-07-19 01:40:15
libjxl updates are already holding back libjxl updates... all of the quality improvements apply to libjxl too, they share a lot of code. So by not releasing an update, libjxl is missing out just as much if not even more than jpegli
2024-07-19 01:41:19
Forget quality improvements; significant bugfixes too.
2024-07-19 01:44:52
Having libjpeg-jpegli integrated more closely with libjxl will hopefully provide more reasons and impetus for people to incorporate libjxl into their projects. But having the complexity and dependencies of cjxl/djxl so deeply entwined into the tree and build system rather than as a separate folder and build system, has an opposite and dampening effect
2024-07-19 01:48:31
Although it would be nice if there were separate commands to install libjpeg headers and library, rather than just having 1 ultimate install command that's modified with -D switches
2024-07-19 01:49:55
I'm not familiar enough with Cmake to know how to do that.
TheBigBadBoy - ๐™ธ๐š›
2024-07-19 08:01:15
is it normal that FFmpeg and ImageMagick say that JPEG "losslessly" reconstructed file does not match JXL ? ```$ jxlinfo 1n17lc6ck2041.jxl JPEG XL file format container (ISO/IEC 18181-2) JPEG bitstream reconstruction data available JPEG XL image, 6071x4299, (possibly) lossless, 8-bit RGB Color space: RGB, D65, sRGB primaries, sRGB transfer function, rendering intent: Relative $ compare -metric pae 1n17lc6ck2041.{jxl,jpg} null: 2056 (0.0313725) $ ffmpeg -v warning -i 1n17lc6ck2041.jxl -map 0 -sn -f streamhash -hash crc32 - 0,v,CRC32=913fb90c $ ffmpeg -v warning -i 1n17lc6ck2041.jpg -map 0 -sn -f streamhash -hash crc32 - 0,v,CRC32=c73984f8```
SwollowChewingGum
TheBigBadBoy - ๐™ธ๐š› is it normal that FFmpeg and ImageMagick say that JPEG "losslessly" reconstructed file does not match JXL ? ```$ jxlinfo 1n17lc6ck2041.jxl JPEG XL file format container (ISO/IEC 18181-2) JPEG bitstream reconstruction data available JPEG XL image, 6071x4299, (possibly) lossless, 8-bit RGB Color space: RGB, D65, sRGB primaries, sRGB transfer function, rendering intent: Relative $ compare -metric pae 1n17lc6ck2041.{jxl,jpg} null: 2056 (0.0313725) $ ffmpeg -v warning -i 1n17lc6ck2041.jxl -map 0 -sn -f streamhash -hash crc32 - 0,v,CRC32=913fb90c $ ffmpeg -v warning -i 1n17lc6ck2041.jpg -map 0 -sn -f streamhash -hash crc32 - 0,v,CRC32=c73984f8```
2024-07-19 08:02:03
JPEG XL's decoder could possibly be applying deblocking filters or other enhancement
2024-07-19 08:02:14
Try decompressing the JXL back to JPEG and the compare them
TheBigBadBoy - ๐™ธ๐š›
2024-07-19 08:03:50
> Try decompressing the JXL back to JPEG and the compare them wdym, that's what I did with `djxl 1n17lc6ck2041.jxl 1n17lc6ck2041.jpg`
SwollowChewingGum
TheBigBadBoy - ๐™ธ๐š› > Try decompressing the JXL back to JPEG and the compare them wdym, that's what I did with `djxl 1n17lc6ck2041.jxl 1n17lc6ck2041.jpg`
2024-07-19 08:04:29
as in ``` cjxl source.jpg image.jxl djxl image.jxl destination.jpg ``` then compare `source.jpg` and `destination.jpg`
2024-07-19 08:04:44
The `image.jxl` may be slightly different due to decoding differences
username
TheBigBadBoy - ๐™ธ๐š› is it normal that FFmpeg and ImageMagick say that JPEG "losslessly" reconstructed file does not match JXL ? ```$ jxlinfo 1n17lc6ck2041.jxl JPEG XL file format container (ISO/IEC 18181-2) JPEG bitstream reconstruction data available JPEG XL image, 6071x4299, (possibly) lossless, 8-bit RGB Color space: RGB, D65, sRGB primaries, sRGB transfer function, rendering intent: Relative $ compare -metric pae 1n17lc6ck2041.{jxl,jpg} null: 2056 (0.0313725) $ ffmpeg -v warning -i 1n17lc6ck2041.jxl -map 0 -sn -f streamhash -hash crc32 - 0,v,CRC32=913fb90c $ ffmpeg -v warning -i 1n17lc6ck2041.jpg -map 0 -sn -f streamhash -hash crc32 - 0,v,CRC32=c73984f8```
2024-07-19 08:05:22
JPEG decoding is not fully defined so different JPEG decoders produce slightly different results and this is just a case of that so when you decode the ".jxl" it's libjxl that is doing the work of decoding it and when you decode the ".jpg" it's whatever JPEG decoder that is bundled into ffmpeg doing the decoding work
SwollowChewingGum
2024-07-19 08:05:31
``` โžœ jxltest md5 * MD5 (destination.jpg) = 330783450a65d4bd60194a57fac6502b MD5 (image.jxl) = 07ae8eb1ba44b51af871c83321ef40ab MD5 (source.jpg) = 330783450a65d4bd60194a57fac6502b ```
username JPEG decoding is not fully defined so different JPEG decoders produce slightly different results and this is just a case of that so when you decode the ".jxl" it's libjxl that is doing the work of decoding it and when you decode the ".jpg" it's whatever JPEG decoder that is bundled into ffmpeg doing the decoding work
2024-07-19 08:05:51
this can also cause it
username
2024-07-19 08:06:49
I'm almost 100% sure it is the case since I almost always get very slightly different results and this is normal
2024-07-19 08:07:32
hashes expect the data to be **exactly** the same and doing hashes on the decoded pixels of different JPEG decoders will have different hashes
SwollowChewingGum
2024-07-19 08:07:44
just tested, can confirm
2024-07-19 08:08:15
`source.jpg` and `destination.jpg` are bit identical (md5 match) but `image.jxl` decodes slightly differently to `source.jpg` or `destination.jpg`
2024-07-19 08:08:23
it's nothing to be worried about
TheBigBadBoy - ๐™ธ๐š›
SwollowChewingGum as in ``` cjxl source.jpg image.jxl djxl image.jxl destination.jpg ``` then compare `source.jpg` and `destination.jpg`
2024-07-19 08:08:30
right JPG matches and I get the same amount of PAE error between reencoded JXL and redecoded JPG
SwollowChewingGum
TheBigBadBoy - ๐™ธ๐š› right JPG matches and I get the same amount of PAE error between reencoded JXL and redecoded JPG
2024-07-19 08:09:01
it's nothing to worry about
2024-07-19 08:09:50
you shouldn't be able to see a difference, and on the off chance you can it should be a good difference (less blocking)
username
2024-07-19 08:14:35
decoding a JPEG is always a lossy process because JPEGs don't really exist as a direct representation of pixels so JPEG decoders producing different results is only really a problem if they fall out of spec or if someone has a poorly system setup that expects decoded pixels to be identical across different decoders
SwollowChewingGum
username decoding a JPEG is always a lossy process because JPEGs don't really exist as a direct representation of pixels so JPEG decoders producing different results is only really a problem if they fall out of spec or if someone has a poorly system setup that expects decoded pixels to be identical across different decoders
2024-07-19 08:14:56
> expects decoded pixels to be identical
2024-07-19 08:15:16
lol kinda silly to use a **lossy** format for anything involving things being "identical"
2024-07-19 08:15:33
even if the format has a well-defined spec without deviations
Demiurge
TheBigBadBoy - ๐™ธ๐š› is it normal that FFmpeg and ImageMagick say that JPEG "losslessly" reconstructed file does not match JXL ? ```$ jxlinfo 1n17lc6ck2041.jxl JPEG XL file format container (ISO/IEC 18181-2) JPEG bitstream reconstruction data available JPEG XL image, 6071x4299, (possibly) lossless, 8-bit RGB Color space: RGB, D65, sRGB primaries, sRGB transfer function, rendering intent: Relative $ compare -metric pae 1n17lc6ck2041.{jxl,jpg} null: 2056 (0.0313725) $ ffmpeg -v warning -i 1n17lc6ck2041.jxl -map 0 -sn -f streamhash -hash crc32 - 0,v,CRC32=913fb90c $ ffmpeg -v warning -i 1n17lc6ck2041.jpg -map 0 -sn -f streamhash -hash crc32 - 0,v,CRC32=c73984f8```
2024-07-19 09:30:50
Yes... It's because of slight pixel differences between different jpeg decoder libraries... libjxl and jpegturbo produce very slightly different pixels from the same JPEG data, and that's normal for different JPEG decoders to have slightly different decode output, since it's a lossy format that can be decoded with varying levels of precision.
2024-07-19 09:33:38
This question gets asked around 5 times a week by different people
Tirr
2024-07-19 09:36:58
lossy image decoders are allowed to decode image differently, and iirc JPEG doesn't really define max error allowed (or it's too permissive)
TheBigBadBoy - ๐™ธ๐š›
2024-07-19 09:51:50
remembers me lossy WebP: FFmpeg gave me pixels that were all with an offset of 1 compared to dwebp [โ €](https://cdn.discordapp.com/emojis/867794291652558888.webp?size=48&quality=lossless&name=av1_dogelol)
JendaLinda
2024-07-19 10:52:09
Pixel values decoded from jpeg would often fall in between the 8 bit levels. Decoders usually do some kind of rounding. lbjxl performs dithering if the pixel values don't match 8 bit values.
2024-07-19 11:01:03
As long as you can't see any difference with your eyes between decoders, the lossy compression is working as intended.
Demiurge
2024-07-19 11:04:56
Still, this question gets asked so often that maybe we should have an FAQ or something.
2024-07-19 11:05:47
People expect all JPEG decoders to produce identical pixel values when converted to an image buffer
2024-07-19 11:08:27
And it doesn't immediately occur to people that different decoders produce slightly different output, even in lossy audio for example, like how libmad is considered a superior quality MP3 decoder compared to others
yoochan
Demiurge Still, this question gets asked so often that maybe we should have an FAQ or something.
2024-07-24 07:13:11
a FAQ was in progress, lead by <@594623456415449088> https://discord.com/channels/794206087879852103/794206087879852106/1247977436478574715 If you want to contribute
Jeebh
2024-07-24 08:39:02
How can I sign individual CLA without Google account?
KKT
yoochan a FAQ was in progress, lead by <@594623456415449088> https://discord.com/channels/794206087879852103/794206087879852106/1247977436478574715 If you want to contribute
2024-07-24 08:30:42
I'll add it.
2024-07-24 08:46:17
How's this? **Is Lossless JPEG Transcoding really bit-for-bit accurate?** Converting a JPEG to JPEG XL using `cjxl` and then back to JPEG with `djxl` results in a bit-identical copy of the original JPEG image. However, itโ€™s important to note the JPEG standard leaves room for some interpretation, leading to potential minor variations between different decoders.
Quackdoc
2024-07-24 08:47:48
... between different decoders, including when decoded directly when using libjxl
TheBigBadBoy - ๐™ธ๐š›
2024-07-25 07:57:07
- encode .jpg to .jxl - decode .jxl to .png - ??? - profit of your better quality image <:KekDog:805390049033191445>
Oleksii Matiash
TheBigBadBoy - ๐™ธ๐š› - encode .jpg to .jxl - decode .jxl to .png - ??? - profit of your better quality image <:KekDog:805390049033191445>
2024-07-25 08:20:54
Reminds me people transcoding mp3 to flac to get lossless quality
RaveSteel
2024-07-25 08:45:01
Even better if they simply rename the file
_wb_
2024-07-25 09:04:07
Lost information is lost, obviously. But how the remaining information is used to reconstruct an image is not completely fixed (at least not in jpeg and not in jxl). A decoder can be sloppy (trading precision for speed, for example), or it can be more precise. They can both be conforming decoders. It generally doesn't make a huge difference, but e.g. when cramming an HDR image in an (8-bit mode) jpeg, it can make noticeable differences: the result of libjpeg-turbo decoding to 8-bit buffers can look significantly worse than the result of djpegli (or djxl) decoding to 16-bit buffers.
Demiurge
2024-07-25 09:38:28
That issue... makes no sense to me. According to the flicker test, the "issue" is close to the realm of hallucinations.
2024-07-25 09:43:03
Ok, looking at it again on another screen, I can see the very slight difference in sharpness of the texture. But it's very, very close to my threshold of hallucinatory perception.
lonjil
2024-07-25 09:50:04
lossless means lossless, not visually lossless...
Demiurge
2024-07-25 10:07:34
It's about lossy jpeg decoder output
2024-07-25 10:07:58
and the minute differences between libjxl and libjpeg-turbo
2024-07-25 10:08:03
or mozjpeg
lonjil
2024-07-25 10:39:42
oh, I didn't scroll up so I thought that was about the lossless bug.
JendaLinda
2024-07-26 05:27:54
The other thing is, if you don't have the original image before jpeg compression, you can't tell which decoder is closer to the original.
Traneptora
TheBigBadBoy - ๐™ธ๐š› - encode .jpg to .jxl - decode .jxl to .png - ??? - profit of your better quality image <:KekDog:805390049033191445>
2024-07-28 12:30:34
djpegli does that btw in one pass
๐‘›๐‘ฆ๐‘•๐‘ฃ๐‘ธ๐‘ฅ๐‘ฉ๐‘ฏ๐‘ฆ | ๆœ€ไธ่ชฟๅ’Œใฎไผๆ’ญ่€… | ็•ฐ่ญฐใฎๅ…ƒ็ด 
2024-07-30 02:21:59
so does jpegli have ready executables ๐Ÿ˜›
Demiurge
2024-07-30 03:09:11
Yes
2024-07-30 03:09:42
cjpegli/djpegli
๐‘›๐‘ฆ๐‘•๐‘ฃ๐‘ธ๐‘ฅ๐‘ฉ๐‘ฏ๐‘ฆ | ๆœ€ไธ่ชฟๅ’Œใฎไผๆ’ญ่€… | ็•ฐ่ญฐใฎๅ…ƒ็ด 
2024-07-30 03:10:01
I meant ready-built
JendaLinda
2024-07-30 03:16:08
They are part of the libjxl compilation.
Demiurge
2024-07-30 03:16:54
You just download it. It's right next to cjxl
๐‘›๐‘ฆ๐‘•๐‘ฃ๐‘ธ๐‘ฅ๐‘ฉ๐‘ฏ๐‘ฆ | ๆœ€ไธ่ชฟๅ’Œใฎไผๆ’ญ่€… | ็•ฐ่ญฐใฎๅ…ƒ็ด 
Demiurge You just download it. It's right next to cjxl
2024-07-30 04:19:56
... why the heck even Nix doesn't bundle that in their jpegxl toolchain...
2024-07-30 04:26:00
oh they updated it, now it's just a bunch of stable distros
frep
JendaLinda They are part of the libjxl compilation.
2024-07-30 11:21:01
Isn't this changing though?https://github.com/libjxl/libjxl/pull/3707/commits/5114b1e098aca67a2daf3db8a46e8dc2cead447c
2024-07-30 11:21:26
https://github.com/google/jpegli
2024-07-30 11:23:04
Yes... I do check commits every other day.. I like jxl a little :)
Demiurge
2024-07-31 12:50:08
It's a shame. Would rather they separated `[cd](jxl|jpegli)` first since that would have a much greater benefit for adoption of the library
2024-07-31 12:50:42
Having the libraries bundled together costs absolutely nothing and makes sense to do since it's free
2024-07-31 12:52:25
But having the tools intertwined into the library adds a lot of complexity and dependencies to the library that makes it harder for people who want to include the library into their own projects
2024-07-31 12:54:04
If anything having jpegli and libjxl as a package deal will just increase interest in the newer format
frep
Demiurge If anything having jpegli and libjxl as a package deal will just increase interest in the newer format
2024-07-31 07:29:04
"I see this as an absolute win" :P
2024-07-31 07:29:14
I think I see your point
_wb_
2024-07-31 07:30:11
Ideally, if you ask me, I think we should at some point have: - a libjxl-core repo that's just the libjxl library, nothing else, only depends on libhwy - a libppf repo (what's now libjxl_extras_codec), depends on libjxl, libpng, etc (does PackedPixelFile internal representation <-> various codecs) - a jxl tools repo, with cjxl/djxl/jxlinfo, depends only on libjxl and libppf - a benchmarking repo, with benchmark_xl/butteraugli/ssimu2/..., depends on libppf and has libjxl as submodule - a hdr tools repo - a jxl dev repo, which has all of the above as git submodules and contains all the unit tests and CI stuff, and remaining tools like jxl_from_tree (the existing libjxl repo could be turned into this) - a libjpegli repo - a jpegli tools repo - a new repo that implements a "combined high-level library" that offers a simple api for doing one-shot encode/decode where the encoder can produce jpeg (with jpegli), jpeg-in-jxl (like what `cjpegli input.png tmp.jpg; cjxl tmp.jpg output.jxl` would produce, but this can be done more efficiently in one go), and jxl (with libjxl), and the decoder can decode both jpeg and jxl. That would be a clean way to separate things imo.
frep
2024-07-31 07:31:29
jpeg-in-jxl is the most cracked idea ever, being able to encode straight to it would be nice
2024-07-31 07:34:37
I don't think keeping jpegli in libjxl's codebase is very realistic honestly, someone might want *just* jpegli without a hypothetical perceived bloat of JXL (as much as I dislike to say that)
2024-07-31 07:37:32
Though people are already packaging it separately from libjxl, as in Arch Linux where the libjxl package does not contain jpegli, while there is an AUR package for jpegli
Demiurge
_wb_ Ideally, if you ask me, I think we should at some point have: - a libjxl-core repo that's just the libjxl library, nothing else, only depends on libhwy - a libppf repo (what's now libjxl_extras_codec), depends on libjxl, libpng, etc (does PackedPixelFile internal representation <-> various codecs) - a jxl tools repo, with cjxl/djxl/jxlinfo, depends only on libjxl and libppf - a benchmarking repo, with benchmark_xl/butteraugli/ssimu2/..., depends on libppf and has libjxl as submodule - a hdr tools repo - a jxl dev repo, which has all of the above as git submodules and contains all the unit tests and CI stuff, and remaining tools like jxl_from_tree (the existing libjxl repo could be turned into this) - a libjpegli repo - a jpegli tools repo - a new repo that implements a "combined high-level library" that offers a simple api for doing one-shot encode/decode where the encoder can produce jpeg (with jpegli), jpeg-in-jxl (like what `cjpegli input.png tmp.jpg; cjxl tmp.jpg output.jxl` would produce, but this can be done more efficiently in one go), and jxl (with libjxl), and the decoder can decode both jpeg and jxl. That would be a clean way to separate things imo.
2024-07-31 11:59:17
Don't you think it's possible to go too far in the other extreme too? Most codec libraries only have a separate "lib" and "tools" tree. As long as you keep the testing/benchmarking stuff optional, and keep the dependencies minimal for people who just want to use the library, then that's as complex as you want to make it. Any additional complexity will just make people confused as hell and wondering what they need to download for their XYZ use case.
2024-07-31 12:00:21
libjpegli does not add any additional dependencies or hardly any extra weight to libjxl at all
2024-07-31 12:01:06
Except the libjpeg headers
2024-07-31 12:02:05
It depends on libjpeg headers and it's not smart enough to use the preinstalled headers in the system include path
2024-07-31 12:03:18
But that's not actually that big of a deal.
CrushedAsian255
2024-07-31 12:18:04
I would say 3 trees is fine, โ€œlibโ€ โ€œtoolsโ€ and โ€œbenchmarksโ€
_wb_
2024-07-31 01:38:31
some of the tools like the metrics and the color conversion / hdr tools are not really specific to jxl at all, except they're made by the same people and they reuse some of the libjxl code
Traneptora
2024-07-31 03:34:16
I don't think you need a separate library that can be exported for libppf or whatever, just have it be a library that's statically linked to the tools at compile-time
Demiurge
2024-08-01 03:07:11
I definitely think cjxl/cjpegli and djxl/djpegli should be bundled together, since they both share 90% of their code...
2024-08-01 03:11:22
libjxl (which includes optional dev tools that don't add any additional dependencies and aren't compiled unless enabled), jxl-tools (which includes everything that adds extra dependencies that library users don't need to have), and libjxl-testdata which I think already exists.
lonjil
Demiurge I definitely think cjxl/cjpegli and djxl/djpegli should be bundled together, since they both share 90% of their code...
2024-08-01 11:30:57
Do they?
CrushedAsian255
2024-08-01 11:35:40
Do they have the same API?
2024-08-01 11:35:53
If so it could help people have an easy upgrade path from JPG to JXL
spider-mario
Demiurge It depends on libjpeg headers and it's not smart enough to use the preinstalled headers in the system include path
2024-08-01 11:36:44
iirc, thatโ€™s because it actually depends on the libjpeg header _templates_ that are used to generate the installed libjpeg headers
2024-08-01 11:36:48
the templates are not installed system-wide
lonjil
CrushedAsian255 Do they have the same API?
2024-08-01 12:54:18
no. Jpegli has the libjpeg-turbo API, which is completely different and totally inadequate for jxl.
Demiurge
2024-08-01 03:10:39
I dunno about _totally_...
JendaLinda
2024-08-01 04:24:57
So, I suppose the jpegli folder in libjxl repository becomes a link to the new jpegli repository?
MSLP
_wb_ Ideally, if you ask me, I think we should at some point have: - a libjxl-core repo that's just the libjxl library, nothing else, only depends on libhwy - a libppf repo (what's now libjxl_extras_codec), depends on libjxl, libpng, etc (does PackedPixelFile internal representation <-> various codecs) - a jxl tools repo, with cjxl/djxl/jxlinfo, depends only on libjxl and libppf - a benchmarking repo, with benchmark_xl/butteraugli/ssimu2/..., depends on libppf and has libjxl as submodule - a hdr tools repo - a jxl dev repo, which has all of the above as git submodules and contains all the unit tests and CI stuff, and remaining tools like jxl_from_tree (the existing libjxl repo could be turned into this) - a libjpegli repo - a jpegli tools repo - a new repo that implements a "combined high-level library" that offers a simple api for doing one-shot encode/decode where the encoder can produce jpeg (with jpegli), jpeg-in-jxl (like what `cjpegli input.png tmp.jpg; cjxl tmp.jpg output.jxl` would produce, but this can be done more efficiently in one go), and jxl (with libjxl), and the decoder can decode both jpeg and jxl. That would be a clean way to separate things imo.
2024-08-01 07:19:59
`libjxl-dec` only repo would be useful too for folks maintaining jxl-capable browser forks
Quackdoc
2024-08-01 07:20:45
browsers might be more into jxl-oxide since rust, and browsers do love rust
MSLP
2024-08-01 07:23:31
true, but that would work for mainline browsers; moving to jxl-oxide for browser forks maintainers would probably be too much overhaul over using pre-existing implementations as base, and only upgrading libjxl version or adding some patches extending functionality (for firefox forks)
lonjil
Quackdoc browsers might be more into jxl-oxide since rust, and browsers do love rust
2024-08-01 11:30:24
I know Firefox does a lot of stuff in Rust, but does Chrome? I thought they were still super into C++.
Quackdoc
2024-08-01 11:37:00
chrome has been pushing towards rust pretty hard
Demiurge
2024-08-02 08:46:59
Just a suggestion, but flac has a -V (verify) mode that decodes the file during encoding to make sure the decoded samples match the input values. It's a pretty good idea for a lossless codec to have this. For relatively immature software like libjxl, it should maybe even be enabled by default for every lossless encode. Just for some assurance that the encode really is mathematically reversible and produces the same exact sample values in the original sample format.
RaveSteel
2024-08-02 08:48:32
Agreed
Demiurge
2024-08-02 08:48:59
That could help foster some more trust and convenience for people who would consider using libjxl to reduce the size of image archive/backups
2024-08-02 08:54:03
If I have a large corpus of images that I want to losslessly preserve and shrink using libjxl, currently it's infeasible because of the amount of work required to ensure that it's truly reversible (At least in the sense of getting the same exact sample values, or in the case of JPEG, the same DCT coefficients and metadata/icc profile)
2024-08-02 08:55:11
So if it was built into the encoder and library in a convenient and automatic way like just flipping a switch, that would be great
2024-08-02 08:55:39
that's my 2 cents :)
CrushedAsian255
2024-08-02 11:59:10
Agreed
JendaLinda
2024-08-03 01:04:33
There are some lossy options usable with `-d 0` as well. The verification would warn the user they're using a lossy feature.
Demiurge
2024-08-03 05:26:45
What, like -d 0 -m 0 -epf 1 or something?
jonnyawsom3
2024-08-03 07:02:35
I assume the non-lossless lossless images pushed this back a while?
_wb_
2024-08-03 07:28:09
Mostly people being on holidays is delaying v0.11, I guess.
A homosapien
2024-08-03 07:58:22
At this pace 1.0 will be released in 2026 instead of 2025 ๐Ÿ˜…
JendaLinda
Demiurge What, like -d 0 -m 0 -epf 1 or something?
2024-08-04 09:54:55
There are options like lossy palette, resampling, bitdepth override etc.
frep
I assume the non-lossless lossless images pushed this back a while?
2024-08-05 09:47:08
Looks like it was just one dev to fix it in about 8 days
2024-08-05 09:48:25
I would assume exhaustive lossless tests will be developed after 0.11
2024-08-05 09:49:52
(Ngl, the fact that this issue came up at all makes me wonder how libpng or libwebp guarantees losslessness.)
_wb_
2024-08-05 10:14:58
This particular regression was a race condition bug in multithreading with relatively rare conditions for happening in practice (you need an image that is just slightly larger than 2048x2048), which caused it to remain undetected.
2024-08-05 10:15:42
Libpng and libwebp don't do multithreading so that makes it easier to avoid those kind of bugs, I guess ๐Ÿ™‚
Tirr
2024-08-05 11:52:47
is there any way to force cjxl to use huffman for entropy coding?
veluca
2024-08-05 12:19:41
with or without source changes?
Tirr
2024-08-05 12:22:04
well I can edit the source code if needed ๐Ÿ™‚
veluca
2024-08-05 12:22:44
https://github.com/libjxl/libjxl/blob/main/lib/jxl/enc_ans_params.h#L83
2024-08-05 12:22:50
make that always true ๐Ÿ™‚
Tirr
2024-08-05 12:23:14
ooh okay
2024-08-05 12:33:19
it seems to work, thanks!
DZgas ะ–
_wb_ Mostly people being on holidays is delaying v0.11, I guess.
2024-08-08 10:41:09
<:This:805404376658739230> ๐Ÿ‘
A homosapien At this pace 1.0 will be released in 2026 instead of 2025 ๐Ÿ˜…
2024-08-08 10:43:54
There really is no hurry. Because it's enough to point a finger to find unfixed bugs and artifacts - that you would not like to see on the release... Although I'm not sure about ambitions and all of like dct 256x256
</Shade>
2024-08-09 04:09:20
When using cjxl to make a lossless jxl file from an existing lossy jpg file, the resulting jxl file is lossy. Below is the command I ran, any ideas why this would be? It only occurs if the source is jpg, while png, webp, etc all work fine. `cjxl source.jpg -q 100 -e 9 --lossless_jpeg=0 --num_threads=15 -x strip=exif -x strip=xmp -x strip=jumbf output.jxl`
2024-08-09 04:13:10
Also, just noticed the "tools" channel. I just posted here because it was the first channel that came up useful when searching "cjxl". Let me know if that's the more appropriate place to ask this.
jonnyawsom3
2024-08-09 04:24:16
How are you checking if it's lossless?
2024-08-09 04:25:58
And is there a reason to use `--lossless_jpeg=0`?
</Shade>
How are you checking if it's lossless?
2024-08-09 04:36:24
I've checked by opening the two files in gimp and checking the color values in different areas, and they are close but not the same. Also, I should be able to convert the original jpg file and the new jxl file to a png and have the hashes match, this is how I noticed the issue originally.
And is there a reason to use `--lossless_jpeg=0`?
2024-08-09 04:37:41
Just for testing. If I disable the pixel re-encode the same thing happens but even more pixel colors are off.
Quackdoc
2024-08-09 04:37:45
could that not just be within the jpeg margin of error?
</Shade>
2024-08-09 04:41:41
If I convert the source jpg file to a png, the pixel colors match exactly between the jpg and the new png file. If I then convert that new png to a jxl file and back to a png again, the colors are still a perfect match. A lossless conversion should not have a margin of error I'd not think?
2024-08-09 04:44:44
Basically, right now in order to preserve my image quality, I'm having to first convert my jpg files to png before making them jxl files. This seems a little silly, especially since I'm loosing out on the lossless recompression feature (--lossless_jpeg=1), maybe I'm missing something obvious? haha
Quackdoc
2024-08-09 04:45:29
jpg itself has a little bit of leeway in how it can be decoded, so im wondering if that's taking effect.
jonnyawsom3
2024-08-09 04:45:53
Mozjpeg vs Jpegli vs JpegTurbo, ect
</Shade>
2024-08-09 04:47:08
I see. That is probably what's going on, as the colors are just changing, there is not any compression artifacts or anything being added.
jonnyawsom3
2024-08-09 04:47:32
And if I recall JXL decoding differs from jpeg decoding slightly too, I know there was a recent discussion about this noting the combinations that do and don't work for verifying
Quackdoc
2024-08-09 04:48:11
interestingly enough, when I encoded a file with this `cjxl source.jpg -q 100 -e 9 --lossless_jpeg=0 --num_threads=15 -x strip=exif -x strip=xmp -x strip=jumbf output.jxl` vs just `cjxl -e 9` the rendering intent changed from relative to perceptual
jonnyawsom3
2024-08-09 04:48:35
Likely the stripped metadata, right?
Quackdoc
2024-08-09 04:48:46
that shouldn't make a difference since it's srgb, but interesting
jonnyawsom3
2024-08-09 04:48:54
Also you can do `-x strip=all`
2024-08-09 04:49:03
Or at least used to be able to
Quackdoc
Likely the stripped metadata, right?
2024-08-09 04:49:04
I dunno what metadata that would be in so I dunno
2024-08-09 04:50:45
```ps โžœ diff <(magick identify -verbose reverse.jxl) <(magick identify -verbose no-reverse.jxl) 26,29c26,29 < standard deviation: 82.0186 (0.321641) < kurtosis: -1.47152 < skewness: 0.177541 < entropy: 0.985663 --- > standard deviation: 81.9497 (0.321371) > kurtosis: -1.47093 > skewness: 0.176782 > entropy: 0.986004 33c33 < mean: 122.941 (0.482123) --- > mean: 122.925 (0.482061) 35,38c35,38 < standard deviation: 83.517 (0.327518) < kurtosis: -1.51971 < skewness: 0.135561 < entropy: 0.982954 --- > standard deviation: 83.5489 (0.327643) > kurtosis: -1.51862 > skewness: 0.135424 > entropy: 0.983019 42c42 < mean: 107.741 (0.422513) --- > mean: 107.881 (0.423062) 44,47c44,47 < standard deviation: 85.414 (0.334957) < kurtosis: -1.353 < skewness: 0.374853 < entropy: 0.972055 --- > standard deviation: 85.4917 (0.335262) > kurtosis: -1.3523 > skewness: 0.375165 > entropy: 0.971709 52c52 < mean: 116.124 (0.45539) --- > mean: 116.161 (0.455533) 54,58c54,58 < standard deviation: 83.6499 (0.328039) < kurtosis: -1.44808 < skewness: 0.229318 < entropy: 0.980224 < Rendering intent: Relative --- > standard deviation: 83.6634 (0.328092) > kurtosis: -1.44729 > skewness: 0.229124 > entropy: 0.980244 > Rendering intent: Perceptual 84,85c84,85 < icc:description: RGB_D65_SRG_Rel_SRG < signature: 2cbc7de4586d92dc4f6a89b6875442a1e8bd1dab4c574bd861fca2830c0d260e --- > icc:description: RGB_D65_SRG_Per_SRG > signature: a3b730ca784b7eb0bfe4e755bfe5b2a9e4b49e9457368372c5df3f0f95347d76 ```
2024-08-09 04:50:52
the magick diff is interesting
</Shade>
2024-08-09 04:51:45
Yeah, I used compare and was just so confused. lol
Quackdoc
2024-08-09 04:52:09
these seem tight enough that it may just be rounding error
jonnyawsom3
</Shade> Just for testing. If I disable the pixel re-encode the same thing happens but even more pixel colors are off.
2024-08-09 04:53:55
That does sound right. Transcode would be using the JXL decoder, the pixels would be using jpegli for decoding and then saving that losslessly, so probably closer to Gimp using mozjpeg (or something similar)
</Shade>
2024-08-09 04:55:23
From an image preservation standpoint, even though I understand that different decoders can show an image with different color values based on how the math is done, I feel like it's not really "safe" to convert to jxl from jpg if these values are physically changing and essentially "lost" when decoding using the same method (gimp).
jonnyawsom3
2024-08-09 04:55:40
Take everything with a grain of salt, but I think I recall the transcode uses Chroma from Luma to try and repair the subsampling, so that would explain the bigger differences
</Shade>
2024-08-09 04:57:41
Unless I'm just ignorant and am missing some fundamental understanding, which to be fair may be possible. I'm coming from webp and avif, where I can convert back to png and get a hash perfect conversion every time (using the same encoder).
jonnyawsom3
2024-08-09 04:59:19
I mean, the jpeg transcode let's you get the exact same file back, so you can't get much more lossless than that
</Shade>
2024-08-09 05:00:04
The jpeg reconstruction?
Quackdoc
</Shade> From an image preservation standpoint, even though I understand that different decoders can show an image with different color values based on how the math is done, I feel like it's not really "safe" to convert to jxl from jpg if these values are physically changing and essentially "lost" when decoding using the same method (gimp).
2024-08-09 05:00:38
well in the end, that's why jpeg reconstruction is important, the file decodes differently in ffmpeg, due to the differences and rounding error and what not, but it reconstructs perfectly ```ps โžœ ffmpeg -hide_banner -loglevel error -i reverse.jxl -c:v ppm -f image2pipe - | b3sum a923cc87c284bc9f9b94eb6df58b605d711e672d8f7a09db05e22ecf88223a0c - โžœ ffmpeg -hide_banner -loglevel error -i input.jpg -c:v ppm -f image2pipe - | b3sum c010d16793e755671204543c237093286701ba2c94d38bc651e45058f069bea4 - โžœ b3sum input.jpg reverse.jxl.jpg faa6b9de237443827a5a41f5c4536e3dab770090ad271005c6f10ee642d93ceb input.jpg faa6b9de237443827a5a41f5c4536e3dab770090ad271005c6f10ee642d93ceb reverse.jxl.jpg ```
jonnyawsom3
</Shade> The jpeg reconstruction?
2024-08-09 05:01:17
Yeah, which is also usually smaller than the standard lossless mode too
Quackdoc
</Shade> The jpeg reconstruction?
2024-08-09 05:01:21
just djxl a transcoded jxl that you didn't do `--lossless_jpeg=0` on
jonnyawsom3
2024-08-09 05:01:40
Also, here's the old thread I mentioned https://discord.com/channels/794206087879852103/804324493420920833/1263767599460778036
</Shade>
2024-08-09 05:02:46
That's the missing piece I did not know! xD
2024-08-09 05:03:22
That's perfect, and it's working as I'd hoped now, lol. Thanks for all the info!
jonnyawsom3
2024-08-09 05:04:02
Always check what you're turning off first! xD
</Shade>
2024-08-09 05:04:51
Yup. I was like, hey with this on converting to png is even worse, but now I see if I use the tooling right it gets me where I want to be. haha
2024-08-09 05:05:17
Thanks again, you two!
jonnyawsom3
2024-08-09 05:11:51
Feel free to ask any more questions in future
spider-mario
2024-08-09 09:42:46
or just `cmp` instead of `sha256sum` and comparing the hashes yourself
Demiurge
2024-08-09 01:04:13
That's a good point... Technically it's NOT lossless to decode a JPEG and encode the resulting pixels. If it's not lossless, then cjxl should exit with error status instead of trying to encode the lossy decoded pixels losslessly
2024-08-12 06:54:19
When I decode a (lossy) jxl that has an embedded ICC profile, djxl creates a new, linear(?!) ICC profile and mangles the colors instead of re-using the same ICC profile embedded in the input file for the output file.
2024-08-12 07:13:56
Can anyone decode this without getting terrible banding? https://cdn.discordapp.com/attachments/794206170445119489/1083872278128885790/imagetest15-Q70.jxl
2024-08-12 07:14:09
From https://discord.com/channels/794206087879852103/847067365891244042/1083980340164558908
2024-08-12 07:17:57
For comparison here it is with `djxl --color_space=RGB_D65_SRG_Rel_SRG` which is still not the same color space embedded in the input file... By default it decodes to `RGB_D65_SRG_Rel_Lin` which would only make sense if decoding to a floating point output buffer...
Traneptora
2024-08-12 12:15:32
Is there a way to force cmake to compile the built-in libpng? I'm cross-compiling on Linux for mingw, and it compiles fine, but cmake can't find the png library bundled in (yes, I ran ./deps.sh) which means that it can't get any pixel data
Demiurge For comparison here it is with `djxl --color_space=RGB_D65_SRG_Rel_SRG` which is still not the same color space embedded in the input file... By default it decodes to `RGB_D65_SRG_Rel_Lin` which would only make sense if decoding to a floating point output buffer...
2024-08-12 12:15:59
this issue has been reported already, iirc
2024-08-12 12:21:46
lemme see if I can find the issue
Traneptora Is there a way to force cmake to compile the built-in libpng? I'm cross-compiling on Linux for mingw, and it compiles fine, but cmake can't find the png library bundled in (yes, I ran ./deps.sh) which means that it can't get any pixel data
2024-08-12 01:01:27
update:
2024-08-12 01:01:38
``` $ wine mingw32/tools/cjxl.exe george.png test.jxl JPEG XL encoder v0.11.0 e10fb685 [AVX2,SSE4,SSE2] ./lib/extras/dec/exr.cc:29: JXL_FAILURE: EXR is not supported ./lib/extras/dec/decode.cc:151: JXL_FAILURE: Codecs failed to decode Getting pixel data failed. ``` well that's probably not supposed to happen now is it
2024-08-12 01:03:10
it's definitely a PNG though
2024-08-12 01:03:20
``` $ umbrielpng george.png PNG signature found: george.png Chunk: IHDR, Size: 25, Offset: 8, CRC32: fab5570d Chunk: sRGB, Size: 13, Offset: 33, CRC32: d9c92c7f Chunk: IDAT, Size: 1281627, Offset: 46, CRC32: c85fd495 Chunk: IEND, Size: 12, Offset: 1281673, CRC32: ae426082 Size: 1080x720, Color: 8-bit RGB ```
2024-08-12 01:03:24
without a fairly complex chunk structure either
KKT
Demiurge For comparison here it is with `djxl --color_space=RGB_D65_SRG_Rel_SRG` which is still not the same color space embedded in the input file... By default it decodes to `RGB_D65_SRG_Rel_Lin` which would only make sense if decoding to a floating point output buffer...
2024-08-12 05:00:39
Not to derail your question, but this image is a perfect example of the bug in macOS's Preview I've submitted to Apple (on an M1 MacBook Pro running 14.6.1). Seems to be any relatively large image with more than 8 bits:
Demiurge
Traneptora lemme see if I can find the issue
2024-08-12 05:42:01
I linked to the issue.
Traneptora
2024-08-12 05:42:16
no, there was a different one
kmadura
2024-08-13 01:16:54
Hello again. I am trying to understand parallel runners in libjxl. Apparently my code doesn't compile due to missing functions in libjxl (linking problem) and I am stuck. My issue: https://github.com/libjxl/libjxl/discussions/3749 Everything else works absolutely fine and beside that, my little program converts a PNG file into lossless JXL without any problems.
2024-08-13 01:17:19
Can somebody take a look at my issue? Ty.
2024-08-13 01:42:14
Nevermind ๐Ÿ˜‰
yoochan
2024-08-13 01:50:49
That was fast ๐Ÿ˜…
gb82
KKT Not to derail your question, but this image is a perfect example of the bug in macOS's Preview I've submitted to Apple (on an M1 MacBook Pro running 14.6.1). Seems to be any relatively large image with more than 8 bits:
2024-08-13 05:13:15
have you reported this in apple's Feedback app?
KKT
2024-08-13 07:16:50
Oh yes.
KKT Oh yes.
2024-08-13 10:16:03
OK, I can confirm at least two of the bugs I reported over the past year are fixed in the latest Sequoia 15.1 Beta (I took the plunge to check!). Preview no longer corrupts as shown in the video above with >8 bits and it now recognizes that it can open JXL files. The other zillion I've filed are a no go, including Preview (and Safari, not that we were expecting that) supporting HDR imagesโ€ฆ
ProfPootis
2024-08-14 05:18:58
I can't seem to compress this image https://www.mediafire.com/file/mo42xgq0f4dw8nx/Combined_100%2525.png/file
2024-08-14 05:19:22
it's 835MB hence the mediafire
Quackdoc
2024-08-14 05:27:19
wow, this image bloody sucks
2024-08-14 05:27:43
magick failed to convert it ```ps โžœ scratch dua 834.93 MiB Combined_100_.png 27.77 GiB magick--yqEFduwpNou4vopFmb6kcq4Pr943Bsr 28.58 GiB total ```
2024-08-14 05:35:35
I cant evennn convert the image to exr using oiiotool
2024-08-14 05:43:00
nothing I have can actually convert this file to anything lmao
2024-08-14 05:50:27
oh wait, png2pnm is streaming, that decoded it
Traneptora
2024-08-14 05:50:30
hydrium?
Quackdoc
2024-08-14 05:54:18
Ill try out hydrium, but trying to encode the ppm file without streaming caused cjxl to oom [av1_dogelol](https://cdn.discordapp.com/emojis/867794291652558888.webp?size=48&quality=lossless&name=av1_dogelol) to be expected since the decoded ppm is 9.77gib lmao
HCrikki
2024-08-14 06:05:35
whats that 834mb png even about? you might need to stream it to encoder
Quackdoc
Traneptora hydrium?
2024-08-14 06:21:31
hit heap max ``` โžœ scratch ~/code/hydrium/build/hydrium Combined_100_.png Combined_100_.png.hydrium.jxl libhydrium version 0.4.1 Max libhydrium heap memory: 2185897 bytes ```
2024-08-14 06:21:49
or is this just output?
2024-08-14 06:24:44
in any case, I can't actually open the image since I don't have a compatible viewer right now lmao, but here are the sizes, i'm assuming hydrium finished, it's quite nice, and that's just output, hydrium certainly didn't kill my PC ```ps โžœ scratch dua 119.69 MiB Combined_100_.png.ppm.default-streaming-in+out.jxl 174.82 MiB Combined_100_.png.hydrium.jxl 591.86 MiB Combined_100_.png.ppm.e3-d0-streaming-in+out.jxl 834.93 MiB Combined_100_.png 9.77 GiB Combined_100_.png.ppm 11.45 GiB total ```
CrushedAsian255
ProfPootis I can't seem to compress this image https://www.mediafire.com/file/mo42xgq0f4dw8nx/Combined_100%2525.png/file
2024-08-14 06:56:15
i wonder why it gives a Segmentation fault
2024-08-14 06:56:39
it doesn't even try to use all the RAM before giving up
2024-08-14 06:57:05
The big hump near the middle is the `cjxl` invocation, system total is 36gb
TheBigBadBoy - ๐™ธ๐š›
Quackdoc nothing I have can actually convert this file to anything lmao
2024-08-14 11:37:38
could a `ect -1` increase compatibility of the image ?
Quackdoc
2024-08-14 11:42:34
no idea, I deleted it since it was sitting in ram
jonnyawsom3
HCrikki whats that 834mb png even about? you might need to stream it to encoder
2024-08-14 12:15:25
"might"
2024-08-14 12:16:57
So out of all the people who downloaded it, what's the actual resolution?
Oleksii Matiash
So out of all the people who downloaded it, what's the actual resolution?
2024-08-14 12:38:46
72622x48128 @ 24 bpp
jonnyawsom3
2024-08-14 12:47:25
So (Roughly) 10x10 8K images stitched together
_wb_
2024-08-14 01:35:35
3.5 gigapixels... That's 10 GB for an 8-bit RGB buffer and 40 GB for a float32 buffer...
gb82
KKT OK, I can confirm at least two of the bugs I reported over the past year are fixed in the latest Sequoia 15.1 Beta (I took the plunge to check!). Preview no longer corrupts as shown in the video above with >8 bits and it now recognizes that it can open JXL files. The other zillion I've filed are a no go, including Preview (and Safari, not that we were expecting that) supporting HDR imagesโ€ฆ
2024-08-14 02:58:43
Awesome!!
jonnyawsom3
_wb_ 3.5 gigapixels... That's 10 GB for an 8-bit RGB buffer and 40 GB for a float32 buffer...
2024-08-14 03:43:18
Being futureproof for high resolutions is paying off already at least
Demiurge
_wb_ 3.5 gigapixels... That's 10 GB for an 8-bit RGB buffer and 40 GB for a float32 buffer...
2024-08-14 07:12:04
Isn't the software supposed to break the input file into square chunks as it reads it? Why does the whole thing need to be loaded into memory at the same time?
2024-08-14 07:17:13
I know every jxl file is divided into big chunky squares and that's how it seems to use multiple threads and how it theoretically can do roi decoding
_wb_
Demiurge Isn't the software supposed to break the input file into square chunks as it reads it? Why does the whole thing need to be loaded into memory at the same time?
2024-08-14 07:19:35
It doesn't have to be. Currently cjxl only has streaming input for ppm though. It does have tiled processing and streaming output, but all input formats are still loaded as full buffers except for ppm...
Demiurge
2024-08-14 07:21:01
I see. Well streaming input is still relatively new.
_wb_
2024-08-14 07:21:54
The libjxl API supports it, that is the main thing. In the end, cjxl is just a demo application for libjxl.
Demiurge
2024-08-14 07:22:11
Speaking of, is there a way for a library or application that uses libjxl to enforce memory limits to prevent denial of service? Would be especially important for libvips for example since it's used a lot on web facing servers
_wb_ The libjxl API supports it, that is the main thing. In the end, cjxl is just a demo application for libjxl.
2024-08-14 07:23:29
Yep, true. The library is more important.
_wb_
Demiurge Speaking of, is there a way for a library or application that uses libjxl to enforce memory limits to prevent denial of service? Would be especially important for libvips for example since it's used a lot on web facing servers
2024-08-14 08:14:18
You can do that if you use your own memory manager. Or you can just decode the header, and decide based on image dimensions (and channel count). We designed things so that memory usage for encode and decode is always at most proportional to the number of samples (though the worst-case constant factor can be quite large, if you e.g. go up the allowed limits of splines, patches and modular MA tree sizes for the highest profile, Main Level 10)
Demiurge
2024-08-14 08:19:21
So vips would have to implement their own malloc wrapper that keeps track of how much heap libjxl is using.
2024-08-14 08:20:12
Sounds like a lot of work.
2024-08-14 08:35:18
And if a malloc were to fail, libjxl won't crash or corrupt the heap by doing out of bounds heap access?
_wb_
Demiurge So vips would have to implement their own malloc wrapper that keeps track of how much heap libjxl is using.
2024-08-14 08:58:10
There is example code for that in the fuzzing decoder, it artificially pretends to have limited memory
Demiurge And if a malloc were to fail, libjxl won't crash or corrupt the heap by doing out of bounds heap access?
2024-08-14 08:58:55
It shouldn't, modulo bugs of course. But this is one of the reasons fuzzing is done...
Demiurge
2024-08-14 09:13:42
That's good... some projects are holding off on including libjxl or adding jxl support because they are waiting for certain issues to be closed on github so they know they can use libjxl without it misbehaving if it encounters a corrupt or maliciously-crafted input file or if a malloc fails. If it's getting fuzzed for these things then maybe those issues can be finally closed until a bug is found or unless there are still some unpredictable cases where it will abort
2024-08-14 09:14:33
Once those issues are closed on github, support for the format and adoption of libjxl is expected to grow since a lot of different projects are waiting on that
CrushedAsian255
Demiurge That's good... some projects are holding off on including libjxl or adding jxl support because they are waiting for certain issues to be closed on github so they know they can use libjxl without it misbehaving if it encounters a corrupt or maliciously-crafted input file or if a malloc fails. If it's getting fuzzed for these things then maybe those issues can be finally closed until a bug is found or unless there are still some unpredictable cases where it will abort
2024-08-14 09:39:26
Sounds like things that will happen when 1.0 releases
Demiurge
2024-08-14 10:26:46
They aren't waiting for 1.0, they're waiting for those github issues to be closed
CrushedAsian255
Demiurge They aren't waiting for 1.0, they're waiting for those github issues to be closed
2024-08-14 10:27:46
No, I mean the issues will probably be closed as part of 1.0
Demiurge
2024-08-14 10:31:52
They should be closed when they're fixed and when there's tests/fuzzing in place, regardless of the version number
2024-08-14 10:33:37
I know they are part of 1.0 goals but I don't think they will wait until 1.0 to close the issue, if it's fixed before then
Traneptora
Quackdoc or is this just output?
2024-08-16 12:40:55
that's just the output. what does it say with `--one-frame`?
2024-08-16 12:41:39
hydrium by default makes jxls that are exceptionally slow to decode with libjxl unless you use `--one-frame`
2024-08-16 12:41:45
but that uses way more RAM so it's off by default
Quackdoc
2024-08-16 12:45:33
unfortunately I already deleted the image, but ill DL and run it again once my PC is done updating
Traneptora
2024-08-16 12:49:07
not a big deal, I'm just curious
Quackdoc
2024-08-16 01:52:27
ran time this time too ```ps โžœ test time ~/code/hydrium/build/hydrium --one-frame Combined_100_.png Combined_100_.png.hydrium.jxl libhydrium version 0.4.1 Max libhydrium heap memory: 4576204876 bytes ~/code/hydrium/build/hydrium --one-frame Combined_100_.png 458.20s user 4.83s system 99% cpu 7:45.33 total avg shared (code): 0 KB avg unshared (data/stack): 0 KB total (sum): 0 KB max memory: 4034 MB โžœ test dua 161.27 MiB Combined_100_.png.hydrium.jxl 834.93 MiB Combined_100_.png 996.20 MiB total ```
Traneptora
2024-08-16 12:12:39
nice, now it'll decode with libjxl maybe
2024-08-16 12:12:40
lol
2024-08-16 12:14:26
what happens if you try to decode it with djxl/libjxl?
2024-08-16 12:16:03
also I'm surprised by that 99% system time, I wonder if it's because it performs a lot of allocs and frees in order to prevent hoarding memory
jonnyawsom3
Traneptora nice, now it'll decode with libjxl maybe
2024-08-16 02:27:34
Did anyone actually try without `--one-frame` anyway? I mean it'd either go at a snail's pace or just give up, but I'd be curious to see what happens anyway
Traneptora
Did anyone actually try without `--one-frame` anyway? I mean it'd either go at a snail's pace or just give up, but I'd be curious to see what happens anyway
2024-08-16 02:28:24
I don't know. for massive massive images then `--one-frame` is significantly more memory-hungry than every other mode. the biggest compromise is `--tile-size=3` which is one LF group per frame, rather than one group per frame
Oleksii Matiash
Did anyone actually try without `--one-frame` anyway? I mean it'd either go at a snail's pace or just give up, but I'd be curious to see what happens anyway
2024-08-16 02:48:17
On Windows it silently fails with WerFault.exe called in the background
Traneptora
2024-08-16 03:12:17
well now that's interesting
2024-08-16 03:12:19
``` leo@gauss ~/Downloads :( $ time hydrium --tile-size=3 combined.png combined.jxl libhydrium version 0.4.1 malloc(): unsorted double linked list corrupted Aborted (core dumped) real 0m4.369s user 0m3.561s sys 0m0.353s leo@gauss ~/Downloads :( $ valgrind hydrium --tile-size=3 combined.png combined.jxl ==507761== Memcheck, a memory error detector ==507761== Copyright (C) 2002-2024, and GNU GPL'd, by Julian Seward et al. ==507761== Using Valgrind-3.23.0 and LibVEX; rerun with -h for copyright info ==507761== Command: hydrium --tile-size=3 combined.png combined.jxl ==507761== libhydrium version 0.4.1 ==507761== Warning: set address range perms: large range [0x4fc6040, 0x1f94b040) (undefined) vex amd64->IR: unhandled instruction bytes: 0xC5 0x79 0xD6 0xC0 0xC5 0x7A 0x7E 0x5 0x41 0xEB vex amd64->IR: REX=0 REX.W=0 REX.R=1 REX.X=0 REX.B=0 vex amd64->IR: VEX=1 VEX.L=0 VEX.nVVVV=0x0 ESC=0F vex amd64->IR: PFX.66=1 PFX.F2=0 PFX.F3=0 ==507761== valgrind: Unrecognised instruction at address 0x485ceb3. ```
2024-08-16 03:15:33
this happens even if I set cflags to `-march=x86_64`
2024-08-16 03:17:29
actually I lied, it doesn't
2024-08-16 03:17:33
which means valgrind may not support fma instructions
2024-08-16 03:17:36
which are here `scratchblock[0][y][k] += encoder->xyb[(posy + n) * 3 + c].f * cosine_lut[k - 1][n];`
2024-08-16 03:29:13
yea, setting `-march=x86_64` doesn't stop malloc from crashing though, but `LD_PRELOAD=/usr/lib/libtcmalloc.so` does
Demiurge
2024-08-16 11:44:57
I heard outstanding things about mimalloc
Quackdoc
Demiurge I heard outstanding things about mimalloc
2024-08-17 05:25:37
mimalloc is nice, but I prefer jemalloc, mimalloc has best performance I find, but it's really aggressive and can cause other applications on the PC to slow down if it's being called a lot
Demiurge
2024-08-17 05:26:25
Easy solution... Just make all apps use mimalloc! ^o^
2024-08-17 05:29:12
I heard it's a good replacement for the default minimal malloc in musl.
2024-08-17 05:30:00
The musl malloc is probably the fewest LoC but it chokes when combined with multiple threads.
ChrisX4Ever
2024-08-25 02:52:38
Hello!! Complete nooby here. Can someone tell me how to make my Windows 11 Notebook to Recognize .jxl files? I mean I already downloaded "jxl-x64-windows-static.zip" file but I'm not sure where to extract it.
A homosapien
2024-08-25 03:05:53
https://github.com/saschanaz/jxl-winthumb ^^^This is for windows thumbnails https://github.com/sylikc/jpegview ^^^This is a solid image viewer for jxl files
ChrisX4Ever
2024-08-25 03:44:07
Thanks for the help. What I'm aiming for is for the Windows 11 OS to recognize .jxl files without the need of third-party software (jxl-winthumb worked, I just want to install libjxl directly). But again, thank you so much for the links, if I don't learn how to install libjxl then i'm definetly using JPEGView as an alternative (also jxl-winthumb)
Demiurge
2024-08-25 03:50:51
Major fail that Microsoft doesn't allow third party wic codecs even though they pretend they do
ChrisX4Ever Thanks for the help. What I'm aiming for is for the Windows 11 OS to recognize .jxl files without the need of third-party software (jxl-winthumb worked, I just want to install libjxl directly). But again, thank you so much for the links, if I don't learn how to install libjxl then i'm definetly using JPEGView as an alternative (also jxl-winthumb)
2024-08-25 03:51:54
XL Converter seems cool, or just use scoop
2024-08-25 03:53:54
Oh, nvm about scoop. if you're a nooby, it's probably not your thing. XL converter is a good interface for converting image files
ChrisX4Ever
2024-08-25 03:55:20
So sad, JPEG XL is an absolutely amazing codec. I understand Google not supporting it because of WebP preference since they made it, but why Microsoft?
Demiurge
2024-08-25 03:56:28
https://codepoems.eu/xl-converter/
ChrisX4Ever So sad, JPEG XL is an absolutely amazing codec. I understand Google not supporting it because of WebP preference since they made it, but why Microsoft?
2024-08-25 03:56:51
Don't get the wrong idea, MS doesn't have it out for JXL specifically
2024-08-25 03:57:35
There is evidence they are secretly working on their own jxl codec
2024-08-25 03:59:13
It's just annoying that people aren't allowed to write their own module and have it integrate seamlessly into Windows. The closest thing is jxl-winthumb
CrushedAsian255
ChrisX4Ever So sad, JPEG XL is an absolutely amazing codec. I understand Google not supporting it because of WebP preference since they made it, but why Microsoft?
2024-08-25 04:00:23
with Microsoft, it's more they haven't gotten round to it / haven't finished
Demiurge
ChrisX4Ever So sad, JPEG XL is an absolutely amazing codec. I understand Google not supporting it because of WebP preference since they made it, but why Microsoft?
2024-08-25 04:02:47
Also, it's not google. It's literally just the webm/avif tech lead and his team from On2 Technologies who Google put in charge of the Chrome Codec Team with no oversight
CrushedAsian255
Demiurge Also, it's not google. It's literally just the webm/avif tech lead and his team from On2 Technologies who Google put in charge of the Chrome Codec Team with no oversight
2024-08-25 04:03:34
so literally just the people who are working on the direct competitor
2024-08-25 04:03:39
so massive conflict of interest
Demiurge
2024-08-25 04:04:03
Yeah. It's literally that cartoonishly ludicrous and shameless
ChrisX4Ever
2024-08-25 04:11:03
Man, such impotence. Not sure if I'm correct on saying this but JPEG XL appears to be objectively superior to WebP. After knowing all the different things it can accomplish while being royalty-free I became obsessed with it. That's why I want my Windows 11 machine to fully recognize JPEG XL images thus making it my default image codec.
2024-08-25 04:27:26
I just used scoop and "'libjxl' (0.10.3) was installed successfully!"
2024-08-25 04:28:23
thanks you so much for the info about scoop, might use it more later
2024-08-25 05:11:10
How do I encode multiple images with the same file extension that are in a folder in the main directory? What I mean by that is that I create a new folder (let's say "Images"), then I put all my image files (let's say they are all PNG files) into that folder, and I want to use a cycle (for/while/etc.) that enters Images folder, encode the images one by one and each encoded image must have it's original name (but now instead of .png they are .jxl) and saves said encoded files in Images folder.
2024-08-25 05:12:47
I know it's very specific but if anyone can help me, I would greatly appreciate it. for now I have been using this code in FFMPEG: for %f in (*.png*) do ffmpeg -i "%f" -c:v libjxl -distance 0.0 "%~nf.jxl"
2024-08-25 05:13:25
but not only it doesn't work for libjxl, it also reads images that are on the main folder
CrushedAsian255
2024-08-25 05:13:57
are you unix or windows
ChrisX4Ever
2024-08-25 05:14:03
Windows 11
CrushedAsian255
2024-08-25 05:14:16
sorry idk then
2024-08-25 05:14:19
i use mac
ChrisX4Ever
2024-08-25 05:14:52
No problem. Thanks anyways for the help!!
Meow
2024-08-25 05:41:11
Maybe PowerToys could implement earlier
ChrisX4Ever
2024-08-25 06:17:59
It seems quite versatile, I'll take a look. Thanks!!
Demiurge
2024-08-25 07:25:44
I love scoop. It's not what newbies usually like but it's perfect for simplifying things with the Terminal
ChrisX4Ever How do I encode multiple images with the same file extension that are in a folder in the main directory? What I mean by that is that I create a new folder (let's say "Images"), then I put all my image files (let's say they are all PNG files) into that folder, and I want to use a cycle (for/while/etc.) that enters Images folder, encode the images one by one and each encoded image must have it's original name (but now instead of .png they are .jxl) and saves said encoded files in Images folder.
2024-08-25 07:29:32
https://xl-docs.codepoems.eu/adding-files
2024-08-25 07:38:26
Keep in mind: Blindly batch converting without verification is danger dragon.
JendaLinda
2024-08-25 07:40:58
Even better is verification using three different decoders.
2024-08-25 07:41:26
That actually helped finding and fixing bugs in the decoders.
jonnyawsom3
2024-08-25 09:12:45
I feel like everyone immediately forgot them saying they're a noob, and started throwing around pretty complicated stuff
2024-08-25 09:25:30
To put it more simply <@383447728685842432> Microsoft are working on adding JXL support, but for now we need separate image viewers Since different programs use libjxl in different ways, I download the static zip and then use cjxl after extracting everything to a folder If you want to set it up for long term use, I made a "Jpeg XL" folder in my Program Files, extracted everything there, and then added the folder to PATH (can look up guides online, it's fairly simple) And this is the command I used to use to convert a folder ```for %i in ("FolderPath\*.png") do cjxl "%i" "FolderPath\%~ni.jxl" -d 0``` If you didn't do the PATH thing, just swap cjxl with the path to it instead The others mentioned verifying the files after, but that can always be done after with a batch file instead
Demiurge
To put it more simply <@383447728685842432> Microsoft are working on adding JXL support, but for now we need separate image viewers Since different programs use libjxl in different ways, I download the static zip and then use cjxl after extracting everything to a folder If you want to set it up for long term use, I made a "Jpeg XL" folder in my Program Files, extracted everything there, and then added the folder to PATH (can look up guides online, it's fairly simple) And this is the command I used to use to convert a folder ```for %i in ("FolderPath\*.png") do cjxl "%i" "FolderPath\%~ni.jxl" -d 0``` If you didn't do the PATH thing, just swap cjxl with the path to it instead The others mentioned verifying the files after, but that can always be done after with a batch file instead
2024-08-25 12:47:06
scoop takes care of all that PATH and installation stuff for ya. if you don't use it already I highly recommend you do.
2024-08-25 12:51:11
And if you're doing a batch lossless conversion job, it's possible and highly wise to add a decode/verification step after each encode to ensure the decode perfectly matches the original...
2024-08-25 12:54:22
If you're writing your own batch conversion script you could ask me or someone else here for advice... For PNG/GIF/BMP etc I would probably use vips to compare pixel data.
ChrisX4Ever
2024-08-25 08:37:40
Thanks for all the help guys, really appreciate it. How do I add a decode/verification step for all my encoded files? Never thought of that but seems smart to do so, considering I mostly want 100% lossless encoded files.
Demiurge
ChrisX4Ever Thanks for all the help guys, really appreciate it. How do I add a decode/verification step for all my encoded files? Never thought of that but seems smart to do so, considering I mostly want 100% lossless encoded files.
2024-08-25 08:47:40
If there is a built in way, in one of the encoding tools, idk of any. I think you need to do it yourself, if you are writing a bulk conversion script...
2024-08-25 08:53:36
For jpeg conversion, I would just decode back to jpeg and make sure the two file streams are identical. For regular lossless comparison? I would probably use vips to compare.
ChrisX4Ever
2024-08-25 08:57:46
Thanks!!
Demiurge
2024-08-25 09:35:56
No prob. If you need help writing a script, just post what you have so far and I or someone else may help suggest improvements.
CrushedAsian255
Demiurge For jpeg conversion, I would just decode back to jpeg and make sure the two file streams are identical. For regular lossless comparison? I would probably use vips to compare.
2024-08-25 09:46:59
how do you use vips to compare images automatically?
Demiurge
2024-08-25 10:15:22
Good question... It's not obvious. If you're using the vips command line utility, then I think you need to use 2 separate commands. `vips relational` will compare two images...
2024-08-25 10:16:09
But then you need to use a second command on the output of `relational` to see if a difference is found, like `vips min`
2024-08-25 10:17:05
if you're using the library, then it can be chained together and optimized so that it stops comparing as soon as a difference is found.
_wb_
2024-08-26 06:39:13
You can also use ImageMagick: `compare -metric pae imageA.png imageB.png null:` should return 0
CrushedAsian255
2024-08-26 06:39:39
Is PAE as efficient as possible? (Single O(N) loop)?
2024-08-26 11:51:20
``` magick "$INPUT" /tmp/image.ppm cjxl /tmp/image.ppm "$OUTPUT" -d 0 djxl "$OUTPUT" /tmp/image2.ppm diff /tmp/image.ppm /tmp/image2.ppm ```
2024-08-26 11:51:28
this seems to work for me
2024-08-26 11:53:15
only thing is this removes iCC profiles
_wb_ You can also use ImageMagick: `compare -metric pae imageA.png imageB.png null:` should return 0
2024-08-26 11:53:54
oh, i forgot the `null:` that's why it was not working for me
_wb_
2024-08-26 11:54:34
Actually `compare` also ignores ICC profiles so it's not perfect
CrushedAsian255
_wb_ Actually `compare` also ignores ICC profiles so it's not perfect
2024-08-26 11:55:13
as in converting to `ppm` stripped ICC profiles
_wb_
2024-08-26 11:56:02
Yes, but `compare` just compares sample values and ignores colorspace info too, so it is kind of the same thing
CrushedAsian255
_wb_ Yes, but `compare` just compares sample values and ignores colorspace info too, so it is kind of the same thing
2024-08-26 11:56:55
converting to `ppm` means the resulting JXL does not have colour space info. `compare` ignoring colour space is not important to me
CrushedAsian255 ``` magick "$INPUT" /tmp/image.ppm cjxl /tmp/image.ppm "$OUTPUT" -d 0 djxl "$OUTPUT" /tmp/image2.ppm diff /tmp/image.ppm /tmp/image2.ppm ```
2024-08-26 11:57:22
the code i gave above converted any input image to `.ppm` which doesn't allow the JXL to have colourspace
RaveSteel
CrushedAsian255 ``` magick "$INPUT" /tmp/image.ppm cjxl /tmp/image.ppm "$OUTPUT" -d 0 djxl "$OUTPUT" /tmp/image2.ppm diff /tmp/image.ppm /tmp/image2.ppm ```
2024-08-26 12:38:45
imagemagick's identify may be better than diff since it only hashes the image data itself
2024-08-26 12:39:45
`identify -format "%#\n" INPUT(s)`
CrushedAsian255
RaveSteel imagemagick's identify may be better than diff since it only hashes the image data itself
2024-08-26 12:42:34
if it's PPM, the only metadata other than image data is the bitdepth and dimensions, so in this instance it's fine
2024-08-26 12:42:44
i would agree in most other situations
Foxtrot
2024-08-26 02:42:35
If I remember correctly libjxl uses abort which is bad for uses by other software and there is plan for fixing this. How far is this goal?
_wb_
2024-08-26 05:18:34
I think it is done, in current git version
Demiurge
2024-08-27 12:03:40
I don't think converting to PPM is a good or efficient idea.
CrushedAsian255
Demiurge I don't think converting to PPM is a good or efficient idea.
2024-08-27 04:27:41
no, it's not
2024-08-27 04:27:47
that's why im thinking of other solutions
2024-08-27 04:27:55
`convert -metric pae` is probably hat ill use
DZgas ะ–
2024-08-29 12:20:57
Why can not **-I 10000** <:Stonks:806137886726553651>
monad
2024-08-29 12:44:33
presumably, there are only ever 100% pixels
HCrikki
2024-08-29 02:08:12
quick question, are there routines for basic resizing using cjxl ?
monad
2024-08-29 02:14:09
you can upsample
2024-08-29 02:21:17
requires `--resampling` with `--already_downsampled`
RaveSteel
HCrikki quick question, are there routines for basic resizing using cjxl ?
2024-08-29 02:21:23
Better use ImageMagick
jonnyawsom3
monad requires `--resampling` with `--already_downsampled`
2024-08-29 02:33:51
Then there's also --upsampling_mode=0 for nearest neighbour, or something like that
HCrikki
2024-08-29 02:35:01
using imagemagick does not cover apps and scripts integrating cjxl
_wb_
2024-08-29 02:37:18
Thanks for the pull request, <@263300458888691714> ! Now e11 is still very slow but an order of magnitude faster, and better. We'll have to be careful, at some point it might actually become a useful effort setting ๐Ÿ™‚
monad
2024-08-29 02:59:25
I was relying on Luca to catch my mistakes, but he missed something.
2024-08-29 04:08:21
Significant density boost when you translate settings correctly. Anyway, yeah, it's still bad, but all the fanatics out there should still be satisfied with this version.
DZgas ะ–
monad presumably, there are only ever 100% pixels
2024-08-29 04:28:43
Iterations, not %
monad
2024-08-29 04:29:37
but it's comparing to some percentage of pixels, represented by 0-100
DZgas ะ–
2024-08-29 04:42:34
<:Thonk:805904896879493180> uhh
jonnyawsom3
2024-08-29 05:03:34
`-I` is the percentage of pixels used for MA tree learning, with 0 disabling it and 100 using the entire group
veluca
monad I was relying on Luca to catch my mistakes, but he missed something.
2024-08-29 07:56:12
never do that, it's a mistake ๐Ÿ˜›
2024-08-29 08:12:15
thanks for the PR ๐Ÿ™‚ it's a very nice improvement
monad
2024-08-29 08:31:34
thanks for taking my memey article seriously
Just me
2024-08-31 09:19:45
Pedantic. 2 keywords are too general right now. Just remove https://github.com/topics/lossy-compression and https://github.com/topics/lossless-compression-algorithm from https://github.com/libjxl/libjxl. The repository has 2 -image-compression keywords and even image-compression. JPEG XL can't compress arbitrary binary files easily...
jonnyawsom3
2024-08-31 09:24:38
You already said that exact message before
CrushedAsian255
You already said that exact message before
2024-08-31 09:31:11
fab 2
Just me Pedantic. 2 keywords are too general right now. Just remove https://github.com/topics/lossy-compression and https://github.com/topics/lossless-compression-algorithm from https://github.com/libjxl/libjxl. The repository has 2 -image-compression keywords and even image-compression. JPEG XL can't compress arbitrary binary files easily...
2024-08-31 09:31:38
Me when I chuck a PPM header on arbitrary data
_wb_
2024-08-31 09:40:18
Lossy compression is not really a thing for arbitrary binary files ๐Ÿ™‚
CrushedAsian255
_wb_ Lossy compression is not really a thing for arbitrary binary files ๐Ÿ™‚
2024-08-31 09:41:50
I tried to force FFmpeg to encode a text file as MP3, you could imagine how well it went
RaveSteel
CrushedAsian255 I tried to force FFmpeg to encode a text file as MP3, you could imagine how well it went
2024-08-31 09:43:11
Of course you have to use aplay first to get a PCM stream lel
Eugene Vert
2024-08-31 06:19:23
https://libjxl.readthedocs.io/en/latest/api_metadata.html#_CPPv4N12JxlLayerInfo17save_as_referenceE says that `save_as_reference = 3` "should not be used by applications", but it seems to be reserved only for patches (`kPatchFrameReferenceId = 3`). Can I use `save_as_reference = 3` in case of disabled patches?
2024-08-31 06:50:15
Oh, there is a tight check in `lib/jxl/encode.cc` for `save_as_reference >= 3`, but it seems to work fine without it
jonnyawsom3
Eugene Vert https://libjxl.readthedocs.io/en/latest/api_metadata.html#_CPPv4N12JxlLayerInfo17save_as_referenceE says that `save_as_reference = 3` "should not be used by applications", but it seems to be reserved only for patches (`kPatchFrameReferenceId = 3`). Can I use `save_as_reference = 3` in case of disabled patches?
2024-08-31 09:21:03
I'd assume no in case decoders tried to interpret it as a patch, but on the other hand if there are no patch references I suppose nothing would happen
CrushedAsian255
2024-09-01 01:33:55
Is this a convention thing or a spec rule?
Eugene Vert
2024-09-01 02:05:11
I think conventional one, as each patch have its own reference frame (`ref`)
CrushedAsian255
2024-09-02 10:42:18
What is this bug report?
2024-09-02 10:42:20
https://github.com/libjxl/libjxl/issues/3764
RaveSteel
CrushedAsian255 What is this bug report?
2024-09-02 10:45:15
One that should be closed
CrushedAsian255
2024-09-02 10:47:02
I donโ€™t get it
2024-09-02 10:47:08
It doesnโ€™t look filled in
2024-09-02 10:47:14
Like they forgot to put the bug
RaveSteel
2024-09-02 10:53:04
That seems to be the case, yes
CrushedAsian255
2024-09-02 10:53:44
So itโ€™s duplicate / spam then ?
RaveSteel
2024-09-02 10:58:44
Maybe the guy wonders why no one has replied to his bug report yet lol
CrushedAsian255
2024-09-02 10:59:29
don't you know that JPEG XL gives you telekenetic powers?
2024-09-02 10:59:35
i thought that was common knowledge
yoochan
2024-09-02 10:59:41
The profile looks like a spam bot
CrushedAsian255
2024-09-02 11:00:35
i guess, that's the only thing they have done on that account
spider-mario
2024-09-02 03:01:59
stack plot from `git-of-theseus` (https://github.com/erikbern/git-of-theseus) on libjxl
monad
2024-09-02 03:25:13
So, since every year trends smaller, eventually there will be no code at all?
jonnyawsom3
2024-09-02 03:52:20
The overall total amount of code has nearly doubled, but every year more old code is replaced by newer
_wb_
2024-09-02 04:44:09
this is for authors
2024-09-02 04:45:34
but a lot of this is "jpegxl-bot" since initially we did the development on a private gitlab repo and the github repo got synced by a bot
2024-09-02 04:47:36
these are the authors when I run it on that private gitlab repo (which was used in the first 2.5 years or so)
lonjil
2024-09-02 05:14:25
shouldn't syncing to another repo retain commit authorship information?
_wb_
2024-09-02 05:28:34
That can be done I guess, but back on gitlab the CI was configured so that the bot would always do the merging, and somehow that's the only thing that survived into the github repo
2024-09-02 05:29:50
Alex Deymo is overrepresented in this gitlab plot since he did the initial PIK code dump commit
2024-09-02 05:30:26
You can see how small FUIF was compared to PIK in lines of code ๐Ÿ™‚
spider-mario
2024-09-02 05:48:31
I guess it should be possible to reconstruct a git history that has both (and maybe even pik)
2024-09-02 05:48:48
by reparenting the first github revision on the last gitlab one, or something along those lines
jonnyawsom3
2024-09-02 06:10:54
Seems like a researcher is trying to compress 10-bit video frames with JXL, but is having issues getting it to convert properly https://github.com/libjxl/libjxl/issues/3787
2024-09-02 06:11:22
Wasn't there Y4M support at some point?
Oleksii Matiash
_wb_ this is for authors
2024-09-02 06:27:51
Just curious, why Eugene has two names ๐Ÿ™‚
_wb_
2024-09-02 06:33:39
E. Upenik is not really a jxl contributor, but he was Touradj' PhD student when that gitlab repo was created and he was the one who created it ๐Ÿ™‚
2024-09-02 06:34:20
I dunno why <@811568887577444363> has his first name spelled in two different ways ๐Ÿ™‚
Oleksii Matiash
2024-09-02 06:35:46
I have an idea, but it depends on where he is from
TheBigBadBoy - ๐™ธ๐š›
spider-mario stack plot from `git-of-theseus` (https://github.com/erikbern/git-of-theseus) on libjxl
2024-09-02 08:41:54
I guess the massive drop is due to jpegli getting its own repo (and being removed from libjxl) ?
spider-mario
2024-09-02 08:45:04
I think jpegli is more recent than the massive drop
2024-09-02 08:45:13
also, Iโ€™m in the process of trying to rerun this on the entire history
2024-09-02 08:45:28
although the pik part will be imperfect because that, in turn, was exported from internal code
2024-09-02 08:45:56
e.g. https://github.com/google/pik/commit/6fa4a5b96131d5b251d588535452b1cfb377d8e3
2024-09-02 08:47:02
and with a mailmap to have only one Eugene (and consolidate a few other people as well)
jonnyawsom3
TheBigBadBoy - ๐™ธ๐š› I guess the massive drop is due to jpegli getting its own repo (and being removed from libjxl) ?
2024-09-02 08:49:08
It's also not been removed from libjxl, just mirrored and then continued
spider-mario
2024-09-02 08:50:26
``` Rebasing (60/2646) ```
2024-09-02 09:05:34
there we go (but as I said, likely distorted by how we exported pik)
CrushedAsian255
2024-09-02 09:06:17
Wait, what parts of JXL came from FUIF/FLIF and Pik
spider-mario
2024-09-02 09:06:47
Modular is from FUIF
2024-09-02 09:07:13
VarDCT is a mixture of PIK and some bits from FUIF
2024-09-02 09:07:23
<@794205442175402004> does that sound about right?
2024-09-02 09:07:57
without authors
_wb_
spider-mario <@794205442175402004> does that sound about right?
2024-09-02 09:10:15
Yes, but that's just the origin, JXL has had ~2 years of new ideas/improvements added on top of the pik/fuif blend before we decided to freeze the bitstream.
2024-09-02 09:16:26
I wonder how well this kind of analysis works when a commit is a big refactor that moves around a lot of code (so lots of lines added and removed if you look at the diff)...
spider-mario
2024-09-02 09:18:28
yeah, probably not so wellย โ€“ it tries to use `git blame`โ€™s line tracking but thatโ€™s easily tripped up
2024-09-02 09:18:47
I can retry with `--ignore-whitespace`, maybe that will help somewhat
2024-09-02 09:20:00
the difference is minor
_wb_
2024-09-02 11:58:02
It's a nice chart, really shows that it was a group effort and many people made substantial contributions. Jan, Lode and Deymo are no longer working on jxl but they did quite a lot in the early years and still a lot of their code is in there. Lines of code is of course not a perfect metric, but still, it's a nice diagram.
TheBigBadBoy - ๐™ธ๐š›
2024-09-03 09:26:32
for a 5000x5000 image, and a 16 hyperthreads/cores CPU, how long can I expect `-e 11` to take ? I need to know an order of magnitude, like is it ~10 hours, ~3 days, ~1 week ?
_wb_
2024-09-03 09:36:45
just don't do it for anything that's not a tiny icon, it's a waste of energy in any scenario
TheBigBadBoy - ๐™ธ๐š›
2024-09-03 09:42:13
yeah but i'm actually curious how much storage I would save <:KekDog:805390049033191445>
CrushedAsian255
TheBigBadBoy - ๐™ธ๐š› yeah but i'm actually curious how much storage I would save <:KekDog:805390049033191445>
2024-09-03 09:45:33
like 4 kB
2024-09-03 09:45:36
at best
Fox Wizard
TheBigBadBoy - ๐™ธ๐š› for a 5000x5000 image, and a 16 hyperthreads/cores CPU, how long can I expect `-e 11` to take ? I need to know an order of magnitude, like is it ~10 hours, ~3 days, ~1 week ?
2024-09-03 09:46:12
1 way to find outโ„ข๏ธ
2024-09-03 09:47:07
Might try it on a 1080p background, but it'll take a while, because 9900K
CrushedAsian255
2024-09-03 09:47:25
https://tryitands.ee
Fox Wizard
2024-09-03 09:52:17
-e 9 -q 100 -I 100 -E 3 -g 3 was 0.048MP/s
2024-09-03 09:52:38
-e 10 -q 100 -I 100 -E 3 -g 3 was 0.016MP/s
2024-09-03 09:52:50
I fear for how slow -e 11 will be XD
2024-09-03 09:52:58
``JPEG XL encoder v0.11.0 960d97c [AVX2,SSE2]``
CrushedAsian255
2024-09-03 09:59:35
i feel bad for your pc
Fox Wizard
2024-09-03 10:00:02
It has done 100% load encodes that took over a month <:KekDog:884736660376535040>
CrushedAsian255
2024-09-03 10:00:29
av1?
Fox Wizard
2024-09-03 10:00:38
At this point I kinda want my 9900K to die, because then I have an excuse to buy a 9950X XD
CrushedAsian255 av1?
2024-09-03 10:01:26
Yup, cpu0 qpsweep 1
2024-09-03 10:01:50
So basically slowest. qpsweep makes it I think about 3 times as slow with tiny benefits XD
2024-09-03 10:02:51
And that was a while ago, so no doubt that I can get better results now with cpu1 without qpsweep which I guess will be 5 - 10 times faster <:KekDog:884736660376535040>
2024-09-03 10:04:32
-e 11 -q 100 was 0.003MP/s
RaveSteel
Fox Wizard At this point I kinda want my 9900K to die, because then I have an excuse to buy a 9950X XD
2024-09-03 10:07:00
Why not threadripper ๐Ÿค”
Fox Wizard
2024-09-03 10:07:12
Because rip gaming performanceโ„ข๏ธ
CrushedAsian255
Fox Wizard -e 11 -q 100 was 0.003MP/s
2024-09-03 10:07:52
yay, you saved 25 kB
RaveSteel
Fox Wizard Because rip gaming performanceโ„ข๏ธ
2024-09-03 10:07:59
Build a dedicated encoding box with 128 core threadripper [LUL~3](https://cdn.discordapp.com/emojis/1246890433041137735.webp?size=128&quality=lossless&name=LUL%7E3)
Fox Wizard
2024-09-03 10:08:55
Or even better, get your own server farm with thousands of cores <:KekDog:884736660376535040>
RaveSteel
2024-09-03 10:09:27
For that we'll then need -e 13 or more xd
Fox Wizard
2024-09-03 10:09:46
Don't give the devs ideas <:RaysShock:686219918030798921>
CrushedAsian255
2024-09-03 10:10:37
im still adament about my "try every possible bitstream" idea to be added
RaveSteel
2024-09-03 10:11:13
Maybe in a hundred years when the hardware makes that possible
CrushedAsian255
2024-09-03 10:11:37
would be massively parallelizable
Tirr
2024-09-03 10:15:23
try every possible bitstreams simultaneously with quantum computing
CrushedAsian255
2024-09-03 10:15:51
quantum bogo-cjxl
2024-09-03 10:15:58
destroy all universes with non-valid bitstreams
jonnyawsom3
TheBigBadBoy - ๐™ธ๐š› for a 5000x5000 image, and a 16 hyperthreads/cores CPU, how long can I expect `-e 11` to take ? I need to know an order of magnitude, like is it ~10 hours, ~3 days, ~1 week ?
2024-09-03 12:05:14
I tried to graph the correlation of Megapixels to seconds spent in effort 11 to estimate what large files would be like... Turns out I can't graph to save my life :P
RaveSteel Build a dedicated encoding box with 128 core threadripper [LUL~3](https://cdn.discordapp.com/emojis/1246890433041137735.webp?size=128&quality=lossless&name=LUL%7E3)
2024-09-03 12:08:09
Over the weekend my friend casually mentioned running a VM for me on 3 gigabit fibre so I can run JXL encoding on an old Google server haha
RaveSteel
Over the weekend my friend casually mentioned running a VM for me on 3 gigabit fibre so I can run JXL encoding on an old Google server haha
2024-09-03 12:08:35
Letsgo ๐Ÿ˜‚
monad
TheBigBadBoy - ๐™ธ๐š› yeah but i'm actually curious how much storage I would save <:KekDog:805390049033191445>
2024-09-03 04:08:58
none because you'll never finish encoding all your files
TheBigBadBoy - ๐™ธ๐š›
2024-09-03 04:11:55
on *one* file
monad
2024-09-03 04:12:47
anyway, intuition says it should take less than 10 hours, but for some edge cases libjxl freaks out and take like 200x as long as expected
2024-09-03 04:13:53
it should be photo-like to avoid that
Jyrki Alakuijala
spider-mario VarDCT is a mixture of PIK and some bits from FUIF
2024-09-04 04:04:04
The VarDCT itself is from Pik, but some of the control fields -- like which transform it is or what is the adaptive quantization level are then compressed by using FUIF-inspired (but re-engineered by Luca to be faster in decoding) decompression
2024-09-04 04:04:18
except Pik only had 8x8
2024-09-04 04:05:04
at least at the time of the initial JPEG XL competition
2024-09-04 04:05:23
we only added variable sizes after the competition
2024-09-04 04:05:57
I designed it (the decomposition to a uniform 8x8 field for previews but still having variable DCT sizes) and Luca did the actual implementation
2024-09-04 04:06:12
here, design was easy and implementation difficult ๐Ÿ˜…
yoochan
2024-09-04 04:30:37
If pik did 8x8 dct what were the difference with jpeg?
_wb_
2024-09-04 04:55:08
XYB, adaptive quantization, better DCT coeff encoding (num_zeroes based, with flexible coeff reordering), ANS, CfL, Gaborish are some of the things I can think of
Demiurge
2024-09-04 10:00:29
Pik seems pretty cool. Much cleaner more organized source tree compared to libjxl, but that's not saying much, since almost anything is...
2024-09-04 10:56:31
And it's basically just lossy 8x8 DCT jxl, optimized for high fidelity, and probably super fast too.
jonnyawsom3
2024-09-04 11:47:41
I guess with such small headers, it's much easier to make a 'valid' file by random https://github.com/libjxl/libjxl/issues/3793
2024-09-04 11:47:45
```JPEG XL image, 848302208x293, (possibly) lossless, 8-bit RGB num_color_channels: 3 num_extra_channels: 0 have_preview: 0 have_animation: 0 Intrinsic dimensions: 848302208x293 Orientation: 1 (Normal) Color space: XYB, D65, gamma(0.333333) transfer function, rendering intent: Perceptual layer: 136x9 at position (64,16) Decoder error```
2024-09-04 11:49:07
250 Gigapixels, gee I wonder why it uses 4 GB of memory just to try and load
Demiurge
2024-09-05 04:30:56
This is why it's nice to have convenient knobs for memory and response time for a codec library... ๐Ÿ˜…
Tirr
2024-09-05 04:33:41
width of 848302208 doesn't conform to Level 10 right? maybe libjxl is checking only frame dimensions
jonnyawsom3
2024-09-05 04:39:53
`1073741824` Max Width and `1099511627776` Max Pixels if I'm reading this right. So it's within spec by 203 million Width still
2024-09-05 04:42:16
Even Level 5 allows for 258 Megapixel images
Tirr
2024-09-05 04:43:47
oh so it allows up to 1 terapixels
2024-09-05 04:44:39
(in base 2)
jonnyawsom3
2024-09-05 04:45:38
Just like the advertising ~~barely~~
CrushedAsian255
2024-09-05 04:46:07
technically it should be able to go to 1 exapixel?
2024-09-05 04:46:09
or is there a reason why not
2024-09-05 04:46:16
(other than practical limitations)
jonnyawsom3
2024-09-05 04:46:26
Based on what?
Tirr
2024-09-05 04:46:27
then it's not like something libjxl can prevent, and we need some image dimension limit or something
CrushedAsian255
Based on what?
2024-09-05 04:46:50
1 Billion width * 1 Billion height?
Tirr
2024-09-05 04:47:15
width * height is limited to `1 << 40`
CrushedAsian255
2024-09-05 04:47:33
is that a level-based limit or a fundamental limit in the bitstream?
Tirr
2024-09-05 04:47:42
level 10 limit
CrushedAsian255
2024-09-05 04:47:56
im saying if someone made a 'level 20' or something
2024-09-05 04:47:56
maybe a "web-safe" level 1 could be created specifically for browsers?
jonnyawsom3
2024-09-05 04:48:14
I remember something about encoding the earth with 1 metre resolution in 1.5 JXL files by bending the spec slightly, so that was probably it
CrushedAsian255
2024-09-05 04:48:16
disabling features that would not make any sense on the web?
Tirr
2024-09-05 04:48:21
level 5 is meant to be web-safe I think
CrushedAsian255
2024-09-05 04:48:40
not sure what web app would need / want 256k by 256k images
2024-09-05 04:49:48
i guess other formats don't have arbitrary limitations
2024-09-05 04:49:50
so it's probably fine
2024-09-05 04:50:04
you can always make a 2 gigapixel png
jonnyawsom3
2024-09-05 04:58:20
Yeah, it is a bit of an odd case... On one hand doing nothing would mean annoying the people reporting it, on the other hand nothing is technically wrong. You requested 250 Gigapixels and it failed to decode :P
2024-09-05 04:58:48
Oh... It's 6am... Maybe I should stop thinking about <:JXL:805850130203934781> for a while
novachromatic
2024-09-07 07:54:21
hiya i keep getting an [error when trying to build](https://github.com/libjxl/libjxl/discussions/3807) on windows 11 with visual studio 2022 (something to do with the `RunRow` function?)
2024-09-07 07:54:27
could anybody help? (tho imma sleep now hehe)
2024-09-08 06:22:31
after researching a bit more, is building with msys easier? i barely have experience compiling stuff and it seems libjxl is one of the harder ones
2024-09-08 06:28:58
oh nice! yeah i'm not compiling for development, just want to try out a new feature
2024-09-08 07:30:54
awesome, i wanted to try out [309b637](https://github.com/libjxl/libjxl/commit/309b637d8a2a78708a2b806d9523ae0de562a997), and i've successfully piped imagemagick output into cjpegli
2024-09-08 07:30:59
the only caveat being that i have to use cmd
2024-09-08 07:31:05
for some reason it doesn't work in powershell: ```powershell โฏ magick "cover (max).png" -colorspace RGB -distort resize 640 -colorspace sRGB -strip - | cjpegli - "cover.jpg" Failed to decode input image - ```
2024-09-08 07:31:08
i might report this as a bug if there isn't anything obviously wrong i'm doing
Demiurge
2024-09-11 12:32:28
If it works in one shell and not the other then it's probably not a bug
Traneptora
novachromatic after researching a bit more, is building with msys easier? i barely have experience compiling stuff and it seems libjxl is one of the harder ones
2024-09-12 07:31:29
I find building with msys to be easier
Demiurge
2024-09-12 07:46:47
The build system for libjxl is kinda stiff and obtuse in my experience.