JPEG XL

sorting out and cleaning up the public gitlab issues a bit, adding some labels

Is there any example on how to use libjxl to decode progressively?

not yet, the closest probably is the DCTest under decode_test.cc. Currently getting the DC image, 8x8 downsampled, is the only form of progressive, more will be added later

I tried JXL_DEC_DC_IMAGE, however it seems get black image only

Is there anything I could have done wrong?

It could be an image without DC. Only VarDCT mode has DC. In Modular mode there is only DC if Squeeze is done (not by default)

I thought the image has DC though. I got both `JXL_DEC_NEED_DC_OUT_BUFFER` and `JXL_DEC_DC_IMAGE` event.

Here is my code and my test image. Basically just add some lines to `decode_oneshot.cc` https://gist.github.com/yllan/0da653b9898090bb5173bbd32eb15201#file-decode_oneshot-cc-L121-L146

<@768090355546587137> can you check?

the image does indeed have DC

When are we pushing the 0.3.1 release? When all open bugs are fixed?

I think we could push it now, I'll ask

Maybe would be good to get this one fixed first: https://gitlab.com/wg1/jpeg-xl/-/issues/135

Yeah, let's wait a bit, don't rush things, 0.3.1 deserves being stable

Zero dot odd dot odd sounds like the opposite of a stable version to me but yes, as stable as possible 😅

Hmm, interesting.

It seems like the QT JPEG-XL plugin isn't working for displaying thumbnails anymore. 🤔

Quite strange.

Let's reinstall it.

<@794205442175402004> Bitcoin Core just had its latest stable release: 0.21.0. And for some reason we've referring to our version numbers as 0.MAJOR.MINOR even. For the next release we're finally going to drop the redundant 0 up front, but that was pretty controversial among contributors even. 0.3.1 isn't too bad ;)

Or when 7zip changed the version number simply to the year of release (no matter how much change there was between versions)

Yeah, time based releases are popular these days. Ubuntu has those too, and Firefox, and Chrome.

For browsers or OS it's fine (because there are always so many changes), but when a small application has corrected a few typos in the documentation and the version number changes significantly (because several years have passed), I'm not a fan of that

For a library, the convention is as in https://semver.org/

First number changes if you break the api

Second number changes if you add stuff in a bw-compatible way

Third number is just bugfixes and minor stuff

Yeah, that's a pretty common one too.

That's more useful than a time based version number, so applications that use the library know what to expect and how to define its dependencies

Yes, absolutely for libraries.

You need to add `image/jxl` to thumbnails.desktop

And update mime database

Thank you. It looks like a KDE stuff update removed the .avif and .jxl stuff. 👍

How many people use KDE in this server?

I recently started trying it again, after 12 years of xmonad.

I use KDE Plasma on my laptop. No DE on my desktop but use a lot of KDE apps.

And my reason is rather silly. I wanted to use my Bluetooth headphones with my desktop, and was too lazy to figure out how to do it with command-line tools (or other configuration that worked without actual desktop environment). In Plasma it just worked...

My reason for still using a full DE pretty much

Question: what kind of stuff do I need to compile JXL on Windows? I'm having to use Windows for school reasons but I'd like to be able to still use JXL. But not used to doing dev on this platform.

https://gitlab.com/wg1/jpeg-xl/-/blob/master/doc/developing_in_windows.md

Can't help you after that, I honestly don't know how to use MSVC 😅

I try to stay as far away from Windows as possible. Successfully, for the past 23 years.

Achieving everyone's dream

I stopped using it as my main OS somewhere around april 2000, I believe. I've used it occasionally as gaming-only OS since, though.

I only achieve about 90% away from Windows

I'm late to the party, November 23rd, 2019 is when I re partitioned my hard drive

It's never too late to come to the Light side.

(Use the source, Luke!)

Install Microsoft's Visual Studio C compiler thingy. Install Microsoft's package manager. Do "install". "Error, file not found." Amazing. Such quality. Wow.

After Windows 95, I briefly used Windows NT 4 before doing the dual boot thing in 1997 and permanently switching to only GNU/Linux in 1998.

I made it as far as Windows ME. That was enough.

Though, I think it's unfair to compare our Windoss experiences in the late 90s with what current Windows platforms are like (in either direction) - I really have no clue.

Microsoft is also a very different company now.

Windows 10 is pretty good, they're trying to shore up their weaknesses

powershell, new terminal, WSL2, winget, Powertoys 10, chromium edge

I dual booted Windows and Linux since I was a small child. Then at some point it stopped working and I was stuck on Windows. Then a bit older and a second family PC ran Ubuntu. At some point dad gave me his old computer, and then I had two, one with Windows and one Linux. During all this I also had a MacBook. Eventually in 2017 I deleted Windows and installed Linux on my main PC because Windows was doing a "bluescreens every day only fix is full reinstall" thing. Then in 2018 I installed Linux on my MacBook because Apple was doing a "make macos unusable" thing.

So I have basically as much experience as anyone with many versions of macos, windows, and linux.

So far, for my use cases anyway, Linux is the closest to "just works".

Drivers pre-installed in the kernel. Tooling made by the same people who have to use it everyday (= actually works).

Anyway, guess I'll just go for some pre-compiled binaries or something.

Does anyone have an up to date link?

libopus 1.3.1 is odd dot odd dot odd, yet it's been stable for almost 2 years 😜

I build it successfully with msys2

something along these lines should do: ``` pacman -S mingw-w64-x86_64-{cmake,ninja,clang} # from build dir in MinGW64 shell: cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -G Ninja $jxl_source_dir ninja ```

after 4 years on Ubuntu followed by 10 years on Arch Linux, I actually made Windows my main OS again recently

it’s not as painful as it used to be

Literally everything changed A LOT since Win95

It's a completely different OS

true, though Windows 8 was still very painful

but admittedly, Windows 7, not as much

Oh god, those tiles, Win10 is actually usable, but still Win8 was step in the right direction in terms of flat design

I think I rate Windows 8 below Vista

I hate those cringy shadows and flares from Win7

visually, I kind of liked Aero from Vista/7

I like it only for nostalgia, but as a daily driver? No way

Things have to look flat

nah, I don’t really like flat

Flat is still hip

I also loved Apple’s Aqua up to OS X 10.5 (inclusive)

if that tells you anything about my tastes 😄

I prefer apple 10.16 lol

after that is seems they want to make MacOS IOS

UI fashion is weird

Always alternates between flat, 3D, and glossy/semitransparent

Windows Classic is the best 🙂

I liked the aesthetics of GEM

Also this

GEM and Amiga were nice GUIs

It went downhill afterwards

I love total minimal, i simply felt so more relaxed

Anyway, back to the channel topic

I'm ready for 0.3.1

http://toastytech.com/guis/ is a site I love, and so is http://www.therestartpage.com/

it feeds my nostalgia

No more crucial fixes?

Can do a 0.3.2 if we forgot something, right?

OK 🙂

XP = Trash XP SP1 = Good XP SP2 or 3 = <:Chrisgasm:277833264331358209> Vista = Good ideas, bad execution, very bad luck 7 = Exact same good ideas, great execution, good luck 8 = Complete garbage ideas, shit execution 10 = Same garbage ideas, but dialed back and with good execution

Windows 8 is the only OS I rate worse than OSX/MacOS

So what I learned today is that there's pretty much no concensus

I like Zorin 🙂 It's Ubuntu pretending to be Windows 7. And I like the *idea* of ReactOS, though I don't know if it will ever become usable in a more practical sense. But it's interesting

It's only been in Alpha for the past 30 years, at this rate we'll see a 1.0 release in time for windows 10 to become public domain

open source meh > closed source public domain, because one can keep being improved upon. Also ReactOS just did a big release recently

dev repo is at 0.3.1, I suppose sync to public repo will happen soon now

Oh, interesting.

I already found a bug with the QT-JPEG-XL plugin.

Fun.

Now, time to try and solve it.

<:FeelsReadingMan:808827102278451241>

<https://github.com/GoogleChromeLabs/squoosh/commit/029c2ebb83651fbf854fa7fd689a571a0338394e> `Bump jxl version and remove workaround for enc memory usage.` <:Hypers:808826266060193874>

Noice

https://tenor.com/view/noice-nice-click-gif-8843762

thanks 😛

https://github.com/phoronix-test-suite/test-profiles/issues/186

https://openbenchmarking.org/test/pts/jpegxl

Perhaps it would be helpful to add lossless modes to the test profiles as well or any adjustments for speeds and options? Because openbenchmarking can have a lot of data over time with a wide variety of hardware

(but it's up to the devs)

oh, it's testing lossless jpeg recompression

that explains why it's so fast...

> `but any improvements are welcome either through GitHub merge requests or simple patches` Oh, that's good, it might also be worth adding another more complex test image(s) if such a thing is allowed, as well as the most necessary speeds for each mode (I would be interested in how the CPUs differ for these modes, including the slowest)

I've updated to 3.0.1 and through trial and error applied https://github.com/tbonfort/gdal/commit/119e1771d21f2bb476eab2e2fef724f1b35e654b . on my single channel 16 bit lossless testcase , the output size went from 133M to 54M , and the result is lossy. Is there any doc about how to use libjxl for this use case, or should I just wait for it to stabilise ?

I think there are explicit flags to set for lossless encoding but I don't really know

well, the lossless compression worked correctly with that code with the previous version of libjxl

Can you share the image?

yes sure, its public : https://storage.cloud.google.com/gcp-public-data-sentinel-2/tiles/31/T/DH/S2A_MSIL1C_20210211T104151_N0209_R008_T31TDH_20210211T125423.SAFE/GRANULE/L1C_T31TDH_A029463_20210211T104437/IMG_DATA/T31TDH_20210211T104151_B08.jp2

Could you perhaps file a bug?

it might be that this is perhaps totally intended behaviour, but at least it would allow me to point the correct person at it 🙂

sure if it's a bug, I'm just wondering if the exposed api in libjxl is just not ready yet for this use-case

I don't know, but no harm in adding an issue 🙂

I've been able to losslessly encode 1 channel and 4 channel images, but documentation always refers to srgb, which my test cases was definitely not

I'll open an issue asap

<https://github.com/phoronix-test-suite/test-profiles/pull/187> Have the lossless encoding/decoding modes been added here? I think that would be more important than JPEG recompression

lossy from pixel is the most important in practice imo

for the web at least

lossless enc/dec is interesting too

lossy from pixel encoding has been added in the 1.1.0 one it seems

he'll add decoding he said, I hope both lossy and lossless

lossless encoding is maybe a bit early to add there, I think we can still improve the speed/density tradeoffs

Yep, lossless speed would also be interesting (as it can be noticeably different from lossy) and for the web as well (and server CPUs)

For web, I think you mostly want lossless for small images like UI elements, emojis, logos etc. Maybe for some infographics too, those can get larger (but maybe good lossy is also ok for those, as long as the text does not get dct artifacts)

Big image lossless is mostly for authoring workflows imo, like when saving/loading an image you're photoshopping

So also relevant, but not so much for the web

Mostly yes, but for example my test set contains popular PNG images, some of which are viewed millions of times, so even for large images on the Web, especially Art/Manga/Comix and such, this is not uncommon

Right, but I guess that's mostly because currently PNG is the only universally available codec that is useful for such images (lossless WebP can start replacing it though). JPEG will just give either worse compression than PNG or ugly artifacts.

With `-d 1` lossy jxl encoding, and maybe patches for the text and other stuff for which lossless works better – patches is effectively the lossless codec, but only for a smaller image (the patch spritesheet), and used with quantized colors (you don't need 256 shades of gray to render black on white text nicely), so that should be cheaper than full-image full-lossless encode/decode

it may be good enough for the web

Lossy JXL still has problems with sharp lines and ringing in Art or b&w manga images (but AVIF and the like are good at that), maybe in the future this will improve noticeably, but I don't think that many sites that currently have PNG images like various boards, reddit, online manga/comics/art will convert this to lossy formats (especially if the lossless JXL can also noticeably reduce size by 30-70% without fear of possible artifacts)

They usually have lossless full images and lossy previews (btw, reddit has also started replacing previews with webp)

hmyes, I guess. Well, the more things we can benchmark and test, the better, of course.

The main problem (apart from support everywhere) can be heavily increased CPU load when converting to lossless JXL (although there are faster modes) and also slower decoding speed for clients (to me it is already sufficient, however, people can have much slower hardware and view a very large image), but not the fear of lossless images sizes compared to lossy (given that they already use less efficient, often unoptimized PNG images)

one of the things on our todo list is a mode that compresses a few types of content "fast and well enough" but more importantly is significantly faster to decompress

you may have noted that -s 3 encoded images also decode faster

Yep, practical modes of encoding and decoding speed with good efficiency are also needed. And however, all I said does not apply to photographic images, they can rarely be found in lossless format for normal web viewing (unless they are memes mixed with text or small images or mistakenly converted Jpeg) and for them also lossless makes much less sense (except for archival storage or use for further editing)

-I 0 images also decode very fast (and encode decently fast with -s 8)

But, it is also necessary that these modes have a noticeable higher average efficiency than PNG (otherwise there will be no reason to change the more supported format)

Which is almost true even for `-s 3`, if improving efficiency for pixelart content

This is also why I would like to see lossless encoding/decoding speeds on different generations of CPUs, with different support for SIMD and other things and how they differ between the fastest and slowest modes

Hello Scope, Could you teach me about this situation? (Lossy JXL still has problems with sharp lines and ringing in Art or b&w manga images), my target image is non-photographic, i very worry those problems...

I can't teach a solution to this problem (except increase the quality of encoding), at the moment this is how JXL works and in this it is worse than for example AVIF, but Jyrki planned some improvements in this direction

I get some issue in non-photographic image, https://gitlab.com/wg1/jpeg-xl/-/issues/147 situation is similar your description, -s 3 is better than -s 7, but that area still look bad, and this area is worse than avif.

A little curious, in your test set this situation will happen in vardct -d 1.0 and -d 0.5?

or this situation will happen in lower quality, like -q 85 -q 80?

It was with -d 1.0, I have not tested -d 0.5, instead I use -d 0.3 (and there it is quite rare, but also the size becomes much larger) and I have not done tests on new builds

Thank you very much, hope we can get encoder improvement about this issue in next release. 🙂

what is that commit?

https://gitlab.com/wg1/jpeg-xl/-/commit/e2b1e60f5fda02f5fb395b0b6113f64d33cbb91d

ah fix to that

https://gitlab.com/wg1/jpeg-xl/-/issues/149

maybe

Look at the commit list, there were two commits in a short time, one with a bugfix and after an hour another one that bumps the version to 0.3.2

Thanks

No problemo.

Has anyone looked into this one? https://gitlab.com/wg1/jpeg-xl/-/issues/151

I have not.

Anyway, I still seem to have the issue of progressive images not being able to be decoded. Is that normal? Here is my build process(with clang of course): ```git clone https://gitlab.com/wg1/jpeg-xl.git --recursive cd jpeg-xl mkdir build && cd build cmake -DCMAKE_BUILD_TYPE=Release -DJPEGXL_ENABLE_PLUGINS=ON -DJPEGXL_ENABLE_DEVTOOLS=ON -DCMAKE_INSTALL_PREFIX=/usr -DBUILD_TESTING=OFF -DJPEGXL_WARNINGS_AS_ERRORS=OFF -DJPEGXL_ENABLE_SJPEG=OFF .. cmake --build . -- -j 16 sudo make install```

I have followed everything down to the latter starting from number 2: https://github.com/novomesk/qt-jpegxl-image-plugin I wonder why progressive images would cause such a problem on my end. 🤔

What kind of progressive images do not decode? And in what sense do they not decode? Not at all, not progressively?

Is it vardct, modular, both?

It does not decode the progressive image at D3, and it uses VARDCT. I have not tested modular.

Here is the error that I am getting: ```nomacs '/home/bluezakm/Pictures/Dark/Low BPP/D3.jxl' [INFO] Hi there [INFO] plugins dir set to: "/usr/lib64/nomacs-plugins" [INFO] local client created in: 2 ms [INFO] theme loaded from "/usr/local/share/nomacs/Image Lounge/themes/Light-Theme.css" [INFO] CSS loaded from: ":/nomacs/stylesheet.css" [INFO] Initialization takes: 78 ms [INFO] [DkImageLoader] 5 containers created in 0 ms [INFO] [DkImageLoader] after sorting: 0 ms [WARNING] Unexpected event 1024 instead of JXL_DEC_NEED_IMAGE_OUT_BUFFER [WARNING] Unexpected event 1024 instead of JXL_DEC_NEED_IMAGE_OUT_BUFFER [WARNING] [Basic Loader] could not load "/home/bluezakm/Pictures/Dark/Low BPP/D3.jxl"```

does it open in gwenveiw?

No.

could havew the file?

Here they are.

all of them open for me in gwenview and qview

although I'm not on the latest qt jxl since compiling the new one fails for me

Yeah, I'd like to fix this as quickly as possible as I need to do a lot of testing for both JPEG-XL and AVIf libaom-av1/rav1e to see what they can do.

https://gitlab.com/wg1/jpeg-xl/-/blob/master/lib/include/jxl/decode.h#L205

Ah, so instead of calling the JXL_DEC_FRAME function, it's doing something else. It must be related to the plugin then 🤔

Seems to me like that application is subscribing to all decode events (see https://gitlab.com/wg1/jpeg-xl/-/blob/master/lib/include/jxl/decode.h#L285) but then throws warnings when it doesn't understand some of them. No idea if that's what causes the problem though

I think the problem might be that the plugin expects events to happen in a specific order, which is not necessarily true

https://github.com/novomesk/qt-jpegxl-image-plugin/blob/main/src/qjpegxlhandler.cpp#L282

Thank you for the small investigation. I will open an issue in the QT-JXL repo.

With progressive images, the order could be different, maybe.

I mean, it could also be a libjxl issue, but I think the application is not using libjxl in a correct way here: you shouldn't expect the decode status events to happen in exactly the same way for all input images. I think. <@768090355546587137> knows better though.

<:megapog:816773962884972565>

The QT-JXL plugin bug has been fixed in regards to progressive images by doing something funky.

you added that demon spawn to this server?

https://github.com/novomesk/qt-jpegxl-image-plugin/commit/5394e0c9dd17faedef9d07d27623ddb198b9a675 that's still not the right way to do it eh..

on the other hand, docs of relevant event really do tell that it it supposed to occur once per frame, before pixel data

but other events aren't so clearly defined so it still seems wrong to me, how it's done right now

should have decoder randomize event order to fuzz library users at some point >:^^DDD

ugh, just noticed that the gdk-pixbuf plugin we have is ignoring ICC profiles

that's not good

uh, is a bug from 0.3.2

in 0.3 it worked, on jpg lossless work

So, is it possible to output the butteraugli_main statistics to a text file? I could read the score from the command line, but that would be a bit harder to make a program in. 😛

well, at least on linux you can just use `./butteraugli_main img1 img2 > textfile`, I think there is some equivalent on windows...

the same thing if it has been built

``` $ ls -l tools/butteraugli_main.exe -rwxr-xr-x 1 Sami Aucun 3779728 5 mars 21:22 tools/butteraugli_main.exe* ```

Yeah, I think I should be able to do pooled butteraugli scores by next week for video.

Yep ```butteraugli_main.exe img1 img2 > textfile.txt```

Seeing a couple vulnerabilities being reported on Twitter.

That doesn't seem to be the best platform to report fuzzerbugs on... Have a link?

https://twitter.com/www_sesin_at/status/1368152063478235137 https://twitter.com/threatintelctr/status/1367847089225531399 https://twitter.com/threatintelctr/status/1367945212497174536

It's good that security researchers are scrutinizing the implementation

What's the best documentation for the bitstream right now, is it just libjxl source? Thinking about writing a decoder in some other language than C++

Rust!!!

Yup, code is actualy best way

Gotcha. And dumb question, but what is the box format container vs the alternative?

I think is becouse the container is not mandatory but I'm not sure

Second that, prevents silly things like these at compile time: https://gitlab.com/wg1/jpeg-xl/-/blob/65825c8629f181dbe8bdd432b15576735af29701/examples/jxlinfo.c#L130-131

yep

<@805471712451100672> if you are interested in writing a new decoder, it would actually be great to do it from spec and not from existing code

Best way to find spec bugs

Is that publicly available, <@!794205442175402004> ?

wait, why did we even write that in C instead of C++? (in C++ linters will usually complain about that kind of things anyway...)

I found the whitepaper but have yet to find a longer form document

it's not - for what it's worth, it's not even published officially (the FDIS might be but I don't know if that's something you can get your hands on, even by paying)

Because declaring variables inside of loops is bad. <:kekw:808717074305122316> Sorry for the joke, I've been learning more C++ these days. 😛

I now understand why some people like using it, while others don't.

I guess we wanted to have an example of using the API from C? oh well...

unless I forgot something basically everything in our codebase is C++ except the header files of the API

if somebody wants to implement a new en/decoder, well, that's nice - certainly want to wish them good luck though 😛

it might be simpler to make a decoder that doesn't need to be super fast

It's certainly going to be more work than making a jpeg decoder

But if speed doesn't matter much, just decoding correctly, it might not be _that_ hard

The spec is about 100 pages

A bit less if you only count the actual meat

well, for the codestream...

I'd start with writing a header parser, then do the modular sub-bitstream without any transforms and without MA trees. Then you have something that you can already test (on outputs of `cjxl -m -I 0 -X 0 -Y 0 -C 0 --palette 0`). Then you can make modular more complete, and then start on VarDCT. The stuff like filters, patches, splines you can do at the very end.

Is this the most up to date spec <@!794205442175402004> ? https://arxiv.org/ftp/arxiv/papers/1908/1908.03565.pdf

hah, no

by a lot

Oh, is it https://www.iso.org/standard/77977.html and https://www.iso.org/standard/80617.html ?

The preview at least says 2020 this time

not quite the latest either, but significantly more recent, at least

Gotcha. Well in any case, I appreciate the tips for where to start for a simple decoder. I think headers, then modular, then VarDCT makes sense as a good approach

I don't know exactly how the IP policy of ISO works - I know we can't make DIS or FDIS public, it might be possible to share it with specific people under appropriate agreements - <@!794205442175402004> do you have an idea?

Worst case, I can just take a look at existing decoder and put together a mental model of what the spec might be

(spoiler: that's how we wrote the spec :P)

it's likely easier for you to write a decoder out of the existing C++ code than out of the spec

but writing one out of the spec would help detecting issues, at least...

there are some tricks that could make modular mode faster to decode that we don't do - such as doing some form of just-in-time compilation of the MA tree to machine code - because they would make security people twitchy 😛 it might be possible to do that in an independent implementation that doesn't get integrated in browsers

Well, if you can translate the MS tree code to JS, then you can just use the browser's JIT 😉

true, but that's likely going to be slower overall than traversing the tree manually 😛

depends on image size of course, but it would take some rather big images

What's the performance of libjxl-compiled-to-wasm like?

I don't remember exact numbers, but not that great without wasm+simd for sure... with wasm+simd, I think it's not too far from native

Yeah, makes sense.

and I don't think wasm+simd is enabled by default anywhere

<@!532010383041363969> So, last week we talked about how XYB is great, yes?

Well, do you think you can point me to some resources to how I could implement usage of the XYB color space in encoders?

I can give you an overview

from the decoder side, it ought to be enough to give the image an ICC profile (you might have seen an XYB JPEG going around in this server at some point...)

Yes.

if you can use the C++ code from libjxl, then it's relatively easy - assuming you can't, you need to parse it 😛 let me see if I can find some equivalent simpler-to-digest description...

I can't because of Rust obviously. 😛 That means I might have to convert this myself: https://gitlab.com/wg1/jpeg-xl/-/blob/master/tools/xyb_range.cc

the problem is more this part: https://gitlab.com/wg1/jpeg-xl/-/blob/master/lib/jxl/enc_xyb.cc#L142

linear RGB to XYB conversion

that's the first part of the trick

(I hope I didn't make typos)

then you need some extra trickery: ``` x' = (x + 0.015386134) * 22.995788804; y' = y * 1.183000077; b' = ((b-y) + 0.27770459) * 1.502141333; ```

after you did that, you get "RGB" pixels that should be displayed correctly if you use that ICC profile

(everything here assumes 0-1 range, btw)

I don't really know, redistribution is not allowed because ISO has copyright, but one might make the argument that it's not a secret/NDA type of document either, and it is a case of "fair use" for JPEG experts to share the spec with others for the purpose of catching typos, unclear parts, or validating the spec with an independent decoder implementation.

Probably there are code of conducts or other ISO 'regulations' that can be interpreted in various ways as allowing it or not allowing it.

What is clearly not allowed is putting the spec on some public website.

What is allowed in terms of private sharing is not fully clear to me.

What does the M variable mean exactly?

OH, eyes.

Long, Medium, Short.

we can very likely find an arrangement if someone wants to write a decoder out of the spec

Uh, no, M was just for Matrix, B is for Bias

Oh, I was worried that it was going to be much more complicated than that. 😛

Lrgb is linear RGB - so you need to convert to that somehow

everyone at ISO and in JPEG XL effort (that I know of) would be very happy to have a verification effort for the spec

how to get the access within the rulebook of ISO is an issue, but I'm pretty sure we will be able to find a way

Mark Adler wrote the verification decoder for Brotli 🙂

from zlib/Adler32 fame, also a lead for some of Mars rover missions

for what it's worth, Swiss copyright law lists a few exceptions :p https://www.fedlex.admin.ch/eli/cc/1993/1798_1798_1798/en#tit_2/chap_5

but best to clarify this with ISO directly

Any help getting it compiled on Windows? Seems to be a real pain. The latest working version I have is 0.3. There were numerous errors I ran into including missing Gmock. It seems to be pointed to the right directory but tells me gmock.lib does not exist. I instead pointed it to the gmock files in googletest (which is a dependency that is not listed, it complains about it when you try to compile, so I downloaded it manually). That seems to work (maybe?), it no longer complains it can't find it. At this point I can get it to compile without any errors but produces an exe that is 1/3 the size it should be and obviously does not work. Running it, it immediately exits with no message.

Ouch

I can't help with Windows building. I would expect the build to be without tests if you don't have that dependency.

I saw some reported build issues on GitLab, I suspect that some are already fixed but still need to be synced to the public repo

You mean gmock? The compile fails when it can't find it. I could try disabling it but I suspect the exe wouldn't be any different. Not sure why the tests would be the cause of an incorrect compile.

There are quite a few warnings, mostly about missing functions that relate to testing.

I applied the newest patch that was suggested since I had the same issue, the incorrect function parameters. Would think that would cause any build to fail, not just windows.

you mean the one with JXL_RESTRICT? I *think* that happens to not change the ABI on some platforms, where it just ends up working...

I forced the GMOCK_INCLUDE_DIR into the json file and it seems to have fixed that, but still making a bad exe.

are you building with MSVC?

for me, it works with MSYS2 / MinGW-w64

Yes, using MSVC with Clang

I went through all the warnings and they are all about unused functions, use of deprecated functions, and disabling AVX due to issues with 32-bit MSVC.

I will try MSYS2

Seems to have failed as well. Ended with this...

`C:\msys64\home\jimbo\downloads\jpeg-xl-2021-03-06\lib\jxl\modular\encoding\enc_encoding.cc(274): message : see reference to function template instantiation 'jxl::pixel_type_w jxl::weighted::State::Predict<true>(size_t,size_t,size_t,jxl::pixel_type_w,jxl::pixel_type_w,jxl::pixel_type_w,jxl::pixel_type_w,jxl::pixel_type_w,jxl::Properties *,size_t)' being compiled [C:\msys64\home\jimbo\downloads\jpeg-xl-2021-03-06\build\lib\jxl_enc-obj.vcxproj] Generating Code...`

It seems to just halt after that with no other messages.

Weird

vcxproj? with msys2?

It is using, from what I can tell, the same MSVC package that MSVC was using. Do you have specific steps you go through with a different compiler?

Ah, the `-G` option. I am trying with that.

wait, you're saying that you don't get a working compile with MSVC?

Nope, and not with MSYS2/MingW64 with MSVC package as default either.

Ming is done compiling. The file size looks much more promising.

ok, that's useful to know and probably something we should fix. Do you think you could file an issue for that, and maybe with more information about the problems you have?

Drat, this file doesn't work either.

Just exits back to the command prompt.

<@!604964375924834314> do you have the same problem?

I will later, not sure how it will help though. Like I said, using the MSVC interface it shows a lot of warnings about deprecated functions and such but no errors. Just produces a faulty exe (~2.5mb). Ming/MSYS with MSVC package gets the dependancies and appears to start compiling but stops and drops to the command line part way through. Ming/MSYS with it's own libraries completes compiling but also with a faulty exe (~5MB).

just for our own tracking 🙂

both sizes are kinda reasonable actually

how are you compiling exactly?

The working 0.3 I have is 6MB. Not sure how 2.5 is. 5 is much more reasonable.

there should be some option to not remove error messages that should perhaps give you more information (not sure how to get that on windows)

a fully working `djxl` binary for me is 1.7 mb on Linux

With the MSVC interface I follow: https://gitlab.com/wg1/jpeg-xl/-/blob/master/doc/developing_in_windows.md However I have to download dependencies manually (not sure why deps.sh keeps failing partway through). Also have to apply patch here: https://gitlab.com/wg1/jpeg-xl/-/issues/166

With Ming I installed the development packages according to the site then followed install according to: https://gitlab.com/wg1/jpeg-xl Replaced the dependency installs with msys/ming package commands.

with MSYS2, I do this in a MinGW shell with `mingw-w64-x86_64-{cmake,ninja,clang}` installed: ``` cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -G Ninja $source_dir ninja ```

ok, can you try `cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo ..` instead?

(i.e. replace `Release` with `RelWithDebInfo` - should give you some error messages)

I am trying with that now. I used -G "MinGW Makefiles" to not use the Windows installer. If I don't it uses MSVC package by default.

<@!604964375924834314> I will try after but that seems like the same setup as when I used the interface.

I think MinGW Makefiles should have been fine as well

Like I said, using -G "MinGW Makefiles" or the interface completes compiling but exes don't work. Using default MSVC packages (no -G) fails part way.

Oh, and I used RelWithDebInfo on when using the interface.

RelWithDebInfo should only help once you have actual binaries I believe

(but if you can give more information on the error message with the interface, that would be really helpful)

There is no error message. Just a bunch of warnings.

<@!179701849576833024> Compile is done. Made a 122MB file. 👀

yeah, that happens 😄 lots of debug info...

Another dud.

and no error whatsoever... that's interesting, if a bit weird

<@!604964375924834314> I will try yours but need to find the ninja executable. I think it's buried in the MSVC path...

I don't think that makes a difference

it should be in the path if you install mingw-w64-$arch-ninja

I kinda hope it doesn't...

is this w32 or w64?

I assume it will be the same as using the interface cause it uses Clang and ninja to compile.

and you run it from the mingw shell

w64

I wonder what's different about your setup...

What is $source_dir ? I thought you wanted me to replace that with the ninja path.

oh, that’s jpeg xl’s root directory

so just `..` if building from a build directory directly in it

Ah, got it.

It is compiling. The output looks exactly like the interface.

Though all warnings are not showing.

Another dud, but the file size is very close to the working 0.3 I have.

This is very strange

Never seen a cjxl/djxl that just goes silent

is there such a thing as a return code on windows, and how can you get it?

it *could* be that it doesn't find some dll it needs and just doesn't even start

you'd get a message for that on Linux but I dunno about windows

I have no clue but I would expect dynamic loading fails not to be silent

Exit code?

That is a funny number

Very odd, here is my 0.3 exit code.

that exit code is 0xC0000135, possibly not completely random

Is there something like `ldd` in windows?

there is in msys2

`0xC0000135 == STATUS_DLL_NOT_FOUND`

well, it's apparently silent then

Ugh

a mingw shell should have everything needed in the $PATH, maybe try running it there instead of PowerShell?

Seems like there is a similar command in ming. I could try that.

Silent errors are annoying. Well at least it sets an exit code

What DLL would it be looking for is the question?

one of the dependencies I guess - libpng/libjpeg/...

libgcc_s, libpng, things like that

I know when I ran the initial setup it did say it couldn't find the avif library, not sure why that would be required.

it’s not

Let me re-run the setup and paste the output. Might give a clue...

That's for benchmark_xl, and should not be a problem if not found

Too big to paste, uploaded. This is from the clang/ninja setup.

so if I had to guess, openexr might be the issue - you could try disabling it, or copying the openexr DLL(s) to your build directory

(possibly also webp and avif)

Webp/avif should not get linked in cjxl/djxl

But openexr might, if enabled

I uninstalled it. Setup says this now: `-- Checking for module 'OpenEXR' -- Package 'OpenEXR', required by 'virtual:world', not found`

If I remember correctly, the interface wouldn't let me compile without it. I think that was something I had to install manually.

Oh. It's supposed to be an optional dependency. Maybe open an issue? If you don't need to do EXR input/output, it should be possible to build a cjxl/djxl that just doesn't support that.

Same problem ☹️

Could also be libjpeg or libpng

I installed the latest ming jpg, png, gif libraries. With the MSVC interface, I downloaded the versions it was expecting. Both produced duds.

Wish it said something like "Cannot find xyz.dll" - would be more helpful.

got anything out of that?

Haven't tried it yet. I am retrying with ming makefiles to see if it makes a difference without openexr, but doubt it.

the problem is not the binary itself, but it's that windows doesn't know how to find some of the dependencies... I am not sure why it would find them during compilation but not during runtime though

Not sure what external dlls it would be looking for that don't exist. I assumed it has some dlls embedded in the exe that it could not find.

I'm relatively convinced the problem is external dlls, but I'm not certain (didn't use windows seriously in more than 10 years...)

Duds again

I will try to object dump

Hm, that package doesn't seem to exist. Maybe not maintained?

Here is the output from a program called Dependency Walker. You were right, it can't seem to find a bunch of Windows related stuff. No idea why...

Oh boy. How can it not find all of that and still you have a terminal running? Must be somehow not looking in the right places...

Maybe it's compiling for some older version of Windows? Is there some way to tell it to target a specific version?

Man this thing is giving me a headache. Gonna go grab some lunch and will try to figure out something when I get back.

it doesn't find [ ? ] LIBGIF-7.DLL [ ? ] LIBJPEG-8.DLL in particular, which *might* explain it

everything else is nested, which might have other explanations

KERNEL32.DLL is a 64-bit module? That sounds weird

It would probably be useful to build statically linked windows binaries to reduce these kind of headaches...

this has its own headaches :p

I’m not surprised by kernel32 being 64 bits, there is probably a lot of stuff that is still named *32 for compatibility

let’s remember that one of the parameters of WinMain was last useful in 16-bit versions of Windows

https://devblogs.microsoft.com/oldnewthing/20040615-00/?p=38873

<@!172952901075992586> does it work if you run the tools from the same shell from which you build them?

specifically, a shell started from e.g. the entry “MSYS2 MinGW (32|64)-bit” in the start menu

it should have everything needed in the $PATH

I am only running MSYS MinGW 64-bit

what confuses me, then, is that your screenshots show PowerShell

Once it compiles I copy the exes to that directory

Oh you mean to run them directly from MSYS? I can try it but don't know why running a 64-bit app from a 32-bit console would even matter.

it’s not so much the fact that it’s 64- or 32-bit but more that the MSYS MinGW shell has the PATH variable properly set up

it contains the directories that have all the “system” libraries required by the executables

copying all those dlls to the output directory should also work but it’s more work

Now I'm confused...

Why would the older 0.3 work and not 0.3.3 when I have the same system/user PATH?

were they built in the same way?

No, downloaded. Though I believe it was made on MinGW.

Still doesn't explain why its looking in the wrong places for Windows. Same happens when I build in the MSVC gui (not MinGW)

I tried running it from the 32-bit Powershell and same issue: nothing happens.

if you run `echo $PATH` in the MinGW shell, you will see many more entries than if you run `echo $env:PATH` in PowerShell

and that’s because the MinGW init script adds those entries but just locally

Opposite happens. PowerShell has many more entries.

sorry, yes, I just tried and it’s not “more”, it’s an entirely different set

but the point is that some of those entries are not in PowerShell, and it’s them that make it work

well, more specifically, `/mingw64/bin` is probably doing all the work

that would be `C:\msys64\mingw64\bin` on your disk, I believe

So how do I get the compiled program to look in the default Windows location rather than mingw or visual studio framework in MSVC?

it already does, it’s just that the dlls are not there

I got it to work in PowerShell by copying the following dlls to the directory of the executables: https://pastebin.com/raw/TCXj3xYC

if some of those are absent from your system, they are probably not needed

for example the Ilm* dlls are from OpenEXR

if you don’t have it installed, then your `cjxl`/`djxl` are built without support for it anyway and won’t require it

using `ldd cjxl.exe` from the MinGW shell can also tell you the list

Ok, I think I got it now. I only copied the libjpeg, libgif, and libpng into the directory and it is working in Powershell now. I assume the person who compiled the other version figured out how to get it to hook into Windows' own image library so they wouldn't be required. I need to see if I can find out how to do that.

Maybe if those dlls are also elsewhere, you can hide the mingw ones and change your path in mingw to look at the system path when compiling things

They are not, I already looked. But I ran the dependency walker on the older version and it is not even looking for libjpeg, libgif, libpng. All the other weird Windows core stuff was there so I guess those do not matter, possibly for older/other versions of Windows.

might have used a static version of those libs

In any case, 0.3.3 is a bit faster, wobbles on file size (+-1KB), and has noticeably less artifacts around edges vs 0.3.

Still a number of images that fail to lossless jpeg transcode.

which ones?

CMYK is a known limitation, we cannot do those

(we can do CMYK, but not with VarDCT so not as lossless transcoded jpeg)

other jpegs that fail: please open issues for them, it's probably a bug that can be fixed

I tried a few that were reported a bit back and still same issue. Also downloaded some of my own and tried and they either fail to encode or appear to complete encoding but end up being nothing but a black bar. I checked, they are RGB.

Ok, will create an issue. I will also go back through older images from different cameras and see if the same issue occurs. Maybe it's just something strange this camera is doing.

Created issue 170: https://gitlab.com/wg1/jpeg-xl/-/issues/170

everything win32 is 64-bit on 64-bit windows. WoW64 on the other hand is 32 bit stuff.

Because that makes sense, or something.

iirc WoW meant Windows on Windows

originally it's so that 16-bit Windows stuff can run on 32-bit Windows

and then WoW64, which is for 32-bit apps on 64-bit Windows

it is kinda confusing, though

Yep

I think the black bar is an application issue in XnView. Could you check what happens if you decode them with `djxl` ?

It works fine in djxl but Squoosh is also not able to open any lossless transcoded jxls (non lossless works fine).

I get the large file restriction, figured that could be an issue.

I wonder if it's the isobmff container that causes trouble

Maybe they don't know it's a jxl because it doesn't start with 0xFF0A

Eh, it still doesn't work with SIMD enabled and people keep using GCC to compile JXL via m-ab-s <:PepeSad:815718285877444619> https://gitlab.com/wg1/jpeg-xl/-/issues/62

```if [[ ${CC##* } = gcc ]]; then do_simple_print -p "${orange}Warning: JPEG-XL compiled with GCC will not use SIMD optimizations!$reset" extra_cxxflags+=('-DHWY_COMPILE_ONLY_SCALAR') fi```

that's because it's buggy, right?

I mean, sounds pretty much like a compiler bug to me...

wouldn't be the first time, or even the tenth time xD

Yep, encoding does not work without disabling SIMD

I remember it working on linux

so platform-specific gcc bug apparently

#yay

although if I recall correctly from the code, there's not a lot of reasons for the compiler to spill variables onto the stack...

I don't think it's a good idea to just disable SIMD, which basically means people will get something that works but very slowly

(and then when the compiler bug gets fixed we may forget to remove that from the build script)

if there's a known bug in gcc on certain platforms, we should 1) report the bug, 2) maybe detect this case in our build script and just exit with a message saying: either use a different compiler, or compile without SIMD but know that speed will suffer, or try anyway (maybe you have a fixed gcc) and know that maybe it will crash due to a compiler bug.

Yep, that's why I mentioned this bug again, because m-ab-s is used by many people to easily and automatically build ffmpeg and other things under Windows <https://github.com/m-ab-s/media-autobuild_suite/commit/af2a73183192147fccec747fe905b74a02633456>

Disable SIMD, enable benchmark tool <:Thonk:805904896879493180>

let's benchmark with crippled builds! yay

Also <https://github.com/m-ab-s/media-autobuild_suite/commit/53ce5a94673637584b2d36769ac4c53a10747170>

to be frank, I don’t really see the point of using it, it doesn’t really look much easier than building those projects oneself

Other things maybe, but ffmpeg includes many libraries and not everyone has time to keep track of all the changes, m-ab-s also includes various patches and fixes to build correctly under Windows

I agree.. this kind of legacy quick fix commonly causes problems in the future (ex. one was just removed from LAME in 3.101) so it’s better to detect the problem and act from there instead of disabling features on compilers unconditionally due to issues we can’t control

well msys2 itself has a PKGBUILD for it, it’s presumably just as easy to build from it (I’ve done it many times to build with fdk_aac support)