JPEG XL

<@167354761270525953> round 2 <https://mega.nz/file/adtFwJDI#iIQY0GmfskdnBc_nUJlK1fRDdoO-m6hG6hnNhCwW9vI>

Nice!

Waiter, Waiter! More screnshots please

https://tenor.com/view/waiter-cat-more-kibble-please-gif-13625215935695637260

I am at 560 screenshots now

I am at 1300 screenshots from over 80 games btw and currently testing the encodes

Sweet.

JXL was so much faster that I had to switch to writing to /tmp

This made no difference at all for AVIF lol

Here is the current draft, feel free to criticise

I will post the encoding results in <#803645746661425173> later

As you are including png decode in your measurement, it is not exactly representative of the use case, but this should just mean the fastest jxl has a bit better ratio. Personally, I would be curious about at least proportion of HDR content, even proportion of different art styles, but I would also suggest it doesn't have to be totally scientific if it can convince Valve to run their own testing. They would be best informed about the requirements of the product.

I'm going to guess a big reason why JXL wouldn't be a priority for Valve is JXL is much harder to share on the web.

> png decode I only measured the encoding from PNG to AVIF/JXL, or was this just a typo? I could split up the images into photographic, pixel art, HDR, but I don't know if that is really needed seeing as the dataset itself has screenshots from so many games that it should be representative. I could add some more pixel art screenshots, but even without them the results speak for themselves I think

That is true, but this is one of the main reasons I am doing this: to gain more adoption for JXL (besides the general advantages). And another point: the current AVIF screenshots gamescope creates cannot be shared over discord either, the main platform of almost any person playing games

Normally gamescope would take an image buffer directly from memory and encode it to the target storage format, right? In this test, a png file needs to be decoded first to a buffer in memory, then that is encoded to the target format. But again, If this just gets the format in front of Valve, maybe they will test it properly in their architecture.

Agreed, I have cloned gamescope to try and get it to encode JXL directly, but I am not a programmer

But still, even with that, it is clear that JXL is much faster than encoding AVIF

Yeah, question is always is it _faster enough_, _denser enough_ ... but I think your presentation is pretty good, you did things much better than most would.

Thanks

That is the question that remains to be answered by Valve indeed

But we'll see

Also I would count it as a win if the default remains AVIF but an option to ouput JXL is added

Yes, that would be the best outcome actually, until JXL has support in Chromium.

Do you have any direct frame buffer captures from console emulators? I imagine screenshots from NES games for example would compress greatly if it's able to utilize the "patches" feature, because those consoles games themselves make heavy use of repeating sprites from a tiny sprite sheet

This is assuming there are JXL encoders capable of utilizing that, I don't know whether there are actually

Oh this is about Steam games, I see... Well, many modern games still use sprites in this way to achieve a retro look

I would happily incoporate this into another comparison, but I don't know of any emulator that supports JXL

I have included some "pixel art" games into the dataset, these images often get blown up massively if encoded to AVIF, even compared to PNG

Ah nice, I'm guessing that's because both PNG and JXL are utilizing palette compression then, as those pixel art games will often have many in-game pixels that are the same color and in the image can be represented as few-bit indices into a color palette. If `JXL_ENC_FRAME_SETTING_PATCHES` is set to `1`, JXL files should be even smaller as it will create not just palettes of pixels but pallets of groups of pixels as well. For example it would only store one of these tiles that I have highlighted with white lines, and the rest of the tiles would reuse the stored one

But the patches need to be an exact match, so as soon as the game does funky upscaling with some sort of interpolation that makes that not the case it won't do it at all (in lossless mode)

luckily most pixel art games use nearest upscaling, so it still looks proper

Ah, that's good

Take a look at this

Same data, but the avif is almost 10(!) times larger

Quite the difference indeed

Unrelated, but now I'm thinking that it might be possible to create a custom encoder specifically for an NES emulator that doesn't even use a frame buffer as the screenshot source, but just directly builds a JXL from sprite data and other relevant information loaded in the emulator's memory. I wonder how small you could get the screenshots

This possibly(?) sounds like something an encoder could be purpose built for. But I wouldn't know, i'm not a programmer

https://github.com/ValveSoftware/gamescope/issues/1525

Here the feature request on gamescope's github

If it were possible I might've included memory usage too

(I only just found the thread)

Since I ran 16 encodes in parallel CPU usage was more the issue, but both encodes ran at the maximum possible speed. It took very little RAM, even with 16 parallel encodes

Currently the patch detection in libjxl is mostly focused on text. Monad made a version that focuses on tiles such as in games, and it does add improvement, but the ideal solution would be directly referencing the sprite sheet in an emulator and essentially rebuilding the scene inside the JXL file. You can also do 2x, 4x and 8x nearest neighbour upsampling in JXL files, so it could be encoded at native resolution and just scaled in metadata instead

Hehe, that's exactly what I came up with a bit further down

Actually yeah, fair point. I immedeately forgot it was e1, although I know I'd have OOM'd with 16 4K files at once

Natually in the 2 messages I skipped before getting distracted by the github issue Dx

Does JXL allow you to transform these sprite like flip them? I know the NES has a handful of sprite transformations like these, and I'm wondering how many of them are natively available in JPEG XL. It's not a problem if there's some things JXL can't do, you'd just need to store some redundant data

It's not something that's likely to appear in real world images *except specifically* screenshots of emulators that can do sprite transformations, so I don't expect much to be possible there

> We considered it `Rotation` (also flips, etc) but decided to keep it simple since it adds a bit of implementation complications to do patch blitting in arbitrary orientations and it is already hard enough to detect patches without considering orientations...

Back on topic - does this version of the encoder used here in lossless mode suffer from some of the file size regressions that have been there since 0.8.X in lossy mode?

If so, it might be more representative of JPEG XL as a format to go with the decoder on 0.8 as a bug is causing larger file sizes and worse quality in recent versions

While not comparing across mutiple releases, this graph showed slight improvement since 0.10.1 https://discord.com/channels/794206087879852103/803645746661425173/1284340338143526997

Oh ok. Yeah I read the issue linked there before ([ Regression in lossy nonphotographic image quality #3530 ](<https://github.com/libjxl/libjxl/issues/3530#issuecomment-2132164263>)) which was about versions up to 0.10 being worse than 0.8 but I guess 0.11 resolved that then, nice (but also that was lossy, idk if lossless was affected)

How do we expect Valve to react to this feature request?

I'm guessing at best they'd want JXL support in embedded chromium first to actually be able to display it

If they chose to implement it I'd hope that that an option to chose JXL as an output format for screenshots would be added. AVIF will very likely remain the default for as long as chromium does not support JXL.

Current libjxl will not detect patches along those white lines. It may detect patches for the tiny decorations on the floor, which would likely make the file larger than without in this case. libjxl often makes bad decisions on non-text content.

In any case, it does not try to do patch detection at all when fast speeds are required.

I only just realised, but you did the encodes running 16 jobs at once, right? Wouldn't that have messed with the multithreading of the encodes?

Was wondering why the 4k set was so much slower than the 1080p, since usually larger images scale near-linearly to cores until a certain extent. Took 10x as long for 4x the pixels

It possibly has, the best way to compare would probably have been to encode one image after the other and compare the encoding times for each image. But even though the 4k encode was *slightly* slower than it would have been (~2 seconds * 2 because of dataset size and * 4 because of pixel amount equals to rougly 16 seconds compared to the 22 seconds real "Encode to RAM" encoding time) it still showed the massive speed increase that is possible with adding support for JXL as an alternative to AVIF

wouldn't a benefit of JXL compared to AVIF for game screenshots be that JXL can actually fully store 16-bit data while AVIF only goes up to 12-bit. the reason I bring this up is because (at least with DirectX and Windows) games can only really output as 8-bit, 10-bit, and 16-bit (ignoring older games and APIs which can get below 8-bit)

only really relevant for native HDR games mostly

https://learn.microsoft.com/en-us/windows/win32/direct3darticles/high-dynamic-range

<@167354761270525953> <@238552565619359744> apologies if the pings are unnecessary but I have no clue if Discord properly notifies people of new messages in threads

No worries, I have seen your message. I wanted to answer when I am not on mobile

Oh I saw it, but with no HDR display I've got limited experience with it. The closest I can get is taking NVIDIA Ansel photos that output float jxr files with imaginary values

I don't think I know of any games that output 16bit, but that could just be a limitation of whatever library they're using to save the image

I know that I can get this game to output as 8-bit or 10-bit or 16-bit by changing around some stuff in the dgVoodoo2 config and can confirm it provides a noticeable bending reduction when I set the internal color formats to a higher bit depth and display it as 10-bit on my 10-bit monitor with the right presentation mode https://store.steampowered.com/app/1304510/SCP_NineTailed_Fox/

the bending reduction is really noticeable for all the dark areas

if uh anyone has the proper software and know how for capturing high bit depth screenshots then this might be a good game to test with

<:banding:804346788982030337>

This is currently only a theoretical advantage, because gamescope will always output a 10 bit AVIF, even though AVIF does support 12 bit indeed, as you mentioned. I have also found no communication from Valve concerning their choice of AVIF. I strongly assume it is because a speed 10 AVIF is often smaller than a barely compressed PNG. This is pure speculation on my part though, I have not tested whether that is actually the case. If anyone wants to benchmark encoding between PNG and AVIF, please do share it. Mayb I'll get around to testing it, but I am much more interested in a possible inclusion of JXL into gamescope.

The people over at the "HDR Den" discord seem to often use SKIV, maybe have a look at that. I haven't used it myself and there is no linux release for it, so no idea as to how it works

I found a game that allowed me to choose between JPEG and PNG for the in-game camera mode, which gives me the normal "screenshot saved" popup from Steam. I guess games are allowed to hook into the screenshot system. Also I found that screenshots taken with the Steam Deck in the Steam UI are JPEG, and I also found some Steam discussions about screenshots being in JPEG by default. Any chance you could clarify in what situation Steam will use lossless AVIF? I don't quite understand (RE: this issue: [ [Feature request] Add support for JPEG XL screenshots due to greater speed and compression #1525](<https://github.com/ValveSoftware/gamescope/issues/1525>)) <@167354761270525953>

there is a setting in Steam that makes it so when you take a screenshot it will save both a JPEG and PNG and the PNGs have their own dedicated folder, the option for AVIF just makes it be used in HDR games instead of tone-mapped PNGs

Ohh I see, thanks

Do note that I could not get the steam deck to save even HDR games as AVIF on the steamdeck. I tested Doom Eternal, but all screenshots were PNG. Just to be sure I even checked /tmp, but no dice.

Does the Deck even support HDR?

<@384009621519597581> gamescope on the desktop will always save screenshots as AVIF if it is executed via `gamescope %command%` in game launhc settings

Yes

It even has a dedicated icon for showing that it is displaying HDR

Only for the steam dekc OLED though

Ah

To be aboslutely sure that no AVIF HDR screenshots were recorded i even attached a usb keyboard and pressed super+S, the screenshot keykombo, but this had no efect either

Yeah, you can do quite a bit with the hook https://github.com/BOLL7708/SuperScreenShotterVR

We love integration

I ended up reading your issue again just now, and thought about our findings from this regression https://github.com/libjxl/libjxl/issues/4447

I'm assuming you used Clang, so the main regression shouldn't have been present, but the standalone speed could've given it a slight boost too depending on how many threads you were using

No, it was a default cmake build from the arch repos

and still it was that much faster

That means you could've got between 2-12x the performance then

Since I opened that issue I have tested more with 4k and HDR on steam

HDR AVIFs take seconds to save

I have vibecoded a very bad implementation of libjxl into gamescope and it was many times faster

of course nothing I would bring into this discussion on a serious basis, since it was just vibecoded

<@274048677851430913> took a shot at integrating simple-lossless JXL into ReShade, works great and thought about making a PR

sounds great!

another pain point of steams HDR AVIFs is that the PNGs derived from them are not identical to the AVIF

ssimulacra2 shows around 96 score, whioch is pretty similar, but still not identical

Already tested HDR on both 10 bit and 16 bit (float) surface format and seems to work well, about time now :p Though I just found out by default reshade HDR PNG implementation always converts that F16 linear sRGB surface into r2020 PQ, while my jxl implementation preserve it (which I do prefer personally)

preserving the original space seems preferable IMO especially since JXL is fully color managed. Also something else but I know that the simple lossless encoder for JXL sadly doesn't support this but it would be cool if the ReShade implementation could save the depth information to the dedicated depth channel that the JXL format supports. <@794205442175402004> how hard would it be to have the [simple lossless encoder](https://github.com/libjxl/simple-lossless-encoder) support writing to other channels besides RGBA?

quickly skimming the code it doesn't seem to be setup in a way that would allow writing to other channels easily but idk maybe I'm wrong

Well Simple is a cut down version of Fast, which says it supports 4 channels <https://github.com/libjxl/libjxl/tree/main/tools/fast_lossless> but is hardcoded for Alpha <https://github.com/libjxl/libjxl/blob/main/lib/jxl/enc_fast_lossless.cc>

And that's something I never considered, with practically all screenshot formats being 10-bit or int, JXL might be the only way to get a truly lossless image from some games (Ignoring JXR because it's support is awful)

one test with 16 bit float

Ah yes, thank you Irfanview for not doing any color management

Good photo though, and I might actually use ReShade to get around an FSR bug with NMS' photo mode

I usually use tev for viewing HDR images

in principle that shouldn't be hard, but part of what makes it fast is that it is hardcoded for taking 1-4 interleaved channels (in uint8 or uint16) as input, so it would need to be extended.

Games generally don't use the Alpha channel, so in this instance Depth could likely be replace it

hmm yeah I forgot that alpha isn't relevent for this use-case so would doing that be as simple as just changing what the final channel type is marked as for alpha or no? (non-upstreamable change)

Yes, actually in general we should probably upstream something like that so it can be used not just for RGBA but also CMYK or RGBD

for the time being does this seem like a correct hack for making the simple lossless encoder use it's input of RGBA to output as a proper RGBD file? ```diff --- simple_lossless.cc +++ simple_lossless_depth-hack.cc @@ -637,32 +637,27 @@ if (frame->bitdepth <= 14) { output->Write(1, 1); // 16-bit-buffer sufficient } else { output->Write(1, 0); // 16-bit-buffer NOT sufficient } if (have_alpha) { output->Write(2, 0b01); // One extra channel - if (frame->bitdepth == 8) { - output->Write(1, 1); // ... all_default (ie. 8-bit alpha) - } else { output->Write(1, 0); // not d_alpha - output->Write(2, 0); // type = kAlpha + output->Write(2, 1); // type = kDepth output->Write(1, 0); // not float if (frame->bitdepth == 10) { output->Write(2, 0b01); // bit_depth.bits_per_sample = 10 } else if (frame->bitdepth == 12) { output->Write(2, 0b10); // bit_depth.bits_per_sample = 12 } else { output->Write(2, 0b11); // 1 + u(6) output->Write(6, frame->bitdepth - 1); } output->Write(2, 0); // dim_shift = 0 output->Write(2, 0); // name_len = 0 - output->Write(1, 0); // alpha_associated = 0 - } } else { output->Write(2, 0b00); // No extra channel } output->Write(1, 0); // Not XYB if (frame->nb_chans > 2) { output->Write(1, 1); // color_encoding.all_default (sRGB) } else { ```

Looks ok. You can use jxlinfo to check if it's correctly tagged.