JPEG XL

Info

rules 58
github 38694
reddit 688

JPEG XL

tools 4270
website 1813
adoption 22069
nomacs 7
UI 11
chrome 34
image-compression-forum 0
glitch-art 1071

General chat

welcome 3957
introduce-yourself 294
color 1687
photography 3532
other-codecs 25116
on-topic 26652
off-topic 23987

Voice Channels

General 2578

Archived

bot-spam 4577

We're actively working on progressive

intelfx
2025-04-05 12:24:02
2025-04-05 12:24:09 I actually wanted to talk about that issue (is a thread ok?)
2025-04-05 12:28:14 if #3823 is only about group placement in the output stream, what exactly prevents libjxl to simply disable streamed output if progressive encoding is used, buffer everything and then reorder the groups as needed?
2025-04-05 12:28:48 even more, from what I see, all the necessary reordering code is already there, so why is [this](https://github.com/intelfx/libjxl/commit/d669a2b2258befa1ff86ee27a5d5f7023ef6d937) not a solution to #3823?
2025-04-05 12:30:13 I also not quite understand this remark: > Downside being the entire image has to be processed as a whole, instead of threading individual groups. Are we speaking about encoding or decoding here?
2025-04-05 12:34:25 And lastly, can you clarify about the chunked encoding — how is it referred to in the libjxl codebase? I tried skimming through the code to understand what exactly it is and how does it work, but I don't understand a thing
2025-04-05 12:34:40 The technical report linked above also does not mention anything by this name
jonnyawsom3
2025-04-05 12:57:43 This is what I also wondered. The TOC already has to be written at the start after the file is encoded, so it should be possible to reorder the chunks too. Streamed output was one reason, but that's already a separate flag as I mentioned in the issue, so it should just be a matter of someone implementing it
2025-04-05 01:00:03 Chunked encoding uses local MA trees, one per group which can be multithreaded, but as a result disables any full-frame features (patches, Global MA tree, Squeeze/Progressive Lossless). It does more, but that has the largest impact on memory usage and encode speed
2025-04-05 01:00:39 I'm currently in London, so responses may be short or delayed
intelfx
2025-04-05 01:13:36 afaics, it seems to be *already* implemented, i.e. simply hacking the code to force non-streaming encoding (see the linked commit) seems to generate a stream that does not have the problem described in #3823
2025-04-05 01:13:40 unless I'm missing something
2025-04-05 01:14:13 the only problem I see is that taking this code path results in a larger file
jonnyawsom3
2025-04-05 01:25:56 Ah, I meant doing the group reordering with chunked, to maintain low memory use and encode speed
2025-04-05 01:29:12 Here's the code <https://github.com/libjxl/libjxl/blob/c496c521f99c13b8205c4fc4ff3eb3d652a1d1c3/lib/jxl/enc_frame.cc#L1734>
intelfx
2025-04-05 01:33:45 yup, this is exactly the piece I tried to hack up
2025-04-05 01:34:26 so what you refer to "chunked encoding" is called "streaming encoding" within libjxl codebase (i.e. `EncodeFrameStreaming()`)? or that's two separate things? I still don't understand this bit
2025-04-05 01:36:25 sorry for bothering you 😅
jonnyawsom3
2025-04-05 02:29:03 It's confusing because encoding can be chunked, and input can be streamed, but streamed requires chunked (hence why it's called streamed internally)
2025-04-05 02:35:28 This may be of use too <https://github.com/libjxl/libjxl/blob/main/lib/include/jxl/encode.h#L343>
intelfx
2025-04-05 02:36:34 afaics, the only place this param is ever used is the same decision logic in CanDoStreamingEncoding
jonnyawsom3
2025-04-05 02:37:04 Yeah, but the comment may help explain what it does internally
intelfx
2025-04-05 02:44:14 yup, the comment makes sense insofar it refers to "streaming input/output" (it looks like the actual treatment of different non-zero values of JXL_ENC_FRAME_SETTING_BUFFERING is not actually implemented)
jonnyawsom3
2025-04-05 02:45:02 Found the last time it was mentioned https://discord.com/channels/794206087879852103/1065165415598272582/1284415979933143071
intelfx
2025-04-05 02:46:48 right, this seems to match what I could gather from perusing the code this evening, but it doesn't add anything to my understanding
2025-04-05 02:47:45 so: "chunked encoding" does not exist in libjxl as some kind of an explicit mode, but rather is just the code path implicitly taken when `PassesEncoderState::streaming_mode != 0`?
jonnyawsom3
2025-04-05 05:52:19 <@179701849576833024> do you think chunked progressive could reorder the LfGlobal similar to the TOC?
2025-04-10 02:36:14 I only just realised, but we enabled local MA trees for lossless progressive and they don't have the same ordering issue as VarDCT. Very strange