|
intelfx
|
|
We're actively working on progressive lossless, if not almost done with it already https://discord.com/channels/794206087879852103/803645746661425173/1357850229247840401
Right now, progressive_dc disabling chunked works around an [issue](<https://github.com/libjxl/libjxl/issues/3823#issuecomment-2351120650>) with the TOC, where the 1:8 LF frame can't be rendered until the end of the file. Downside being the entire image has to be processed as a whole, instead of threading individual groups.
Squeeze, used for progressive lossless, also disables chunked due to downsampling the image as part of the transform
|
|
2025-04-05 12:24:02
|
|
|
2025-04-05 12:24:09
|
I actually wanted to talk about that issue (is a thread ok?)
|
|
2025-04-05 12:28:14
|
if #3823 is only about group placement in the output stream, what exactly prevents libjxl to simply disable streamed output if progressive encoding is used, buffer everything and then reorder the groups as needed?
|
|
2025-04-05 12:28:48
|
even more, from what I see, all the necessary reordering code is already there, so why is [this](https://github.com/intelfx/libjxl/commit/d669a2b2258befa1ff86ee27a5d5f7023ef6d937) not a solution to #3823?
|
|
2025-04-05 12:30:13
|
I also not quite understand this remark:
> Downside being the entire image has to be processed as a whole, instead of threading individual groups.
Are we speaking about encoding or decoding here?
|
|
2025-04-05 12:34:25
|
And lastly, can you clarify about the chunked encoding — how is it referred to in the libjxl codebase? I tried skimming through the code to understand what exactly it is and how does it work, but I don't understand a thing
|
|
2025-04-05 12:34:40
|
The technical report linked above also does not mention anything by this name
|
|
|
jonnyawsom3
|
|
intelfx
if #3823 is only about group placement in the output stream, what exactly prevents libjxl to simply disable streamed output if progressive encoding is used, buffer everything and then reorder the groups as needed?
|
|
2025-04-05 12:57:43
|
This is what I also wondered. The TOC already has to be written at the start after the file is encoded, so it should be possible to reorder the chunks too. Streamed output was one reason, but that's already a separate flag as I mentioned in the issue, so it should just be a matter of someone implementing it
|
|
|
intelfx
And lastly, can you clarify about the chunked encoding — how is it referred to in the libjxl codebase? I tried skimming through the code to understand what exactly it is and how does it work, but I don't understand a thing
|
|
2025-04-05 01:00:03
|
Chunked encoding uses local MA trees, one per group which can be multithreaded, but as a result disables any full-frame features (patches, Global MA tree, Squeeze/Progressive Lossless). It does more, but that has the largest impact on memory usage and encode speed
|
|
2025-04-05 01:00:39
|
I'm currently in London, so responses may be short or delayed
|
|
|
|
intelfx
|
|
This is what I also wondered. The TOC already has to be written at the start after the file is encoded, so it should be possible to reorder the chunks too. Streamed output was one reason, but that's already a separate flag as I mentioned in the issue, so it should just be a matter of someone implementing it
|
|
2025-04-05 01:13:36
|
afaics, it seems to be *already* implemented, i.e. simply hacking the code to force non-streaming encoding (see the linked commit) seems to generate a stream that does not have the problem described in #3823
|
|
2025-04-05 01:13:40
|
unless I'm missing something
|
|
2025-04-05 01:14:13
|
the only problem I see is that taking this code path results in a larger file
|
|
|
|
jonnyawsom3
|
|
intelfx
afaics, it seems to be *already* implemented, i.e. simply hacking the code to force non-streaming encoding (see the linked commit) seems to generate a stream that does not have the problem described in #3823
|
|
2025-04-05 01:25:56
|
Ah, I meant doing the group reordering with chunked, to maintain low memory use and encode speed
|
|
2025-04-05 01:29:12
|
Here's the code
<https://github.com/libjxl/libjxl/blob/c496c521f99c13b8205c4fc4ff3eb3d652a1d1c3/lib/jxl/enc_frame.cc#L1734>
|
|
|
|
intelfx
|
|
Here's the code
<https://github.com/libjxl/libjxl/blob/c496c521f99c13b8205c4fc4ff3eb3d652a1d1c3/lib/jxl/enc_frame.cc#L1734>
|
|
2025-04-05 01:33:45
|
yup, this is exactly the piece I tried to hack up
|
|
2025-04-05 01:34:26
|
so what you refer to "chunked encoding" is called "streaming encoding" within libjxl codebase (i.e. `EncodeFrameStreaming()`)? or that's two separate things? I still don't understand this bit
|
|
2025-04-05 01:36:25
|
sorry for bothering you 😅
|
|
|
|
jonnyawsom3
|
2025-04-05 02:29:03
|
It's confusing because encoding can be chunked, and input can be streamed, but streamed requires chunked (hence why it's called streamed internally)
|
|
2025-04-05 02:35:28
|
This may be of use too <https://github.com/libjxl/libjxl/blob/main/lib/include/jxl/encode.h#L343>
|
|
|
|
intelfx
|
|
This may be of use too <https://github.com/libjxl/libjxl/blob/main/lib/include/jxl/encode.h#L343>
|
|
2025-04-05 02:36:34
|
afaics, the only place this param is ever used is the same decision logic in CanDoStreamingEncoding
|
|
|
|
jonnyawsom3
|
2025-04-05 02:37:04
|
Yeah, but the comment may help explain what it does internally
|
|
|
|
intelfx
|
2025-04-05 02:44:14
|
yup, the comment makes sense insofar it refers to "streaming input/output" (it looks like the actual treatment of different non-zero values of JXL_ENC_FRAME_SETTING_BUFFERING is not actually implemented)
|
|
|
|
jonnyawsom3
|
2025-04-05 02:45:02
|
Found the last time it was mentioned https://discord.com/channels/794206087879852103/1065165415598272582/1284415979933143071
|
|
|
|
intelfx
|
2025-04-05 02:46:48
|
right, this seems to match what I could gather from perusing the code this evening, but it doesn't add anything to my understanding
|
|
2025-04-05 02:47:45
|
so: "chunked encoding" does not exist in libjxl as some kind of an explicit mode, but rather is just the code path implicitly taken when `PassesEncoderState::streaming_mode != 0`?
|
|
|
|
jonnyawsom3
|
2025-04-05 05:52:19
|
<@179701849576833024> do you think chunked progressive could reorder the LfGlobal similar to the TOC?
|
|
2025-04-10 02:36:14
|
I only just realised, but we enabled local MA trees for lossless progressive and they don't have the same ordering issue as VarDCT. Very strange
|
|