JPEG XL

but it's also not inherently lossy - it's just hard/rare for it to not be

do you mean that if you took a VarDCT JXL, sent that into another JXL encoder, as pixel data, it could theoretically output the exact same blocks and be lossless?

I think you can go one step further, DCT is a mathematically reversible (lossless) transform. it's just we throw away some of it where we can afford it. but if you don't throw anything away, and don't quantize, I think vardct can in theory be lossless?

I'm not sure if you can setup quantization tables with 'multiplier values' of 1 so that no coefficients are rounded by being multiplied?

I mean more that if you had a hypothetical camera sensor that transmitted it's data as a DCT, you could just plug that into VarDCT and it be lossless

ahh right

wait don't many camera sensors like laptop webcams basically output a stream of JPEGs?

MJPEG is a thing, yes

though they can often output a raw stream as well

but because USB 2.0 bandwidth is what it is, for anything remotely high res it's MJPEG

ah that makes sense

on a semi-related note - there is eUSB2V2 that "solves" this issue by allowing one side of things to effectively be overclocked up to 10x the speed, while the other stays as-is, so that you could have internal cameras that can transmit raw high-res data without needing complex receivers unlike if they used USB 3.0

And yes we've made MJXL with an FFMPEG hack before

Doubled my processing speed on single thread and did a 0.512 GB in 16 minutes and 27 seconds.

Speed is not what is impressive about it, breaking basic math is

Even if it would take a billion years per block, it would still be mathematically impossible

would be really neat to store and retrieve data in the cpu branch prediction between the encoder and decoder i'm sure some security researcher has figured out how to do that

Interesting, what I was doing before was very slow. So, this speed up is huge. I was mostly focused on compressing binary and that is a fringe benefit of my process. So, even though I can lossless compress binary my speed and rate of compression isn't useful? (Compared to other methods?)

Speed is nice, but if you are achieving such compression, that alone is worth it

I have to more and more believe this is same story as Sloot Digital Coding System, from misunderstanding shared dictionary to intentional cheating

Yep... It's the modern version of alchemy, snake oil and perpetuum mobile inventions, I guess.

so brotli >:3

(I joke, a compression algorithm with a shared dictionary is perfect for use on the web where a lot of data will be html/js/json)

There is a more fancy shared dictionary: next-token-prediction language model as the probability distribution of arithmetic coding

The bit size of every encoded token is the cross entropy of model prediction vs ground truth (one-hot)

If a dictionary is used you have to account for the space used in the dictionary as overhead. My reports are the actual storage size and do not require a dictionary each block has enough data to self describe and self define back to the original value.

That's... What they're doing. They're getting it reviewed by someone else

He think his secret power worth 500 euro

Not likely. The more messages posted, the more they read like a formulaic troll tactic.

https://www.gamedev.net/forums/topic/11259-mp3-beating-compression/

if you havent seen this ancient forum thread yet you should haha

``` Original post by kieren_j Well actually, I posted that infamous first message because thats what my compressor produced - then I found a little (!) problem in my code that meant instead of reading and writing all the bits of a byte, it was only using the first. I didn''t really have the guts to say it went wrong like that... I still feel really bad though. --------------- kieren_j ```

I am just updating the chat and potentially interested parties as I make breakthroughs. Since, I last discussed in the chat I am now processing around 1000 blocks a second where as before I was only getting 10 blocks a second. I am not trying to troll anyone. Just enjoying the process and learning as I go. Some people give good constructive feedback. So, I continue to participate in dialogues.

the only feedback you've gotten is that what you've claimed is impossible

people have been trying to get through to you with this

How did you do the benchmark I asked you to do (taking 1000 blocks of 512 bytes from /dev/random and trying to compress them), if you're not using linux but Windows? I haven't used Windows in 27 years but last time I checked it didn't have a /dev/random...

I might as well add now that "Linux is not my OS" is a bs excuse. Unlike the safely guarded bit-discarding method, it is open and available to anyone.

This is his first time that he censored his log screenshot to hide the data size If he has the willing to modify the log, he can fake the data size too

looks like <@991011772381163581> deleted all those original messages? Maybe they finally figured out the flaw and are now very embarrassed and hiding the traces

His log is already censored before he deleted message

No, I was told not to use that forum. So, now I am in this forum. There is random block generation and it doesn't need to be truly random just random enough. A million blocks is a large enough sample size.

That is why I wrote code to test my algorithm. I wouldn't be pursing further if my code was not producing the results I claim.

https://en.wikipedia.org/wiki/Mechanical_Turk#/media/File:Kempelen_chess1.jpg

In the day without computer, at least you need a very clever player for such magic show

I don't think I fit inside the code... If my algorithm didn't work it wouldn't be able to compress a single block. Random bits are not supposed to be compressible.

ok I'll play along one more time. What does this block compress to? Show me a hexdump of the compressed version please. ``` 9961fe4013b95b1de9c2305990fdc817cc9a46d35cb920065a866995d7bd bc87f0d74ce4e10e193f7a9da1d11542e80ef1c14c494628dee873d7e5b8 a93cc5d4532c146c039fb001d15e6308832b43c95e288a5e293de9e564db 571d702c3d4fc386aca22c0e5f3c42643757977048be328579440a15f50f 243bcf670a1c4066c339bc82ea3b0b0326fbd503eef9a08f4cbc1b7c1015 66364c38d3e9eeb809f8bf21464449eb43449b628b317b97e09ef652fe26 e14a4db6fdb70e368bcd128a68dc463ae4b72171fb474eae70140849a947 e195c7d4307cfb2b7c5aa8eca70d93b4c110deffa18ff2bf41e2639a0dee 9e05553361fbb7528c3294a0b723eebb7bc3c0dedbcef1168844bc2690ac 4d49d238dc59e609c5fe0806e295640bac12d2bcc1a9d6042bde35678a42 3fd7cd1327dda3db9c389f6ba95644797ff144ad8b4751f69627ba3e1609 6a11e67e33a64679e1d9ac1e3b86c9428a577cb2f834449f360d43b89cd1 e581ee9edb3d4d5bbf45b0aba3aa5294adddf05db2e7f7518a7af6fa2353 8ef50a7e4b5811e20ae76a42f044231bb7cb8859290b9e0f2172dea2b337 d8716b1ab041da64d0bcb3da8ac6bf9c0204bacbf4789f06984afad2002d 58b678698ec0e80229d71d4be8718aaf2012053aab49a6fa15f163374c95 a151e9eb1d9f71b637df2f5d02d62881280d7e9c95a0b20cc5c40efd23f6 0082 ```

write your encoded data into file system, and show the file properties by right click menu You claim you are using Windows

``` <!doctype html> <html> <head> <meta charset="utf-8" /> <title>Download 1024-bit Random File</title> </head> <body> <button id="downloadBtn">Download 1024-bit random file</button> <script> function downloadRandomBitsFile(bits = 1024, filename = 'random-1024bit.bin') { if (bits % 8 !== 0) { throw new Error('bits must be a multiple of 8') } const byteLen = bits / 8 const arr = new Uint8Array(byteLen) crypto.getRandomValues(arr) const blob = new Blob([arr], { type: 'application/octet-stream' }) const url = URL.createObjectURL(blob) const a = document.createElement('a') a.href = url a.download = filename document.body.appendChild(a) a.click() a.remove() setTimeout(() => URL.revokeObjectURL(url), 5000) } document.getElementById('downloadBtn').addEventListener('click', () => { downloadRandomBitsFile(1024, 'random-1024bit.bin') }) </script> </body> </html> ```

This is a random file generated by `JS_RNG.html`

I ran it for intellectual curiosity. It compressed. I don't want to have an issue with reverse engineering so I am keeping the compressed version to myself. I have someone I somewhat trust to evaluate my code. Hopefully I find out within the month. Thank you for taking the time to respond to me. I am not seeking validation from anyone on this discord. Thank you for your constructive feedback.

Did you compress the ASCII text or binary?

and compression is one thing, the most difficult part is decompression <:KekDog:805390049033191445>

Hex is just 4 bits... And, both hex and bits are just numbers... Compressing numbers and decompressing numbers.

Yes, it is a lossless process. I don't think you can compress numbers lossy and have that be useful for files.

You confirm your transform from ASCII text to binary are correct?

That is the reason to use file system

ASCII is significantly larger per character than 4 bits. Hex is an INT value...

ASCII: 7 bits per character (commonly stored in 8-bit bytes). Hex: not a character encoding — it's a numeral system. Each hex digit represents 4 bits; two hex digits commonly represent one byte (8 bits). UTF‑8: variable-length encoding — 1 to 4 bytes per Unicode code point: 1 byte (8 bits) for U+0000..U+007F (same as ASCII), 2 bytes (16 bits) for U+0080..U+07FF, 3 bytes (24 bits) for U+0800..U+FFFF, 4 bytes (32 bits) for U+10000..U+10FFFF.

In previous screenshot you read text from cmd, write text to cmd

\> numerical mumbo jumbo ChatGPT is the compressor.

I don't want the transform chain [ASCII (used by cmd) -- hex -- binary] I want directly read binary file, directly write binary file

Meanwhile, real compressors.

Still optimizing the code working on speed. I want to be able to get similar speeds to MB/s like other compressors. I already did a separate compressor and de-compressor code. Printing the compressed hex and using it to decompress in the de-compressor code and comparing the original input to code 1 and the final output of code 2.

That's a very verbose way of saying nothing at all.

I have limited development time. I can remake the code to their liking or I can keep experimenting and optimizing the process...

The only possible situation is: He confuse - ASCII text annotation of hexadecimal digit (nibble in byte) - hexadecimal digit (nibble) - binary file (byte)

An amazing result as even sophisticated methods can spectacularly fail given antagonistic data. ```22406 sharnd.0.bin 22614 sharnd.0.bin.rz 22406 per.6 74368 per.6.rz```

Most general data compression schemes have a fallback to just encode raw bytes with some minimal header overhead

if you don't have that, then yes, random bits can blow up

OK, keep the compressed version to yourself. Now pass it to your decompressor and check what output you get from it.

It wouldn't be the first time someone thinks they have found something amazing, only to then understand they missed something big. I remember someone once claimed super lossy image compression performance, outperforming anything else by a factor of 3 or more. Turned out he was measuring compression rate in bytes per pixel while everyone else was using bits per pixel. Oopsie!

(actually I don't remember this, it's a story I heard from Touradj of something that happened during JPEG 2000 development)

We've been chatting in the voice channel occasionally, last night I explained the concept of VarDCT

Who wants to bet the compression he gets is due to counting the hex input as 1 char = 1 byte of data but then checking the binary sized of the "compressed" data buffer?

No, its all bits each hex is just 4 bits. Hex is human readable whereas binary is not. The completed code will work on bits only when done and not have the overhead of converting hexadecimal to binary.

"Trust me bro, I don't know ASM or C or C++, I also don't use hex viewer, but I can transform bits correctly"

https://www.youtube.com/watch?v=2sRS1dwCotw

> There is random block generation and it doesn't need to be truly random just random enough is your compressor tuned to the way your "random enough" random generator works?

it compressed how well?

do you see this 40% with your own random data or with any random data from any source?

if by "it compressed" you just meant "it produced an output" then, yeah

actually nvm, don't worry about it

Day 1 of working on my super good FPGA video codec: I said that I will be using 18-bit math, since FPGAs tend to have 18-bit multipliers. I'm realizing now that the most annoying part of implementing dct1d will be producing sqrt and cosine values of adequete precision without too big a LUT, without too many LEs, and without spending too many cycles.

not planning to unroll the multiplications then, I take it :p

we'll see :p

the chip I'm using has 144 18-bit multipliers, and there probably aren't *that* many other things I might want to use them for on the same SoC, so I'd feel comfortable using a large chunk of them.

To start with, I'll use a very advanced tool (Desmos :v) to figure out how many LUT values I actually need.

IIRC you can get away with basically N for a N-DCT

i.e. a N-DCT needs (N/2) constants + those for a (N/2)-DCT

(exception being the 2-DCT which needs 0)

at least, that's with the jxl-rs and libjxl impl

(and something needs a sqrt2)

mm I'll be referencing jxl-rs for this, I think. And jxl-tiny, considering what it's for.

pretty sure it uses the same DCT impl

this is such weird statement to me

<https://www.sony.co.jp/en/Products/LDAC/info/>

It is a weird statement because it is not based on reality

This is the old Debatte of analog vs digital, but extended into lossless vs lossy

what they're saying boils down to "lossy compression of a high-resolution signal is better than lossless compression of a low-resolution signal" which is not a fair comparison to make

it seems sony never confirmed what LDAC stands for

it's definitely a lossy codec, though Wikipedia says it's an acronym for "Lossless Digital Audio Codec"

Tbf, if the lossy high res has the same bit rate as the lossless low res signal, it's a good comparison if you're goal is to show to shower that higher res is a good idea.

I meant bit rate with resolution

or bit depth

in the graphic, they're claiming the top right signal is better than the bottom left signal, essentially

sony's marketing claims are dubious but the previous editor's statement about them was also false

also it seems we don't really know what LDAC stands for, apparently some people assume it stands for 'Lossless Digital Audio Codec' which would be categorically wrong

I could elaborate about the dubious marketing and acronym that suspiciously implies lossless encoding but then I'd have to dig up a bunch of reliable secondary sources that talk about exactly this and I'm too low energy for that right now <:SadCheems:890866831047417898>

also, huffman coding <:BanHammer:805396864639565834>

The part at the bottom suggests a bandwidth limit to me. FLAC may be lossless but detail is lost in transmission due to the network/decoder not keeping up. Lossy allows greater range in the same bandwidth Maybe...

there is a bandwidth limit since it's designed for bluetooth yeah, and somewhere else it says lossless encoding is not feasible for bluetooth because of that limit. but as for the claim it's making, it's saying lossy encoding of *high-res audio* is better than lossless encoding of *high-quality* content, and it considers *high-quality audio* to be lower resolution/bit depth than *high-res audio* you can see in the named graph, that the highly quantized ones at the bottom are called "high-quality" and the lesser quantized ones are called "high-res"

I... Think I understand? High res has larger range, but high quality is quantized less

I think what they're saying is akin to saying a 2000 x 2000 JPEG ("lossy High-Res Audio content") is better than a 1000 x 1000 PNG ("lossless High quality content") that's nearest-neighbor upsampled to 2000 x 2000 again

which is true, but a lossless representation of a downsampled signal is not to be considered lossless relative to the source

and unless you're sure what they mean with high-quality and high-res it's really confusing

also what happened to that ultra hdr article? was it so bad it was decided to just delete it? <:KekDog:805390049033191445>

oh it's back? I could've sworn I couldn't find it just a few days ago

oh no right, it was this. so effectively, "yes"

Key point: They think lossy 192k Hz > lossless 48k Hz

lmao what

(1) isn't even true either

it isn't?

https://people.xiph.org/~xiphmont/demo/neil-young.html

Totally bullshit

It's marketing for "audiophiles"

The people at xiph also published a 30 minute explainer some years ago regarding this

https://wiki.xiph.org/Videos/A_Digital_Media_Primer_For_Geeks

https://wiki.xiph.org/Videos/Digital_Show_and_Tell

Most audiophiles are uneducated, can't understand Nyquist-Shannon sampling theorem (why 44k Hz sample point can represent 20k Hz sine wave, won't become triangular wave or square wave)

also, Bluetooth LE Audio is genuinely a great technology lc3 (the mandatory codec) has good quality at low complexity and low latency and auracast is great for sharing audio, accessibility at venues and (soon™) announcements at airports and stuff

think it obsoletes most if not all the proprietary codecs

The only real problem with bluetooth is with its massive quality limiation for bluetooth headsets

everything else is pretty great

LE Audio fixes this (to some extent)

oh, and that one piece of the headset has double the energy usage due to it transmitting audio to the other piece

this isn't actually necessarily true in practice

But for the most part is how it is integrated

https://assets.qualcomm.com/rs/385-TWS-803/images/whitepaper-the-evolution-of-true-wireless-technology-2020.pdf

not only dubious, it's an outright lie

and there are LE Audio tws that still operate like that instead of acting as 2 devices

Is there any headset that implements this?

anything that uses a qualcomm soc, presumably?

LDAC marketing is contingent on the fact that 990 BT audio exists, but not it's feasibility

LC3 chad

I like how LC3Plus is mostly useless now since they punted all the goodies to LC3

LC3Plus literally just offers a lower latency and 96khz audio pretty much

got these 2 days ago, zero issues with le audio so far

and they support fully parametric 10 band eq (some assembly required)

ill have to try them some time, but I dont have anything currently that supports LC3 properly, I just use it for my local speaker setup.

hm, will have to have a look then for a new pair then

my current headset, which is 5 years old, has 5x energy usage on the right piece due to aged battery compared to the left piece

which boils down to an hour max of listening

why do youtube's encoders incorrectly assume they can disproportionally reduce quality in dark areas?

or does it drop quality equally everywhere regardless of brightness, and we're just more sensitive to those lower brightness differences? I know the latter is true, but I assume encoders usually account for that?

do they assume dark things are less *important* maybe?

The whole point of the transfer curve in a colorspace is to take the very nonlinear human perception of brightness into account...

but it's kind of hard to know the actual viewing conditions. If the entire image is dark (not just a dark region in an otherwise bright image), and the room is also dark, then your eyes will adapt and you are much more sensitive than in a condition where bright stuff is also in view

so if you zoom in on a dark region until your entire screen is only dark stuff, you will not only see more artifacts because of the zooming, but also because of adaptation

if encoders have to take into account the full dynamic range of humans including adaptation, an insane precision is required in the darks since at some point it's not just cones but also rods and we can almost detect individual photons with those when it's sufficiently dark

hm that makes sense

still, dark segments in youtube videos are so bad that creators sometimes have to artificially boost the brightness so half the frame's content doesn't get quantized away

it sort of reminds me of using a low intensity target in jxl

what you said should apply to encoders in general, but I haven't noticed images with these issues, nor videos of dark areas I've taken myself, so it doesn't really explain why youtube specifically does really bad in dark scenes and more or less fine in well lit scenes

> we can almost detect individual photons with those when it's sufficiently dark also, that's incredible, I didn't know our eyes were anything close to this sensitive

Once or twice I've ended up in countryside without a light, it gets to a point you can practically see noise like a camera. Blotchy areas of grey in otherwise pure black

The quantizer sees a difference of 1 regardless if it's bright or dark, even though visually the range is weighted 80% towards bright colors There's good demos by the AV1 guys https://gitlab.com/AOMediaCodec/SVT-AV1/-/blob/master/Docs/Appendix-Variance-Boost.md

overfit PSNR

I think there are a few things working together to make video codecs often very bad at dark scenes/regions, turning them into a blocky mess: - the transfer curves typically used (Rec709 etc) don't really have a lot of precision in the darks to begin with - using tv-range instead of full range further reduces the precision - YCbCr is perceptually not a very good colorspace, especially in the darks you actually need the contribution from the chroma components to make gradients etc look good - doing 4:2:0 and using aggressive chroma quantization (both are typical for video codecs) makes this even worse - still too much PSNR optimization in video codecs, and PSNR is basically not sensitive to banding/blocking issues — no per-pixel error metric that doesn't consider the spatial aspect can detect that since these artifacts are creating low per-pixel error yet they are very obvious due to the spatial structure

its cenrosed

that's my name (idk why that shows there)

is there an algorithim that uncencors it?

pretty sure ai can fill the gaps

AI is not magic, the best it could do in this case is guess

... Why would they help you uncensor their own image?

I thought that maybe they were interested in that

two chatbots (Phind 70B and GPT-4o-mini) claiming JPEG XL supports up to 16-bit per channel 😵‍💫

oh good

crazy how JPEG was already so well adopted before it was invented

both also claim webp supports a bit depth higher of 8 bpc in those tables btw...

Not *technically* wrong if it meant integers

if it's integers, wouldn't that make it 24-bit for VarDCT and 32-bit (unsigned) for modular?

Just 24-bit

modular is implemented with fp32 so it only works if the integers aren't too big

Theoretically there's nothing stopping f64 in Modular, just spec and implementation limits

interesting, I thought Modular supported using i32/u32 internally as it stands

it does

there's some postprocessing in fp32 in libjxl but it doesn't *need* to be done that way

https://discord.com/channels/794206087879852103/805176455658733570/812257462379872287

> JPEG XL can losslessly represent 32-bit floating point > data. For integer data, in principle up to 31-bit unsigned > integer or 32-bit signed integer data can be stored losslessly, > but the reference implementation libjxl is limited to an > integer precision of 24-bit since it internally uses a 32-bit > float representation in most of the encoding and decoding > implementation.

is jxl-rs limited to f32 for everything as well, or does it already support integer channels that an encoder could produce in the future?

I am not sure I see much of a point of having more than 24 bits, tbh

the only use case I can think of is losslessly recompressing other formats that support larger bit depths. not that I know whether there are any

like for archival

JPEG 2000 supports lossless up to 38-bit integers 😄

oh damn

I can respect this, also knowing you might be the one that would have to write new SIMD code for those bit depths 😅

As far as I understood, jxl-rs was targeting Level 5 of the spec for faster decoding in browsers (Using float16 instead of float32). Any extra support is just bonus, but I could be wrong

I don't think we ever *explicitly* decided that, but I will say that if we want to try non-fp32 things out they will be fp16 before fp64 or int32 😛

Yeah, maybe I was getting mixed up with the simple render pipeline or something

*I have a suggestion* https://doc.rust-lang.org/std/primitive.f128.html

/j

fp16 should be especially good on aarch64, and on x86 we can be fast enough with fp32 on sufficiently modern machines, I believe

f16 would be neat for sure

(my laptop does ~75MP/s on a single thread, that's fast enough :P)

(with libjxl, I mean)

Clang or MSVC?

clang

Ah good, because Clang is still between 20-30% faster (Or 200-300% for fast lossless)... One day we'll get the github actions using it instead

Meanwhile, my poor poor CPU xD ```JPEG XL decoder v0.12.0 6efa0f5a [_AVX2_] {Clang 20.1.8} Decoded to pixels. 3840 x 2160, geomean: 29.503 MP/s [27.203, 30.894], 20 reps, 0 threads.```

That's desktop by the way, time has not treated first gen Ryzen well

is that vardct?

Yeah

the difference between my desktop and my laptop is not too big (on one thread)

wow, ok xD

I guess it is an 8 year old cpu

I know first gen Ryzen takes 2 clock cycles for AVX2, so that's probably a good chunk of it

yeah I believe they only have 128 bit SIMD execution units

One day I'll do a comparison of my 8 year old phone vs my 8 year old desktop haha

will want to try the 9xxx series eventually, those have full 512-bit SIMD execution units IIRC

Oooh, I know I wanted to see fast-lossless on AVX512 since AVX2 was a 5x speedup, also a threadripper/EPYC since we discovered the libjxl 'port' has worse thread utilization compared to standalone

maybe once we're done with the decoder side of jxl-rs we'll start adding some encoders 🙂

Right now I'm unravelling Jyrki's VarDCT code. I think I just got it performing better than 0.8 (finally), better quality and less density but needs more tuning

I hope I might be able to contribute somehow. I probably would have already, I don't really like working in C and I don't know it that well

there's plenty to do! 🙂

that reminds me, if I have any time between school and the hobby video codec, I want to try to make a slow but flexible encoder that makes decisions using a user provided script (probably Scheme or Lua or something) would make quality vs size tradeoff experiments easier, though you wouldn't get any data on the perf impact it'd have in practice

Similar to MA trees or something else?

the idea would be that all decisions made by the encoder would be done by a script, so more like the code in libjxl that decides which MA tree to use, decides how much to quantize stuff, etc.

I've been thinking about making specialized encoders for screenshot tools in apps, like for games, where game content is encoded in VarDCT but the UI layer is encoded losslessly and alpha blended on top currently I'd be more interesting in making an analyzer though, something that lets you view block selection and MA tree decisions over the image

the analyzer also seems easier since it would not necessarily require decoding anything to pixel data, I can use a complete decoder library for that part

some things I really want to try out encoder side: - find optimal k-way splits along each MA tree property, maybe for a combination of 2 properties (would make for a very fast modular decoder, and I think it should be quite powerful) - figure out how to train some small neural network to replace the current encoder decisions (biggest issue is figuring out what training data to use...) - try to see if we can use some gradient descent based method to optimize a bitstream more or less from scratch, like https://arxiv.org/abs/2412.00505

I forgot, I got Sapien to make me the slowest build possible. Emulated AVX (I think) ```JPEG XL decoder v0.12.0 e5943b1a [_EMU128_] {Clang 21.1.4} Decoded to pixels. 3840 x 2160, geomean: 19.515 MP/s [18.772, 20.104], 20 reps, 0 threads.``` ```JPEG XL decoder v0.12.0 6efa0f5a [_AVX2_] {Clang 20.1.8} Decoded to pixels. 3840 x 2160, geomean: 29.503 MP/s [27.203, 30.894], 20 reps, 0 threads.``` Honestly surprised it's not more of a difference, guess that 128bit unit really struggles

I know Jon has talked about a JIT MA tree decoder, but it'd cause security scrutiny

I also mentioned that a few times

as one of the things that would be very cool but we'll probably never do 😛

Also, I found out recently that libjxl does background and screen content detection in the encoder, but screen content doesn't seem to do much and background detection adds dots(?) but then encodes over them anyway (I think)

oh another thing I wanted to explore was recursive patches. for example, for a large text screenshot, you get one layer of patches that has all the letters, and then you patch-copy those onto a new 'free' frame to create patches for common words, and then in the final image you can in some cases replace entire words with a single larger patch instead of with individual letters, which should be cheaper to signal

Still lots of mysteries to unravel in libjxl, yet alone the new techniques jxl-rs could allow haha

The patch locations are compressed, so I think it'd only be saving bytes compared to compressing the patch frame itself more efficiently (Right now it's an effort 2 encode IIRC)

speaking of the decoder side, once it's in a really good or "finished" state it would maybe be cool to look into having [Knusperli](https://github.com/google/knusperli/) like/style decoding for transcoded JPEGs?

Didn't Jyrki say it was in the original design, but was just too slow?

no/yes

that was for the VarDCT spec

I'm just talking about transcoded JPEGs

the way to decode VarDCT is way more tightly defined then regular JPEGs

hmm, you might be right. I'm not sure how patch locations for individual letters would be signaled, if they are signaled as the relative distance from the previous patch then the letters from identical words in different places could be LZ77 deduplicated, but if it stores absolute coordinates then LZ77 won't deduplicate those I don't think

Ah right, I found the old discussion https://discord.com/channels/794206087879852103/805176455658733570/1345882517005275308

another semi related thing: inter-frame compression of animations where only a thiny bit moves by patch-copying over large chunks that don't move from the previous frame

would be cool for recompressing some GIFs

Right now lossless struggles with low color content, which is what patches usually are due to text. The time doing recursive patches could be spent improving overall compression instead, either making the patch frame a few percent smaller, or the entire image a few percent smaller, instead of saving 1KB~ of compressed coordinate info

Of course that depends how much text and how many patches, but then that makes the few percent increase in savings too

We kinda already have that with differential transparency, though libjxl doesn't do it automatically, only by copying existing GIFs/APNGs

maybe, but it wouldn't hurt for me to experiment with this separately (I wasn't suggesting for others to work on this, it was something I would like to explore eventually)

Also this https://github.com/libjxl/libjxl/issues/3134

nice, I didn't know it did anything like this already

how's the compression compared to APNGs/GIFs generally? is it competitive? 👀

Due to that, only at effort 9 or 10 get gains over GIF. Pallete sorting is a big problem, but for APNG it does pretty well

oh that's not bad

reinvent motion compensation <:galaxybrain:821831336372338729>

we'll have to, jxl can't natively do that 😔

> The Patches coding tool exploits such redundancies. It works as follows. Up to four previously coded frames can be ‘saved’, i.e. the frame header indicates to a decoder that the contents of a decoded frame must be buffered for future reference. There are four ‘slots’ available for this. The Patches tool allows referencing these frames, selecting a rectangular region in them, and then adding that ‘patch’ to the current frame in one or more arbitrary positions. Different blend modes are supported, and when there are... <:tfw:843857104439607327>

this is why we need MPEG XL <:Stonks:806137886726553651>

https://discord.com/channels/794206087879852103/794206170445119489/1178541355639783424

knusperli's smoothing behavior looks unnatural. seems an appeal thing over fidelity

Hi, i was wondering if there are options to recompress with a higher effort later down the lane for already processed .jxl? (libjxl) Ho and if there will/could be breaking changes that could make current .jxl decompress differently later?

no encoders currently optimize existing lossy encodes. different decoders can make different rendering decisions, whether related to bugs or exercising the flexibility of the spec.

Sorry i forgot to precise that it's for lossless compressing.

And if we don't take into account metadatas as i've backed up those before converting and can set them back. (not sure if this is an option directly using libjxl or if it's out of scope, it's only 1-2kb per metadatas file after all)

For lossless you can already do this, cjxl takes jxl input so you can do things like `cjxl input.ppm -e 1 -d 0 fast.jxl` and then later `cjxl fast.jxl -e 7 -d 0 slow.jxl`

Alright good to know thanks. I'm consistently getting 30% space savings with effort 5 for now!

how does it compare to jpeg2png ?

it's wayy more subtle but doesn't have the downside of causing a lot of images to look worse/oversmoothed. only problem is the core JPEG decoder in that project seems to be problematic in some way because it's decodes don't seem to have the right colors for some reason. also it doesn't decode in a high precision like jpegli/libjxl does.

thanks

Fake news

I meant the spammer who faked Trump

is there still a thread going for "lossless examples where other codecs perform better"? I found one but it says only for screenshots

There is one for high effort. https://discord.com/channels/794206087879852103/1187313089612369991

the difference between avif and jpeg from nintendo.com

ohhhh I'm stupid

there's only a difference on my end because I keep forgetting that my viewer has no color management 😭

doesn't their website also serve JXL?

no

huh I remember some random domain of theirs serving JXLs

guess it's not the main website

It was cloudinary and someone changed the url from auto to JXL

Nintendo of Europe is a Cloudinary customer so they might serve jxl, not sure if they have it enabled though

will there be a point where existing customers have support implicitly enabled if they have it on auto? instead of having to explicitly enable it? that's how it is now right

I can't think of any reason not to

I'll give you a reason: we still have lots of contracts where our revenue directly depends on bandwidth consumption. Enabling more efficient formats is shooting ourselves in the foot, our revenue will just go down. So we do that only by default when customers are switching to the new pricing model, which is based on impressions, not just on bandwidth.

ohhh yeah I remember reading about that, I'd forgotten since

that's completely fair

forcing it with a redirector works, but I think some things are under normal scenarios

for all this time I was under the impression that "chroma from luma" was just a different term for chroma subsampling... it's something different

it's predicting chroma from luma, which in jxl we actually only do when there's no chroma subsampling because that just complicates such a prediction.

look what I found, must've fallen off the back of a truck. for legal reasons, I really appreciate the choice of wording and layout, and the positioning of the logos could not have been better if I were to review it critically in a matter that constitutes fair use as recognized by the united states copyright law which is enforced on this social medium

no one tell them this is on the internet archive <:KekDog:805390049033191445>

I was actually hoping it was on there "officially" with digital controlled lending so I could reserve it for a while, but no, someone just put it on there as a pdf

lol nice

ok no snitching

of course I deeply condemn this

you wouldn't download an open standard

presumably this retains a lot more color than chroma subsampling, how competitive is it in terms of compression?

it doesn't look like I can control these with cjxl

Control what?

chroma subsampling and chroma from luma

Subsampling is JPEG transcode only

oh!

does the format not support it or is it just not in the encoder?

I really dislike chroma subsampling so you don't hear me complaining either way

*Technically* you could make a bastardized encoder that does, but you're not meant to

that's actually great

There's a cool Xiph article explaining chroma from luma for AV1 https://people.xiph.org/~xiphmont/demo/av1/demo1.shtml

is this by using modular tools or can you actually use the coding tools from lossless JPEG transcoding on their own?

I was under the impression that JPEG transcoding was all or nothing, and that you couldn't use its coding tools on their own

there is not separate lossless jpeg mode

it's just vardct

some coding tools won't be available if you use subsampling IIRC

that's interesting

does this mean we can create otherwise normal JXLs that use Huffman coding instead of ANS <:KekDog:805390049033191445>

oh you explicitly can lmao > At a high level, JPEG XL’s entropy coding method > works as follows. Every entropy coded stream can use either > prefix coding (Huffman coding) or ANS coding

Huffman is much easier for hardware

Double checked the standard. chrome subsampling is only allowed for YCbCr frames 🙂

ah, so they really don't want you to use this for anything other than JPEG compression ^^

https://discord.com/channels/794206087879852103/803645746661425173/1362825741728874597

> Chroma Subsampling is a sledgehammer when you should be using a scalpel YES thank you

CfL exists as shaders, I use it when the quality is not very good, it works well on AVC as a post-algorithm

minesweeper anyone?

it's interesting to use only one block type for the entire image. it gives you an idea of how the they work because you see the artifacts they produce in places where they usually wouldn't be picked

you can see how the hornuss transform bases pixels in the 4x4 chunk on the average values in that chunk for example

> Associated alpha (as in e.g. the OpenEXR image format) is more expressive than unassociated alpha (as in e.g. the PNG image format) since it allows modeling layers with light-emitting but transparent pixels, such as a candle flames or reflections in glass. Such pixels would correspond to an alpha value of zero and color components that are non-zero. what would an alpha of zero and non-zero color values look like?

I also don't intuitively understand how associated alpha lets things appear emissive while unassociated alpha doesn't

> what would an alpha of zero and non-zero color values look like? when you do this does the effective transparency become a function of the brightness/intensity of that pixel, and is that why it lets you create an effect of emissiveness maybe?

basically acting like a different blend mode

Yes it only becomes useful when layered on something else For modelling light transport, it's very intuitive as well (make a plane in blender, set transparency to the alpha channel and color to the emission channel), and it's very easy to generalize to having three alpha channels (RGB) for filtering effects Once you start getting filtering effects in though the only way to actually be accurate is considering every wavelength individually, but it's close enough for most things

associated: out = (1−alpha)·bg + fg unassociated: out = (1−alpha)·bg + alpha·fg

so with alpha=0, associated: out = bg + fg (emissive); unassociated: out = bg

candle light

another one is something like a gas it has no physical solid, you can't see it unless something is behind it

it has color and energy but no surface for it to reflect on until you put something behind it

okay I understand that. so basically, it's like allowing to you *add* intensity of the foreground pixels to the background pixels, instead of numerically replacing some ratio of the background pixel intensity with the foreground pixel intensity. and as a result, if you *do* want to replicate this 'partially replace' blending that unassociated alpha does, using associated alpha, your pixels need to already be multiplied by the proportional amount of alpha, hence "premultiplied"

I think I finally understand, thanks!

now I also understand why associated is a better term than premultiplied, you don't *need* premultiplication for associated alpha to be useful, if you only have emissive elements you don't do it at all. I'd heard that before but never understood why

I just noticed that in the captain disillusion video about the alpha channel, in this scene, none of these images are truly transparent, they all have a different transparency checkerboard which implies they're part of the images instead of rendered by the browser. just like how in practice you'll find a considerable amount of images with false transparency <:KekDog:805390049033191445>

now someone will demand a new metadata chunk in PNG and JXL to declare which size/colour the checkerboard should be

if we're redesigning the alpha system, I propose a new system with an alpha, beta and gamma channel that define the image subject's transparency to alpha, beta and gamma radiation, respectively

for what it’s worth, OpenEXR supports per-channel alpha (AR, AG, AB)

https://openexr.com/en/latest/TechnicalIntroduction.html#channel-names

ah that's for filtering like adrian mentioned earlier

neat

Add a layer of JXL art :>

would be cool if browsers automatically picked the checkerboard shade that would give the most contrast

my experience from ~2006 about PNG alpha optimization for Google Maps: ~32 levels is roughly enough, one level needs to be 100 % opaque, one level needs to be 100 % transparent, ideally a few more levels for highest transparency values -- I think I ended up adding 1 (or 3, don't remember) levels more for high transparency values, to get drop shadows without banding

Very interesting, I didn't know you worked on Maps, yet alone that long ago

you were talking about subsampling the alpha channel right

i wonder if you could get better gains by doing like 32 transparency levels and palette encoding them, then?

I had typed out about 5 or 6bit alpha, but pallete should already be doing that either locally or globally, so I don't think it would help much

...you can use bit depths lower than 8 in modular?

huh

or do you mean effective bit depth, by not using more than some amount of values

Both I guess

Both are possible and will end up encoding more or less to the same thing

it's a bit different on quantization? With palette you could "quantize" to any free value, which could exploit the non-linear usage of alpha, where some ranges are more used / important than others (where just dropping bit-depth would lead to a consistent drop in precision across the entire range)

Yes, and also it doesn't need to be an integer power of two

NASA's so-called PNG

actually, this might be Firefox's fault

yep

is it possible (or rather probable) for a true lossless image to still show up as 92% quality with imagemagick identify?

What does that even mean? JPEG quality?

I believe it tries to guess JPEG quality from the artifacts or quantization yeah

this is also interesting, there are no pixels with a value lower than three in the green channel

might be an outlier, a different image of theirs doesn't have that

What's the link to the image?

the png download on this page: https://science.nasa.gov/asset/webb/cats-paw-nebula-nircam-image/

direct link: <https://assets.science.nasa.gov/content/dam/science/missions/webb/science/2025/07/STScI-01JY2AYZ2PYBDDSMX83K5248DD.jpg/jcr:content/renditions/Full%20Res%20(For%20Display).png>

they also have a JPEG variant (right) which has much more obvious compression artifacts, but is also reported as having a quality of 92

> revealing a subset of mini toe bean-reminiscent structures Best NASA quote so far

first signs of arirals

guys, I think if we all stop working on jxl and pivot to cloning then jxl development will go a lot faster since we can clone Luca

off-topic but is that [IPvFoo](https://addons.mozilla.org/firefox/addon/ipvfoo/) I spot in the top right of your screenshot? Some other extensions in a similar vein of functionality if you interested include: [HTTP Version Indicator](https://addons.mozilla.org/firefox/addon/http2-indicator/) (shows if a site is using HTTP 1.1 or 2 or 3) [IndicateTLS](https://addons.mozilla.org/firefox/addon/indicatetls/) (shows if a site is using TLS 1.2 or 1.3 or etc)

yes it is :)

how to spot a good website:

perhaps YUV420

imagemagick looks at the quantization matrices -- it is possible to fool it by sending a very fine quantization matrix and then not use -3,-2,-1, 1, 2, 3 -- and compress a lot with crappy quality but get 100 as a reported quality

interesting

so that PNG was definitely derived from a lossy compressed image then :/

A PNG doesn't have quant matrices though

from what I know identify looks at the pixel data directly, I assumed Jyrki meant that it estimates the matrices from those

It doesn't, it looks at the quant tables of JPEGs

from testing, it seems to *always* report 92 for PNGs

imagemagick's identify looks at the quant matrices, the image can be fully black (the same pixels) and exactly the same for different qualities, still image magick will compute something resembling the original encoding quality

I know, I'm saying it always reports 92 for PNG files, which of course don't have any quant matrices.

oh, well that's confusing

ah yeah for PNG files and I guess other lossless formats, it pretends it is quality 92 so that will also be the quality it uses by default when you convert png to jpeg, since the default is to keep the quality

which kind of makes sense, if you want to avoid generation loss then it's better to avoid requantizing with a different quant table, though of course even better is not to decode to pixels and then encode from pixels again

wait, so you're saying it reports that quality for PNGs so that the last three steps in the PNG -> JPEG -> PNG -> JPEG pipeline are lossless?

or am I misunderstanding

"default is to keep the quality" I don't think refers to anything lossless I assume it's just so that converting between different lossy formats has a similar relative quality level

oh got it

"format=gbrp", yes, this, It's just hilarious that without this parameter you can't save RGB24 downscale. It's literally "YUV420 in core" for anything when converting video to PNG the ffmpeg.

when suddenly, suspenders

I like how Nano Banana Pro looks sort of like a traditional image codec generation loss test

a couple of them do

giggling at the thought of putting lossless WebP in there, calling it something else and seeing how many people get suspicious

some in the comments have hypothesised that what we’re seeing is the “this is AI-generated” watermark

which sounds plausible

?

uh lossy WebP I mean <:galaxybrain:821831336372338729>

<@1028567873007927297> you might find this paper interesting re noise shaping https://doi.org/10.1109/ICIP.2010.5651757

They set up spatial noise shaping as a convex optimization problem: minimize the l1 norm (for sparsity) of the transform coefficients (wavelet for them since they're working with JPEG 2000) subject to the spatial error staying below a certain level per pixel, which means you can allow more error in places where it's less noticeable due to masking

Also they have this whole complicated scheme for it but I'm pretty sure the problem they set up is linear, and I bet a couple iterations of an interior point method would do wonders

(but ig technically what you want to solve is the integer linear problem where you're looking for the best quantized values, and maybe that's why they're doing their whole "let's see which bits we can zero out" thing?)

Earlier versions of lossy WebP would turn it to some nice smokes

We know what we must do https://x.com/dimenpsyonal/status/1994440264086147391

effort 11 takes it to 54 bytes

where is the 186B SVG ?

> Some manual tweaking takes it down to 42 bytes: `data:image/jxl;base64,/wq6Euh8YAASCAYBAGwASzhD2c0D1fBK9HQNJxh05Qk9XQQAuvIEIMEA` fitting that into a URL the length of a sentence is such a flex. it's a crazy feeling that these few characters form the whole image