|
|
haaaaah
|
2021-12-30 10:44:15
|
Context: https://discord.com/channels/587033243702788123/677266163994198020/925852795788214284
|
|
|
nathanielcwm
|
|
haaaaah
Context: https://discord.com/channels/587033243702788123/677266163994198020/925852795788214284
|
|
2021-12-30 11:20:12
|
yep
|
|
|
Scope
|
2021-12-30 11:52:34
|
https://twitter.com/richgel999/status/1476325003662667777
|
|
2021-12-30 11:52:39
|
Also in the future it may be worth trying a fast JXL-GPU encoder using GPU-friendly methods
|
|
2021-12-30 07:34:18
|
https://twitter.com/richgel999/status/1476623312658776066
|
|
|
fab
|
2021-12-30 07:38:31
|
Followed him on Twitter
|
|
|
Scope
|
2021-12-31 10:06:47
|
https://twitter.com/richgel999/status/1476710105047576576
|
|
2022-01-03 07:27:18
|
π€
https://github.com/lvandeve/lodepng/issues/155
|
|
2022-01-03 07:44:41
|
Fast paths for encoding/decoding would be interesting, also maybe it could be done even more efficiently than in fpng
And on the one hand it would be more useful to promote and make something like this based on new formats (considering that Lode is also a JXL dev), but on the other hand - PNG is still the most popular and common format
|
|
|
190n
|
2022-01-03 07:54:49
|
why would lodepng, a png encoder/decoder, integrate a different png encoder/decoder?
|
|
2022-01-03 07:55:18
|
lodepng description:
> PNG encoder and decoder in C and C++, without dependencies
|
|
2022-01-03 07:55:52
|
maybe they could add a fast decoding path for fpng-produced files since there's that ancillary chunk as the submitter of that issue stated
|
|
|
Scope
|
2022-01-03 07:57:15
|
Yep, just pure fpng integration is not very good, but using similar fast path methods would make sense
|
|
|
190n
|
2022-01-03 07:57:45
|
it doesn't sound like that's what they're asking for <:Thonk:805904896879493180>
|
|
2022-01-03 07:57:53
|
next someone will ask lodepng to support qoi <:kekw:808717074305122316>
|
|
|
Scope
|
2022-01-03 07:58:34
|
https://twitter.com/richgel999/status/1478036725687541765
|
|
|
190n
|
2022-01-03 07:59:24
|
fair, the issue is a little unclear. i guess he's referring to fpng the "format," not the library?
|
|
|
Scope
|
2022-01-03 08:04:02
|
Except that Lode will need to spend time on this implementation in the old format instead of improving and working on JXL
|
|
|
190n
|
2022-01-03 08:05:05
|
there's already been work done on detecting fjxl-produced files and using a faster decode path, right?
|
|
|
Scope
|
2022-01-03 08:07:14
|
Yep, Luca and Jon do this in their spare time
|
|
|
190n
|
2022-01-03 08:07:58
|
<:Stonks:806137886726553651>
|
|
2022-01-03 08:08:02
|
chad shit
|
|
|
Traneptora
|
2022-01-04 02:01:05
|
what is fpng, is it a backward compatible png extension?
|
|
2022-01-04 02:01:39
|
how does it work, does it just encode the idat in LZ4 instead of Zlib or something as an ancillary chunk?
|
|
|
Scope
|
2022-01-04 02:05:15
|
Encoding - yes
<https://github.com/richgel999/fpng>
```The files written by fpng conform to the PNG standard, are readable using any PNG decoder, and validate successfully using pngcheck. PNG files written using fpng can also be read using fpng significantly faster than other PNG libraries, due to its explicit use of Length-Limited Prefix Codes and an optimized decoder that exploits the properties of these codes```
|
|
|
Traneptora
|
2022-01-04 02:56:06
|
> Also, the code has only been tested with `-fno-strict-aliasing`
|
|
2022-01-04 02:56:07
|
<:Madge:859968580996956200>
|
|
|
|
veluca
|
2022-01-04 03:15:36
|
ahhhhh, aliasing rules
|
|
2022-01-04 03:15:48
|
one of the most abused language rules ever
|
|
|
Reddit β’ YAGPDB
|
|
Kagamiin~ Saphri
|
|
190n
lol zstd compresses one audio file i had lying around far, far worse than flac
|
|
2022-01-04 11:26:06
|
FLAC uses linear prediction + Golomb-coded residual, it's an algorithm optimized for audio data yet simple enough that it decodes very fast. I wonder how Zlib or ZStandard would perform as a coder for the residual bitstream of FLAC since it can be quite stochastic due to the nature of linear prediction.
|
|
|
190n
|
2022-01-04 11:45:45
|
yeah sure i know what all that means <:kekw:808717074305122316>
|
|
|
improver
|
2022-01-04 11:54:27
|
basically flac for audio is kinda like png for images (does predictor magix, is combined with another compression thing ontop of that), and it'd be interesting to see how deflate or zstd would work to replace golomb thingy (kind of experiment like replacing deflate with zstd in png)
|
|
|
190n
|
2022-01-04 11:54:57
|
very interesting
|
|
|
_wb_
|
2022-01-05 12:34:13
|
I wonder what could be done with jxl-style MA context trees + hybriduint ANS
|
|
|
Kagamiin~ Saphri
|
|
improver
basically flac for audio is kinda like png for images (does predictor magix, is combined with another compression thing ontop of that), and it'd be interesting to see how deflate or zstd would work to replace golomb thingy (kind of experiment like replacing deflate with zstd in png)
|
|
2022-01-05 12:46:38
|
To put it simply, LZ-like coders probably won't bring much improvement for FLAC over Golomb coding because it's very hard to isolate repetition in audio. You'd essentially just be using the entropy coding part of Zlib or ZStandard.
|
|
|
_wb_
|
2022-01-05 12:55:58
|
Something like the weighted predictor of jxl might work well in audio too then...
|
|
|
|
veluca
|
2022-01-05 09:13:50
|
sooooo... I wrote a fast PNG encoder on the same lines as fjxl
|
|
2022-01-05 09:14:14
|
depending on image size and a few other things, 180~800MP/s on a single core
|
|
|
Scope
|
2022-01-05 09:21:42
|
Faster than FPNG? And what about compression?
|
|
|
|
veluca
|
2022-01-05 09:22:34
|
faster than FPNG: yes, if you disable adaptive predictor choices, otherwise who knows
|
|
2022-01-05 09:23:43
|
it seems to be, but not by a lot
|
|
2022-01-05 09:23:53
|
compression: ehhhh... I didn't test it too much
|
|
2022-01-05 09:24:50
|
atm it only supports RGB8, but extending it to (G|GA|RGB|RGBA)(8|16) should be reasonably trivial
|
|
|
Scope
|
2022-01-05 09:27:09
|
Considering that FPNG used a lot of different tricks and fast Adler/CRC32 implementations and other things, as far as I understand
|
|
|
|
veluca
|
2022-01-05 09:29:27
|
so with fixed predictor I'd say it's anywhere from 2x to 4x faster than fpng, but compression suffers a bit
|
|
2022-01-05 09:29:47
|
like, 10% or so
|
|
2022-01-05 09:29:56
|
(depending on the predictor)
|
|
2022-01-05 09:32:07
|
even ~1% if you use paeth
|
|
|
Scope
|
2022-01-05 09:32:35
|
Is it fully PNG compatible or in FPNG implementation just haven't really tried other predictors?
|
|
|
|
veluca
|
2022-01-05 09:33:50
|
it produces PNGs
|
|
2022-01-05 09:34:12
|
as in, Chrome/eog/imagemagick open them
|
|
2022-01-05 09:34:22
|
FPNG doesn't try other predictors AFAIU
|
|
|
Scope
|
2022-01-05 09:35:26
|
http://www.libpng.org/pub/png/apps/pngcheck.html
|
|
|
|
veluca
|
2022-01-05 09:36:55
|
it's 100% PNG spec compliant
|
|
2022-01-05 09:37:06
|
i.e. I wrote it while reading the PNG spec π
|
|
2022-01-05 09:37:15
|
(and the deflate/zlib specs...)
|
|
|
Scope
|
2022-01-05 09:38:39
|
Very cool then, except that having a fast decoder for such PNGs would also be nice (because for those who need such speeds, decoders are probably even more important)
|
|
|
_wb_
|
2022-01-05 09:39:19
|
It's interesting that really fast decode is harder than really fast encode
|
|
2022-01-05 09:39:55
|
Considering that codecs tend to be designed asymmetrically for fast decode and not necessarily as fast encode
|
|
|
|
veluca
|
2022-01-05 09:40:13
|
well, really fast huffman π
|
|
|
_wb_
|
2022-01-05 09:40:46
|
ANS/AC are more symmetrical in that part, I suppose
|
|
|
|
veluca
|
2022-01-05 09:40:58
|
in the end, fpnge and fjxl get a 3x speedup from basically the same trick - a very fancy Huffman encoder for some special cases π
|
|
|
_wb_
|
2022-01-05 09:41:55
|
But still, predict is easier to simd than unpredict, and thread parallelism is easier to do in encode than in decode
|
|
|
|
veluca
|
2022-01-05 09:42:17
|
not so sure about parallelism
|
|
2022-01-05 09:42:23
|
but otherwise yes
|
|
|
_wb_
|
2022-01-05 09:42:48
|
At least in png you need custom chunks to get parallel decode, while parallel encode can be done within regular png
|
|
|
|
veluca
|
2022-01-05 09:42:53
|
I mean, modulo having offsets
|
|
|
Scope
|
2022-01-05 09:43:50
|
Is parallelism in PNG encoding not quite standard, or just not all decoders can accept it?
|
|
|
|
veluca
|
2022-01-05 09:44:38
|
you can produce 100% spec-compliant PNGs with parallel encoding
|
|
|
_wb_
|
2022-01-05 09:44:50
|
Encode side you can do things in parallel and concatenate bitstreams
|
|
2022-01-05 09:45:53
|
Decode side you don't know without decoding (or having offsets stored somewhere) where rows start
|
|
2022-01-05 09:47:44
|
Fundamentally, an encoder has all information from the beginning, while a decoder has to reconstruct everything as it is decoding
|
|
|
Scope
|
2022-01-05 09:49:19
|
So, for own encoded PNGs it's not a problem π€ (if the decoder knows how they were encoded)
|
|
|
|
veluca
|
2022-01-05 09:52:07
|
or if you write a specific chunk to give you that extra info
|
|
|
Scope
|
2022-01-05 09:55:00
|
Also <https://github.com/w3c/PNG-spec/issues/54>
<https://github.com/w3c/PNG-spec/issues/60>
|
|
2022-01-05 10:21:55
|
https://twitter.com/richgel999/status/1478846234446209029
|
|
2022-01-06 12:04:12
|
However, if the decoder is fast enough, a specialized one will not be as necessary
https://twitter.com/richgel999/status/1478878192307945473
|
|
|
|
veluca
|
|
Scope
https://twitter.com/richgel999/status/1478846234446209029
|
|
2022-01-06 08:14:17
|
rust's decoder is pretty fast too
|
|
|
Scope
|
2022-01-06 12:25:47
|
It's weird, but rust-png is not enabled for me or I'm doing something wrong and also maybe this benchmark does not work properly in Windows, very often spng and wuffs have the same timings
<https://github.com/veluca93/spngt>
```libpng: 31243 usec
spng: 15623 usec
stb_image: 31248 usec
lodepng: 46871 usec
wuffs: 15622 usec```
|
|
2022-01-06 12:29:39
|
Also maybe it would be easier to optimize some things in lodepng (CRC-32, Huffman, etc) or is it quite time consuming and dangerous to add new bugs/vulnerabilities?
And in the future add fast modes
|
|
|
|
veluca
|
2022-01-06 04:06:56
|
for rust: it only works with external modules, you need to pass an env variable and compile the crate separately
|
|
2022-01-06 04:07:33
|
no idea about windows, dlopen exists I think so it should work but IDK how
|
|
2022-01-06 04:09:23
|
` const char *external_lib = getenv("EXTERNAL_LIB");`
|
|
2022-01-06 04:09:39
|
so you should do something like EXTERNAL_LIB=path_to_stuff.dll
|
|
|
Scope
|
2022-01-07 04:01:43
|
https://www.contentful.com/blog/2021/11/29/load-avif-webp-using-html/
|
|
2022-01-07 04:05:03
|
But, also a strange comparison only by the resulting size after encoding
|
|
|
_wb_
|
2022-01-07 08:33:24
|
https://www.reddit.com/r/AV1/comments/ry8qvg/intel_completely_disables_avx512_on_alder_lake/
|
|
2022-01-07 08:46:06
|
So they'll have dedicated hardware av1 decode on those chips and also general-purpose 512-bit simd (which can e.g. be used to make software jxl encode/decode faster), but the 512-bit simd will be disabled (even though it takes up quite some of the die area), presumably to have a better sales pitch for their more expensive products...
|
|
2022-01-07 08:50:19
|
Basically we're evolving from "general purpose computers" to more and more of a bifurcation between ubiquitous "consumer grade computers" (which is good to decode/consume; extreme examples being the iOS devices with their walled gardens of Apple-approved consumables) and more expensive "professional/enterprise computers" that have actual general-purpose computing power.
|
|
2022-01-07 08:55:17
|
Maybe I'm a bit too pessimistic but that's what I feel we're moving to. At some point the consumer grade computers will stop being general-purpose computers but they'll only run "approved" software and have DRM built in β and it will be in the name of security (malware prevention etc) and stopping child porn, of course.
|
|
|
improver
|
2022-01-07 09:10:28
|
AVX-512 on consumer devices was never really viable because it reduced clock speeds to account for thermals & power draw, and made everything else slower
|
|
2022-01-07 09:11:00
|
also iirc ander lake stuff involves mixing with power-efficient cores which can't do AVX-512 at all
|
|
2022-01-07 09:12:54
|
it's also kinda annoyingly fragmented (with some CPUs supporting some of subset of AVX-512, and others supporting other), and was never supported by AMD cpus, so I'd say good ridance
|
|
|
Scope
|
2022-01-07 09:16:48
|
I think Intel just rushed to push out the new generation CPUs to get their market share back, but they didn't have time to do something to make it convenient to use AVX-512 only on part of the cores, since they are not available on E-cores
However, for ML, compression and many codecs the AVX-512 would be very useful
https://twitter.com/richgel999/status/1479281825084284932
|
|
|
_wb_
|
2022-01-08 07:16:17
|
Well don't use it for memcpy if that's all you're going to use it for
|
|
2022-01-08 07:21:02
|
So Linus prefers to get more cores instead of wider simd. I think both are useful.
|
|
2022-01-08 07:24:32
|
I think he has a point that for typical applications, more/faster cores is more useful than wider simd.
|
|
2022-01-08 07:25:15
|
But as libjxl shows, simd is really not just for HPC and AI applications
|
|
2022-01-08 07:26:31
|
Anything related to images or video can benefit from simd
|
|
2022-01-08 07:26:58
|
(and from more cores too)
|
|
|
The_Decryptor
|
2022-01-08 07:27:35
|
I like the direction ARM/RISC-V are going with variable length vectors, instead of having a separate SIMD instructions for each length, though I will admit to having no idea how effective that'd be in practise for most apps
|
|
|
_wb_
|
2022-01-08 08:06:34
|
> Supported targets: scalar, S-SSE3, SSE4, AVX2, AVX-512, AVX3_DL (~Icelake, requires opt-in by defining HWY_WANT_AVX3_DL), NEON (ARMv7 and v8), SVE, WASM SIMD.
|
|
2022-01-08 08:07:39
|
No idea why AVX1 is not a target
|
|
|
|
veluca
|
2022-01-08 08:15:17
|
because there's a lot of stuff that is useful but doesn't exist for AVX1 - in particular many many integer and shuffle ops
|
|
|
_wb_
|
2022-01-08 08:22:35
|
For jxl we don't really need the integer and shuffle ops, or do we?
|
|
2022-01-08 08:23:19
|
The FMA introduced in AVX2 is very useful though, I imagine
|
|
|
Scope
|
2022-01-11 04:47:22
|
<https://github.com/libjxl/libjxl/commit/f413c429cfd5f2e1f65b5ebfc0c969b67ddbcbaf>
Also, what about adding fpnge to the fast_lossless utility, like if the output image is png, then it's a job for the fpng encoder or is it not ready yet?
|
|
|
|
veluca
|
2022-01-11 07:12:43
|
fast_lossless doesn't decompress π
|
|
|
Scope
|
2022-01-11 07:21:27
|
Yes, I mean this (same as fjxl, but for png and in the same fast_lossless_main.cc tool)
https://discord.com/channels/794206087879852103/805176455658733570/928395898638172251
|
|
|
|
veluca
|
2022-01-11 07:40:02
|
ah, I'm making a separate repository
|
|
2022-01-11 07:40:11
|
saying it's in the scope of libjxl is a bit of a stretch
|
|
|
Scope
|
2022-01-11 07:51:58
|
I think it's convenient to have a universal tool for fast encoding, but with different files/headers for png/jxl, something like:
```
fjxl.cc
fjxl.h
fpng.cc
fpng.h
fast_lossless_main.cc```
And encode depending on the output image extension, like:
`fast_lossless image.png image.jxl` (fjxl)
`fast_lossless image.png image.png` (fpng)
|
|
|
|
veluca
|
2022-01-11 08:10:25
|
ah, mayyybe
|
|
2022-01-11 08:31:00
|
<@!111445179587624960> want to try it out? https://github.com/veluca93/fpnge
|
|
|
Scope
|
|
|
veluca
|
2022-01-11 08:36:35
|
I don't expect it will do super well
|
|
2022-01-11 08:37:22
|
you can `-DFPNGE_FIXED_PREDICTOR=[1 to 4]` to get the Very Fast versions
|
|
|
Scope
|
2022-01-11 09:01:29
|
Anyway I can not compare the speed accurately, I think this needs a comparison from Jon, because I'm not a coder and can not create a tool and the same conditions for different codecs to avoid the impact of writing/reading from disk, PNG decoding time, etc.
|
|
2022-01-11 09:05:53
|
<@!179701849576833024> π€
`Assertion failed: length >= 8, file ../fpnge.cc, line 369`
|
|
2022-01-11 09:07:23
|
This happens quite often
|
|
|
|
veluca
|
2022-01-11 09:08:55
|
uh
|
|
2022-01-11 09:09:01
|
could be, didn't test it super well
|
|
2022-01-11 09:09:36
|
can you link me a failing image?
|
|
|
Scope
|
|
|
veluca
|
2022-01-11 09:16:17
|
should be fixed π
|
|
|
Scope
|
2022-01-11 11:17:35
|
https://twitter.com/richgel999/status/1481039953828233225
|
|
2022-01-11 11:17:36
|
https://twitter.com/richgel999/status/1481040512383660039
|
|
|
|
veluca
|
2022-01-11 11:31:21
|
added support for RGBA input too, while I was at it
|
|
2022-01-11 11:31:27
|
not exactly well tested
|
|
|
Scope
|
2022-01-11 11:33:04
|
> Never try predictor 0
It is also useless for this fast implementation?
|
|
|
|
veluca
|
2022-01-11 11:34:03
|
yeah it's very bad always
|
|
2022-01-11 11:34:35
|
TOP is nearly as fast to compute and compresses loads better
|
|
2022-01-11 11:35:05
|
I mean, if you do proper lz77 then 0 *can* be useful
|
|
|
Scope
|
2022-01-11 11:38:39
|
Looks like the fpnge has been spotted <:PeepoDiamondSword:805394101340078092>
https://twitter.com/richgel999/status/1481043266728603651
|
|
|
|
veluca
|
2022-01-11 11:40:20
|
Hmm...this page doesnβt exist. Try searching for something else.
|
|
|
Scope
|
2022-01-11 11:42:27
|
It seems that the post was deleted, it was about encoders that will soon compete with FPNG, so some comparisons have not yet been shown
|
|
|
|
veluca
|
2022-01-11 11:43:46
|
hehe
|
|
2022-01-11 11:44:01
|
I've been chatting with Richard anyway, I think he'll write something soonish
|
|
|
Scope
|
2022-01-11 11:48:09
|
Also, are the predictors selected by brute force, or is there some sort of heuristic?
Because it's likely that some will be efficient very often and others very rarely
|
|
|
|
veluca
|
2022-01-11 11:49:01
|
in fpnge?
|
|
2022-01-11 11:49:12
|
by default, try all 4 and pick the best
|
|
2022-01-11 11:49:30
|
if you compile with FIXED_PREDICTOR, well
|
|
|
Scope
|
2022-01-11 11:51:34
|
Yes, for example, if I am not mistaken, fpng always uses filter/predictor 2 (most likely because it proved to be more efficient for the tested images)
|
|
|
|
veluca
|
2022-01-11 11:54:31
|
yup
|
|
2022-01-11 11:55:06
|
also because 4 may be better but it's more annoying to compute
|
|
|
Scope
|
2022-01-12 12:08:53
|
Maybe also add Wuffs decoder support to FJXL/FPNGE, because decoding can take significant time and probably even more than encoding?
However, as far as I remember, it needs some gigantic C source file, and I don't know how much that would affect the overall encoder binary size
https://twitter.com/richgel999/status/1481027198530248714
|
|
|
|
veluca
|
2022-01-12 12:10:03
|
meh
|
|
2022-01-12 12:10:14
|
proof of concept, not serious production thingy
|
|
2022-01-12 12:10:53
|
also, if you use it by reading from a PNG, you're using it wrong xD
|
|
|
Scope
|
2022-01-12 12:14:52
|
I see, also as far as I understand, the main speedup of fast decoders comes from better optimized CRC-32/Adler-32 and Deflate implementations
|
|
|
|
veluca
|
2022-01-12 12:15:29
|
also filters/predictors and pixel format conversions
|
|
2022-01-12 12:15:53
|
but I'd bet most of it is deflate
|
|
|
Kagamiin~ Saphri
|
2022-01-12 12:45:42
|
does <#805176455658733570> cover audio codecs too?
|
|
|
w
|
2022-01-12 12:47:08
|
why not
|
|
|
Kagamiin~ Saphri
|
2022-01-12 12:47:25
|
anybody heard of "SSDPCM"?
|
|
2022-01-12 12:47:33
|
someone from the demoscene came up with this codec
|
|
2022-01-12 12:48:15
|
turns out it basically boils down to DPCM split into short blocks, where deltas are vector-quantized and each block has a tiny codebook
|
|
2022-01-12 12:48:27
|
it's meant for like 1bpp and 2bpp so I really mean tiny
|
|
2022-01-12 12:48:43
|
and it turns out it's a surprisingly good codec
|
|
2022-01-12 12:49:19
|
i made up an implementation of it in C, I'm kinda ashamed to share it but I did encode some stuff in it some time ago...
|
|
2022-01-12 12:49:36
|
uhhh the author never released a toolkit for it
|
|
2022-01-12 12:49:50
|
i did figure it out on my own and made the most basic brute-force encoder
|
|
2022-01-12 12:50:01
|
know what, guess I'll just throw the code out here
|
|
2022-01-12 12:50:52
|
I can't remember if I ever made a makefile for this but you just gotta `gcc -O2 ssdpcm.c` and it should work
|
|
2022-01-12 12:51:10
|
don't judge, it's old hackish code, it just works
|
|
2022-01-12 12:51:47
|
sample output at 1 bit per sample:
|
|
2022-01-12 12:51:53
|
|
|
2022-01-12 12:52:22
|
(this is from an older version where I did assymetric deltas, I gave up on it and it encodes an order of magnitude faster because it's basically a brute force search over the codebook lol)
|
|
2022-01-12 12:55:57
|
1 bit vs. 2 bit comparison at the same sample rate
|
|
2022-01-12 12:56:11
|
here are the raw bitstreams for those two in case someone wants to mess with those... it's a completely custom format I made for testing the codec
|
|
2022-01-12 12:57:43
|
I don't, I deleted my GitHub account quite some time ago
|
|
2022-01-12 12:58:19
|
nope nope
|
|
2022-01-12 12:58:22
|
none of those
|
|
2022-01-12 12:59:14
|
I do have a local git repo for it but that's it lol
|
|
2022-01-12 12:59:54
|
I really couldn't care less, this was a prototype I did once, messed with it a while later and never touched it later
|
|
2022-01-12 01:00:07
|
I thereby release it as public domain, really
|
|
2022-01-12 01:00:25
|
it was not even my invention, just my implementation of it based on what I could guess it was
|
|
2022-01-12 01:00:56
|
the original author actually did some more advanced stuff on the encoding side with shotgun hill-climbing instead of the greedy DPCM encoding + brute force codebooks I did
|
|
2022-01-12 01:01:29
|
the original author is known as "Algorithm" in the scene
|
|
2022-01-12 01:02:42
|
not at all
|
|
2022-01-12 01:03:03
|
it's closer to SNES audio than it is to that
|
|
2022-01-12 01:03:24
|
you know DPCM? where you encode each sample as the difference between the last sample and the current one?
|
|
|
Scope
|
2022-01-12 01:04:35
|
π€
https://youtu.be/OPBoAQMDoVs?t=11
|
|
|
Kagamiin~ Saphri
|
|
Scope
π€
https://youtu.be/OPBoAQMDoVs?t=11
|
|
2022-01-12 01:04:50
|
yuuup that's it, I couldn't find the channel for some reason
|
|
2022-01-12 01:05:22
|
I basically reverse-engineered their codec based on vague descriptions collected from pouet.net and youtube comments
|
|
2022-01-12 01:05:33
|
it was originally meant to run on low-end hardware
|
|
2022-01-12 01:06:10
|
there's a very impressive demo of it running on an Amiga, it's basically a 800K floppy demo that plays a full rendition of some theme song from Rocky and it sounds amazing for what it is
|
|
|
Kagamiin~ Saphri
you know DPCM? where you encode each sample as the difference between the last sample and the current one?
|
|
2022-01-12 01:07:12
|
basically, it was a common thing to encode lossy DPCM using a fixed table of possible difference values between the last sample and the current one...
|
|
2022-01-12 01:07:57
|
but Algorithm thought "what if instead of fixed values we divided our stream into blocks and we had a list of values for every block of samples"
|
|
2022-01-12 01:08:47
|
and somehow this sounds better than ADPCM which is technically more advanced than this and is still used today in modern audio codecs such as apt-X, with the tradeoff that it takes way more computational power to encode than it does to decode because it relies on vector quantization after all
|
|
2022-01-12 01:10:02
|
(btw, if you didn't know about it, apt-X, the fancy proprietary codec used in Bluetooth audio, is basically just ADPCM applied over 4 sub-bands with fixed bit allocation...)
|
|
2022-01-12 01:12:50
|
I can easily ELI5 PCM vs. DPCM, but DSM takes a whole another level of understanding
|
|
2022-01-12 01:13:42
|
Anyway: in PCM, you have a set of numbers representing different amplitude levels of a sound pressure wave... I'd guess you do understand that, right <@456226577798135808>
|
|
2022-01-12 01:14:56
|
So, in DPCM, instead of storing the instant amplitude level of the waveform, you store the difference between the waveform itself and its last value, it's as simple as that, but the real deal comes when you wanna do lossy DPCM compression
|
|
2022-01-12 01:15:24
|
You see, in lossy PCM, you basically have to take bits away so you have less possible positions for the sound pressure wave, and you get quantization noise
|
|
2022-01-12 01:15:56
|
But in lossy DPCM, depending on your content, the difference between the current value and the next one might usually be a small number
|
|
2022-01-12 01:17:01
|
So, if you allocate more possible values to small differences between the next sample and the previous one, and reserve the last few values for those moments where the difference between the next sample and the previous one is large, you'll tend to get better quality audio on average
|
|
2022-01-12 01:17:09
|
This is the basic principle behind DPCM
|
|
2022-01-12 01:19:05
|
But in reality it doesn't work that well, you either get lots of noise during loud high-frequency-laden passages, or you get an effect known as slope overload where the DPCM-encoded waveform can't keep up with the original waveform, it's a tradeoff as to how you allocate the possible difference values between the last sample and the previous one
|
|
2022-01-12 01:21:40
|
ADPCM improves on DPCM by automagically predicting when "slope overload" occurs and scaling the difference values around to compensate for it... the same prediction algorithm is applied both during encoding and decoding to generate a reproducible audio stream... and most ADPCM codecs are able to get away with linear quantization (meaning each difference value is linearly spaced and scaled by the predicted scale value)... it automatically minimizes noise in both quiet and loud passages
|
|
2022-01-12 01:22:28
|
uwu I'm happy I was able to teach you this
|
|
2022-01-12 01:39:26
|
>python script by Kagamiin~
Waaaait, I did do that too lol, I published it on pret (a discord server related to researching stuff about PokΓ©mon games)
|
|
2022-01-12 01:40:20
|
it's a script to encode 1-bit PCM audio for using in PokΓ©mon Yellow to replace Pikachu's voice samples
|
|
2022-01-12 01:41:12
|
Really simple stuff, it just converts 8-bit PCM to 1-bit MSB-first PCM
|
|
2022-01-12 01:41:32
|
(and it skips over WAV headers for convenience lol)
|
|
2022-01-12 01:56:47
|
wtf there's like a serious bug in this, how does this even work?
|
|
2022-01-12 01:58:10
|
wait does this work because of Python's bigint support?
|
|
2022-01-12 01:59:04
|
wtf I'm friggin' flattered over my terrible code from the past... I swear it has to be older than a year... really?
|
|
2022-01-12 01:59:23
|
so, I did `while all(x for x in samples):`, fair enough
|
|
2022-01-12 02:00:08
|
then I did `for s in samples:` wtf this is going to consume every single byte of the file, it doesn't make sense
|
|
2022-01-12 02:00:20
|
I should stop looking at my old code
|
|
2022-01-12 02:00:33
|
and maybe make a brand new GitHub account?
|
|
2022-01-12 02:01:58
|
I do have reasons to not own a Github account... but I do also have reasons to make a new one, oh well idk I'll just make one when I feel like it so I can make a pull request for this broken code of mine, I swear I tested this at the time and it did work?
|
|
2022-01-12 02:02:05
|
I'd better go sleep... damn
|
|
2022-01-12 02:06:46
|
oh wait nevermind, I reread it and this code actually should work lol
|
|
2022-01-12 02:07:03
|
I was just misunderstanding Python's iterators and forgetting to read the line of code just above it
|
|
2022-01-12 02:07:09
|
I'm really rusty in Python
|
|
2022-01-12 02:09:36
|
I had to delete my old GitHub account because of reasons I shall not disclose, and I've been waiting for my trace to die off so that I don't get caught up on GitHub's TOS which prohibits one from owning more than one account
|
|
2022-01-12 02:11:00
|
Mostly C, some Python, and quite a bit of shell scripting, I can also do Java but I'm not great at it, done some Groovy too... NASM is an assembler and not a language, you gotta learn to program for whatever processor you want to start with and whatever hardware or system features it offers
|
|
2022-01-12 02:16:16
|
gtg
|
|
|
w
|
2022-01-12 02:16:22
|
when i tried to make a second account it was instantly blocked
|
|
|
Kagamiin~ Saphri
|
|
w
|
2022-01-12 02:17:53
|
probably ip
|
|
|
Kagamiin~ Saphri
|
2022-01-12 02:17:58
|
Fingerprinting
|
|
2022-01-12 02:18:02
|
Basically multiple sources
|
|
2022-01-12 02:18:31
|
Including IP address, cookies, third-party cookies shared around, also device ID (which usually makes up most of one's fingerprint)
|
|
2022-01-12 02:19:21
|
lol GitHub mostly works without Javascript at least for browsing code
|
|
2022-01-12 02:19:30
|
meanwhile Gitlab doesn't even render without Javascript
|
|
2022-01-12 02:19:45
|
it sounds ironic but it's a fact
|
|
2022-01-12 02:20:42
|
true
|
|
2022-01-12 02:20:52
|
anyway I should sleep so bye
|
|
|
Scope
|
2022-01-12 10:39:41
|
https://twitter.com/richgel999/status/1481376407968272385
|
|
2022-01-12 10:40:11
|
There's also Mango, with sources, but which are kind of "hidden", so it's probably considered proprietary, but I still don't understand how it can have this speed or is it cheating with a single-threaded vs multithreaded comparison?
|
|
|
|
veluca
|
2022-01-12 10:46:30
|
is that a PNG en/decoder?
|
|
|
Scope
|
2022-01-12 10:49:54
|
Yep, PNG/JPEG/...
|
|
|
|
veluca
|
2022-01-12 10:50:29
|
I have strong doubts you can get that speed with a single thread
|
|
|
Scope
|
2022-01-12 10:50:34
|
https://www.liimatta.org/mango/
|
|
|
|
veluca
|
2022-01-12 10:53:56
|
it does support multithreading
|
|
2022-01-12 10:54:05
|
so I assume it uses it to get those numbers π
|
|
2022-01-12 10:55:17
|
if you craft your PNG properly (or if you're encoding), I'm sure with MT en/decoding you'll be only limited by memory bandwidth
|
|
|
_wb_
|
2022-01-13 06:31:47
|
Yes, iirc it does require the png to have a certain structure to get those decode numbers, if this is the png implementation I think it is
|
|
|
Scope
|
2022-01-17 07:03:46
|
<:Thonk:805904896879493180>
https://twitter.com/richgel999/status/1483127512259608576
|
|
|
_wb_
|
2022-01-17 07:20:02
|
PNG and JPEG were both quite nice for their time, imo.
|
|
2022-01-17 07:26:02
|
PNG did some things very right imo:
- chunks that can be parsed (skipped) and partially understood even if you don't understand them
- the 'modular' approach of separating the filters (predictors) from the compression (deflate), so in principle new filters and new compression methods could have been proposed to improve things
- supporting high bit depth (16-bit!) at a time when most people still couldn't even see more than rgb565 on their screen
|
|
2022-01-17 07:28:19
|
unfortunately the thing with 'extensible' chunks and compression methods etc is that it doesn't really work for anything non-optional β once you have a critical mass of decoders deployed, you cannot really produce new kinds of bitstreams that they don't decode
|
|
2022-01-17 07:29:43
|
which is why we don't have JPEG-with-an-alpha-channel (technically perfectly possible, JPEG can encode 4 channels just fine), and no PNG-with-something-better-than-deflate
|
|
|
Scope
|
2022-01-17 08:04:25
|
Some new discussions about JXL, FLIF, WebP2, etc
<https://encode.su/threads/3520-WebP-2-experimental-successor-of-the-WebP-image-format/page3>
|
|
2022-01-17 08:10:03
|
And also about the Jpeg XL speed, my pretty old builds are faster most likely because they did not have compression improvements for the slowest modes, I do not think that any compiler options can have that much effect
|
|
2022-01-17 09:00:56
|
<:Thonk:805904896879493180>
|
|
2022-01-17 09:01:29
|
`FLIF is quite poor, and it seems the lossless mode for JPEG-XL is nothing to write home about either.`
`So hopefully WebP 2 turns out to actually be much better than FLIF and JPEG-XL at lossless compression.`
|
|
|
Cool Doggo
|
2022-01-17 10:03:13
|
are they only comparing encoding speed? i dont get what they are comparing
|
|
|
Scope
|
2022-01-17 10:12:34
|
Encoding speed and compression, but the comparison is still quite strange, because tested only one image and does not take into account that the speed of decoding is also very important, so that symmetric codecs are also not suitable, even if the encoding speed is like 5-10 seconds, waiting for 5-10 seconds to decode the image for viewing is not acceptable, and the faster symmetric encoder/decoder can not get that much better compression
|
|
2022-01-17 10:16:49
|
Multithreading can speed up decoding, but Jpeg XL also uses it and splits the image into tiles/groups
|
|
|
BlueSwordM
|
|
Scope
`FLIF is quite poor, and it seems the lossless mode for JPEG-XL is nothing to write home about either.`
`So hopefully WebP 2 turns out to actually be much better than FLIF and JPEG-XL at lossless compression.`
|
|
2022-01-17 11:28:36
|
I like the fact that he doesn't give us an image for us to test out his fact.
|
|
|
Scope
|
2022-01-17 11:29:17
|
It's one big image in the posts above
|
|
2022-01-17 11:29:51
|
https://cdn.discordapp.com/attachments/399068055092723712/928065449722282005/Doom_Eternal_Screenshot_2021.07.26_-_12.14.04.24.png
|
|
|
_wb_
|
2022-01-18 06:31:25
|
I can make a super fast and super dense codec for that image.
|
|
2022-01-18 06:31:49
|
π
|
|
2022-01-18 07:52:15
|
Making a codec for a single image is trivial :)
|
|
2022-01-18 07:52:57
|
Hardcode the image, define that image to be represented by 1 byte, store any other image uncompressed.
|
|
2022-01-18 07:55:04
|
When evaluating codecs, it's essential to do it on a corpus that is representative of the images you actually want to compress. Testing on a single image is as silly as testing on all possible random images, for which (due to the pigeonhole principle) no codec can do better than uncompressed.
|
|
|
Fox Wizard
|
2022-01-18 08:24:41
|
I like how optimizers made that huge PNG like 30MB smaller XD
|
|
|
|
Deleted User
|
|
_wb_
Hardcode the image, define that image to be represented by 1 byte, store any other image uncompressed.
|
|
2022-01-18 09:31:21
|
Why use that 1 byte? Define the image to be represented by *0 bytes*, store any other image uncompressed! I've just saved you 1 byte π
|
|
2022-01-18 09:33:18
|
Wait, that sounds familiar! I wonder where I've seen it before... <:Thonk:805904896879493180>
||~~LenPEG 2~~||
|
|
|
190n
|
|
Why use that 1 byte? Define the image to be represented by *0 bytes*, store any other image uncompressed! I've just saved you 1 byte π
|
|
2022-01-18 09:34:23
|
infinity% improvement <:Poggers:805392625934663710>
|
|
|
The_Decryptor
|
2022-01-19 01:56:04
|
For web pages, similar to how browsers support Brotli
|
|
|
Scope
|
2022-01-19 03:35:44
|
> **mpais**
> Slicing/tiling in squares is also very much an artefact of thinking of lossless compression from the point of view of the techniques used for lossy compression.
> For lossy compression you'll almost always want to use a transform for dimensionality reduction (DCT, DWT/WPD, SVD, etc), and using it on the whole image at once is usually bad for performance.
> For lossless compression you get really good results from simple LPC (on most natural images), so for better performance you can simply split the image into horizontal slices, each the full width.
> That gives you better cache locality and is usually mostly irrelevant in terms of compression ratio change. You can think of the filtering (not the actual DEFLATE compression stage) in PNG as an extreme case of this, where each line is a slice.
<https://encode.su/threads/3520-WebP-2-experimental-successor-of-the-WebP-image-format?p=72737&viewfull=1#post72737>
I also wonder if square tiles are really something not efficient and bad for lossless compression or do they still have benefits and this solution is pretty practical?
|
|
|
The_Decryptor
|
2022-01-19 05:13:53
|
I think that'd just be on the encoder, if it breaks the image into tiles before encoding it, then each "encoding unit" is just operating on a contiguous memory chunk anyway
|
|
2022-01-19 05:15:24
|
Can also do a multi step process, horizontal slices of the tile height, then break that slice into tile sized units before processing
|
|
|
Nova Aurora
|
|
190n
infinity% improvement <:Poggers:805392625934663710>
|
|
2022-01-19 05:36:00
|
Ah, The metagolfscript gambit
|
|
|
_wb_
|
2022-01-19 06:18:09
|
Slicing is a bit better for speed than tiling, and independent tiles have the compression downside of more poorly-predicted pixels (since it's the top row and left column of every tile that becomes poorly-predicted, not just those of the entire image)
|
|
2022-01-19 06:24:04
|
But tiles have advantages over slices too: better 2D locality for compression (e.g. for local palette or for RLE), you can do ROI decode (only decode part of the image), can deal with very wide images, etc.
|
|
|
VEG
|
2022-01-19 02:31:10
|
https://twitter.com/richgel999/status/1483746169629163524
|
|
2022-01-19 02:31:19
|
Interesting experiments with lossy PNG
|
|
|
Scope
|
2022-01-19 04:52:36
|
But, there are quite visible distortions on the edges
https://twitter.com/richgel999/status/1483818811853783043
|
|
|
VEG
|
2022-01-19 05:24:43
|
Yeah, that's true.
|
|
|
_wb_
|
2022-01-19 05:29:31
|
Any lossless codec can be used in a lossy way like that, I also did that in flif
|
|
|
190n
|
2022-01-19 05:33:26
|
lossy flif was just truncated progressive flif, right?
|
|
|
_wb_
|
2022-01-19 05:37:57
|
Not just that. You could also preprocess it to be more predictable, by chopping off lsbs of the predictor residuals
|
|
2022-01-19 05:38:21
|
(and/or deadzoning low amplitude residuals to zero)
|
|
2022-01-22 08:31:30
|
https://www.reddit.com/r/apple/comments/s5xcgd/why_is_heic_not_popular/
|
|
|
improver
|
2022-01-22 09:05:05
|
tbh it started getting relatively popular with pirating groups lately (anime or not)
|
|
|
w
|
2022-01-22 09:08:43
|
heic?
|
|
2022-01-22 09:08:45
|
nobody uses heic
|
|
|
Nova Aurora
|
|
improver
tbh it started getting relatively popular with pirating groups lately (anime or not)
|
|
2022-01-22 09:11:46
|
HEVC? Yeah, that's fairly common. HEIC? No, only Apple devices put them out by default and nothing actually takes them as input.
|
|
|
_wb_
|
2022-01-22 09:19:32
|
Well I made heic encode/decode work on Cloudinary so maybe "nothing" is a bit exaggerated, but yes, outside the Apple ecosystem support is very poor, for good reasons (it's a patent minefield)
|
|
2022-01-22 09:23:09
|
The way Apple uses heic seems to be using hardware encode in 512x512 tiles that are independently coded (to the point of having independent h265 codestreams in the heif file, which are then assembled using the `grid` construct of heif)
|
|
2022-01-22 09:24:04
|
While file sizes are small, the perceptual quality is also lower than if you configure it to encode to jpeg
|
|
2022-01-22 09:24:32
|
(and or course it gets even worse if you have to transcode the heic to jpeg to be able to share it)
|
|
|
Nova Aurora
|
|
_wb_
Well I made heic encode/decode work on Cloudinary so maybe "nothing" is a bit exaggerated, but yes, outside the Apple ecosystem support is very poor, for good reasons (it's a patent minefield)
|
|
2022-01-22 09:26:00
|
I haven't seen an online service other than cloudinary that actually takes HEIC. All my assignments ban using HEIC by name because the professors can't open them.
|
|
|
_wb_
|
2022-01-22 09:26:32
|
I am assuming they calibrated their encode settings using a bad metric like psnr, which h265 can optimize very well for but that doesn't mean it actually looks good
|
|
2022-01-22 09:29:11
|
Anyway, besides the technical issues, the main thing is that it's very much patent-encumbered, which is a clear no-go if you want an interoperable image format
|
|
|
Nova Aurora
|
|
Nova Aurora
I haven't seen an online service other than cloudinary that actually takes HEIC. All my assignments ban using HEIC by name because the professors can't open them.
|
|
2022-01-22 09:29:43
|
I'm fairly certain if I started using exotic formats I could expand the list given the amount of tech savvy some professors have
|
|
|
monad
|
2022-01-22 09:31:18
|
Use JXL <:YEP:808828808127971399>
|
|
|
_wb_
|
2022-01-22 09:31:23
|
I don't know what Apple was thinking when they started using heic - maybe they were hoping for (even) more vendor lock-in by using a format they know other platforms are not going to support
|
|
|
Nova Aurora
|
|
_wb_
I don't know what Apple was thinking when they started using heic - maybe they were hoping for (even) more vendor lock-in by using a format they know other platforms are not going to support
|
|
2022-01-22 09:33:56
|
Another thing is probably to get the most mileage out of their custom silicon with a HEVC encoder
|
|
|
_wb_
|
2022-01-22 09:35:43
|
With WDP (JPEG XR), I think Microsoft genuinely just wanted to have an HDR-ready image format, and they standardized it and made sure it was not patent encumbered. But that codec has the problem of being very "meh" in compression (it positions itself as somewhere between JPEG and JPEG 2000 in terms of complexity and compression results), and the only public implementation is abandonware...
|
|
|
Nova Aurora
Another thing is probably to get the most mileage out of their custom silicon with a HEVC encoder
|
|
2022-01-22 09:37:11
|
Yes, that could also be the case. "We have this shiny hardware, now let's use it!". I think a similar reasoning is pushing avif btw.
|
|
|
Nova Aurora
|
|
monad
Use JXL <:YEP:808828808127971399>
|
|
2022-01-22 09:40:36
|
>student has turned in, let's see it
>File won't open in photo viewer
>Don't give student grades
(In the future)
I WILL NOT accept. .jxl .avif .flif .HEIC .DDS .qoi .bpg or .JP2000 files.
|
|
|
The_Decryptor
|
2022-01-22 09:57:19
|
Nvidia switched from EXR to JPEG XR for HDR screenshots on Windows within the last year or so
|
|
2022-01-22 09:58:53
|
The default picture viewer in Windows doesn't do any form of tone mapping, but I suppose that's better than something most people couldn't open
|
|
|
_wb_
|
2022-01-22 10:04:01
|
I need to make a 10-bit version of fjxl, it would be great for making HDR screenshots...
|
|
|
The_Decryptor
|
2022-01-22 11:37:12
|
I've got a process for converting Xbox HDR screenshots to JXL, lots of sharp edges that make it annoying
|
|
2022-01-22 11:38:09
|
The MS JXR decoder has limited output support, it says it can emit the 10 bit values to TIF/BMP, but then nothing I've found supports reading those files properly
|
|
2022-01-22 11:39:58
|
iirc GIMP encounters an error trying to load the TIF files, while ImageMagick sees it as pure noise, while they treat the BMP file as 8bit
|
|
|
_wb_
|
2022-01-22 11:46:04
|
The JXR tooling is a mess, finding input/output formats that actually work is very tricky with it
|
|
2022-01-22 11:49:10
|
I dunno if 10-bit TIF is well-standardized, I suppose the main question is if you pad with msb zeroes or with bogus lsb (either zeroes or convert more properly to 16-bit but ignore the 6 lsb). If applications do not agree on that, the result will be garbage
|
|
|
The_Decryptor
|
2022-01-22 11:55:17
|
I assumed if neither ImageMagick or GIMP couldn't open it (Or any other app I tried) that it had to be a custom format that wasn't worth them supporting
|
|
2022-01-22 11:56:15
|
The issues with BMP surprised me a bit, since it does actually have a properly specified way to pack >8bit values into a 32bit image
|
|
2022-01-22 11:57:08
|
ImageMagick reports the image as being "8/16bit", but shows the individual channels as 8bit only
|
|
|
improver
|
|
Nova Aurora
HEVC? Yeah, that's fairly common. HEIC? No, only Apple devices put them out by default and nothing actually takes them as input.
|
|
2022-01-22 12:51:52
|
oh. yeah ive misread that
|
|
|
Scope
|
2022-01-22 01:22:09
|
<:Poggers:805392625934663710>
<https://github.com/veluca93/fpnge/commit/a1e8f5b7029716f6c73cc9846e9126da822d05b7>
|
|
|
|
Deleted User
|
2022-01-22 01:23:18
|
So no forced RGB to RGBA conversion?
|
|
|
|
veluca
|
2022-01-22 01:24:35
|
I feel a bit stalkered :P
|
|
2022-01-22 01:24:50
|
Yup
|
|
2022-01-22 01:25:09
|
Now I want to see if I can improve compression
|
|
2022-01-22 01:25:19
|
I think I can at minimal speed impact
|
|
|
_wb_
|
2022-01-22 01:39:38
|
https://youtu.be/hmmIl9aMz4I
|
|
|
Fox Wizard
|
2022-01-22 01:40:25
|
Good old memories
|
|
|
_wb_
|
2022-01-22 01:42:43
|
libpng-turbo :)
|
|
|
BlueSwordM
|
|
_wb_
I don't know what Apple was thinking when they started using heic - maybe they were hoping for (even) more vendor lock-in by using a format they know other platforms are not going to support
|
|
2022-01-22 03:58:44
|
I can think of one reason.
|
|
2022-01-22 03:58:51
|
It's money and being able to use HW encoders.
|
|
|
Nova Aurora
|
2022-01-22 06:40:28
|
I guess it depends on whether their computer has the webp image extensions
|
|
|
Scope
|
|
veluca
Now I want to see if I can improve compression
|
|
2022-01-22 07:04:34
|
Also, what makes FPNG have better compression even with one filter?
And since there are some changes, maybe copy build.sh from FJXL as well (although I don't know if it's really necessary, but it's a kind of unification)
|
|
|
|
veluca
|
2022-01-22 07:05:42
|
well, one reason is that their huffman tables are not completely horrible xD (fixing that...)
|
|
|
_wb_
|
2022-01-22 07:58:01
|
How does fpng do lz77? Probably in a less constrained way than fpnge, I'd guess?
|
|
|
|
veluca
|
2022-01-22 08:01:11
|
likely
|
|
|
190n
|
|
w
nobody uses heic
|
|
2022-01-22 08:54:33
|
pretty sure iphones use it by default
|
|
|
_wb_
|
2022-01-22 09:27:01
|
Until you share it anywhere, then it gets transcoded to jpeg
|
|
2022-01-22 09:27:18
|
Anywhere outside the apple ecosystem at least
|
|
|
|
veluca
|
2022-01-22 10:39:48
|
<@!111445179587624960> I submitted the code to build better Huffman tables, if you want to give it a try... on my set of benchmark images it's a 5-20% improvement (depending on the image), with an average somewhere around 10%
|
|
|
Scope
|
2022-01-22 10:41:51
|
Yep, it will be interesting to compare, 5-20% is pretty noticeable
|
|
|
|
veluca
|
2022-01-22 10:43:47
|
probably on different corpora it's not as good
|
|
2022-01-22 10:44:08
|
but still, should have some decent improvements
|
|
|
Scope
|
2022-01-22 10:48:39
|
Also, how hard is it to make encoding multi-threaded?
Then FPNGE would beat the other encoders with a big advantage (at least until something similar is added to FPNG)
|
|
|
|
veluca
|
2022-01-22 10:50:18
|
I mean... with the speedup from today, I think I already am leaving fpng in the dust π
|
|
|
Scope
|
2022-01-22 10:56:41
|
Yes, but it can still be many times faster if there are no needs to encode multiple files in parallel
Btw, FPNG still has higher compression on some sets (of those that have already been encoded)
|
|
|
|
veluca
|
2022-01-22 10:59:32
|
yeah I'm not surprised
|
|
2022-01-22 11:00:49
|
the way I'm doing RLE is pretty barbaric (it heavily trades off density in favour of simplicity & speed) so fpng doing better in some cases is not a huge surprise
|
|
2022-01-22 11:01:33
|
about multithreading: I'm not actually sure how much faster this could get even with MT, it's getting uncomfortably close to memory bandwidth xD
|
|
|
Scope
|
2022-01-22 11:02:52
|
Even without fixed predictors?
|
|
|
|
veluca
|
2022-01-22 11:08:04
|
no, that's a bit slower
|
|
2022-01-22 11:08:23
|
but tbh the density gain from choosing predictors is very much in the "dimnishing returns" area
|
|
2022-01-22 11:08:44
|
I rarely see more than 1-2%, and it doesn't *really* seem worth it
|
|
2022-01-22 11:09:12
|
well, ok, except on screen content
|
|
|
Scope
|
2022-01-22 11:14:38
|
Is it possible to fast identify such images and also is it possible to encode black and white and grayscale images more efficiently or for PNG as a format it doesn't really matter?
|
|
2022-01-22 11:17:25
|
Maybe do something like a quick heuristic only for images where predictors can have a noticeable effect, or would that make the code too complicated?
|
|
|
|
veluca
|
2022-01-22 11:19:24
|
I could do a few more tricks for images that are less than RGB888
|
|
2022-01-22 11:19:30
|
I didn't bother trying yet π
|
|
2022-01-22 11:19:51
|
identifying images where changing predictors would help... IDK
|
|
2022-01-22 11:20:19
|
in interesting news, out of a few images I was trying out I found this interesting one:
```
luca@desktop ~/fpnge ξ main $ FILE=/data/jpeg_xl_data/imagecompression.info.rgb8/spider_web.ppm.png; ~/fpng/bin/fpng_test $FILE && ./build.sh -DFPNGE_FIXED_PREDICTOR=4 && ./build/fpnge $FILE /tmp/fast.png 20 && compare -metric pae $FILE /tmp/fast.png null:; echo; ls -l /tmp/fast.png
SSE 4.1 supported: 0
Filename: /data/jpeg_xl_data/imagecompression.info.rgb8/spider_web.ppm.png
Dimensions: 4256x2848, Has Alpha: 0, Total Pixels: 12121088, bytes: 36363264 (34.678711 MB)
** Encoding:
FPNG: 0.069215 secs, 14317733 bytes, 13.654 MB, 167.010 MP/sec
lodepng: 5.287667 secs, 10890859 bytes, 10.386 MB, 2.186 MP/sec
stbi: 1.788477 secs, 17305221 bytes, 16.504 MB, 6.463 MP/s
qoi: 0.103213 secs, 14334346 bytes, 13.670 MB, 111.997 MP/sec
** Decoding:
FPNG: 0.074284 secs, 155.613 MP/s
lodepng: 0.265825 secs, 43.486 MP/s
stbi: 0.195687 secs, 59.072 MP/s
qoi: 0.091144 secs, 126.828 MP/s
398.386 MP/s
6.919 bits/pixel
0 (0)
-rw-r--r-- 1 luca luca 10482891 Jan 23 00:17 /tmp/fast.png
luca@desktop ~/fpnge ξ main $ convert $FILE /tmp/x.ppm && convert /tmp/x.ppm /tmp/libpng.png && ls -l /tmp/libpng.png
-rw-r--r-- 1 luca luca 10660099 Jan 23 00:17 /tmp/libpng.png
```
|
|
2022-01-22 11:20:50
|
(where fpnge is faster *and* denser than anything else, including libpng/lodepng)
|
|
|
190n
|
|
_wb_
Until you share it anywhere, then it gets transcoded to jpeg
|
|
2022-01-23 01:37:23
|
i think you can also configure the camera to save as jpeg in the first place
|
|
|
w
|
2022-01-23 01:39:22
|
iphones use heic
|
|
2022-01-23 01:39:25
|
but nobody uses heic
|
|
|
190n
|
2022-01-23 03:06:52
|
<:Thonk:805904896879493180>
|
|
2022-01-23 03:07:00
|
a lot of people use iphones and don't change default settings
|
|
|
w
|
2022-01-23 03:19:27
|
what i mean is that nobody actively works with heic
|
|
2022-01-23 03:19:54
|
actively and knowingly
|
|
2022-01-23 03:20:03
|
those people use images
|
|
2022-01-23 03:20:14
|
some would call them jpegs
|
|
|
_wb_
|
2022-01-23 07:13:56
|
https://twitter.com/tunetheweb/status/1484819698592137217?t=VX5cgjNklwXa2tDQMQ5MDA&s=19
|
|
|
Scope
|
2022-01-25 10:56:58
|
https://twitter.com/richgel999/status/1485878580898934785
|
|
2022-01-25 11:48:26
|
<@!794205442175402004> Also, can this be useful for the JXL/FJXL palette or is it very different?
|
|
|
_wb_
|
2022-01-25 11:58:51
|
maybe
|
|
|
|
veluca
|
2022-01-25 01:45:24
|
https://twitter.com/LucaVersari3/status/1485971553892323333 uploaded some slides with brief explanations of how fjxl & fpnge work
|
|
2022-01-25 01:45:32
|
also, a few fancy plots π
|
|
|
Scope
|
2022-01-25 02:10:38
|
Except that I would like to see `ECT -1`, instead of or in addition to OptiPNG, because it is more relevant and much faster and efficient, OptiPNG is pretty old
<https://github.com/fhanau/Efficient-Compression-Tool>
|
|
|
|
veluca
|
2022-01-25 02:15:13
|
ehhh Jon picked those π
|
|
|
Scope
|
2022-01-25 02:19:08
|
Well, then maybe after some further improvements for the FJXL and new comparisons
|
|
|
|
veluca
|
2022-01-25 02:22:50
|
that I can do
|
|
2022-01-25 02:44:37
|
yup, they were for a presentation
|
|
2022-01-25 02:44:40
|
but nope, no video
|
|
2022-01-25 02:51:21
|
who's "we"? π
|
|
2022-01-25 02:51:25
|
it should be easy to do so
|
|
|
Scope
|
2022-01-25 02:53:57
|
Such an encoder still has a rather niche use to replace other popular encoders, although a similar fast mode in some of them would be useful
Also, presentations to promote JXL or just some kind of research?
|
|
|
|
veluca
|
2022-01-25 02:54:46
|
just me finding out that people were interested in how fjxl and fpnge work and being asked to give a presentation
|
|
2022-01-25 02:54:55
|
and at that point, might as well publish the slides
|
|
2022-01-25 02:58:06
|
it's not exactly *hard* to make a SSE4 or so version... but yes
|
|
2022-01-25 02:58:17
|
I was too lazy to write it in a more portable way
|
|
|
Scope
|
2022-01-25 03:01:03
|
But at least there is also FPNG
https://twitter.com/richgel999/status/1485978449231683587
|
|
2022-01-25 03:13:00
|
And its -s variant is also pretty comparable (on average), although slower, but even that speed is usually excessive for normal use and if not required to encode petabytes of images every day
|
|
2022-01-25 03:20:22
|
And FPNG also has a very fast decoder for its own encoded images
`FPNG: 0.000657 secs, 2378.959 MP/s` <:monkaMega:809252622900789269>
|
|
2022-01-25 03:35:56
|
Almost lossless? π€
https://twitter.com/nikq/status/1485975181156057091
|
|
|
|
veluca
|
|
Scope
Almost lossless? π€
https://twitter.com/nikq/status/1485975181156057091
|
|
2022-01-25 03:36:09
|
not sure what that's supposed to mean π
|
|
|
Scope
And FPNG also has a very fast decoder for its own encoded images
`FPNG: 0.000657 secs, 2378.959 MP/s` <:monkaMega:809252622900789269>
|
|
2022-01-25 03:36:25
|
I guess that's with multithreading though
|
|
|
Scope
|
2022-01-25 03:37:19
|
As far as I know FPNG does not support multithreading (yet), most likely some images are just good for RLE
|
|
|
|
veluca
|
2022-01-25 03:46:17
|
that decoding speed seems too fast even for very good rle
|
|
2022-01-25 03:46:21
|
could be wrong though
|
|
|
Scope
|
2022-01-25 03:50:47
|
For some other images it's not that fast
|
|
|
|
veluca
|
2022-01-25 04:13:45
|
ah that makes sense then
|
|
|
Kleis Auke
|
2022-01-25 05:36:59
|
fpnge is interesting! They are currently dusting off and updating the PNG spec, one of the things that looks promising is the `mARK` chunk that allows parallel encoding and decoding (<https://github.com/w3c/PNG-spec/issues/60>).
|
|
|
Scope
|
|
veluca
https://twitter.com/LucaVersari3/status/1485971553892323333 uploaded some slides with brief explanations of how fjxl & fpnge work
|
|
2022-01-26 09:15:29
|
I'm also thinking, what if FPNGE used a combination of, say, the two most efficient for different content predictors to get the speed a little slower than one fixed predictor, but the efficiency is closer as like if all were used or is it useless? π€
|
|
|
|
veluca
|
2022-01-26 09:16:37
|
FPNGE tries all 4 unless you compile it in a special way
|
|
2022-01-26 09:16:49
|
if you switch to PAETH only it's not much worse
|
|
|
Scope
|
2022-01-26 09:18:33
|
Yes, I understand, but I meant to use only the 2 most useful instead of 4, or it would not much change the encoding speed?
|
|
2022-01-26 09:27:09
|
Still, all 4 predictors on some images or sets give a significant advantage compared to only PAETH and I am thinking that maybe a duo of some predictors will work better, with a small speed loss, compared to only one
|
|
|
_wb_
|
2022-01-26 09:27:51
|
Yes, something like fpnge24 or 34 could be a good idea
|
|
|
Scope
|
2022-01-26 09:33:36
|
Hmm, theoretically I can test it, if only I know exactly where in the code I can "remove unnecessary" predictors, I think it's somewhere here:
<https://github.com/veluca93/fpnge/blob/main/fpnge.cc#L441>
|
|
|
|
veluca
|
2022-01-26 09:51:03
|
here: https://github.com/veluca93/fpnge/blob/main/fpnge.cc#L708
|
|
2022-01-26 09:51:24
|
just comment out any subset of those "trypredictor<x>" lines
|
|
2022-01-26 09:51:40
|
well, as long as you leave at least 2 π
|
|
2022-01-26 09:51:50
|
(only 1 is equivalent to using a fixed predictor)
|
|
|
Scope
|
2022-01-26 11:37:16
|
According to preliminary tests, FPNGE34 does not differ much in speed from FPNGE4, but has better compression
|
|
|
|
veluca
|
2022-01-26 11:38:40
|
how much better?
|
|
|
Scope
|
2022-01-26 11:39:18
|
I will finish the spreadsheet for more accurate results
|
|
|
Nova Aurora
|
2022-01-27 07:40:49
|
WTF is Gralic?
I was on the compression subreddit and someone was recommending it for lossless archival purposes, and I've never heard of it before.
|
|
|
Scope
|
2022-01-27 07:43:40
|
One of the closed/proprietary formats from Alex Rhatushnyak, but it is efficient only for photo images (also a similar predictor is used in JXL `-e 3`)
http://qlic.altervista.org/LPCB.html
|
|
2022-01-27 07:51:39
|
Also, it's the same guy who once invented a revolutionary format (but it was lossy, so it had such compression) and I also once answered in a similar thread
<https://www.reddit.com/r/jpegxl/comments/mepoun/modern_lossless_jpegxl_still_defeated_by_old/>
|
|
2022-01-27 07:52:13
|
And also EMMA is better than Gralic
As a storage format they can be useful, but for other purposes very questionable because they have sometimes like 100-1000x slower decoding than Jpeg XL
Although, I would be afraid to store anything valuable and crucial for archiving in closed, very little tested, experimental formats
|
|
|
_wb_
|
2022-01-27 07:56:43
|
I don't know what gralic does, but the weighted predictor of jxl was also made by Alex Rhatushnyak
|
|
2022-01-27 08:01:07
|
Luca and me changed it a bit from Alex' original version, which had hardcoded parametrizations (3 variants for 8-bit, 1 for 16-bit) so we replaced that with signalled parameters, and the original also had an integer division per pixel (which is a crazy slow operation), which we replaced by something that can be implemented with a lookup instead.
|
|
2022-01-27 08:02:34
|
Also the original WP had its own hardcoded context model, and we replaced that with MA trees, which is strictly more expressive.
|
|
2022-01-27 08:04:02
|
(the original WP context model can be represented as a fixed MA tree that uses only the WGH property, which is what cjxl -e 3 does: it skips MA tree learning and just uses that fixed tree)
|
|
|
Scope
|
2022-01-27 11:02:54
|
<https://www.reddit.com/r/compression/comments/scvzmf/question_im_here_as_an_ignorant_looking_for/>
|
|
|
_wb_
|
2022-01-27 11:26:24
|
I don't know what tests he has seen, but the mention of PSNR is a red flag...
|
|
2022-01-27 11:28:00
|
PSNR correlates extremely poorly with subjective quality assessment
|
|
|
Scope
|
2022-01-27 11:53:39
|
https://cloudinary.com/blog/detecting_the_psychovisual_impact_of_compression_related_artifacts_using_ssimulacra
|
|
|
_wb_
|
2022-01-27 12:44:37
|
ssimulacra is better than psnr but it's still far from perfect β I made it when jpeg, j2k, webp, jxr were the main codecs of interest, and it was tuned to correlate with subjective eval of those.
|
|
2022-01-27 12:46:23
|
ssimulacra is quite forgiving for blurring artifacts, so I think it might be overestimating the perceptual quality of encoders that go heavy on the smoothing
|
|
|
|
Deleted User
|
2022-01-27 02:22:03
|
Why is there a difference between `cwebp -lossless` and `cwebp -lossless -q 100`?
|
|
|
_wb_
|
2022-01-27 02:24:34
|
is there? maybe different default effort?
|
|
|
Scope
|
2022-01-27 02:33:20
|
As far as I remember -m 6 and -q 100 used to activate the crunch mode, maybe it's something like that (but it was in older versions)
|
|
|
|
Deleted User
|
2022-01-27 02:47:32
|
```
A:\>cwebp -lossless -z 9 -m 6 -q 100 g8d.png -o 100.webp
Saving file '100.webp'
File: g8d.png
Dimension: 3840 x 2160
Output: 170476 bytes (0.16 bpp)
Lossless-ARGB compressed size: 170476 bytes
* Header size: 2302 bytes, image data size: 168148
* Lossless features used: PALETTE
* Precision Bits: histogram=6 transform=4 cache=2
* Palette size: 247
A:\>cwebp -lossless -z 9 -m 6 g8d.png -o ll.webp
Saving file 'll.webp'
File: g8d.png
Dimension: 3840 x 2160
Output: 824374 bytes (0.80 bpp)
Lossless-ARGB compressed size: 824374 bytes
* Header size: 14312 bytes, image data size: 810036
* Lossless features used: PREDICTION CROSS-COLOR-TRANSFORM SUBTRACT-GREEN
* Precision Bits: histogram=6 transform=4 cache=7
* Palette size: 247
A:\>cwebp -version
1.2.2
```
|
|
|
Scope
|
2022-01-27 02:50:14
|
Yes, it's probably a crunch mode
Also, since WebP was mentioned, does JXL use any palette reordering or sorting techniques for more efficient compression?
For example as it has been used here in WebP from the beginning of this source code:
<https://github.com/webmproject/libwebp/blob/master/src/enc/vp8l_enc.c>
Maybe also use something fast for the FJXL palette mode?
|
|
|
|
Deleted User
|
2022-01-27 02:51:32
|
Do you know how to use crunch mode with near lossless?
|
|
|
Scope
|
2022-01-27 02:53:15
|
Perhaps it works in the same way, but maybe not at all
|
|
|
_wb_
|
2022-01-27 02:55:51
|
both fjxl and cjxl sort the palette on luma and then on alpha (not quite in the same way though, fjxl looks at the Y from YCbCr multiplied by A, cjxl sorts lexicographically on YCoCgA)
|
|
2022-01-27 02:56:16
|
that's probably not optimal, but at least it's better than doing nothing
|
|
|
Scope
|
2022-01-27 02:57:55
|
And also the modified Zeng method that was recently mentioned:
<https://github.com/webmproject/libwebp/blob/master/src/enc/vp8l_enc.c#L152>
|
|
2022-01-27 03:01:08
|
Perhaps it will also be useful to use Select predictor and some techniques from lossless WebP for palette images π€
|
|
|
|
Deleted User
|
|
Scope
Perhaps it works in the same way, but maybe not at all
|
|
2022-01-27 03:51:19
|
ok, thx! I didn't made a difference between ll and nll in that case but it generally works that way.
|
|
|
_wb_
|
|
Scope
And also EMMA is better than Gralic
As a storage format they can be useful, but for other purposes very questionable because they have sometimes like 100-1000x slower decoding than Jpeg XL
Although, I would be afraid to store anything valuable and crucial for archiving in closed, very little tested, experimental formats
|
|
2022-01-27 04:14:28
|
Gralic for archiving is probably a very bad idea. I had to do some searching to find a "demo version", now installing wine so I can try to run the .exe
|
|
2022-01-27 04:15:57
|
a closed-source, proprietary, experimental, non-standardized format is pretty much the opposite of what you want for long-term archival purposes
|
|
|
Scope
|
2022-01-27 04:19:19
|
Yep, among such closed formats only BMF is relatively stable, at least it has a long history of existence and use and is more versatile for different content, not only for photos
|
|
2022-01-27 04:28:54
|
And also about Gralic unbeatability:
<https://www.reddit.com/r/jpegxl/comments/mepoun/modern_lossless_jpegxl_still_defeated_by_old/gtm8geq/>
|
|
|
_wb_
|
2022-01-27 04:48:09
|
well it does seem to get very dense results and at least on photos, jxl doesn't beat it in density
|
|
2022-01-27 04:48:53
|
or at least not yet - perhaps with future encoder improvements we can beat it, we're still quite far from exhaustively making full use of the bitstream expressivity
|
|
2022-01-27 04:54:44
|
on some photographic images, jxl _can_ already beat GraLIC, if you throw enough effort at it:
```
9871305 ClassA_8bit_BIKE_2048x2560_8b_RGB.ppm.png
7590448 ClassA_8bit_BIKE_2048x2560_8b_RGB.ppm.png.jxl3
7168248 ClassA_8bit_BIKE_2048x2560_8b_RGB.ppm.png.jxl7
6909524 ClassA_8bit_BIKE_2048x2560_8b_RGB.ppm.png.jxl9
6187395 ClassA_8bit_BIKE_2048x2560_8b_RGB.ppm.png.jxl9E3g3
15728657 ClassA_8bit_BIKE_2048x2560_8b_RGB.ppm.png.ppm
6566562 ClassA_8bit_BIKE_2048x2560_8b_RGB.ppm.png.ppm.gra
```
|
|
2022-01-27 05:00:57
|
for lossless I think there are three main actual use cases:
- fast encoding during authoring workflows: it's not that density doesn't matter here, but speed matters more - you don't want to stare at a "saving..." progress bar every time you hit ctrl-S
- archival: density matters, speed is less of an issue, but long-term reliability and interoperability is important: how likely is it that in the future, it will be possible to open the file? Standards are a must here.
- web delivery for images that just don't benefit (enough) from lossy, e.g. logos etc: here encode speed typically doesn't matter much, decode speed and browser support do matter a lot
|
|
2022-01-27 05:02:02
|
(and then of course there are niche use cases like medical, science, etc, where the requirements can be different)
|
|
|
Scope
|
2022-01-27 05:06:23
|
Yes, but also another advantage of GraLIC is that it has quite fast encoding for such results (but it takes the same time to decode)
Also this EMMA can be even denser, with some more encoding time, but still fast enough for these results
https://globalcompetition.compression.ru/assets/files/compressors/EMMA.7z
But, yes, they are suitable mostly for the theoretical storage use case (if imagining that encoders are very stable and safe)
|
|
|
_wb_
|
2022-01-27 05:07:09
|
I kind of wonder why Gralic and BMF are closed-source and don't just have a github repo β it doesn't look like the authors are planning to actually have commercial applications so they want to keep their secret techniques secret
|
|
2022-01-27 05:08:15
|
it's obviously too late to change the jxl bitstream at this point, but if they have interesting encoder-side heuristics, some ideas might be transferable
|
|
|
Scope
|
2022-01-27 05:11:58
|
Yes, BMF is almost 25 years old and it was not used for commercial purposes, it was just closed source, as well as in a recent discussion on encode.su, the EMMA author was not impressed with the JXL lossless results and said it was pretty easy to make something much more efficient and fast, but he also did not open the source code for his compressors (mostly because it is like bad code)
But if such more efficient codecs were open-source, it would be easier to use and practically test their ideas
|
|
|
_wb_
|
2022-01-27 05:12:18
|
decoder being as slow as the encoder doesn't sound good in that respect though - smells like it's mostly a good predictor / ctx model / entropy coder that is complex and symmetric, and not so much an expressive bitstream with fast decode but search-intensive encode
|
|
2022-01-27 05:13:24
|
not open sourcing an experimental encoder because it is bad code sounds silly to me - who cares about code quality of something experimental, just put a disclaimer on it that it's experimental
|
|
2022-01-27 05:14:13
|
if you can share binaries and expect people to run them, you can also share code
|
|
2022-01-27 05:20:44
|
if someone is claiming that it is pretty easy to make something much more efficient and fast for lossless compression, then I'd encourage them to submit their ideas as input to JPEG (WG1) so we can consider it for a potential extension of jxl, if there is really a low hanging fruit that we're missing.
|
|
2022-01-27 05:21:12
|
that or just make a github repo
|
|
|
Scope
|
2022-01-27 05:21:59
|
From fast encoders/decoders I found LEA the most interesting, moreover, as the author said, it is not optimized at all and is very easy to parallelize both encoding and decoding, by compression LEA can often be equal or better even to JXL -e 9, but with very high encoding speed, with some optimization theoretically comparable with JXL decoding speed
https://globalcompetition.compression.ru/assets/files/compressors/LEA.7z
|
|
|
_wb_
|
2022-01-27 05:22:06
|
but just putting magic .exe files on the internet is not going to lead to any long-term progress
|
|
2022-01-27 05:24:09
|
what is needed is at least a sufficiently detailed explanation of how the algorithms work β or even better, some code that can be studied
|
|
2022-01-27 05:25:41
|
claims of "not optimized at all, can be much faster" are something can only really be demonstrated by actually doing it
|
|
|
Scope
|
2022-01-27 05:26:02
|
Yes, one of the recent discussions was here, but similar ones were before, I think also such developers want to somehow monetize their knowledge, but perhaps this does not happen and the research results and ideas essentially just disappear into oblivion and the authors often switch to completely different fields
<https://encode.su/threads/3520-WebP-2-experimental-successor-of-the-WebP-image-format/page3>
|
|
|
_wb_
|
2022-01-27 05:28:14
|
I mean if you want to monetize your expertise, and get a contract somewhere, then putting something on github is going to be more effective than distributing some .exe files in obscure russian forums π
|
|
2022-01-27 05:29:49
|
it's mostly just a pity that there might be great ideas hidden inside GraLIC, BMF, EMMA, LEA, etc β and they are likely to just going to have to be rediscovered by someone else
|
|
|
Scope
|
2022-01-27 05:31:40
|
Except that some monetization is possible if winning GDCC and similar competitions, but this kind of monetization is not very stable
|
|
2022-01-27 05:38:50
|
Yes, for example such quotes:
> FLIF is quite poor, and it seems the lossless mode for JPEG-XL is nothing to write home about either.
> To elaborate why, allow me to use my poorly written entry from the GDCC'20, "LEA".
> It's about 700 loc (excluding the LZW code), written in an afternoon, single-threaded, doesn't use any SIMD, and since I was lazy I used my existing code for a planar image codec and didn't even bother converting it, i.e., it goes through the whole image 3 times, encoding a single color component in each pass.
> Now, LEA is trivially parallelizable, so let's simulate what we'd get with a multi-threaded version, by splitting the image into 4 slices (2 by 2). In that case, compression actually improves, down to just 36,271,421 bytes.
> And even accounting for obviously less than linear scaling, with 6 threads, decoding time would likely already be close to JPEG-XL at maximum effort (-e 9).
> With proper optimization, it would likely be on par with it.
> As for LEA, I have no doubts that either Shelwien or some of the other amazing coders in this forum could in less than 24h post a completely rewritten and much optimized version with proper multi-threading that beats even FLIC in terms of pareto-frontier.
> I can't do it, I'm a lousy coder on my best day, but it seems quite possible, even if it wouldn't be suited as a proper codec because it isn't amenable to hardware implementations or such.
>
> I just feel that a next-gen lossless format should actually be, you know, next-gen. So if it provides options that put its decoding performance in the same ballpark as other codecs, it better at least consistently beat them on compression ratio.
|
|
2022-01-27 05:38:59
|
> FLIF has its merits, but simply disregarding closed-source codecs, even if experimental, doesn't change the fact that better solutions exists.
|
|
|
Koromaru KorΓΌko
|
2022-01-27 05:42:20
|
this man sounds like a lunatic with 0 practical knowledge.
"in less than 24h post a completely rewritten and much optimized version"
|
|
2022-01-27 05:42:35
|
yea F\*\*\*\*\*\*\* Right
|
|
|
Scope
|
2022-01-27 05:46:39
|
It sounds like it, but if I didn't know who it was, I wouldn't have paid attention, but this man has won a lot of compression competitions (where very experienced compression engineers have also participated) and contributed a lot of code to various compressors
|
|
|
_wb_
|
2022-01-27 05:59:15
|
It's a funny mix to trash talk like that and say everything else sucks, you made something better in just an afternoon, but then at the same time saying you're a lousy coder and you're too embarrassed to show your code.
|
|
|
lithium
|
2022-01-27 06:14:30
|
For some game repacker they usually use LOLZ to compress texture,
LOLZ also good for general image compress?
|
|
|
BlueSwordM
|
|
Scope
<https://www.reddit.com/r/compression/comments/scvzmf/question_im_here_as_an_ignorant_looking_for/>
|
|
2022-01-27 06:14:38
|
My god.
|
|
2022-01-27 06:14:45
|
Someone measuring compression performance using PSNR <:whatisthis:672847617796997165>
|
|
|
Scope
|
|
lithium
For some game repacker they usually use LOLZ to compress texture,
LOLZ also good for general image compress?
|
|
2022-01-27 06:27:34
|
No, but it has special methods for compressing textures
|
|
|
lithium
|
|
Scope
No, but it has special methods for compressing textures
|
|
2022-01-27 06:51:20
|
I understand, thank you π
|
|
|
Nova Aurora
|
|
BlueSwordM
Someone measuring compression performance using PSNR <:whatisthis:672847617796997165>
|
|
2022-01-27 10:03:10
|
|
|
|
BlueSwordM
|
|
Nova Aurora
|
|
2022-01-27 10:04:51
|
<:megareee:900504155632828457>
|
|
|
_wb_
|
2022-01-27 10:19:42
|
So many things wrong with that quote
|
|
2022-01-27 10:20:35
|
One thing being that psnr is not lower-is-better but higher-is-better
|
|
|
BlueSwordM
|
2022-01-28 01:44:45
|
Please help me.
|
|
2022-01-28 01:44:53
|
https://old.reddit.com/r/compression/comments/scvzmf/question_im_here_as_an_ignorant_looking_for/huik5fh/
|
|
|
Nova Aurora
|
2022-01-28 02:49:04
|
https://www.youtube.com/watch?v=CriuRH9q-hU
|
|
|
Scope
|
2022-01-28 03:56:29
|
Measuring RGB values to determine quality?
|
|
2022-01-28 04:08:25
|
It is strange to use only LZ4 on RAW image formats and consider it unbeatable, I think even something like FPNG will win easily, also having very high decoding speed (and for more modern formats this can be done even more efficiently)
|
|
2022-01-28 04:22:54
|
Also about PSNR
https://videoprocessing.ai/metrics/ways-of-cheating-on-popular-objective-metrics.html
|
|
2022-01-28 04:23:34
|
|
|
2022-01-28 04:50:16
|
Yes, a very smoothed, outdated, bad for tests image (I do not understand why this image is still used, not even considering some moral aspects)
For both lossy and lossless, instead of getting a more varied set of more modern and higher quality images
|
|
|
Nova Aurora
|
|
Scope
Yes, a very smoothed, outdated, bad for tests image (I do not understand why this image is still used, not even considering some moral aspects)
For both lossy and lossless, instead of getting a more varied set of more modern and higher quality images
|
|
2022-01-28 05:08:01
|
Smoothed is a critical part of this to me since AVIF's first noticeable artifacts are usually smoothing out any fine details
|
|
|
Scope
|
|
Scope
It is strange to use only LZ4 on RAW image formats and consider it unbeatable, I think even something like FPNG will win easily, also having very high decoding speed (and for more modern formats this can be done even more efficiently)
|
|
2022-01-28 06:14:55
|
However, as I said before, it would be interesting to use methods good for very fast decoding in JXL (even if the encoding is not that fast) and maybe have some kind of specialized fast decoder for this in the future, even though it is not very good for spreading a complete format, but still, it is a pretty demanding thing
|
|
2022-01-28 07:10:12
|
Also for the sake of interest, while reading some blogs, does JXL use only rANS?
Something from FSE/tANS is not very suitable for such cases or does it not give any benefits?
<https://fgiesen.wordpress.com/2015/12/21/rans-in-practice/>
|
|
|
_wb_
|
2022-01-28 07:20:19
|
The entropy coding of jxl is MA context + configurable hybriduint (meaning a symbol gets split into an entropy coded token and some number of raw bits, depending on the token), optional lz77, and then either prefix (huffman) or ANS for the entropy-coded tokens (with a prefix code/ANS distribution that depends on the context)
|
|
|
Scope
|
2022-01-28 07:39:08
|
Yes, I meant among the ANS family
|
|
|
_wb_
|
2022-01-28 08:05:20
|
We do use some kind of tabling (the AliasTable) in ANS, so I am not sure what to call it. <@179701849576833024> maybe we need to come up with a name for the specific flavor of ANS we use, and describe it somewhere
|
|
|
|
veluca
|
2022-01-28 08:15:07
|
It is a rANS
|
|
2022-01-28 08:15:34
|
With this fancy alias table trick that allows to use O(symbols) memory instead of O(state) at decode time
|
|
|
Nova Aurora
|
|
Scope
Yes, a very smoothed, outdated, bad for tests image (I do not understand why this image is still used, not even considering some moral aspects)
For both lossy and lossless, instead of getting a more varied set of more modern and higher quality images
|
|
2022-01-28 08:20:43
|
> Revolutionalredstone
>
> Here's a quick reference comparison: https://imgur.com/a/vBgZNp3
>
> Definitely difficult to see any difference when viewed without zoom.
>
> I'm sure I won't need to mention how horrid a 24kb jpeg would look.
>
> I do think for such a grainy image a slightly higher setting would be good.
>
> Ta
|
|
2022-01-28 08:21:13
|
I find that not visually lossless at all <:Thonk:805904896879493180>
|
|
2022-01-28 08:21:25
|
It's smoothed out all the grain
|
|
2022-01-28 08:22:14
|
Also how did an adult image become part of the standard testset?
|
|
|
Scope
|
2022-01-28 08:25:09
|
Yes, because such comparisons are very subjective, some people like smoothness even more than the original
About the test image, I think because the engineers were mostly young men π€
|
|
|
_wb_
|
2022-01-28 08:28:24
|
https://en.wikipedia.org/wiki/Lenna
|
|
2022-01-28 08:31:04
|
To explain Lenna's popularity, David C. Munson, editor-in-chief of IEEE Transactions on Image Processing, noted that it was a good test image because of its detail, flat regions, shading, and texture. However, he also noted that its popularity was largely because an image of an attractive woman appealed to the males in a male-dominated field.[10]
|
|
2022-01-28 08:31:20
|
I assume it's mostly the latter
|
|
2022-01-28 08:40:48
|
It's not a particularly good test image imo, since
- it is a scan of a print
- a scan from 1973
- only a narrow part of the color gamut is used
- most of the image is noisy and out of focus
- it's only a thumbnail by today's standards
|
|
|
Scope
|
|
Nova Aurora
> Revolutionalredstone
>
> Here's a quick reference comparison: https://imgur.com/a/vBgZNp3
>
> Definitely difficult to see any difference when viewed without zoom.
>
> I'm sure I won't need to mention how horrid a 24kb jpeg would look.
>
> I do think for such a grainy image a slightly higher setting would be good.
>
> Ta
|
|
2022-01-28 08:50:19
|
And I also do not see anything magical for AVIF at these bitrates, for example 24kb JXL
|
|