JPEG XL

It's not perfect but I found it useful to clean up images before downscaling.

I would recommend trying jpeg2png as well

they both have their use cases

QuantSmooth is included in IrfanView, that's how I discovered it.

Should work excellently on text, graphics

probably smoothes much more than ideal on photographic content

Seems to work best for non-photo content in jpeg of quality above 75. It can also decrease the impact of chroma subsampling.

Have anybody tried using zopfli compression in TIFF images? Might be useful for images unsupported by PNG, like CMYK.

<@219525188818042881> you won't beat me this time <:PeepoDiamondSword:805394101340078092>

What if I do? <:KittyThink:1126564678835904553>

🍕

but you won't

Still waiting for all the other pizzas smh

mmmmh i don't remember from when I owe you one [⠀](https://cdn.discordapp.com/emojis/1088952745471516815.gif?size=48&quality=lossless&name=laugh)

I do and not just 1 <:KittyLaugh:1126563616343216268>

I mean, you always said "pizza if you want to know how" but I still don't know lmao

Maybe it's time to buy that knowledge then <:KittyUwU:1147753612529913938>

What if I gift you a 2 week Nitro trial thing if I can't beat you? <:KekDog:884736660376535040>

The two jpeg foxes fighting in my head

<@693503208726986763> made it smaller

I don't need Nitro [⠀](https://cdn.discordapp.com/emojis/862625638238257183.webp?size=48&quality=lossless&name=av1_chad)

# F

[⠀](https://cdn.discordapp.com/emojis/654081051768913941.webp?size=48&quality=lossless&name=av1_PepeHands)

Why did you think I couldn't make it smaller though? <:KittyThink:1126564678835904553>

because I managed to shrink just a little more after jpegultrascan -b 13

but apparently not enough

shit

Can't beat me, I'm very experienced with small things™️

Am I missing something? Is there anything better than ECT?

I've found a PNG, where ECT just couldn't beat PNGOut. Maybe it could beat it eventually, but it ran for more than 24 hours with no success. PNGOut is just too good in compressing flat colored lineart.

for JPGs ? yeah https://encode.su/threads/2489-jpegultrascan-an-exhaustive-JPEG-scan-optimizer

Looks like a wrapper for jpegtran. ECT uses mozjpeg jpegtran as well.

Are there precompiled Windows binaries for mozjpeg tools somewhere?

yeah but it's a bruteforcing script, with exhaustive scan search so really slow but always compresses better than ECT

I see. I thought the options for jpeg in ECT are quite sparse compared to options for zopfli/png.

Can you send the png? Also pingo and oxipng are also very good for optimizing pngs, they don't offer brute forcing capabilities but their lossless reductions are top notch. ECT falls behind in that area.

Here is png benchmark showcasing which lossless reductions are possible: https://css-ig.net/benchmark/png-lossless

again, you can go further than just `-9` with ECT and ECT is the best Deflate optimizer I know The only way to beat it in PNG filesize is to have somehow a better filter than the ones it tries

I haven't tried -99999 yet, but I don't feel like crunching a single png for a week. I've tried some lower values, though. PNGOut excels on images with solid colors, where the zero filter is the optimal choice for the entire image as the png predictors won't do any better.

-99999 is overkill, usually `-30069` works fine with `--mt-deflate`

`69` [⠀](https://cdn.discordapp.com/emojis/586100936544616450.webp?size=48&quality=lossless&name=sataniasmirk)

<a:RaysNut:737691574221406299>

I've tried -100, -30060, -30100 and -90100

What are the results?

Saved 0B out of the file.

The file was previously compressed using `pngout /f0`

Here is the file in question if somebody would like to tinker with it.

anyone already tested this ? https://github.com/thorfdbg/libjpeg/

Tested what about it, visuals, performance, features?

quality/filesize ratio (but I doubt anyone has tested it yet tho)

`jpegtran ect leanify pingo jpegoptim curtail jpgcrush jhead` you didn't use any of them I hate you, I wasted 4 hours searching and testing, to save 0 byte <:KekDog:805390049033191445>

I don't even know a good JPG analyser (to see the difference between yours and mine)

It has overwhelming assortment of features. I wonder which viewers can decode those fancy jpegs.

just tested it and the lossless compression it provides is [⠀](https://cdn.discordapp.com/emojis/852007419474608208.webp?size=48&quality=lossless&name=av1_woag)

```$ cjxl -d 0 -e 10 --brotli_effort=11 -x strip=all -I 100 -g 3 -E 11 1094077 Bytes 1m16.536s ~3 threads $ jpeg -c -ls 1 -cls 1289537 Bytes 0.868s 1 thread $ ect -60199 --mt-deflate 1642880 Bytes 49.927s ~3 threads``` This is quite really good lossless compression, and more importantly speed 0.o

but I feel like this lossless mode isn't supported by anything while using a worse lossless JPG compression (just `-p`) is still supported by e.g. FFmpeg

also supports a "normal" JPG encoding using XYZ colorspace

this encoder is really full of features

but at the same time it comes with so many arguments for the CLI <:KekDog:805390049033191445>

It's the reference implementation of JPEG, made by long-time JPEG member Thomas Richter. It's the first implementation that is actually implementing the full JPEG spec 🙂

(and also it implements JPEG XT, if I remember correctly)

(and JPEG LS too, I think)

[⠀](https://cdn.discordapp.com/emojis/872898333239279666.webp?size=48&quality=lossless&name=av1_frogstudy)

and yeah it implements XT and LS

Good, I also hate myself <:KittyUwU:1147753612529913938>

Gotta waste some more hours :D

<:KekDog:805390049033191445>

I gave up trying to compress my png file further. It's attached above if anybody is interested. Ryzen 5 1600 is not the fastest CPU in the world, although I've just recently upgraded from Core i5 3570 😄

gonna try it later currently optimizing a 8GB FLAC album <:KekDog:805390049033191445>

Ah FLAC, that sounds interesting too.

well, I could use something else for better compression but FLAC support is so good

SAC my beloved <:trolldoge:1200049081880432660>

no never again

I mean it's nice and all but you compile it with different compilers and you won't have same encoded (or decoded) file <:KekDog:805390049033191445>

If I had to pick something else than FLAC I would happily go with LA

ECT was able to optimize mp3s, although it seems it just tried to optimize jpeg and png images embedded inside mp3. My mp3s don't have any images in them, so no use for me.

so you want a MP3 lossless optimizer? [⠀](https://cdn.discordapp.com/emojis/1042213453189873716.webp?size=48&quality=lossless&name=esmirk)

MP3packer <:KittyUwU:1147753612529913938>

nooooo you just threw away my chance of getting a pizza

I thought mp3 uses set of fixed bitrates.

b...b...but, you discovered through me so doesn't that reset the pizza count ? <:KekDog:805390049033191445> same for FLACcid xD

I didn't <:KekDog:884736660376535040>

shit

well MP3Packer all it does is minimizing the frames' padding so it can convert losslessly CBR to VBR (and inversely)

but even with VBR input it manages to gain some kB

I see it can be useful to clean up mp3s

well then for FLACcid I'm sure lol this was the first battle we had, good times <:CatSmile:805382488293244929>

I was just curious what ECT does with mp3 but it only optimizes embedded images which I've already thrown away.

you're on Linux or Windows

Windows

you might want give it a try <https://hydrogenaud.io/index.php/topic,32379.0.html>

Is there any thing that does that on Mac? It’s exactly what i need but I only have Macs

do Linux executables work on Mac ?

if no, you'll have to compile it yourself and it's really a hassle because it's OCaml, with unsafe-string

No, as Mac uses Mach-O instead of ELF

I could try WINE

yeah works with wine

I’ll try that when I get hoe

Home*

how dense is the jxl effort with the closest speed?

```$ cjxl -d 0 -e 4 1208525 Bytes 0.878s ~3 threads $ cjxl -d 0 -e 3 --num_threads=0 1210889 Bytes 0.404s $ cjxl -d 0 -e 4 --num_threads=0 1208525 Bytes 1.134s```orig benchmark:```$ cjxl -d 0 -e 10 --brotli_effort=11 -x strip=all -I 100 -g 3 -E 11 1094077 Bytes 1m16.536s ~3 threads $ jpeg -c -ls 1 -cls 1289537 Bytes 0.868s 1 thread $ ect -60199 --mt-deflate 1642880 Bytes 49.927s ~3 threads```

Yes, but that encoder was broken lmao

I filed an issue and the author solved it lmao

for flac, I recently played a little with http://cue.tools/wiki/FLACCL

it certainly is fast

density is fine but only occasionally better than the official encoder

yeah, as expected for a GPU encoder

Yes, but as far as I know there isn't a compiled version of the fixed encoder and I'm too lazy to learn how to compile <:KekDog:884736660376535040>

there's one I made but for Linux <:kekw:808717074305122316>

I know. Which is why I'm not using FLACCID anymore XD

https://hydrogenaud.io/index.php/topic,123248.msg1043482.html#msg1043482 just do a little search [⠀](https://cdn.discordapp.com/emojis/895863009820414004.webp?size=48&quality=lossless&name=av1_thinkies)

What about... no <:KittyUwU:1147753612529913938>

well since I gave you the link it's not a search anymore for you <:KekDog:805390049033191445>

That's why I don't have to search anymore :D

But sadly won't be using FLACCID anymore anyways since some applications can't decode it

true but at least all my apps support it

~~shit I should have asked for a pizza~~

Can any of these tools merge duplicate copies of quantization tables?

[⠀](https://cdn.discordapp.com/emojis/586168843781668876.webp?size=48&quality=lossless&name=AkkoShrug)

the only thing I know is that these can't improve compression over jpegultrascan

But it would only be worth a pizza if it's valuable information to me <:RaysEvilUglyClone:1126526202178441277>

right <:PepeHands:808829977608323112>

Some people use regular jpeg at q=100 in a hope it would be lossless. It's not the best idea but these jpegs do exist. In these files, luma and chroma quantization tables are identical, all contains only 1s. So using just one copy for both luma and chroma easily saves 65 bytes.

Most jpeg optimizers seem to be based on jpegtran. jpegtran will discard unused quantization tables so the SOF marker can be modified, so both luma and chroma will use one copy of the quantization table and jpegtran will remove the redundant copy. I edited the SOF using hex editor but it shouldn't be too hard to add check for multiple identical quantization tables in the optimizer programs.

Meanwhile, I discovered even more pngs compressed by pngout where ECT is failing.

indeed, but I won't try `-99999 --palsort=120 --allfilters-b` on the file you send yesterday <:KekDog:805390049033191445>

I wonder: how much bytes less is the pingo output compared to something like `ect -90099 --mt-deflate` ?

I haven't tried pingo, the sparse documentation is not convincing.

I meant pngout lol

Also ECT doesn't say how much the file size would be when it fails.

Bu I can do the comparison by starting with "unoptimized" file.

yeah it only shows the nbr of saved bytes

According to my tests, `--mt-deflate` actually increases the file size by several bytes quite often.

I takes much longer though.

how can mt-deflate take longer lmao

Sorry, omitting --mt-deflate takes longer.

So yeah, `--mt-deflate` is sometimes better and sometimes worse. It depends on the image.

Alright. I will do more tests. I suppose the result will be tomorrow 😄

[⠀](https://cdn.discordapp.com/emojis/674256399412363284.webp?size=48&quality=lossless&name=av1_pepelove)

If there are already optimizers for audio files, the rabbit hole can go much deeper. What about optimizing video files?

I can imagine hardware video encoder do quite sloppy job.

Alright, didn't take too long. `pngout /f0` does 336683 bytes `ect -90099 --mt-deflate` does 336869 bytes

I know some audio optimizers but never heard of any video ones [⠀](https://cdn.discordapp.com/emojis/654081052108652544.webp?size=48&quality=lossless&name=av1_Hmmm)

<:FeelsReadingMan:808827102278451241>

https://github.com/danielrh/losslessh264

isn't that just `-crf 0` ?

The name is misleading, it's a recoder/recompressor

Made by dropbox along with their jpeg compression

https://github.com/dropbox/avrecode

ah, a sort of brunsli for h.264?

avrecode acts like an archive (only avrecode can decompress the encoded files)

nice, thanks !

Or at least that's what it says

I just remember finding it a few months ago then wanting this after https://elecard.com/products/video-analysis/streameye

suprisingly brunsli gives the smallest output for the unoptimized JPG and the biggest output is the most optimized JPG (with jpegultrascan)

and to **de**optimize a JPG, the only command I know is `jpegtran -revert`, idk if there's something else (or something better)

```bash $ find *jpg -printf '%s %p\n' | sort -n 365977 1.jpg 366319 2.jpg 376318 3.jpg 391187 4.jpg $ find *jxl -printf '%s %p\n' | sort -n 314107 3.jxl 314323 2.jxl 314346 1.jxl 314381 4.jxl $ find *brn -printf '%s %p\n' | sort -n 313474 4.brn 313566 3.brn 313743 2.brn 313759 1.brn ```they're all the same file (losslessly speaking)

JXL done with `cjxl -e 10 -d 0 --brotli_effort=11 -I 100 -g 3 -E 11` ofc but idk if that changes anything for lossless_jpeg=1

What a strange image, I unoptimized it then tried recompressing it with pngOUT. I could never even get close to the 328 KB file you sent me. The best result I got with pngOUT `/f0` was 352 KB. Here is how the other png optimizers did: ECT `-9`: 329 KB Pingo `-s4 -l`: 330 KB Oxipng `-o max`:330 KB

mmmmh note that running some optimizers after other ones might help compression

for example try pngOUT on the output of ECT, I wonder if it can shave off some bytes

Sorry I forgot to tell you that pngout is quite quirky and it's compression depends on previous compression. So, first run pngout without options and then pngout /f0

Ok now I got that image down to 328 KB

But that just raises more questions than answers

Like why does running the same image twice through the same program make it smaller?

Does it have something to do with block splitting?

This is just a quirk of pngout. The block splitting seems to be the culprit.

Yup, just got a smaller image tweaking the block splitting setting with pngout

I managed to save like 100 bytes just by manually trying different block sizes

Yes, that's possible. Pngout seems to choose suboptimal block splitting when the filter type is enforced right away. But it's not an issue if the png was optimized by pngout beforehand. Unfortunately pngout is closed source so who knows what's going on. It could be a bug. I'm trying to move to open source optimizers.

Yup, I love open source optimizers. I got a bug fixed in oxipng recently.

I guess to compress an animation for web use, AV1 would be a decent codec. But is there a way to get over 8 bits per channel (let's say 12) and still decode it in a browser, reading the 12bit data in JS? If the app has to resort to Wasm, maybe JPEG XL is a competitive option even for HDR video 🤔

JXL will never be competitive for lossy video, AV1 supports 10bit on most profiles, and 12bit is usually only supported in software. if you need greater then 12bit then av1 is not usable. now if you are using JXL in a professional editing workflow where you are pretty much just using complete image sequences, then yes, JXL is quite good there

Use case is needing to access the decoded frame in JavaScript. I'm using lossless JXL for the raw data and wondering how to deliver compressed data to the client without losing too much flexibility. I guess one option is to decode AV1 in Wasm and provide a separate desktop app if someone is serious about using the data instead of just testing and poking around.

Could Motion JXL(?) be a decent way to compress my giant ProRes 422HQ archive folder?

How does the compression ratio compare?

https://discord.com/channels/794206087879852103/794206170445119489/1116377019928805426

Hmm, I’ll do some testing myself on my own machine but it might work

How do I stuff JXL images into an MKV though?

Other issue would be converting from YUV422-bt2020 to whatever JXL uses

I guess you could use the HDR canvas instead of the regular image canvas to get access to 16-bit buffers from 12-bit avif / apng / jxl images/animations. But I think only Chrome supports the HDR canvas...

Thanks, will test! I'm fine with some features only on Chrome, it's still better than a custom desktop app.

In principle JXL can represent yuv422 losslessly (at any bitdepth, in any underlying rgb colorspace), as long as the yuv matrix is the jpeg one (full range). In practice though, libjxl does not have any API to pass buffers as yuv (or to skip the decoder conversion from yuv to rgb), since the API is based on interleaved RGB(A), so there currently is no way to do that except by hacking libjxl in nontrivial ways.

Likely trying to do a lossless conversion will not be effective anyway, unless somehow ProRes frames can be mapped to VarDCT frames in an exact way (like we do for JPEG recompression) — I don't know if that's possible though, I don't know ProRes well enough.

If some additional loss is allowed, I guess you can just convert the yuv422 to RGB and do standard lossy jxl encoding on that.

you need a build of ffmpeg using this patch you can remove the qoi part if you want, change the +34,8 to +34,7 if you remove it ```diff diff --git a/libavformat/riff.c b/libavformat/riff.c index df7e9df31b..16e37fb557 100644 --- a/libavformat/riff.c +++ b/libavformat/riff.c @@ -34,6 +34,8 @@ * files use it as well. */ const AVCodecTag ff_codec_bmp_tags[] = { + { AV_CODEC_ID_JPEGXL, MKTAG('J', 'X', 'L', ' ') }, + { AV_CODEC_ID_QOI, MKTAG('Q', 'O', 'I', ' ') }, { AV_CODEC_ID_H264, MKTAG('H', '2', '6', '4') }, { AV_CODEC_ID_H264, MKTAG('h', '2', '6', '4') }, { AV_CODEC_ID_H264, MKTAG('X', '2', '6', '4') }, ```

Some time ago, I discovered a particular jpeg file. https://discord.com/channels/794206087879852103/804324493420920833/1242368843461169252

I can't figure out, what's wrong with the file, I don't see anything suspicious. djpegli can't decode the jpeg file. cjxl can't transcode the jpeg file to jxl losslessly. jpehtran will fix the file so both djpegli and cjxl will be able to process it.

the thing to take note when using JXL to compress a "working file" I **highly** reccomend using faster_decoding=3, it does increase the filesize a good chunk, but the video actually remains usable in an NLE or whatever you use

also its very important to note that the comparison I made is from the same source, and it's comparing prores HQ from ffmpeg's prores_ks encoder. so it should still save more data, but always keep in mind you will need to do a lossy encode to actually save the data

If you want to figure out why things fail in libjxl, it's very useful to build your libjxl with `./ci.sh opt` instead of building the release build where debug output has been stripped. Likely you'll see a more useful error string that way.

Well, I guess I will have to figure out, how to compile libjxl. I know that lossless transcoding is sometimes picky about jpegs, but jpegli should be able to decode all jpegs.

can it do cmyk

I haven't tested CMYK yet. The problematic file is supposedly just an ordinary YCbCr progressive JPEG, jpegtran can convert it losslessly to baseline JPEG.

**JPEG2000 support removed from Safari 18.** As of the beta, apparently, JPEG2000 support is completely removed. I have not tested this yet but as the Webkit team notes: *"WebKit for Safari 18 beta removes support for the JPEG2000 image format. Safari was the only browser to ever provide support. If you’ve been serving JPEG2000 files using best practices, then your site is using the picture element to offer multiple file format options to every browser. Safari 18 beta will simply no longer choose JPEG2000, and instead use a file compressed in JPEG XL, AVIF, WebP, HEIC, JPG/JPEG, PNG, or Gif — choosing the file that’s best for each user. Only one image will be downloaded when you use <picture>, and the browser does all the heavy lifting. We have noticed that some Content Deliver Networks (CDN) use User Agent sniffing to provide one file to each UA, offering only JPEG2000 images to Safari — especially on iPhone and iPad. If you expect this might be happening with your site, we recommend testing in Safari 18 beta on both macOS Sequoia and iOS or iPadOS 18. If you see problems, contact your SaaS provider or change your image delivery settings to ensure your website provides fallback images using industry best practices."*

there should be no issue with that. i think pdf docs with jp2 images dont rely on safari/webkit to render anyway

unlike as implied by the myth, formats can absolutely be shelved, especially if theyre almost exclusively only served by CDNs as 'optimized' substitutes for the original images that are typically jpg or png (like with webp) sites maintained in the last few years can offset any adverse consequence of their use by simply using any other format available or the originals, unlike geocities-era abandonware that doesnt suffer such an issue anyway

*.swf used to be a good way of publishing animation for web. Infinite resolution and a small size.

It's incredibly upsetting that nothing replaced Flash for animation. We have GPU accelerated vector libraries now, it's way easier to make a performant animated vector format these days than when Flash started

Animated SVG, meanwhile, can bring even i9 to knees

Oh no Nintendo can no longer give me Switch thumbnails in .jp2<:PepeHands:808829977608323112>

Some Cloudinary users were actually still using j2k on Safari/iOS. For now it still seems to work (it still decodes it) but we're trying to track down users who are doing hardcoded `f_jp2` in Cloudinary (instead of using our `f_auto` where we can make sure we don't send a format that will not work), which is a bad idea anyway but it will become even worse when Safari really pulls the plug on it.

But yes, this is actually nice to see, that a browser can actually deprecate a format it used to support.

This has been used many times as an argument against adding JXL support: "the bar for new formats is incredibly high because once added they have to be there forever" has basically been the dogma so far, although even before the news of Safari pulling j2k, there were some precedents: Edge dropping JXR (WDP), all browsers dropping XBM.

at least Safari has JXL and so does my Thorium install

as i use mac and iphone (although thinking about Android)

i installed through Homebrew, how would i go changing the compilation?

not sure, I dont really use homebrew so I couldn't really say

because i could compile it but then i do not know how to link that into anything else

like mpv

if the application is linked against the ffmpeg it should just work, mpv wont work oob since it uses it's own mkv demuxer, using the same patch you can mux into a nut and that will work with mpv better

my mpv build is linking against homebrew's ffmpeg /opt/homebrew/opt/ffmpeg/lib/libavcodec.61.dylib /opt/homebrew/opt/ffmpeg/lib/libavfilter.10.dylib /opt/homebrew/opt/ffmpeg/lib/libavformat.61.dylib /opt/homebrew/opt/ffmpeg/lib/libavutil.59.dylib /opt/homebrew/opt/ffmpeg/lib/libswresample.5.dylib /opt/homebrew/opt/ffmpeg/lib/libswscale.8.dylib /opt/homebrew/opt/ffmpeg/lib/libavdevice.61.dylib

maybe ill just decode in NUT and then pass as raw video to other programs as a pipe

Are there any tools that do that that are more up to date?

MP3Packer breaks on joint stereo for me

MP3Packer doesn't break for 100+ files I gave it, could you share your problematic file ?

and no, there's no other tool than MP3Packer, and not "more up to date" because there's nothing more MP3Packer can do <:KekDog:805390049033191445>

Maybe wine is screwing with something

Hang on I’m on phone I’ll send u the file in a bit

wait, you're trying it under Linux ?

then use the Linux executable <:KekDog:805390049033191445>

No I’m on MacOS

oh

right

I guess I could magic wormhole the files up to my Linux vps

~~you could also compile it yourself~~

wait, is your MAC arm64 ?

Screw ocaml

Yes

then I already compiled a static version for it [⠀](https://cdn.discordapp.com/emojis/895863009820414004.webp?size=48&quality=lossless&name=av1_thinkies)

How?

well for aarch64 (my phone) but it's the same arch

Can you send makefile?

I couldn’t get OCaml to compile it as something something immutable strings

I'll send you the exec lmao I compiled it more than a year ago, I don't remember how I did it (and it was a fucking headache bc of OCaml) <:kekw:808717074305122316>

yeah, so you have to even build OCaml yourself with `--use-unsafe-string` (something like that) during the config <:kekw:808717074305122316>

<@386612331288723469> MP3Packer, version 2.04, 64-bit, compiled with optimizations (`-O3 -flto`), generic, statically linked, stripped. Compressed using `upx -9 --ultra-brute` 4.0.2 https://cdn.discordapp.com/attachments/1042536514783023124/1080563572960133161/mp3packer_aarch64_clang_static

Is it ELF or MachO?

I don't understand

it's the same as if I did `clang -flto -s -static -O3 -mtune=generic -o mp3packer mp3packer.c`

it's a build from my arm64 (aarch64) phone butperhaps what you mean is that it needs something special to work on MacOS ?

I thought it didn't need anything bc MacOS is based on Linux [⠀](https://cdn.discordapp.com/emojis/654081052108652544.webp?size=48&quality=lossless&name=av1_Hmmm)

nah, macOS is based on Unix, but not Linux specifically

so it uses the Mach-O

wait where do you get a .c file from?

i could just build the .c file using MacOS clang

nowhere, I just used this command to show how it was build and with which params but ofc there's no C file

could you try the executable I sent?

it won't work `ELF 64-bit LSB executable, ARM aarch64` MacOS uses `Mach-O 64-bit executable arm64` sorry

huh it's working now

odd behaviour

maybe wine update fixed it

it was this file btw

nvm it's broken for this file

it gives this

it's dropping the MS encoded audio packets i think

optimized using MP3Packer x86-64: `0,a,murmur3=20f7d9636eed5d59a08677f1f4e79584` same hash, no prob

again, no prob on my end

probably a bug in the version of mp3packer i'm using

updating to 2.04 fixed the bug for me

https://github.com/apple-oss-distributions/xnu

It's its own thing that uses a lot of FreeBSD code (for some reason)

It has no relation to Linux and is very different than Linux. Linux was loosely inspired by Unix which is why Linux loosely/vaguely resembles BSD, if BSD was dropped on its head several times as a baby

I didn't want to say Linux, but rather Unix

MacOS is POSIX compliant though

It's just that I thought any static prog compiled on Unix could be run on any Unix machine but pparently and sadly no [⠀](https://cdn.discordapp.com/emojis/654081051768913941.webp?size=48&quality=lossless&name=av1_PepeHands)

very much no

you can take the simplest x86 program and run it on windows and linux

no, it still has to be compiled for that specific environment which at minimum includes kernel, and often times libc, though some static compilation like musl usually will link libc into it

that latter thing only works on linux

I think id seen a couple others where similar things work, but regardless it still kernel dependant, and ofc any other deps

linux is the only kernel with a stable kernel interface 😄

every OS that doesn't use the Linux kernel, has you use some kind of dynamically linked library to access the kernel

On POSIX OSs, this is libc

Which is a horrible idea, incidentally... the only reason why Linux does this is because they want to avoid making an actual complete OS with a stable high-level interface.

Having a stable low level kernel interface makes it much harder for Linux to improve since they have to worry about breaking compatibility... if they just developed a full OS then they wouldn't have to worry about changing low level interfaces as long as the high level interfaces are the same. But for some reason the Linux maintainers never wanted to take on the responsibility of adding essential userspace stuff to their source tree

They would rather let someone else worry about it even if it's a complete holy mess like glibc

It seems completely arbitrary too, since they act like writing basic userspace tools is some magical line they cannot cross that their mind simply cannot comprehend. It started with Linus talking like that

"I can write a kernel and (userspace tool) git but I just can't include a c library and basic daemons."

can you elaborate on this? because I can't see the path of logic here

They don't want to provide any high level userspace stuff. They make a ton of arbitrary exceptions to this rule of course because of how insane and impractical it would be to stick to that rule.

So, they provide a low level kernel interface and try to keep it stable in order to make it easier for some outside group to do the rest of the work for them.

I dont think the kernel developers providing high level userspace stuff really makes a lot of sense

They don't have a rule against userspace stuff

But they're all kernel devs, not OS devs

Making an OS would be a big shift

You mean like every other open source OS project and source tree?

if you take a look at what happened with windows, it's a way worse situation, they have to actively go out and emulate previous versions of windows to retain backwards compatibility, the maintenance burden is fairly high for that

osx just doesn't care enough to really be bothered, "if it breaks it breaks" kind of thing

Sure but it makes sense to keep things in the same source tree and provide everything under one project umbrella for many different obvious practical reasons. The decision bites Linux in the ass often IMO

not really, it provides great flexibility and support, having the linux kernel actually be stable is the reason why linux dominates in pretty much every single market aside from desktop, and the reason it doesn't dominate desktop is because no one cares about it

A stable kernel api has nothing to do with why people use it

it has excellent backwards compatibility, it's really rare that a program actually breaks when a kernel updates

The only reason they do that is because they don't want to include libc in their source tree for irrational reasons

unlike with something like say windows server, every major upgrade you need to update your program or deal with the usually spotty backwards compatibility issues

there are loads of reasons not to do so

like the fact that actually supporting varying libc implementations is great for security, as well as perf

You don't have to worry about kernel updates, you have to worry about glibc updates breaking compatibility with everything...

that's because glibc is dogshit, just use musl or a different libc

or dont update libc outside of major security updates

They trust GNU to do everything for them and them they regret it over and over again and had to fork glibc several times even

no they dont

the explicitly don't trust gnu to do everything for them.

They do though. That is their whole reason why they don't just include musl into the linux source tree

no, they dont

they actively support many libc impls

<#806898911091753051> <:Hypers:808826266060193874>

They would rather someone else take responsibility even if it ends up making Linux suck

off the top of my head I actively use bionc, musl, and glibc, which is three extremely seperate things

[av1_dogelol](https://cdn.discordapp.com/emojis/867794291652558888.webp?size=48&quality=lossless&name=av1_dogelol)

to be fair, older versions of Windows were using an entirely different kernel altogether

so just keeping a kernel’s interface stable wouldn’t have helped

indeed, but even say winxp -> windows vista/7 was a massive shift in kernel so a lot of things broke, a lot broke with the 7->10 too, but windows emulation is genuinely quite good so it's not a massive issue, but it is a whole thing they still need to do.

from what I recall, most userspace breakage from XP to Vista was because of the emphasis on “not running everything as admin by default”

“what do you mean I can’t just write the user’s preferences to a file at the root of `C:\`?”

well there was that too :D but we can't forget that a large amount of windows' emulation is "transparent" to the user, a good example of this is some of the old directx stuff, it still works on windows, but a lot of those old libraries needed to be updated for newer technology stacks. this is partially the reason why windows is so bloated now, they have to ship all these really old libraries to get backwards compat

and ofc, this sometimes doesn't work at all, and you need to emulate the full environment which is the "compatibility mode" which... usually works

Windows still understands near and far pointers.

Microsoft only deprecated features that were so buggy and exploitable so fixing them would be too much work.

<:YEP:808828808127971399>

is a stable C library interface really meaningfully higher level than a stable syscall interface

a bit

a c library interface can do fun userspace side stuff instead of jumping into the kernel

i'd be more open to linux having a stable C interface instead of a stable syscall interface if the C library in question were easier to compile backwards-compatible binaries for

like rn to make broadly compatible binaries for linux you need to either jump through hoops to link against a really old glibc, or make a static binary

true

?

This music makes me feel nostalgic.

You're not supposed to interface directly with the kernel. Everything is done through windows.h and the kernel interface is unofficial and behind the scenes and private

That's how it is on every single other OS except Linux because Linus literally just wanted to avoid the responsibility of providing a slightly more complete and more functional system

Having bionic/musl/glibc is not a strength of Linux

It is a contra

There is no advantage in not having a single included libc everyone can expect to be present and take for granted without worrying about what version

And relying on GNU as the de-facto provider for everything in the early days was simply done because the legality of BSD was in question during a lawsuit at the time and there was no other alternative available at the time. GNU and Linux were just in the right place at the right time unfortunately

That's why we ended up with this awkward poorly thought out patchwork OS gaining lots of popularity in a very short time

And everyone doing their own weird mutually incompatible thing and not caring about everybody else

the issue with this sentiment is the idea that any single "libc" can be good, it can't each distro that ships their own libc surprise surprise, they all wind up having issues. instead linux has excellent support for many different libc implementations, allowing you to use whatever libc may be required

Literally no other OS does this. It's not because Linux somehow knows better than literally every other OS project. It's because Linus literally didn't feel comfortable with the responsibility of providing the bare minimum userspace libraries for his kernel.

big disagree, it saves a lot of headaches, if this really was a massive issue, linux wouldn't dominate in nearly every single market segment

There's no advantage. The only reason Bionic exists is because Google's corporate policy said to avoid anything GPL and there weren't any non-GPL alternatives that were good enough, so they just forked BSD libc.

musl exists because glibc is a dumpster fire, and I think it was also partially forked from BSD too.

all musl, bionic, and glibc have their own respective issues, and the flexibility of swapping between them is really nice

also simply not using a libc is really nice too

glibc was the de-facto standard and Linus just relied on it because it was just there at the time and convenient since it avoids work and responsibility

https://youtu.be/8czf2lc1nXI

maybe that is the case, but it has reaped many benefits

It literally hasn't reaped anything except misery. Compatibility problems and political problems because the GNU and kernel guys try to fight each other.

wat

There was a time when libc actually was included with Linux in the past, I think it was called klibc

But like I said the only reason this isn't done is because Linus really really wants to avoid it for irrational reasons.

Essentially afraid of responsibility

if you say so lmao

You know, even if libc is included with Linux, it can still be customized during build time like the rest of Linux? And still replaced completely if desired, since it's open source?

BSD has everything included but nothing stops people from writing or easily installing more sophisticated replacements for the built in components.

There is literally, only a huge advantage to be had, in providing a basic bare minimum for people to rely on and take for granted

Nothing forces you to use 100% of the included features either. You can exclude major parts of the Linux kernel when compiling it if you don't need all its features. And if you don't need all of the tools BSD comes with, you can make a stripped down or customized OS image for your hardware and simply not compile or include things that aren't going to be used or needed.

People have just gotten accustomed to the mediocrity of Linux that people will make excuses to defend it without considering if it's truly an advantage or a liability.

When comparing it to everything else, it just seems really half assed

It's shocking that it got that popular but it was at the right place at the right time when people wanted a free system.

Linus is also afraid to make any fundamental improvements to OS and kernel design or to learn any of the lessons from the book he read about OS design. And the stable low-level kernel interface probably contributes to the stagnation of Linux.

Linus's professor essentially said he was pretty disappointed in his student.

At least in how little he seemed to take from his book.

and yet, the OS he develops is still the king

what are you even talking about? klibc's last release was last year.

Its goal was to be a very tiny and incomplete libc to be used during early boot so that userspace could handle some initialization tasks instead of the kernel. It's still used for that.

I know nowadays there is a kernelspace libc called klibc, but there used to be a userspace libc bundled with Linux at some point too when they thought they needed to move away from their dependency on glibc because of irreconcilable disagreements with the stubborn project management, which mostly consists of Red Hat employees.

wat

you don't know what you're talking about

klibc is a regular library like any other libc

Maybe it was called something else and I'm not remembering the name correctly

> Linus's professor essentially said he was pretty disappointed in his student. > At least in how little he seemed to take from his book. Pretty sure Linus wasn't a student of Andrew Tanenbaum.

imagine having the balls to publically state you are dissapointed in the lead maintainer/developer of the most successful kernel on the planet

regardless of who said it

it feels like children trying to punch adults lmao

https://man.archlinux.org/man/libc.7.en

I don't remember if Linus was in a classroom with him or not, but he definitely used his book...

He wasn't disappointed with his success, just disappointed with how uninspired the design of Linux is

"man, he could have done all this funky dumb stuff and made the kernel worse instead of making sensible design decisions that will eventually turn it into a behemoth powerhouse"

It wasn't sensible though

It was a monolithic design

No separation of code or anything

It's like someone just did the bare minimum without thinking about how it should be designed to be elegant or secure or modular or easy to maintain and expand, or anything.

It brought absolutely nothing to the table other than barely being able to run

sure, thats why people kept adopting it, because it was a terrible design that they couldnt add anything to

Because there was nothing else at the time lol.

and?

Every other OS has some sort of purpose in the design

if linux was really that bad, people would have invested in an alternative, they didn't because linux was not, and is not that bad

The kernel brings some sort of innovation or unique feature

Linux had no features at all other than being able to barely boot

No self healing anything

sounds like major copeage to me

No separation of anything

No lessons learned from the past at all

That's why Linux is still dealing with embarrassing fragility problems whenever a driver crashes or the system runs out of RAM

Linux is only chosen if compatibility with existing Linux software ecosystem is a factor driving the decision.

[av1_kekw](https://cdn.discordapp.com/emojis/758892021191934033.webp?size=48&quality=lossless&name=av1_kekw)

funny

BSD was a very popular system at the time and it was a lot more functional and complete than Linux at the time but there was an ongoing lawsuit and people were not sure if it was legal to use BSD.

Basically the only reason Linux is successful is because of that lawsuit and Linux and GNU both being available at the right time.

Even today, if you compare a Linux system to a BSD system, you will see how... strangely sloppy and half-assed Linux seems by comparison.

Then again MacOS also feels kinda half-ass cobbled together like Linux

Like a deranged cousin of FreeBSD

And FreeBSD is already pretty deranged

BSDs would be a lot more popular if this was actually the case

Look I'm an equal opportunity hater. I just try to keep it real and if something sucks I admit to it with no excuses

I love using Linux but it's sloppy and half-assed and I wish it wasn't

I would still much rather use it than Windows

If people stopped making excuses for genuine deficiencies of Linux and stopped saying "yeah but it's soooo successful so it must be good" then maybe they'd actually improve it so it could actually get more successful.

It looks pretty stagnant to me, considering how many billions of dollars of man hours are being spent on it and how little actually improves or changes in return

<:monkaMega:809252622900789269> long text

Sorry 😅

I highly disagree that a lot of what is brought up is an actual deficiency. it's not like there are none, there are loads, however I highly disagree that libc is one of them

Not just libc, just it's incompleteness and not bundling anything and relying on other people to take responsibility because you're afraid of providing basic foundational bedrock

That's a real problem and I think a lot of people are actually in denial about this. There is a good reason why literally no one else does this outside of Linux

webp good

it's not denial, linux is designed in a way for flexibility, if linux "did everything the way other operating systems do" linux would just be another piece of trash

That's their cope

this "foundational bedrock" thats missing allows "many foundational bedrocks"

No it doesn't.

it's a core part of the reason *why* linux has been so succsessful

Providing basic stuff doesn't prevent people from installing a replacement if and when it's needed.

tfw I don't agree with anyone in this conversation

progressive line—by-line display during loading and the decoding speed make webp the only format that is better than jpeg but not more complicated than jpeg. it seems to me that webp is what jpeg should have been from its very beginning. For Internet

disagree, it ads in inherent bias to development that doesnt exist in the linux kernel

Yes it does exist

while I continue to use jpeg to transfer files in archives. for the reason that yuv444 and q98 are better than webp. but png file size too much, too much

It already is biased toward glibc

[av1_whatisthis](https://cdn.discordapp.com/emojis/939666933119324222.webp?size=48&quality=lossless&name=av1_whatisthis)

The linux kernel literally includes glibc-specific documentation

And there's already a huge existing bias that Linux = glibc

Just ask Lennart

don't forget JPEG has "progressive refinement decoding"

no one actually has this bias, if you are aware of what glibc actually is, you know musl is also a fairly decent sized player and you may even also have a good idea that bionic is also massive, if you don't know what glibc is, then well, you don't know what it is

the inherent bias already exists, and worse, it exists for the most overcomplicated and highest-resource-requirement libc. It would be much better if the bias was for something much more simple and basic and foundational that was included with Linux, rather than the current state of affairs.

this is the most useless advantage in 2024

It's much better than line-by-line decoding

It's not useless, it makes images load visibly faster by a factor of ten

I would always used it. but this slows down the decoding speed extremely much. I just can't use it on images like 14k x 28k.

let's move the OS talk <#806898911091753051>

Tell that to Lennart. He says Linux = glibc and if you use musl then you shouldn't expect to be able to run systemd or any software that depends on it.

lennart is an idiot

and systemd is trash, this is nothing ground breaking

<:JPEG_XL:805860709039865937> jpeg xl just... it just can't be used at all. I am not writing on this server because in the last six months I have never encoded jxl and I used it only for lossless compression in my personal archives

the big problem is that the AI processing of Cunet and Cugan neural networks makes an almost perfect image from Art in Jpeg without problems

I vaguely recall Linus saying something along the lines that the fallacy of microkernels is the belief that complexity lies in individual modules, when in fact it's between them

iirc, this is how he justified the monolithic design

that you end up with less total complexity

He'd be right on fully memory safe hardware

But as it is right now, the kernel can't be protected from faults in individual parts, so a microkernel would be better.

microkernels aren't a fallacy, it's the standard these days for all modern software projects. And it's just a basic principle of software design to not write a monolith.

In the future with CHERI we can have fully independent modules sharing an address space allowing for all the advantages of monolithic kernels and microkernels to be mixed.

Yeah, microkernels have been hugely successful everywhere except big computers.

Code should be resilient and modular and faults in one area should not cause everything else in other areas to fail

MacOS was based on a microkernel (Mach) a long time ago, but they made it more monolithic. But they've spent the last 10 years making it more microkernel-y again.

that's not even specific to kernels

Even filesystems are in userspace on iPhone now

there's nothing radical or crazy about applying such obvious consideration to kernel design

that's why every single kernel written these days is a microkernel afaik

cuz it's just obvious to do things that way

only reason linux doesn't is laziness and people making excuses instead of making it better

Re-architecting Linux would be in the 10s of millions dollars minimum

So it not being done isn't laziness

I kinda doubt it.

Most of linux is just drivers.

like probably 90%

Fixing up all the drivers whenever someone makes a small change to the internal interfaces is already a lot of work

Yeah, too bad there isn't a more abstract, higher level interface drivers can use

The fact that Linux has so many drivers is what would make re-architecting so expensive.

Easier to just make a new microkernel and run Linux in VMs to use as driver servers.

That's part of the whole flaw with Linux

Since Linux never bothered actually designing it well, it increases the maintenance burden. If it was a microkernel with slightly more abstract interfaces, it would reduce the breakage and maintenance burden.

Linux consumes a huge amount of man hours of labor but it results in very little new features and improvements and it still takes years for major issues to get addressed if ever

It's almost like it's stagnant by design

Other OS projects innovate and come up with new ideas and build upon successful existing ideas and Linux just stays stagnant

And occasionally something cool from another OS gets ported over to Linux. Like pledge or sudo.

But how long did it take to even get something as basic and essential as arc4random/getentropy?

not really relevant to my point. Re-architecting would always require large changes to drivers. Like QNX and Mach are *super* different, even though they're both microkernels.

sudo is an independent software project that is unaffiliated with any specific OS. Pledge for Linux is a userspace library that uses Linux's pre-existing SECCOMP system. io_uring was an innovation in Linux that was later copied by Windows :)

Wait isn't sudo mostly off kernel? There are distros that don't have any sudo support

Oh you beat me to it

also I just saw that the pledge port to Linux was done by Justine Tunney, an actual honest to goodness fascist 🤢

???

idk who this person is but I don't see any fascist advocacy...

She hasn't done public advocacy for around 10 years, but she also has never apologized to any of the people she hurt, so I don't reckon her views have changed.

Fascism is just when government and corporations team up to screw everyone over and everyone is hypnotized into worshiping the contrived authority of these arbitrary and self-serving institutions, so in my opinion the majority of the human race seems to support fascism since everyone keeps saying it's a necessary evil or something

Or that I supposedly signed a social contract before I was born and had the ability to consent to one

And that if I ever benefit from it in any way then I'm not allowed to be against people committing crimes in my name and cloaking themselves in the supposed consent of the people

But that's just my hot take.

Lonnie, apparently I was thinking of "Linux libc" not klibc. It's also known as libc5. It was a fork of glibc that was used for a long time as the standard libc on Linux

Aside from that I'm also aware of the eglibc fork

which was a similar situation where relying on the glibc maintainers was inadequate

found the quote from Torvalds

> The theory behind the microkernel is that operating systems are complicated. So you try to get some of the complexity out by modularizing it a lot. The tenet of the microkernel approach is that the kernel, which is the core of the core of the core, should do as little as possible. Its main function is to communicate. All the different things that the computer offers are services that are available through the microkernel communications channels. In the microkernel approach, you’re supposed to split up the problem space so much that none of it is complex. > > I thought this was stupid. Yes, it makes every single piece simple. But the interactions make it far more complex than it would be if many of the services were included in the kernel itself, as they are in Linux. Think of your brain. Every single piece is simple, but the interactions between the pieces make for a highly complex system. It’s the whole-is-bigger-than-the-parts problem. If you take a problem and split it in half and say that the halves are half as complicated, you’re ignoring the fact that you have to add in the complication of communication between the two halves. The theory behind the microkernel was that you split the kernel into fifty independent parts, and each of the parts is a fiftieth of the complexity. But then everybody ignores the fact that the communication among the parts is actually more complicated than the original system was—never mind the fact that the parts are still not trivial. > > That’s the biggest argument against microkernels. The simplicity you try to reach is a false simplicity.

(2001)

good example of linus being clueless about things outside his own work

though to be fair, many of the academics working on microkernels in the 90s weren't exactly competent either