Libvips: Vips CLI specify file format with pipes

Hi Felix, this has been talked about a bit, there's even an experimental branch with support for streams:

https://github.com/jcupitt/libvips/tree/load-from-stream

It can run this C program:

https://gist.github.com/jcupitt/ff21f5b8d1dd6ba54c6f

It reads a series of images from a socket, processes them, and sends them off to another socket, all without having to load the whole image. The aim was to support AWS buckets, but it would work for command-line pipes too, with a small amount of extra work.

There was a long chat about it on this issue:

https://github.com/lovell/sharp/issues/30

I gave up for a bit, then it bitrotted, then I tried to clean it up ready to merge to master, and hit some problems. It needs a bit more work still :/

I wasn't talking about streaming multiple images, although that would be a cool feature.

The main idea is to be able to set the loader or saver (or roughly the file type), when it can not be inferred from the file extension. I'm sure there are more issues for filetypes that require seeking, but eg. jpeg, png, ppm would work fine. This would also be helpful, when there are multiple implementations that can emit or load the same file type. For example PDF could be loaded with either pdfload or magickload.

I think you'd need the streaming stuff for /dev/stdin to work. For example, this fails:

$ cat k2.jpg | vips jpegload /dev/stdin x.jpg
VipsJpeg: Not a JPEG file: starts with 0x87 0xe5
...

The file loaders expect to be able to open their input several times. The streaming stuff would add a read buffer that would make this possible -- the many images from a stream thing is just a bonus.

This is added in 8.9. You can write eg.

aws s3 cp s3://mybucket/input.jpg - | \
  vips thumbnail_source [descriptor=0] .jpg 128 | \
    aws s3 cp - s3://mybucket/output.jpg

The release notes have more information:

https://libvips.github.io/libvips/2019/12/11/What's-new-in-8.9.html