An alternative to gdk-pixbuf

classic Classic list List threaded Threaded
39 messages Options
12
Reply | Threaded
Open this post in threaded view
|

An alternative to gdk-pixbuf

Magnus Bergman-3
Over the years it has been discussed from time to time to replace
gdk-pixbuf with something else[1][2]. Something was even in the making
(I guess over ten years ago) but it never replaced gdk-pixbuf
apparently. Now I don't even remember what it was called. And something
else called pig was proposed more recently.

One major reason to replace gdk-pixbuf has been the long term goal to
phase out GDK in favour of cairo. And some people (including me) has
been displeased with the gdk-pixbuf API for more subtle reasons. It has
also been brough up that it lacks some features (which I guess are hard
to implement as an afterthought). I finally took some time to design an
image loading library on top of cairo called abydos, which at least
suits my needs. And also some needs mentioned in this list over the
years. First I thought it could suit the needs of GTK+ as well. But I
just learned that even cairo is now considered "legacy technology" by
the GTK+ developers[3], so I guess cairo is also being phased out now.
But in favour of what? Maybe the basic design of my API could still be
of some interest. And perhaps I could even port my code over to whatever
is the next hot drawing library.

It's mainly a proof of concept right now. But it works well enough to be
tested. It doesn't have many loaders, but I assume it will be a trivial
task to port them over from gdk-pixbuf since the plugin API:s are quite
similar. The API for handling variants (see below) is work in
progress and not even implemented yet. And I plan to add support for
stereoscopic images as well.


The main features of abydos

* It's easy to use, especially for the common use cases.

* Depends only on cairo. (The main library that is, loaders may of
  course have other dependancies.)

* Selects what loader to use based on MIME types. (But does nothing to
  determine the MIME types by itself.)

* Full support for vector graphics. (Scaling and other transformations
  can be done using cairo.)

* Supports animations, both frame based and continous. (But does
  nothing to measure clock time or schedule events by itself.) Where is
  also no intention to support proper video (streaming) formats.

* Supports layers. As in toggling layers on and off. Just delivering
  each layer as a separate image might not be feasible since a layer
  can be more like a filter operation than a pixmap.

* Supports multi page images. This means image files that contains more
  than one image. Which for example TIFF files can.

* Supports multi variant images. This is somewhat like multi page images
  except that it is known that the images are all (differently sized)
  variants of the same image. It could be embedded thumbnails (which
  could be intelligently cropped and not just scaled down versions of
  the full size image). Or it could be icons with different details
  level for different sizes. Or it could be mipmap textures.

* Deals with non square pixels. By default it scales images to the
  correct aspect ratio, but can optionally deliver them pixel by pixel
  as well.


What it lacks in comparison to gdk-pixbuf

* It (currently) lacks support for saving images. Of course images can
  be saved to PDF, PNG and SVG using cairo. I would like to add for
  some more general purpose animation format to complement those (MNG
  or APNG probably). As far as I know the image saving functionality of
  gdk-pixbuf isn't heavily used. At least not for something other than
  PNG.

* It lacks support for indexed colors, since cairo lacks that support.
  (So this is more of a question about what cairo lacks in comparison
  to GDK.)

* It lacks support for determining the MIME type of files. This is a
  hard problem to solve and I think it's better solved elsewhere. Using
  the shared MIME database for sniffing goes a long way, and is surely
  enough for the needs of GTK+. But not all types can be detected this
  way. Using libextractor (with special plugins) goes a bit further.

* It doesn't use GObject. But adding GObject bindings would be possible
  I guess.


Some examples

This is how to convert an SVG file to a PNG file.

    cairo_surface_t *surface = abydos_load("image/svg+xml",
    "example.svg"); cairo_surface_write_to_png(surface, "example.png");
    cairo_surface_destroy(surface);

This is how to convert the second frame of a GIF animation to PNG.

   abydos_t *ar = abydos_create_from_file("image/gif", "example.gif");
   abydos_set_frame(ar, 1);
   cairo_surface_t *surface = abydos_get_image_surface(ar, 0);
   abydos_destroy(ar);
   cairo_surface_write_to_png(surface, "example.png");
   cairo_surface_destroy(surface);

This is how to display a GIF animation to something cairo can draw on.
Of course some kind of event scheduling should be used insted of a
delay in the real world.

    abydos_t *ar = abydos_create_from_file("image/gif", "example.gif");
    double last_timestamp = now();
    for(;;) {
        abydos_render(ar, cr, 0);
        delay(abydos_get_delay(ar));
        double timestamp = now()
        abydos_add_time(ar,timestamp - last_timestamp);
        last_timestamp = timestamp;
    }


Less is more from a security perspective

It has often been noted that support for more than a core set of image
formats is a severe security risk. One conclusion from this is that the
library GTK+ uses for loading images should only support a core set of
image formats. It has been said that if a dedicated imageviewer wants
to support more formats (which they should think twice about since it's
also a security risk) they should have their own plugin system. But
image viewers generally don't do this. And it would also be a bit
impractical to maintain image loading plugins for severeral image
loaders. I've been in contact with a two creators of image viewers and
they found it unnecessary to have their own plugin system since
gdk-pixbuf already has one. And I think it's a valid point. This
argument could however be nullified by simply disallowing third party
plugins.

What it all comes down to is that using a bigger variety of software
creates a bigger attack surface. Viewing a lot of images is less secure
than never viewing any images at all. But viewing images with a larger
variety of loaders (for the same formats) could also be considered less
secure than using only one loader. So forcing every image viewer to
have their own loaders is also not optimal from a security perspective.
Sharing the same loaders could also be a good thing if get gets more
attention and care. (This is also a good argument against writing yet
another image loading library on the other hand.)

I think a reasonable compromize would be to block all but a core set of
formats (MIME types) in specially vulnerable situations (like for auto
displaying images from internet). And for extra security all but a core
set of loaders could be removed. But this should be done on a distro
level, not enforced by developers, I think.


For more information about abydos, including API documentation, see:
http://snisurset.net/code/abydos/

[1]
https://mail.gnome.org/archives/gtk-devel-list/2007-October/msg00034.html
[2]
https://mail.gnome.org/archives/gtk-devel-list/2014-October/msg00017.html
[3]
https://gitlab.gnome.org/GNOME/gdk-pixbuf/issues/13
_______________________________________________
gtk-devel-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: An alternative to gdk-pixbuf

Gtk+ - Dev - General mailing list
Hi;

On Tue, 4 Sep 2018 at 23:19, Magnus Bergman <[hidden email]> wrote:
Over the years it has been discussed from time to time to replace
gdk-pixbuf with something else[1][2]. Something was even in the making
(I guess over ten years ago) but it never replaced gdk-pixbuf
apparently. Now I don't even remember what it was called. And something
else called pig was proposed more recently.

One major reason to replace gdk-pixbuf has been the long term goal to
phase out GDK in favour of cairo. And some people (including me) has
been displeased with the gdk-pixbuf API for more subtle reasons. It has
also been brough up that it lacks some features (which I guess are hard
to implement as an afterthought). I finally took some time to design an
image loading library on top of cairo called abydos, which at least
suits my needs. And also some needs mentioned in this list over the
years. First I thought it could suit the needs of GTK+ as well. But I
just learned that even cairo is now considered "legacy technology" by
the GTK+ developers[3], so I guess cairo is also being phased out now.
But in favour of what?

We're phasing out Cairo in favour of the CSS rendering model, implemented on top of OpenGL and Vulkan, as it's the API that most closely matches the requirements of GTK.

Cairo is still used as a fallback mechanism — both when running on platforms without support for OpenGL or Vulkan, and in the interim period while the GL/Vulkan rendering pipelines inside GTK don't support a CSS feature. Additionally, you may still use Cairo for 2D high quality rendering, for printing and for custom drawing.

Internally, GTK only cares about GdkPixbuf in as much as it provides us with a way to load graphic assets like icons, which are typically in a very limited amount of formats. As far as we're concerned, image data coming from GdkPixbuf and Cairo gets loaded into memory buffers that get submitted to the GPU almost immediately, and all transformations and rendering happen using shaders, on the GPU.

Anything bigger than an icon should probably be loaded and displayed through Gegl, the same library used by the GIMP; this is especially true if you're planning to process the image using filters. Images, these days, are pretty big buffers — and so are displays; using a simple linear buffer like GdkPixbuf's does not scale.

From the perspective of a consumer of GdkPixbuf, the only thing that an image loading library should do is, pretty literally, load images. No scaling, no compositing, no rendering. Saving to a file may be interesting — but that opens the whole transcoding can of worms, so maybe it should just be a debugging feature. Ideally, I/O operations should happen in a separate thread, and possible use multi-threading to chunk out the work on multiple cores; it definitely needs to integrate with GIO, as it's a modern API that well maps to GUIs.

In the near future, I'll very likely deprecate most of GdkPixbuf's API, except for the I/O operations; I'd also be happy to seal off most of its internals, within the ABI stability promise, to avoid leakage of internal state.

Ciao,
 Emmanuele.

--

_______________________________________________
gtk-devel-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: An alternative to gdk-pixbuf

Magnus Bergman-3
On Wed, 5 Sep 2018 17:28:22 +0100
Emmanuele Bassi <[hidden email]> wrote:

> Hi;
>
> On Tue, 4 Sep 2018 at 23:19, Magnus Bergman
> <[hidden email]> wrote:
>
> > Over the years it has been discussed from time to time to replace
> > gdk-pixbuf with something else[1][2]. Something was even in the
> > making (I guess over ten years ago) but it never replaced gdk-pixbuf
> > apparently. Now I don't even remember what it was called. And
> > something else called pig was proposed more recently.
> >
> > One major reason to replace gdk-pixbuf has been the long term goal
> > to phase out GDK in favour of cairo. And some people (including me)
> > has been displeased with the gdk-pixbuf API for more subtle
> > reasons. It has also been brough up that it lacks some features
> > (which I guess are hard to implement as an afterthought). I finally
> > took some time to design an image loading library on top of cairo
> > called abydos, which at least suits my needs. And also some needs
> > mentioned in this list over the years. First I thought it could
> > suit the needs of GTK+ as well. But I just learned that even cairo
> > is now considered "legacy technology" by the GTK+ developers[3], so
> > I guess cairo is also being phased out now. But in favour of what?  
>
>
> We're phasing out Cairo in favour of the CSS rendering model,
> implemented on top of OpenGL and Vulkan, as it's the API that most
> closely matches the requirements of GTK.

I'm not sure I quite understand what you are saying. What does this
mean for image loading if terms of actual implementation? What would
ideally happen then GTK+ needs to load an image (because the
application called gtk_image_new_from_file() for example)?

Will GTK+ have a limited set of hard coded loaders which can render
images directly using both OpenGL and Vulcan? So one custom SVG loader
for OpenGL and one custom SVG loader for Vulcan. And likewise for every
other supported image format (I guess at least PNG, JPEG, GIF, WEBP and
Windows Icon).


> Cairo is still used as a fallback mechanism — both when running on
> platforms without support for OpenGL or Vulkan, and in the interim
> period while the GL/Vulkan rendering pipelines inside GTK don't
> support a CSS feature. Additionally, you may still use Cairo for 2D
> high quality rendering, for printing and for custom drawing.

But then it comes to printing and custom drawing I'm on my own then it
comes to loading the images? Which is okey of course since gdk-pixbuf
is kind of a separate library already. But isn't it a good thing to
share common image loading code?


> Internally, GTK only cares about GdkPixbuf in as much as it provides
> us with a way to load graphic assets like icons, which are typically
> in a very limited amount of formats. As far as we're concerned, image
> data coming from GdkPixbuf and Cairo gets loaded into memory buffers
> that get submitted to the GPU almost immediately, and all
> transformations and rendering happen using shaders, on the GPU.
>
> Anything bigger than an icon should probably be loaded and displayed
> through Gegl, the same library used by the GIMP; this is especially
> true if you're planning to process the image using filters. Images,
> these days, are pretty big buffers — and so are displays; using a
> simple linear buffer like GdkPixbuf's does not scale.

I'm not convinced that you should need to use one image loader for
loading small images and another for loading full scale images (such as
ordinary photos). Then it comes to really large images (many times the
RAM size) I can agree that something else might be needed.

Gegl is great for image editing. But not as much for simple viewing. It
doesn't do animation and doesn't support vector graphics (apart from
converting them into pixmaps). Also it only loads images from the
file system and is a bit bulky to use. It simply doesn't fit the needs
for a simple image viewer.


> From the perspective of a consumer of GdkPixbuf, the only thing that
> an image loading library should do is, pretty literally, load images.
> No scaling, no compositing, no rendering. Saving to a file may be
> interesting — but that opens the whole transcoding can of worms, so
> maybe it should just be a debugging feature. Ideally, I/O operations
> should happen in a separate thread, and possible use multi-threading
> to chunk out the work on multiple cores; it definitely needs to
> integrate with GIO, as it's a modern API that well maps to GUIs.

Yes, I very much agree with this. Except I think it has to do rendering
of some sort. If an image contains a triangle, it has to be converted to
a function call that renders it (using OpenGL/Vulcan/cairo) somehow.
Just handing over a set of geometrical objects hasn't taken you very far
from the original coded data, basically just converted it into another
vector data format. Spreading the work over multiple threads I mostly
consider an implementation detail. Or do you think it needs to be
considered then designing an API for image loading? And if you have
looked at the API of abydos, do you think it requires some changes to
fit the needs you mention?


> In the near future, I'll very likely deprecate most of GdkPixbuf's
> API, except for the I/O operations; I'd also be happy to seal off
> most of its internals, within the ABI stability promise, to avoid
> leakage of internal state.

Will the loader plugin API go away, you think?
_______________________________________________
gtk-devel-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: An alternative to gdk-pixbuf

Gtk+ - Dev - General mailing list
In reply to this post by Magnus Bergman-3
hi,

On Tue, Sep 4, 2018, 6:19 PM Magnus Bergman <[hidden email]> wrote:
Over the years it has been discussed from time to time to replace
gdk-pixbuf with something else[1][2].
[...]
I finally took some time to design an
image loading library on top of cairo
[...]
 abydos, which at least
suits my needs. And also some needs mentioned in this list over the
years. First I thought it could suit the needs of GTK+ as well.
So one thing i think a modern image library needs is sandboxing.  the loader should be contained with bubblewrap and it should just give back a chunk of pixels in shared memory as a memfd or similar.

then if it crashes it doesnt take down the app.  if its exploited the most the attacker can do is make the caller show the wrong image (which could still be bad of course)

--Ray

_______________________________________________
gtk-devel-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: An alternative to gdk-pixbuf

Nicolas Dufresne-4


Le mer. 5 sept. 2018 17:48, Ray Strode via gtk-devel-list <[hidden email]> a écrit :
hi,

On Tue, Sep 4, 2018, 6:19 PM Magnus Bergman <[hidden email]> wrote:
Over the years it has been discussed from time to time to replace
gdk-pixbuf with something else[1][2].
[...]
I finally took some time to design an
image loading library on top of cairo
[...]
 abydos, which at least
suits my needs. And also some needs mentioned in this list over the
years. First I thought it could suit the needs of GTK+ as well.
So one thing i think a modern image library needs is sandboxing.  the loader should be contained with bubblewrap and it should just give back a chunk of pixels in shared memory as a memfd or similar.

then if it crashes it doesnt take down the app.  if its exploited the most the attacker can do is make the caller show the wrong image (which could still be bad of course)

For foreign image, yes, but for system icons, that's just an overhead.



--Ray
_______________________________________________
gtk-devel-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-devel-list

_______________________________________________
gtk-devel-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: An alternative to gdk-pixbuf

Christian Hergert
On 09/05/2018 05:03 PM, Nicolas Dufresne wrote:
>
> For foreign image, yes, but for system icons, that's just an overhead.

System icons should be using mmap'able caches that avoid any runtime
overhead and allows read-only page-sharing between processes.
_______________________________________________
gtk-devel-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: An alternative to gdk-pixbuf

Magnus Bergman-3
In reply to this post by Gtk+ - Dev - General mailing list
On Wed, 5 Sep 2018 17:47:57 -0400
Ray Strode <[hidden email]> wrote:

> hi,
>
> On Tue, Sep 4, 2018, 6:19 PM Magnus Bergman
> <[hidden email]> wrote:
>
> > Over the years it has been discussed from time to time to replace
> > gdk-pixbuf with something else[1][2].  
>
> [...]
>
> > I finally took some time to design an
> > image loading library on top of cairo  
>
> [...]
>
> >  abydos, which at least
> > suits my needs. And also some needs mentioned in this list over the
> > years. First I thought it could suit the needs of GTK+ as well.  
>
> So one thing i think a modern image library needs is sandboxing.  the
> loader should be contained with bubblewrap and it should just give
> back a chunk of pixels in shared memory as a memfd or similar.
>
> then if it crashes it doesnt take down the app.  if its exploited the
> most the attacker can do is make the caller show the wrong image
> (which could still be bad of course)

Yes, I though about that and there is an example implementation in the
netpbm plugin of abydos that takes this approach. It currently doesn't
drop privileges, but that can easily be added. But it only works with
pixmaps. I'm not sure how to deal with vector graphics this way. I will
take a look at bubblewrap and think this over.

But if opening images is considered dangerous, opening files in general
must be considered equally dangerous. And if third party image loaders
are considered dangerous, all third party code must be considered
equally dangerous. I think it would be better with some kind of
framework that can separate applications into run-time modules with
different privileges. So that all kind of data processing happens in
contexts without access to the file system for example. Then the whole
image loader (in this case better described as an image decoder) could
be placed in the same isolated context together with the rest of the
data processing stuff. It would be kind of cool if every application
always did this. But I think this technique is primarily needed in
especially sensitive situations (like handling complex and potentially
malicious data like from the internet), not as much for loading local
images in general.
_______________________________________________
gtk-devel-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: An alternative to gdk-pixbuf

Nicolas Dufresne-4
In reply to this post by Christian Hergert
Le mercredi 05 septembre 2018 à 17:49 -0700, Christian Hergert a
écrit :
> On 09/05/2018 05:03 PM, Nicolas Dufresne wrote:
> >
> > For foreign image, yes, but for system icons, that's just an
> > overhead.
>
> System icons should be using mmap'able caches that avoid any runtime
> overhead and allows read-only page-sharing between processes.

Is there any benchmark that would justify this added complexity ? Also,
there will be more context switch, so cache misses will take more time
then just loading the icon directly.

> _______________________________________________
> gtk-devel-list mailing list
> [hidden email]
> https://mail.gnome.org/mailman/listinfo/gtk-devel-list

_______________________________________________
gtk-devel-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: An alternative to gdk-pixbuf

Christian Hergert
On 09/05/2018 06:18 PM, Nicolas Dufresne wrote:
> Is there any benchmark that would justify this added complexity ? Also,
> there will be more context switch, so cache misses will take more time
> then just loading the icon directly.

Just because you've decoded into a non-shareable page of memory doesn't
mean it wont swap and be trivialized into the same as the cached case
but with everything else worse.

Also, read-only caches can have the benefit of shoving the index table
into the head of the file and embedding it in the inode ensuring that
even your lookup table is fast and share-able.

_______________________________________________
gtk-devel-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: An alternative to gdk-pixbuf

Nicolas Dufresne-4
Le mercredi 05 septembre 2018 à 18:43 -0700, Christian Hergert a écrit :

> On 09/05/2018 06:18 PM, Nicolas Dufresne wrote:
> > Is there any benchmark that would justify this added complexity ? Also,
> > there will be more context switch, so cache misses will take more time
> > then just loading the icon directly.
>
> Just because you've decoded into a non-shareable page of memory doesn't
> mean it wont swap and be trivialized into the same as the cached case
> but with everything else worse.
>
> Also, read-only caches can have the benefit of shoving the index table
> into the head of the file and embedding it in the inode ensuring that
> even your lookup table is fast and share-able.

I've replied to a comment about sandboxing image loading to prevent
possible crash buffer overflow from happening. You are now focusing on
an optimization. I believe you should start a new thread.

Nicolas

_______________________________________________
gtk-devel-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: An alternative to gdk-pixbuf

Christian Hergert
On 09/05/2018 06:57 PM, Nicolas Dufresne wrote:
> I've replied to a comment about sandboxing image loading to prevent
> possible crash buffer overflow from happening. You are now focusing on
> an optimization. I believe you should start a new thread.

It's relevant because it is the same as the bubblewrap case. You pass an
FD to the mmap'able file across the IPC boundary.

_______________________________________________
gtk-devel-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: An alternative to gdk-pixbuf

Gtk+ - Dev - General mailing list
In reply to this post by Magnus Bergman-3
On Wed, 5 Sep 2018 at 19:25, Magnus Bergman <[hidden email]> wrote:
On Wed, 5 Sep 2018 17:28:22 +0100
Emmanuele Bassi <[hidden email]> wrote:

> We're phasing out Cairo in favour of the CSS rendering model,
> implemented on top of OpenGL and Vulkan, as it's the API that most
> closely matches the requirements of GTK.

I'm not sure I quite understand what you are saying. What does this
mean for image loading if terms of actual implementation? What would
ideally happen then GTK+ needs to load an image (because the
application called gtk_image_new_from_file() for example)?


We're still using GdkPixbuf, and for the 4.0 API series we still have GdkPixbuf types exposed in the GTK API.

For future API series (5.x, 6.x, …) we may revisit this, with the option of moving icon loading into GTK itself.
 
> Cairo is still used as a fallback mechanism — both when running on
> platforms without support for OpenGL or Vulkan, and in the interim
> period while the GL/Vulkan rendering pipelines inside GTK don't
> support a CSS feature. Additionally, you may still use Cairo for 2D
> high quality rendering, for printing and for custom drawing.

But then it comes to printing and custom drawing I'm on my own then it
comes to loading the images? Which is okey of course since gdk-pixbuf
is kind of a separate library already. But isn't it a good thing to
share common image loading code?

Not really; icons—especially ones that are identified via icon theme names—are a very narrow subset of "all the possible images and formats in human history".

Gegl is great for image editing. But not as much for simple viewing.

This is debatable. If I'm viewing a 4000x4000 RGB image on a hidpi display I'm already pushing gdk-pixbuf and cairo to their limits because of the scaling factor applied to the window — not only the buffer gets loaded uncompressed to allow for zooming, but the image viewer needs to render a CPU-scaled down copy of the image.

Sure, for viewing a 500x400px image macro for a meme we're fine; but we're fine with gdk-pixbuf as well, so there's really no need to change to a different image loading library.
 
It doesn't do animation

Animated formats are a dying breed, and they have all but killed by VP9 and new, web-friendly video formats. Social platforms will take a GIF and turn it into a video transparently.

Additionally, GTK 4.x already has new API to render videos and other frame-based media sources through GStreamer.

> From the perspective of a consumer of GdkPixbuf, the only thing that
> an image loading library should do is, pretty literally, load images.
> No scaling, no compositing, no rendering. Saving to a file may be
> interesting — but that opens the whole transcoding can of worms, so
> maybe it should just be a debugging feature. Ideally, I/O operations
> should happen in a separate thread, and possible use multi-threading
> to chunk out the work on multiple cores; it definitely needs to
> integrate with GIO, as it's a modern API that well maps to GUIs.

Yes, I very much agree with this. Except I think it has to do rendering
of some sort. If an image contains a triangle, it has to be converted to
a function call that renders it (using OpenGL/Vulcan/cairo) somehow.

That's up to the toolkit, as it's going to be the one deciding how to render the rest of the UI. There's no way for you to know, from the outside, how to efficiently render anything.

That's why it's much, much better to get a memory buffer with the contents of the image, and let the toolkit pass it to the GPU.

Vector formats are different, but that's why we have API like Cairo or Skia; I'm also waiting for web browsers to implement SVG rendering on the GPU where possible, with the help of new primitives from the GL/Vulkan implementation.

> In the near future, I'll very likely deprecate most of GdkPixbuf's
> API, except for the I/O operations; I'd also be happy to seal off
> most of its internals, within the ABI stability promise, to avoid
> leakage of internal state.

Will the loader plugin API go away, you think?

No API will ever go away: there are no plans for a gdk-pixbuf-3.0. The deprecations would mostly apply to API that is either long since been replaced by Cairo (scale/composite), or that is weirdly ad hoc, like saturate_and_pixelate(). Ideally, I'd like to deprecate the option to build gdk-pixbuf without depending on GIO to do MIME type sniffing, as GIO has been fixed to work on Windows and macOS, and it is a required dependency anyway. The animation API is pretty much a garbage fire, so it may go on the chopping block as well. Of course, deprecated API will keep working as well—or as badly—as it works today, so people can still use it. Moving pixbuf loaders to separate processes, and wrap them in sandboxes, would be a fairly good thing to do; it need to be decided at run time, though, because there are many users of GdkPixbuf that already run in a sandbox, which prevents creating smaller sandboxes inside it.

Ciao,
 Emmanuele.

--

_______________________________________________
gtk-devel-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: An alternative to gdk-pixbuf

Gtk+ - Dev - General mailing list
On Thu, 6 Sep 2018 at 11:40, Emmanuele Bassi via gtk-devel-list
<[hidden email]> wrote:
> On Wed, 5 Sep 2018 at 19:25, Magnus Bergman <[hidden email]> wrote:
>> Gegl is great for image editing. But not as much for simple viewing.
>
> This is debatable. If I'm viewing a 4000x4000 RGB image on a hidpi display I'm already pushing gdk-pixbuf and cairo to their limits because of the scaling factor applied to the window — not only the buffer gets loaded uncompressed to allow for zooming, but the image viewer needs to render a CPU-scaled down copy of the image.

It doesn't have to be gegl, of course, you could use any image
processing library to load and scale down the images. I made a tiny
image viewer based on libvips:

https://github.com/jcupitt/vipsdisp-tiny

It's 300 lines, but does multi-threaded, asynchronous painting of many
image formats. It can display gigapixel images on very modest
hardware, and it should be quick.

There's a more complete image viewer here:

https://github.com/jcupitt/vipsdisp

That adds most of the usual navigation stuff, though it's not quite
done, I must get back to it. The image display part is a widget you
could easily cut out and paste into other applications. It's in C, but
libvips is a GObject-based library, so it'd be easy to write in any
language.

Here it is displaying a 18k x 8k 16-bit TIFF:

http://www.rollthepotato.net/~john/summer.png

That image takes about 8s to load on this laptop. Here it is
displaying a 120k x 100k pixel ndpi slide image:

http://www.rollthepotato.net/~john/slide.png

That image loads instantly, since ndpi supports random access and
vipsdisp can just load the parts it needs to paint the screen.

John
_______________________________________________
gtk-devel-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: An alternative to gdk-pixbuf

Federico Mena Quintero-4
In reply to this post by Gtk+ - Dev - General mailing list
On Wed, 2018-09-05 at 17:28 +0100, Emmanuele Bassi via gtk-devel-list
wrote:

> In the near future, I'll very likely deprecate most of GdkPixbuf's
> API, except for the I/O operations; I'd also be happy to seal off
> most of its internals, within the ABI stability promise, to avoid
> leakage of internal state.

Related to this - I just wrote a braindump on gdk-pixbuf issues and
historical baggage.  I hope it's useful for people trying to clean up
this swamp:

https://people.gnome.org/~federico/blog/my-gdk-pixbuf-braindump.html

Magnus - thanks for showing us Abydos.  The parts of the API that
intersect with gdk-pixbuf's look okay.  I think you may find this
useful:

https://developer.gnome.org/programming-guidelines/stable/index.html.en

I think your API could improve with some of the material there around
memory management and error reporting.  Internally, the code needs
checks in all the malloc()-like functions; some of the arithmetic
(especially in assertions) needs to avoid overflows.  I couldn't find
any tests.  The SVG plug-in needs to handle error results from librsvg.
The Magick plug-in for PNG and JPEG has no error handling; also you may
want to call cairo_surface_mark_dirty() when you are done writing
pixels to it.  The netpbm plugin does a bunch of system calls and
doesn't check for errors.

The parts of your API that deal with pagination / variants / animation
frames - I'm not sure if a gdk-pixbuf replacement would need them, but
I'm coming from the viewpoint of my braindump above.

  Federico

_______________________________________________
gtk-devel-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: An alternative to gdk-pixbuf

Magnus Bergman-3
In reply to this post by Gtk+ - Dev - General mailing list
On Thu, 6 Sep 2018 11:39:59 +0100
Emmanuele Bassi <[hidden email]> wrote:

> On Wed, 5 Sep 2018 at 19:25, Magnus Bergman
> <[hidden email]> wrote:
>
> > On Wed, 5 Sep 2018 17:28:22 +0100
> > Emmanuele Bassi <[hidden email]> wrote:
> >  
> > > We're phasing out Cairo in favour of the CSS rendering model,
> > > implemented on top of OpenGL and Vulkan, as it's the API that most
> > > closely matches the requirements of GTK.  
> >
> > I'm not sure I quite understand what you are saying. What does this
> > mean for image loading if terms of actual implementation? What would
> > ideally happen then GTK+ needs to load an image (because the
> > application called gtk_image_new_from_file() for example)?
> >
> >  
> We're still using GdkPixbuf, and for the 4.0 API series we still have
> GdkPixbuf types exposed in the GTK API.
>
> For future API series (5.x, 6.x, …) we may revisit this, with the
> option of moving icon loading into GTK itself.

Okay, sound reasonable.


> > > Cairo is still used as a fallback mechanism — both when running on
> > > platforms without support for OpenGL or Vulkan, and in the interim
> > > period while the GL/Vulkan rendering pipelines inside GTK don't
> > > support a CSS feature. Additionally, you may still use Cairo for
> > > 2D high quality rendering, for printing and for custom drawing.  
> >
> > But then it comes to printing and custom drawing I'm on my own then
> > it comes to loading the images? Which is okey of course since
> > gdk-pixbuf is kind of a separate library already. But isn't it a
> > good thing to share common image loading code?
> >  
>
> Not really; icons—especially ones that are identified via icon theme
> names—are a very narrow subset of "all the possible images and
> formats in human history".
>
> > Gegl is great for image editing. But not as much for simple viewing.
>
> This is debatable. If I'm viewing a 4000x4000 RGB image on a hidpi
> display I'm already pushing gdk-pixbuf and cairo to their limits
> because of the scaling factor applied to the window — not only the
> buffer gets loaded uncompressed to allow for zooming, but the image
> viewer needs to render a CPU-scaled down copy of the image.

I often deal with images of those sizes and I haven't experienced any
problems with this myself (sure it's not as fast as viewing small
images). But I will look into it. I though cairo was capable of
utilising the rendering back end for it's scaling. I don't see why
cairo NEEDS to use CPU-scaling, so maybe it can be fixed.


> > It doesn't do animation  
>
> Animated formats are a dying breed, and they have all but killed by
> VP9 and new, web-friendly video formats. Social platforms will take a
> GIF and turn it into a video transparently.
>
> Additionally, GTK 4.x already has new API to render videos and other
> frame-based media sources through GStreamer.

I must agree this sounds like a good decision. Even though I don't
think animated GIFs will die any time soon, it's not really anything
GTK+ should need to pay special attention to. From my perspective,
being interested in the art scene, animation might not be very common,
but formats supporting animation certainly are. It's not only in the
form of a series of frames (like GIF), it's also palette cycling, or
simply one blinking color I include in the concept of animation. I've
also seen image formats that shows the current time. Those, I think,
should ideally be displayed with the time updating. So that's also a
kind of animation. Does GStreamer support vector animations, by the way?


> > > From the perspective of a consumer of GdkPixbuf, the only thing
> > > that  
> > > an image loading library should do is, pretty literally, load
> > > images. No scaling, no compositing, no rendering. Saving to a
> > > file may be interesting — but that opens the whole transcoding
> > > can of worms, so maybe it should just be a debugging feature.
> > > Ideally, I/O operations should happen in a separate thread, and
> > > possible use multi-threading to chunk out the work on multiple
> > > cores; it definitely needs to integrate with GIO, as it's a
> > > modern API that well maps to GUIs.  
> >
> > Yes, I very much agree with this. Except I think it has to do
> > rendering of some sort. If an image contains a triangle, it has to
> > be converted to a function call that renders it (using
> > OpenGL/Vulcan/cairo) somehow.
>
> That's up to the toolkit, as it's going to be the one deciding how to
> render the rest of the UI. There's no way for you to know, from the
> outside, how to efficiently render anything.
>
> That's why it's much, much better to get a memory buffer with the
> contents of the image, and let the toolkit pass it to the GPU.
>
> Vector formats are different, but that's why we have API like Cairo or
> Skia; I'm also waiting for web browsers to implement SVG rendering on
> the GPU where possible, with the help of new primitives from the
> GL/Vulkan implementation.

That means one code path for images that are certainly raster based and
another code path for images that could contain vector elements? My
thought was to always use cairo (or something similar) and just ignore
if the images are raster based or vector based (or a combination of the
two). As long as there is no need to directly access the pixels, there
is generally no need to even know difference then using the API. But you
say there is no way for cairo to know how to effectively render a
pixmap? Could this problem possibly be solved in cairo? I guess the
problem is that cairo can't keep track of things such as images stored
in the texture RAM. Would it be theoretically possible to expand cairos
API to support such things, you think? So that a PDF viewer for example
could store embedded images in texture RAM and render the whole thing
in different sizes super fast, by only using cairo (or at least
something like cairo).


> > In the near future, I'll very likely deprecate most of GdkPixbuf's  
> > > API, except for the I/O operations; I'd also be happy to seal off
> > > most of its internals, within the ABI stability promise, to avoid
> > > leakage of internal state.  
> >
> > Will the loader plugin API go away, you think?
> >  
>
> No API will ever go away: there are no plans for a gdk-pixbuf-3.0. The
> deprecations would mostly apply to API that is either long since been
> replaced by Cairo (scale/composite), or that is weirdly ad hoc, like
> saturate_and_pixelate(). Ideally, I'd like to deprecate the option to
> build gdk-pixbuf without depending on GIO to do MIME type sniffing,
> as GIO has been fixed to work on Windows and macOS, and it is a
> required dependency anyway. The animation API is pretty much a
> garbage fire, so it may go on the chopping block as well. Of course,
> deprecated API will keep working as well—or as badly—as it works
> today, so people can still use it. Moving pixbuf loaders to separate
> processes, and wrap them in sandboxes, would be a fairly good thing
> to do; it need to be decided at run time, though, because there are
> many users of GdkPixbuf that already run in a sandbox, which prevents
> creating smaller sandboxes inside it.

There was talk about ignoring the sniffing patterns (as well as the
file extensions) provided by the image loader plugins and only relying
on glib (and the shared MIME database) for sniffing the type. But it's
now a compile time option you say?

How does file type handling work on Windows nowadays? Last time I tried
it (about the time GTK+ 3,0 was released), it didn't work at all because
glib on Windows relied on Windows' own type system which is (or was)
completely broken. (I needed to ship the MIME database and the Unix
version of the type checking code from glib compiled for Windows with
my application in order to get it to work reliably.)

As I wrote in another answer, I don't think it's ideal to put small
tasks such as image decoding in micro sandboxes. It's probably better
to put even more data processing stuff in the same sandbox (like
applying filters to an image for example). Especially if the two
approaches conflicts with one another.
_______________________________________________
gtk-devel-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: An alternative to gdk-pixbuf

Bastien Nocera
In reply to this post by Gtk+ - Dev - General mailing list
On Thu, 2018-09-06 at 11:39 +0100, Emmanuele Bassi via gtk-devel-list
wrote:

> On Wed, 5 Sep 2018 at 19:25, Magnus Bergman <
> [hidden email]> wrote:
> > On Wed, 5 Sep 2018 17:28:22 +0100
> > Emmanuele Bassi <[hidden email]> wrote:
> >
> > > We're phasing out Cairo in favour of the CSS rendering model,
> > > implemented on top of OpenGL and Vulkan, as it's the API that
> > most
> > > closely matches the requirements of GTK.
> >
> > I'm not sure I quite understand what you are saying. What does this
> > mean for image loading if terms of actual implementation? What
> > would
> > ideally happen then GTK+ needs to load an image (because the
> > application called gtk_image_new_from_file() for example)?
> >
>
> We're still using GdkPixbuf, and for the 4.0 API series we still have
> GdkPixbuf types exposed in the GTK API.
>
> For future API series (5.x, 6.x, …) we may revisit this, with the
> option of moving icon loading into GTK itself.

This uncertainty is the reason why I didn't work on moving the
thumbnailing code somewhere else, as it's bound to be made irrelevant
sooner rather than later.

<snip>

> > Gegl is great for image editing. But not as much for simple
> > viewing.
>
> This is debatable. If I'm viewing a 4000x4000 RGB image on a hidpi
> display I'm already pushing gdk-pixbuf and cairo to their limits
> because of the scaling factor applied to the window — not only the
> buffer gets loaded uncompressed to allow for zooming, but the image
> viewer needs to render a CPU-scaled down copy of the image.
>
> Sure, for viewing a 500x400px image macro for a meme we're fine; but
> we're fine with gdk-pixbuf as well, so there's really no need to
> change to a different image loading library.

I concur, 4000x4000 would likely OOM your machine, and it's not really
fixable:
https://gitlab.gnome.org/GNOME/gdk-pixbuf/issues/30

Viewers should use GEGL, and GEGL should be taught about more formats.
That still leaves many GIF files unhandled though, and I'm not sure
we'd want apps writing their own GIF loader, seeing how complicated
that is.


> In the near future, I'll very likely deprecate most of
> > GdkPixbuf's
> > > API, except for the I/O operations; I'd also be happy to seal off
> > > most of its internals, within the ABI stability promise, to avoid
> > > leakage of internal state.
> >
> > Will the loader plugin API go away, you think?
>
> No API will ever go away: there are no plans for a gdk-pixbuf-3.0.
> The deprecations would mostly apply to API that is either long since
> been replaced by Cairo (scale/composite), or that is weirdly ad hoc,
> like saturate_and_pixelate(). Ideally, I'd like to deprecate the
> option to build gdk-pixbuf without depending on GIO to do MIME type
> sniffing, as GIO has been fixed to work on Windows and macOS, and it
> is a required dependency anyway.

That's probably fine, though the current code in the stack is dumb as
rocks, and will try to thumbnail a JPG file with the PNG decoder if the
suffix is incorrect. The error message is also obnoxious, so that'll
need fixing before removing the content type sniffing.

>  The animation API is pretty much a garbage fire, so it may go on the
> chopping block as well. Of course, deprecated API will keep working
> as well—or as badly—as it works today, so people can still use it.
> Moving pixbuf loaders to separate processes, and wrap them in
> sandboxes, would be a fairly good thing to do; it need to be decided
> at run time, though, because there are many users of GdkPixbuf that
> already run in a sandbox, which prevents creating smaller sandboxes
> inside it.

That'd be best left to an API that can do that natively, with the API
being thought out ahead of time.

In the short term, my wishlist/TODO list would be:
- somebody writes a GIF loader that is readable, and passes the test
suite.
- we disable every loader by default except the ones that the core
desktop needs
- image viewers that rely on gdk-pixbuf ship their additional loaders
in the app's Flatpak[1].

That would already reduce the attack surface for gdk-pixbuf.

On a sidenote, I also use gdk-pixbuf as a pixel-perfect drawing API for
thumbnailers:
https://gitlab.gnome.org/Archive/gnome-nds-thumbnailer/blob/master/gnome-nds-thumbnailer.c#L72

[1]: I recently did this to add webp comics support to the evince
Flatpak:
https://gitlab.gnome.org/GNOME/evince/merge_requests/50

_______________________________________________
gtk-devel-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: An alternative to gdk-pixbuf

Debarshi Ray-2
In reply to this post by Magnus Bergman-3
Hey Magnus,

I haven't yet worked my way through the whole thread. It's pretty
long and will take me a while longer, but I did want to mention a
few things before the weekend draws me away from the computer.

On Wed, Sep 05, 2018 at 12:02:45AM +0200, Magnus Bergman wrote:
> Over the years it has been discussed from time to time to replace
> gdk-pixbuf with something else[1][2]. Something was even in the making
> (I guess over ten years ago) but it never replaced gdk-pixbuf
> apparently. Now I don't even remember what it was called. And something
> else called pig was proposed more recently.

As Emmanuele mentioned, if you ignore the use-case of loading icons
in GTK, and focus on image applications dealing with high resolution
images, then GEGL [1] is the way to go. It's a GObject-based image
processing framework that's used by GIMP and GNOME Photos, and is
way more advanced than anything that we have in the GNOME platform.

If you use GIMP 2.10 or practically any version of GNOME Photos, then
all the image decoding and processing is happening with GEGL.

A GeglBuffer is the equivalent of GdkPixbuf in GEGL. It can:

(a) Be converted to and from a GdkPixbuf. That makes porting trivial
    and boosts interoperability.

(b) Handle massive images that are larger than the amount of physical
    RAM available on the system. This is achieved by spliting the
    pixels into smaller tiles that get transparently swapped out of
    memory into a disk-backed cache when necessary.

    It can be dumbed down into a GdkPixbuf-like linear sequence of
    bytes, if need be.

(c) Represent a whole horde of pixel formats, colour spaces, bit depths,
    etc. [2], which can be programmatically extended at run-time.

(d) Automatically convert between pixel formats, colour spaces, bit
    depths, etc., with accelerated fast paths.

(e) Be serialized into a named file, which is interesting for passing
    pixels across process boundaries.

(f) Use mipmaps for scaling, rendering and processing the pixels.

(g) Be used with OpenCL.

(h) Do all that and is still fast. Try zooming in and out in GNOME
    Photos.

Not to mention a vast array of image processing operations. If it's in
GIMP, it's in GEGL. Interesting operations are ported from elsewhere,
and there's a constant flow of new things under development.

> One major reason to replace gdk-pixbuf has been the long term goal to
> phase out GDK in favour of cairo.

De/encoding images and manipulating the pixels is related to rendering
them on the screen, but aren't necessarily the same thing. It's
important to ensure that the pixels can be efficiently drawn, but I
wouldn't add a hard dependency on the drawing framework. For example,
GEGL has fast paths for drawing with Cairo, but doesn't mandate it.
It also doesn't require GTK, even though it has helper widgets for it.

That's why GIMP 2.10 and GNOME Photos can use the same libgegl-0.4.so,
even though they use different widget toolkits (GTK 2.x versus 3.x).

> This is how to convert an SVG file to a PNG file.
>
>     cairo_surface_t *surface = abydos_load("image/svg+xml",
>     "example.svg"); cairo_surface_write_to_png(surface, "example.png");
>     cairo_surface_destroy(surface);
>
> This is how to convert the second frame of a GIF animation to PNG.
>
>    abydos_t *ar = abydos_create_from_file("image/gif", "example.gif");
>    abydos_set_frame(ar, 1);
>    cairo_surface_t *surface = abydos_get_image_surface(ar, 0);
>    abydos_destroy(ar);
>    cairo_surface_write_to_png(surface, "example.png");
>    cairo_surface_destroy(surface);

I don't see any error handling. Maybe I should read the actual code.

Either way, as Emmanuele mentioned, it's really important to have
support for GIO facilities like GCancellable, GFile, streams,
GAsyncResult-based asynchronous APIs, etc..

If you look closely at GEGL, you'll see that while it already has
codecs, they are missing these requirements. It's an artifact of
GIMP historically having it's own in-house codecs.

However, that's changing. I have a work-in-progress GNOME Photos
branch [3] that prototypes a better codec API for GeglBuffer.

See:
* https://gitlab.gnome.org/GNOME/gnome-photos/blob/wip/rishi/buffer-decoder/src/photos-gegl-buffer-io.h
* https://gitlab.gnome.org/GNOME/gnome-photos/blob/wip/rishi/buffer-decoder/src/photos-gegl-buffer-loader.h

One obvious thing is that it looks almost identical to the corresponding
GdkPixbuf API. That's an explicit goal to make porting trivial.

Of course, one could use the GdkPixbuf codecs for I/O and then covert
to and from a GeglBuffer. But then you wouldn't be able to handle RAWs
or higher bit-depth PNGs and will likely run into problems with your
80 megapixel images [4] - things that a good image application will
care about. :)

> I've been in contact with a two creators of image viewers

Which ones? Just curious. :)

> What it all comes down to is that using a bigger variety of software
> creates a bigger attack surface. Viewing a lot of images is less secure
>
> [...]
>
> formats (MIME types) in specially vulnerable situations (like for auto
> displaying images from internet). And for extra security all but a core
> set of loaders could be removed.

Flatpak plays a big role here. If the application is sandboxed, the
attack surface decreases quite a bit.

As Ray said, you can also move the decoding into a separate process
that's even more tightly locked down than the main application. This
is what GIMP does. You can serialize the GeglBuffer to transfer the
pixels, without using any OS-specific facility, which is a nice touch
if you want to be portable across Linux, Windows, OS X, etc..

Happy hacking,
Rishi

[1] http://gegl.org/
[2] http://gegl.org/babl/
[3] https://gitlab.gnome.org/GNOME/gnome-photos/tree/wip/rishi/buffer-decoder
[4] https://en.wikipedia.org/wiki/Phase_One_(company)
_______________________________________________
gtk-devel-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: An alternative to gdk-pixbuf

Debarshi Ray-2
In reply to this post by Magnus Bergman-3
On Wed, Sep 05, 2018 at 08:25:05PM +0200, Magnus Bergman wrote:
> Gegl is great for image editing. But not as much for simple viewing. It
> doesn't do animation

People have been creating and playing videos with it:
http://gegl.org/gcut.html

> Also it only loads images from the
> file system and is a bit bulky to use.

The existing codecs aren't good, but they do support GIO URIs through
GFile and streams.
_______________________________________________
gtk-devel-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: An alternative to gdk-pixbuf

Magnus Bergman-3
In reply to this post by Federico Mena Quintero-4
On Thu, 06 Sep 2018 13:03:03 -0500
Federico Mena Quintero <[hidden email]> wrote:

> On Wed, 2018-09-05 at 17:28 +0100, Emmanuele Bassi via gtk-devel-list
> wrote:
>
> > In the near future, I'll very likely deprecate most of GdkPixbuf's
> > API, except for the I/O operations; I'd also be happy to seal off
> > most of its internals, within the ABI stability promise, to avoid
> > leakage of internal state.  
>
> Related to this - I just wrote a braindump on gdk-pixbuf issues and
> historical baggage.  I hope it's useful for people trying to clean up
> this swamp:
>
> https://people.gnome.org/~federico/blog/my-gdk-pixbuf-braindump.html

This was an interesting read. It touched many of my own ideas and
concerns.

Animation

I actually think the concept of animations in gdk-pixbuf is okay, so
it's pretty much the same in abydos, with the main difference that
abydos fully supports vector animation. Apart from the frame based view
of animations abydos also support continuous animations (which it seems
like gdk-pixbuf also does). It's mostly the API of gdk-pixbuf that is
suboptimal. Apart from GIF there are a few formats common in the art
scene that supports animation. It's not just flip book animations but
also palette cycling or simply one blinking color. There are also
formats that show the current time (or depend on other variables that
can change over time). Ideally an image viewer should update those
then appropriate, so they can also be considered animations. (As you
might have noticed I'm quite interested in the odd corner cases.)

Mutable pixbufs

The problem with mutable pixbufs is pretty much the same problem
described in the documentation for abydos_get_image_surface(). The user
can promise to either not modify pixels or not use the image for
anything else after the modification (this is of course the fastest
alternative and currently the default). Or the user can request the
pixels to be copied or transferred, in which case it's safe to both
modify the pixels and use the image for something else. This solution
is a bit awkward, but since I think the copy/transfer scenarios are
actually corner cases (which I still want to support) the main use case
should remain as fast as possible, and also the one not requiring
extra typing.

Pluggable loaders

Pluggable loaders it's something I very much appreciate since I can
write my own loaders for the additional formats I care about (ten or
so) and use common image viewers to view them. Otherwise I would need to
write the whole viewer myself as well. The only alternatives are image
viewers on original hardware (emulators may or may not cut it since the
image viewers often use hardware tricks that are difficult to emulate
and most of them doesn't even try very hard to correct the aspect
ratio). Or try to get some strange and glitchy Windows application
running in Wine, often with a suboptimal result since it doesn't
support all details I care about (such as a correct aspect ratio and
palette cycling). Some art groups release their own viewers, that are
usually quite good. But they usually only support the subset of
features their own paint applications support. One viewer to view them
all would be better. But all your favourite viewers to view them all
would be the best IMHO.

Multi page files

Multi page TIFF images and SVG images with multiple icons inside are
conceptually the same to abydos. The same API could support both.

Limitation of supported image formats

It's clear that GTK+ doesn't need to load that many image formats. Most
applications doesn't either. And it's clear that more supported formats
is a bigger security risk. (But I would like to add that this isn't
limited to image loading plugins, in general more software is less
secure than less software). For image viewers I think the priorities
are different. Most would opt for more more more. (It's not like people
would opt for less games on steam just because that would reduce the
attack surface, to draw a parallel.) Also, distros would probably
package the more exotic loaders in different packages, not installed by
default.

In case applications shouldn't be allowed to read any kind of image, a
filtering could be installed that only let blessed MIME types through.
It could easily be added to a wrapper around abydos. Apart from the
security aspect I think it's a good thing if an application can open
anything you throw at it. In the word processor example you mentioned
it's especially true IMHO. Personally I use Lyx, and it gives you the
ability to teach it about new formats if you just tell it how to
convert it to a format it already knows about. I use that a lot and
find it very useful and time saving. But it's a problem if you expect
others to be able to open it (which I don't, I export it to PDF
instead). So for any kind of format intended for data exchange it
should be limited, and clearly defined, what you can embed in it.

In the end it seams to boil down to this. You can only pick two:
  1) comprehensive image format support.
  2) security.
  3) speed.
I think everybody could be happy if the choice could be made at
run-time.

Progressive loading

I also doubt there are some actual use case that needs progressive
loading. A progress bar would probably do in the cases it takes a
little time to load an image I think. In abydos that API is always
supported, but it's not required for plugins to support it. In case
they don't abydos will buffer the bytes, call plugin with all data
available, then invoke the callbacks to the user supplied functions.
I'm quite sure it wouldn't break anything if gdk-pixbuf did the same.
It could be mentioned that some image formats cannot be progressively
loaded anyway since crucial data is placed at the very end (or they
have no predefined height, just add rows until there is no more data).

The feature to request another size than the reported size was mainly
added to support vector images at larger scale I think. I'm even quite
sure it wasn't intended for taking a short path then a smaller image
was requested. Once I suggested that if an image contained an embedded
thumbnail, that could be returned if the user requested that size (or
smaller), especially then creating thumbnails. I was told no, images
should always be loaded at their largest size and then scaled down.

By the way. Back 1998 I don't think loading images over a slow internet
connection was the only use case for progressive loading. You could
definitely see images load progressively even then loading from disk too
(and I'm not even talking about floppy disks).

Non-raster image formats

For non-raster data abydos solves the problem by being agnostic to what
an image can be. It's simply anything that can be rendered using cairo.
This made it especially easy to make plugins for SVG and PDF since
librsvg and poppler has this exact same approach.

Meta data

Then it comes to meta data abydos is almost useless, it only cares
about meta data that affects viewing the image. Trying do do more will
either be overly ambitious or half assed. I think there is no middle
ground. For general purpose meta data extraction libextractor is most
useful (but not perfect). And I don't think this is something GTK+ needs
at all. For loading one format and saving to another users might expect
the meta data to be preserved to some degree, but to what degree is
completely arbitrary and I guess it's impossible to find one that makes
everybody happy. I think it's better do to what netpbm does, simply
ditch all meta data. But I'm glad you mentioned color space support.
That's something I have to think about.

MIME sniffing

I was pretty sure the custom MIME sniffing was already removed from
gdk-pixbuf (according to another answer it seems to be a compile time
option). Otherwise it's done twice. Because then I write a loader I need
to add the corresponding MIME type to the shared MIME database as well
for it to work. In case it has been removed, I think the pattern field
(as well as the extension field) in all loaders should be left empty. (I
assume the ABI isn't intended to be stable in that direction.)

Rust

I also like the idea of writing plugins in Rust. My experience with
Rust is very limited (and a bit dated). But to me Rust seams to be best
suited for large applications. It's quite possible to create very tiny
binaries, but as soon as you use one something from a library tons of
stuff gets dragged in (as with any language). And it gets very
noticeable since everything is statically linked. It seams you have
better experience with Rust and I look forward to see what you come up
with.

The main question, at least for me, is: Should the image loading
library that GTK+ uses to load images be a general purpose image loading
library also usable for image viewing applications?

It seams clear to me that the general opinion points to no. And that
the goals of abydos doesn't match the needs for GTK+. The problem
with the current situation is that creators of image viewing
applications perceive gdk-pixbuf as a general purpose image loading
library and uses it for that purpose. One could say that the interest
of GTK+ developers are in conflict with the interest of users of image
viewing applications. In that case the obvious solution is to clearly
state that gdk-pixbuf is not a general purpose image loading library,
but only intended for loading icons for GUI elements. (And possibly let
the whole thing die in time for GTK+ 5.)

> Magnus - thanks for showing us Abydos.  The parts of the API that
> intersect with gdk-pixbuf's look okay.  I think you may find this
> useful:
>
> https://developer.gnome.org/programming-guidelines/stable/index.html.en
>
> I think your API could improve with some of the material there around
> memory management and error reporting.  Internally, the code needs
> checks in all the malloc()-like functions; some of the arithmetic
> (especially in assertions) needs to avoid overflows.  I couldn't find
> any tests.  The SVG plug-in needs to handle error results from
> librsvg. The Magick plug-in for PNG and JPEG has no error handling;
> also you may want to call cairo_surface_mark_dirty() when you are
> done writing pixels to it.  The netpbm plugin does a bunch of system
> calls and doesn't check for errors.

Yes, the code is a bit sketchy right now. I'm still in a phase then I
consider throwing away and rewriting large parts. I will definitely add
better error handling before 1.0 (and well before that hopefully). I'm
mainly focusing on getting the API right at the moment. But I appreciate
you point these things out (I'll write them down so I don't forget
about them).

There are tests (no comprehensive test suit though), but they all rely
on non-free images, so I decided to leave that out for now.

This is a bit off topic. But are there actually modern (non real time)
systems that can recover from a state there malloc() has failed?
At least in Linux it will basically only happen if you run out of
memory addresses (which will not happen on a 64-bit system at least).
And if that happens you're long since screwed anyway. Since the actual
allocation will happen then you write to the memory, malloc() will
continue to give you memory that doesn't exist. I just assumed any
modern non real time system takes this approach. (And if one aims for
true real time compatibility it opens up a whole new can of worms.)

> The parts of your API that deal with pagination / variants / animation
> frames - I'm not sure if a gdk-pixbuf replacement would need them, but
> I'm coming from the viewpoint of my braindump above.

It's not pagination, as in chopping up an image for printing it on
several pages. The concept of pages (in lack of a better term I guess)
is about files containing multiple images with an otherwise unspecified
relationship. Such as SVG files containing multiple icons. The API is
also designed so that everything related to pages, layers, variants and
animations can just be ignored, both by applications and plugins, then
not needed. (And the extra bytes it adds to the binary is not
overwhelming.)

Anyway, thank you very much for your valuable feedback!
_______________________________________________
gtk-devel-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: An alternative to gdk-pixbuf

Magnus Bergman-3
In reply to this post by Bastien Nocera
On Fri, 07 Sep 2018 12:51:32 +0200
Bastien Nocera <[hidden email]> wrote:

> > > Gegl is great for image editing. But not as much for simple
> > > viewing.  
> >
> > This is debatable. If I'm viewing a 4000x4000 RGB image on a hidpi
> > display I'm already pushing gdk-pixbuf and cairo to their limits
> > because of the scaling factor applied to the window — not only the
> > buffer gets loaded uncompressed to allow for zooming, but the image
> > viewer needs to render a CPU-scaled down copy of the image.
> >
> > Sure, for viewing a 500x400px image macro for a meme we're fine; but
> > we're fine with gdk-pixbuf as well, so there's really no need to
> > change to a different image loading library.  
>
> I concur, 4000x4000 would likely OOM your machine, and it's not really
> fixable:
> https://gitlab.gnome.org/GNOME/gdk-pixbuf/issues/30
>
> Viewers should use GEGL, and GEGL should be taught about more formats.
> That still leaves many GIF files unhandled though, and I'm not sure
> we'd want apps writing their own GIF loader, seeing how complicated
> that is.

How do you mean it's not fixable? Of course it's not fixable that
processing larger amounts of data will require more processing power.
So I think it's kind of obvious that creating thumbnails for many large
images will take it's time. But that shouldn't need be too much of a
problem. Without looking too deeply into it, it looks like a problem
with nautilus to me. Obviously it's a bad idea to to run the
thumbnailers synchronously (which I can't imagine Nautilus does), and
it's also bad to run too many memory hungry tasks in parallel. But
that must be fixable, right? Like limiting the the total size in bytes
of files being allowed to be processed in parallel, for example.

At least now I see the problem with large images. I have no problem with
4000x4000, such images load instantly and zooms and pans smoothly. But
the 11000x14000 image was a bit sluggish to zoom and pan. With Gimp it
took a lot longer to load it, but panning and zooming was fast and
smooth. While gdk-pixbuf is optimised for delivering one pixmap, GEGL is
optimised for editing. I have some idea of how to optimise for viewing
by doing a simplified form of prescaling, compared to GEGL (and
hopefully outperform them both for that particular use case). Otherwise
it would be possible (but probably not always optimal) to use GEGL as a
backend for abydos then dealing with potentially large images. I need to
investigate this. Since it's common with multi gigabyte images in many
fields of science, this is something I obviously can't ignore.

> > In the near future, I'll very likely deprecate most of  
> > > GdkPixbuf's  
> > > > API, except for the I/O operations; I'd also be happy to seal
> > > > off most of its internals, within the ABI stability promise, to
> > > > avoid leakage of internal state.  
> > >
> > > Will the loader plugin API go away, you think?  
> >
> > No API will ever go away: there are no plans for a gdk-pixbuf-3.0.
> > The deprecations would mostly apply to API that is either long since
> > been replaced by Cairo (scale/composite), or that is weirdly ad hoc,
> > like saturate_and_pixelate(). Ideally, I'd like to deprecate the
> > option to build gdk-pixbuf without depending on GIO to do MIME type
> > sniffing, as GIO has been fixed to work on Windows and macOS, and it
> > is a required dependency anyway.  
>
> That's probably fine, though the current code in the stack is dumb as
> rocks, and will try to thumbnail a JPG file with the PNG decoder if
> the suffix is incorrect. The error message is also obnoxious, so
> that'll need fixing before removing the content type sniffing.

Then I ran gdk-pixbuf in a debugger I noticed that it ignored the file
name and only used the content then determining the file type. So
is there another code path which uses the filename, or is it a compile
time option? I noticed that the MIME type detection code (which I think
is more or less a copy paste of the reference implementation) uses the
filename alone to determine the MIME type if the extension is
considered reliable enough. Just as a secondary method it will use the
supplied content for sniffing. It's also possible that it returns more
than one MIME type and in that case gdk-pixbuf will always use the
first one. This is annoying since a completely made up MIME type (like
"application/x-extension-png") can be prepended to the list sometimes,
so the real one gets ignored. This is apparently a feature I don't
understand.

> >  The animation API is pretty much a garbage fire, so it may go on
> > the chopping block as well. Of course, deprecated API will keep
> > working as well—or as badly—as it works today, so people can still
> > use it. Moving pixbuf loaders to separate processes, and wrap them
> > in sandboxes, would be a fairly good thing to do; it need to be
> > decided at run time, though, because there are many users of
> > GdkPixbuf that already run in a sandbox, which prevents creating
> > smaller sandboxes inside it.  
>
> That'd be best left to an API that can do that natively, with the API
> being thought out ahead of time.
>
> In the short term, my wishlist/TODO list would be:
> - somebody writes a GIF loader that is readable, and passes the test
> suite.

I've written loader for GIF that simply wraps abydos. In lines of
code it's about a quarter the size of the current loader, even including
the GIF plugin for abydos. It might even be slightly smaller with the
whole of abydos included in the equation. On the downside it probably
doesn't pass the test suite since I haven't tried it. But I will, and
hopefully publish the whole thing in a couple of days.

> - we disable every loader by default except the ones that the core
> desktop needs
> - image viewers that rely on gdk-pixbuf ship their additional loaders
> in the app's Flatpak[1].

I don't care much for Flatpak in particular. But generalised and
rephrased as, leave it to the distributors to decide, I agree that
this is absolutely the best approach.

> On a sidenote, I also use gdk-pixbuf as a pixel-perfect drawing API
> for thumbnailers:
> https://gitlab.gnome.org/Archive/gnome-nds-thumbnailer/blob/master/gnome-nds-thumbnailer.c#L72
>
> [1]: I recently did this to add webp comics support to the evince
> Flatpak:
> https://gitlab.gnome.org/GNOME/evince/merge_requests/50

Nice!
_______________________________________________
gtk-devel-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-devel-list
12