Can't launch apps, fork() returns ENOMEM

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Can't launch apps, fork() returns ENOMEM

Daniel Drake
Hi,

At Endless we're looking at how to make GNOME run better on systems
with low amounts of RAM.

One issue biting on a regular basis particularly on systems with (say)
1GB RAM is that apps will refuse to launch from gnome-shell even when
there's a decent enough chunk of memory available.

The user sees error message:
   fork(): Cannot allocate memory

When gnome-shell is launching apps (via glib) it ultimately comes down
to fork() + exec(). In this case the fork() fails with ENOMEM, because
the Linux kernel worries that the process being forked may end up
duplicating all of the memory allocations of the shell. By default the
memory map is set up so that the new process has a view on the exact
same pages as the parent process, however they are set up as
copy-on-write, so if the child writes to such memory it'll silently
cause new memory to be allocated. Under that limited perspective, the
kernel is not totally out of line in worrying about this situation,
especially because gnome-shell is a RAM-heavy process.

In reality we only want to fork() to immediately exec() which replaces
the child process memory map with a blank slate, but this
misinterpretation of intentions is a limitation of the fork() API
combined with a conflict with Linux's memory overcommit model.

The solution to this leads us towards alternatives to fork(). vfork()
can create the forked process using the exact same memory address
space as the parent process, meaning that there's no memory that might
need to be duplicated, avoiding this failure condition.
clone(CLONE_VM) can do the same thing. But two processes using the
exact same memory draws in a whole bunch of nasty complications, it's
dangerous.
https://ewontfix.com/7/
https://gist.github.com/nicowilliams/a8a07b0fc75df05f684c23c18d7db234

Discussions here have lead to glibc's posix_spawn() being recently
reimplemented to use clone(CLONE_VM) while solving practically all the
danger there. How can we do something similar in GNOME?

The complications mostly boil down to that you need to be rather
careful in what you do in the child process before you exec(), since
you are sharing the parent's memory. And the underlying
g_spawn_async_with_pipes() is a powerful interface which can do a
number of things in that danger zone, depending on flags passed in.

I started to implement an alternative codepath for
g_spawn_async_with_pipes() to execute when the flags aren't
particularly complex (which appeared to be the case for app
launching), based on the posix_spawn() design. But I ran into too many
headaches and brick walls there. For example you have to allocate a
stack for the child process so you need to know if the stack grows up
or down on your architecture. glibc knows these architectural details
well, but we don't. Then the games with signal handling - ideally we
need to block glibc-internal signals (e.g. NPTL stuff) in the child
process, but that's also a hairy task to do outside of the glibc
codebase. etc.

Then I considered making glib just call posix_spawn() directly when
appropriate. There are a number of things in that API which let
relevant actions be done in the child process before it does the exec
- managing file descriptors, signal handlers, etc. The big issue there
is that glib lets you pass an arbitrary GSpawnChildSetupFunc to be run
in the child before the exec(), this is used by gnome-shell, and there
is no equivalent hook in posix_spawn().

In gnome-shell app_child_setup() this function is used to send stdout
and stderr to journald. Also in glib gdesktopappinfo.c child_setup()
the same mechanism is used to set GIO_LAUNCHED_DESKTOP_FILE_PID in the
child process to it's own process PID (can't find the background
here).

I think there are 2 viable paths forward:

 1. Eliminate the child_setup calls from these codepaths to allow
posix_spawn() to be used. The gspawn code already allows for file
descriptor redirection but this would have to be exposed via
additional parameters to g_desktop_app_info_launch_uris_as_manager() -
an API change. If GIO_LAUNCHED_DESKTOP_FILE_PID can't be reimplemented
another way then we could first exec a binary wrapper that sets this
env variable before execing the app itself.

or

 2. Propose to http://www.opengroup.org/austin/ that posix_spawn()
grows the capability to call a user-specified child setup func. There
is some unused padding in posix_spawnattr_t which could be used to
store the function pointer and data pointer. Then implement this in
glibc. I guess this is a lengthy process and it's not a great idea
from the glibc standpoint where such code would run in dangerous
context.

I'm inclined to persue the first approach, of removing child_setup
from this codepath and then implementing an optimized gspawn codepath
that uses posix_spawn() when the conditions are right.

I'd appreciate any comments before I continue though, does this design
sound acceptable? Am I missing anything?

Thanks,
Daniel
_______________________________________________
gtk-devel-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: Can't launch apps, fork() returns ENOMEM

Alan Coopersmith-2
On 04/ 2/18 12:15 PM, Daniel Drake wrote:
> Discussions here have lead to glibc's posix_spawn() being recently
> reimplemented to use clone(CLONE_VM) while solving practically all the
> danger there.

We've done something similar in the Solaris libc recently, so I'd
support using posix_spawn if possible.

> Then I considered making glib just call posix_spawn() directly when
> appropriate. There are a number of things in that API which let
> relevant actions be done in the child process before it does the exec
> - managing file descriptors, signal handlers, etc. The big issue there
> is that glib lets you pass an arbitrary GSpawnChildSetupFunc to be run
> in the child before the exec(), this is used by gnome-shell, and there
> is no equivalent hook in posix_spawn().
>
> In gnome-shell app_child_setup() this function is used to send stdout
> and stderr to journald. Also in glib gdesktopappinfo.c child_setup()
> the same mechanism is used to set GIO_LAUNCHED_DESKTOP_FILE_PID in the
> child process to it's own process PID (can't find the background
> here).
>
> I think there are 2 viable paths forward:
>
>  1. Eliminate the child_setup calls from these codepaths to allow
> posix_spawn() to be used. The gspawn code already allows for file
> descriptor redirection but this would have to be exposed via
> additional parameters to g_desktop_app_info_launch_uris_as_manager() -
> an API change. If GIO_LAUNCHED_DESKTOP_FILE_PID can't be reimplemented
> another way then we could first exec a binary wrapper that sets this
> env variable before execing the app itself.

This also seems like the better path to me.  It does seem strange that
you need to set an environment variable in the child process to tell
it what it's PID is - perhaps it's for children of that child to find
the spawn point of their family?

>  2. Propose to http://www.opengroup.org/austin/ that posix_spawn()
> grows the capability to call a user-specified child setup func. There
> is some unused padding in posix_spawnattr_t which could be used to
> store the function pointer and data pointer. Then implement this in
> glibc. I guess this is a lengthy process and it's not a great idea
> from the glibc standpoint where such code would run in dangerous
> context.

The Austin Group does not like specifying API that no one has yet
implemented and proven to be working and useful - they'd rather you
get one of the Linux, BSD, or Unix libc's to first implement and show
the design works, then standardize it.

--
        -Alan Coopersmith-               [hidden email]
         Oracle Solaris Engineering - https://blogs.oracle.com/alanc
_______________________________________________
gtk-devel-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: Can't launch apps, fork() returns ENOMEM

Daniel Drake
In reply to this post by Daniel Drake
Spotted another complication here.

On Mon, Apr 2, 2018 at 1:15 PM, Daniel Drake <[hidden email]> wrote:

> Then I considered making glib just call posix_spawn() directly when
> appropriate. There are a number of things in that API which let
> relevant actions be done in the child process before it does the exec
> - managing file descriptors, signal handlers, etc. The big issue there
> is that glib lets you pass an arbitrary GSpawnChildSetupFunc to be run
> in the child before the exec(), this is used by gnome-shell, and there
> is no equivalent hook in posix_spawn().
>
> In gnome-shell app_child_setup() this function is used to send stdout
> and stderr to journald. Also in glib gdesktopappinfo.c child_setup()
> the same mechanism is used to set GIO_LAUNCHED_DESKTOP_FILE_PID in the
> child process to it's own process PID (can't find the background
> here).
>
> I think there are 2 viable paths forward:
>
>  1. Eliminate the child_setup calls from these codepaths to allow
> posix_spawn() to be used. The gspawn code already allows for file
> descriptor redirection but this would have to be exposed via
> additional parameters to g_desktop_app_info_launch_uris_as_manager() -
> an API change.

I was wrong in thinking that gspawn lets you say "run this binary, and
here's the fd that you should use for stdout".

The parameters that exist in the current interfaces either let you
inherit stdin/out/err from the parent, or create an entirely new pipe
in each case.

So, in order to avoid needing gnome-shell's app_child_setup(), we
would either need to extend the gspawn API to allow specific file
descriptors to be passed in, or perhaps another option to consider is
to have gdesktopappinfo call posix_spawn directly (instead of gspawn).

Daniel
_______________________________________________
gtk-devel-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: Can't launch apps, fork() returns ENOMEM

Simon McVittie
On Wed, 11 Apr 2018 at 14:02:57 -0600, Daniel Drake wrote:
> I was wrong in thinking that gspawn lets you say "run this binary, and
> here's the fd that you should use for stdout".

The interface for that is the newer GSubprocess and GSubprocessLauncher.

> So, in order to avoid needing gnome-shell's app_child_setup(), we
> would either need to extend the gspawn API to allow specific file
> descriptors to be passed in, or perhaps another option to consider is
> to have gdesktopappinfo call posix_spawn directly (instead of gspawn).

... or make GDesktopAppInfo use GSubprocess instead of g_spawn, and
have a way for GSubprocess to use posix_spawn instead of g_spawn
in the (hopefully common) case where its ChildData.child_setup_func
is NULL.

    smcv
_______________________________________________
gtk-devel-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-devel-list