gtk performance

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

gtk performance

ente
Hi,

I am facing performance issues in GTK upon presenting a big amount of tabular data. First I used GtkListbox for it's layout flexibility. Handling more than 10,000 items gets inacceptable slow. Switching to GtkTreeview I can handle some 300,000 items after applying a few optimizations (column sizing fixed, fixed height mode): a full load takes about 4 seconds, i.e. the application feels anything but "responsive" although this is ... ok. Currently I see 2 options: dropping Gtk in favor of a HTML frontend (that feels awkward to say) or implementing paging on the GtkTreeview.

At this moment I am analyzing the reasons for the performance issues. I measured the time it takes to generate a number of labels in a simple loop (no call ot GtkMain) and to add those labels to a GtkBox:
* 100 labels in 1.7 ms
* 1,000 labels in 20.9 ms
* 10,000 labels in 610 ms
* 100,000 labels in 1m18s
After the loop I pack the GtkBox into a window and show it. That takes another century to show up. 
If I adjust the loop to skip the packing of the labels into the box, the times are down to:
* 10,000 labels in 115 ms
* 100,000 labels in 940ms

The windows shows up instantly (although empty of course).

So my performance issue is related to the packing. In my tests with the GtkTreeview it seems speficially to boil down to the sizing and signalling of the items: The performance of the treeview was highly impacted by the fixed height mode & the fixed column sizing (more than ordering by a column or calling GtkMain while adding rows). So my question goes down to: How can I further optimize my way of using GTK in order to speed up UI? Is there any way to add 1,000,000 labels to a GtkBox in less than a second? Could I somehow suspend the signalling during UI creation and replace it by a "I am done, now calculate all the sizes" signal? Am I wrong about my assumption that the sizing signalling is the cause for the low performance?

With regards to the treeview: After initially creating the data rows I am calculating values that affect 4 columns out of 10 columns. So far I determine the GtkTreeIter of the affected row and update all cells in that row using gtk_list_store_set. Should I adjust to update the affected columns only? When I implement paging for the TreeView: should I rather drop the existing ListStore and create a new or is it faster to overwrite all elements in the Liststore with the new items?

I tried to use the GtkBuilder in order to setup a box with 100,000 labels. This performance was somewhat the same as creating the labels in a loop (I didn't keep the measurements).

I could isolate the problem to UI rendering. If I don't assign my ListStore to the Treeview, all is fine. As soon as my ListStore is filled, I assign the model to the treeview. That's the most time consuming step.

Last question: is there any way to create callbacks from the javascript world using the Webkit2Gtk webview? More specificly: I am working in go language. I am aware of the webview widget. Unfortunately that's a window. I would prefer to rather place a widget into a native GtkApplicationWindow. Using Eval I can inject anything into the DOM. How do I get events from the DOM back to go?


Background on the application:
I am developing a text analysis application. Text fragments get precalculated attributes assigned. Based on these attributes and the Levensthein distance between fragments, someone is supposed to (for now: manually) define text fragment categories (categories). Roughly speaking the application has a paned window: on the left I show the category list or an editor for a new / existing category; on the right I show a list of text fragments with the major 3 attributes or a details view on a specific fragment: all attributes and a list of similar fragments.

The smallest amount of text fragments to be analyzed and categorized is about 300,000 items. The number of items goes easily up to 5,000,000 or even much more. Categorizing all fragments requires about 100 to at maximum 1,000 categories, i.e. there is quite some clustering among those text fragments. On the interface I work with highlight colors: a human can easily realize which fragements are already clustered and which aren't. Usually only a small number of text fragment categories are of special interest. One could compare it to a log file: I don't care so much about how often someone sucessfully authorized; it is much more interesting if someone suddenly fails to authorize.


thanx,


ente

_______________________________________________
gtk-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-list
Reply | Threaded
Open this post in threaded view
|

Re: gtk performance

Gergely Polonkai
Hello,

I don’t know about GtkListBox, but when using GtkTree, are you detaching
your model from the treeview while you are doing mass operations on it?

I remember reading it in a tutorial years (decades?) ago that you should
do this.

Sorry, but i have no answers at hand for your other questions.

Best,
Gergely

On Mon, Oct 08, 2018 at 12:24:03PM +0200, ente wrote:

> Hi,
>
> I am facing performance issues in GTK upon presenting a big amount of
> tabular data. First I used GtkListbox for it's layout flexibility.
> Handling more than 10,000 items gets inacceptable slow. Switching to
> GtkTreeview I can handle some 300,000 items after applying a few
> optimizations (column sizing fixed, fixed height mode): a full load
> takes about 4 seconds, i.e. the application feels anything but
> "responsive" although this is ... ok. Currently I see 2 options:
> dropping Gtk in favor of a HTML frontend (that feels awkward to say) or
> implementing paging on the GtkTreeview.
>
> At this moment I am analyzing the reasons for the performance issues. I
> measured the time it takes to generate a number of labels in a simple
> loop (no call ot GtkMain) and to add those labels to a GtkBox:
> * 100 labels in 1.7 ms
> * 1,000 labels in 20.9 ms
> * 10,000 labels in 610 ms
> * 100,000 labels in 1m18s
> After the loop I pack the GtkBox into a window and show it. That takes
> another century to show up.
> If I adjust the loop to skip the packing of the labels into the box,
> the times are down to:
> * 10,000 labels in 115 ms
> * 100,000 labels in 940ms
>
> The windows shows up instantly (although empty of course).
>
> So my performance issue is related to the packing. In my tests with the
> GtkTreeview it seems speficially to boil down to the sizing and
> signalling of the items: The performance of the treeview was highly
> impacted by the fixed height mode & the fixed column sizing (more than
> ordering by a column or calling GtkMain while adding rows). So my
> question goes down to: How can I further optimize my way of using GTK
> in order to speed up UI? Is there any way to add 1,000,000 labels to a
> GtkBox in less than a second? Could I somehow suspend the signalling
> during UI creation and replace it by a "I am done, now calculate all
> the sizes" signal? Am I wrong about my assumption that the sizing
> signalling is the cause for the low performance?
>
> With regards to the treeview: After initially creating the data rows I
> am calculating values that affect 4 columns out of 10 columns. So far I
> determine the GtkTreeIter of the affected row and update all cells in
> that row using gtk_list_store_set. Should I adjust to update the
> affected columns only? When I implement paging for the TreeView: should
> I rather drop the existing ListStore and create a new or is it faster
> to overwrite all elements in the Liststore with the new items?
>
>
> I tried to use the GtkBuilder in order to setup a box with 100,000
> labels. This performance was somewhat the same as creating the labels
> in a loop (I didn't keep the measurements).
>
> I could isolate the problem to UI rendering. If I don't assign my
> ListStore to the Treeview, all is fine. As soon as my ListStore is
> filled, I assign the model to the treeview. That's the most time
> consuming step.
>
> Last question: is there any way to create callbacks from the javascript
> world using the Webkit2Gtk webview? More specificly: I am working in go
> language. I am aware of the webview widget. Unfortunately that's a
> window. I would prefer to rather place a widget into a native
> GtkApplicationWindow. Using Eval I can inject anything into the DOM.
> How do I get events from the DOM back to go?
>
>
> Background on the application:
> I am developing a text analysis application. Text fragments get
> precalculated attributes assigned. Based on these attributes and the
> Levensthein distance between fragments, someone is supposed to (for
> now: manually) define text fragment categories (categories). Roughly
> speaking the application has a paned window: on the left I show the
> category list or an editor for a new / existing category; on the right
> I show a list of text fragments with the major 3 attributes or a
> details view on a specific fragment: all attributes and a list of
> similar fragments.
>
> The smallest amount of text fragments to be analyzed and categorized is
> about 300,000 items. The number of items goes easily up to 5,000,000 or
> even much more. Categorizing all fragments requires about 100 to at
> maximum 1,000 categories, i.e. there is quite some clustering among
> those text fragments. On the interface I work with highlight colors: a
> human can easily realize which fragements are already clustered and
> which aren't. Usually only a small number of text fragment categories
> are of special interest. One could compare it to a log file: I don't
> care so much about how often someone sucessfully authorized; it is much
> more interesting if someone suddenly fails to authorize.
>
>
> thanx,
>
>
> ente

> _______________________________________________
> gtk-list mailing list
> [hidden email]
> https://mail.gnome.org/mailman/listinfo/gtk-list


--
You must believe in things that are not true.
Otherwise, how will they become?
_______________________________________________
gtk-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-list
Reply | Threaded
Open this post in threaded view
|

Re: gtk performance

Gtk+ - General mailing list
In reply to this post by ente
about this:

> is there any way to create callbacks from the javascript world using the Webkit2Gtk webview?

I've created a JSON communication channel between GJS and the WebView through title notifications:


The nitty-gritty is that you change your title from the page

```js
document.title = `${SECRET}:js=${JSON.stringify(data)}`;
```

and you intercept it from GJS side

```js
webView.connect('notify::title', (self, params) => {
  if (`${SECRET}`:js=/.test(self.title)) {
    const data = JSON.parse(self.title.slice(SECRET.length + 4));
    // do stuff ... then ... 
    const response = {};
    self.run_javascript(
      `document.dispatchEvent(
        new CustomEvent(
          '${SECRET}:gjs',
          {detail: ${JSON.stringify(response)}}
        )
      );`,
      null,
      (self, result, error) => {
        self.run_javascript_finish(result);
      }
    );
  }
});
```

This way you can pass any serializable data to the listener and send eventually data back. The SECRET can be any channel, for circular data just use one of the many circular JSON parsers/stringifiers (i.e. flatted or CircularJSON)

Hope this helps at least with the last part of the question.

P.S. on DOM, a table with 1.000.000 rows would be slow as well, the way I'm scaling these kind of tables on DOM is by rendering only what's visible (fixed rows height helps a lot with this) virtually updating the scroll bar which is **not** the native one or performance will be super bad regardless.



On Mon, Oct 8, 2018 at 12:44 PM ente <[hidden email]> wrote:
Hi,

I am facing performance issues in GTK upon presenting a big amount of tabular data. First I used GtkListbox for it's layout flexibility. Handling more than 10,000 items gets inacceptable slow. Switching to GtkTreeview I can handle some 300,000 items after applying a few optimizations (column sizing fixed, fixed height mode): a full load takes about 4 seconds, i.e. the application feels anything but "responsive" although this is ... ok. Currently I see 2 options: dropping Gtk in favor of a HTML frontend (that feels awkward to say) or implementing paging on the GtkTreeview.

At this moment I am analyzing the reasons for the performance issues. I measured the time it takes to generate a number of labels in a simple loop (no call ot GtkMain) and to add those labels to a GtkBox:
* 100 labels in 1.7 ms
* 1,000 labels in 20.9 ms
* 10,000 labels in 610 ms
* 100,000 labels in 1m18s
After the loop I pack the GtkBox into a window and show it. That takes another century to show up. 
If I adjust the loop to skip the packing of the labels into the box, the times are down to:
* 10,000 labels in 115 ms
* 100,000 labels in 940ms

The windows shows up instantly (although empty of course).

So my performance issue is related to the packing. In my tests with the GtkTreeview it seems speficially to boil down to the sizing and signalling of the items: The performance of the treeview was highly impacted by the fixed height mode & the fixed column sizing (more than ordering by a column or calling GtkMain while adding rows). So my question goes down to: How can I further optimize my way of using GTK in order to speed up UI? Is there any way to add 1,000,000 labels to a GtkBox in less than a second? Could I somehow suspend the signalling during UI creation and replace it by a "I am done, now calculate all the sizes" signal? Am I wrong about my assumption that the sizing signalling is the cause for the low performance?

With regards to the treeview: After initially creating the data rows I am calculating values that affect 4 columns out of 10 columns. So far I determine the GtkTreeIter of the affected row and update all cells in that row using gtk_list_store_set. Should I adjust to update the affected columns only? When I implement paging for the TreeView: should I rather drop the existing ListStore and create a new or is it faster to overwrite all elements in the Liststore with the new items?

I tried to use the GtkBuilder in order to setup a box with 100,000 labels. This performance was somewhat the same as creating the labels in a loop (I didn't keep the measurements).

I could isolate the problem to UI rendering. If I don't assign my ListStore to the Treeview, all is fine. As soon as my ListStore is filled, I assign the model to the treeview. That's the most time consuming step.

Last question: is there any way to create callbacks from the javascript world using the Webkit2Gtk webview? More specificly: I am working in go language. I am aware of the webview widget. Unfortunately that's a window. I would prefer to rather place a widget into a native GtkApplicationWindow. Using Eval I can inject anything into the DOM. How do I get events from the DOM back to go?


Background on the application:
I am developing a text analysis application. Text fragments get precalculated attributes assigned. Based on these attributes and the Levensthein distance between fragments, someone is supposed to (for now: manually) define text fragment categories (categories). Roughly speaking the application has a paned window: on the left I show the category list or an editor for a new / existing category; on the right I show a list of text fragments with the major 3 attributes or a details view on a specific fragment: all attributes and a list of similar fragments.

The smallest amount of text fragments to be analyzed and categorized is about 300,000 items. The number of items goes easily up to 5,000,000 or even much more. Categorizing all fragments requires about 100 to at maximum 1,000 categories, i.e. there is quite some clustering among those text fragments. On the interface I work with highlight colors: a human can easily realize which fragements are already clustered and which aren't. Usually only a small number of text fragment categories are of special interest. One could compare it to a log file: I don't care so much about how often someone sucessfully authorized; it is much more interesting if someone suddenly fails to authorize.


thanx,


ente
_______________________________________________
gtk-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-list

_______________________________________________
gtk-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-list
Reply | Threaded
Open this post in threaded view
|

Re: gtk performance

ente
On Mon, 2018-10-08 at 14:06 +0200, Andrea Giammarchi wrote:
about this:

is there any way to create callbacks from the javascript world using the
Webkit2Gtk webview?

I've created a JSON communication channel between GJS and the WebView
through title notifications:

https://gist.github.com/WebReflection/7ab0addec037508cc8380a9c37d285f2#gistcomment-2245132


The nitty-gritty is that you change your title from the page

```js
document.title = `${SECRET}:js=${JSON.stringify(data)}`;
```

and you intercept it from GJS side

```js
webView.connect('notify::title', (self, params) => {
  if (`${SECRET}`:js=/.test(self.title)) {
    const data = JSON.parse(self.title.slice(SECRET.length + 4));
    // do stuff ... then ...
    const response = {};
    self.run_javascript(
      `document.dispatchEvent(
        new CustomEvent(
          '${SECRET}:gjs',
          {detail: ${JSON.stringify(response)}}
        )
      );`,
      null,
      (self, result, error) => {
        self.run_javascript_finish(result);
      }
    );
  }
});
```

This way you can pass any serializable data to the listener and send
eventually data back. The SECRET can be any channel, for circular data just
use one of the many circular JSON parsers/stringifiers (i.e. flatted or
CircularJSON)

Hope this helps at least with the last part of the question.

P.S. on DOM, a table with 1.000.000 rows would be slow as well, the way I'm
scaling these kind of tables on DOM is by rendering only what's visible
(fixed rows height helps a lot with this) virtually updating the scroll bar
which is **not** the native one or performance will be super bad regardless.



On Mon, Oct 8, 2018 at 12:44 PM ente <
[hidden email]
> wrote:

Hi,

I am facing performance issues in GTK upon presenting a big amount of
tabular data. First I used GtkListbox for it's layout flexibility. Handling
more than 10,000 items gets inacceptable slow. Switching to GtkTreeview I
can handle some 300,000 items after applying a few optimizations (column
sizing fixed, fixed height mode): a full load takes about 4 seconds, i.e.
the application feels anything but "responsive" although this is ... ok.
Currently I see 2 options: dropping Gtk in favor of a HTML frontend (that
feels awkward to say) or implementing paging on the GtkTreeview.

At this moment I am analyzing the reasons for the performance issues. I
measured the time it takes to generate a number of labels in a simple loop
(no call ot GtkMain) and to add those labels to a GtkBox:
* 100 labels in 1.7 ms
* 1,000 labels in 20.9 ms
* 10,000 labels in 610 ms
* 100,000 labels in 1m18s
After the loop I pack the GtkBox into a window and show it. That takes
another century to show up.
If I adjust the loop to skip the packing of the labels into the box, the
times are down to:
* 10,000 labels in 115 ms
* 100,000 labels in 940ms

The windows shows up instantly (although empty of course).

So my performance issue is related to the packing. In my tests with the
GtkTreeview it seems speficially to boil down to the sizing and signalling
of the items: The performance of the treeview was highly impacted by the
fixed height mode & the fixed column sizing (more than ordering by a column
or calling GtkMain while adding rows). So my question goes down to: How can
I further optimize my way of using GTK in order to speed up UI? Is there
any way to add 1,000,000 labels to a GtkBox in less than a second? Could I
somehow suspend the signalling during UI creation and replace it by a "I am
done, now calculate all the sizes" signal? Am I wrong about my assumption
that the sizing signalling is the cause for the low performance?

With regards to the treeview: After initially creating the data rows I am
calculating values that affect 4 columns out of 10 columns. So far I
determine the GtkTreeIter of the affected row and update all cells in that
row using gtk_list_store_set. Should I adjust to update the affected
columns only? When I implement paging for the TreeView: should I rather
drop the existing ListStore and create a new or is it faster to overwrite
all elements in the Liststore with the new items?

I tried to use the GtkBuilder in order to setup a box with 100,000 labels.
This performance was somewhat the same as creating the labels in a loop (I
didn't keep the measurements).

I could isolate the problem to UI rendering. If I don't assign my
ListStore to the Treeview, all is fine. As soon as my ListStore is filled,
I assign the model to the treeview. That's the most time consuming step.

Last question: is there any way to create callbacks from the javascript
world using the Webkit2Gtk webview? More specificly: I am working in go
language. I am aware of the webview widget. Unfortunately that's a window.
I would prefer to rather place a widget into a native GtkApplicationWindow.
Using Eval I can inject anything into the DOM. How do I get events from the
DOM back to go?


Background on the application:
I am developing a text analysis application. Text fragments get
precalculated attributes assigned. Based on these attributes and the
Levensthein distance between fragments, someone is supposed to (for now:
manually) define text fragment categories (categories). Roughly speaking
the application has a paned window: on the left I show the category list or
an editor for a new / existing category; on the right I show a list of text
fragments with the major 3 attributes or a details view on a specific
fragment: all attributes and a list of similar fragments.

The smallest amount of text fragments to be analyzed and categorized is
about 300,000 items. The number of items goes easily up to 5,000,000 or
even much more. Categorizing all fragments requires about 100 to at maximum
1,000 categories, i.e. there is quite some clustering among those text
fragments. On the interface I work with highlight colors: a human can
easily realize which fragements are already clustered and which aren't.
Usually only a small number of text fragment categories are of special
interest. One could compare it to a log file: I don't care so much about
how often someone sucessfully authorized; it is much more interesting if
someone suddenly fails to authorize.


thanx,


ente
_______________________________________________
gtk-list mailing list
[hidden email]

https://mail.gnome.org/mailman/listinfo/gtk-list



A brilliant idea!

thanks,

ente

_______________________________________________
gtk-list mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/gtk-list