gtk regression test suite

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

gtk regression test suite

Jon Dowland-4
I've been working on a small patch for GTK+ which changes a
GtkScrolledWindow property to be a style property (or GtkSetting)[1].
This has been both to `scratch an itch' and to begin learning my way
around GTK+ with a view to future hacking.

There have been two noteable problems so far that I have been able to
diagnose with the help of test cases (one of which is in bugzilla[2],
the rest are pending addition).

Constructing these test cases led me to look at the current GTK testing
framework. I see ./tests/ underneath the source tree, which looks to me
like these are tests to run on an ad-hoc basis to determine whether
things look correct visually.  None of these appear to be regression
tests, i.e. non-interactive with a pass/fail return value. The GTK+
manual describes the gtk-demo program, which looks like it is used by
end-users to test that they've built GTK+ correctly.  (please enlighten
me if I'm mis-representing these).

I was wondering if there was any interest amongst the GTK+ hackers to
investigate creating a regression test suite. I think API changes and
not-API-changes-but-similar (like [1]) could be caught with such a
framework. I'm not experienced enough with the GTK+ internals to know
how appropriate a non-interactive suite would be for the rest of the
toolkit, but an interactive one (e.g. `click ok if the scrollbar is on
the left, cancel otherwise') is also feasible.



[1] http://bugzilla.gnome.org/show_bug.cgi?id=157025
[2] http://bugzilla.gnome.org/attachment.cgi?id=47003&action=view


Yours,

--
Jon Dowland
http://jon.dowland.name/
PGP fingerprint: 7032F238
_______________________________________________
gtk-devel-list mailing list
[hidden email]
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: gtk regression test suite

muppet-6

On Jun 13, 2005, at 4:48 AM, Jon Dowland wrote:

> I've been working on a small patch for GTK+ which changes a
> GtkScrolledWindow property to be a style property (or GtkSetting)[1].
> This has been both to `scratch an itch' and to begin learning my way
> around GTK+ with a view to future hacking.

This is likely to break code that expects that property to be an  
object property.  You never know who is using these things.  Would it  
be reasonable to have the object property be a proxy for the style  
proxy, to retain compatibility?


> I was wondering if there was any interest amongst the GTK+ hackers to
> investigate creating a regression test suite. I think API changes and
> not-API-changes-but-similar (like [1]) could be caught with such a
> framework.

The perl bindings[1] have an extensive API regression test suite[2].  
This suite uses the standard perl test framework with some Gtk2-
related enhancements (auto-init of gtk+, and some helpers to make  
deferred tests cleaner), and exists mainly to verify that we call the  
bound APIs correctly.  That is, we test the validity of the bindings,  
not the correctness of the bound API itself.

That said, we have, with our test suite, found quite a handful of  
bugs in gtk+ itself (memory leaks, missed assertions, corner-case  
crashes, and even functionality regressions).

So, i can vouch for the usefulness of an automated suite, and point  
to an example of how we set one up.

The basic trick is to know what things require a main loop in order  
to function correctly.  Some things just can't be tested simply.  For  
example, drag'n'drop, or just about anything to do with user input,  
will require help from something that plays back events (at least two  
such projects exist, one in gnome cvs, iirc).  Other things can be  
very easily tricked; for every release of gtk+, we have to repair our  
tests for GtkFileChooser, because the behavior changes slightly under  
the hood (this file used to be selected immediately, but now is  
selected after an idle, etc).


[1] http://gtk2-perl.sourceforge.net/
[2] http://cvs.sourceforge.net/viewcvs.py/gtk2-perl/gtk2-perl-xs/Gtk2/t/

--
Holy crap, dude, we have kids!
     -- Elysse, six days after giving birth to twins

_______________________________________________
gtk-devel-list mailing list
[hidden email]
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: gtk regression test suite

Mike Emmel
In reply to this post by Jon Dowland-4
It would be nice if a regression suite allowed use of a high level
language to write the tests  for example python. I  know this adds a
extra layer on one hand but I think it makes it much easier to write
the larget number of test cases needed. Also object oriented methods
in python could make it easy to resuse existing code in  new tests.

Java has aspect/contract oriented programming I think python almost
has the abilitiy by default it a useful methodolgy for testing.

Mike


On 6/13/05, Jon Dowland <[hidden email]> wrote:

> I've been working on a small patch for GTK+ which changes a
> GtkScrolledWindow property to be a style property (or GtkSetting)[1].
> This has been both to `scratch an itch' and to begin learning my way
> around GTK+ with a view to future hacking.
>
> There have been two noteable problems so far that I have been able to
> diagnose with the help of test cases (one of which is in bugzilla[2],
> the rest are pending addition).
>
> Constructing these test cases led me to look at the current GTK testing
> framework. I see ./tests/ underneath the source tree, which looks to me
> like these are tests to run on an ad-hoc basis to determine whether
> things look correct visually.  None of these appear to be regression
> tests, i.e. non-interactive with a pass/fail return value. The GTK+
> manual describes the gtk-demo program, which looks like it is used by
> end-users to test that they've built GTK+ correctly.  (please enlighten
> me if I'm mis-representing these).
>
> I was wondering if there was any interest amongst the GTK+ hackers to
> investigate creating a regression test suite. I think API changes and
> not-API-changes-but-similar (like [1]) could be caught with such a
> framework. I'm not experienced enough with the GTK+ internals to know
> how appropriate a non-interactive suite would be for the rest of the
> toolkit, but an interactive one (e.g. `click ok if the scrollbar is on
> the left, cancel otherwise') is also feasible.
>
>
>
> [1] http://bugzilla.gnome.org/show_bug.cgi?id=157025
> [2] http://bugzilla.gnome.org/attachment.cgi?id=47003&action=view
>
>
> Yours,
>
> --
> Jon Dowland
> http://jon.dowland.name/
> PGP fingerprint: 7032F238
> _______________________________________________
> gtk-devel-list mailing list
> [hidden email]
> http://mail.gnome.org/mailman/listinfo/gtk-devel-list
>
_______________________________________________
gtk-devel-list mailing list
[hidden email]
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: gtk regression test suite

Gustavo J. A. M. Carneiro
On Mon, 2005-06-13 at 10:10 -0400, Mike Emmel wrote:
> It would be nice if a regression suite allowed use of a high level
> language to write the tests  for example python. I  know this adds a
> extra layer on one hand but I think it makes it much easier to write
> the larget number of test cases needed. Also object oriented methods
> in python could make it easy to resuse existing code in  new tests.

  PyGTK, like perl bindings, also already has unit tests.  So just
use/extend them, but let's not add pygtk dependency to gtk+, it has to
be the other way around :)

  Regards.

--
Gustavo J. A. M. Carneiro
<[hidden email]> <[hidden email]>
The universe is always one step beyond logic.

_______________________________________________
gtk-devel-list mailing list
[hidden email]
http://mail.gnome.org/mailman/listinfo/gtk-devel-list

smime.p7s (4K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: gtk regression test suite

Tor Lillqvist
Hmm, would LDTP (the Linux Desktop Testing Project) be of any
relevance to this discussion? Check
http://gnomebangalore.org/ldtp/index.php/Main_Page .

--tml

_______________________________________________
gtk-devel-list mailing list
[hidden email]
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: gtk regression test suite

Mike Emmel
In reply to this post by Gustavo J. A. M. Carneiro
On 6/13/05, Gustavo J. A. M. Carneiro <[hidden email]> wrote:

> On Mon, 2005-06-13 at 10:10 -0400, Mike Emmel wrote:
> > It would be nice if a regression suite allowed use of a high level
> > language to write the tests  for example python. I  know this adds a
> > extra layer on one hand but I think it makes it much easier to write
> > the larget number of test cases needed. Also object oriented methods
> > in python could make it easy to resuse existing code in  new tests.
>
>   PyGTK, like perl bindings, also already has unit tests.  So just
> use/extend them, but let's not add pygtk dependency to gtk+, it has to
> be the other way around :)
>
>   Regards.
>

It does not have to be especially in software :)
One of the primary purposes of gtk is to support high level language
bindings by using one of them extensivily in the test suite this
binding ability is also tested.  The test framework should be its own
project thats a peer with gtk so its dependencies are different. They
rapidly grow beyond a simple directory of test cases. I suspect with a
little thought the language binding tools used to generate the binding
i.e the defs files could be used to generate a skelton test cases. If
assertion defs were added then they could even generate real cases.
Python is easier the generate but you could do C code also.

Generally my point is look at automating as much as possible moving to
a dynamic language like python is a step towards highly automated
testing. My experience with programmers and testing is that by giving
them a easy to use automated framework you increase the probability of
a useful test case being written by like 90%.


> --
> Gustavo J. A. M. Carneiro
> <[hidden email]> <[hidden email]>
> The universe is always one step beyond logic.
>
>
>
_______________________________________________
gtk-devel-list mailing list
[hidden email]
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: gtk regression test suite

Stefan Sauer-4
In reply to this post by Tor Lillqvist
Tor Lillqvist wrote:
> Hmm, would LDTP (the Linux Desktop Testing Project) be of any
> relevance to this discussion? Check
> http://gnomebangalore.org/ldtp/index.php/Main_Page .
>
> --tml
>
or even better using Gerd (see gnome CVS) which Sven Herzberg has brought
up-to-date just now. Gerd is a gtk+ event recorder originally developed by tj.

Stefan
_______________________________________________
gtk-devel-list mailing list
[hidden email]
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: gtk regression test suite

Jon Dowland-4
In reply to this post by muppet-6
On Mon, Jun 13, 2005 at 08:02:40AM -0400, muppet wrote:
 
> This is likely to break code that expects that property to be an  
> object property.  You never know who is using these things.  

This is absolutely right - Federico raised this concern, and the first
test I talked about was an attempt to prove this.

> Would it  be reasonable to have the object property be a proxy for the
> style  proxy, to retain compatibility?

I think so - the current revision of my patch installs it as both a
g_object property and a GtkSetting. In honesty, I don't know what 'be a
proxy for the style property' means!

[Note that all revisions of my patch that use GtkSetting are broken in a
fundamental way - previous revisions which installed it as a gtk style
property appeared to work. I've yet to figure out why]

> The perl bindings[1] have an extensive API regression test suite[2].  
> This suite uses the standard perl test framework with some Gtk2-
> related enhancements (auto-init of gtk+, and some helpers to make  
> deferred tests cleaner), and exists mainly to verify that we call the  
> bound APIs correctly.  That is, we test the validity of the bindings,  
> not the correctness of the bound API itself.
> ...
> So, i can vouch for the usefulness of an automated suite, and point  
> to an example of how we set one up.

I have heard good things about perl's testing framework. I've never used
it from a developer end but I have had several pieces of software fail
to install as a result of their regression tests failing :-) I think it
would be an excellent idea to study their framework.

--
Jon Dowland
http://jon.dowland.name/
PGP fingerprint: 7032F238
_______________________________________________
gtk-devel-list mailing list
[hidden email]
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: gtk regression test suite

Jon Dowland-4
In reply to this post by Mike Emmel
On Mon, Jun 13, 2005 at 10:47:39AM -0400, Mike Emmel wrote:
> One of the primary purposes of gtk is to support high level language
> bindings by using one of them extensivily in the test suite this
> binding ability is also tested.  The test framework should be its own
> project thats a peer with gtk so its dependencies are different.

Good point: not everyone who will be building GTK+ would be interested
in a regression test suite, so they shouldn't be saddled with it (or
it's dependencies).

--
Jon Dowland
http://jon.dowland.name/
PGP fingerprint: 7032F238
_______________________________________________
gtk-devel-list mailing list
[hidden email]
http://mail.gnome.org/mailman/listinfo/gtk-devel-list
Reply | Threaded
Open this post in threaded view
|

Re: gtk regression test suite

Federico Mena Quintero
In reply to this post by Jon Dowland-4
On Mon, 2005-06-13 at 09:48 +0100, Jon Dowland wrote:

> I was wondering if there was any interest amongst the GTK+ hackers to
> investigate creating a regression test suite.

The right way to do this is through LDTP:

http://gnomebangalore.org/ldtp/index.php/Main_Page

The testing scripts are written in Python.  At the low levels of the
framework, they communicate with apps via the accessibility interfaces.
So this immediately lets you test two things:

- that the a11y interfaces are sufficient to operate the user interface

- that the interface is doing the right thing

The interesting part, of course, is actually writing the testing
scripts.  For GTK+'s purposes, it would probably be good to start by
testing high-level functionality (e.g. click on a treeview column
header, extract the contents of the rows, and see that they are indeed
sorted).  The low-level functionality can be tested later (e.g. we are
pretty certain that clicking a button will emit the "clicked" signal).

  Federico

_______________________________________________
gtk-devel-list mailing list
[hidden email]
http://mail.gnome.org/mailman/listinfo/gtk-devel-list