tests/ and feature tests reorganized

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

tests/ and feature tests reorganized

Tim Janik-6
Hey Stefan,

I've just moved tests/ to use a single non-recursive makefile.
In the process, all audio feature tests got moved into af-tests/
also using a non-recursive makefile.

This means all test from tests/ and af-tests/ are now executed
in parallel, plus the resampler tests also became more
parallelized by splitting up the test rules.

Since the tests run a lot quicker on multi-core computers now
(I could save more than 5/10 seconds for test runs here), I've
added the af-tests back into 'make check' with a few exceptions:

1) Partymonster is not enabled by default. Though it passes,
  it blocks the CPU for a solid 7 seconds on my laptop, while
  all other tests complete in half the time. I'd love to add
  it in though, but it needs to be faster, what about cutting
  its rendering to a few seconds? Maybe 60 seconds or so
  into the song?

2) A bunch of tests are never completing now, they keep rendering
   into the WAV file forever, filling up the disc. The culprits
   are commented out for now:
# FIXME: check-af-tests: af-tests/artscompressor
# FIXME: check-af-tests: af-tests/minisong
# FIXME: check-af-tests: af-tests/organsong
# FIXME: check-af-tests: af-tests/simple-loop

Not sure I'll get around to fixing this soon,
but it definitley must be addressed.

Also, it could really help to fix up resamplertest and bsefcompare
to *only* put out '  PASS       blah short text blah' unless -v is
given or a test is failing. That way the mixed-line output of
parallel test runs becomes parsable again.

I.e. test should in general not produce more than a single
line under normal conditions to play nicely with other
tests running parallel.

Btw, the slow suites of resampler tests are still available to
be run manually with:

        make tests-testresampler-check-perf # passes fine

        make tests-testresampler-check-all


The later fails here, with:

tests/testresampler accuracy --up --precision=24 --freq-scan=50,18000,50 --max-threshold=126.5  # ideally: 144dB
# accuracy test for factor 2 upsampling using SSE instructions
#   input frequency range used [ 50.00 Hz, 18000.00 Hz ] (SR = 44100.0 Hz, freq increment = 50.00)
#   max difference between correct and computed output: 0.000000 = -126.489977 dB
#                             (threshold given by user: -126.500000 dB)
tests/testresampler.cc:523: assertion failed: max_diff_db < options.max_threshold_db

Not sure how bad this is, input is appreciated.


--
Yours sincerely,
Tim Janik

https://testbit.eu/timj/
Free software author.
_______________________________________________
beast mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/beast
Reply | Threaded
Open this post in threaded view
|

Re: tests/ and feature tests reorganized

Stefan Westerfeld
    Hi!

Let me reply only partially,

On Mon, Oct 16, 2017 at 01:10:13AM +0200, Tim Janik wrote:

> Btw, the slow suites of resampler tests are still available to
> be run manually with:
>
> make tests-testresampler-check-perf # passes fine
>
> make tests-testresampler-check-all
>
>
> The later fails here, with:
>
> tests/testresampler accuracy --up --precision=24 --freq-scan=50,18000,50 --max-threshold=126.5  # ideally: 144dB
> # accuracy test for factor 2 upsampling using SSE instructions
> #   input frequency range used [ 50.00 Hz, 18000.00 Hz ] (SR = 44100.0 Hz, freq increment = 50.00)
> #   max difference between correct and computed output: 0.000000 = -126.489977 dB
> #                             (threshold given by user: -126.500000 dB)
> tests/testresampler.cc:523: assertion failed: max_diff_db < options.max_threshold_db
>
> Not sure how bad this is, input is appreciated.
The threshold problem is not bad, I looked at the actual output. This can be
easily fixed by a minimal change to the threshold option. Fix is in my branch.

However we have a bigger problem, which I found while looking at the issue:

Both, the FPU and SSE variant of the resampler code need to be tested
(depending on the testresampler --fpu option). From looking at the source code
testresampler tries to enforce this by passing --bse-force-fpu bse_init_test()
if the FPU variant is to be tested.  However, currently this doesn't have the
desired effect. So we always test SSE (and never test FPU), which is bad.

Option #1: make --bse-force-fpu do what it used to do.

Option #2: I looked at what bse/tests/blocktests.cc does. Apparently the FPU is
used if we init without Bse::cstrings_to_vector ("load-core-plugins=1", NULL)

I've committed a complete solution based on that on the resampler-test-fix
branch of my beast repo. If you want to go that way, I'd recommend removing
the bse-force-fpu option completely.

https://github.com/swesterfeld/beast/tree/resampler-test-fix

   Cu... Stefan
--
Stefan Westerfeld, http://space.twc.de/~stefan

_______________________________________________
beast mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/beast

res-sse-fpu.diff (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: tests/ and feature tests reorganized

Stefan Westerfeld
In reply to this post by Tim Janik-6
   Hi!

On Mon, Oct 16, 2017 at 01:10:13AM +0200, Tim Janik wrote:

> I've just moved tests/ to use a single non-recursive makefile.
> In the process, all audio feature tests got moved into af-tests/
> also using a non-recursive makefile.
>
> This means all test from tests/ and af-tests/ are now executed
> in parallel, plus the resampler tests also became more
> parallelized by splitting up the test rules.
>
> Since the tests run a lot quicker on multi-core computers now
> (I could save more than 5/10 seconds for test runs here), I've
> added the af-tests back into 'make check' with a few exceptions:
>
> 1) Partymonster is not enabled by default. Though it passes,
>   it blocks the CPU for a solid 7 seconds on my laptop, while
>   all other tests complete in half the time. I'd love to add
>   it in though, but it needs to be faster, what about cutting
>   its rendering to a few seconds? Maybe 60 seconds or so
>   into the song?

Yes, maybe it should be a full test if you call make slowcheck (do we still
have a slow variant of make check?) or whatever, and check the first minute
otherwise, which sounds like a reasonable thing to do.

> 2) A bunch of tests are never completing now, they keep rendering
>    into the WAV file forever, filling up the disc. The culprits
>    are commented out for now:
> # FIXME: check-af-tests: af-tests/artscompressor
> # FIXME: check-af-tests: af-tests/minisong
> # FIXME: check-af-tests: af-tests/organsong
> # FIXME: check-af-tests: af-tests/simple-loop
>
> Not sure I'll get around to fixing this soon,
> but it definitley must be addressed.

Definitely.

> Also, it could really help to fix up resamplertest and bsefcompare
> to *only* put out '  PASS       blah short text blah' unless -v is
> given or a test is failing. That way the mixed-line output of
> parallel test runs becomes parsable again.

Ok, I found some time to implement one line test output, the merge
request for that is

  https://github.com/tim-janik/beast/pull/23

   Cu... Stefan
--
Stefan Westerfeld, http://space.twc.de/~stefan
_______________________________________________
beast mailing list
[hidden email]
https://mail.gnome.org/mailman/listinfo/beast