JUnitBenchmarks

Run method before each @Test

Details

  • Type: Improvement Improvement
  • Status: Resolved Resolved
  • Priority: Trivial Trivial
  • Resolution: Won't Fix
  • Affects Version/s: 0.3.0
  • Fix Version/s: 0.4.0
  • Component/s: Core

Description

@BeforeClass methods only run once; @Before methods will run on every round. There should be another level for methods that run before each @Test benchmark.

I have created a @BeforeBenchmark annotation that does just this. Attached is a patch that includes code to modify BenchmarkStatement.evaluate() so that the @BeforeBenchmark methods are run at the correct place. This patch also forces GC after calling the benchmark befores and at the end of the evaluate() since I think that was something that was missing. Perhaps this should be another BenchmarkOption to call GC between tests? I can open a separate defect if desired.

Note that similar code could be added for an @AfterBenchmark annotation. I can submit a patch for that if needed too. Not sure if that's as useful. I didn't need it in my usecase but maybe others do.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Dawid Weiss added a comment -

Thanks for your contribution, but I'd rather not put it in. Here is my motivation:

  • I think a single benchmarks pass should be as simple as possible, without any additional surrounding code. This code can interfere with the benchmark by allocating memory or even by influencing the jit so that it takes different optimization paths.
  • The GC is called before every round (in BaseEvaluator#evaluateInternally) so that measurements can be taken after a round is complete. It doesn't make sense to invoke it after or from other places that are not part of the measurement. GC measurement is not that reliable either – it could be improved by observing major GC counters via JMX, but I don't think it's worth the effort, honestly.
  • I just wouldn't want too many features to creep in, especially in the light of potential refactorings.

Your code is fine, don't get me wrong. It is a biased opinion not to commit it in. If you desperately need a before/ after benchmark code then your patch would be fine and it can be applied separately. Feel free to fork the code on github if you wish.

Show
Dawid Weiss added a comment - Thanks for your contribution, but I'd rather not put it in. Here is my motivation:
  • I think a single benchmarks pass should be as simple as possible, without any additional surrounding code. This code can interfere with the benchmark by allocating memory or even by influencing the jit so that it takes different optimization paths.
  • The GC is called before every round (in BaseEvaluator#evaluateInternally) so that measurements can be taken after a round is complete. It doesn't make sense to invoke it after or from other places that are not part of the measurement. GC measurement is not that reliable either – it could be improved by observing major GC counters via JMX, but I don't think it's worth the effort, honestly.
  • I just wouldn't want too many features to creep in, especially in the light of potential refactorings.
Your code is fine, don't get me wrong. It is a biased opinion not to commit it in. If you desperately need a before/ after benchmark code then your patch would be fine and it can be applied separately. Feel free to fork the code on github if you wish.
Hide
Permalink
Dawid Weiss added a comment -

Feature creep.

Show
Dawid Weiss added a comment - Feature creep.
Hide
Permalink
AngerClown added a comment -

I can agree with that. It's mostly a design decision on how you want to structure tests, though with benchmarking, it may make a difference what you do when.

FYI for anyone else who finds this issue, what I was trying to do was have a single class that did:
Test 1 - config 1, option 1
Test 2 - config 1, option 2
Test 3 - config 2, option 1
...

I was using the TestName JUnit rule and using the test name to determine which config to create. I didn't need to do that on each round, but couldn't do it at the class level either. Arguably, there's a 'code smell' to this setup. I was able to avoid it, and avoid the need for this feature by creating:

Class 1, setup config 1 with @BeforeClass
Test 1 - option 1
Test 2 - option 2

Class 2, setup config 2 with @BeforeClass
Test 1 - option 1
...

It's a little more tedious to setup, but probably cleaner especially when the common stuff is factored into a base class. It also means I need to run all the Tests in a package rather than a single class, but that's not a big deal.

Show
AngerClown added a comment - I can agree with that. It's mostly a design decision on how you want to structure tests, though with benchmarking, it may make a difference what you do when. FYI for anyone else who finds this issue, what I was trying to do was have a single class that did: Test 1 - config 1, option 1 Test 2 - config 1, option 2 Test 3 - config 2, option 1 ... I was using the TestName JUnit rule and using the test name to determine which config to create. I didn't need to do that on each round, but couldn't do it at the class level either. Arguably, there's a 'code smell' to this setup. I was able to avoid it, and avoid the need for this feature by creating: Class 1, setup config 1 with @BeforeClass Test 1 - option 1 Test 2 - option 2 Class 2, setup config 2 with @BeforeClass Test 1 - option 1 ... It's a little more tedious to setup, but probably cleaner especially when the common stuff is factored into a base class. It also means I need to run all the Tests in a package rather than a single class, but that's not a big deal.
Hide
Permalink
Dawid Weiss added a comment -

With this description I think it would be very interesting to experiment with a fusion of JUnitBenchmarks and RandomizedRunner, which supports parameterized tests (and shows their parameters in a nice way). RandomizedRunner is still in early development, in particular @Rule support is something I haven't done intense testing of, but feel free to try and provide feedback. An example of a parameterized test is here as a JUnit test case:

https://github.com/carrotsearch/randomizedtesting/blob/master/runner/src/test/java/com/carrotsearch/randomizedtesting/TestParameterized.java

You don't need to use randomization stuff, but you'll have to @RunWith(RandomizedRunner.class). Then, you should be able to both have parameterized tests in a single class and benchmark them accordingly. Let us know if you've tried and if it worked or not.

Show
Dawid Weiss added a comment - With this description I think it would be very interesting to experiment with a fusion of JUnitBenchmarks and RandomizedRunner, which supports parameterized tests (and shows their parameters in a nice way). RandomizedRunner is still in early development, in particular @Rule support is something I haven't done intense testing of, but feel free to try and provide feedback. An example of a parameterized test is here as a JUnit test case: https://github.com/carrotsearch/randomizedtesting/blob/master/runner/src/test/java/com/carrotsearch/randomizedtesting/TestParameterized.java You don't need to use randomization stuff, but you'll have to @RunWith(RandomizedRunner.class). Then, you should be able to both have parameterized tests in a single class and benchmark them accordingly. Let us know if you've tried and if it worked or not.

People

Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved:

Time Tracking

Estimated:
4h
Original Estimate - 4h
Remaining:
4h
Remaining Estimate - 4h
Logged:
Not Specified
Time Spent - Not Specified
Atlassian JIRA (v4.4.3#663-r165197) | Report a problem
Powered by a free Atlassian JIRA open source license for Carrot2. Try JIRA - bug tracking software for your team.