How to implement performance regression tests?

General Usage Performance
1

I added a performance test to my package KiteModels.jl: Benchmark simplify operation by ufechner7 · Pull Request #250 · OpenSourceAWE/KiteModels.jl · GitHub.

If you run the script, you get an output like:

julia> include("test/bench_simplify.jl")
[ Info: Simplifying the system
[ Info: Simplify took 37.283457024 seconds

I want a unit test that fails in case of a performance regression of a commit of 20% or more. How can I achieve that?

The absolute time depends on:

  • the computer it runs on
  • the load on the computer
  • If it runs on a laptop, it depends on the question of whether it is connected to power or running on battery

etc

How can I catch regressions independent of the hardware on which the code is running?

Ideas:

  • Create a dictionary with reference values per computer
  • Disable the test if the computer is running on battery (how can I detect that?)
  • Scale the time with the clock frequency of the core on which the main thread is running (how can I detect that?)
  • Disable the test if the system load is too high (how can I detect that?)
2

I once did something like this

cpu_model_raw = getproperty(first(Sys.cpu_info()), :model)
cpu_model = lowercase(replace(cpu_model_raw, r"\(.*?\)" => "", ' '=>'-'))
cpu_name = lowercase(replace(Sys.CPU_NAME, '_'=>'-'))
reference_benchmark = "benchmark/profile_$(cpu_name)_$cpu_model.json"

it won’t get the battery vs power stuff, or system load, but at least let me partition by cpu

1 Like
3

Take a look into AirSpeedVelocity.jl. You can adapt the benchmarks and GitHub Action we have in Meshes.jl for example. We did setup the action so that every PR triggers benchmark results as comments:

3 Likes
4

But how do you handle the fact that the results are hardware dependant?

5

The benchmarks are run on the same remote machines. But I would double check the exact infrastructure that AirSpeedVelocity.jl provides.

1 Like
6

A simple way I think is to benchmark against a reference benchmark which contains simple code which does not change. Like some king of stupid loop.