1.5.0

1.5.0 (2025-10-21)

Finally max_sample_size is here to alleviate a variety of woes resulting from gathering too many samples!
Also finally, Benchee lets you exclude outliers!

Features (User Facing)

• Introduce max_sample_size which guides how many samples will be gathered at most for a given scenario.
  This avoids a variety of issues when scenarios gather too many samples (memory consumption, statistics taking long to calculate, formatters hanging/not working).
  Defaults to 1_000_000, setting it to nil gathers unlimited samples again (behavior before this version).
• Introduce exclude_outliers option which when set to true will automatically exclude outliers from the samples gathered.
  Especially important for run time, you can remove samples caused by garbage collection or external factors.
  Defaults to false.
  Shout out to @NickNeck who implemented this long wished for feature over in Statistex.
• Display input_name entries in Livebook/Table.Reader protocol. Thanks @madlep!

Bugfixes (User Facing)

• fixed a bug where if times were supplied as 0 instead of 0.0 we'd sometimes gather a single measurement
• elixir 1.19 compilation warnings have been fixed

Features (Plugins)

• The %Benchee.Statistics{} struct now comes with values to accompany the outlier exclusion feature:
  • outliers - if outlier exclusion was enabled, may include any samples of outliers that were found, empty list otherwise
  • lower_outlier_bound - value below which values are considered an outlier
  • upper_outlier_bound - value above which values are considered an outlier

1.4.0

Some nice features (pre_check: :all_same is cool) along with adding support for some new stuff (tprof) and fixing some bugs.

Features (User Facing)

• pre_check now accepts the option :all_same - which raises if any job returns a value different from another for the same input. Useful when benchmarking multiple variants of the same function. Thanks to @sabiwara!
• Enable the tprof profiler that shipped with OTP 27, requires OTP 27+ and elixir 1.17+.

Bugfixes (User Facing)

• Make gathering of system data more resilient, if the commands we ran didn't exist it may have crashed previously. Most notably, occurring on Windows 11 due to te removal of the tool we use to gather processor information. Now "N/A" is returned.
• An incompatibility with the upcoming OTP 28 that'd emit a warning on newer elixir versions was removed. Thanks TBK145!

1.3.1

Small release to remove warnings for the upcoming elixir 1.17 as well as add proper solaris support.

Features (User Facing)

• Solaris systems will now correctly be identified and have their CPU and memory parsed. Thanks to @brianewell!

Bugfixes (User Facing)

• warnings on elixir-main/1.17-rc have been removed. Thanks to @mhanberg & @alco!

1.3.0

A big swath of bug fixes and improvements. The highlights certainly are fixes and conveniences around saved benchmarks and loading them again via the new Benchee.report/1. The other big one is saving a lot of memory (and time!) when processing big inputs. Sadly the latter comes with some breaking changes for plugins, but they are well justified and shouldn't actually affect any plugin in practice.

There's also a known issue for elixir 1.14.0 to 1.16.0-rc.0 - read up about it.

Features (User Facing)

• System information now includes whether or not the JIT is enabled (erlang docs).
• Benchee now let's you know when it's calculating statistics or running the formatters. Helps when you wonder what's taking so long or blows up memory.
• Benchee.report/1 introduced if you just want to load saved benchmarks and report on them (---> run formatters).
• Configuration times will now be displayed in a more human friendly format (1 min 12 s vs. 1.2 min). Thanks to @drobnyd.

Bugfixes (User Facing)

• Memory usage should be massively reduced when dealing with larger sets of data in inputs or benchmarking functions. They were needlessly sent to processes calculating statistics or formatters which could lead to memory blowing up.
• Similarly, inputs and benchmarking functions will no longer be saved when using the :save option, this makes it immensely faster and depending on the size of the data a lot slower (I have an example with a factor 200x for the size). The side effect of this is that you also can't use :load and run the benchmarks saved again from just the file, this was never an intended use case though (as loading happens after benchmarking by default). You also still should have the benchmarking script so it's also not needed.
• Fix a bug where relative statistics would always rely on the inputs provided in the config, which can break when you load saved benchmarks but don't specify the inputs again.

Breaking Changes (Plugins)

Woopsie, didn't wanna do any of these in 1.x, sorry but there's good reason :( However, counting them as bugs.

• Formatters have lost access to benchmarking functions and the inputs, this is to enable huge memory and run time savings when using a lot of data. I also believe they should not be needed for formatters, please get in touch if this is a problem so we can work it out. In detail this means:
  • Each Benchee.Scenario struct in a formatter will have :function and :input set to nil
  • The inputs list in Configuration retains the input names, but the values will be set to :scrubbed_see_1_3_0_changelog. It may be completely scrubbed in the future, use the newly introduced input_names instead if you need easy access to all the input names at once.
  • Technically speaking formatters haven't generally lost access, only if they are processed in parallel - so not if it's the only formatter or if it's used via a function (formatters: [fn suite -> MyFormatter.output(suite) end]. Still, should not be used or relied upon.

Features (Plugins)

• jit_enabled? is exposed as part of the suite.system struct
• Yes, Benchee.System is now a struct so feel easier about relying on the fields
• Configuration now has an input_names key that holds the name of all inputs, for the reasoning, see above.
• The more human friendly formats are accessible via Format.format_human/2 or Duration.format_human/1 (& friends)

1.2.0 (2023-11-09)

Long time, huh? 😉 I'm not gonna repeat all that, but I'm happy benchee is in a place where it just works and doesn't need too much support. Biggest feature here is the implementation of the Table.Reader protocol for a better Livebook experience.

Features (User Facing)

• Trying to benchmark evaluated functions will now result in a warning. Thanks @BrooklinJazz, @czrpb, @aar2dee2,@ReecesPeanutButterCodes.
• Add support for the Table.Reader protocol so that benchee works out of the box in Livebook. See #369, big shoutout to @akoutmos and sorry for keeping it unreleased for so long.

Bugfixes (User Facing)

• Removed Elixir 1.16 compiler warnings around +0.0 and -0.0. Thanks @tomciop.
• Building an escript and running it is fixed. See #384 and thanks @Munksgaard!

1.1.0

Long time, huh? I'm sorry, combination of priorities, difficult to fix bugs, stress and arm problems kept me away too long.
This release brings major features long developed and now finally released (reduction measurements + profiler after run), along with a critical bugfix around measurement accuracy for very fast functions (nanoseconds).

Features (User Facing)

• Reduction counting/measurements was implemented. Basically, it's a rather stable unit of execution implemented by the BEAM that measures in more abstract manner how much work was done. It's helpful, as it shouldn't be affected by load on the system. Check out the docs.
• You can now dive straight into profiling from your benchmarks by using the profiling feature. See the docs - thanks @pablocostass

Default Behavior Changes (User Facing)

• measuring function call overhead is now turned off by default, as it only helps with extreme nano benchmarks and has some potential for causing wrong results, so it should only be opt in. We now also print out the measured overhead for vsibility.

Bugfixes (User Facing)

• Benchee now correctly looks for the time resolution as reported by :erlang.system_info(:os_monotonic_time_source) to accomodate when determining if a measurement is "precise" enough. Benche also works around an erlang bug we discovered present in erlang <= 22.2. Issue for reference.
• The annoying stacktrace warning has been removed - thanks @mad42

Noteworthy

• a new dependency statistex will show up - it's a part of continued efforts to extract reusable libraries from Benchee.

1.0.1

Bugfixes (User Facing)

• When memory measurements were actually different extended statistics was displayed although the option was not provided. Now correctly only displayed if the option is provided and values actually had variance.

1.0.0

It's 0.99.0 without the deprecation warnings. Specifically:

• Old way of passing formatters (:formatter_options) vs. new :formatters with modules, tuples or functions with one arg
• The configuration needs to be passed as the second argument to Benchee.run/2
• Benchee.collect/1 replaces Benchee.measure/1
• unit_scaling is a top level configuration option, not for the console formatter
• the warning for memory measurements not working on OTP <= 18 will also be dropped (we already officially dropped OTP 18 support in 0.14.0)

We're aiming to follow Semantic Versioning as we go forward. That means formatters should be safe to use ~> 1.0 (or even >= 0.99.0 and < 2.0.0).

0.99.0

The "we're almost 1.0!" release - all the last small features, a bag of polish and deprecation warnings. If you run this release succesfully without deprecation warnings you should be safe to upgrade to 1.0.0, if not - it's a bug :)

Breaking Changes (User Facing)

• changed official Elixir compatibility to ~> 1.6, 1.4+ should still work but aren't guaranteed or tested against.

Features (User Facing)

• the console comparison now also displays the absolute difference in the average (like +12 ms) so that you have an idea to how much time that translates to in your applications not just that it's 100x faster
• Overhaul of README, documentation, update samples etc. - a whole lot of things have also been marked @doc false as they're considered internal

Bugfixes (User Facing)

• Remove double empty line after configuration display
• Fix some wrong type specs

Breaking Changes (Plugins)

• Scenario made it to the big leagues, it's no longer Benchee.Benchmark.Scenario but Benchee.Scenario - as it is arguably one of our most important data structures.
• The Scenario struct had some keys changed (last time before 2.0 I promise!) - instead of :run_times/:run_time_statistics you now have one run_time_data key that contains Benchee.CollectionData which has the keys :samples and :statistics. Same for memory_usage. This was done to be able to handle different kinds of measurements more uniformly as we will add more of them.

Features (Plugins)

• Benchee.Statistics comes with 3 new values: :relative_more, :relative_less, :absolute_difference so that you don't have to calculate these relative values yourself :)

0.14.0

Breaking Changes (User Facing)

• dropped support for Erlang 18.x
• Formatters no longer have an output/1 method, instead use Formatter.output/3 please
• Usage of formatter_options is deprecated, instead please use the new tuple way

Features (User Facing)

• benchee now uses the maximum precision available to measure which on Linux and OSX is nanoseonds instead of microseconds. Somewhat surprisingly :timer.tc/1 always cut down to microseconds although better precision is available.
• The preferred way to specify formatters and their options is to specify them as a tuple {module, options} instead of using formatter_options.
• New Formatter.output/1 function that takes a suite and uses all configured formatters to output their results
• Add the concept of a benchmarking title that formatters can pick up
• the displayed percentiles can now be adjusted
• inputs option can now be an ordered list of tuples, this way you can determine their order
• support FreeBSD properly (system metrics) - thanks @kimshrier

Bugfixes (User Facing)

• Remove extra double quotes in operating system report line - thanks @kimshrier

Breaking Changes (Plugins)

• all reported times are now in nanoseconds instead of microseconds
• formatter methods format and write now take 2 arguments each where the additional arguments is the options specified for this formatter so that you have direct access to it without peeling it from the suite
• You can no longer use Benchee.Formatter - just adopt the behaviour (no more auto generated output/1 method, but Formatter.output/3 takes that responsibility now)

Features (Plugins)

• An optional title is now available in the suite for you to display
• Scenarios are now sorted already sorted (first by run time, then memory usage) - no need to sort them yourself!
• Add Scenario.data_processed?/2 to check if either run time or memory data has had statistics generated