Skip to content

Add a HistFactory-style likelihood benchmark generator - #2002

Open
guitargeek wants to merge 1 commit into
vgvassilev:masterfrom
guitargeek:histfactory-benchmark
Open

Add a HistFactory-style likelihood benchmark generator#2002
guitargeek wants to merge 1 commit into
vgvassilev:masterfrom
guitargeek:histfactory-benchmark

Conversation

@guitargeek

@guitargeek guitargeek commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator

benchmark/histfactory.py generates a scaled-down proxy of the code RooFit emits for a real 1470-parameter ATLAS likelihood, compiles it with the clad plugin, and runs it, printing the compilation time in the same table format as the benchmark executable's primal/gradient/hessian timings. The .cpp is a transient artifact in a temp directory (kept with --cpp); the compiler is read from the clad build tree's CMakeCache, so histfactory.py --clad-build <dir> is all that is needed to compare clad revisions.

The generated model reproduces the statement patterns of the full likelihood at 1/16 scale: 91 parameters, 4 channels with straight-line per-sample yields (flexibleInterp over nuisance subsets, product factors, a lumi parameter), bin loops with uniformBinNumber shape lookups feeding binned nll terms, and a constraint block (gaussian/gaussianIntegral and poisson/poissonIntegral ratios into constraintSum). The driver validates the gradient against central differences of the primal and sampled hessian rows against central differences of the gradient. Measured against the full benchmark, the mini preserves the relative cost profile (gradient ~7x primal vs 12x; hessian-vector product ~2.2x gradient vs 2.9x) while compiling in seconds instead of minutes.

benchmark/HistFactoryMathFuncs.h carries the RooFit::Detail::MathFuncs subset the model calls, with TMath/ROOT::Math replaced by std:: equivalents, the Cephes incomplete gamma behind poissonIntegral, and its custom clad derivatives copied verbatim from ROOT's Math/CladDerivator.h (kept verbatim on purpose, including the Cephes do-while loops and the ROOT formatting, which clang-format off/on markers protect so the file stays diffable against the originals).
constraintSum, the one function carrying #pragma clad checkpoint loop, is emitted into the generated main file instead: clad currently mis-attributes a checkpoint pragma living in an included header, because the planner selects pragmas by raw SourceLocation order, which does not match translation-unit order across files.

The benchmark registers as the CTest test clad-HistFactory (labels "benchmark;short", RUN_SERIAL) under CLAD_ENABLE_BENCHMARKS. Because one of its metrics is the compilation time of the generated code, it compiles at test time via the script instead of going through CB_ADD_GBENCHMARK; CMake passes the tested clang explicitly through the new --compiler option, since the CMakeCache fallback would pick the host compiler on builds configured with gcc.

Output in the CI looks as follows:

[7](https://github.com/vgvassilev/clad/actions/runs/32480829289/job/96766612084?pr=2002#step:30:628)
      Start  6: clad-HistFactory
6: Test command: /usr/bin/python3.12 "/home/runner/work/clad/clad/benchmark/histfactory.py" "--clad-build" "/home/runner/work/clad/clad/obj" "--compiler" "/usr/lib/llvm-20/bin/clang++"
6: Working Directory: /home/runner/work/clad/clad/obj/benchmark
6: Test timeout computed to be: 1200
6: clad build    : /home/runner/work/clad/clad/obj
6: compilation   :     22.867 s
6: n params      : 91
6: function value: 4413.41174684329
6: primal        :      0.023 ms/call
6: gradient      :      0.184 ms/call  (8.1x primal)
6: hessian       :     37.619 ms/call  (204.9x gradient)
6: gradient vs numerical: worst rel. deviation 1.15e-06 (at index 10), 0 outside tolerance
6: hessian vs numerical (8 sampled rows): worst rel. deviation 5.62e-10 (at [18,39]), 0 outside tolerance
6: hessian asymmetry: worst rel. deviation 9.37e-16
6: validation passed
 6/11 Test  #6: clad-HistFactory .................   Passed   23.84 sec

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

There were too many comments to post at once. Showing the first 10 out of 262. Check the log or trigger a new build to see more.

Comment thread benchmark/HistFactoryMathFuncs.h
Comment thread benchmark/HistFactoryMathFuncs.h
Comment thread benchmark/HistFactoryMathFuncs.h
Comment thread benchmark/HistFactoryMathFuncs.h
Comment thread benchmark/HistFactoryMathFuncs.h
Comment thread benchmark/HistFactoryMathFuncs.h
Comment thread benchmark/HistFactoryMathFuncs.h
Comment thread benchmark/HistFactoryMathFuncs.h
Comment thread benchmark/HistFactoryMathFuncs.h
Comment thread benchmark/HistFactoryMathFuncs.h
@codecov

codecov Bot commented Aug 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@guitargeek
guitargeek force-pushed the histfactory-benchmark branch from d9b56f9 to ba83b9f Compare August 21, 2026 12:43

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

There were too many comments to post at once. Showing the first 10 out of 22. Check the log or trigger a new build to see more.

Comment thread benchmark/HistFactoryMathFuncs.h
Comment thread benchmark/HistFactoryMathFuncs.h
Comment thread benchmark/HistFactoryMathFuncs.h
Comment thread benchmark/HistFactoryMathFuncs.h
Comment thread benchmark/HistFactoryMathFuncs.h
Comment thread benchmark/HistFactoryMathFuncs.h
Comment thread benchmark/HistFactoryMathFuncs.h
Comment thread benchmark/HistFactoryMathFuncs.h
Comment thread benchmark/HistFactoryMathFuncs.h
Comment thread benchmark/HistFactoryMathFuncs.h
benchmark/histfactory.py generates a scaled-down proxy of the code RooFit
emits for a real 1470-parameter ATLAS likelihood, compiles it with the clad
plugin, and runs it, printing the compilation time in the same table format
as the benchmark executable's primal/gradient/hessian timings. The .cpp is
a transient artifact in a temp directory (kept with --cpp); the compiler is
read from the clad build tree's CMakeCache, so `histfactory.py --clad-build
<dir>` is all that is needed to compare clad revisions.

The generated model reproduces the statement patterns of the full
likelihood at 1/16 scale: 91 parameters, 4 channels with straight-line
per-sample yields (flexibleInterp over nuisance subsets, product factors, a
lumi parameter), bin loops with uniformBinNumber shape lookups feeding
binned nll terms, and a constraint block (gaussian/gaussianIntegral and
poisson/poissonIntegral ratios into constraintSum). The driver validates
the gradient against central differences of the primal and sampled hessian
rows against central differences of the gradient. Measured against the full
benchmark, the mini preserves the relative cost profile (gradient ~7x
primal vs 12x; hessian-vector product ~2.2x gradient vs 2.9x) while
compiling in seconds instead of minutes.

benchmark/HistFactoryMathFuncs.h carries the RooFit::Detail::MathFuncs
subset the model calls, with TMath/ROOT::Math replaced by std::
equivalents, the Cephes incomplete gamma behind poissonIntegral, and its
custom clad derivatives copied verbatim from ROOT's Math/CladDerivator.h.
The copied code is kept verbatim on purpose -- including the Cephes
do-while loops, the ROOT naming and formatting, and the pullbacks'
snapshots of not-yet-initialized values (their restores are dead by
construction, but they trip the uninitialized-use compiler diagnostics
that clang-tidy surfaces under -Wall) -- so the file stays diffable
against the originals. clang-format off/on markers and a blanket
NOLINTBEGIN/END block shield exactly that region; an explicit check list
proved futile because the surfaced compiler diagnostics vary with the
compilation flags. The benchmark's own glue (includes, header guard)
conforms.
constraintSum, the one function carrying `#pragma clad checkpoint loop`,
is emitted into the generated main file instead: clad currently
mis-attributes a checkpoint pragma living in an included header, because
the planner selects pragmas by raw SourceLocation order, which does not
match translation-unit order across files.

The benchmark registers as the CTest test clad-HistFactory (labels
"benchmark;short", RUN_SERIAL) under CLAD_ENABLE_BENCHMARKS. Because one
of its metrics is the compilation time of the generated code, it compiles
at test time via the script instead of going through CB_ADD_GBENCHMARK;
CMake passes the tested clang explicitly through the new --compiler
option, since the CMakeCache fallback would pick the host compiler on
builds configured with gcc.
@guitargeek
guitargeek force-pushed the histfactory-benchmark branch from ba83b9f to 2027578 Compare August 21, 2026 12:57
@github-actions

Copy link
Copy Markdown
Contributor

clang-tidy review says "All clean, LGTM! 👍"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant