Skip to content

Add benchmarking for knn and knn nanoflann#6267

Merged
mvieth merged 4 commits intoPointCloudLibrary:masterfrom
jmackay2:benchmark_kdtree
Apr 18, 2025
Merged

Add benchmarking for knn and knn nanoflann#6267
mvieth merged 4 commits intoPointCloudLibrary:masterfrom
jmackay2:benchmark_kdtree

Conversation

@jmackay2
Copy link
Contributor

@jmackay2 jmackay2 commented Apr 16, 2025

This updates the radius search benchmark to also benchmark kdtree and kdtree_nanoflann radius searches. Additional options are added to modify the radius or number of neighbors.

Results:

./benchmark_radius_search ../../test/table_scene_mug_stereo_textured.pcd
-----------------------------------------------------------------------------
Benchmark                                   Time             CPU   Iterations
-----------------------------------------------------------------------------
OrganizedNeighborSearch                 0.617 us        0.619 us      1094508
KdTree                                   47.3 us         47.3 us        16495
KdTreeAll/iterations:1/manual_time        225 us     47072497 us            1 items_per_second=928.178M/s
KdTreeNanoflann                          35.8 us         35.9 us        24727

Changing the neighbor limit:

./benchmark_radius_search ../../test/table_scene_mug_stereo_textured.pcd 0.1 5
-----------------------------------------------------------------------------
Benchmark                                   Time             CPU   Iterations
-----------------------------------------------------------------------------
OrganizedNeighborSearch                 0.671 us        0.674 us      1050794
KdTree                                   2.57 us         2.59 us       278435
KdTreeAll/iterations:1/manual_time       2.50 us       523936 us            1 items_per_second=83.5884G/s
KdTreeNanoflann                          1.31 us         1.32 us       510518

Copy link
Member

@mvieth mvieth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for extending this benchmark! I have added a few comments.

@mvieth
Copy link
Member

mvieth commented Apr 17, 2025

  1. Filtering out the invalid points with removeNaNFromPointCloud does not work here, unfortunately: the output point cloud will not be organized any more. For KdTree and KdTreeNanoflann that is not a problem, but OrganizedNeighbor cannot work with that cloud any more (that also explains the unrealistically low time of 0.617 us/0.671 us for OrganizedNeighborSearch in your benchmark above). So you have to pass the cloud unchanged to the search classes (at least to OrganizedNeighbor), then make sure you only call radiusSearch with valid query points (for example using my suggestion from above: while(!pcl::isFinite((*cloudIn)[searchIdx])) searchIdx = radiusSearchIdx++ % cloudIn->size();)
  2. I noticed that two pcd files are given to the benchmark (in CMakeLists.txt), but only the first one is used (argv[1]). You wrote it so that the search radius is read from the second command line argument (argv[2]), but now it tries to interpret a pcd file name as a search radius (failing obviously). Please remove milk_cartoon_all_small_clorox.pcd from the command line arguments in CMakeLists.txt

Edit: and please make sure that the formatting is correct, otherwise the checks will not pass.

Copy link
Member

@mvieth mvieth left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good now, thanks! By the way, here are the results if I run the benchmarks:

./benchmarks/benchmark_search_radius_search ../test/table_scene_mug_stereo_textured.pcd
-----------------------------------------------------------------------------
Benchmark                                   Time             CPU   Iterations
-----------------------------------------------------------------------------
OrganizedNeighborSearch                  19.8 us         19.9 us        43569
KdTree                                   49.0 us         49.1 us        16203
KdTreeAll/iterations:1/manual_time        217 us     45313606 us            1 items_per_second=966.331M/s
KdTreeNanoflann                          37.1 us         37.2 us        21170

and

./benchmarks/benchmark_search_radius_search ../test/table_scene_mug_stereo_textured.pcd 0.1 5
-----------------------------------------------------------------------------
Benchmark                                   Time             CPU   Iterations
-----------------------------------------------------------------------------
OrganizedNeighborSearch                  5.78 us         5.79 us       100000
KdTree                                   1.41 us         1.42 us       470067
KdTreeAll/iterations:1/manual_time       1.24 us       258355 us            1 items_per_second=169.229G/s
KdTreeNanoflann                         0.772 us        0.774 us       891644

So seems like OrganizedNeighbor is currently not the fastest if only a limited number of neighbors is needed.

@mvieth mvieth merged commit c0d8f86 into PointCloudLibrary:master Apr 18, 2025
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants