Benchmark a set of planners on a problem instance. More...

#include <ompl/tools/benchmark/Benchmark.h>

Classes
struct	Status
	This structure contains information about the activity of a benchmark instance. If the instance is running, it is possible to find out information such as which planner is currently being tested or how much. More...
struct	RunProperties
	The data collected from a run of a planner is stored as key-value pairs. More...
struct	PlannerExperiment
	The data collected after running a planner multiple times. More...
struct	CompleteExperiment
	This structure holds experimental data for a set of planners. More...
struct	Request
	Representation of a benchmark request. More...

Public Types
using	RunProgressData = std::vector<std::map<std::string, std::string>>
using	PreSetupEvent = std::function<void(const base::PlannerPtr &)>
	Signature of function that can be called before a planner execution is started.
using	PostSetupEvent = std::function<void(const base::PlannerPtr &, RunProperties &)>
	Signature of function that can be called after a planner execution is completed.

Public Member Functions
	Benchmark (geometric::SimpleSetup &setup, const std::string &name=std::string())
	Constructor needs the SimpleSetup instance needed for planning. Optionally, the experiment name (name) can be specified.
	Benchmark (control::SimpleSetup &setup, const std::string &name=std::string())
	Constructor needs the SimpleSetup instance needed for planning. Optionally, the experiment name (name) can be specified.
void	addExperimentParameter (const std::string &name, const std::string &type, const std::string &value)
	Add an optional parameter's information to the benchmark output. Useful for aggregating results over different benchmark instances, e.g., parameter sweep. type is typically "BOOLEAN", "INTEGER", or "REAL".
const std::map< std::string, std::string > &	getExperimentParameters () const
	Get all optional benchmark parameters. The map key is 'name type'.
std::size_t	numExperimentParameters () const
	Return the number of optional benchmark parameters.
void	setExperimentName (const std::string &name)
	Set the name of the experiment.
const std::string &	getExperimentName () const
	Get the name of the experiment.
void	addPlanner (const base::PlannerPtr &planner)
	Add a planner to use.
void	addPlannerAllocator (const base::PlannerAllocator &pa)
	Add a planner allocator to use.
void	clearPlanners ()
	Clear the set of planners to be benchmarked.
void	setPlannerSwitchEvent (const PreSetupEvent &event)
	Set the event to be called before any runs of a particular planner (when the planner is switched).
void	setPreRunEvent (const PreSetupEvent &event)
	Set the event to be called before the run of a planner.
void	setPostRunEvent (const PostSetupEvent &event)
	Set the event to be called after the run of a planner.
virtual void	benchmark (const Request &req)
	Benchmark the added planners on the defined problem. Repeated calls clear previously gathered data.
const Status &	getStatus () const
	Get the status of the benchmarking code. This function can be called in a separate thread to check how much progress has been made.
const CompleteExperiment &	getRecordedExperimentData () const
	Return all the experiment data that would be written to the results file. The data should not be changed, but it could be useful to quickly extract cartain statistics.
virtual bool	saveResultsToStream (std::ostream &out=std::cout) const
	Save the results of the benchmark to a stream.
bool	saveResultsToFile (const char *filename) const
	Save the results of the benchmark to a file.
bool	saveResultsToFile () const
	Save the results of the benchmark to a file. The name of the file is the current date and time.

Protected Attributes
geometric::SimpleSetup *	gsetup_
	The instance of the problem to benchmark (if geometric planning).
control::SimpleSetup *	csetup_
	The instance of the problem to benchmark (if planning with controls).
std::vector< base::PlannerPtr >	planners_
	The set of planners to be tested.
CompleteExperiment	exp_
	The collected experimental data (for all planners).
Status	status_
	The current status of this benchmarking instance.
PreSetupEvent	plannerSwitch_
	Event to be called when the evaluated planner is switched.
PreSetupEvent	preRun_
	Event to be called before the run of a planner.
PostSetupEvent	postRun_
	Event to be called after the run of a planner.

Detailed Description

Benchmark a set of planners on a problem instance.

Definition at line 48 of file Benchmark.h.

Member Typedef Documentation

◆ PostSetupEvent

using ompl::tools::Benchmark::PostSetupEvent = std::function<void(const base::PlannerPtr &, RunProperties &)>

Signature of function that can be called after a planner execution is completed.

Definition at line 82 of file Benchmark.h.

◆ PreSetupEvent

using ompl::tools::Benchmark::PreSetupEvent = std::function<void(const base::PlannerPtr &)>

Signature of function that can be called before a planner execution is started.

Definition at line 79 of file Benchmark.h.

◆ RunProgressData

using ompl::tools::Benchmark::RunProgressData = std::vector<std::map<std::string, std::string>>

Definition at line 76 of file Benchmark.h.

Constructor & Destructor Documentation

◆ Benchmark() [1/2]

ompl::tools::Benchmark::Benchmark	(	geometric::SimpleSetup &	setup,
		const std::string &	name = std::string() )

Constructor needs the SimpleSetup instance needed for planning. Optionally, the experiment name (name) can be specified.

Definition at line 196 of file Benchmark.cpp.

◆ Benchmark() [2/2]

ompl::tools::Benchmark::Benchmark	(	control::SimpleSetup &	setup,
		const std::string &	name = std::string() )

Constructor needs the SimpleSetup instance needed for planning. Optionally, the experiment name (name) can be specified.

Definition at line 202 of file Benchmark.cpp.

Member Function Documentation

◆ addExperimentParameter()

void ompl::tools::Benchmark::addExperimentParameter	(	const std::string &	name,
		const std::string &	type,
		const std::string &	value )

Add an optional parameter's information to the benchmark output. Useful for aggregating results over different benchmark instances, e.g., parameter sweep. type is typically "BOOLEAN", "INTEGER", or "REAL".

Definition at line 208 of file Benchmark.cpp.

◆ addPlanner()

void ompl::tools::Benchmark::addPlanner ( const base::PlannerPtr & planner )

Add a planner to use.

Definition at line 234 of file Benchmark.cpp.

◆ addPlannerAllocator()

void ompl::tools::Benchmark::addPlannerAllocator ( const base::PlannerAllocator & pa )

Add a planner allocator to use.

Definition at line 242 of file Benchmark.cpp.

◆ benchmark()

void ompl::tools::Benchmark::benchmark ( const Request & req )

virtual

Benchmark the added planners on the defined problem. Repeated calls clear previously gathered data.

Parameters

req	The parameters for the execution of the benchmark

Note

The values returned for memory consumption may be misleading. Memory allocators often free memory in a lazy fashion, so the returned values for memory consumption indicate the increase in memory usage for each run. Since not all the memory for the previous run was freed, the increase in usage may be close to

To get correct averages for memory usage, use req.runCount = 1 and run the process multiple times.

Definition at line 433 of file Benchmark.cpp.

◆ clearPlanners()

void ompl::tools::Benchmark::clearPlanners ( )

Clear the set of planners to be benchmarked.

Definition at line 247 of file Benchmark.cpp.

◆ getExperimentName()

const std::string & ompl::tools::Benchmark::getExperimentName ( ) const

Get the name of the experiment.

Definition at line 229 of file Benchmark.cpp.

◆ getExperimentParameters()

const std::map< std::string, std::string > & ompl::tools::Benchmark::getExperimentParameters ( ) const

Get all optional benchmark parameters. The map key is 'name type'.

Definition at line 214 of file Benchmark.cpp.

◆ getRecordedExperimentData()

const ompl::tools::Benchmark::CompleteExperiment & ompl::tools::Benchmark::getRecordedExperimentData ( ) const

Return all the experiment data that would be written to the results file. The data should not be changed, but it could be useful to quickly extract cartain statistics.

Definition at line 274 of file Benchmark.cpp.

◆ getStatus()

const ompl::tools::Benchmark::Status & ompl::tools::Benchmark::getStatus ( ) const

Get the status of the benchmarking code. This function can be called in a separate thread to check how much progress has been made.

Definition at line 269 of file Benchmark.cpp.

◆ numExperimentParameters()

std::size_t ompl::tools::Benchmark::numExperimentParameters ( ) const

Return the number of optional benchmark parameters.

Definition at line 219 of file Benchmark.cpp.

◆ saveResultsToFile() [1/2]

bool ompl::tools::Benchmark::saveResultsToFile ( ) const

Save the results of the benchmark to a file. The name of the file is the current date and time.

Definition at line 300 of file Benchmark.cpp.

◆ saveResultsToFile() [2/2]

bool ompl::tools::Benchmark::saveResultsToFile ( const char * filename ) const

Save the results of the benchmark to a file.

Definition at line 279 of file Benchmark.cpp.

◆ saveResultsToStream()

bool ompl::tools::Benchmark::saveResultsToStream ( std::ostream & out = std::cout ) const

virtual

Save the results of the benchmark to a stream.

Definition at line 306 of file Benchmark.cpp.

◆ setExperimentName()

void ompl::tools::Benchmark::setExperimentName ( const std::string & name )

Set the name of the experiment.

Definition at line 224 of file Benchmark.cpp.

◆ setPlannerSwitchEvent()

void ompl::tools::Benchmark::setPlannerSwitchEvent ( const PreSetupEvent & event )

Set the event to be called before any runs of a particular planner (when the planner is switched).

Definition at line 252 of file Benchmark.cpp.

◆ setPostRunEvent()

void ompl::tools::Benchmark::setPostRunEvent ( const PostSetupEvent & event )

Set the event to be called after the run of a planner.

Definition at line 264 of file Benchmark.cpp.

◆ setPreRunEvent()

void ompl::tools::Benchmark::setPreRunEvent ( const PreSetupEvent & event )

Set the event to be called before the run of a planner.

Definition at line 258 of file Benchmark.cpp.

Member Data Documentation

◆ csetup_

control::SimpleSetup* ompl::tools::Benchmark::csetup_

protected

The instance of the problem to benchmark (if planning with controls).

Definition at line 276 of file Benchmark.h.

◆ exp_

CompleteExperiment ompl::tools::Benchmark::exp_

protected

The collected experimental data (for all planners).

Definition at line 282 of file Benchmark.h.

◆ gsetup_

geometric::SimpleSetup* ompl::tools::Benchmark::gsetup_

protected

The instance of the problem to benchmark (if geometric planning).

Definition at line 273 of file Benchmark.h.

◆ planners_

std::vector<base::PlannerPtr> ompl::tools::Benchmark::planners_

protected

The set of planners to be tested.

Definition at line 279 of file Benchmark.h.

◆ plannerSwitch_

PreSetupEvent ompl::tools::Benchmark::plannerSwitch_

protected

Event to be called when the evaluated planner is switched.

Definition at line 288 of file Benchmark.h.

◆ postRun_

PostSetupEvent ompl::tools::Benchmark::postRun_

protected

Event to be called after the run of a planner.

Definition at line 294 of file Benchmark.h.

◆ preRun_

PreSetupEvent ompl::tools::Benchmark::preRun_

protected

Event to be called before the run of a planner.

Definition at line 291 of file Benchmark.h.

◆ status_

Status ompl::tools::Benchmark::status_

protected

The current status of this benchmarking instance.

Definition at line 285 of file Benchmark.h.

The documentation for this class was generated from the following files:

ompl/tools/benchmark/Benchmark.h
ompl/tools/benchmark/src/Benchmark.cpp

Classes

Public Types

Public Member Functions

Protected Attributes

Detailed Description

Member Typedef Documentation

◆ PostSetupEvent

◆ PreSetupEvent

◆ RunProgressData

Constructor & Destructor Documentation

◆ Benchmark() [1/2]

◆ Benchmark() [2/2]

Member Function Documentation

◆ addExperimentParameter()

◆ addPlanner()

◆ addPlannerAllocator()

◆ benchmark()

◆ clearPlanners()

◆ getExperimentName()

◆ getExperimentParameters()

◆ getRecordedExperimentData()

◆ getStatus()

◆ numExperimentParameters()

◆ saveResultsToFile() [1/2]

◆ saveResultsToFile() [2/2]

◆ saveResultsToStream()

◆ setExperimentName()

◆ setPlannerSwitchEvent()

◆ setPostRunEvent()

◆ setPreRunEvent()

Member Data Documentation

◆ csetup_

◆ exp_

◆ gsetup_

◆ planners_

◆ plannerSwitch_

◆ postRun_

◆ preRun_

◆ status_