Getting Started

To get a jump start on HEBench, users can benchmark one of the published reference backends. These backends have been designed to build and run with minimal configuration requirements.

Benchmarking a Reference Backend

This guide will walk you through the steps of running the benchmark for the reference Cleartext backend. Note the default configuration for this backend may take an extended amount of time on some systems. If you’d like to reduce the amount of workloads run, please refer to the Test Harness User Guide and the Benchmark Configuration File Reference on modifying backend configuration files.

Also note that this is a general Getting Started guide using the Cleartext Reference Backend, please find quickstart guides to other reference backends below:

  1. Pick the Cleartext backend repository from the list of published backends, or visit direct link to Cleartext backend repository.
  2. Check the readme for requirements to build the backend.
    Make sure the machine where the build will occur meets such requirements.
  3. Clone the repo to your local machine*. Assume the repo will be cloned cloned to the location contained in environment variable $CLEARTEXT_BACKEND. Note that repo URL may change. Obtain the correct URL from the GitHub repo itself.
cd $CLEARTEXT_BACKEND
git clone https://github.com/hebench/backend-cpu-cleartext.git

*Users must have, at least, read access to the GitHub repository. Otherwise, cloning and/or building steps may fail.

  1. Build with default settings. The following commands will create a build directory, pull down all the required dependencies, build all dependencies and the backend itself, and install the backend and Test Harness.
cd $CLEARTEXT_BACKEND/backend-cpu-cleartext
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=$CLEARTEXT_BACKEND/backend-cpu-cleartext/install -DCMAKE_BUILD_TYPE=Release ..
make -j
make install
  1. Create a directory to hold the report of the benchmark we are about to run:
mkdir $HOME/backend-cpu-cleartext-report
  1. Navigate to directory where the test harness was installed:
cd $CLEARTEXT_BACKEND/backend-cpu-cleartext/install/bin
  1. Run the benchmark by executing test_harness while pointing it to the shared library containing the compiled Cleartext reference backend. If these instructions have been followed correctly, the shared library for the Cleartext reference backend should have been installed in $CLEARTEXT_BACKEND/backend-cpu-cleartext/install/lib .
./test_harness --backend_lib_path $CLEARTEXT_BACKEND/backend-cpu-cleartext/install/lib/libhebench_cleartext_backend.so --report_root_path $HOME/backend-cpu-cleartext-report
  1. After the execution completes, the report will be saved to the specified location $HOME/backend-cpu-cleartext-report. The report is a collection of CSV files organized in directories that represent the benchmark parameters used for each workload benchmarked. Each report has a report.csv file and, if the corresponding benchmark completed successfully, a summary.csv file in the same location. View the summary.csv for the readable information about the benchmark, or report.csv if the benchmark failed for the cause of failure.

Benchmarking a Custom Backend

The power of HEBench is in the ability for users to create custom backends to benchmark their own implementations of the supported workloads, running on whichever framework or hardware they prefer.

Creating a custom backend requires creating a shared library that exposes the API Bridge functionality. Once this is accomplished, the steps to run the benchmark are similar to those listed above.

To create a custom backend, it is required to clone and build HEBench Frontend repository. This will pull and build Test Harness, API Bridge, documentation, and the tools required to create a custom backend (technically, only API Bridge is required to create a backend, but frontend contains the tools to benchmark the backend). Once cloned, follow the tutorial Backend Creation using C++ in the HEBench documentation. Note that it requires knowledge of C++ to complete.

For more information about creating custom backends, read Backend Overview in the documentation.

Adding New Workloads

If there is a need to benchmark a workload that is not officially supported by HEBench, there are a couple of features that allow extending the frontend to support other workloads. 

A user can create a custom workload using the Generic Workload operation, which doesn’t require changes to the framework codebase; or, for more flexibility, add support for the new workload into Test Harness. The former is the preferred method when the operation matches the characteristics of the Generic Workload. The latter is a topic intended for advanced users and requires knowledge and modification of the code base. 

Both methods are well documented in Working with Unsupported Workloads. 

References

Using HEBench

Tutorial: Creating Custom Backend

Tutorial: Adding a New Workload to Test Harness

HEBench Frontend

Published Backends

HEBench Documentation