Homomorphic Encryption Benchmarking Framework – HEBench

What is HEBench?

A flexible benchmarking framework for Homomorphic Encryption that allows fair performance comparison among different implementations (in hardware and/or software) of a collection of homomorphic encryption workloads.

The HEBench project is hosted at https://github.com/hebench

Check out the video below for a quick introduction:

To get a jump start on HEBench, users can benchmark one of the published reference backends. These backends have been designed to build and run with minimal configuration requirements.

Benchmarking a Reference Backend

This guide will walk you through the steps of running the benchmark for the reference Cleartext backend. Note the default configuration for this backend may take an extended amount of time on some systems. If you’d like to reduce the amount of workloads run, please refer to the Test Harness User Guide and the Benchmark Configuration File Reference on modifying backend configuration files.

Also note that this is a general Getting Started guide using the Cleartext Reference Backend, please find quickstart guides to other reference backends below:

  1. Pick the Cleartext backend repository from the list of published backends, or visit direct link to Cleartext backend repository.
  2. Check the readme for requirements to build the backend.
    Make sure the machine where the build will occur meets such requirements.
  3. Clone the repo to your local machine*. Assume the repo will be cloned cloned to the location contained in environment variable $CLEARTEXT_BACKEND. Note that repo URL may change. Obtain the correct URL from the GitHub repo itself.
cd $CLEARTEXT_BACKEND
git clone https://github.com/hebench/backend-cpu-cleartext.git

*Users must have, at least, read access to the GitHub repository. Otherwise, cloning and/or building steps may fail.

  1. Build with default settings. The following commands will create a build directory, pull down all the required dependencies, build all dependencies and the backend itself, and install the backend and Test Harness.
cd $CLEARTEXT_BACKEND/backend-cpu-cleartext
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=$CLEARTEXT_BACKEND/backend-cpu-cleartext/install -DCMAKE_BUILD_TYPE=Release ..
make -j
make install
  1. Create a directory to hold the report of the benchmark we are about to run:
mkdir $HOME/backend-cpu-cleartext-report
  1. Navigate to directory where the test harness was installed:
cd $CLEARTEXT_BACKEND/backend-cpu-cleartext/install/bin
  1. Run the benchmark by executing test_harness while pointing it to the shared library containing the compiled Cleartext reference backend. If these instructions have been followed correctly, the shared library for the Cleartext reference backend should have been installed in $CLEARTEXT_BACKEND/backend-cpu-cleartext/install/lib .
./test_harness --backend_lib_path $CLEARTEXT_BACKEND/backend-cpu-cleartext/install/lib/libhebench_cleartext_backend.so --report_root_path $HOME/backend-cpu-cleartext-report
  1. After the execution completes, the report will be saved to the specified location $HOME/backend-cpu-cleartext-report. The report is a collection of CSV files organized in directories that represent the benchmark parameters used for each workload benchmarked. Each report has a report.csv file and, if the corresponding benchmark completed successfully, a summary.csv file in the same location. View the summary.csv for the readable information about the benchmark, or report.csv if the benchmark failed for the cause of failure.

Where to go From Here?

There are several ways to use HEBench, from testing pre-existing backends to running your own. Use the flowchart below as a guide:

Start with Introduction. Then follow the flow guiding towards specific use-cases of HEBench, and click on a subject for more information
Start with Introduction. Then follow the flow guiding towards specific use-cases of HEBench, and click on a subject for more information
A. Contributing to HEBench
A. Contributing to…
1. Introduction
1. Introduction
User Type
User Type
3. Backend overview
3. Backend overvie…

I want to benchmark my own 

implementation of a workload

(Backend developer)

I want to benchmark my own…

My workload is not supported

(Workload developer)

My workload is not supported…
7. Creating a Backend (tutorial)
7. Creating a Back…
Add new workload to framework?
Add new workload to fra…
10. Use Generic Workload
10. Use Generic Wo…
13. Implement new Workload into the Framework (advanced)
13. Implement new…
Yes
Yes
No
No
9. Intro to workload addition
9. Intro to worklo…
12. Using Custom Datasets
12. Using Custom D…
2. Benchmarking a workload with an existing backend
2. Benchmarking a…

I want to run an 

existing benchmark

(HEBench consumer)

I want to run an…
5. Test Harness User Guide
5. Test Harness Us…
6. Using config files
6. Using config fi…
8. Using report compiler
8. Using report co…
4. Intro to Report
4. Intro to Report
11. Generic Workload Example (tutorial)
11. Generic Worklo…
Text is not SVG – cannot display