Homomorphic Encryption Benchmarking Framework – HEBench

What is HEBench?

A flexible benchmarking framework for Homomorphic Encryption that allows fair performance comparison among different implementations (in hardware and/or software) of a collection of homomorphic encryption workloads.

The HEBench project is hosted at https://github.com/hebench

Check out the video below for a quick introduction:

To get a jump start on HEBench, users can benchmark one of the published reference backends. These backends have been designed to build and run with minimal configuration requirements.

Benchmarking a Reference Backend

This guide will walk you through the steps of running the benchmark for the reference Cleartext backend. Note the default configuration for this backend may take an extended amount of time on some systems. If you’d like to reduce the amount of workloads run, please refer to the Test Harness User Guide and the Benchmark Configuration File Reference on modifying backend configuration files.

Also note that this is a general Getting Started guide using the Cleartext Reference Backend, please find quickstart guides to other reference backends below:

• PALISADE Reference Backend Quickstart Guide
• SEAL Refence Backend Quickstart Guide
• HElib Reference Backend Quickstart Guide

1. Pick the Cleartext backend repository from the list of published backends, or visit direct link to Cleartext backend repository.
2. Check the readme for requirements to build the backend.
   Make sure the machine where the build will occur meets such requirements.
3. Clone the repo to your local machine*. Assume the repo will be cloned cloned to the location contained in environment variable $CLEARTEXT_BACKEND. Note that repo URL may change. Obtain the correct URL from the GitHub repo itself.

cd $CLEARTEXT_BACKEND
git clone https://github.com/hebench/backend-cpu-cleartext.git

*Users must have, at least, read access to the GitHub repository. Otherwise, cloning and/or building steps may fail.

4. Build with default settings. The following commands will create a build directory, pull down all the required dependencies, build all dependencies and the backend itself, and install the backend and Test Harness.

cd $CLEARTEXT_BACKEND/backend-cpu-cleartext
mkdir build
cd build
cmake -DCMAKE_INSTALL_PREFIX=$CLEARTEXT_BACKEND/backend-cpu-cleartext/install -DCMAKE_BUILD_TYPE=Release ..
make -j
make install

5. Create a directory to hold the report of the benchmark we are about to run:

mkdir $HOME/backend-cpu-cleartext-report

6. Navigate to directory where the test harness was installed:

cd $CLEARTEXT_BACKEND/backend-cpu-cleartext/install/bin

7. Run the benchmark by executing test_harness while pointing it to the shared library containing the compiled Cleartext reference backend. If these instructions have been followed correctly, the shared library for the Cleartext reference backend should have been installed in $CLEARTEXT_BACKEND/backend-cpu-cleartext/install/lib .

./test_harness --backend_lib_path $CLEARTEXT_BACKEND/backend-cpu-cleartext/install/lib/libhebench_cleartext_backend.so --report_root_path $HOME/backend-cpu-cleartext-report

8. After the execution completes, the report will be saved to the specified location $HOME/backend-cpu-cleartext-report. The report is a collection of CSV files organized in directories that represent the benchmark parameters used for each workload benchmarked. Each report has a report.csv file and, if the corresponding benchmark completed successfully, a summary.csv file in the same location. View the summary.csv for the readable information about the benchmark, or report.csv if the benchmark failed for the cause of failure.

Where to go From Here?

There are several ways to use HEBench, from testing pre-existing backends to running your own. Use the flowchart below as a guide: