FAQ – HEBench Organization

What is HEBench?

A flexible benchmarking framework for Homomorphic Encryption that allows fair performance comparison among different implementations (in hardware and/or software) of a collection of homomorphic encryption workloads.

HEBench project is hosted at https://github.com/hebench .

What is Homomorphic Encryption?

Homomorphic Encryption (HE) refers to a form of encryption which enables computation in the encrypted domain without requiring access to a private key. This technology enables unprecedented capabilities for secure outsourced computation.

More information can be found at Homomorphic Encryption Standardization.

What Does HEBench Offer?

The main objective of HEBench is to allow users to create backends that implement their solutions to a set of supported workloads to be benchmarked by HEBench’s Test Harness. To this end, HEBench offers a flexible C API that enables implementation of such backends to test HE APIs, algorithms, optimizations, hardware, etc.

A backend is a shared library that exposes its functionality through the API Bridge interface, compatible with the Test Harness. A backend can focus on specific workloads to benchmark; it does not need to implement every possible workload supported by Test Harness.

With this release, some reference backends have been published as well. As reference, they show how users might implement their own backends to fit their needs.

Software Components and Architecture

Test Harness and backend Loader (w/ extensible workload framework): Main frontend component that drives the benchmarking of the workload operations.

Currently supported categories:

Latency: measures end-to-end latency (time to complete) for a single operation
Offline: measures throughput (ops/sec) that a backend can perform on a batch of data.

Currently supported workloads:

Element-wise vector addition
Element-wise vector multiplication
Vector dot product
Matrix multiplication
Logistic regression inference

The dynamic backend loader allows the Test Harness to dynamically link to any backend that implements the API Bridge interface.

API Bridge with C++ Wrapper (w/ documentation and example): API Bridge offers a uniform C interface that bridges the communication between Test Harness and backends. Test Harness communicates and issues commands to backends for the operations to perform via the API Bridge. Operations and their results are validated and timed by Test Harness.

C++ wrapper is a thin wrapper around the API Bridge. It offers a utility library that encapsulates common functionality expected by the API Bridge and hides boilerplate code, easing the development of backends by users.

A backend example is shipped with the frontend that contains a simple implementation of a workload in clear-text. This is intended to be used as a starting point for any new backends.

Clear text backend reference (CPU): A reference backend that performs the operations of all supported workloads in clear text. This is intended as an extended reference on how to implement a fully functional backend, and as a unit test for the Test Harness.

SEAL backend reference (CPU): A reference backend that performs the operations of some of the supported workloads. This is intended as a reference to an actual implementation of the workloads in the HE space. It uses Microsoft SEAL library for the HE operations.

PALISADE backend reference (CPU): A reference backend that performs the operations of some of the supported workloads. This is intended as a reference to an actual implementation of the workloads in the HE space. It uses PALISADE lattice cryptography library for the HE operations.

HElib backend reference (CPU): A reference backend that performs the operations of some of the supported workloads. This is intended as a reference to an actual implementation of the workloads in the HE space. It uses the HElib homomorphic encryption library for the HE operations.

What Can Be Done with HEBench?

Out of the box, users can run the benchmark from the API Bridge example, clear text, reference SEAL, and reference PALISADE backends. These run on CPU using the specified API support.

Flexibility to create new backends is already in place and users are encouraged to create their own to benchmark their HE implementations in whichever hardware or system it may be hosted.

Users can benchmark their own implementations with HEBench (by creating their own backends) and compare results with other implementations of published backends. The framework is designed to give a fair shot to all users to put their best foot forward when implementing their solutions.

How can I contribute to HEBench?

As an open-source project, anyone can contribute to the development of the HEBench framework. Users that are not part of the organization typically contribute by forking the repo to which contributions are being added, and then submit Pull Requests against the development branch. See the contribution guidelines for each repository.

Users looking to submit their benchmark solution to an HEBench workload can submit it to HEBench for review and turn their solution into an official HEBench submission. Submission benchmarking results will be published on the HEBench website, and the backend code will be available in the HEBench GitHub organization. (Submission procedures and implementation are currently work-in-progress. In the meantime, users can publish their backend in GitHub, referencing HEBench).

Anyone looking forward to be a more active collaborator with the HEBench framework or community can Join Us as a member of the HEBench Organization. Once a collaborator has joined as member, they can volunteer to contribute actively in one of the work teams for framework development, pathfinding to search and propose new workload benchmarks or features, or review backend submissions. Members seeking even more active roles in swaying the direction of the Homomorphic Encryption benchmarking community can also request to become HEBench Organization Advisors.