Skip to content

README.md

Latest commit

 

History

History
158 lines (121 loc) · 7.01 KB

README.md

File metadata and controls

158 lines (121 loc) · 7.01 KB
Raw

OpenVINO Backend for ExecuTorch

The OpenVINO backend enables optimized execution of deep learning models on Intel hardware, leveraging Intel's OpenVINO toolkit for inference acceleration.

Supported Hardware

OpenVINO backend supports the following hardware:

  • Intel CPUs
  • Intel integrated GPUs
  • Intel discrete GPUs
  • Intel NPUs

For more information on the supported hardware, please refer to OpenVINO System Requirements page.

Quick Start (pip wheel)

On Linux and Windows, the OpenVINO backend is included in the ExecuTorch pip wheel. Install the OpenVINO runtime to activate it:

pip install executorch[openvino]

For now the extra installs no runtime on Python 3.15, or on Linux aarch64 with Python 3.14. OpenVINO publishes no build for Python 3.15, and its aarch64 builds for Python 3.14 need glibc 2.35 or newer.

The backend automatically discovers the OpenVINO C library from the pip-installed package, so no library search path setup is needed.

If auto-discovery fails (e.g. non-standard install), set OPENVINO_LIB_PATH to the full path of the OpenVINO C library: a libopenvino_c.so file on Linux, or openvino_c.dll on Windows. For the pip package on Linux:

export OPENVINO_LIB_PATH="$(python3 -c 'import glob, openvino, os; print(sorted(glob.glob(os.path.join(os.path.dirname(openvino.__file__), "libs", "libopenvino_c.so*")))[0])')"

On Windows, in PowerShell:

$env:OPENVINO_LIB_PATH = python -c "import openvino, os; print(os.path.join(os.path.dirname(openvino.__file__), 'libs', 'openvino_c.dll'))"

On Windows, PATH and the current folder are not searched for the library or for its dependencies, such as tbb12.dll. Those must sit beside openvino_c.dll, as they do in the pip package, or in a folder added with os.add_dll_directory (AddDllDirectory in C++). Importing the openvino Python package adds its folders this way, including the ones Intel's setupvars.bat lists.

Verify the backend is available:

from executorch.extension.pybindings.portable_lib import (
    _get_registered_backend_names,
)
print(_get_registered_backend_names())
# Should include 'OpenvinoBackend'

Directory Structure

executorch
├── backends
│   └── openvino
│       ├── quantizer
│           ├── observers
│               └── nncf_observers.py
│           ├── __init__.py
│           └── quantizer.py
│       ├── runtime
│           ├── OpenvinoApi.h
│           ├── OpenvinoBackend.cpp
│           └── OpenvinoBackend.h
│       ├── scripts
│           └── openvino_build.sh
│       ├── tests
│       ├── CMakeLists.txt
│       ├── README.md
│       ├── __init__.py
│       ├── partitioner.py
│       ├── preprocess.py
│       └── requirements.txt
└── examples
    └── openvino
        ├── aot_optimize_and_infer.py
        └── README.md

Build Instructions

Prerequisites

Before you begin, ensure you have openvino installed and configured on your system.

Use OpenVINO from Release Packages

  1. Download the OpenVINO release package from here. Make sure to select your configuration and click on OpenVINO Archives under the distribution section to download the appropriate archive for your platform.

  2. Extract the release package from the archive and set the environment variables.

    tar -zxf openvino_toolkit_<your_release_configuration>.tgz
cd openvino_toolkit_<your_release_configuration>
source setupvars.sh

(Optional) Build OpenVINO from Source

git clone https://github.com/openvinotoolkit/openvino.git
cd openvino
git submodule update --init --recursive
sudo ./install_build_dependencies.sh
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DENABLE_PYTHON=ON
make -j$(nproc)

cd ..
cmake --install build --prefix <your_preferred_install_location>
cd <your_preferred_install_location>
source setupvars.sh

For more information about OpenVINO build, refer to the OpenVINO Build Instructions.

Setup

Follow the steps below to setup your build environment:

  1. Create a Virtual Environment
  • Create a virtual environment and activate it by executing the commands below. 
    python -m venv env
source env/bin/activate
  1. Clone ExecuTorch Repository from Github
  • Clone Executorch repository by executing the command below. 
    git clone --recurse-submodules https://github.com/pytorch/executorch.git
  1. Build ExecuTorch with OpenVINO Backend

  • Ensure that you are inside executorch/backends/openvino/scripts directory. The following command builds and installs ExecuTorch with the OpenVINO backend, also compiles the C++ runtime libraries and binaries into <executorch_root>/cmake-out for quick inference testing.

    openvino_build.sh

  • Optionally, openvino_build.sh script can be used to build python package or C++ libraries/binaries separately.

    Build OpenVINO Backend Python Package with Pybindings: To build and install the OpenVINO backend Python package with Python bindings, run the openvino_build.sh script with the --enable_python argument as shown in the below command. This will compile and install the ExecuTorch Python package with the OpenVINO backend into your Python environment. This option will also enable python bindings required to execute OpenVINO backend tests and aot_optimize_and_infer.py script inside executorch/examples/openvino folder.

    ./openvino_build.sh --enable_python

    Build C++ Runtime Libraries for OpenVINO Backend: Run the openvino_build.sh script with the --cpp_runtime flag to build the C++ runtime libraries as shown in the below command. The compiled libraries files and binaries can be found in the <executorch_root>/cmake-out directory. The binary located at <executorch_root>/cmake-out/executor_runner can be used to run inference with vision models.

    ./openvino_build.sh --cpp_runtime

    Build C++ Runtime Libraries with LLM Extension: Run the openvino_build.sh script with the --cpp_runtime_llm flag to build the C++ runtime libraries with LLM extension as shown in the below command. Use this option instead of --cpp_runtime for LLM extension support which is required by LLM examples.

    ./openvino_build.sh --cpp_runtime_llm

For more information about ExecuTorch environment setup, refer to the Environment Setup guide.

Run

Please refer to README.md for instructions on running examples of various of models with openvino backend.