VERIFAI 2.0

Table of Contents

• VERIFAI 2.0
  • Table of Contents
  • Overview
    • Metrics
  • Prerequisites
  • Installation
    • Clone the Repository
    • Setup Docker
    • Configure the Environment
    • Build Docker Containers
  • Usage
    • Starting the Containers
    • Stopping the Containers
  • Docker Setup
    • Python Image
    • Building the Docker Image
  • Poetry for Dependency Management
    • Adding New Packages
  • MKdocs
  • Filenames and Folders
    • Important Naming Conventions
  • Running the Application
  • Further Notes
  • Contributing
  • License
  • Contact

Overview

This is a research project for supporting responsible AI and builds upon my master's thesis VERIFAI. The goal of the framework is evaluating Machine Learning Models with different metrics. During research, it became clear, that the Metrics can be grouped into one out of these categories:

Metrics

Ethical Metrics

• Bias/Fairness Metrics
  • NLP
    • Adversarial Bias (Paper Link): Explanation

Privacy Metrics

• Privacy Leakage Metrics
  • Aspect
    • Metric (Paper Link): Explanation

Security Metrics

• Robustness Metrics
  • Aspect
    • Metric (Paper Link): Explanation

Explainability Metrics

• XAI Evaluation Metrics
  • Aspect
    • Metric (Paper Link): Explanation

The library also supports standard performance metrics like Accuracy Score etc.

Prerequisites

• Docker
• Docker Compose
• Python 3.11
• Poetry

Installation

Clone the Repository

git clone https://gitlab.com/sabrinagoellner/verifai.git

Setup Docker

Ensure Docker and Docker Compose are installed and running on your machine.

Configure the Environment

Two environment files are needed, neither of which is committed. Copy the templates and fill them in:

cp .env.example .env
cp backend/.env.example backend/.env

• .env (repository root) holds the MongoDB credentials. Compose reads it and substitutes the values into the compose files. MONGO_ROOT_PASSWORD has no default — pick one before the first start, because Mongo only creates the root user on an empty data volume.
• backend/.env holds the external API keys. Both are optional; leaving one blank only disables the metrics that depend on it.

Build Docker Containers

docker-compose build

Usage

Starting the Containers

docker-compose up

Stopping the Containers

docker-compose down

Docker Setup

Python Image

• Base Image: python:3.11-slim
• Description: This image is based on Debian but is stripped down for efficiency. It uses glibc, which ensures better compatibility with most Python packages, reducing the need for additional adjustments when building and installing dependencies.

Building the Docker Image

Buildkit for faster image building:

export DOCKER_BUILDKIT=1
docker-compose build <image>

Building docker image and write to the logfile:

cd backend
docker-compose build <image> --no-cache > build.log 2>&1

Build with no cache (build from scratch):

docker-compose build <image> --no-cache

Poetry for Dependency Management

Poetry is used for managing dependencies in the backend. It is loaded by the Dockerfile and installed during the build process.

Adding New Packages

To add new packages, navigate to the backend folder and use:

poetry add <library>

Then, rebuild the Docker image for the backend.

MKdocs

• How-To implement new metrics
• please refer to MkDocs for a more detailed documentation
• Serve the docs at port 8080 because 8000 is used by the frontend:

cd verifai-docs;
mkdocs serve -a 'localhost:8080'

Filenames and Folders

The project is structured with three main apps:

• Frontend
• Backend
• MongoDB

Important Naming Conventions

• Outside Docker, the applications are named frontend and backend.
• Inside Docker, they are referred to as app.
• Each app has a single Dockerfile and Image that can be built separately, simplifying maintenance.

Running the Application

To access the frontend of the application navigate to:

http://localhost:8050/

Further Notes

• Make sure to follow the above steps carefully when adding dependencies or making changes to the Docker setup.
• For any further customization or troubleshooting, refer to the official Docker and Poetry documentation.

Contributing

We welcome contributions! Please follow these steps:

1. Fork the repository.
2. Create a new branch (git checkout -b feature-branch).
3. Make your changes.
4. Commit your changes (git commit -m 'Add some feature').
5. Push to the branch (git push origin feature-branch).
6. Open a pull request.

License

TO DO

Contact

For issues, questions, or discussions, please open an issue on GitHub or contact us at [your-email@example.com].

---