README.md

Backend

This project contains the backend of the Data Abstraction Platform.

It is comprised of a collection of applications with the main goal is to run a http server which will offer API services. Contains a set of applications that serve different purposes as follows:

• cmd_app_create_super_user - cli app to create a system user with the necessary system administration roles in the system.
• cmd_app_init_database - cli app to initialize database with default values.
• job_service - service that combines all job services into one.
• web_service - http server that combines all web api services into one.

This section contains information about setting up and running the various backend applications.

Table of Contents

1. Environment Variables
2. Development
   1. Setting Environment Variables
   2. Database Migrations
      • Create New Migrations
      • Up migrations
      • Down migrations
      • Fix dirty migrations
   3. Applications
      • Web Service
      • Cmd App Create Super User
      • Cmd App Init Database
3. Containerization
   • Web Service
4. Miscellaneous

Containerization

Containerization-Web Service

The script builds a data_abstraction_platform/web_service container using this Dockerfile.

The script accepts the following flags during execution.

Flag	Example	Default	Purpose
-c	podman	docker	Container engine to use Optional
-t	v1alpha3	latest	Container Tag Optional

Example:

#!/bin/bash

CONTAINER_TAG="v4.18.3"
CONTAINER_ENGINE="podman"

bash scripts/build/web_service_container_image.sh -t $CONTAINER_TAG -c $CONTAINER_ENGINE

Development

The following pre-requisites must be installed/already setup:

• Go.
• Postgres.
• OpenID Provider
• Storage - Local Folder or S3.
• Container Engine - Optional. For building and runing container images.

NB. For consistency, execute shell scripts at the root of the backend project.

Applications

Some environment variables are required to be setup before each application runs otherwise it will crash.

Before the applications can be used, dependencies need to be downloaded.

#!/bin/bash

go mod tidy

Applications-Web Service

You can build and run the web_service application in the following ways:

1. Build+run. Useful for quick iterations

#!/bin/bash

go run cmd/web_service/main.go

2. Build alone using the script. Then run the application with the command: bin/web_service.

#!/bin/bash

bash scripts/build/web_service.sh

The application uses one port which defaults to 5174.

Applications-Cmd App Create Super User

You can build and run the cmd_app_create_super_user application in the following ways:

1. Build+run. Useful for quick iterations

#!/bin/bash

go run cmd/cmd_app_create_super_user/main.go

2. Build alone using the script. Then run the application with the command: bin/cmd_app_create_super_user.

#!/bin/bash

bash scripts/build/cmd_app_create_super_user.sh

Applications-Cmd App Init Database

You can build and run the cmd_app_init_database application in the following ways:

1. Build+run. Useful for quick iterations

#!/bin/bash

go run cmd/cmd_app_init_database/main.go

2. Build alone using the script. Then run the application with the command: bin/cmd_app_init_database.

#!/bin/bash

bash scripts/build/cmd_app_init_database.sh

Database Migrations

Before any of the services are ran, database migrations need to be executed against a postgres database.

Currently, the tool used to do so is called golang migrate.

This script can be used to download the cli tool into a bin folder (will automatically create if it does not exist) in the project.

Requires tar command to be available.

The script accepts the following flags during execution, refer to the releases page for more information.

Flag	Example	Accepted Values	Purpose
-v	v4.18.3		CLI Tool Version
-o	linux	darwin linux windows	Operating System
-a	amd64	amd64 armv7 386	Computer CPU Architecture

Below is a sample execution of the said script to be installed in a linux based machine that uses x86 architecture.

#!/bin/bash

VERSION="v4.18.3"
OS="linux"
ARCH="amd64"

bash scripts/download.sh -v $VERSION -o $OS -a $ARCH

All postgres database migrations are stored in database/psql_migrations/ folder.

Once the cli tool has been installed, you can run it directly or use the helper scripts found here.

Create new migrations

The script accepts the arguments and flags below:

Argument	Example	Purpose
$1	table_name	Required name of migration no spaces Pass as first argument

Flag	Example	Purpose
-m	migrate	Optional. Will default to using the downloaded migrate tool in the bin folder. Path to the migrate cli tool.

Example:

#!/bin/bash

MIGRATION_NAME="test_name"

bash scripts/golang_migrate/create.sh $MIGRATION_NAME

Up migrations

The script accepts the arguments and flags below:

Argument	Example	Purpose
$1	2	Number of up migrations to run. Optional. Will run all remaining up migrations

Flag	Example	Purpose
-m	migrate	Optional. Will default to using the downloaded migrate tool in the bin folder. Path to the migrate cli tool.

Example:

#!/bin/bash

NO_OF_MIGRATIONS="2"

bash scripts/golang_migrate/up.sh $NO_OF_MIGRATIONS

Down migrations

The script accepts the arguments and flags below:

Argument	Example	Purpose
$1	2	Number of down migrations to run. Optional. Will run all remaining down migrations

Flag	Example	Purpose
-m	migrate	Optional. Will default to using the downloaded migrate tool in the bin folder. Path to the migrate cli tool.

Example:

#!/bin/bash

MIGRATION_COUNT="3"

bash scripts/golang_migrate/down.sh $MIGRATION_COUNT

Fix dirty migrations

The script accepts the arguments and flags below:

Argument	Example	Purpose
$1	20250402092853	New migration version

Flag	Example	Purpose
-m	migrate	Optional. Will default to using the downloaded migrate tool in the bin folder. Path to the migrate cli tool.

Example:

#!/bin/bash

MIGRATION_VERSION="20250402092853"

bash scripts/golang_migrate/force.sh $MIGRATION_VERSION

Set Environment Variables

Before any cli tool or service is ran, ensure the required environment variables are already set.

To quickly set the environment variables in the current terminal session, you can use the template bash script. DO NOT edit the script. Instead copy it into a file such as env.sh, edit it appropriately the execute it in a shell/bash terminal e.g. source env.sh.

Environment variables

WEB_SERVICE

Used during startup of web services.

Name/Key	Example	Default	Description	Used In
WEB_SERVICE_CORS_URLS	http://0.0.0.0:5173 https://dap.icipe.org http://0.0.0.0:5173		List of server accepted origns. Separated by space.	web_service
WEB_SERVICE_APP_PREFIX	data_abstraction_platform	data_abstraction_platform	Used for purposes such as prefixing session id keys in the cache database.	web_service
WEB_SERVICE_PORT	5174	5174	Port the web service should run on.	web_service
WEB_SERVICE_BASE_PATH	/dap	/	Base path to listen for http requests.	web_service

PSQL

Used for connecting to a postgres database.

Name/Key	Example	Default	Description	Used In
PSQL_USER	postgres		Postgres database user.	web_service cmd_app_create_super_user cmd_app_init_database golang migrate job_service
PSQL_PASSWORD	postgres2025		Postgres database password	web_service cmd_app_create_super_user cmd_app_init_database golang migrate job_service
PSQL_HOST	10.88.0.100		Postgres database host	web_service cmd_app_create_super_user cmd_app_init_database golang migrate job_service
PSQL_PORT	5432		Postgres database port	web_service cmd_app_create_super_user cmd_app_init_database golang migrate job_service
PSQL_DATABASE	data_abstraction_platform		Postgres database	web_service cmd_app_create_super_user cmd_app_init_database golang migrate
PSQL_SCHEMA	public	public	Postgres database schema to use	web_service cmd_app_create_super_user cmd_app_init_database golang migrate job_service
PSQL_SEARCH_PARAMS	sslmode=disable		Optional List of postgres search params. Separated by space. golang migrate job_service	web_service cmd_app_create_super_user cmd_app_init_database golang migrate job_service

Storage

Will look at S3 config first before local.

Local

Name/Key	Example	Default	Description	Used In
STORAGE_LOCAL_FOLDER_PATH	/mnt/data		Path to folder to store files	web_service job_service

S3

Name/Key	Example	Default	Description	Used In
STORAGE_S3_ENDPOINT	play.min.io 10.88.0.60:9000			web_service job_service
STORAGE_S3_ACCESS_KEY				web_service job_service
STORAGE_S3_SECRET_KEY				web_service job_service
STORAGE_S3_USE_SSL	true	true		web_service job_service
STORAGE_S3_BUCKET	data-abstraction-platform	data-abstraction-platform	Bucket service will use to store its files.	web_service job_service

IAM

Used for user authentication.

Description

Name/Key	Example	Default	Used In
IAM_ENCRYPTION_KEY	9lWLZlzJBCO4xuWe9hrLD97oI87EBdlL		Random sequence of characters Used for encrypting data like JWT tokens. Key MUST be 16, 24, or 32 characters in length ONLY.	web_service
IAM_SIGNING_KEY	9lWLZlzJBCO4xuWe9hrLD97oI87EBdlL		Random sequence of characters Used for signing JWT tokens. Key MUST be 16, 24, or 32 characters in length ONLY.	web_service
IAM_COOKIE_HTTP_ONLY	true	true		web_service
IAM_COOKIE_SAME_SITE	1	3	Accepted options: Value Represents 1 SameSiteDefaultMode 2 SameSiteLaxMode 3 SameSiteStrictMode 4 SameSiteNoneMode	web_service
IAM_COOKIE_SECURE	true	true	Send cookie to server only if connection is secure	web_service
IAM_COOKIE_DOMAIN	localhost		Match domain that hosts the website	web_service

TELEMETRY

Setting up telemetry infrastructure.

Name/Key	Example	Default	Description	Used In
TELEMETRY_LOG_LEVEL	0	1	Level of detail of logs generated. Range Meaning -4 to -1 debug 0 to 3 info 4 to 7 warning 8 error	web_service cmd_app_create_super_user cmd_app_init_database job_service
TELEMETRY_LOG_USE_JSON	false	false	Generate logs in JSON form	web_service cmd_app_create_super_user cmd_app_init_database job_service
TELEMETRY_LOG_COINCISE	true	true	Emit non-detailed logs which excludes info like some http request details.	web_service cmd_app_create_super_user cmd_app_init_database job_service
TELEMETRY_LOG_REQUEST_HEADERS	true	true	Logs should include http request details.	web_service cmd_app_create_super_user cmd_app_init_database job_service
TELEMETRY_LOG_APP_VERSION	true	true	Version of the deployed applications.	web_service cmd_app_create_super_user cmd_app_init_database job_service

miscellaneous

View telemetry:

go run golang.org/x/telemetry/cmd/gotelemetry@latest view