This project hosts the code for the data ingestor module of real data management workflow in Synthema. Data ingestor component is responsible for:
- Ingesting a new dataset and saving it to a MinIO bucket
- Deleting an existing dataset from the MinIO bucket
The data-ingestor module is structured in the following folders:
- The folder src provides utilities, datasets, models, fastapi, requirements and Dockerfile
- The folder k8s includes kubernetes manifests
- The folder jenkins contains the Jenkinsfile to run unit and functional tests.
- Create the network
docker network create mynetwork
- Build the Dockerfile in src/Dockerfile:
docker build -t data-ingestor-image
- Run the image:
docker run --network mynetowrk --name data-ingestor-container-name -p 8002:8002 data-ingestor-image
- Go to http://localhost:8002/docs in your browser and upload the AML_DATA_ES.csv in Example_data (after uploading a valid schema).
N.B. Remember to put data-annotation-container-name and data-catalogue-container-name in ANNOTATION_ENDPOINT and CATALOGUE_ENDPOINT respectively in main.py
This project is licensed under the MIT License.
This project extends and uses the following Open Softwares, which are compliant with MIT License:
- FastAPI: MIT License
- Pandas: BSD License
- psycopg2-binary: PostgreSQL License
- Uvicorn: BSD License
- python-multipart: MIT License
- pytest: MIT License
- jsonschema: MIT License
- sqlalchemy: MIT License
- sqlmodel: MIT License
- requests: Apache 2.0 License
- httpx: BSD License
- MinIO: GNU AGPL v3