Clinical Trials Intelligence Engine
The scope of this project moves like:
Clinical CSV Files --> Landing (S3) : raw files will be placed here it acts as inbound --> Working (S3) : here when are processing the file they should move here as well archiving the file from Landing (S3) to processed (s3) duing this we consuming 2 scripts file_read.py and file_movement.py
--> once we have file in working (s3) --> Airflow DAG --> PySpark Cleaning --> Data Validation --> then we put the data into staging (truccate and load) --> INT (SCD1 full load) --> DWH (SCD2 full load) --> {tbd}: we build sample dashboard/ ML models / Implement AI agents