Deletion of large datasets takes a while #1763

If a user has a large dataset worth TBs stored in any ObjectStore like S3, GCS, etc. its deletion takes a while and the user is still able to see the dataset in the UI.

Presently, the deletion API tries to delete the dataset files from object store and then respond back to the UI with a 200 OK

Proposed solution: make a .tombstone file while marks a dataset for deletion and prevents any new actions from being performed on that dataset.

For example

• User sends a deletion request for dataset A
• Query node creates a .tombstone file in the root of that dataset and informs other ingest nodes of the same and sends back a 200 OK to the user. It also kicks off a background task in charge of deletion.
• All the nodes delete references to the dataset from their memory while the query node keeps deleting in the background
• During live ingestion, the ingestor should be able to deny incoming requests saying the dataset is being deleted
• Once the deletion is complete (the background job returns 200 OK), the query node can delete the .tombstone file and the folder
• If the query node restarts in the middle of deletion, while loading the datasets it will read the .tombstone file and start deleting in background without loading the dataset