Implementing data proxy functionality on GKE (Google Cloud)

[eecavanna]

Today, on NERSC Spin, each of our Data Portal instances (e.g. production, development) includes a portal-data-proxy pod that runs the nginxinc/nginx-unprivileged:stable container image. The containers in the pod mount this NERSC filesystem path (/global/cfs/cdirs/m3408/ficus/pipeline_products) to a local filesystem path (/usr/share/nginx/html) within the container.

Given that a container running on Google Cloud doesn't have access to the NERSC filesystem, @sanjaypjana and I cannot use the configuration as is (in the context of a Google Kubernetes nodes, /global/cfs/cdirs/m3408/ficus/pipeline_products is an arbitrary file path—not something that contains NMDC data).

I'm creating this discussion thread in an attempt to (a) learn more about the role of the portal-data-proxy workload in our current Data Portal instances; and (b) determine a path forward for me and @sanjaypjana to follow in order to achieve equivalent functionality (assuming it is necessary functionality) on the instance of the Data Portal we deploy to Google Cloud.

I'll tag people that I think will be interested in this conversation:
@shreddd , @sanjaypjana , @eecavanna , @jeffbaumes , @naglepuff , @pkalita-lbl

[jeffbaumes]

I mentioned some of this in the meeting 2 weeks ago, but I'll restate it here. I believe the proxy is there solely to host the data from NERSC to the web. We use nginx mod zip to conveniently make streaming zip downloads. This requires all file URLs to be hosted under a single domain (since files could be coming from multiple sources), which also means that all files need to be hosted somewhere accessible by URLs.

This file shows the config proxies for the two data-hosting locations we are using, as well as the use of mod_zip in the first line, in order to put all the pieces together.

Perhaps just that hosting nginx would need to remain on Spin (because something with access to that filesystem has to run the file hosting service) and everything else could be moved to the cloud.