[Feature]: Migrating Cron policies to Reconciler

[csirmazbendeguz]

Scheduled jobs

For example:

• Keeping N latest version of a record (prune job)
• Remove all records older than 10 days that have no signatures (prune job)
• Remove all records with severity >= HIGH older than 10 days (prune job)
• Synchronize records (sync job).
• Publish all records (publish job).

We already have a basic version of this (see #2128).

The issue is it's quite limited in functionality and only works on Kubernetes.
We don't want to depend on external schedulers.
We need to migrate the Kubernetes CronJobs to the reconciler.

Storage

Where to store the scheduled jobs?

Reconciler config

PROS:

• Easy to implement.

CONS:

• The jobs are coupled to the reconciler. If the jobs change, reconciler needs a restart.
  It's probably fine though, it's not a big problem.

Solutions

KISS solution

The simple solution is to create some pre-defined reconciler jobs.
So, for example, the reconciler config would have a pre-defined entry prune-untrusted.

PROS:

• Easy to implement. It's a good first step.

CONS:

• Not as flexible - it's a step back from Kubernetes CronJobs in terms of flexibility.

ETL solution

ETL (Extract-Transform-Load) is a pattern for data integration.

The Kubernetes CronJobs already resemble an ETL:

• "Extract" =match
• "Load" =action

dir/install/charts/dirctl/values.yaml

Lines 105 to 111 in ad1c172

	prune-untrusted:
	enabled: false
	schedule: '*/30 * * * *'
	action: prune
	match:
	trusted: false
	scan-severity: MEDIUM

We could implement something similar in the reconciler.

PROS:

• Flexible. It gives admins freedom to use logic we can't predict.

CONS:

• It's more complex to implement.

OPA

Rego -> SQL

OPA supports compiling Rego policies to SQL.
We could use this in ETLs. One limitation of the SQL compiler is it can't JOIN tables.
In theory, we could implement a compiler for dirctl search flags as well, which is more flexible.

With this approach, we could re-use Rego policies for ETLs:

• Extract = Query the data with the SQL compiled from Rego. Alternatively, use dirctl search with flags compiled from Rego.
• Transform = Can use Rego for filtering (similar to trigger-based policies).
• Load = The action to perform, e.g. prune.

In practice, I'm not sure if this could work, there may be limitations. But I think it's worth a try as it would tie everything together.