Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 24 additions & 6 deletions app/_event_gateway_policies/mask-fields-consume/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,9 +152,13 @@ columns:
key: description
rows:
- failure: "`skip`"
description: Doesn't deliver the record to the client.
description: |
Doesn't deliver the record to the client.
Recommended for most situations.
- failure: "`error`"
description: Doesn't deliver the batch to the client.
description: |
Doesn't deliver the batch to the client.
Use `error` sparingly, because an error on a batch makes clients stop at the problematic offset, and an operator must intervene to skip it.
- failure: "`passthrough`"
description: Delivers the record to the client without a mask.
- failure: "`mark`"
Expand All @@ -163,10 +167,24 @@ rows:
The value of the header is the reason for the failure.
{% endtable %}

For a redaction policy:
* In most situations, use `skip`.
* Use `error` sparingly, because an error on a batch makes clients stop at the problematic offset, and an operator must intervene to skip it.
* Don't use `passthrough` or `mark`, because both deliver the original value to the client.
Set the parent [Schema Validation Consume policy](/event-gateway/policies/schema-validation-consume/) to `skip` or `error` to use this policy.

The parent policy must not use `passthrough` or `mark`.
With these failure modes, the parent delivers the record without parsing it. The Mask Fields Consume policy then never runs, and the client receives the fields unmasked.

## Selecting fields from schema metadata

Instead of listing paths one by one, the `paths` entry can take a single expression that reads the record's schema metadata and returns the tagged field paths. For example, with a Confluent Schema Registry, an expression can return every field path that carries a `PII` tag:

```
paths: 'record.value.schema.metadata.tags["PII"]'
```

This is the recommended way to mask fields. The mask follows the schema, so when the schema evolves and a new field gets tagged, it is masked without any policy change.

In this case, the schema becomes the input that decides which fields are masked, so anyone who can edit the schema tags can change the masking. Keep schema registry write access restricted to trusted roles.

See the full walkthrough in [Mask Kafka message fields selected by schema registry tags](/event-gateway/mask-kafka-message-fields-from-schema/).

## Policy order

Expand Down
23 changes: 21 additions & 2 deletions app/_event_gateway_policies/mask-fields-produce/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,9 @@ columns:
key: description
rows:
- failure: "`reject`"
description: Rejects the record batch that holds the record.
description: |
Rejects the record batch that holds the record.
Recommended for most situations.
- failure: "`passthrough`"
description: Sends the record to the backend cluster without a mask.
- failure: "`mark`"
Expand All @@ -155,7 +157,24 @@ rows:
The value of the header is the reason for the failure.
{% endtable %}

For a redaction policy, use `reject`. Don't use `passthrough` or `mark`, because both write the original value to the cluster.
Set the parent [Schema Validation Produce policy](/event-gateway/policies/schema-validation-produce/) to `reject` to use this policy.

The parent policy must not use `passthrough` or `mark`.
With these failure modes, the parent sends the record without parsing it. The Mask Fields Produce policy then never runs, and the cluster stores the fields unmasked.

## Selecting fields from schema metadata

Instead of listing paths one by one, the `paths` entry can take a single expression that reads the record's schema metadata and returns the tagged field paths. For example, with a Confluent Schema Registry, an expression can return every field path that carries a `PII` tag:

```
paths: 'record.value.schema.metadata.tags["PII"]'
```

This is the recommended way to mask fields. The mask follows the schema, so when the schema evolves and a new field gets tagged, it is masked without any policy change.

In this case, the schema becomes the input that decides which fields are masked, so anyone who can edit the schema tags can change the masking. Keep schema registry write access restricted to trusted roles.

See the full walkthrough in [Mask Kafka message fields selected by schema registry tags](/event-gateway/mask-kafka-message-fields-from-schema/).

## Policy order

Expand Down
Loading