Skip to content

Commit 897350a

Browse files
authored
feat!: implement experimentation tracking (#37)
* build: require flagsmith >=5.5 for the experimentation API Signed-off-by: wadii <wadii.zaim@flagsmith.com> * feat: populate reason, variant and flag_metadata on resolutions Signed-off-by: wadii <wadii.zaim@flagsmith.com> * feat: support transient identities via the evaluation context Signed-off-by: wadii <wadii.zaim@flagsmith.com> * fix!: track() crashed on flagsmith >=5.4 and hid value in metadata Signed-off-by: wadii <wadii.zaim@flagsmith.com> * feat: route feature_flag.exposure tracking events to Flagsmith Signed-off-by: wadii <wadii.zaim@flagsmith.com> * feat: add FlagsmithExposureHook for evaluation-coupled exposures Signed-off-by: wadii <wadii.zaim@flagsmith.com> * feat: re-export provider, hook and exposure constant from package root Signed-off-by: wadii <wadii.zaim@flagsmith.com> * docs: document exposure tiers and fix broken tracking example Signed-off-by: wadii <wadii.zaim@flagsmith.com> * feat: emit SPLIT for multivariate assignments and gate exposures on it Signed-off-by: wadii <wadii.zaim@flagsmith.com> * test: pin the cross-SDK wire contract with literal assertions Signed-off-by: wadii <wadii.zaim@flagsmith.com> * feat: remove ia slop Signed-off-by: wadii <wadii.zaim@flagsmith.com> * fix: warn and drop exposures carrying a non-string variant Signed-off-by: wadii <wadii.zaim@flagsmith.com> * ci: run the test suite against the minimum supported flagsmith Signed-off-by: wadii <wadii.zaim@flagsmith.com> * refactor: drop the hook dedupe LRU in favor of SDK-side deduplication Signed-off-by: wadii <wadii.zaim@flagsmith.com> * feat: forward engine evaluation reasons from flagsmith >=6.2 Signed-off-by: wadii <wadii.zaim@flagsmith.com> * feat: map the engine DEFAULT reason to STATIC Signed-off-by: wadii <wadii.zaim@flagsmith.com> --------- Signed-off-by: wadii <wadii.zaim@flagsmith.com>
1 parent 20c9f20 commit 897350a

11 files changed

Lines changed: 1387 additions & 124 deletions

File tree

‎.github/workflows/pytest.yml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,3 +37,28 @@ jobs:
3737
3838
- name: Run Tests
3939
run: poetry run pytest
40+
41+
test-sdk-floor:
42+
runs-on: ubuntu-latest
43+
name: Pytest against the minimum supported flagsmith version
44+
45+
steps:
46+
- name: Cloning repo
47+
uses: actions/checkout@v4
48+
with:
49+
fetch-depth: 0
50+
51+
- name: Set up Python
52+
uses: actions/setup-python@v5
53+
with:
54+
python-version: '3.12'
55+
56+
- name: Install Dependencies
57+
run: |
58+
python -m pip install --upgrade pip
59+
pip install poetry
60+
poetry install --no-root
61+
poetry run pip install flagsmith==5.5.0
62+
63+
- name: Run Tests
64+
run: poetry run pytest

‎README.md‎

Lines changed: 68 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -60,40 +60,85 @@ provider = FlagsmithProvider(
6060
The provider can then be used with the OpenFeature client as per
6161
[the documentation](https://openfeature.dev/docs/reference/concepts/evaluation-api#setting-a-provider).
6262

63-
### Tracking
63+
### Tracking and experimentation
6464

65-
The provider supports the [OpenFeature tracking API](https://openfeature.dev/specification/sections/tracking/), which lets you associate user actions with feature flag evaluations for experimentation.
65+
The provider supports the [OpenFeature tracking API](https://openfeature.dev/specification/sections/tracking/) (an experimental OpenFeature capability), which lets you record custom events and flag **exposures** for experimentation.
6666

67-
Tracking requires pipeline analytics to be enabled on the **Flagsmith client** (available from `flagsmith` version 5.2.0). The provider acts as a thin delegate — all buffering and flushing is managed by the client.
67+
Tracking requires events to be enabled on the **Flagsmith client** (`flagsmith` ≥5.5). The provider acts as a thin delegate — all buffering and flushing is managed by the client.
6868

6969
```python
70-
from flagsmith import Flagsmith, PipelineAnalyticsConfig
70+
from flagsmith import Flagsmith
7171
from openfeature import api
72-
from openfeature.evaluation_context import EvaluationContext
73-
from openfeature.track import TrackingEventDetails
74-
from openfeature_flagsmith.provider import FlagsmithProvider
72+
from openfeature_flagsmith import FlagsmithProvider
7573

76-
# Enable pipeline analytics on the Flagsmith client
7774
client = Flagsmith(
7875
environment_key="your-environment-key",
79-
pipeline_analytics_config=PipelineAnalyticsConfig(
80-
analytics_server_url="https://analytics-collector.flagsmith.com/",
81-
max_buffer_items=1000, # optional, default 1000
82-
flush_interval_seconds=10, # optional, default 10s
83-
),
76+
enable_events=True,
8477
)
8578

86-
api.set_provider(FlagsmithProvider(client=client))
79+
provider = FlagsmithProvider(client=client)
80+
api.set_provider(provider)
8781
of_client = api.get_client()
82+
```
83+
84+
If events are not enabled on the Flagsmith client, all tracking calls are silently dropped.
85+
86+
#### Recording exposures
87+
88+
An **exposure** marks an identity as having experienced an experiment variant. Exposures are never recorded automatically: evaluating a flag does not expose anyone. There are three ways to record them, from most to least recommended.
89+
90+
**1. The exposure hook (recommended).** Attach `FlagsmithExposureHook` to the evaluations that *are* your experiment — attaching the hook is the experiment declaration:
8891

89-
# Flag evaluations are tracked automatically — no extra code needed
90-
variant = of_client.get_string_value(
91-
"checkout-variant",
92+
```python
93+
from openfeature.evaluation_context import EvaluationContext
94+
from openfeature.flag_evaluation import FlagEvaluationOptions
95+
from openfeature_flagsmith import FlagsmithExposureHook
96+
97+
hook = FlagsmithExposureHook(provider)
98+
99+
details = of_client.get_string_details(
100+
"my_experiment_flag",
92101
"control",
102+
EvaluationContext(targeting_key="user-123"),
103+
FlagEvaluationOptions(hooks=[hook]),
104+
)
105+
```
106+
107+
The hook records an exposure only when the flag resolved with a variant and reason `SPLIT` — a multivariate percentage-split assignment (enabled, identified, not offline). With `flagsmith` ≥6.2 resolution reasons come from the Flagsmith engine (e.g. `SPLIT; weight=30`; the engine's `DEFAULT` maps to `STATIC`); on older SDKs or APIs the provider infers them. Repeated evaluations are safe: duplicate exposures are deduplicated downstream.
108+
109+
**2. Explicit `track()`.** Use the reserved `feature_flag.exposure` event name when you need to record an exposure decoupled from evaluation:
110+
111+
```python
112+
from openfeature.track import TrackingEventDetails
113+
from openfeature_flagsmith import EXPOSURE_TRACKING_EVENT
114+
115+
# With an explicit variant: sent as rendered.
116+
of_client.track(
117+
EXPOSURE_TRACKING_EVENT,
118+
evaluation_context=EvaluationContext(targeting_key="user-123"),
119+
tracking_event_details=TrackingEventDetails(
120+
attributes={"flag_key": "my_experiment_flag", "variant": "treatment"}
121+
),
122+
)
123+
124+
# Without a variant: the provider resolves the flag for the targeting key and
125+
# records the exposure only if the flag exists, is enabled and has a variant.
126+
of_client.track(
127+
EXPOSURE_TRACKING_EVENT,
93128
evaluation_context=EvaluationContext(targeting_key="user-123"),
129+
tracking_event_details=TrackingEventDetails(
130+
attributes={"flag_key": "my_experiment_flag"}
131+
),
94132
)
133+
```
134+
135+
**3. The native Flagsmith client.** `client.get_experiment_flag(...)` / `client.track_exposure_event(...)` work as documented in the [Flagsmith docs](https://docs.flagsmith.com/) and share the same event pipeline.
95136

96-
# Track a custom event explicitly
137+
#### Custom events
138+
139+
Any other event name is forwarded as a plain Flagsmith event. `TrackingEventDetails.value` must be numeric and is sent as the event value; `attributes` become event metadata; context traits are attached to the event.
140+
141+
```python
97142
of_client.track(
98143
"purchase",
99144
evaluation_context=EvaluationContext(
@@ -107,7 +152,11 @@ of_client.track(
107152
)
108153
```
109154

110-
If `pipeline_analytics_config` is not set on the Flagsmith client, calls to `track()` are silently ignored.
155+
#### Caveats
156+
157+
- **Anonymous contexts**: exposures require a `targeting_key`; without one they are skipped (logged at info).
158+
- **Reserved names**: event names starting with `$` are reserved for Flagsmith system events and are dropped with a warning — use `EXPOSURE_TRACKING_EVENT` to record exposures.
159+
- **Transient identities** (Python provider only, remote evaluation only): set the context attribute `"transient": True` to evaluate an identity without persisting it. The variant-less exposure path honors it too.
111160

112161
### Evaluation Context
113162

‎openfeature_flagsmith/__init__.py‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
from openfeature_flagsmith.hooks import FlagsmithExposureHook
2+
from openfeature_flagsmith.provider import FlagsmithProvider
3+
from openfeature_flagsmith.tracking import EXPOSURE_TRACKING_EVENT
4+
5+
__all__ = [
6+
"EXPOSURE_TRACKING_EVENT",
7+
"FlagsmithExposureHook",
8+
"FlagsmithProvider",
9+
]

‎openfeature_flagsmith/hooks.py‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
import logging
2+
import typing
3+
4+
from openfeature.flag_evaluation import FlagEvaluationDetails, Reason
5+
from openfeature.hook import Hook, HookContext, HookHints
6+
from openfeature.track import TrackingEventDetails
7+
8+
from openfeature_flagsmith.tracking import EXPOSURE_TRACKING_EVENT
9+
10+
if typing.TYPE_CHECKING:
11+
from openfeature_flagsmith.provider import FlagsmithProvider
12+
13+
logger = logging.getLogger(__name__)
14+
15+
16+
def _is_split_reason(reason: typing.Union[str, Reason, None]) -> bool:
17+
# The engine annotates reasons ("SPLIT; weight=30"); compare the
18+
# leading token.
19+
if reason is None:
20+
return False
21+
return str(reason).split(";", 1)[0].strip() == Reason.SPLIT.value
22+
23+
24+
class FlagsmithExposureHook(Hook):
25+
"""
26+
Records a Flagsmith exposure as a side effect of a flag evaluation::
27+
28+
hook = FlagsmithExposureHook(provider)
29+
client.get_string_details(
30+
"my_experiment_flag",
31+
"control",
32+
context,
33+
FlagEvaluationOptions(hooks=[hook]),
34+
)
35+
36+
Attaching the hook at a call site is the experiment declaration:
37+
evaluations without it never record exposures. Exposures only fire for
38+
flags resolved with a variant and reason ``SPLIT``; duplicate exposures
39+
are deduplicated downstream.
40+
"""
41+
42+
def __init__(self, provider: "FlagsmithProvider") -> None:
43+
self._provider = provider
44+
45+
def after(
46+
self,
47+
hook_context: HookContext,
48+
details: FlagEvaluationDetails,
49+
hints: HookHints,
50+
) -> None:
51+
# An uncaught after-hook error would flip the evaluation to ERROR.
52+
try:
53+
variant = details.variant
54+
if not isinstance(variant, str):
55+
return
56+
if not _is_split_reason(details.reason):
57+
logger.debug(
58+
'Exposure for "%s" skipped: resolution reason is %s, not SPLIT.',
59+
details.flag_key,
60+
details.reason,
61+
)
62+
return
63+
self._provider.track(
64+
EXPOSURE_TRACKING_EVENT,
65+
hook_context.evaluation_context,
66+
TrackingEventDetails(
67+
attributes={"flag_key": details.flag_key, "variant": variant}
68+
),
69+
)
70+
except Exception:
71+
logger.warning(
72+
'Failed to record the exposure for "%s".',
73+
details.flag_key,
74+
exc_info=True,
75+
)

0 commit comments

Comments
 (0)