Skip to content

Commit bf595a5

Browse files
cosmicBboyclaude
andcommitted
feat(github): form-encoded deliveries, App installation tokens, installation events
Gaps found by studying how the internal agents (Cally, Nodey, and the SWE-agent webhook receiver) actually use GitHub: - parse() only accepted application/json bodies, but GitHub's Add-webhook form *defaults* to application/x-www-form-urlencoded, which wraps the JSON in a payload= field. The receiver now unwraps both shapes into the same WebhookEvent (and the same dedupe key), so a webhook left on the default content type works instead of 4xx-ing every delivery. - Cally's github_app.py, its acknowledged copy in opencode_chat's mint_gh_token.py, and Nodey's get_github_app_token are three copies of the same GitHub App installation-token minting. mint_installation_token and clone_url now live here under the [auth] extra (PyJWT[crypto]), with the same degrade-to-None-with-a-logged-reason contract and GITHUB_TOKEN/GH_TOKEN fallbacks those agents rely on. - events gains Installation and InstallationRepositories constants, which every GitHub App webhook receives unconditionally. Tests are modeled on the agents' real payloads (the /swe_agent fix issue_comment trigger; Cally's JWT/fallback/outage matrix), the example's --local mode now replays the form-encoded shape too, and the README documents all three additions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NwQixBcyR5va6BC75jaQx3
1 parent 156d378 commit bf595a5

10 files changed

Lines changed: 437 additions & 15 deletions

File tree

plugins/github/README.md

Lines changed: 48 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
# flyteplugins-github
22

3-
Receive GitHub webhooks in Flyte.
3+
Receive GitHub webhooks in Flyte — JSON or form-encoded, GitHub signs both the
4+
same way — plus human review gates on pull requests and GitHub App
5+
installation tokens for agents that clone, push, or open PRs.
46

57
```bash
68
pip install "flyteplugins-github[app]"
@@ -82,8 +84,9 @@ python examples/github_webhooks.py # deploy the receiver to Flyte
8284

8385
`--local` posts this plugin's `SAMPLE_DELIVERY` through the app with FastAPI's
8486
test client, so you see a delivery verified, normalized, and dispatched — plus
85-
an unsigned one refused with a 401, and the same delivery replayed to show the
86-
dedupe key is stable.
87+
an unsigned one refused with a 401, the same delivery replayed to show the
88+
dedupe key is stable, and the same delivery form-encoded (GitHub's default
89+
content type) landing on that same key.
8790

8891
## Setup
8992

@@ -92,14 +95,49 @@ dedupe key is stable.
9295
flyte create secret GITHUB_WEBHOOK_SECRET --value <secret>
9396
```
9497
2. Point GitHub at `<app-url>/webhook/github`, from
95-
repository Settings → Webhooks → Add webhook, content type `application/json`.
98+
repository Settings → Webhooks → Add webhook. Either content type works —
99+
the form's default `application/x-www-form-urlencoded` wraps the JSON in a
100+
`payload=` field and is unwrapped automatically; `application/json` keeps
101+
the deliveries readable in *Recent Deliveries*.
96102

97103
GitHub sends a `ping` when the webhook is created; it is answered automatically, so a green check in *Recent Deliveries* means the app is reachable.
98104

99-
**Verification:** HMAC-SHA256 over the raw body (`X-Hub-Signature-256`).
105+
**Verification:** HMAC-SHA256 over the raw body (`X-Hub-Signature-256`), whichever content type the webhook uses.
100106

101107
Comment and review events fold the comment id into `resource_id`, so two comments on one issue are two events rather than a redelivery of the first.
102108

109+
## GitHub App tokens
110+
111+
Agents that clone, push, or open PRs authenticate best as a GitHub App: hold
112+
no personal access token, mint a short-lived installation token per operation.
113+
Tokens live one hour — plenty for a clone or a `gh pr create`, useless to an
114+
attacker who exfiltrates one from a log:
115+
116+
```python
117+
import asyncio
118+
119+
from flyteplugins.github import clone_url, mint_installation_token
120+
121+
122+
@env.task
123+
async def open_fix_pr(repo: str) -> str:
124+
# One HTTPS round trip; keep it off the event loop.
125+
token = await asyncio.to_thread(mint_installation_token)
126+
url = clone_url(repo, token) # https://x-access-token:<token>@github.com/...
127+
...
128+
```
129+
130+
Configuration comes from three secrets, mounted as environment variables on
131+
the task's environment — `GITHUB_APP_ID`, `GITHUB_APP_INSTALLATION_ID`, and
132+
`GITHUB_APP_PRIVATE_KEY` (the app's PEM key). `GITHUB_TOKEN`/`GH_TOKEN` are
133+
honored as fallbacks so a deployment can migrate one secret at a time, and a
134+
deployment with none of them gets `None` back — with a logged reason — rather
135+
than a crash, so unauthenticated paths keep working.
136+
137+
```bash
138+
pip install "flyteplugins-github[auth]"
139+
```
140+
103141
## Event constants
104142

105143
`events` spells every event this plugin can dispatch, as `str` enums grouped by
@@ -108,6 +146,8 @@ Raw strings still work, for events the constants do not cover yet.
108146

109147
## What this plugin does not do
110148

111-
Call the GitHub API. Use `PyGithub` directly from your tasks — see
112-
`examples/external_saas_integrations`. This plugin owns only the part that is
113-
Flyte's: authenticating an inbound delivery and turning it into a run.
149+
Wrap the GitHub API. Use `PyGithub` directly from your tasks — see
150+
`examples/external_saas_integrations`. This plugin owns the parts every
151+
GitHub agent otherwise duplicates: authenticating an inbound delivery and
152+
turning it into a run, gating a run on a human review, and minting the App
153+
token the outbound side authenticates with.

plugins/github/examples/github_webhooks.py

Lines changed: 11 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,10 +8,13 @@
88
`--local` runs the app through FastAPI's test client and posts this plugin's
99
`SAMPLE_DELIVERY` — a `pull_request.opened` delivery — signed with a throwaway secret. You see the
1010
delivery verified, normalized, and dispatched to a handler, which is the whole
11-
path a real webhook takes.
11+
path a real webhook takes. The same delivery is then replayed form-encoded —
12+
GitHub's *default* content type, the JSON under a `payload=` field — and lands
13+
on the same dedupe key.
1214
1315
To receive real events, deploy it and point GitHub at `<app-url>/webhook/github`
14-
from repository Settings -> Webhooks -> Add webhook, content type `application/json`.
16+
from repository Settings -> Webhooks -> Add webhook. Either content type works;
17+
`application/json` keeps the deliveries readable in *Recent Deliveries*.
1518
1619
Setup for the real thing:
1720
flyte create secret GITHUB_WEBHOOK_SECRET --value <secret>
@@ -21,6 +24,7 @@
2124

2225
import os
2326
import sys
27+
import urllib.parse
2428

2529
import flyte
2630
from flyte.extras.webhooks import WebhookAppEnvironment
@@ -101,6 +105,11 @@ def _try_locally() -> None:
101105
again = client.post("/webhook/github", content=body, headers=build_headers(body, secret))
102106
print(f" {again.status_code} {again.json()}\n")
103107

108+
print("the same delivery form-encoded (GitHub's default content type) — same dedupe_key:")
109+
form = urllib.parse.urlencode({"payload": body.decode()}).encode()
110+
encoded = client.post("/webhook/github", content=form, headers=build_headers(form, secret))
111+
print(f" {encoded.status_code} {encoded.json()}\n")
112+
104113
print("an unsigned delivery is refused:")
105114
bad = client.post("/webhook/github", content=body, headers={})
106115
print(f" {bad.status_code} {bad.json()}\n")

plugins/github/pyproject.toml

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,8 @@ dependencies = ["flyte"]
1212
app = ["fastapi>=0.115", "uvicorn>=0.30"]
1313
# review_pr reads the pull request through PyGithub; webhook-only installs skip it.
1414
review = ["PyGithub>=2"]
15+
# mint_installation_token signs the app JWT with PyJWT; webhook-only installs skip it.
16+
auth = ["PyJWT[crypto]>=2"]
1517

1618
[build-system]
1719
requires = ["setuptools", "setuptools_scm"]
@@ -21,6 +23,7 @@ build-backend = "setuptools.build_meta"
2123
dev = [
2224
"pytest>=8.3.5",
2325
"PyGithub>=2",
26+
"PyJWT[crypto]>=2",
2427
"pytest-asyncio>=0.26.0",
2528
"fastapi>=0.115",
2629
"uvicorn>=0.30",

plugins/github/src/flyteplugins/github/__init__.py

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,14 +48,31 @@ async def gated_merge(repo: str, number: int) -> str:
4848
pull request is `PyGithub`'s job, and this calls it directly rather than
4949
wrapping it — install `flyteplugins-github[review]` for that extra.
5050
51-
Calling the GitHub API for anything else is not this plugin's job either; use
51+
## GitHub App tokens
52+
53+
Agents that clone, push, or open PRs authenticate best as a GitHub App,
54+
minting a short-lived installation token per operation instead of holding a
55+
personal access token:
56+
57+
```python
58+
from flyteplugins.github import clone_url, mint_installation_token
59+
60+
token = mint_installation_token() # GITHUB_APP_ID / _INSTALLATION_ID / _PRIVATE_KEY
61+
url = clone_url("octo/repo", token)
62+
```
63+
64+
It lives here because every agent otherwise carries its own copy of the same
65+
minting logic — install `flyteplugins-github[auth]` for that extra.
66+
67+
Wrapping the GitHub API for anything else is not this plugin's job; use
5268
`PyGithub` from your tasks. See `examples/external_saas_integrations`.
5369
"""
5470

5571
import hashlib
5672
import hmac
5773

5874
from . import events
75+
from ._app_auth import clone_url, mint_installation_token
5976
from ._provider import GitHubProvider, handshake, parse, verify
6077
from ._review import (
6178
DEFAULT_TOKEN_ENV_VAR,
@@ -79,10 +96,12 @@ async def gated_merge(repo: str, number: int) -> str:
7996
"ReviewDecision",
8097
"Verdict",
8198
"build_review_prompt",
99+
"clone_url",
82100
"collect_review_context",
83101
"condition_name_for",
84102
"events",
85103
"handshake",
104+
"mint_installation_token",
86105
"parse",
87106
"parse_review_payload",
88107
"review_pr",
Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
"""GitHub App installation tokens, for agents that clone, push, or open PRs.
2+
3+
The pattern: hold no personal access token. Authenticate as a GitHub App and
4+
mint a short-lived installation token whenever one is needed. Tokens live one
5+
hour — plenty for a clone or a `gh pr create`, useless to an attacker who
6+
exfiltrates one from a log.
7+
8+
This belongs in the plugin because every agent otherwise carries its own copy
9+
of the same fifty lines: sign an RS256 JWT as the app, trade it for an
10+
installation token, splice it into a clone URL. The webhook side of a GitHub
11+
agent already imports this package, so the auth side comes from it too.
12+
13+
Inputs, all injected as Flyte secrets (or passed explicitly):
14+
15+
GITHUB_APP_ID the app's numeric id
16+
GITHUB_APP_INSTALLATION_ID the installation's numeric id
17+
GITHUB_APP_PRIVATE_KEY the app's PEM private key
18+
19+
`GITHUB_TOKEN` and `GH_TOKEN` are honored as fallbacks so a deployment can
20+
migrate one secret at a time; once the app secrets exist the fallback never
21+
fires.
22+
23+
`PyJWT[crypto]` signs the app JWT, so a webhook-only install stays lean:
24+
25+
```bash
26+
pip install "flyteplugins-github[auth]"
27+
```
28+
"""
29+
30+
from __future__ import annotations
31+
32+
import json
33+
import logging
34+
import os
35+
import time
36+
import urllib.request
37+
from typing import Any
38+
39+
logger = logging.getLogger(__name__)
40+
41+
GITHUB_API = "https://api.github.com"
42+
#: App tokens authenticate with this literal username in clone URLs.
43+
GIT_USERNAME = "x-access-token"
44+
45+
#: Environment variables the app credentials default to.
46+
DEFAULT_APP_ID_ENV = "GITHUB_APP_ID"
47+
DEFAULT_INSTALLATION_ID_ENV = "GITHUB_APP_INSTALLATION_ID"
48+
DEFAULT_PRIVATE_KEY_ENV = "GITHUB_APP_PRIVATE_KEY"
49+
50+
#: Plain-token fallbacks, checked in order when the app credentials are absent.
51+
FALLBACK_TOKEN_ENVS = ("GITHUB_TOKEN", "GH_TOKEN")
52+
53+
54+
def _post_json(url: str, *, bearer: str) -> dict[str, Any]:
55+
"""POST to the GitHub API with a bearer credential and return the JSON."""
56+
request = urllib.request.Request(
57+
url,
58+
method="POST",
59+
headers={"Authorization": f"Bearer {bearer}", "Accept": "application/vnd.github+json"},
60+
)
61+
with urllib.request.urlopen(request, timeout=30) as response:
62+
return json.loads(response.read().decode("utf-8"))
63+
64+
65+
def mint_installation_token(
66+
*,
67+
app_id: str | None = None,
68+
installation_id: str | None = None,
69+
private_key: str | None = None,
70+
) -> str | None:
71+
"""A fresh installation token, or None with a logged reason.
72+
73+
None means "proceed unauthenticated or not at all" — treat it the way a
74+
missing token is treated today, so a half-configured deployment degrades
75+
instead of crashing. Synchronous, one HTTPS round trip: call through
76+
`asyncio.to_thread` from handlers and other async code.
77+
78+
Args:
79+
app_id: The app's numeric id; otherwise read from `GITHUB_APP_ID`.
80+
installation_id: The installation's numeric id; otherwise read from
81+
`GITHUB_APP_INSTALLATION_ID`.
82+
private_key: The app's PEM private key; otherwise read from
83+
`GITHUB_APP_PRIVATE_KEY`.
84+
"""
85+
app_id = app_id or os.environ.get(DEFAULT_APP_ID_ENV)
86+
installation_id = installation_id or os.environ.get(DEFAULT_INSTALLATION_ID_ENV)
87+
private_key = private_key or os.environ.get(DEFAULT_PRIVATE_KEY_ENV)
88+
89+
if not (app_id and installation_id and private_key):
90+
for env in FALLBACK_TOKEN_ENVS:
91+
fallback = os.environ.get(env)
92+
if fallback:
93+
logger.info("GitHub App credentials not set; using %s fallback", env)
94+
return fallback
95+
logger.warning(
96+
"Neither the %s/%s/%s secrets nor a fallback token (%s) are set; "
97+
"authenticated GitHub operations will be skipped",
98+
DEFAULT_APP_ID_ENV,
99+
DEFAULT_INSTALLATION_ID_ENV,
100+
DEFAULT_PRIVATE_KEY_ENV,
101+
"/".join(FALLBACK_TOKEN_ENVS),
102+
)
103+
return None
104+
105+
try:
106+
import jwt # PyJWT[crypto]
107+
except ModuleNotFoundError as exc: # pragma: no cover - depends on extras
108+
raise ModuleNotFoundError(
109+
"PyJWT is not installed. Install 'flyteplugins-github[auth]' to mint GitHub App tokens."
110+
) from exc
111+
112+
now = int(time.time())
113+
# iat is backdated 60s because GitHub rejects JWTs it considers issued in
114+
# the future, and clocks drift.
115+
app_jwt = jwt.encode({"iat": now - 60, "exp": now + 600, "iss": app_id}, private_key, algorithm="RS256")
116+
try:
117+
data = _post_json(f"{GITHUB_API}/app/installations/{installation_id}/access_tokens", bearer=app_jwt)
118+
return data["token"]
119+
except Exception as exc: # callers degrade, they don't crash
120+
logger.warning("Could not mint a GitHub App installation token: %s", exc)
121+
return None
122+
123+
124+
def clone_url(repo: str, token: str | None = None) -> str:
125+
"""An https clone URL for `repo` ("owner/name"), authenticated when a token is given."""
126+
if token:
127+
return f"https://{GIT_USERNAME}:{token}@github.com/{repo}.git"
128+
return f"https://github.com/{repo}.git"

plugins/github/src/flyteplugins/github/_provider.py

Lines changed: 44 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,20 @@
1-
"""GitHub webhook verification and payload normalization."""
1+
"""GitHub webhook verification and payload normalization.
2+
3+
GitHub delivers the same payload in two body shapes, chosen per webhook in the
4+
*Add webhook* form and signed the same way:
5+
6+
* content type `application/json` — the JSON is the body;
7+
* content type `application/x-www-form-urlencoded` — the form's *default* —
8+
the JSON arrives under a `payload=` form field.
9+
10+
`verify` covers both, since the HMAC signs the raw body regardless of encoding.
11+
`parse` normalizes both into the same `WebhookEvent`, so a webhook left on the
12+
default content type still works.
13+
"""
214

315
from __future__ import annotations
416

17+
import urllib.parse
518
from typing import Any, ClassVar, Mapping
619

720
from flyte.extras.webhooks import (
@@ -23,6 +36,30 @@ def verify(body: bytes, headers: Mapping[str, str], secret: str) -> bool:
2336
return constant_time_equals(hex_hmac_sha256(secret, body), signature.removeprefix("sha256="))
2437

2538

39+
def _form_payload(body: bytes) -> dict[str, Any] | None:
40+
"""Decode a form-encoded delivery's `payload` field, or None when the body is JSON.
41+
42+
GitHub's *Add webhook* form defaults the content type to
43+
`application/x-www-form-urlencoded`, which wraps the JSON in a `payload=`
44+
form field. Sniffing the body rather than trusting Content-Type keeps
45+
`parse` a pure function of the delivery, which is what the conformance
46+
harness replays.
47+
"""
48+
if body[:1] in (b"{", b"["):
49+
return None
50+
try:
51+
decoded = body.decode("utf-8")
52+
except UnicodeDecodeError:
53+
return None
54+
if "=" not in decoded.split("&", 1)[0]:
55+
return None
56+
fields = {key: values[0] for key, values in urllib.parse.parse_qs(decoded, keep_blank_values=True).items()}
57+
raw = fields.get("payload")
58+
if raw is None:
59+
raise SignatureError("form-encoded delivery carries no `payload` field")
60+
return json_body(raw.encode("utf-8"))
61+
62+
2663
def handshake(headers: Mapping[str, str], body: bytes) -> dict[str, Any] | None:
2764
"""Answer the `ping` GitHub sends when a webhook is created."""
2865
if lower_headers(headers).get("x-github-event") == "ping":
@@ -31,12 +68,13 @@ def handshake(headers: Mapping[str, str], body: bytes) -> dict[str, Any] | None:
3168

3269

3370
def parse(headers: Mapping[str, str], body: bytes) -> WebhookEvent:
34-
"""Normalize a GitHub delivery into a `WebhookEvent`."""
71+
"""Normalize a GitHub delivery — JSON or form-encoded — into a `WebhookEvent`."""
3572
lowered = lower_headers(headers)
3673
event_type = lowered.get("x-github-event")
3774
if not event_type:
3875
raise SignatureError("missing X-GitHub-Event header")
39-
payload = json_body(body)
76+
form = _form_payload(body)
77+
payload = form if form is not None else json_body(body)
4078

4179
repo = payload.get("repository") or {}
4280
issue_or_pr = payload.get("pull_request") or payload.get("issue") or {}
@@ -77,6 +115,9 @@ class GitHubProvider(Provider):
77115
app_env = WebhookAppEnvironment(name="webhooks", providers=[GitHubProvider()])
78116
```
79117
118+
Either content type in GitHub's *Add webhook* form works: `application/json`
119+
and the default `application/x-www-form-urlencoded` normalize identically.
120+
80121
`WebhookAppEnvironment` mounts `default_secret_env` for you, so it does not
81122
need naming again in `secrets=`.
82123

0 commit comments

Comments
 (0)