Skip to content

Commit 29e09d3

Browse files
committed
feat: render torch.profiler GPU profiles in the Flyte report
Add torch_profile() to flyteplugins-pytorch: a context manager (sync and async) that runs torch.profiler over a region of a task body and renders the result into a Flyte report tab — summary tiles, top-ops tables, and an interactive Perfetto timeline (the gzipped chrome trace is base64- embedded and pushed into a lazy ui.perfetto.dev iframe via its postMessage deep-link API), plus a trace download button. Traces over an embed cap are uploaded to blob storage and linked instead. Rendering is best-effort and never fails the task; body exceptions are never suppressed. Verified end-to-end on the demo cluster (L4 GPU). Claude-Session: https://claude.ai/code/session_01FoFxbPpsYcWMny59J63BZM Signed-off-by: Kevin Su <pingsutw@apache.org>
1 parent c521246 commit 29e09d3

4 files changed

Lines changed: 597 additions & 1 deletion

File tree

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
"""GPU profiling with torch.profiler, rendered in the Flyte report.
2+
3+
The task runs a few matmul+add steps under `torch_profile()`; the resulting report tab shows
4+
summary tables and an interactive Perfetto timeline (ProfilerStep#N / matmul_add / aten::mm
5+
spans on CPU and CUDA tracks) right in the Flyte UI.
6+
7+
Run:
8+
make dist && FLYTE_PLUGIN_DIST=plugins/pytorch make dist-plugins
9+
flyte run plugins/pytorch/examples/profile_matmul.py profile_matmul
10+
"""
11+
12+
import flyte
13+
14+
from flyteplugins.pytorch import torch_profile
15+
16+
image = (
17+
flyte.Image.from_debian_base(name="torch-profile", python_version=(3, 12))
18+
.with_pip_packages("torch")
19+
.with_local_v2()
20+
.with_local_v2_plugins(["flyteplugins-pytorch"])
21+
)
22+
23+
env = flyte.TaskEnvironment(
24+
name="torch-profile",
25+
image=image,
26+
resources=flyte.Resources(cpu="4", memory="16Gi", gpu="L4:1"),
27+
)
28+
29+
30+
@env.task(report=True)
31+
def profile_matmul(steps: int = 8, n: int = 4096) -> str:
32+
import torch
33+
from torch.profiler import record_function, schedule
34+
35+
device = "cuda" if torch.cuda.is_available() else "cpu"
36+
a = torch.randn(n, n, device=device)
37+
b = torch.randn(n, n, device=device)
38+
39+
# repeat=1: keep the completed profiling cycle — without it, stepping past the active
40+
# window starts a new cycle and clears the collected events (empty report).
41+
with torch_profile(profile_memory=True, schedule=schedule(wait=1, warmup=2, active=4, repeat=1)) as prof:
42+
for _ in range(steps):
43+
with record_function("matmul_add"):
44+
c = a @ b + a
45+
if device == "cuda":
46+
torch.cuda.synchronize()
47+
prof.step()
48+
49+
return f"done on {device}: {c.shape}"
50+
51+
52+
if __name__ == "__main__":
53+
flyte.init_from_config()
54+
run = flyte.run(profile_matmul)
55+
print(run.url)
Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1-
__all__ = ["Elastic"]
1+
__all__ = ["Elastic", "torch_profile"]
22

3+
from flyteplugins.pytorch._profile import torch_profile
34
from flyteplugins.pytorch.task import Elastic

0 commit comments

Comments
 (0)