Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
0eb5554
add sdk quality assureance skill
chunyu3 Jul 30, 2026
8551591
iterative assurance sdk quality
chunyu3 Jul 30, 2026
c576be9
Potential fix for pull request finding
chunyu3 Jul 30, 2026
ca96c86
Potential fix for pull request finding
chunyu3 Jul 30, 2026
3921116
Potential fix for pull request finding
chunyu3 Jul 30, 2026
cd4392c
update the reference
chunyu3 Jul 30, 2026
0817aca
remove breaking change resolve from generate-sdk-locally skill
chunyu3 Aug 3, 2026
ddde45a
format the breaking change display
chunyu3 Aug 3, 2026
79382bc
remove unused tool
chunyu3 Aug 3, 2026
1e02cce
format and resolve comments
chunyu3 Aug 4, 2026
6b328d6
add evals
chunyu3 Aug 4, 2026
220ebb9
refine the instruction
chunyu3 Aug 4, 2026
12050f5
typo
chunyu3 Aug 10, 2026
d135bb3
add eval test
chunyu3 Aug 10, 2026
1541f7b
resolve comments
chunyu3 Aug 20, 2026
fa17b5d
format eval yaml
chunyu3 Aug 20, 2026
c600f47
resolve comments
chunyu3 Aug 21, 2026
99bdc36
Sync .github/skills directory with azure-sdk-tools repository for Too…
azure-sdk Sep 24, 2026
38271dd
use editScope name in skill
chunyu3 Aug 28, 2026
02d49f6
add start mcp server instruction
chunyu3 Aug 31, 2026
b0cd8c0
Potential fix for pull request finding
chunyu3 Sep 2, 2026
1d42cf6
Potential fix for pull request finding
chunyu3 Sep 2, 2026
e1c6c74
Add generate-only eval coverage for local SDK generation
Copilot Sep 2, 2026
e614abc
update the eval case
chunyu3 Sep 3, 2026
423f56f
update the timeout
chunyu3 Sep 3, 2026
426952d
set editCode CutomCode when in SDK repo
chunyu3 Sep 16, 2026
02f729b
Sync .github/skills directory with azure-sdk-tools repository for Too…
azure-sdk Sep 24, 2026
5654b5b
Sync .github/skills directory with azure-sdk-tools repository for Too…
azure-sdk Sep 24, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 19 additions & 8 deletions .github/skills/azsdk-common-generate-sdk-locally/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
name: azsdk-common-generate-sdk-locally
license: MIT
metadata:
version: "1.1.0"
version: "1.1.1"
distribution: shared
description: 'Generate, build, and test Azure SDKs locally from TypeSpec with automatic customization. WHEN: "generate SDK locally", "build SDK", "run SDK tests", "run CI checks", "validate package", "run checks", "update changelog", "fix SDK build errors", "fix breaking changes", "resolve SDK generation errors", "customize TypeSpec", "rename SDK client", "rename SDK model", "hide operation from SDK", "fix analyzer errors", "resolve customization drift", "create subclient", "update metadata", "update version". DO NOT USE FOR: publishing to package registries, CI pipeline configuration, API design review. INVOKES: azsdk_verify_setup, azsdk_package_generate_code, azsdk_package_build_code, azsdk_package_run_check, azsdk_package_run_tests, azsdk_customized_code_update, azsdk_package_update_changelog_content, azsdk_package_update_metadata, azsdk_package_update_version.'
description: 'Generate and finalize Azure SDKs locally from TypeSpec, including build, checks, tests, changelog, metadata, and version updates by default. WHEN: "generate SDK locally", "build SDK", "run SDK tests", "run CI checks", "validate package", "run checks", "update changelog", "fix SDK build errors", "resolve SDK generation errors", "customize TypeSpec", "rename SDK client", "rename SDK model", "hide operation from SDK", "fix analyzer errors", "resolve customization drift", "create subclient", "update metadata", "update version". INVOKES: azsdk_verify_setup, azsdk_package_generate_code, azsdk_package_build_code, azsdk_package_run_check, azsdk_package_run_tests, azsdk_customized_code_update, azsdk_package_update_changelog_content, azsdk_package_update_metadata, azsdk_package_update_version.'
compatibility: "azure-sdk-mcp server, local azure-sdk-for-{language} clone, language build tools"
---

Expand All @@ -14,12 +14,23 @@ This skill generates, builds, and tests Azure SDKs locally from TypeSpec with au

## Triggers

USE FOR: generate, build, and test Azure SDKs locally from TypeSpec with automatic customization; update changelog; fix SDK build errors; fix breaking changes; resolve SDK generation errors; customize TypeSpec; rename SDK client or model; hide operation from SDK; fix analyzer errors; resolve customization drift; create subclient; update metadata; update version
WHEN: "generate SDK locally", "build SDK", "run SDK tests", "update changelog", "fix SDK build errors", "fix breaking changes", "resolve SDK generation errors", "customize TypeSpec", "rename SDK client", "rename SDK model", "hide operation from SDK", "fix analyzer errors", "resolve customization drift", "create subclient", "update metadata", "update version"
DO NOT USE FOR: publishing to package registries, CI pipeline configuration, API design review
USE FOR: generate, build, and test Azure SDKs locally from TypeSpec with automatic customization; update changelog; fix SDK build errors; resolve SDK generation errors; customize TypeSpec; rename SDK client or model; hide operation from SDK; fix analyzer errors; resolve customization drift; create subclient; update metadata; update version
WHEN: "generate SDK locally", "build SDK", "run SDK tests", "update changelog", "fix SDK build errors", "resolve SDK generation errors", "customize TypeSpec", "rename SDK client", "rename SDK model", "hide operation from SDK", "fix analyzer errors", "resolve customization drift", "create subclient", "update metadata", "update version"

## Boundaries

Do not use this skill for:

- Publishing SDK packages to package registries
- CI pipeline configuration
- API design review

## Rules

- Interpret **"generate SDK"** and **"generate SDK locally"** as requests for the complete workflow: generate, build, validate, test, update changelog, update metadata, and update version. Do not treat the word "generate" by itself as a generate-only request.
- Stop after generation and a successful build when the user explicitly says **"generate only"**, **"only generate"**, or otherwise explicitly asks to skip checks, tests, changelog, metadata, and version updates. Bind `only` to the generation action even when a target qualifier follows: **"generate SDK only for Web PubSub"** is a generate-only request scoped to Web PubSub. By contrast, **"generate the SDK for only the Web PubSub service"** scopes the complete workflow to one service. Once a request is classified as generate-only, do not prompt for a commit or run steps 8–11. If the user's intent is unclear, default to the complete workflow.
- This skill generates an SDK **locally** from a local clone. Use it to generate an SDK **only when the user indicates local generation** — they say "local"/"locally", or are working from a local clone. If the user asks to "generate SDK for **all languages**", to generate **for a release plan / release ID** (even a single language), to generate **without a local clone**, or wants SDK **pull requests created** for them, do **not** generate locally — use the `azsdk-common-generate-sdk-pipeline` skill, which runs the SDK generation pipeline for each language and produces the SDK pull requests.
Comment thread
chunyu3 marked this conversation as resolved.
Resolved
- Never use `azure-sdk-mcp:azsdk_get_sdk_pull_request_link` or `azure-sdk-mcp:azsdk_get_pull_request` to _generate_ an SDK; those only retrieve links for SDKs that were already generated.
- Requires the `azure-sdk-mcp` server for the MCP workflow; without MCP, use `npm exec --prefix eng/common/tsp-client -- tsp-client` CLI.
- Verify the target language repo and the correct TypeSpec configuration file before generation.
- After generation or customization, run the check and test steps before updating metadata or finalizing changes.
Expand Down Expand Up @@ -49,8 +60,8 @@ Prerequisites: azure-sdk-mcp server must be running. Without MCP, use `npx tsp-c
- From an SDK language repo: use path to `tsp-location.yaml`.
4. **Verify setup** — Run `azure-sdk-mcp:azsdk_verify_setup` to confirm environment.
5. **Generate** — Run `azure-sdk-mcp:azsdk_package_generate_code` with the config file path.
6. **Build** — Run `azure-sdk-mcp:azsdk_package_build_code`. If build succeeds, proceed to step 8.
7. **Customize** — If build fails, or if user requests SDK modifications, run `azure-sdk-mcp:azsdk_customized_code_update` with the build errors or user request. The tool handles the full workflow internally: it classifies the issue, applies TypeSpec decorators and/or code patches, regenerates the SDK, and builds — all in one call. See [customization workflow](references/customization-workflow.md).
6. **Build** — Run `azure-sdk-mcp:azsdk_package_build_code`. For .NET, use the `additionalArguments: "/p:RunApiCompat=false"` to skip API compatibility checks if user requests to skip API compatibility check.
7. **Customize** — If build fails, or if user requests SDK modifications, run `azure-sdk-mcp:azsdk_customized_code_update` with the build errors or user request. The tool handles the full workflow internally: it classifies the issue, applies TypeSpec decorators and/or code patches, regenerates the SDK, and builds — all in one call. See [customization workflow](references/customization-workflow.md). _(If the user requested "generate only", stop here — skip steps 8–11.)_
8. **Commit checkpoint** — Prompt the user to commit generated changes before proceeding. See [commit checkpoint details](references/detailed-workflow.md).
9. **Validate** — Run `azure-sdk-mcp:azsdk_package_run_check` and `azure-sdk-mcp:azsdk_package_run_tests`.
10. **Metadata** — Run `azure-sdk-mcp:azsdk_package_update_changelog_content`, `azure-sdk-mcp:azsdk_package_update_metadata`, and `azure-sdk-mcp:azsdk_package_update_version`. _(Note: For .NET data plane, skip this step — metadata, changelog, and version updates are per-commit tasks, not part of the generate/build/test workflow.)_
Expand All @@ -70,7 +81,6 @@ Prerequisites: azure-sdk-mcp server must be running. Without MCP, use `npx tsp-c
- "Build and test the Python SDK package"
- "Run CI checks for my SDK package"
- "Fix the SDK build errors on this PR"
- "The SDK generation has breaking changes, resolve them"
- "Rename FooClient to BarClient for .NET"
- "Hide the internal polling operation from the Python SDK"
- "Fix .NET analyzer errors AZC0030 and AZC0012"
Expand All @@ -83,6 +93,7 @@ Prerequisites: azure-sdk-mcp server must be running. Without MCP, use `npx tsp-c

## Troubleshooting

- For "generate SDK for all languages", pipeline-based generation, or when no local SDK clone exists, call `azure-sdk-mcp:azsdk_run_generate_sdk` instead of the local generate flow, and never substitute `azure-sdk-mcp:azsdk_get_sdk_pull_request_link` / `azure-sdk-mcp:azsdk_get_pull_request` for generation.
- Run `azure-sdk-mcp:azsdk_verify_setup` to confirm MCP and tools.
- If build fails with type conflicts, breaking changes, analyzer errors, or customization drift, use `azure-sdk-mcp:azsdk_customized_code_update` to apply customizations.
- The customization tool uses a two-phase approach: TypeSpec decorators first (Phase A), then code repairs if needed (Phase B).
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,44 @@ stimuli:
required:
- name: azure-sdk-mcp-azsdk_verify_setup
- name: azure-sdk-mcp-azsdk_package_generate_code
- name: azure-sdk-mcp-azsdk_package_build_code
- name: azure-sdk-mcp-azsdk_package_run_check
- name: azure-sdk-mcp-azsdk_package_run_tests
- name: azure-sdk-mcp-azsdk_package_update_changelog_content
- name: azure-sdk-mcp-azsdk_package_update_metadata
- name: azure-sdk-mcp-azsdk_package_update_version

- name: sdk-local-generate-only-001
prompt: "Generate only the Python SDK locally from my TypeSpec project at specification/compute/resource-manager/Microsoft.Compute/ComputeRP/tspconfig.yaml. My azure-sdk-for-python repo is at ~/repos/azure-sdk-for-python."
graders:
- type: tool-calls
config:
required:
- name: azure-sdk-mcp-azsdk_verify_setup
- name: azure-sdk-mcp-azsdk_package_generate_code
- name: azure-sdk-mcp-azsdk_package_build_code
disallowed:
- name: azure-sdk-mcp-azsdk_package_run_check
- name: azure-sdk-mcp-azsdk_package_run_tests
- name: azure-sdk-mcp-azsdk_package_update_changelog_content
- name: azure-sdk-mcp-azsdk_package_update_metadata
- name: azure-sdk-mcp-azsdk_package_update_version

- name: sdk-local-generate-only-002
prompt: "Only generate the Python SDK locally from my TypeSpec project at specification/compute/resource-manager/Microsoft.Compute/ComputeRP/tspconfig.yaml. My azure-sdk-for-python repo is at ~/repos/azure-sdk-for-python."
graders:
- type: tool-calls
config:
required:
- name: azure-sdk-mcp-azsdk_verify_setup
- name: azure-sdk-mcp-azsdk_package_generate_code
- name: azure-sdk-mcp-azsdk_package_build_code
disallowed:
- name: azure-sdk-mcp-azsdk_package_run_check
- name: azure-sdk-mcp-azsdk_package_run_tests
- name: azure-sdk-mcp-azsdk_package_update_changelog_content
- name: azure-sdk-mcp-azsdk_package_update_metadata
- name: azure-sdk-mcp-azsdk_package_update_version

- name: sdk-local-negative-001
prompt: "I want to use the Azure DevOps pipeline to generate SDKs automatically and have PRs created for me. I don't want to set up local tools."
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
name: azsdk-common-generate-sdk-locally-trigger-eval
description: Trigger tests for the azsdk-common-generate-sdk-locally skill
type: capability

environment: azsdk-mcp-mock

tags:
area: azsdk-common-generate-sdk-locally
type: ci-gate

defaults:
runs: 1
timeout: "90s"
model: gpt-5.6-sol
executor: copilot-sdk

scoring:
threshold: 0.8

stimuli:
- name: trigger-generate-python-sdk-locally
prompt: "Generate the Python SDK locally from my TypeSpec project at specification/compute/resource-manager/Microsoft.Compute/ComputeRP/tspconfig.yaml."
graders:
- type: skill-invocation
config:
required: ["azsdk-common-generate-sdk-locally"]

# Should NOT trigger
- name: anti-trigger-mitigate-breaking-change
prompt: "Fix the SDK breaking changes"
graders:
- type: skill-invocation
config:
disallowed: ["azsdk-common-generate-sdk-locally"]
51 changes: 51 additions & 0 deletions .github/skills/azsdk-common-sdk-breaking-change/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
---
name: azsdk-common-sdk-breaking-change
license: MIT
metadata:
version: "1.0.0"
distribution: shared
description: 'Detect and mitigate SDK Breaking changes for an SDK package which is generated from TypeSpec. WHEN: "Detect SDK breaking changes for a service", "Detect SDK breaking changes for an SDK package", "Detect and mitigate SDK breaking changes for a service", "Detect and mitigate SDK breaking changes for an SDK package". INVOKES: skill: azsdk-common-generate-sdk-locally; MCP tools: azure-sdk-mcp:azsdk_customized_code_update, azure-sdk-mcp:azsdk_package_detect_breaking_change.'
compatibility: "azure-sdk-mcp server, local azure-sdk-for-{language} clone, language build tools"
---

# SDK Breaking Change Detection and Mitigation

## MCP Tools

| Tool | Purpose |
| ---------------------------------------------------- | --------------------------- |
| `azure-sdk-mcp:azsdk_customized_code_update` | Apply customizations |
| `azure-sdk-mcp:azsdk_package_detect_breaking_change` | Detect SDK breaking changes |

Prerequisites: azure-sdk-mcp server must be running.

## Steps

1. **Select language** — First extract one or more target languages from the user's prompt. If the prompt specifies any target languages, use them without asking the user to confirm. If it does not specify a target language, prompt the user to choose one or more from this list, presented in exactly this order: Go, Java, JavaScript, Python, .NET.

- For each selected language in step 1, execute steps 2 through 6 **one language at a time** — fully complete all steps for one language before starting the next. Process multiple selected languages in the same order they appear in the list above.

2. Follow `azsdk-common-generate-sdk-locally` skill to generate SDK only. For .NET, require the preparatory build to disable API compatibility checks using additionalArguments: "/p:RunApiCompat=false". This is a requirement of this workflow: compatibility findings must reach the detection and user-selection steps rather than be automatically repaired during generation.
3. **Detect SDK breaking Changes** - Run `azure-sdk-mcp:azsdk_package_detect_breaking_change`
4. **Display detected breaking changes** — If step 3 fails, display the `ResponseError` message and stop processing the current language. If step 3 detects breaking changes, display a `## Detected Breaking Changes` title followed by a Markdown table. Include exactly one breaking change per row with these columns: `Breaking Change`, `Category`, and `Resolution`. Preserve the category and resolution returned by the detection tool; do not replace them with a summary. If no breaking changes are detected, report that result and stop processing the current language.
5. **Prompt the user to choose breaking changes to mitigate** — If the user requested detection only, stop processing the current language after displaying the results. Otherwise, present the list from step 3 as a multiple-choice selection.
6. **Mitigate selected SDK breaking changes** — If the user selects no changes in step 5, stop processing the current language. Otherwise, run `azure-sdk-mcp:azsdk_customized_code_update` for the selected changes with these parameters:

- packagePath: The SDK package path
- customizationRequest: resolve the 'breakingChanges' chosen in step 5
- tspProjectPath: the typespec project path
- editScope: Set this based on the repository where the skill is running. When running in the target SDK language repository, use `'CustomCode'`. When running in `azure-rest-api-specs`, use `'All'` if the target SDK language repository is also available as a local clone; otherwise, use `'SpecInputs'`.

After step 6, begin the next iteration. If the customization modified TypeSpec, first repeat step 2 to regenerate the SDK; otherwise, resume at step 3. Then repeat steps 3 through 6. Run at most three detection-and-mitigation iterations per language, stopping earlier if no breaking changes remain or the user declines further mitigation. After the third mitigation, run step 3 once more to identify and report any remaining breaking changes, then stop without offering a fourth mitigation.

## Examples

- "Detect and mitigate SDK breaking changes for the service"
- "Detect SDK breaking changes for the service"
- "Detect SDK breaking changes for Go SDK package"
- "Detect and mitigate SDK breaking changes for Go SDK package"

## Troubleshooting

- Requires the `azure-sdk-mcp` server. If it is unavailable, direct the user to the [quick-start guide](https://github.com/Azure/azure-sdk-tools/blob/main/tools/azsdk-cli/Azure.Sdk.Tools.Cli/README.md#quick-start) to configure it before continuing.
- Requires a local clone of the target SDK repository. If unavailable, prompt the user to clone it before continuing.
41 changes: 41 additions & 0 deletions .github/skills/azsdk-common-sdk-breaking-change/evals/eval.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
name: azsdk-common-sdk-breaking-change-eval
description: tests for the azsdk-common-sdk-breaking-change skill
type: capability

environment: azsdk-mcp

tags:
area: azsdk-common-sdk-breaking-change
type: ci-gate

defaults:
runs: 1
timeout: "900s"
model: gpt-5.6-sol
executor: copilot-sdk

scoring:
threshold: 0.8

stimuli:
- name: detect-sdk-breaking-changes-for-service-001
prompt: "Detect and mitigate Go SDK breaking changes for the service webpubsub"
environment:
commands:
- git clone --filter=blob:none --sparse https://github.com/Azure/azure-sdk-for-go.git && cd azure-sdk-for-go && git sparse-checkout set eng documentation sdk/resourcemanager/webpubsub/armwebpubsub
- git clone --filter=blob:none --sparse https://github.com/Azure/azure-rest-api-specs.git && cd azure-rest-api-specs && git sparse-checkout set specification/webpubsub/resource-manager/Microsoft.SignalRService/SignalRService
- node -e "require('fs').copyFileSync('models.tsp', 'azure-rest-api-specs/specification/webpubsub/resource-manager/Microsoft.SignalRService/SignalRService/models.tsp')"
files:
- src: ../fixtures/detect-sdk-breaking-changes-for-service-001/models.tsp
dest: models.tsp
graders:
- type: skill-invocation
config:
required:
[
"azsdk-common-sdk-breaking-change",
"azsdk-common-generate-sdk-locally",
]
- type: tool-calls
config:
required: ["azure-sdk-mcp-azsdk_package_detect_breaking_change"]
Comment thread
chunyu3 marked this conversation as resolved.
Resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
name: azsdk-common-sdk-breaking-change-trigger-eval
description: Trigger tests for the azsdk-common-sdk-breaking-change skill
type: capability

environment: azsdk-mcp-mock

tags:
area: azsdk-common-sdk-breaking-change
type: ci-gate

defaults:
runs: 1
timeout: "90s"
model: gpt-5.6-sol
executor: copilot-sdk

scoring:
threshold: 0.8

stimuli:
- name: trigger-detect-sdk-breaking-changes-for-service
prompt: "Detect and mitigate SDK breaking changes for the service webpubsub"
graders:
- type: skill-invocation
config:
required: ["azsdk-common-sdk-breaking-change"]
Loading
Loading