Repository navigation
Sync .github/skills directory with azure-sdk-tools for PR 16634 #50577
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Crystal YU (chunyu3)
merged 28 commits into
main
from
sync-.github-skills-yuc/sdk-quality-skill-16634
Sep 24, 2026
Merged
Changes from all commits
Commits
Show all changes
28 commits
Select commit
Hold shift + click to select a range
0eb5554
add sdk quality assureance skill
chunyu3 8551591
iterative assurance sdk quality
chunyu3 c576be9
Potential fix for pull request finding
chunyu3 ca96c86
Potential fix for pull request finding
chunyu3 3921116
Potential fix for pull request finding
chunyu3 cd4392c
update the reference
chunyu3 0817aca
remove breaking change resolve from generate-sdk-locally skill
chunyu3 ddde45a
format the breaking change display
chunyu3 79382bc
remove unused tool
chunyu3 1e02cce
format and resolve comments
chunyu3 6b328d6
add evals
chunyu3 220ebb9
refine the instruction
chunyu3 12050f5
typo
chunyu3 d135bb3
add eval test
chunyu3 1541f7b
resolve comments
chunyu3 fa17b5d
format eval yaml
chunyu3 c600f47
resolve comments
chunyu3 99bdc36
Sync .github/skills directory with azure-sdk-tools repository for Too…
azure-sdk 38271dd
use editScope name in skill
chunyu3 02d49f6
add start mcp server instruction
chunyu3 b0cd8c0
Potential fix for pull request finding
chunyu3 1d42cf6
Potential fix for pull request finding
chunyu3 e1c6c74
Add generate-only eval coverage for local SDK generation
Copilot e614abc
update the eval case
chunyu3 423f56f
update the timeout
chunyu3 426952d
set editCode CutomCode when in SDK repo
chunyu3 02f729b
Sync .github/skills directory with azure-sdk-tools repository for Too…
azure-sdk 5654b5b
Sync .github/skills directory with azure-sdk-tools repository for Too…
azure-sdk File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
34 changes: 34 additions & 0 deletions
34
.github/skills/azsdk-common-generate-sdk-locally/evals/trigger.eval.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| name: azsdk-common-generate-sdk-locally-trigger-eval | ||
| description: Trigger tests for the azsdk-common-generate-sdk-locally skill | ||
| type: capability | ||
|
|
||
| environment: azsdk-mcp-mock | ||
|
|
||
| tags: | ||
| area: azsdk-common-generate-sdk-locally | ||
| type: ci-gate | ||
|
|
||
| defaults: | ||
| runs: 1 | ||
| timeout: "90s" | ||
| model: gpt-5.6-sol | ||
| executor: copilot-sdk | ||
|
|
||
| scoring: | ||
| threshold: 0.8 | ||
|
|
||
| stimuli: | ||
| - name: trigger-generate-python-sdk-locally | ||
| prompt: "Generate the Python SDK locally from my TypeSpec project at specification/compute/resource-manager/Microsoft.Compute/ComputeRP/tspconfig.yaml." | ||
| graders: | ||
| - type: skill-invocation | ||
| config: | ||
| required: ["azsdk-common-generate-sdk-locally"] | ||
|
|
||
| # Should NOT trigger | ||
| - name: anti-trigger-mitigate-breaking-change | ||
| prompt: "Fix the SDK breaking changes" | ||
| graders: | ||
| - type: skill-invocation | ||
| config: | ||
| disallowed: ["azsdk-common-generate-sdk-locally"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,51 @@ | ||
| --- | ||
| name: azsdk-common-sdk-breaking-change | ||
| license: MIT | ||
| metadata: | ||
| version: "1.0.0" | ||
| distribution: shared | ||
| description: 'Detect and mitigate SDK Breaking changes for an SDK package which is generated from TypeSpec. WHEN: "Detect SDK breaking changes for a service", "Detect SDK breaking changes for an SDK package", "Detect and mitigate SDK breaking changes for a service", "Detect and mitigate SDK breaking changes for an SDK package". INVOKES: skill: azsdk-common-generate-sdk-locally; MCP tools: azure-sdk-mcp:azsdk_customized_code_update, azure-sdk-mcp:azsdk_package_detect_breaking_change.' | ||
| compatibility: "azure-sdk-mcp server, local azure-sdk-for-{language} clone, language build tools" | ||
| --- | ||
|
|
||
| # SDK Breaking Change Detection and Mitigation | ||
|
|
||
| ## MCP Tools | ||
|
|
||
| | Tool | Purpose | | ||
| | ---------------------------------------------------- | --------------------------- | | ||
| | `azure-sdk-mcp:azsdk_customized_code_update` | Apply customizations | | ||
| | `azure-sdk-mcp:azsdk_package_detect_breaking_change` | Detect SDK breaking changes | | ||
|
|
||
| Prerequisites: azure-sdk-mcp server must be running. | ||
|
|
||
| ## Steps | ||
|
|
||
| 1. **Select language** — First extract one or more target languages from the user's prompt. If the prompt specifies any target languages, use them without asking the user to confirm. If it does not specify a target language, prompt the user to choose one or more from this list, presented in exactly this order: Go, Java, JavaScript, Python, .NET. | ||
|
|
||
| - For each selected language in step 1, execute steps 2 through 6 **one language at a time** — fully complete all steps for one language before starting the next. Process multiple selected languages in the same order they appear in the list above. | ||
|
|
||
| 2. Follow `azsdk-common-generate-sdk-locally` skill to generate SDK only. For .NET, require the preparatory build to disable API compatibility checks using additionalArguments: "/p:RunApiCompat=false". This is a requirement of this workflow: compatibility findings must reach the detection and user-selection steps rather than be automatically repaired during generation. | ||
| 3. **Detect SDK breaking Changes** - Run `azure-sdk-mcp:azsdk_package_detect_breaking_change` | ||
| 4. **Display detected breaking changes** — If step 3 fails, display the `ResponseError` message and stop processing the current language. If step 3 detects breaking changes, display a `## Detected Breaking Changes` title followed by a Markdown table. Include exactly one breaking change per row with these columns: `Breaking Change`, `Category`, and `Resolution`. Preserve the category and resolution returned by the detection tool; do not replace them with a summary. If no breaking changes are detected, report that result and stop processing the current language. | ||
| 5. **Prompt the user to choose breaking changes to mitigate** — If the user requested detection only, stop processing the current language after displaying the results. Otherwise, present the list from step 3 as a multiple-choice selection. | ||
| 6. **Mitigate selected SDK breaking changes** — If the user selects no changes in step 5, stop processing the current language. Otherwise, run `azure-sdk-mcp:azsdk_customized_code_update` for the selected changes with these parameters: | ||
|
|
||
| - packagePath: The SDK package path | ||
| - customizationRequest: resolve the 'breakingChanges' chosen in step 5 | ||
| - tspProjectPath: the typespec project path | ||
| - editScope: Set this based on the repository where the skill is running. When running in the target SDK language repository, use `'CustomCode'`. When running in `azure-rest-api-specs`, use `'All'` if the target SDK language repository is also available as a local clone; otherwise, use `'SpecInputs'`. | ||
|
|
||
| After step 6, begin the next iteration. If the customization modified TypeSpec, first repeat step 2 to regenerate the SDK; otherwise, resume at step 3. Then repeat steps 3 through 6. Run at most three detection-and-mitigation iterations per language, stopping earlier if no breaking changes remain or the user declines further mitigation. After the third mitigation, run step 3 once more to identify and report any remaining breaking changes, then stop without offering a fourth mitigation. | ||
|
|
||
| ## Examples | ||
|
|
||
| - "Detect and mitigate SDK breaking changes for the service" | ||
| - "Detect SDK breaking changes for the service" | ||
| - "Detect SDK breaking changes for Go SDK package" | ||
| - "Detect and mitigate SDK breaking changes for Go SDK package" | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| - Requires the `azure-sdk-mcp` server. If it is unavailable, direct the user to the [quick-start guide](https://github.com/Azure/azure-sdk-tools/blob/main/tools/azsdk-cli/Azure.Sdk.Tools.Cli/README.md#quick-start) to configure it before continuing. | ||
| - Requires a local clone of the target SDK repository. If unavailable, prompt the user to clone it before continuing. |
41 changes: 41 additions & 0 deletions
41
.github/skills/azsdk-common-sdk-breaking-change/evals/eval.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,41 @@ | ||
| name: azsdk-common-sdk-breaking-change-eval | ||
| description: tests for the azsdk-common-sdk-breaking-change skill | ||
| type: capability | ||
|
|
||
| environment: azsdk-mcp | ||
|
|
||
| tags: | ||
| area: azsdk-common-sdk-breaking-change | ||
| type: ci-gate | ||
|
|
||
| defaults: | ||
| runs: 1 | ||
| timeout: "900s" | ||
| model: gpt-5.6-sol | ||
| executor: copilot-sdk | ||
|
|
||
| scoring: | ||
| threshold: 0.8 | ||
|
|
||
| stimuli: | ||
| - name: detect-sdk-breaking-changes-for-service-001 | ||
| prompt: "Detect and mitigate Go SDK breaking changes for the service webpubsub" | ||
| environment: | ||
| commands: | ||
| - git clone --filter=blob:none --sparse https://github.com/Azure/azure-sdk-for-go.git && cd azure-sdk-for-go && git sparse-checkout set eng documentation sdk/resourcemanager/webpubsub/armwebpubsub | ||
| - git clone --filter=blob:none --sparse https://github.com/Azure/azure-rest-api-specs.git && cd azure-rest-api-specs && git sparse-checkout set specification/webpubsub/resource-manager/Microsoft.SignalRService/SignalRService | ||
| - node -e "require('fs').copyFileSync('models.tsp', 'azure-rest-api-specs/specification/webpubsub/resource-manager/Microsoft.SignalRService/SignalRService/models.tsp')" | ||
| files: | ||
| - src: ../fixtures/detect-sdk-breaking-changes-for-service-001/models.tsp | ||
| dest: models.tsp | ||
| graders: | ||
| - type: skill-invocation | ||
| config: | ||
| required: | ||
| [ | ||
| "azsdk-common-sdk-breaking-change", | ||
| "azsdk-common-generate-sdk-locally", | ||
| ] | ||
| - type: tool-calls | ||
| config: | ||
| required: ["azure-sdk-mcp-azsdk_package_detect_breaking_change"] | ||
|
chunyu3 marked this conversation as resolved.
Resolved
|
||
26 changes: 26 additions & 0 deletions
26
.github/skills/azsdk-common-sdk-breaking-change/evals/trigger.eval.yaml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| name: azsdk-common-sdk-breaking-change-trigger-eval | ||
| description: Trigger tests for the azsdk-common-sdk-breaking-change skill | ||
| type: capability | ||
|
|
||
| environment: azsdk-mcp-mock | ||
|
|
||
| tags: | ||
| area: azsdk-common-sdk-breaking-change | ||
| type: ci-gate | ||
|
|
||
| defaults: | ||
| runs: 1 | ||
| timeout: "90s" | ||
| model: gpt-5.6-sol | ||
| executor: copilot-sdk | ||
|
|
||
| scoring: | ||
| threshold: 0.8 | ||
|
|
||
| stimuli: | ||
| - name: trigger-detect-sdk-breaking-changes-for-service | ||
| prompt: "Detect and mitigate SDK breaking changes for the service webpubsub" | ||
| graders: | ||
| - type: skill-invocation | ||
| config: | ||
| required: ["azsdk-common-sdk-breaking-change"] |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.