ci: enable blocking deptry DEP001 with uv-installed deps (#7161) - #7630
Thibaut Rouxel (throuxel) wants to merge 2 commits into
Conversation
Thibaut Rouxel (throuxel)
added
filigran team
There was a problem hiding this comment.
Pull request overview
Updates the existing deptry-based CI workflow to catch missing dependency declarations at PR time by running deptry with connector dependencies installed via uv, making DEP001 blocking while keeping DEP002/DEP003 informational.
Changes:
- Run deptry inside an isolated
uvenvironment with each connector’srequirements.txtinstalled, and scan changed connectors in parallel. - Treat
DEP001(imported but not declared) as a failing condition; keepDEP002/DEP003as warnings surfaced via annotations and a PR comment. - Extend the deptry package→module mapping with
python-magic=magic.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| .github/workflows/ci-unused-deps.yml | Reworks the CI job to run deptry with uv-installed deps, adds DEP001 blocking behavior, and enhances annotations/commenting. |
| .github/deptry-package-map.txt | Adds python-magic=magic mapping to reduce false positives and correctly attribute the magic import. |
Suppressed comments (1)
.github/workflows/ci-unused-deps.yml:166
- Same as warnings:
printf '%s\n' "$errors"produces a non-empty output (newline) whenerrorsis empty, which can incorrectly trip thesteps.deptry.outputs.errors != ''condition. Avoid forcing a newline in the output value.
printf '%s\n' "$errors"
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| - 'stream/**' | ||
| - '.github/deptry-package-map.txt' | ||
| - '.github/workflows/unused-deps.yml' | ||
| - '.github/workflows/ci-unused-deps.yml' |
There was a problem hiding this comment.
issue: This triggers does nothing because you will run deptry only if the code of a connector has changed due to https://github.com/OpenCTI-Platform/connectors/pull/7630/changes#diff-007841c83bdb5286672b72b04f8c6b914aa6bf9c1c8a08705e19f8babe73ed9dR134. What you can do is to run deptry on all connectors when you update this file and deptry-package-map.txt
There was a problem hiding this comment.
I removed those triggers, it will just make a lot of noises to apply to all connectors.
| - name: Install uv | ||
| uses: astral-sh/setup-uv@v6 | ||
|
|
||
| - name: Run deptry on changed connectors |
There was a problem hiding this comment.
suggestion: This step is becoming quite large and difficult to read in a YAML file. Perhaps you could move it into a proper script file and simply call it here.
| per_rule="DEP001=__main__" | ||
| [ -n "${IGNORE_PKGS:-}" ] && per_rule="$per_rule,DEP002=$IGNORE_PKGS" | ||
|
|
||
| out=$(uv run --isolated --no-project \ |
There was a problem hiding this comment.
2>&1 | grep ... || true discards uv run 's exit code, so if the env fails to build. out is empty → return 0 → connector reported clean, job green. Maybe separate "deptry ran and found nothing" from "deptry never ran".
There was a problem hiding this comment.
done, exit codes are now different.
Install each changed connector's dependencies in an isolated uv environment before running deptry so transitive imports resolve as DEP003 instead of false DEP001. Make DEP001 (undeclared) blocking; keep DEP002 (unused) and DEP003 (transitive) as non-blocking warnings. Auto-detect first-party modules, exclude tests, ignore __main__, and add python-magic=magic mapping. Scan connectors in parallel.
a84ada1 to
b81bdbb
Compare
Split the deptry logic into .github/scripts/deptry_scan.py (JSON output) and scan each changed connector in parallel via a discover -> matrix -> report workflow. Distinguish 'deptry could not run' (env build failure -> exit 2, blocking) from 'no findings'. Cover connectors declaring dependencies via pyproject.toml (generate a requirements file from [project.dependencies]) in addition to requirements.txt, scan the whole connector root so every layout is handled, and detect first-party modules across it. Drop the workflow/map files from the path trigger.
b81bdbb to
d5fd801
Compare
Test PR to see results: #7631
Proposed changes
uvenvironment before runningdeptry, so transitive imports (provided bypycti) resolve asDEP003instead of falseDEP001.DEP001(undeclared import) blocking — the job fails so undeclared dependencies are caught at PR time instead of at OCTI release time.DEP002(unused) andDEP003(transitive) as non-blocking warnings surfaced via annotations and a PR comment.src/, exclude test directories, and ignore the__main__pydanticBaseSettingsfalse positive.xargs -P 6).python-magic=magicto.github/deptry-package-map.txt(the package whose transitive removal originally broke the release).Related issues
Checklist
Further comments
Follow-up from the
python-magicrelease breakage (revert PR #7158).Validated locally against the whole repo with
deptry==0.25.1anduv:DEP001reported 530 findings (mostlystix2/pydantic/requestsprovided transitively bypycti, plus localsrcimports).__main__ignore, realDEP001findings drop to 0 false positives; a genuinely undeclared package (e.g.tabulate) is still correctly flagged and would block.DEP003(~630 findings, allpycti-transitive:requests,yaml,stix2,dateutil, …) stays informational.