chore(scanner): upgrade, fix llm_analyzer in Modal, and improve backfill - #344
Open
maresb wants to merge 5 commits into
Open
chore(scanner): upgrade, fix llm_analyzer in Modal, and improve backfill#344maresb wants to merge 5 commits into
maresb wants to merge 5 commits into
Conversation
…rror rotations
The crawler image build kept failing inside Modal with 404s for
libssl3 / openssl on the Debian security mirror — apt_install("git")
pulls openssh-client as a recommended dep, which then drags in the
openssl chain whose security revisions get rotated faster than the
cached apt package list. Cascade rebuilds (e.g. after a uv.lock bump)
made this trip the live mirror instead of the cached layer.
The crawler clones repos over HTTPS using GitHub App installation
tokens (mint_installation_token), never over SSH — verified that the
git@github.com: references in tracker.py / discovery.py are only URL
parsing, not transport. So --no-install-recommends is safe and also
shrinks the crawler image.
Made-with: Cursor
Pulls in upstream improvements ahead of the validation batch run for issue pymc-labs#329. Also re-syncs the lockfile to the current declared pyproject versions (dhub-cli 0.12.3, dhub-core 0.2.10) and drops the google-generativeai stack — the new scanner now reaches Gemini via litellm only, and no first-party code imports google.* directly. Made-with: Cursor
…r loads in Modal The scanner's LLMProviderConfig raises ImportError for Gemini models when google-genai is missing, even if LiteLLM (which also supports Gemini) is installed — see cisco-ai-defense/skill-scanner#97. Our local dev env had a stray google-genai installation that masked the bug; in Modal the llm_analyzer was silently disabled. Pulling google-genai via the [google] extra makes llm_analyzer available on every scan in the deployed image until the upstream is fixed. Made-with: Cursor
Pulls google-genai and its transitive deps into the workspace lock so the [google] extra from the previous commit resolves deterministically. Made-with: Cursor
Two related changes to make the scan-report backfill usable at scale and able to produce representative samples: - Replace the ROW_NUMBER() OVER (PARTITION BY skill_id ORDER BY created_at DESC) subquery with a direct join on the denormalized skills.latest_semver pointer. The window-function variant trips the 30s statement_timeout on the current versions table; the new query runs in well under a second. - Add a --random flag that swaps the ORDER BY from (org_slug, name) to random(). Alphabetical ordering concentrated --limit N batches on a handful of orgs (e.g. 77% of the first 200-skill run landed on a single prolific publisher); random order spreads the same sample across ~80 orgs with no single org over 10%, which is what we want for the distribution stats in pymc-labs#329. Made-with: Cursor
maresb
force-pushed
the
scanner-validation-infra
branch
from
April 20, 2026 09:58
d90b7e6 to
c9bb2e0
Compare
maresb
marked this pull request as ready for review
April 20, 2026 10:01
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Three #329 blockers and one follow-on. Each was required to produce trustworthy distribution numbers for the Cisco scanner validation batch — they're not "cleanup that fell out of #329".
Why each commit was necessary
3e5cf11— fix(modal): drop openssh-client from crawler imagemake deploy-devwas failing intermittently with404 Not Found [IP: …] libssl3 / opensslafter Debian mirror snapshot rotations, blocking any iteration on the server for #329. Root cause:image.apt_install("git")pulledopenssh-clientas a recommended dep, which dragged inlibssl3/opensslthat were briefly missing from the mirror. The crawler only clones over HTTPS with GitHub App tokens (never SSH), so switching toimage.run_commands("apt-get install --no-install-recommends git …")is safe and makes deploys reliable.03a9f49— chore(deps): upgrade cisco-ai-skill-scanner 2.0.6 → 2.0.9 (uv.lock)Pulls upstream scanner improvements (including ones that affect verdict classification) so that #329's numbers reflect the current scanner rather than a stale fork. Also re-syncs the lockfile to declared pyproject versions and drops the old
google-generativeaistack (the new scanner reaches Gemini vialitellm).792f37d+de95818— fix(scanner): depend oncisco-ai-skill-scanner[google]During the first #329 backfill run, Modal logs showed
WARNING | Could not load LLM analyzeron every scan — Cisco was running 2 of its 3 analyzers (behavioral + trigger) and silently skippingllm_analyzer. Any statistics gathered in that state would have understated Cisco's capability and led to the wrong replace/don't-replace decision for #328.Root cause is an upstream logic flaw in
cisco-ai-skill-scanner(filed ascisco-ai/cisco-ai-skill-scanner#97): for Gemini models it raisesImportErrorwhengoogle-genaiis absent, even iflitellmis installed. My local env had a straygoogle-genaiinstall that masked it; Modal didn't.Fix: depend on
cisco-ai-skill-scanner[google], which pullsgoogle-genaiinto the Modal image. After deploy all three analyzers load and the LLM analyzer contributes findings to the #329 batch.Lockfile changes are in the separate commit
de95818per repo convention.c9bb2e0— refactor(backfill): useskills.latest_semverjoin and add--randomflagTwo fixes to
backfill_scan_reports.py, both blockers:_find_skills_needing_scanusedROW_NUMBER() OVER (PARTITION BY skill_id ORDER BY semver DESC)to pick the latest version per skill. Against the currentversionstable this hits the 30sstatement_timeoutbefore returning a single row, so the script aborted on startup. Rewrote to join on the denormalizedskills.latest_semvercolumn — same result, well under a second.--randomwhich appendsORDER BY random()at the SQL level, used to produce the corrected distribution on Run fresh scan batch and compute distribution stats #329.Test plan
ruff check+ruff format --checkcleanmake deploy-dev— crawler image builds reliably (used throughout Run fresh scan batch and compute distribution stats #329 validation)--random, all three analyzers load per Modal logsIndependent of #337, #342, #343.