build(deps): bump github.com/google/osv-scalibr from 0.3.4 to 0.4.5 - #6179
build(deps): bump github.com/google/osv-scalibr from 0.3.4 to 0.4.5#6179dependabot[bot] wants to merge 11 commits into
Conversation
ef68922 to
6a1c5e9
Compare
80adf4f to
74901c5
Compare
74901c5 to
c955573
Compare
c955573 to
4e2b677
Compare
Bumps [github.com/google/osv-scalibr](https://github.com/google/osv-scalibr) from 0.3.4 to 0.4.5. - [Release notes](https://github.com/google/osv-scalibr/releases) - [Commits](google/osv-scalibr@v0.3.4...v0.4.5) --- updated-dependencies: - dependency-name: github.com/google/osv-scalibr dependency-version: 0.4.5 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
4e2b677 to
da5b197
Compare
da5b197 to
ca585a0
Compare
| if scanResults.Status.Status == scalibr_plugin.ScanStatusSucceeded { | ||
| return scanResults.Inventory.Packages, nil | ||
| } | ||
|
|
||
| return scanResults.Inventory.Packages, nil | ||
| // Scalibr runs a lot of plugins and aggregates the result. Some of these are picky, and | ||
| // fail for random reasons. Accept partial success, but log the failing plugins. | ||
| if scanResults.Status.Status == scalibr_plugin.ScanStatusPartiallySucceeded { |
There was a problem hiding this comment.
Do you prefer this set of if statements, or a switch like in scalibr.go, line 94?
(Yeah, we need to unify these two code paths or retire one)
|
@JAORMX - can you take a look at this? I had to do some code updates to support the new scalibr evaluation (and ignore a couple scanners that were over-sensitive). |
|
This PR needs additional information before we can continue. It is now marked as stale because it has been open for 30 days with no activity. Please provide the necessary details to continue or it will be closed in 30 days. |
…es/github.com/google/osv-scalibr-0.4.5
|
@dakshhhhh16 or @krrish175-byte Could you take a look at this? I'd like to get scalibr updated to their most recent release in prep for clearing out the pin on OCI v1.2.1 spec. |
| _ = os.RemoveAll(tmpDir) | ||
| }() | ||
| cfg := scalibr_cfg.PluginConfig{ | ||
| MaxFileSizeBytes: 1024 * 1024, |
There was a problem hiding this comment.
Could we avoid setting a global 1 MiB file-size limit here and in internal/engine/ingester/diff/diff.go?
This configuration is passed to every selected plugin, including the package-lock.json extractor, which silently skips valid lockfiles larger than this limit. I reproduced a valid lockfile over 1 MiB returning an empty dependency inventory. Previously, an unset value of 0 meant no limit.
If this cap is intended only for binary extractors, could we configure those plugins specifically and add a large-lockfile regression test?
There was a problem hiding this comment.
Good point! We're trying to avoid blowing up memory during rule evaluation on the Minder server side, so let me bump this to 2MB, and see if we can check the stats for ignored files.
GitHub security alerts apparently only trigger for <0.5MB, so there is precedent for this sort of limit.
| return nil, err | ||
| } | ||
| // unknownbinariesextr uses file extension to determine "binary-ness", and triggers on e.g. .py files | ||
| skipPlugins := []string{"ffa/unknownbinariesextr"} |
There was a problem hiding this comment.
Should go/binary remain in this skip list?
Before this PR, Minder explicitly removed the Go-binary extractor because it can panic on certain files. The v0.4.5 implementation retains the same extraction path, and SCALIBR does not recover extractor panics, so removing that exclusion appears to reintroduce a process-level crash risk.
Could we keep it excluded unless there is an upstream fix we can reference and cover with a regression test?
There was a problem hiding this comment.
It looks like google/osv-scalibr#2316 may have (incidentally) fixed this. I added a regression test.
evankanderson
left a comment
There was a problem hiding this comment.
Feedback addressed and test cases added! Thanks for the review!
| _ = os.RemoveAll(tmpDir) | ||
| }() | ||
| cfg := scalibr_cfg.PluginConfig{ | ||
| MaxFileSizeBytes: 1024 * 1024, |
There was a problem hiding this comment.
Good point! We're trying to avoid blowing up memory during rule evaluation on the Minder server side, so let me bump this to 2MB, and see if we can check the stats for ignored files.
GitHub security alerts apparently only trigger for <0.5MB, so there is precedent for this sort of limit.
| return nil, err | ||
| } | ||
| // unknownbinariesextr uses file extension to determine "binary-ness", and triggers on e.g. .py files | ||
| skipPlugins := []string{"ffa/unknownbinariesextr"} |
There was a problem hiding this comment.
It looks like google/osv-scalibr#2316 may have (incidentally) fixed this. I added a regression test.
|
A newer version of github.com/google/osv-scalibr exists, but since this PR has been edited by someone other than Dependabot I haven't updated it. You'll get a PR for the updated version as normal once this PR is merged. |
…es/github.com/google/osv-scalibr-0.4.5
| return cmp.Or( | ||
| cmp.Compare(a.Name, b.Name), | ||
| cmp.Compare(a.Version, b.Version), | ||
| cmp.Compare(a.Location.PathOrEmpty(), b.Location.PathOrEmpty()), |
There was a problem hiding this comment.
Could we include Location.Related in this comparison too? If the base and head packages have the same name, version, and descriptor but the head adds another related file, this comparator returns 0. setDifference then considers them identical, so the new file never makes it into the dependency diff.
The old comparison included all locations. A focused test where only Related changes would help make sure we don’t lose that case.
There was a problem hiding this comment.
I'll add a.Id as a tie-breaker to avoid this comparison. The old set of affected files was the best we could do with the previous API, but we're really interested in collecting packages and the corresponding file-points (e.g. package.json and package-lock.json), which should be consistent given a particular packaging ecosystem.
There was a problem hiding this comment.
I checked this against the current SCALIBR output, and I don’t think ID works as the tie-breaker here. I ran the same requirements.txt scan twice, and the package ID was empty both times. That means when only Location.Related changes, this comparator still returns 0 and setDifference drops the update.
IDs can also be randomly generated in some SCALIBR enrichment paths, so they don’t look like a stable cross-scan identity. Could we compare Related directly, or build a stable key from the PURL plus descriptor and related locations? A focused setDifference test would make this much easier to verify.
There was a problem hiding this comment.
Can we guard the PURL before calling String() here? SCALIBR returns nil when a package has no PURLType, so one such inventory item would panic the whole scan.
We already account for a nil PURL in inventoryToEcosystem; I think this path should do the same and simply omit the identifier when there isn’t one. A small regression test would be useful too.
There was a problem hiding this comment.
Done. I'll see if I can find an example scan that doesn't produce a PURL; my experience has been that all the scanners for the types that we detect (at least on Linux) produce PURLs.
There was a problem hiding this comment.
Confirmed, this is fixed now. Moving the conversion into nodeFromPackage, guarding the nil PURL, and covering the empty-package case addresses the panic. Thanks!
| }() | ||
| cfg := scalibr_config.PluginConfig{ | ||
| ProtoConfig: &scalibr_cfg.PluginConfig{ | ||
| MaxFileSizeBytes: 1024 * 1024, |
There was a problem hiding this comment.
I think one half of the earlier size-limit feedback is still open. scanFilesystem wires in the stats collector and logs files skipped because of this limit, but the new-deps path here doesn’t install that collector.
That means a large lockfile is silently omitted from a PR dependency diff. Could we share the scan configuration/stats wiring between the two paths and add a regression test for this path as well?
There was a problem hiding this comment.
Updated the code. Given the copying between the two, I'm not sure that a regression test here adds a lot. (I'd like to figure out how to retire this path in favor of the SBOM-output variety.)
There was a problem hiding this comment.
Confirmed, the diff path now installs the same stats collector and logs skipped files. I’m okay with not duplicating the heavier scan test while this path is being phased out. Thanks!
| assert.Contains(t, string(logOutput.String()), tc.expectedLog) | ||
| require.NoError(t, err) | ||
| if tc.expectedLog == "" { | ||
| assert.Equal(t, logOutput.String(), tc.expectedLog) |
There was a problem hiding this comment.
I think requiring a completely empty log makes this test platform-dependent. On macOS, the broader plugin set reports a legitimate partial-success warning from vex/os-duplicate/rpm (“not implemented outside of Linux”), so the focused test fails here even though dependency extraction succeeds.
Could we only assert on the log when the test case provides an expected message, rather than requiring every normal case to produce no warnings?
There was a problem hiding this comment.
I was concerned about known cases producing ScanStatusPartiallySucceeded and clogging up the logs. What do you think about only performing this check on GOOS=linux?
There was a problem hiding this comment.
The Linux-only check works for me. It still catches unexpected server-side log spam without breaking local macOS or Windows runs. I reran the focused tests on macOS and they pass now.
…es/github.com/google/osv-scalibr-0.4.5
|
@dakshhhhh16 -- ready for another round of review. |
Bumps github.com/google/osv-scalibr from 0.3.4 to 0.4.5.
Release notes
Sourced from github.com/google/osv-scalibr's releases.
... (truncated)
Commits
6510600Bump SCALIBR version in preparation for a new release.9c92834Merge pull request #1870 from cuixq:npmea5d428fix: escape package names in manifest writerc7f2e9bAdd new structured Location fields to protos.352fc38Merge pull request #1855 from cleanstart-community-admin:feat/cleanstart-ecos...6742f9cMerge pull request #1862 from ackama:refactor/remove-unneeded-parenthesesafc5764Merge pull request #1861 from ackama:refactor/use-packagist-directly0bf9e87Merge pull request #1853 from ackama:adjust-version886caf3Merge pull request #1859 from ackama:enrichers-root8bd4660Merge pull request #1858 from ackama:adjust-comment