Skip to content

Reduce check-shims indexing work - #2729

Merged
KaanOzkan merged 1 commit into
mainfrom
ko-check-shims-filtered-index
Oct 28, 2026
Merged

KaanOzkan merged 1 commit into
mainfrom
ko-check-shims-filtered-index

Conversation

@KaanOzkan

@KaanOzkan KaanOzkan commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

check-shims only reports generated definitions that overlap with shim or TODO definitions, but it previously built a complete index for every payload, gem, DSL, and annotation RBI.

Part of #2728

Implementation

Before:

  1. Parse every RBI into a complete syntax tree.
  2. Send each complete tree back from its worker.
  3. Add every definition to the shared index.
  4. Discard unrelated definitions only while looking for duplicates.

After:

  1. Parse shim and TODO RBIs first to collect the exact index IDs that can produce a duplicate.
  2. While parsing generated RBIs, retain only nodes matching those candidate IDs.
  3. Detach retained nodes from their full trees before returning them from workers.
  4. Index shim and TODO RBIs normally and run the existing duplicate checks against the smaller generated-RBI index.

This keeps the duplicate-detection behavior unchanged while reducing worker serialization and shared-index work.

Tests

Ran the focused check-shims tests, typecheck, and RuboCop.

@KaanOzkan
KaanOzkan force-pushed the ko-check-shims-filtered-index branch 3 times, most recently from 75272ed to 50458d9 Compare September 30, 2026 20:37
#: (RBI::Index index, String kind, String dir, number_of_workers: Integer?) -> void
def index_rbis(index, kind, dir, number_of_workers:)
index_rbis_with_filter(index, kind, dir, number_of_workers: number_of_workers, index_ids: nil)
end

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I opted to make the index_ids parameter contract more explicit by creating 2 separate functions with different names.

@KaanOzkan
KaanOzkan marked this pull request as ready for review September 30, 2026 20:43
@KaanOzkan
KaanOzkan requested a review from a team as a code owner September 30, 2026 20:43
@KaanOzkan KaanOzkan self-assigned this Sep 30, 2026

@amomchilov amomchilov left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make sense to me, just a few small suggestions

Comment thread lib/tapioca/helpers/rbi_files_helper.rb Outdated
Comment on lines +143 to +144
return unless File.exist?(file)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TOC-TOU

Just let File.read fail, and rescue the exception.

Comment thread lib/tapioca/helpers/rbi_files_helper.rb Outdated

#: (RBI::Index index, String kind, String dir, index_ids: Set[String], number_of_workers: Integer?) -> void
def index_filtered_rbis(index, kind, dir, index_ids:, number_of_workers:)
index_rbis_with_filter(index, kind, dir, number_of_workers: number_of_workers, index_ids: index_ids)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
index_rbis_with_filter(index, kind, dir, number_of_workers: number_of_workers, index_ids: index_ids)
index_rbis_with_filter(index, kind, dir, number_of_workers:, index_ids:)

Comment thread lib/tapioca/helpers/rbi_files_helper.rb Outdated

#: (RBI::Index index, String kind, String dir, number_of_workers: Integer?, index_ids: Set[String]?) -> void
def index_rbis_with_filter(index, kind, dir, number_of_workers:, index_ids:)
return unless Dir.exist?(dir) && !Dir.empty?(dir)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Likewise. Just try the access, and rescue if missing or not a dir

@KaanOzkan
KaanOzkan force-pushed the ko-check-shims-filtered-index branch from 50458d9 to b6f5463 Compare October 26, 2026 20:51
@KaanOzkan

Copy link
Copy Markdown
Contributor Author

Addressed suggestions, initial tests show 2.2x speedup in Core 🚀

@KaanOzkan KaanOzkan added refactor Code refactor with no behaviour change and removed bugfix labels Oct 28, 2026
@KaanOzkan
KaanOzkan merged commit 5f92c2b into main Oct 28, 2026
19 checks passed
@KaanOzkan
KaanOzkan deleted the ko-check-shims-filtered-index branch October 28, 2026 16:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

refactor Code refactor with no behaviour change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants