Skip to content

fix(general_text): keep refined text in ReferenceRemoverRefiner output - #555

Merged
SunnyHaze merged 1 commit into
OpenDCAI:mainfrom
raams95:fix/reference-remover-refiner-output
Oct 19, 2026
Merged

SunnyHaze merged 1 commit into
OpenDCAI:mainfrom
raams95:fix/reference-remover-refiner-output

Conversation

@raams95

@raams95 raams95 commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

Summary

ReferenceRemoverRefiner computes the cleaned text (with <ref> tags and {{cite ...}} templates removed) but then appends the original item to the output list. The result is that the operator never changes anything, and references are still in the output column.

ReferenceRemoverRefiner().run(storage, input_key="text")
# "Paris is the capital of France.<ref>Smith 2020</ref>" comes back unchanged

The "Total modified items" log was also off: numbers was incremented twice for each modified row.

Fix

Append refined_text instead of item, and drop the extra numbers += 1 so each modified row is counted once. Rows without references are returned as-is, same as before.

Tests

I added test/cpu_only/test_reference_remover_refiner.py. It runs the refiner on a small dataframe with a <ref> tag, a {{cite ...}} template and a plain row, then checks the output column.

  • Before the fix: 1 failed (output equals the input)
  • After the fix: 1 passed
pytest -m "cpu or api" test/cpu_only/test_reference_remover_refiner.py -q
1 passed

Fixes #554

@SunnyHaze
SunnyHaze merged commit 0c203ac into OpenDCAI:main Oct 19, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ReferenceRemoverRefiner leaves the text unchanged (refined result is discarded)

2 participants