Skip to content

fix: preserve multiline fields during import - #27596

Merged
gwossum merged 4 commits into
influxdata:master-1.xfrom
1fanwang:stewang-linkedin-fix-influx-multiline-import
Aug 25, 2026
Merged

fix: preserve multiline fields during import#27596
gwossum merged 4 commits into
influxdata:master-1.xfrom
1fanwang:stewang-linkedin-fix-influx-multiline-import

Conversation

@1fanwang

@1fanwang 1fanwang commented Aug 21, 2026

Copy link
Copy Markdown

influx_inspect export preserves literal newlines inside quoted string fields. influx -import counted those physical lines toward its 5,000-line batch, so a batch could end inside a field and fail to round trip with unbalanced quotes.

The importer now batches complete line-protocol records. Record framing uses models.ScanLine for quotes and escapes, rejects a single record above the server's default 25 MB request-body limit, and flushes valid records collected before a later read error.

Fixes #27589

Testing

5,000-line red/green repro and package output
Setup:
$ go build -o /tmp/influx-pkg-config github.com/influxdata/pkg-config

Before (afddccc2baec3ffc771cd1a120dbe8ace28f7bc2):
$ PKG_CONFIG=/tmp/influx-pkg-config go test ./importer/v8 -run '^TestImporter_MultilineFieldsAcrossBatchBoundary$' -count=1 -v
hhhhhhhhhh': unbalanced quotes
import failed: 6012 points were not inserted
--- FAIL: TestImporter_MultilineFieldsAcrossBatchBoundary (0.01s)
FAIL github.com/influxdata/influxdb/importer/v8 0.661s

After (2615fa1c145ed0affb433727b07ec82949e21d5a):
$ PKG_CONFIG=/tmp/influx-pkg-config go test ./importer/v8 -run '^TestImporter_(MultilineFieldsAcrossBatchBoundary|FlushesBatchOnReadError|UnterminatedFinalPoint)$' -count=1 -v
--- PASS: TestImporter_MultilineFieldsAcrossBatchBoundary (0.01s)
--- PASS: TestImporter_UnterminatedFinalPoint (0.00s)
--- PASS: TestImporter_FlushesBatchOnReadError (0.00s)
PASS
ok github.com/influxdata/influxdb/importer/v8 0.763s

$ PKG_CONFIG=/tmp/influx-pkg-config go test ./importer/v8 ./models -count=1
ok github.com/influxdata/influxdb/importer/v8 0.532s
ok github.com/influxdata/influxdb/models 2.293s

$ PKG_CONFIG=/tmp/influx-pkg-config go test -race ./importer/v8 -count=1
ok github.com/influxdata/influxdb/importer/v8 1.732s
Required for all non-trivial PRs
  • Sign CLA (if not already signed). The repository check reports this after PR creation.

Signed-off-by: 1fanwang <1fannnw@gmail.com>
Signed-off-by: 1fanwang <1fannnw@gmail.com>

@gwossum gwossum left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! This correctly identifies the issue, that influx import assumes 1 line == 1 point and disregards quoting.

My main request is to re-use models.scanLine instead of copying it. Most of the rest is matching the projects modern test code style, as well as a few questions.

Comment thread importer/v8/importer.go Outdated
Comment thread importer/v8/importer_test.go Outdated
Comment thread importer/v8/importer_test.go Outdated
Comment thread importer/v8/importer_test.go Outdated
Comment thread importer/v8/importer_test.go Outdated
Comment thread importer/v8/importer_test.go Outdated
Comment thread importer/v8/importer_test.go Outdated
Comment thread importer/v8/importer_test.go Outdated
Comment thread importer/v8/importer_test.go
Comment thread importer/v8/importer.go Outdated
Signed-off-by: 1fanwang <1fannnw@gmail.com>
Comment thread importer/v8/importer_test.go
Comment thread importer/v8/importer.go Outdated
Comment thread models/points_test.go Outdated
Comment thread models/points_test.go Outdated
Comment thread importer/v8/importer.go

@gwossum gwossum left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Almost there! Just a couple of tweaks and this can get merged. I also added some non-blocking comments. Nothing needs to be done on those for approval.

Signed-off-by: 1fanwang <1fannnw@gmail.com>
testImportRecords(t, []string{"test value=1i 1"}, false)
}

func TestImporter_FlushesBatchOnReadError(t *testing.T) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like that a test was added for this.

@gwossum gwossum left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks for the PR, and congratulations on your first InfluxDB PR!

@gwossum
gwossum merged commit 8a2663a into influxdata:master-1.x Aug 25, 2026
6 of 10 checks passed
@1fanwang
1fanwang deleted the stewang-linkedin-fix-influx-multiline-import branch August 28, 2026 12:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants