Cannot opt out of Dataverse tabular ingest on upload (tabIngest not passed)

[adamlabadorf]

Summary

Uploads always let Dataverse run tabular ingest. #307 / #314 addressed the
download side of this (always fetch format=original, so annex verification
passes — thank you, that works well). The upload side is still unaddressed:
ingest happens, and there is no way to ask Dataverse not to do it.

Dataverse's native API accepts a tabIngest field in jsonData on
/datasets/:persistentId/add ("controls whether or not the file skips tabular
ingest. If the tabIngest parameter is not specified, it defaults to true").
datalad-dataverse does not send it: DataverseDataset.upload_file builds a
pyDataverse Datafile with a fixed field set (label, filename,
directoryLabel, pid) and serializes with datafile.json(), which emits only
declared fields, so there is no way to add it from the outside.

Why it still matters after #314

Content retrieval is correct, so this is not a data-integrity bug. It is three
smaller problems:

1. filetree mode produces a misleading human-readable record. The whole
   point of --mode filetree is a tree a person can browse and download
   without DataLad. After ingest they see all_counts.tab, and the default
   web-UI/API download is the derived file, whose checksum does not match the
   md5 Dataverse records for it (the recorded md5 is the original's). On a
   125-column clinical table I measured, ingest rewrote NA to empty string in
   3614 of 26375 cells — semantically meaningful for a "not assessed" code, and
   invisible to anyone who does not know to ask for format=original.
2. It is not reversible by the depositor. uningest is superuser-only. On
   Harvard Dataverse a normal depositor gets
   User @… is not permitted to perform requested action, so there is no
   remediation after the fact — it has to be prevented at upload.
3. Wasted work on annex-mode remotes. Annex keys ending in .csv/.tsv
   get ingested too, so the instance stores and converts a derived copy of
   opaque key blobs that nothing will ever read. In my case that included an
   82 MB key.

Reproducer

Verified against demo.dataverse.org 6.11 and dataverse.harvard.edu 6.10.1.
Note ingest is asynchronous — the upload response always reports
tabularData: false; you have to re-query the file listing.

printf 'a,b,c\n1,NA,3\n4,5,NA\n' > t.csv

# default -> becomes t.tab, tabularData: true
curl -H "X-Dataverse-key: $TOK" -X POST \
  "$HOST/api/datasets/:persistentId/add?persistentId=$DOI" \
  -F file=@t.csv -F 'jsonData={"directoryLabel":"x"}'

# with tabIngest -> stays t.csv, tabularData: false
curl -H "X-Dataverse-key: $TOK" -X POST \
  "$HOST/api/datasets/:persistentId/add?persistentId=$DOI" \
  -F file=@t.csv -F 'jsonData={"directoryLabel":"x","tabIngest":"false"}'

tabIngest needs no special privileges — it worked from an ordinary account.

Suggested shape

Uploads happen inside git-annex TRANSFER STORE / EXPORT, where there is no
place to thread per-call options, so this wants to be special-remote
configuration persisted in remote.log alongside doi, exporttree and
credential — surfaced as an add-sibling-dataverse option, defaulting to
current behavior so nothing changes for existing siblings.

On scope, I'd suggest a small explicit allowlist of passthrough jsonData
fields rather than arbitrary passthrough, because the extension's correctness
depends on owning four of them: label, filename and directoryLabel drive
the key-to-remote-path mapping, and pid identifies the dataset. Letting a
caller override those would silently break content retrieval and the file-id
bookkeeping. An allowlist seeded with tabIngest — with restrict,
description and categories as obvious later members, and the four owned
fields hard-rejected — gets the flexibility without that hazard.

Both call sites need it: upload_datafile and replace_datafile.

Version compatibility

I noticed the concern in #307 about supporting Dataverse 4.x/5.x alongside 6.x.
tabIngest is documented in the 5.x+ native API. I have not tested how a 4.x
instance reacts to an unknown jsonData key, so if that matters it is worth
checking before defaulting anything; keeping the option opt-in avoids the
question entirely for existing users.

Environment

• datalad-dataverse 1.0.3, datalad 1.6.2, git-annex 10.20260601
• Verified end to end by depositing a ~95 MB DataLad dataset in both annex
  and filetree modes to Harvard Dataverse and demo.