Skip to content

Commit 292cfa5

Browse files
authored
docs: deprecate the enableSnappy option in bigtable-beam-import (#4635)
The flag has been a no-op since #4338 (released in 2.14.0), which moved the import job to HBase 2.x / Hadoop 3. Its only effect was swapping in a custom SDK container image and enabling use_runner_v2 so that workers could decompress Snappy; that block was deleted in #4338 and nothing has read getEnableSnappy() since. Snappy compressed snapshots are now read natively. Mark the option deprecated rather than removing it so existing invocations keep working: - @deprecated on the getter/setter, and the --help description now says the option is ignored. - Log a warning when --enableSnappy=true is passed so it is not silently dropped. - README: collapse the duplicated "Snappy compressed Snapshots" command block (identical to the preferred method except for the flag) into a note that no extra configuration is needed, plus a deprecation warning. - SNAPSHOT_IMPORT_USAGE.md: mark ENABLE_SNAPPY deprecated. - run-snapshot-import.sh: stop passing --enableSnappy; warn if ENABLE_SNAPPY is set in the environment. - EndToEndIT: drop the setEnableSnappy(true) call; the test still imports the snappy snapshot. Change-Id: I7113f112309c7813fba7cc0d7c9b73401029d620
1 parent a674aba commit 292cfa5

5 files changed

Lines changed: 42 additions & 31 deletions

File tree

‎bigtable-dataflow-parent/bigtable-beam-import/README.md‎

Lines changed: 9 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -167,32 +167,15 @@ workers accordingly.
167167
168168
### Snappy compressed Snapshots
169169
170-
1. Set the environment variables.
171-
```
172-
PROJECT_ID=your-project-id
173-
INSTANCE_ID=your-instance-id
174-
TABLE_NAME=your-table-name
175-
REGION=us-central1
176-
177-
SNAPSHOT_GCS_PATH="$BUCKET_NAME/hbase-migration-snap"
178-
SNAPSHOT_NAME=your-snapshot-name
179-
```
180-
181-
1. Run the import.
182-
```
183-
java -jar bigtable-beam-import-2.3.0.jar importsnapshot \
184-
--runner=DataflowRunner \
185-
--project=$PROJECT_ID \
186-
--bigtableInstanceId=$INSTANCE_ID \
187-
--bigtableTableId=$TABLE_NAME \
188-
--hbaseSnapshotSourceDir=$SNAPSHOT_GCS_PATH/data \
189-
--snapshotName=$SNAPSHOT_NAME \
190-
--stagingLocation=$SNAPSHOT_GCS_PATH/staging \
191-
--gcpTempLocation=$SNAPSHOT_GCS_PATH/temp \
192-
--maxNumWorkers=$(expr 3 \* $CLUSTER_NUM_NODES) \
193-
--region=$REGION \
194-
--enableSnappy=true
195-
```
170+
Snappy compressed snapshots are read natively, so they are imported with the
171+
exact same command as above, with no extra configuration.
172+
173+
> [!WARNING]
174+
> The `--enableSnappy` flag is **deprecated**. Since
175+
> [#4338](https://github.com/googleapis/java-bigtable-hbase/pull/4338) (released
176+
> in 2.14.0) the import job runs on HBase 2.x / Hadoop 3, which decompresses
177+
> Snappy without the custom worker container the flag used to install. The flag
178+
> is accepted but ignored, and it will be removed in a future release.
196179
197180
### Sequence Files
198181

‎bigtable-dataflow-parent/bigtable-beam-import/SNAPSHOT_IMPORT_USAGE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ The script relies on the following environment variables. You should set them be
2626
| `DISK_SIZE_GB` | *Optional* | Worker disk size in Gigabytes. | `500` (Default) |
2727
| `MAX_NUM_WORKERS` | *Optional* | Maximum number of active Dataflow workers per job. | `10` (Default) |
2828
| `USE_PUBLIC_IPS` | *Optional* | Whether Dataflow workers should have public IPs. | `false` (Default) |
29-
| `ENABLE_SNAPPY` | *Optional* | Whether to enable Snappy compression for transit files. | `true` (Default) |
29+
| `ENABLE_SNAPPY` | **Deprecated** | Ignored. Snappy compressed snapshots are read natively, so no flag is needed. Setting it only prints a warning. | N/A |
3030

3131
## Understanding Sharding
3232

‎bigtable-dataflow-parent/bigtable-beam-import/bin/run-snapshot-import.sh‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,11 @@ WORKER_MACHINE_TYPE="${WORKER_MACHINE_TYPE:-n1-highmem-4}"
114114
DISK_SIZE_GB="${DISK_SIZE_GB:-500}"
115115
MAX_NUM_WORKERS="${MAX_NUM_WORKERS:-10}"
116116
USE_PUBLIC_IPS="${USE_PUBLIC_IPS:-false}"
117-
ENABLE_SNAPPY="${ENABLE_SNAPPY:-true}"
117+
118+
# Deprecated: Snappy compressed snapshots are read natively, this variable is ignored.
119+
if [ -n "${ENABLE_SNAPPY:-}" ]; then
120+
echo "⚠️ Warning: ENABLE_SNAPPY is deprecated and ignored. Snappy compressed snapshots are supported natively."
121+
fi
118122

119123
# Generate a safe, unique job name prefix to prevent collisions
120124
SAFE_TABLE_NAME=$(echo "${TABLE_NAME}" | tr '[:upper:]' '[:lower:]' | tr '_' '-' | tr -cd '[:alnum:]-')
@@ -284,7 +288,6 @@ for (( i=START_SHARD; i<=END_SHARD; i++ )); do
284288
--region="${REGION}" \
285289
"${SERVICE_ACCOUNT_ARGS[@]}" \
286290
--usePublicIps="${USE_PUBLIC_IPS}" \
287-
--enableSnappy="${ENABLE_SNAPPY}" \
288291
--skipRestoreStep="${SKIP_RESTORE}" \
289292
--deleteRestoredSnapshots=false \
290293
--restorePath="${RESTORE_DIR}" \

‎bigtable-dataflow-parent/bigtable-beam-import/src/main/java/com/google/cloud/bigtable/beam/hbasesnapshots/ImportJobFromHbaseSnapshot.java‎

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -120,10 +120,23 @@ public interface ImportOptions extends ImportJob.ImportOptions {
120120
@SuppressWarnings("unused")
121121
void setSnapshotName(String snapshotName);
122122

123-
@Description("Is importing Snappy compressed snapshot.")
123+
/**
124+
* @deprecated This option is a no-op since the job moved to HBase 2.x / Hadoop 3. Snappy
125+
* compressed snapshots are read natively, so the flag no longer needs to be set and is
126+
* ignored. It will be removed in a future release.
127+
*/
128+
@Deprecated
129+
@Description(
130+
"Deprecated: this option is ignored. Snappy compressed snapshots are supported natively"
131+
+ " and no longer require a flag.")
124132
@Default.Boolean(false)
125133
Boolean getEnableSnappy();
126134

135+
/**
136+
* @deprecated This option is a no-op since the job moved to HBase 2.x / Hadoop 3. See {@link
137+
* #getEnableSnappy()}.
138+
*/
139+
@Deprecated
127140
@SuppressWarnings("unused")
128141
void setEnableSnappy(Boolean enableSnappy);
129142

@@ -260,6 +273,8 @@ public static void main(String[] args) throws Exception {
260273
// To determine the Google Cloud Storage file scheme (gs://)
261274
FileSystems.setDefaultPipelineOptions(options);
262275

276+
warnIfSnappyFlagSet(options);
277+
263278
LOG.info("Building Pipeline");
264279
Pipeline pipeline = null;
265280
ImportConfig importConfig = null;
@@ -286,6 +301,16 @@ public static void main(String[] args) throws Exception {
286301
}
287302
}
288303

304+
@SuppressWarnings("deprecation")
305+
private static void warnIfSnappyFlagSet(ImportOptions options) {
306+
if (Boolean.TRUE.equals(options.getEnableSnappy())) {
307+
LOG.warn(
308+
"--enableSnappy is deprecated and ignored. Snappy compressed snapshots are read natively"
309+
+ " since the job moved to HBase 2.x / Hadoop 3; the flag will be removed in a future"
310+
+ " release.");
311+
}
312+
}
313+
289314
@VisibleForTesting
290315
static ImportConfig buildImportConfigFromConfigFile(String configFilePath) throws Exception {
291316
Gson gson = new GsonBuilder().create();

‎bigtable-dataflow-parent/bigtable-beam-import/src/test/java/com/google/cloud/bigtable/beam/hbasesnapshots/EndToEndIT.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -419,8 +419,8 @@ public void testHBaseSnapshotImportWithCorruptions() throws Exception {
419419
@Test
420420
public void testSnappyCompressedHBaseSnapshotImport() throws Exception {
421421
// Start import
422+
// Snappy compressed snapshots are read natively, no flag needed.
422423
ImportOptions importOpts = createImportOptions();
423-
importOpts.setEnableSnappy(true);
424424
importOpts.setSnapshotName(TEST_SNAPPY_SNAPSHOT_NAME);
425425

426426
// run pipeline

0 commit comments

Comments
 (0)