Title: SNAC decoding expands duration (extra silence) when using encode → decode

[humair-m]

Everything works, but when I encode → decode the audio back, the reconstructed waveform becomes much longer than the original — for example:

Original duration: 2 seconds

Decoded output: ~10 seconds

The first ~2 seconds are correct

Extra ~8 seconds are silence, not noise

This happens even if I do nothing to the codes except flatten and feed them back.

Output length is consistently 3–5x longer than input, with silence appended at the end.

✅ Why it matters

If the decoded audio is longer than the original, it makes:

round-trip quality evaluation misleading

downstream training impossible (aligned speech–text pairs break)

sequence-to-sequence training drift because silence dominates

The below code I use

import numpy as np
from snac import SNAC
from datasets import load_dataset, Features, Value
from tqdm import tqdm
import warnings
warnings.filterwarnings('ignore')

# =========================================================
# CONFIGURATION - RESEARCH-BACKED VALUES
# =========================================================
OUTPUT_FILE = "dataset.txt"
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
BATCH_SIZE = 64 if DEVICE == "cuda" else 16  # Adaptive batch size
NUM_PREFETCH = 8  # Prefetch batches for speed

# SNAC 24kHz specifications (verified from official docs)
SAMPLE_RATE = 24000
CODEBOOK_SIZE = 4096  # Each codebook: 0-4095
NUM_CODEBOOKS = 3  # 3 RVQ levels for 24kHz
# Token rates: 12Hz, 23Hz, 47Hz (from documentation)

# CRITICAL: Use offsets to distinguish codebook levels for LM training
# Level 0: 0-4095, Level 1: 4096-8191, Level 2: 8192-12287
USE_CODEBOOK_OFFSET = True  # Set False for no offset (not recommended)

print(f"🔧 Device: {DEVICE}")
print(f"🔧 Batch size: {BATCH_SIZE}")
print(f"🔧 Codebook size: {CODEBOOK_SIZE} entries per level")
print(f"🔧 Using offsets: {USE_CODEBOOK_OFFSET}")

# =========================================================
# LOAD MODEL (ONCE - CRITICAL FOR SPEED)
# =========================================================
print("\n🔄 Loading SNAC model...")
model = SNAC.from_pretrained("hubertsiuzdak/snac_24khz").eval().to(DEVICE)
if DEVICE == "cuda":
    model = model.half()  # FP16 for 2x GPU speed
    torch.backends.cudnn.benchmark = True  # Auto-optimize
print("✅ Model loaded")

# =========================================================
# LOAD DATASET
# =========================================================
print("\n📂 Loading dataset...")
features = Features({
    "id": Value("int64"),
    "audio_bytes": Value("binary"),
    "transcript": Value("string"),
    "voice": Value("string"),
    "text": Value("string"),
    "timestamp": Value("string"),
    "error": Value("string"),
})
#dataset = load_dataset("humair025/myTTS", features=features, split="train")
print(f"✅ Loaded {len(dataset)} samples")

# =========================================================
# OPTIMIZED BATCH PROCESSING
# =========================================================
def bytes_to_waveform(audio_bytes):
    """Fastest bytes->waveform conversion"""
    pcm = np.frombuffer(audio_bytes, dtype=np.int16)
    return pcm.astype(np.float32) * (1.0 / 32768.0)  # Normalized float32

def prepare_batch(samples):
    """Ultra-fast batch preparation with vectorization"""
    waveforms = []
    metadata = []
    
    for s in samples:
        try:
            wav = bytes_to_waveform(s['audio_bytes'])
            waveforms.append(wav)
            metadata.append({'text': s['text'], 'voice': s['voice']})
        except:
            continue
    
    if not waveforms:
        return None, []
    
    # Vectorized padding
    max_len = max(len(w) for w in waveforms)
    batch = np.zeros((len(waveforms), max_len), dtype=np.float32)
    for i, w in enumerate(waveforms):
        batch[i, :len(w)] = w
    
    # Convert to tensor [B, 1, T]
    tensor = torch.from_numpy(batch).unsqueeze(1).to(DEVICE)
    if DEVICE == "cuda":
        tensor = tensor.half()
    
    return tensor, metadata

@torch.inference_mode()
def tokenize_batch(waveforms):
    """Batch tokenization with optimized flattening"""
    # Encode all waveforms at once
    codebooks = model.encode(waveforms)
    
    # Flatten with optional offset (flattening pattern: level0, level1, level2)
    batch_tokens = []
    B = waveforms.shape[0]
    
    for b in range(B):
        flat_codes = []
        for level, codes in enumerate(codebooks):
            # Extract codes for this sample
            sample_codes = codes[b].cpu().numpy().flatten()
            
            # Apply offset if enabled (recommended for LM training)
            if USE_CODEBOOK_OFFSET:
                offset = level * CODEBOOK_SIZE
                flat_codes.extend([int(c) + offset for c in sample_codes])
            else:
                flat_codes.extend([int(c) for c in sample_codes])
        
        # Convert to token strings
        tokens = [f"<audio_{x}>" for x in flat_codes]
        batch_tokens.append(tokens)
    
    return batch_tokens

# =========================================================
# FILTER VALID SAMPLES (SINGLE PASS)
# =========================================================
print("\n📋 Filtering valid samples...")
valid_samples = []

for row in tqdm(dataset, desc="Filtering", ncols=80):
    text = (row.get("text") or "").strip()
    voice = (row.get("voice") or "").strip()
    audio_bytes = row.get("audio_bytes")
    error = (row.get("error") or "").strip()
    
    if text and audio_bytes and not error:
        valid_samples.append({
            'text': text,
            'voice': voice,
            'audio_bytes': audio_bytes
        })

print(f"✅ Filtered: {len(valid_samples)} valid ({len(dataset) - len(valid_samples)} skipped)")

# =========================================================
# BATCH PROCESS & WRITE
# =========================================================
print(f"\n🎵 Processing in batches of {BATCH_SIZE}...\n")

successful = 0
failed = 0

with open(OUTPUT_FILE, "w", encoding="utf-8", buffering=8192*16) as f:  # Large buffer
    for i in tqdm(range(0, len(valid_samples), BATCH_SIZE), 
                  desc="Processing", ncols=80, unit="batch"):
        batch = valid_samples[i:i + BATCH_SIZE]
        
        try:
            # Prepare and tokenize batch
            waveforms, metadata = prepare_batch(batch)
            if waveforms is None:
                failed += len(batch)
                continue
            
            batch_tokens = tokenize_batch(waveforms)
            
            # Write results (buffered for speed)
            for meta, tokens in zip(metadata, batch_tokens):
                tokens_str = " ".join(tokens)
                line = (
                    f"<user><text>{meta['text']}</text></user>"
                    f"<voice {meta['voice']}><generate_audio>{tokens_str}</generate_audio>\n"
                )
                f.write(line)
                successful += 1
            
        except Exception as e:
            failed += len(batch)
            print(f"\n⚠️ Batch error: {e}")
            continue

# =========================================================
# SUMMARY
# =========================================================
print(f"\n{'='*60}")
print("✅ PROCESSING COMPLETE")
print(f"{'='*60}")
print(f"Total samples:          {len(dataset)}")
print(f"Valid samples:          {len(valid_samples)}")
print(f"Successfully processed: {successful}")
print(f"Failed:                 {failed}")
print(f"Output file:            {OUTPUT_FILE}")
print(f"Offset mode:            {'Enabled' if USE_CODEBOOK_OFFSET else 'Disabled'}")
if USE_CODEBOOK_OFFSET:
    print(f"Token range:            0-{NUM_CODEBOOKS * CODEBOOK_SIZE - 1}")
else:
    print(f"Token range:            0-{CODEBOOK_SIZE - 1} (per level)")
print(f"{'='*60}")

# Sample output
print(f"\n📄 Sample output (first 400 chars):")
print("-"*60)
try:
    with open(OUTPUT_FILE, "r", encoding="utf-8") as f:
        sample = f.readline()
        print(sample[:400] + ("..." if len(sample) > 400 else ""))
except:
    print("No output generated")
print("-"*60)

print("\n💡 Tip: Set USE_CODEBOOK_OFFSET=False if your LM uses separate")
print("   embeddings per codebook level (not recommended for most cases)")

What I get :

The dataset is available at : [!https://huggingface.co/datasets/humair025/tts-tokens]

sample

txt<user><text>اگر آپ چاہیں تو میں اس مضمون کو مزید طول دے سکتا ہوں</text></user><voice nova><generate_audio><audio_413> <audio_3490> <audio_882> <audio_3515> <audio_2831> <audio_1893> <audio_1594> <audio_2748> <audio_746> <audio_2045> <audio_1775> <audio_114> <audio_2612> <audio_2612> <audio_1288> <audio_2595> <audio_1850> <audio_277> <audio_2525> <audio_3850> <audio_1579> <audio_2872> <audio_73> <audio_426> <audio_1780> <audio_3224> <audio_19> <audio_321> <audio_1500> <audio_212> <audio_73> <audio_1306> <audio_3771> <audio_538> <audio_2016> <audio_138> <audio_3530> <audio_3850> <audio_3773> <audio_3216> <audio_898> <audio_492> <audio_1675> <audio_353> <audio_2907> <audio_2907> <audio_1942> <audio_810> <audio_717> <audio_3981> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_2068> <audio_3981> <audio_4571> <audio_5060> <audio_6986> <audio_4869> <audio_6384> <audio_6481> <audio_5144> <audio_4433> <audio_5597> <audio_6851> <audio_4925> <audio_7721> <audio_7168> <audio_4415> <audio_4862> <audio_6114> <audio_4208> <audio_7484> <audio_7226> <audio_4662> <audio_7464> <audio_6321> <audio_6316> <audio_7294> <audio_8073> <audio_4139> <audio_4139> <audio_4460> <audio_6079> <audio_7261> <audio_7261> <audio_6829> <audio_5144> <audio_7494> <audio_7100> <audio_4571> <audio_5486> <audio_6775> <audio_4758> <audio_6314> <audio_6532> <audio_4381> <audio_5481> <audio_7353> <audio_4426> <audio_6457> <audio_6521> <audio_6378> <audio_4478> <audio_6449> <audio_4941> <audio_7619> <audio_7944> <audio_6233> <audio_5569> <audio_5933> <audio_6229> <audio_5831> <audio_5795> <audio_7482> <audio_7011> <audio_7484> <audio_6400> <audio_6811> <audio_6373> <audio_6337> <audio_6206> <audio_6980> <audio_8013> <audio_5382> <audio_8061> <audio_4751> <audio_6778> <audio_4149> <audio_4288> <audio_4758> <audio_8022> <audio_4506> <audio_6094> <audio_6591> <audio_6433> <audio_7366> <audio_4896> <audio_6287> <audio_5636> <audio_6399> <audio_6919> <audio_4139> <audio_7119> <audio_5925> <audio_7119> <audio_6084> <audio_4516> <audio_7823> <audio_6831> <audio_5275> <audio_5028> <audio_5214> <audio_6705> <audio_6522> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_5832> <audio_6522> <audio_6522> <audio_9976> <audio_8690> <audio_11610> <audio_9141> <audio_10736> <audio_9032> <audio_9575> <audio_12226> <audio_8470> <audio_10533> <audio_10076> <audio_11977> <audio_11466> <audio_9737> <audio_12163> <audio_8541> <audio_9092> <audio_10796> <audio_11531> <audio_11154> <audio_11510> <audio_11575> <audio_9424> <audio_11806> <audio_11655> <audio_11839> <audio_9585> <audio_9067> <audio_8538> <audio_11354> <audio_9251> <audio_9297> <audio_10620> <audio_12150> <audio_9963> <audio_12172> <audio_11449> <audio_11086> <audio_11369> <audio_11377> <audio_8312> <audio_9535> <audio_10105> <audio_8411> <audio_9903> <audio_10191> <audio_9890> <audio_9362> <audio_11148> <audio_9019> <audio_10148> <audio_11845> <audio_9661> <audio_8368> <audio_9370> <audio_9803> <audio_10628> <audio_9030> <audio_8307> <audio_10393> <audio_10884> <audio_11039> <audio_11798> <audio_11983> <audio_11156> <audio_8811> <audio_10624> <audio_9501> <audio_11832> <audio_8246> <audio_9075> <audio_12004> <audio_9966> <audio_12188> <audio_10523> <audio_11097> <audio_12278> <audio_11895> <audio_11693> <audio_9582> <audio_10117> <audio_9481> <audio_10228> <audio_8662> <audio_11599> <audio_11439> <audio_9959> <audio_9202> <audio_10874> <audio_10801> <audio_10209> <audio_10022> <audio_11902> <audio_8808> <audio_11970> <audio_10838> <audio_11476> <audio_11680> <audio_8650> <audio_9612> <audio_10516> <audio_10508> <audio_9731> <audio_9904> <audio_8555> <audio_11377> <audio_11618> <audio_9035> <audio_11268> <audio_10994> <audio_10831> <audio_9569> <audio_8401> <audio_11835> <audio_10902> <audio_9917> <audio_10684> <audio_9627> <audio_8332> <audio_12250> <audio_11678> <audio_10553> <audio_8231> <audio_8695> <audio_12093> <audio_10228> <audio_11855> <audio_8256> <audio_11529> <audio_9855> <audio_10227> <audio_10325> <audio_11460> <audio_10151> <audio_11853> <audio_11011> <audio_10328> <audio_11935> <audio_10782> <audio_11039> <audio_10461> <audio_10413> <audio_9229> <audio_12277> <audio_11834> <audio_8726> <audio_10349> <audio_8472> <audio_12075> <audio_11097> <audio_8365> <audio_8356> <audio_8826> <audio_9132> <audio_11500> <audio_8353> <audio_9035> <audio_9312> <audio_10628> <audio_11898> <audio_10276> <audio_10560> <audio_9682> <audio_10793> <audio_8862> <audio_8702> <audio_8349> <audio_9830> <audio_11602> <audio_9481> <audio_11356> <audio_11150> <audio_10845> <audio_8407> <audio_10771> <audio_10256> <audio_9667> <audio_9506> <audio_10796> <audio_10033> <audio_8778> <audio_9147> <audio_11350> <audio_8287> <audio_9396> <audio_9192> <audio_8831> <audio_11305> <audio_10038> <audio_10470> <audio_8709> <audio_8248> <audio_9235> <audio_8883> <audio_11808> <audio_10701> <audio_10003> <audio_9760> <audio_9760> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_11614> <audio_9760> <audio_9724> <audio_11794> <audio_10047></generate_audio>

[lexkoro]

could be cause you are padding with zeros

[hasaki321]

Hello, there is a preprocess step inside the "encode" method which pad the audio to the multiplicaion of attention stride, try to check it out. Besides, you dont have to iterate through the batch dim, since the model support batch inference originally.
I wish it would help.