Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 96 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,102 @@ swift format --in-place --recursive --configuration .swift-format Sources/ Tests
- Thread safety: Use actors, `@MainActor`, or proper locking - never `@unchecked Sendable`
- Control flow: Prefer flattened if statements with early returns/continues over nested if statements. Use guard statements and inverted conditions to exit early. Nested if statements should be absolutely avoided.

## Swift 6 Concurrency Migration

### Common Warning Patterns & Fixes

#### 1. Non-Sendable Struct Types
When a struct is used across actor boundaries or in concurrent contexts, add `Sendable` conformance:

```swift
// Before
struct MyConfig {
let timeout: Int
}

// After
struct MyConfig: Sendable {
let timeout: Int
}
```

**Examples from codebase:**
- `TdtDecoderState` (ASR/TDT/TdtDecoderState.swift) - LSTM state management
- `AssignmentConfig` (Diarizer/Clustering/SpeakerOperations.swift) - speaker assignment config
- `DownloadConfig` (DownloadUtils.swift) - download timeout settings

#### 2. Function Type Aliases
Closure types that cross actor boundaries must be marked `@Sendable`:

```swift
// Before
public typealias DataWriter = (Data, URL) throws -> Void

// After
public typealias DataWriter = @Sendable (Data, URL) throws -> Void
```

**Applied to:** AssetDownloader.swift - `DataWriter` and `FileMover` typealias

#### 3. Singleton/Shared Static Properties
Global actor isolation for non-Sendable classes:

```swift
// Before
static let shared = EspeakG2P()

// After
@MainActor static let shared = EspeakG2P()
```

**Applied to:** EspeakG2P.swift - eSpeak NG wrapper singleton

#### 4. Mutable Global State (#MutableGlobalVariable)
Mutable static properties require proper synchronization or actor isolation. Options:

**Option A: Use @MainActor for entire type**
```swift
@MainActor
class MyService {
static var cache: [String: Data] = [:]
static let cacheLock = NSLock()
}
```

**Option B: Use actor for concurrent access**
```swift
actor CacheManager {
private var cache: [String: Data] = [:]

func set(_ key: String, _ value: Data) {
cache[key] = value
}

func get(_ key: String) -> Data? {
cache[key]
}
}
```

**Current cases:** KokoroSynthesizer.swift - voiceEmbeddingPayloads, voiceEmbeddingVectors

#### 5. Non-Sendable Framework Types (MLMultiArray)
CoreML's `MLMultiArray` doesn't conform to Sendable. When passing across actor boundaries:
- Wrap in a Sendable struct/class
- Use `@MainActor` for related processing
- Create separate Sendable representations for cross-actor data

### Migration Checklist

When fixing concurrency warnings:
1. Run `swift build` and capture full warning output
2. Identify warning categories (Sendable, @MainActor, #MutableGlobalVariable, etc.)
3. Start with "low-hanging fruit": simple struct/typealias additions
4. Address mutable state with proper synchronization
5. Handle framework non-Sendable types last (most complex)
6. Run `swift build` again to verify warning reduction
7. Never use `@unchecked Sendable` as a shortcut

## Clean code

- When adding new interfaces, make sure that the API is consistent with the other model managers
Expand Down
29 changes: 26 additions & 3 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,9 @@ FluidAudio/
- **Persistent States**: Decoder states maintained across chunks for streaming
- **Memory Management**: Automatic cleanup and ANE optimization
- **Parallel Processing**: Multi-stream support for batch operations
- **Swift 6 Strict Concurrency Migration** (in progress):
- **Completed**: TdtDecoderState, AssignmentConfig, DownloadConfig, AssetDownloader, EspeakG2P
- **Remaining**: KokoroSynthesizer mutable state, MLMultiArray non-Sendable issues

### Model Management
- **Automatic Downloads**: Models fetched from HuggingFace on first use
Expand Down Expand Up @@ -297,11 +300,31 @@ The project uses GitHub Actions with the following workflows:
9. **Git Operations**: NEVER run `git push` unless explicitly requested by the user. Only commit when asked.
10. **Code Formatting**: All code must pass swift-format checks before merge

## Swift 6 Concurrency Migration Status

### Completed Fixes
- **TdtDecoderState** (ASR/TDT/TdtDecoderState.swift): Added `Sendable` conformance for LSTM state across actor boundaries
- **AssignmentConfig** (Diarizer/Clustering/SpeakerOperations.swift): Added `Sendable` conformance for clustering configuration
- **DownloadConfig** (DownloadUtils.swift): Added `Sendable` conformance for download settings
- **AssetDownloader** (Shared/AssetDownloader.swift): Marked `DataWriter` and `FileMover` typealias as `@Sendable`
- **EspeakG2P** (TextToSpeech/Kokoro/Assets/Lexicon/EspeakG2P.swift): Added `@MainActor` to shared singleton

### Remaining Work
- **KokoroSynthesizer** (TextToSpeech/Kokoro/Pipeline/Synthesize/KokoroSynthesizer.swift):
- Mutable static state: `voiceEmbeddingPayloads`, `voiceEmbeddingVectors` (lines 58-59)
- Requires `@MainActor` annotation or actor-based refactoring
- Multiple MLMultiArray sending warnings (framework limitation)

### Framework Limitations
- **MLMultiArray** (CoreML): Does not conform to Sendable - wrap in Sendable types for cross-actor use
- **Non-Sendable Types**: Create Sendable wrapper structs when passing non-Sendable data across actor boundaries

## Next Steps

1. **Multi-file validation**: Test optimal config on all AMI files
2. **Real-world testing**: Validate on non-AMI audio
3. **Documentation**: Update API documentation
1. **Complete Swift 6 Migration**: Finish KokoroSynthesizer mutable state handling
2. **Multi-file validation**: Test optimal config on all AMI files
3. **Real-world testing**: Validate on non-AMI audio
4. **Documentation**: Update API documentation

## Testing Strategy

Expand Down
10 changes: 8 additions & 2 deletions Package.swift
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,10 @@ let package = Package(
"FastClusterWrapper",
],
path: "Sources/FluidAudio",
exclude: ["Frameworks"]
exclude: ["Frameworks"],
swiftSettings: [
.enableUpcomingFeature("StrictConcurrency"),
]
),
.target(
name: "FastClusterWrapper",
Expand All @@ -48,7 +51,10 @@ let package = Package(
),
.testTarget(
name: "FluidAudioTests",
dependencies: ["FluidAudio"]
dependencies: ["FluidAudio"],
swiftSettings: [
.enableUpcomingFeature("StrictConcurrency"),
]
),
],
cxxLanguageStandard: .cxx17
Expand Down
2 changes: 1 addition & 1 deletion Sources/FluidAudio/ASR/AsrManager.swift
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import AVFoundation
@preconcurrency import AVFoundation
import CoreML
import Foundation
import OSLog
Expand Down
2 changes: 1 addition & 1 deletion Sources/FluidAudio/ASR/Streaming/StreamingAsrManager.swift
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
import AVFoundation
@preconcurrency import AVFoundation
import Foundation
import OSLog

Expand Down
2 changes: 1 addition & 1 deletion Sources/FluidAudio/ASR/TDT/TdtDecoderState.swift
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import CoreML
import Foundation

/// Manages LSTM hidden and cell states for the Parakeet decoder
struct TdtDecoderState {
struct TdtDecoderState: Sendable {
var hiddenState: MLMultiArray
var cellState: MLMultiArray
/// Stores the last decoded token from the previous audio chunk.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ public enum SpeakerUtilities {
// MARK: - Configuration

/// Platform-specific configuration for speaker assignment
public struct AssignmentConfig {
public struct AssignmentConfig: Sendable {
public let maxDistanceForAssignment: Float
public let maxDistanceForUpdate: Float
public let minSpeakerDuration: Float
Expand Down Expand Up @@ -128,7 +128,7 @@ public enum SpeakerUtilities {
// MARK: - Speaker Assignment Decision

/// Decision result for speaker assignment
public struct AssignmentDecision {
public struct AssignmentDecision: Sendable {
public let shouldAssign: Bool
public let shouldUpdate: Bool
public let confidence: Float
Expand Down Expand Up @@ -199,7 +199,7 @@ public enum SpeakerUtilities {
// MARK: - Speaker Creation

/// Validated speaker creation parameters
public struct SpeakerCreationParams {
public struct SpeakerCreationParams: Sendable {
public let id: String
public let name: String
public let duration: Float
Expand Down Expand Up @@ -379,7 +379,7 @@ public enum SpeakerUtilities {
// MARK: - Complete Speaker Update Operations

/// Complete speaker update operation including raw tracking
public struct SpeakerUpdateResult {
public struct SpeakerUpdateResult: Sendable {
public let updatedMainEmbedding: [Float]?
public let updatedRawEmbeddings: [RawEmbedding]
public let updatedDuration: Float
Expand Down
2 changes: 1 addition & 1 deletion Sources/FluidAudio/DownloadUtils.swift
Original file line number Diff line number Diff line change
Expand Up @@ -248,7 +248,7 @@ public class DownloadUtils {
public typealias ProgressHandler = (Double) -> Void

/// Download configuration
public struct DownloadConfig {
public struct DownloadConfig: Sendable {
public let timeout: TimeInterval

public init(timeout: TimeInterval = 1800) { // 30 minutes for large models
Expand Down
2 changes: 1 addition & 1 deletion Sources/FluidAudio/FluidAudioSwift.swift
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,6 @@ public typealias SpeakerDiarizationError = DiarizerError
// Types like RawEmbedding and SendableSpeaker are already public in their respective files
// and will be available when importing FluidAudio module

public struct FluidAudio {
public struct FluidAudio: Sendable {
// Empty struct for namespace - all functionality is in the module's types
}
9 changes: 6 additions & 3 deletions Sources/FluidAudio/Shared/AppLogger.swift
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@ import OSLog

/// Lightweight logger that writes to Unified Logging and, optionally, to console.
/// Use this instead of `OSLog.Logger` so CLI runs can surface logs without `print`.
public struct AppLogger {
public struct AppLogger: Sendable {
/// Default subsystem for all loggers in FluidAudio.
/// Keep this consistent; categories should vary per component.
public static var defaultSubsystem: String = "com.fluidinference"
public static let defaultSubsystem: String = "com.fluidinference"

public enum Level: Int {
public enum Level: Int, Sendable {
case debug = 0
case info
case notice
Expand Down Expand Up @@ -82,6 +82,9 @@ public struct AppLogger {
}

private func logToConsole(_ level: Level, _ message: String) {
let level = level
let category = category
let message = message
Task.detached(priority: .utility) {
await LogConsole.shared.write(level: level, category: category, message: message)
}
Expand Down
8 changes: 4 additions & 4 deletions Sources/FluidAudio/Shared/AssetDownloader.swift
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ import OSLog
/// Provides reusable logic for simple file/data transfers used across FluidAudio modules.
public enum AssetDownloader {

public typealias DataWriter = (Data, URL) throws -> Void
public typealias FileMover = (URL, URL) throws -> Void
public typealias DataWriter = @Sendable (Data, URL) throws -> Void
public typealias FileMover = @Sendable (URL, URL) throws -> Void

public static let defaultDataWriter: DataWriter = { data, destination in
try data.write(to: destination, options: [.atomic])
Expand All @@ -19,12 +19,12 @@ public enum AssetDownloader {
try FileManager.default.moveItem(at: tempURL, to: destination)
}

public enum TransferMode {
public enum TransferMode: Sendable {
case data(DataWriter = AssetDownloader.defaultDataWriter)
case file(FileMover = AssetDownloader.defaultFileMover)
}

public struct Descriptor {
public struct Descriptor: Sendable {
public let description: String
public let remoteURL: URL
public let destinationURL: URL
Expand Down
Loading
Loading