Skip to content

Commit 9c712db

Browse files
BrandonWengBrandonWeng
committed
migrate espeakg2p
1 parent a04b4d5 commit 9c712db

5 files changed

Lines changed: 159 additions & 45 deletions

File tree

‎AGENTS.md‎

Lines changed: 96 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,102 @@ swift format --in-place --recursive --configuration .swift-format Sources/ Tests
3737
- Thread safety: Use actors, `@MainActor`, or proper locking - never `@unchecked Sendable`
3838
- Control flow: Prefer flattened if statements with early returns/continues over nested if statements. Use guard statements and inverted conditions to exit early. Nested if statements should be absolutely avoided.
3939

40+
## Swift 6 Concurrency Migration
41+
42+
### Common Warning Patterns & Fixes
43+
44+
#### 1. Non-Sendable Struct Types
45+
When a struct is used across actor boundaries or in concurrent contexts, add `Sendable` conformance:
46+
47+
```swift
48+
// Before
49+
struct MyConfig {
50+
let timeout: Int
51+
}
52+
53+
// After
54+
struct MyConfig: Sendable {
55+
let timeout: Int
56+
}
57+
```
58+
59+
**Examples from codebase:**
60+
- `TdtDecoderState` (ASR/TDT/TdtDecoderState.swift) - LSTM state management
61+
- `AssignmentConfig` (Diarizer/Clustering/SpeakerOperations.swift) - speaker assignment config
62+
- `DownloadConfig` (DownloadUtils.swift) - download timeout settings
63+
64+
#### 2. Function Type Aliases
65+
Closure types that cross actor boundaries must be marked `@Sendable`:
66+
67+
```swift
68+
// Before
69+
public typealias DataWriter = (Data, URL) throws -> Void
70+
71+
// After
72+
public typealias DataWriter = @Sendable (Data, URL) throws -> Void
73+
```
74+
75+
**Applied to:** AssetDownloader.swift - `DataWriter` and `FileMover` typealias
76+
77+
#### 3. Singleton/Shared Static Properties
78+
Global actor isolation for non-Sendable classes:
79+
80+
```swift
81+
// Before
82+
static let shared = EspeakG2P()
83+
84+
// After
85+
@MainActor static let shared = EspeakG2P()
86+
```
87+
88+
**Applied to:** EspeakG2P.swift - eSpeak NG wrapper singleton
89+
90+
#### 4. Mutable Global State (#MutableGlobalVariable)
91+
Mutable static properties require proper synchronization or actor isolation. Options:
92+
93+
**Option A: Use @MainActor for entire type**
94+
```swift
95+
@MainActor
96+
class MyService {
97+
static var cache: [String: Data] = [:]
98+
static let cacheLock = NSLock()
99+
}
100+
```
101+
102+
**Option B: Use actor for concurrent access**
103+
```swift
104+
actor CacheManager {
105+
private var cache: [String: Data] = [:]
106+
107+
func set(_ key: String, _ value: Data) {
108+
cache[key] = value
109+
}
110+
111+
func get(_ key: String) -> Data? {
112+
cache[key]
113+
}
114+
}
115+
```
116+
117+
**Current cases:** KokoroSynthesizer.swift - voiceEmbeddingPayloads, voiceEmbeddingVectors
118+
119+
#### 5. Non-Sendable Framework Types (MLMultiArray)
120+
CoreML's `MLMultiArray` doesn't conform to Sendable. When passing across actor boundaries:
121+
- Wrap in a Sendable struct/class
122+
- Use `@MainActor` for related processing
123+
- Create separate Sendable representations for cross-actor data
124+
125+
### Migration Checklist
126+
127+
When fixing concurrency warnings:
128+
1. Run `swift build` and capture full warning output
129+
2. Identify warning categories (Sendable, @MainActor, #MutableGlobalVariable, etc.)
130+
3. Start with "low-hanging fruit": simple struct/typealias additions
131+
4. Address mutable state with proper synchronization
132+
5. Handle framework non-Sendable types last (most complex)
133+
6. Run `swift build` again to verify warning reduction
134+
7. Never use `@unchecked Sendable` as a shortcut
135+
40136
## Clean code
41137

42138
- When adding new interfaces, make sure that the API is consistent with the other model managers

‎CLAUDE.md‎

Lines changed: 26 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -232,6 +232,9 @@ FluidAudio/
232232
- **Persistent States**: Decoder states maintained across chunks for streaming
233233
- **Memory Management**: Automatic cleanup and ANE optimization
234234
- **Parallel Processing**: Multi-stream support for batch operations
235+
- **Swift 6 Strict Concurrency Migration** (in progress):
236+
- **Completed**: TdtDecoderState, AssignmentConfig, DownloadConfig, AssetDownloader, EspeakG2P
237+
- **Remaining**: KokoroSynthesizer mutable state, MLMultiArray non-Sendable issues
235238

236239
### Model Management
237240
- **Automatic Downloads**: Models fetched from HuggingFace on first use
@@ -297,11 +300,31 @@ The project uses GitHub Actions with the following workflows:
297300
9. **Git Operations**: NEVER run `git push` unless explicitly requested by the user. Only commit when asked.
298301
10. **Code Formatting**: All code must pass swift-format checks before merge
299302

303+
## Swift 6 Concurrency Migration Status
304+
305+
### Completed Fixes
306+
- **TdtDecoderState** (ASR/TDT/TdtDecoderState.swift): Added `Sendable` conformance for LSTM state across actor boundaries
307+
- **AssignmentConfig** (Diarizer/Clustering/SpeakerOperations.swift): Added `Sendable` conformance for clustering configuration
308+
- **DownloadConfig** (DownloadUtils.swift): Added `Sendable` conformance for download settings
309+
- **AssetDownloader** (Shared/AssetDownloader.swift): Marked `DataWriter` and `FileMover` typealias as `@Sendable`
310+
- **EspeakG2P** (TextToSpeech/Kokoro/Assets/Lexicon/EspeakG2P.swift): Added `@MainActor` to shared singleton
311+
312+
### Remaining Work
313+
- **KokoroSynthesizer** (TextToSpeech/Kokoro/Pipeline/Synthesize/KokoroSynthesizer.swift):
314+
- Mutable static state: `voiceEmbeddingPayloads`, `voiceEmbeddingVectors` (lines 58-59)
315+
- Requires `@MainActor` annotation or actor-based refactoring
316+
- Multiple MLMultiArray sending warnings (framework limitation)
317+
318+
### Framework Limitations
319+
- **MLMultiArray** (CoreML): Does not conform to Sendable - wrap in Sendable types for cross-actor use
320+
- **Non-Sendable Types**: Create Sendable wrapper structs when passing non-Sendable data across actor boundaries
321+
300322
## Next Steps
301323

302-
1. **Multi-file validation**: Test optimal config on all AMI files
303-
2. **Real-world testing**: Validate on non-AMI audio
304-
3. **Documentation**: Update API documentation
324+
1. **Complete Swift 6 Migration**: Finish KokoroSynthesizer mutable state handling
325+
2. **Multi-file validation**: Test optimal config on all AMI files
326+
3. **Real-world testing**: Validate on non-AMI audio
327+
4. **Documentation**: Update API documentation
305328

306329
## Testing Strategy
307330

‎Sources/FluidAudio/TextToSpeech/Kokoro/Assets/Lexicon/EspeakG2P.swift‎

Lines changed: 18 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ import Foundation
33

44
/// Thread-safe wrapper around eSpeak NG C API to get IPA phonemes for a word.
55
/// Uses espeak_TextToPhonemes with IPA mode.
6-
final class EspeakG2P {
6+
actor EspeakG2P {
77
enum EspeakG2PError: Error, LocalizedError {
88
case frameworkBundleMissing
99
case dataBundleMissing
@@ -30,38 +30,33 @@ final class EspeakG2P {
3030
static let shared = EspeakG2P()
3131
private let logger = AppLogger(subsystem: "com.fluidaudio.tts", category: "EspeakG2P")
3232

33-
private let queue = DispatchQueue(label: "com.fluidaudio.tts.espeak.g2p")
3433
private var initialized = false
3534
private var currentVoice: String = ""
3635

3736
private init() {}
3837

3938
deinit {
40-
queue.sync {
41-
if initialized {
42-
espeak_Terminate()
43-
}
39+
if initialized {
40+
espeak_Terminate()
4441
}
4542
}
4643

4744
func phonemize(word: String, espeakVoice: String = "en-us") throws -> [String]? {
48-
return try queue.sync {
49-
try initializeIfNeeded(espeakVoice: espeakVoice)
50-
return word.withCString { cstr -> [String]? in
51-
var raw: UnsafeRawPointer? = UnsafeRawPointer(cstr)
52-
let modeIPA = Int32(espeakPHONEMES_IPA)
53-
let textmode = Int32(espeakCHARS_AUTO)
54-
guard let outPtr = espeak_TextToPhonemes(&raw, textmode, modeIPA) else {
55-
logger.warning("espeak_TextToPhonemes returned nil for word: \(word)")
56-
return nil
57-
}
58-
let phonemeString = String(cString: outPtr)
59-
if phonemeString.isEmpty { return nil }
60-
if phonemeString.contains(where: { $0.isWhitespace }) {
61-
return phonemeString.split { $0.isWhitespace }.map { String($0) }
62-
} else {
63-
return phonemeString.unicodeScalars.map { String($0) }
64-
}
45+
try initializeIfNeeded(espeakVoice: espeakVoice)
46+
return word.withCString { cstr -> [String]? in
47+
var raw: UnsafeRawPointer? = UnsafeRawPointer(cstr)
48+
let modeIPA = Int32(espeakPHONEMES_IPA)
49+
let textmode = Int32(espeakCHARS_AUTO)
50+
guard let outPtr = espeak_TextToPhonemes(&raw, textmode, modeIPA) else {
51+
logger.warning("espeak_TextToPhonemes returned nil for word: \(word)")
52+
return nil
53+
}
54+
let phonemeString = String(cString: outPtr)
55+
if phonemeString.isEmpty { return nil }
56+
if phonemeString.contains(where: { $0.isWhitespace }) {
57+
return phonemeString.split { $0.isWhitespace }.map { String($0) }
58+
} else {
59+
return phonemeString.unicodeScalars.map { String($0) }
6560
}
6661
}
6762
}

‎Sources/FluidAudio/TextToSpeech/Kokoro/Pipeline/Preprocess/KokoroChunker.swift‎

Lines changed: 18 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ enum KokoroChunker {
4040
hasLanguageToken: Bool,
4141
allowedPhonemes: Set<String>,
4242
phoneticOverrides: [TtsPhoneticOverride]
43-
) throws -> [TextChunk] {
43+
) async throws -> [TextChunk] {
4444
let trimmed = text.trimmingCharacters(in: .whitespacesAndNewlines)
4545
guard !trimmed.isEmpty else { return [] }
4646

@@ -60,7 +60,7 @@ enum KokoroChunker {
6060
return []
6161
}
6262

63-
let mergedSentences = try mergeShortSentences(
63+
let mergedSentences = try await mergeShortSentences(
6464
refinedSentences,
6565
lexicon: wordToPhonemes,
6666
caseSensitiveLexicon: caseSensitiveLexicon,
@@ -74,7 +74,7 @@ enum KokoroChunker {
7474
segmentsByPunctuations.reserveCapacity(segmentsByPeriods.count)
7575

7676
for (periodIndex, segment) in segmentsByPeriods.enumerated() {
77-
let count = try tokenCountForSegment(
77+
let count = try await tokenCountForSegment(
7878
for: segment,
7979
lexicon: wordToPhonemes,
8080
caseSensitiveLexicon: caseSensitiveLexicon,
@@ -84,7 +84,7 @@ enum KokoroChunker {
8484

8585
if count > capacity {
8686
let fragments = splitByPunctuation(segment)
87-
let reassembled = try reassembleFragments(
87+
let reassembled = try await reassembleFragments(
8888
fragments,
8989
lexicon: wordToPhonemes,
9090
caseSensitiveLexicon: caseSensitiveLexicon,
@@ -120,7 +120,7 @@ enum KokoroChunker {
120120
chunks.reserveCapacity(segmentsByPunctuations.count)
121121

122122
for chunkText in segmentsByPunctuations {
123-
let built = try buildChunks(
123+
let built = try await buildChunks(
124124
from: chunkText,
125125
lexicon: wordToPhonemes,
126126
caseSensitiveLexicon: caseSensitiveLexicon,
@@ -181,7 +181,7 @@ enum KokoroChunker {
181181
caseSensitiveLexicon: [String: [String]],
182182
allowed: Set<String>,
183183
capacity: Int
184-
) throws -> [String] {
184+
) async throws -> [String] {
185185
guard !sentences.isEmpty else { return [] }
186186

187187
let threshold = max(1, min(capacity, TtsConstants.shortSentenceMergeTokenThreshold))
@@ -203,7 +203,7 @@ enum KokoroChunker {
203203
let trimmed = sentence.trimmingCharacters(in: .whitespacesAndNewlines)
204204
guard !trimmed.isEmpty else { continue }
205205

206-
let sentenceTokens = try tokenCountForSegment(
206+
let sentenceTokens = try await tokenCountForSegment(
207207
for: trimmed,
208208
lexicon: lexicon,
209209
caseSensitiveLexicon: caseSensitiveLexicon,
@@ -231,7 +231,7 @@ enum KokoroChunker {
231231
}
232232

233233
let candidate = appendSegment(buffer, with: trimmed)
234-
let candidateTokens = try tokenCountForSegment(
234+
let candidateTokens = try await tokenCountForSegment(
235235
for: candidate,
236236
lexicon: lexicon,
237237
caseSensitiveLexicon: caseSensitiveLexicon,
@@ -270,7 +270,7 @@ enum KokoroChunker {
270270
wordIndex: inout Int,
271271
overrides: [TtsPhoneticOverride],
272272
overrideIndex: inout Int
273-
) throws -> [TextChunk] {
273+
) async throws -> [TextChunk] {
274274
let atoms = tokenizeAtoms(text)
275275
guard !atoms.isEmpty else { return [] }
276276

@@ -345,7 +345,7 @@ enum KokoroChunker {
345345

346346
if resolved == nil {
347347
guard
348-
let fallback = try resolvePhonemes(
348+
let fallback = try await resolvePhonemes(
349349
for: original,
350350
normalized: normalized,
351351
lexicon: lexicon,
@@ -482,7 +482,7 @@ enum KokoroChunker {
482482
caseSensitiveLexicon: [String: [String]],
483483
allowed: Set<String>,
484484
missing: inout Set<String>
485-
) throws -> [String]? {
485+
) async throws -> [String]? {
486486
var phonemes = caseSensitiveLexicon[original]
487487

488488
if phonemes == nil, let exactNormalized = caseSensitiveLexicon[normalized] {
@@ -493,7 +493,7 @@ enum KokoroChunker {
493493
phonemes = lexicon[normalized]
494494
}
495495

496-
if phonemes == nil, let ipa = try EspeakG2P.shared.phonemize(word: normalized) {
496+
if phonemes == nil, let ipa = try await EspeakG2P.shared.phonemize(word: normalized) {
497497
let mapped = PhonemeMapper.mapIPA(ipa, allowed: allowed)
498498
if !mapped.isEmpty {
499499
phonemes = mapped
@@ -510,7 +510,7 @@ enum KokoroChunker {
510510
for spelled in spelledTokens {
511511
var segment = lexicon[spelled]
512512

513-
if segment == nil, let ipa = try EspeakG2P.shared.phonemize(word: spelled) {
513+
if segment == nil, let ipa = try await EspeakG2P.shared.phonemize(word: spelled) {
514514
let mapped = PhonemeMapper.mapIPA(ipa, allowed: allowed)
515515
if !mapped.isEmpty {
516516
segment = mapped
@@ -574,7 +574,7 @@ enum KokoroChunker {
574574
caseSensitiveLexicon: [String: [String]],
575575
allowed: Set<String>,
576576
capacity: Int
577-
) throws -> Int {
577+
) async throws -> Int {
578578
let atoms = tokenizeAtoms(text)
579579
guard !atoms.isEmpty else { return 0 }
580580

@@ -589,7 +589,7 @@ enum KokoroChunker {
589589
let normalized = normalize(original)
590590
guard !normalized.isEmpty else { continue }
591591
guard
592-
let phonemes = try resolvePhonemes(
592+
let phonemes = try await resolvePhonemes(
593593
for: original,
594594
normalized: normalized,
595595
lexicon: lexicon,
@@ -625,7 +625,7 @@ enum KokoroChunker {
625625
caseSensitiveLexicon: [String: [String]],
626626
allowed: Set<String>,
627627
capacity: Int
628-
) throws -> [String] {
628+
) async throws -> [String] {
629629
guard !fragments.isEmpty else { return [] }
630630

631631
var assembled: [String] = []
@@ -647,7 +647,7 @@ enum KokoroChunker {
647647
current.isEmpty
648648
? trimmedFragment
649649
: appendSegment(current, with: trimmedFragment)
650-
let candidateTokens = try tokenCountForSegment(
650+
let candidateTokens = try await tokenCountForSegment(
651651
for: candidate,
652652
lexicon: lexicon,
653653
caseSensitiveLexicon: caseSensitiveLexicon,
@@ -660,7 +660,7 @@ enum KokoroChunker {
660660
} else {
661661
flushCurrent()
662662
current = trimmedFragment
663-
let fragmentTokens = try tokenCountForSegment(
663+
let fragmentTokens = try await tokenCountForSegment(
664664
for: current,
665665
lexicon: lexicon,
666666
caseSensitiveLexicon: caseSensitiveLexicon,

‎Sources/FluidAudio/TextToSpeech/Kokoro/Pipeline/Synthesize/KokoroSynthesizer.swift‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,7 @@ public struct KokoroSynthesizer {
6868
try await loadSimplePhonemeDictionary()
6969
let hasLang = false
7070
let lexicons = await lexiconCache.lexicons()
71-
return try KokoroChunker.chunk(
71+
return try await KokoroChunker.chunk(
7272
text: text,
7373
wordToPhonemes: lexicons.word,
7474
caseSensitiveLexicon: lexicons.caseSensitive,

0 commit comments

Comments
 (0)