Skip to content

Commit 6a58b25

Browse files
nnnkkk7nnnkkk7
authored
docs: update readme (#26)
1 parent 225e49c commit 6a58b25

1 file changed

Lines changed: 91 additions & 127 deletions

File tree

‎README.md‎

Lines changed: 91 additions & 127 deletions
Original file line numberDiff line numberDiff line change
@@ -1,33 +1,35 @@
11
# go-simdcsv
22

3-
> **⚠️ Experimental Project**: This is a showcase project for Go 1.26's experimental `simd/archsimd` package. The SIMD API is unstable and may change in future Go releases. Not recommended for production use.
3+
[![CI](https://github.com/nnnkkk7/go-simdcsv/actions/workflows/ci.yml/badge.svg)](https://github.com/nnnkkk7/go-simdcsv/actions/workflows/ci.yml)
4+
[![Go 1.26+](https://img.shields.io/badge/Go-1.26%2B-00ADD8?logo=go&logoColor=white)](https://go.dev/)
5+
[![Go Report Card](https://goreportcard.com/badge/github.com/nnnkkk7/go-simdcsv)](https://goreportcard.com/report/github.com/nnnkkk7/go-simdcsv)
6+
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
47

5-
A high-performance CSV parser for Go using SIMD (Single Instruction Multiple Data) instructions. Drop-in replacement for the standard library's `encoding/csv` package with performance improvements on AMD64 processors with AVX-512 support.
68

7-
## Features
9+
SIMD-accelerated CSV parser for Go - Drop-in replacement for `encoding/csv` with AVX-512 optimization.
810

9-
- **SIMD-accelerated parsing**: Uses Go 1.26's experimental `simd/archsimd` package with 256-bit vectors (requires AVX-512BW for `ToBits()` operation)
10-
- **API compatible**: Drop-in replacement for `encoding/csv.Reader` and `encoding/csv.Writer`
11-
- **RFC 4180 compliant**: Full support for quoted fields, escaped quotes, multiline fields, and CRLF normalization
12-
- **Automatic fallback**: Gracefully falls back to scalar implementation on CPUs without AVX-512
13-
- **Zero-copy API**: `ParseBytes` function for direct byte slice parsing without io.Reader overhead
11+
> **Experimental**: Requires Go 1.26+ with `GOEXPERIMENT=simd`. The SIMD API is unstable and may change　now. Not recommended for production use.
1412
15-
## Requirements
13+
## Quick Start
14+
15+
```go
16+
import csv "github.com/nnnkkk7/go-simdcsv"
1617

17-
- **Go 1.26+** with `GOEXPERIMENT=simd` build tag (experimental)
18-
- **AMD64 architecture** (x86-64) only
19-
- **AVX-512 support** for SIMD acceleration (AVX512F, AVX512BW, AVX512VL required)
18+
reader := csv.NewReader(strings.NewReader("name,age\nAlice,30\nBob,25"))
19+
records, _ := reader.ReadAll()
20+
// [[name age] [Alice 30] [Bob 25]]
21+
```
2022

21-
### Limitations
23+
Same API as `encoding/csv` - just change the import.
2224

23-
- **Experimental SIMD API**: The `simd/archsimd` package is experimental and AMD64-specific. A portable high-level SIMD package is planned for future Go releases.
24-
- **AVX-512 dependency**: Despite using 256-bit vectors (`Int8x32`), the `ToBits()` method requires AVX-512BW instruction (VPMOVB2M). This means SIMD acceleration is **not available** on:
25-
- Most CI environments (GitHub Actions `ubuntu-latest`, etc.)
26-
- CPUs without AVX-512 (Intel before Skylake-X, most AMD before Zen 4)
27-
- Apple Silicon (ARM64)
28-
- **Memory**: Currently reads entire input into memory (streaming I/O planned)
25+
## Features
2926

30-
> **Note**: On unsupported CPUs, the library automatically falls back to a scalar implementation with no SIMD acceleration.
27+
| Feature | Description |
28+
|---------|-------------|
29+
| **API Compatible** | Drop-in replacement for `encoding/csv.Reader` and `Writer` |
30+
| **RFC 4180** | Quoted fields, escaped quotes (`""`), multiline fields, CRLF normalization |
31+
| **Auto Fallback** | Gracefully falls back to scalar on non-AVX-512 CPUs |
32+
| **Direct Byte API** | `ParseBytes()` and `ParseBytesStreaming()` for `[]byte` input |
3133

3234
## Installation
3335

@@ -37,133 +39,75 @@ go get github.com/nnnkkk7/go-simdcsv
3739

3840
## Usage
3941

40-
### Basic Reading (Drop-in Replacement)
42+
### Basic Reading
4143

4244
```go
43-
package main
44-
45-
import (
46-
"fmt"
47-
"strings"
48-
49-
csv "github.com/nnnkkk7/go-simdcsv"
50-
)
51-
52-
func main() {
53-
data := "name,age,city\nAlice,30,Tokyo\nBob,25,Osaka\n"
54-
reader := csv.NewReader(strings.NewReader(data))
55-
56-
records, err := reader.ReadAll()
57-
if err != nil {
58-
panic(err)
59-
}
60-
61-
for _, record := range records {
62-
fmt.Println(record)
63-
}
64-
}
45+
reader := csv.NewReader(strings.NewReader(data))
46+
records, err := reader.ReadAll()
6547
```
6648

67-
### Zero-Copy Parsing
68-
69-
For maximum performance when you already have data in a byte slice:
49+
### Record-by-Record
7050

7151
```go
72-
package main
73-
74-
import (
75-
"fmt"
76-
77-
csv "github.com/nnnkkk7/go-simdcsv"
78-
)
79-
80-
func main() {
81-
data := []byte("a,b,c\n1,2,3\n4,5,6\n")
82-
83-
records, err := csv.ParseBytes(data, ',')
84-
if err != nil {
85-
panic(err)
86-
}
87-
88-
for _, record := range records {
89-
fmt.Println(record)
52+
reader := csv.NewReader(r)
53+
for {
54+
record, err := reader.Read()
55+
if err == io.EOF {
56+
break
9057
}
58+
// process record
9159
}
9260
```
9361

94-
### Streaming API
62+
### Direct Byte Parsing
9563

96-
Process records one at a time with a callback:
64+
For maximum performance with `[]byte` input:
9765

9866
```go
99-
package main
100-
101-
import (
102-
"fmt"
103-
104-
csv "github.com/nnnkkk7/go-simdcsv"
105-
)
106-
107-
func main() {
108-
data := []byte("name,value\nfoo,100\nbar,200\n")
109-
110-
err := csv.ParseBytesStreaming(data, ',', func(record []string) error {
111-
fmt.Printf("Record: %v\n", record)
112-
return nil
113-
})
114-
if err != nil {
115-
panic(err)
116-
}
117-
}
67+
records, err := csv.ParseBytes(data, ',')
11868
```
11969

120-
### Writing CSV
70+
### Streaming with Callback
12171

12272
```go
123-
package main
124-
125-
import (
126-
"os"
127-
128-
csv "github.com/nnnkkk7/go-simdcsv"
129-
)
130-
131-
func main() {
132-
writer := csv.NewWriter(os.Stdout)
73+
csv.ParseBytesStreaming(data, ',', func(record []string) error {
74+
fmt.Println(record)
75+
return nil
76+
})
77+
```
13378

134-
records := [][]string{
135-
{"name", "age", "city"},
136-
{"Alice", "30", "Tokyo"},
137-
{"Bob", "25", "Osaka"},
138-
}
79+
### Writing
13980

140-
writer.WriteAll(records)
141-
}
81+
```go
82+
writer := csv.NewWriter(os.Stdout)
83+
writer.WriteAll([][]string{
84+
{"name", "age"},
85+
{"Alice", "30"},
86+
})
14287
```
14388

144-
### Configuration Options
89+
### Configuration
14590

146-
The Reader supports all standard `encoding/csv` options:
91+
All standard `encoding/csv` options are supported:
14792

14893
```go
14994
reader := csv.NewReader(r)
150-
reader.Comma = ';' // Custom field delimiter
95+
reader.Comma = ';' // Field delimiter (default: ',')
15196
reader.Comment = '#' // Comment character
152-
reader.FieldsPerRecord = 3 // Expected fields per record (0 = auto-detect)
153-
reader.LazyQuotes = true // Allow bare quotes in unquoted fields
97+
reader.LazyQuotes = true // Allow bare quotes
15498
reader.TrimLeadingSpace = true // Trim leading whitespace
155-
reader.ReuseRecord = true // Reuse record slice for performance
99+
reader.ReuseRecord = true // Reuse slice for performance
100+
reader.FieldsPerRecord = 3 // Expected fields (0 = auto-detect, -1 = variable)
156101
```
157102

158-
Extended options with `NewReaderWithOptions`:
103+
Extended options:
159104

160105
```go
161106
reader := csv.NewReaderWithOptions(r, csv.ReaderOptions{
162107
SkipBOM: true, // Skip UTF-8 BOM if present
163108
})
164109
```
165110

166-
167111
## Architecture
168112

169113
```
@@ -173,32 +117,58 @@ reader := csv.NewReaderWithOptions(r, csv.ReaderOptions{
173117
└─────────────┘ └──────────────┘ └──────────────┘ └──────────────┘
174118
```
175119

176-
1. **`scanBuffer()`**: Scans input in 64-byte chunks using 256-bit SIMD vectors (`archsimd.Int8x32`). Detects positions of structural characters (`"`, `,`, `\n`, `\r`) and outputs bitmasks. Handles CRLF normalization and tracks quote state across chunk boundaries.
120+
| Stage | Function | Description |
121+
|-------|----------|-------------|
122+
| **Scan** | `scanBuffer()` | SIMD scanning in 64-byte chunks. Detects `"`, `,`, `\n`, `\r` positions as bitmasks. Handles CRLF and quote state across boundaries. |
123+
| **Parse** | `parseBuffer()` | Iterates bitmasks to find field boundaries. Outputs `fieldInfo` (offset, length) and `rowInfo` (field count). |
124+
| **Build** | `buildRecords()` | Extracts strings from positions. Applies `""` → `"` unescaping. |
125+
126+
## Requirements
177127

178-
2. **`parseBuffer()`**: Iterates through bitmasks to determine field boundaries. Outputs `fieldInfo` (start offset, length) and `rowInfo` (field count per row). Correctly handles quoted fields containing commas or newlines.
128+
| Requirement | Details |
129+
|-------------|---------|
130+
| **Go** | 1.26+ with `GOEXPERIMENT=simd` |
131+
| **Architecture** | AMD64 (x86-64) only |
132+
| **SIMD Acceleration** | AVX-512 (F, BW, VL) required |
179133

180-
3. **`buildRecords()`**: Extracts strings from byte positions. Applies double-quote unescaping (`""` → `"`) for fields that were marked during scanning.
134+
### Without AVX-512
181135

182-
## Building and Testing
136+
The library **still works** but falls back to scalar implementation (no speedup). This includes:
137+
- Most CI environments (GitHub Actions `ubuntu-latest`, etc.)
138+
- Intel CPUs before Skylake-X
139+
- AMD CPUs before Zen 4
140+
- Apple Silicon (ARM64)
183141

184-
### AMD64 Environment
142+
## Building & Testing
185143

186144
```bash
187-
# Build with SIMD support
145+
# Build
188146
GOEXPERIMENT=simd go build ./...
189147

190-
# Run tests
148+
# Test
191149
GOEXPERIMENT=simd go test -v ./...
192150

193-
# Run benchmarks
151+
# Benchmark
194152
GOEXPERIMENT=simd go test -bench=. -benchmem
195153
```
196154

197155
## Performance
198156

199-
Benchmarks comparing `go-simdcsv` against `encoding/csv` (run on AMD64 with AVX-512):
157+
Benchmarks comparing `go-simdcsv` against `encoding/csv`:
200158

201-
TODO: Add benchmark results here.
159+
| Benchmark | encoding/csv | go-simdcsv | Speedup |
160+
|-----------|--------------|------------|---------|
161+
| Small (1KB) | - | - | - |
162+
| Medium (100KB) | - | - | - |
163+
| Large (10MB) | - | - | - |
164+
165+
> TODO: Add benchmark results from AVX-512 environment.
166+
167+
## Known Limitations
168+
169+
- **Experimental API**: `simd/archsimd` may have breaking changes in future Go releases
170+
- **Memory**: Reads entire input into memory (streaming I/O planned for future)
171+
- **Custom delimiters**: Some edge cases with non-comma delimiters may differ from `encoding/csv`
202172

203173
## Contributing
204174

@@ -207,9 +177,3 @@ Contributions are welcome! Please open issues or pull requests on GitHub.
207177
## License
208178

209179
MIT License - see [LICENSE](LICENSE) file for details.
210-
211-
## Known Issues
212-
213-
- **CI environments**: Most CI runners (GitHub Actions, etc.) do not have AVX-512 support. Tests pass using the scalar fallback, but SIMD acceleration is not tested in CI.
214-
- **Apple Silicon**: Not supported. This library is AMD64-specific.
215-
- **Go SIMD API stability**: The `simd/archsimd` package is experimental. Future Go releases may introduce breaking changes.

0 commit comments

Comments
 (0)