Skip to content

Commit 167cece

Browse files
author
Henry Barnes
committed
Fix HuggingFace upload for N3 and deploy
- Fix generate_model_card: handle None quant_acc for N3 entries, fix dataset tag lookup for _n3 suffixed keys, add N3 paper DOI - Update leaderboard README: replace "coming soon" with N3 results - Deployed: leaderboard Space, SHD/SSC/NMNIST N3 model cards
1 parent 5c41dad commit 167cece

2 files changed

Lines changed: 51 additions & 36 deletions

File tree

huggingface/leaderboard/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ Community benchmark comparison for spiking neural networks across neuromorphic h
2222

2323
Covers SHD, SSC, N-MNIST, GSC KWS, and DVS128 Gesture benchmarks. Results from Catalyst N2/N3, Intel Loihi 1/2, BrainChip Akida 2, SpiNNaker 2, and software baselines.
2424

25-
**Catalyst N3 benchmark results** (with precision sweeps, FACTOR compression, and N3 hardware feature demonstrations) coming soon.
25+
**Catalyst N3 results**: SHD 91.0% (#1 SOTA), SSC 76.4% (#1 SOTA), N-MNIST 99.2%.
2626

2727
**Submit your results**: Open a discussion or PR on this Space to add your platform's numbers.
2828

huggingface/upload.py

Lines changed: 50 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -126,16 +126,56 @@ def generate_model_card(benchmark):
126126
float_acc = info['float_acc']
127127
quant_acc = info['quant_acc']
128128
params = info['params']
129+
generation = info.get('generation', 'N2')
129130
neuron_model = info.get('neuron_model', 'Adaptive Leaky Integrate-and-Fire (adLIF)')
130131

132+
# Strip generation suffix for dataset/display names
133+
base_benchmark = benchmark.replace('_n3', '').replace('_n2', '')
134+
131135
# Dataset tag mapping
132136
dataset_tags = {
133137
'shd': 'shd',
134138
'ssc': 'ssc',
135139
'nmnist': 'n-mnist',
136140
'gsc': 'google-speech-commands',
137141
}
138-
dataset_tag = dataset_tags.get(benchmark, benchmark.replace('_', '-'))
142+
dataset_tag = dataset_tags.get(base_benchmark, base_benchmark.replace('_', '-'))
143+
144+
# Build metrics YAML (skip quant if None)
145+
metrics_yaml = f""" - name: Float Accuracy ({generation})
146+
type: accuracy
147+
value: {float_acc}"""
148+
if quant_acc is not None:
149+
metrics_yaml += f"""
150+
- name: Quantized Accuracy (int16)
151+
type: accuracy
152+
value: {quant_acc}"""
153+
154+
# Build results table (skip quant row if None)
155+
results_rows = f"| Float accuracy | {float_acc}% |\n| Parameters | {params} |"
156+
if quant_acc is not None:
157+
results_rows = (
158+
f"| Float accuracy | {float_acc}% |\n"
159+
f"| Quantized accuracy (int16) | {quant_acc}% |\n"
160+
f"| Parameters | {params} |\n"
161+
f"| Quantization loss | {float_acc - quant_acc:.1f}% |"
162+
)
163+
164+
# Reproduce command
165+
train_dir = base_benchmark
166+
train_cmd = f"python {train_dir}/train.py --device cuda:0"
167+
if generation == 'N3' and base_benchmark == 'shd':
168+
train_cmd = f"python {train_dir}/train.py --neuron adlif --hidden 1536 --epochs 200 --device cuda:0 --amp"
169+
170+
# Paper links
171+
paper_links = """- **Benchmark repo**: [catalyst-neuromorphic/catalyst-benchmarks](https://github.com/catalyst-neuromorphic/catalyst-benchmarks)
172+
- **Cloud API**: [catalyst-neuromorphic.com](https://catalyst-neuromorphic.com)"""
173+
if generation == 'N3':
174+
paper_links += """
175+
- **N3 paper**: [Zenodo DOI 10.5281/zenodo.18881283](https://zenodo.org/records/18881283)"""
176+
paper_links += """
177+
- **N2 paper**: [Zenodo DOI 10.5281/zenodo.18728256](https://zenodo.org/records/18728256)
178+
- **N1 paper**: [Zenodo DOI 10.5281/zenodo.18727094](https://zenodo.org/records/18727094)"""
139179

140180
return f"""---
141181
language: en
@@ -147,13 +187,13 @@ def generate_model_card(benchmark):
147187
- surrogate-gradient
148188
- benchmark
149189
- catalyst
150-
- {benchmark.replace('_', '-')}
190+
- {base_benchmark}
151191
datasets:
152192
- {dataset_tag}
153193
metrics:
154194
- accuracy
155195
model-index:
156-
- name: Catalyst {benchmark.upper()} SNN Benchmark
196+
- name: Catalyst {base_benchmark.upper()} SNN Benchmark ({generation})
157197
results:
158198
- task:
159199
type: {info['task']}
@@ -162,63 +202,38 @@ def generate_model_card(benchmark):
162202
name: {info['dataset']}
163203
type: {dataset_tag}
164204
metrics:
165-
- name: Float Accuracy
166-
type: accuracy
167-
value: {float_acc}
168-
- name: Quantized Accuracy (int16)
169-
type: accuracy
170-
value: {quant_acc}
205+
{metrics_yaml}
171206
---
172207
173-
# Catalyst {benchmark.upper()} SNN Benchmark
208+
# Catalyst {base_benchmark.upper()} SNN Benchmark ({generation})
174209
175210
{info['description']}
176211
177212
## Model Description
178213
179-
- **Architecture**: {info['architecture']}
214+
- **Architecture ({generation})**: {info['architecture']}
180215
- **Neuron model**: {neuron_model}
181216
- **Training**: Surrogate gradient BPTT, fast-sigmoid surrogate (scale=25)
182-
- **Hardware target**: Catalyst N1/N2/N3 neuromorphic processors
183-
- **Quantization**: Float32 weights -> int16, membrane decay -> 12-bit fixed-point
217+
- **Hardware target**: Catalyst {generation} neuromorphic processor
184218
185219
## Results
186220
187221
| Metric | Value |
188222
|--------|-------|
189-
| Float accuracy | {float_acc}% |
190-
| Quantized accuracy (int16) | {quant_acc}% |
191-
| Parameters | {params} |
192-
| Quantization loss | {float_acc - quant_acc:.1f}% |
223+
{results_rows}
193224
194225
## Reproduce
195226
196227
```bash
197228
git clone https://github.com/catalyst-neuromorphic/catalyst-benchmarks.git
198229
cd catalyst-benchmarks
199230
pip install -e .
200-
python {benchmark}/train.py --device cuda:0
201-
```
202-
203-
## Deploy to Catalyst Hardware
204-
205-
```python
206-
import catalyst_cloud
207-
208-
client = catalyst_cloud.Client()
209-
result = client.simulate(
210-
model="catalyst-neuromorphic/{benchmark}-snn-benchmark",
211-
input_data=your_spikes,
212-
processor="n2"
213-
)
231+
{train_cmd}
214232
```
215233
216234
## Links
217235
218-
- **Benchmark repo**: [catalyst-neuromorphic/catalyst-benchmarks](https://github.com/catalyst-neuromorphic/catalyst-benchmarks)
219-
- **Cloud API**: [catalyst-neuromorphic.com](https://catalyst-neuromorphic.com)
220-
- **N2 paper**: [Zenodo DOI 10.5281/zenodo.18728256](https://zenodo.org/records/18728256)
221-
- **N1 paper**: [Zenodo DOI 10.5281/zenodo.18727094](https://zenodo.org/records/18727094)
236+
{paper_links}
222237
223238
## Citation
224239

0 commit comments

Comments
 (0)