Repository navigation
Commit 1138b85
authored
model-conversion : use save_output_data for causual embeddings [no ci] (ggml-org#26890)
This commit updates the python script that runs the original model to
generate embeddings for the causal model, to use save_output_data which
stores the token ids and the prompt in addition to logits.
The motivation for this is that the embedding logits verification will
fail as it expects these files (-prompt.txt and -tokens.bin) to exist.
With the changes in this commit the causal-verify-embeddings target
works again.1 parent 9afff1b commit 1138b85
1 file changed
Lines changed: 6 additions & 17 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| 5 | + | |
5 | 6 | | |
6 | 7 | | |
7 | 8 | | |
8 | 9 | | |
9 | 10 | | |
10 | | - | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
11 | 14 | | |
12 | 15 | | |
13 | 16 | | |
| |||
54 | 57 | | |
55 | 58 | | |
56 | 59 | | |
| 60 | + | |
57 | 61 | | |
58 | 62 | | |
59 | 63 | | |
| |||
74 | 78 | | |
75 | 79 | | |
76 | 80 | | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | | - | |
81 | | - | |
82 | | - | |
83 | | - | |
84 | 81 | | |
85 | | - | |
86 | | - | |
87 | | - | |
88 | | - | |
89 | | - | |
90 | | - | |
91 | | - | |
| 82 | + | |
92 | 83 | | |
93 | 84 | | |
94 | 85 | | |
| |||
110 | 101 | | |
111 | 102 | | |
112 | 103 | | |
113 | | - | |
114 | | - | |
0 commit comments