Skip to content

Commit e4d8750

Browse files
alvationsclaude
andcommitted
Release 0.3.0: emoji support
New charguana.emoji module bundles parsed Unicode emoji data (latest): - emoji_chars: 1,219-item frozenset of Emoji_Presentation codepoints - emoji_sequences: 2,339 keycap/flag/modifier sequences - emoji_zwj_sequences: 1,614 ZWJ compositions (family, skintone, etc.) - emoji_properties: char -> frozenset of UCD emoji properties - is_emoji(s): True for a known single emoji or multi-char sequence - fetch_emoji(version='latest'): runtime fetch from unicode.org as an escape hatch (stdlib urllib, no requests dep) All bundled data is lazy-loaded via PEP 562 __getattr__; importing charguana.emoji is free. get_charset('emoji') now works. Maintainer tooling: - scripts/refresh_emoji.py regenerates data/emoji/*.txt from unicode.org - scripts/generate_perluniprops.sh moved from repo root for consistency Package sizes: - 1,219 emoji chars, 2,339 sequences, 1,614 ZWJ, 3,679 properties rows - Wheel grows 197 KB -> 222 KB (+13%) Tests: 32 passing on 3.10-3.14 (up from 27). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent 55ea170 commit e4d8750

11 files changed

Lines changed: 8563 additions & 3 deletions

README.md

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,22 @@ Install
1313
pip install charguana
1414
```
1515

16+
### What's new in 0.3.0
17+
18+
- **Emoji support.** New `charguana.emoji` module bundles parsed Unicode
19+
emoji data (latest version).
20+
- `get_charset('emoji')` returns the list of single-codepoint
21+
Emoji_Presentation characters.
22+
- `is_emoji(s)` — True for a single emoji char or a known multi-char
23+
sequence (flags, ZWJ combos, keycaps, modifier sequences).
24+
- `charguana.emoji.emoji_sequences`, `emoji_zwj_sequences`,
25+
`emoji_properties` for finer access. All lazy-loaded.
26+
- `fetch_emoji(version='latest')` hits unicode.org for fresh data at
27+
runtime (requires network; no side effects on disk).
28+
- Maintainer tooling: `scripts/refresh_emoji.py` regenerates the bundled
29+
emoji data files from unicode.org. Alongside the existing
30+
`scripts/generate_perluniprops.sh`.
31+
1632
### What's new in 0.2.0
1733

1834
- `get_charset(name)` now returns a **list** instead of a generator. Use
@@ -133,6 +149,36 @@ True
133149
'0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmn'
134150
```
135151

152+
**Emoji**
153+
154+
```python
155+
>>> from charguana import get_charset, is_emoji
156+
>>> get_charset('emoji')[:5]
157+
['', '', '', '', '']
158+
>>> is_emoji('😀')
159+
True
160+
>>> is_emoji('🇺🇸') # regional-indicator sequence (US flag)
161+
True
162+
>>> is_emoji('👨\u200d👩\u200d👧\u200d👦') # ZWJ: family
163+
True
164+
>>> is_emoji('A')
165+
False
166+
167+
# Finer access to sequences and properties (lazy-loaded):
168+
>>> from charguana.emoji import emoji_sequences, emoji_zwj_sequences, emoji_properties
169+
>>> '🇺🇸' in emoji_sequences
170+
True
171+
>>> emoji_properties['😀']
172+
frozenset({'Emoji', 'Emoji_Presentation'})
173+
174+
# Runtime fetch from unicode.org (needs network):
175+
>>> from charguana import fetch_emoji
176+
>>> parsed = fetch_emoji() # or fetch_emoji('16.0')
177+
>>> sorted(parsed.keys())
178+
['data_properties', 'sequences', 'variation_sequences', 'zwj_sequences']
179+
```
180+
181+
136182
**Thai**
137183

138184
```python

charguana/__init__.py

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
from charguana.perluniprops import *
77
from charguana.thai import *
88
from charguana.viet import *
9+
from charguana.emoji import is_emoji, fetch_emoji
910

1011
cjk_charsets = {'chinese': han_utf8, 'zh': han_utf8, 'cn': han_utf8,
1112
'japanese': jap_utf8, 'ja': jap_utf8, 'jp': jap_utf8,
@@ -104,13 +105,18 @@ def get_charset_ranges(charset_ranges):
104105
# (either an explicit list, a long string, or an expanded range). The Thai
105106
# block is stored as (start, end) tuples in ``thai_utf8`` for reference; we
106107
# expand it once here so the dict is shape-consistent with the others.
108+
#
109+
# ``'emoji'`` is lazy-resolved: the underlying frozenset is loaded on first
110+
# access via ``charguana.emoji`` (see ``iter_charset`` below).
107111
other_charsets = {
108112
'thai': list(get_charset_ranges(thai_utf8)),
109113
'viet': viet_utf8,
110114
'traditional_chinese': big5,
111115
'simplified_chinese': gbk,
112116
}
113117

118+
_lazy_other_charsets = {'emoji'}
119+
114120

115121
def iter_charset(charset_name):
116122
"""Return a lazy iterator over the characters in the named charset.
@@ -120,7 +126,7 @@ def iter_charset(charset_name):
120126
121127
Valid names: any key of ``cjk_charsets``, ``perluniprops_charsets``, or
122128
``other_charsets`` — e.g. 'chinese', 'japanese', 'korean', 'thai', 'viet',
123-
'traditional_chinese', 'simplified_chinese', 'IsAlpha', 'IsN',
129+
'traditional_chinese', 'simplified_chinese', 'emoji', 'IsAlpha', 'IsN',
124130
'Open_Punctuation', 'Currency_Symbol', etc. Unknown names yield nothing.
125131
"""
126132
if charset_name in cjk_charsets:
@@ -129,6 +135,9 @@ def iter_charset(charset_name):
129135
return iter(perluniprops_charsets[charset_name])
130136
if charset_name in other_charsets:
131137
return iter(other_charsets[charset_name])
138+
if charset_name in _lazy_other_charsets:
139+
from charguana import emoji as _emoji
140+
return iter(sorted(getattr(_emoji, f'{charset_name}_chars')))
132141
return iter(())
133142

134143

@@ -179,6 +188,8 @@ def all_in_charset(string, charset):
179188
'thai_utf8', 'thai_block',
180189
# Vietnamese
181190
'viet_utf8', 'viet_tones', 'viet_consonants', 'viet_ime',
191+
# Emoji
192+
'is_emoji', 'fetch_emoji',
182193
# perluniprops
183194
'open_punctuation', 'close_punctuation', 'currency_symbol',
184195
'is_sc', 'is_alnum', 'is_alpha', 'is_lower', 'is_upper', 'is_n', 'is_so',

0 commit comments

Comments
 (0)