Skip to content

Commit 8e43097

Browse files
authored
Improve device and bot detection for 3.1.0
- Add is_tv for smart TVs and streaming devices, which were reported as touch capable tablets, phones or PCs - Catch bots ua-parser misses, using product tokens, contact URLs, contact emails and headless browsers - Treat Generic Tablet as a tablet and Dalvik clients as phones, and add KaiOS to the mobile OS families - Accept None and bytes in parse() instead of raising TypeError - Move to the current ua-parser API, requiring ua-parser>=1.0.0, and add a fast extra for the Rust-backed parser - Run CI against both parsers and update the publish workflow actions
1 parent c553408 commit 8e43097

7 files changed

Lines changed: 163 additions & 22 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,3 +29,8 @@ jobs:
2929
3030
- name: Run tests
3131
run: python -m unittest -v user_agents.tests
32+
33+
- name: Run tests with the fast parser
34+
run: |
35+
python -m pip install -e ".[fast]"
36+
python -m unittest -v user_agents.tests

‎.github/workflows/publish.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,8 @@ jobs:
1212
name: Build distributions
1313
runs-on: ubuntu-latest
1414
steps:
15-
- uses: actions/checkout@v5
16-
- uses: actions/setup-python@v6
15+
- uses: actions/checkout@v7
16+
- uses: actions/setup-python@v7
1717
with:
1818
python-version: "3.12"
1919
- name: Build sdist and wheel

‎README.md‎

Lines changed: 22 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,11 @@ to do the actual parsing of the raw user agent string.
2525

2626
pip install user-agents-ng
2727

28+
For a roughly 20x parsing speedup on uncached user agents, install the optional
29+
Rust-backed parser too. It's a drop-in change, no code needs updating:
30+
31+
pip install "user-agents-ng[fast]"
32+
2833
### Migrating from `user-agents`
2934

3035
`user-agents-ng` is a drop-in replacement. Swap the dependency and your code
@@ -81,8 +86,13 @@ Currently these attributes are supported:
8186
* `is_mobile`: whether user agent is identified as a mobile phone (iPhone, Android phones, Blackberry, Windows Phone devices etc)
8287
* `is_tablet`: whether user agent is identified as a tablet device (iPad, Kindle Fire, Nexus 7 etc)
8388
* `is_pc`: whether user agent is identified to be running a traditional "desktop" OS (Windows, OS X, Linux)
89+
* `is_tv`: whether user agent is identified as a smart TV or streaming device (Samsung Tizen, LG webOS, Android TV, Fire TV, Roku, Chromecast etc)
8490
* `is_touch_capable`: whether user agent has touch capabilities
85-
* `is_bot`: whether user agent is a search engine crawler/spider
91+
* `is_bot`: whether user agent is a crawler, spider or headless browser. Besides the bots ua-parser knows about, this catches any user agent with a bot-style product token (`Applebot/0.1`), a contact URL (`+http://example.com/bot`) or a contact email address
92+
* `is_email_client`: whether user agent is an email client (Outlook, Thunderbird, Apple Mail etc)
93+
94+
`parse()` also accepts `bytes` and `None`, so a raw header or a request with no
95+
`User-Agent` header at all won't raise an exception.
8696

8797
For example:
8898

@@ -97,7 +107,7 @@ user_agent.is_tablet # returns False
97107
user_agent.is_touch_capable # returns False
98108
user_agent.is_pc # returns False
99109
user_agent.is_bot # returns False
100-
str(user_agent) # returns "BlackBerry 9700 / BlackBerry OS 5 / BlackBerry 9700"
110+
str(user_agent) # returns "BlackBerry 9700 / BlackBerry OS 5.0.0 / BlackBerry 9700"
101111

102112
# Now a Samsung Galaxy S3
103113
ua_string = 'Mozilla/5.0 (Linux; U; Android 4.0.4; en-gb; GT-I9300 Build/IMM76D) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30'
@@ -117,7 +127,7 @@ user_agent.is_tablet # returns False
117127
user_agent.is_touch_capable # returns True
118128
user_agent.is_pc # returns True
119129
user_agent.is_bot # returns False
120-
str(user_agent) # returns "PC / Windows 8 / IE 10"
130+
str(user_agent) # returns "PC / Windows 8 / IE 10.0"
121131
```
122132

123133
## Running Tests
@@ -126,6 +136,15 @@ str(user_agent) # returns "PC / Windows 8 / IE 10"
126136

127137
## Changelog
128138

139+
### Version 3.1.0
140+
141+
* Added `is_tv` for smart TVs and streaming devices, which no longer count as touch capable tablets, phones or PCs.
142+
* `is_bot` now catches bots missed by ua-parser, like Applebot, Amazonbot, Bytespider and headless Chrome.
143+
* Added the `user-agents-ng[fast]` extra, which parses about 20x faster.
144+
* `parse()` accepts `None` and `bytes` instead of raising `TypeError`.
145+
* Firefox on Android tablets is now a tablet, Dalvik clients are no longer tablets, and KaiOS is mobile.
146+
* Now uses the current `ua-parser` API and requires `ua-parser>=1.0.0`, which keeps detection data up to date.
147+
129148
### Version 3.0.0
130149

131150
* Released as `user-agents-ng`, a maintained successor to `user-agents`.

‎pyproject.toml‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
44

55
[project]
66
name = "user-agents-ng"
7-
version = "3.0.0"
7+
version = "3.1.0"
88
description = "A maintained successor to user-agents: identify devices (phones, tablets) and their capabilities by parsing browser user agent strings."
99
readme = "README.md"
1010
requires-python = ">=3.9"
@@ -18,7 +18,7 @@ maintainers = [
1818
]
1919
keywords = ["user agent", "user-agents", "browser", "device", "detection", "parser", "mobile", "tablet", "bot"]
2020
dependencies = [
21-
"ua-parser>=0.10.0",
21+
"ua-parser>=1.0.0",
2222
]
2323
classifiers = [
2424
"Development Status :: 5 - Production/Stable",
@@ -38,6 +38,11 @@ classifiers = [
3838
"Topic :: Software Development :: Libraries :: Python Modules",
3939
]
4040

41+
[project.optional-dependencies]
42+
fast = [
43+
"ua-parser[regex]>=1.0.0",
44+
]
45+
4146
[project.urls]
4247
Homepage = "https://github.com/SAKMZ/python-user-agents"
4348
Repository = "https://github.com/SAKMZ/python-user-agents"

‎user_agents/__init__.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
VERSION = (3, 0, 0)
1+
VERSION = (3, 1, 0)
22

33
from .parsers import UserAgent, parse
44

‎user_agents/parsers.py‎

Lines changed: 53 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
1-
from typing import Any, NamedTuple, Optional
1+
import re
2+
from typing import Any, NamedTuple, Optional, Union
23

3-
from ua_parser import user_agent_parser
4+
import ua_parser
45

56

67
MOBILE_DEVICE_FAMILIES = (
@@ -26,6 +27,7 @@
2627
'Windows CE',
2728
'Windows Mobile',
2829
'Maemo',
30+
'KaiOS',
2931
)
3032

3133
MOBILE_BROWSER_FAMILIES = (
@@ -47,6 +49,7 @@
4749
'Galaxy Tab',
4850
'Xoom',
4951
'Dell Streak',
52+
'Generic Tablet',
5053
)
5154

5255
TABLET_DEVICE_BRANDS = (
@@ -89,6 +92,23 @@
8992
'YahooMobileMail',
9093
}
9194

95+
# Catches bots ua-parser doesn't know about yet
96+
BOT_UA_PATTERN = re.compile(
97+
r'\+https?://' # Contact URL, e.g. (+http://example.com/bot.html)
98+
r'|@[\w-]+\.[a-z]{2,}' # Contact email address
99+
r'|(?:bot|crawler|spider)/\d' # Product token, e.g. Applebot/0.1
100+
r'|compatible; [\w.-]*(?:bot|crawler|spider)\b' # e.g. compatible; Bytespider
101+
r'|HeadlessChrome/|PhantomJS/',
102+
re.IGNORECASE
103+
)
104+
105+
TV_UA_PATTERN = re.compile(
106+
r'smart[- ]?tv|hbbtv|google ?tv|android ?tv|apple ?tv|web0s|netcast|vidaa'
107+
r'|\broku\b|bravia|crkey'
108+
r'|(?-i:\bAFT[A-Z0-9]{1,6}\b)', # Amazon Fire TV models, e.g. AFTKA
109+
re.IGNORECASE
110+
)
111+
92112

93113
def verify_attribute(attribute: Any) -> Any:
94114
if isinstance(attribute, str) and attribute.isdigit():
@@ -163,12 +183,19 @@ def parse_device(family: str, brand: Optional[str], model: Optional[str]) -> Dev
163183

164184
class UserAgent:
165185

166-
def __init__(self, user_agent_string: str):
167-
ua_dict = user_agent_parser.Parse(user_agent_string)
186+
def __init__(self, user_agent_string: Union[str, bytes, None]):
187+
# Frameworks return None for a missing header, raw headers are bytes
188+
if user_agent_string is None:
189+
user_agent_string = ''
190+
elif isinstance(user_agent_string, bytes):
191+
user_agent_string = user_agent_string.decode('latin-1')
192+
193+
result = ua_parser.parse(user_agent_string).with_defaults()
194+
browser, os = result.user_agent, result.os
168195
self.ua_string = user_agent_string
169-
self.os = parse_operating_system(**ua_dict['os'])
170-
self.browser = parse_browser(**ua_dict['user_agent'])
171-
self.device = parse_device(**ua_dict['device'])
196+
self.os = parse_operating_system(os.family, os.major, os.minor, os.patch, os.patch_minor)
197+
self.browser = parse_browser(browser.family, browser.major, browser.minor, browser.patch, browser.patch_minor)
198+
self.device = parse_device(result.device.family, result.device.brand, result.device.model)
172199

173200
def __str__(self) -> str:
174201
return "{device} / {os} / {browser}".format(
@@ -182,9 +209,9 @@ def __repr__(self) -> str:
182209

183210
def _is_android_tablet(self) -> bool:
184211
# Newer Android tablets don't have "Mobile" in their user agent string,
185-
# older ones like Galaxy Tab still have "Mobile" though they're not
186-
return ('Mobile Safari' not in self.ua_string and
187-
self.browser.family != "Firefox Mobile")
212+
# older ones like Galaxy Tab still have "Mobile" though they're not.
213+
# Dalvik never says either way, so assume the far more common phone.
214+
return 'Mobile' not in self.ua_string and 'Dalvik' not in self.ua_string
188215

189216
def _is_blackberry_touch_capable_device(self) -> bool:
190217
# A helper to determine whether a BB phone has touch capabilities
@@ -206,6 +233,9 @@ def get_browser(self) -> str:
206233

207234
@property
208235
def is_tablet(self) -> bool:
236+
# ua-parser reports Android TVs as Generic_Android_Tablet
237+
if self.is_tv:
238+
return False
209239
if self.device.family in TABLET_DEVICE_FAMILIES:
210240
return True
211241
if self.device.brand in TABLET_DEVICE_BRANDS:
@@ -225,7 +255,7 @@ def is_mobile(self) -> bool:
225255
# First check for mobile device and mobile browser families
226256
if self.device.family in MOBILE_DEVICE_FAMILIES:
227257
return True
228-
if self.is_tablet or self.is_pc:
258+
if self.is_tablet or self.is_pc or self.is_tv:
229259
return False
230260
if self.browser.family in MOBILE_BROWSER_FAMILIES:
231261
return True
@@ -254,6 +284,8 @@ def is_mobile(self) -> bool:
254284
@property
255285
def is_touch_capable(self) -> bool:
256286
# TODO: detect touch capable Nokia devices
287+
if self.is_tv:
288+
return False
257289
if self.os.family in TOUCH_CAPABLE_OS_FAMILIES:
258290
return True
259291
if self.device.family in TOUCH_CAPABLE_DEVICE_FAMILIES:
@@ -271,7 +303,8 @@ def is_touch_capable(self) -> bool:
271303
def is_pc(self) -> bool:
272304
if self.device.family in MOBILE_DEVICE_FAMILIES or \
273305
self.device.family in TABLET_DEVICE_FAMILIES or \
274-
self.device.brand in TABLET_DEVICE_BRANDS:
306+
self.device.brand in TABLET_DEVICE_BRANDS or \
307+
self.is_tv:
275308
return False
276309
# Returns True for "PC" devices (Windows, Mac and Linux)
277310
if 'Windows NT' in self.ua_string or self.os.family in PC_OS_FAMILIES or \
@@ -291,14 +324,20 @@ def is_pc(self) -> bool:
291324
return True
292325
return False
293326

327+
@property
328+
def is_tv(self) -> bool:
329+
return TV_UA_PATTERN.search(self.ua_string) is not None
330+
294331
@property
295332
def is_bot(self) -> bool:
296-
return self.device.family == 'Spider'
333+
if self.device.family == 'Spider':
334+
return True
335+
return BOT_UA_PATTERN.search(self.ua_string) is not None
297336

298337
@property
299338
def is_email_client(self) -> bool:
300339
return self.browser.family in EMAIL_PROGRAM_FAMILIES
301340

302341

303-
def parse(user_agent_string: str) -> UserAgent:
342+
def parse(user_agent_string: Union[str, bytes, None]) -> UserAgent:
304343
return UserAgent(user_agent_string)

‎user_agents/tests.py‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -245,6 +245,78 @@ def test_unicode_strings(self):
245245
self.assertEqual(ua_str, "iPhone / iOS 5.1 / Mobile Safari 5.1")
246246
self.assertIsInstance(ua_str, str)
247247

248+
def test_missing_user_agent(self):
249+
for ua_string in (None, '', b''):
250+
ua = parse(ua_string)
251+
self.assertEqual(ua.ua_string, '')
252+
self.assertEqual(str(ua), 'Other / Other / Other')
253+
self.assertFalse(ua.is_bot)
254+
255+
def test_bytes_user_agent(self):
256+
ua = parse(iphone_ua_string.encode('latin-1'))
257+
self.assertEqual(ua.ua_string, iphone_ua_string)
258+
self.assertTrue(ua.is_mobile)
259+
260+
def test_is_bot_undetected_by_ua_parser(self):
261+
for ua_string in (
262+
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)',
263+
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)',
264+
'Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)',
265+
'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/140.0.0.0 Safari/537.36',
266+
'ExampleFetcher/1.0 (contact: ops@example.com)',
267+
'Mozilla/5.0 (compatible; ExampleCrawler; http://example.com)',
268+
'SomeBot/2.3',
269+
):
270+
self.assertTrue(parse(ua_string).is_bot, msg=ua_string)
271+
272+
def test_is_bot_ignores_real_browsers(self):
273+
for ua_string in (
274+
'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36',
275+
'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:140.0) Gecko/20100101 Firefox/140.0',
276+
'Mozilla/5.0 (Linux; Android 11; CUBOT_X50) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Mobile Safari/537.36',
277+
'Mozilla/5.0 (Linux; Android 9; CUBOT P20 Build/PPR1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Mobile Safari/537.36',
278+
iphone_ua_string,
279+
'curl/8.7.1',
280+
'okhttp/4.12.0',
281+
):
282+
self.assertFalse(parse(ua_string).is_bot, msg=ua_string)
283+
284+
def test_is_tv(self):
285+
for ua_string in (
286+
'Mozilla/5.0 (SMART-TV; Linux; Tizen 7.0) AppleWebKit/537.36 (KHTML, like Gecko) 94.0.4606.31/7.0 TV Safari/537.36',
287+
'Mozilla/5.0 (SMART-TV; X11; Linux armv7l) AppleWebKit/537.42 (KHTML, like Gecko) Safari/537.42',
288+
'Mozilla/5.0 (Web0S; Linux/SmartTV) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36 WebAppManager',
289+
'Mozilla/5.0 (Linux; Android 12; BRAVIA 4K UR3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
290+
'Mozilla/5.0 (Linux; Android 9; AFTKA) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
291+
'Mozilla/5.0 (X11; Linux aarch64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 CrKey/1.56.500000',
292+
'Roku/DVP-12.5 (12.5.0.4130-AE)',
293+
):
294+
ua = parse(ua_string)
295+
self.assertTrue(ua.is_tv, msg=ua_string)
296+
self.assertFalse(ua.is_tablet, msg=ua_string)
297+
self.assertFalse(ua.is_mobile, msg=ua_string)
298+
self.assertFalse(ua.is_pc, msg=ua_string)
299+
self.assertFalse(ua.is_touch_capable, msg=ua_string)
300+
301+
def test_is_not_tv(self):
302+
for ua in (iphone_ua, ipad_ua, galaxy_s3_ua, nexus_7_ua, ie_ua, mac_safari_ua, ubuntu_firefox_ua):
303+
self.assertFalse(ua.is_tv, msg=ua.ua_string)
304+
305+
def test_android_firefox_tablet(self):
306+
ua = parse('Mozilla/5.0 (Android 14; Tablet; rv:140.0) Gecko/140.0 Firefox/140.0')
307+
self.assertTrue(ua.is_tablet)
308+
self.assertFalse(ua.is_mobile)
309+
310+
def test_android_dalvik_is_mobile(self):
311+
ua = parse('Dalvik/2.1.0 (Linux; U; Android 7.1.2; Nexus 6P Build/N2G47O)')
312+
self.assertFalse(ua.is_tablet)
313+
self.assertTrue(ua.is_mobile)
314+
315+
def test_kaios_is_mobile(self):
316+
ua = parse('Mozilla/5.0 (Mobile; Nokia 8110 4G; rv:48.0) Gecko/48.0 Firefox/48.0 KAIOS/2.5')
317+
self.assertTrue(ua.is_mobile)
318+
self.assertFalse(ua.is_tablet)
319+
248320

249321
with open(os.path.join(os.path.dirname(__file__), 'devices.json')) as f:
250322
devices = json.load(f)
@@ -254,6 +326,7 @@ def test_wrapper(items):
254326
def test_func(self):
255327
attrs = ('is_bot', 'is_mobile',
256328
'is_pc', 'is_tablet', 'is_touch_capable')
329+
self.assertFalse(items['user_agent'].is_tv)
257330
for attr in attrs:
258331
self.assertEqual(
259332
getattr(items['user_agent'], attr), items[attr], msg=attr)

0 commit comments

Comments
 (0)