ai.robots.txt/haproxy-block-ai-bots.txt at main · saar-twito/ai.robots.txt · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
AddSearchBot
AgentTimes
AI2Bot
AI2Bot-DeepResearchEval
Ai2Bot-Dolma
aiHitBot
AIWebIndex
amazon-kendra
amazon-QBusiness
Amazonbot
AmazonBuyForMe
Amzn-SearchBot
Amzn-User
Andibot
Anomura
anthropic-ai
ApifyBot
ApifyWebsiteContentCrawler
Applebot
Applebot-Extended
Aranet-SearchBot
atlassian-bot
Awario
AzureAI-SearchBot
bedrockbot
bigsur.ai
Bravebot
Brightbot
Brightbot 1.0
BuddyBot
Bytespider
CCBot
Channel3Bot
ChatGLM-Spider
ChatGPT Agent
ChatGPT-User
Claude-Code
Claude-SearchBot
Claude-User
Claude-Web
ClaudeBot
Cloudflare-AutoRAG
CloudVertexBot
Code
cohere-ai
cohere-training-data-crawler
Cotoyogi
CragCrawler
Crawl4AI
Crawlspace
Cursor
Datenbank Crawler
DeepSeekBot
Devin
Diffbot
DuckAssistBot
Echobot Bot
EchoboxBot
ExaBot
ExaSearchBot
FacebookBot
facebookexternalhit
Factset_spyderbot
FirecrawlAgent
FriendlyCrawler
GeistHaus-PageFetcher
Gemini-Deep-Research
Google-Agent
Google-CloudVertexBot
Google-Extended
Google-Firebase
Google-Gemini-CLI
Google-NotebookLM
GoogleAgent-Mariner
GoogleAgent-URLContext
GoogleOther
GoogleOther-Image
GoogleOther-Video
GPTBot
HenkBot
iAskBot
iaskspider
iaskspider/2.0
ICC-Crawler
ImagesiftBot
imageSpider
img2dataset
ISSCyberRiskCrawler
kagi-fetcher
Kangaroo Bot
Kimi-User
KlaviyoAIBot
KunatoCrawler
laion-huggingface-processor
LAIONDownloader
LCC
Lightpanda
LinerBot
Linguee Bot
LinkupBot
Manus-User
meta-externalagent
Meta-ExternalAgent
meta-externalfetcher
Meta-ExternalFetcher
meta-webindexer
MistralAI-User
MistralAI-User/1.0
Mozilla-Tabstack
MyCentralAIScraperBot
NagetBot
netEstate Imprint Crawler
newsai
NotebookLM
NovaAct
OAI-SearchBot
omgili
omgilibot
OpenAI
opencode
Operator
PanguBot
Panscient
panscient.com
Perplexity-User
PerplexityBot
PetalBot
PhindBot
Poggio-Citations
Poseidon Research Crawler
QualifiedBot
Querit-SearchBot
QueritBot
QuillBot
quillbot.com
Reflectionbot
SBIntuitionsBot
Scrapy
SemrushBot-OCOB
SemrushBot-SWA
Shap-User
ShapBot
Sidetrade indexer bot
Spider
TavilyBot
Terra Cotta
TerraCotta
Thinkbot
TikTokSpider
Timpibot
TongyiBot
Trae
TwinAgent
UseAI
VelenPublicWebCrawler
WARDBot
Webzio-Extended
webzio-extended
wpbot
WRTNBot
YaK
YandexAdditional
YandexAdditionalBot
YiyanBot
YouBot
ZanistaBot