fix(robots): update robots.txt

- don't block bingbot as that would delist the site from
  - Bing
  - Yahoo
  - DuckDuckGo
- follow best practices on some Disallow statements and using Allow

Author: Splaktar

adev-es/src/robots.txt

@@ -1,19 +1,13 @@
 # Block OpenAI
 User-agent: GPTBot
-Disallow: /
 User-agent: OAI-SearchBot
 Disallow: /
 
 # Block Anthropic (Claude)
 User-agent: anthropic-ai
-Disallow: /
 User-agent: Claude-Web
 Disallow: /
 
-# Block Microsoft/Bing AI
-User-agent: bingbot
-Disallow: /
-
 # Block Google AI (Gemini training)
 User-agent: Google-Extended
 Disallow: /
@@ -44,4 +38,4 @@ Disallow: /
 
 # Allow all standard search engines (like Google)
 User-agent: *
-Disallow:
+Allow: /