Consider creating a "trustedBot" list of IPs for fine tuned SSR (to filter out aggressive bots)

[tdonohue]

Is your feature request related to a problem? Please describe.

In order to provide better protections against aggressive bots, DSpace should consider adding a "trusted bot IPs" configuration, to allow trusted bots easier harvesting access to DSpace (for SEO or similar).

What special behaviors would trusted bots possibly have?

1. Trusted bots would always have SSR enabled (to allow for harvesting). This would also allow sites to potentially disable SSR site-wide, blocking out any un-trusted bots which cannot parse Javascript.
2. Trusted bots might be treated differently by our ui > rateLimiter. Or, perhaps there's just a separate configuration for trusted bots in the rateLimiter, to allow sites to potentially loosen rules for trusted bots and tighten rules elsewhere.
3. Trusted bots would be known, verifiable bots which could also be configured to serve cached SSR content (see cache settings) if necessary for performance.

How would trusted bots be identified?

Trusted bots would need to be identified by IP address/range. Identifying bots by user-agent is no longer recommended practice because it's trivial to fake.

DSpace could populate an initial list of trusted bots from trusted sources such as:

• https://www.gstatic.com/ipranges/goog.json -> IPs of all Googlebots
• https://www.bing.com/toolbox/bingbot.json -> IPs of all Bing bots
• https://duckduckgo.com/duckduckbot.json -> IPs of all DuckDuckGo bots
• Any bot IPs which are auto-populated would also need to be auto-updated from the trusted sources on a regular basis, e.g. once per week. These lists won't change frequently, but IPs may be added/removed.

Individual DSpace sites should also have a configuration (or similar) which allows them to add/remove/update additional "trusted bots". This allows sites to give access to other trusted crawlers / harvesters, or maybe remove access if a previously-trusted bot becomes problematic.

How will this alleviate issues with aggressive bots?

By providing a way to identify the good/trusted bots, DSpace sites could then turn off SSR without any impact on SEO (because sites could configure SSR to only be used for trusted bots).

Turning off SSR helps to deter aggressive bots since many of those bots currently attempt to harvest/parse HTML. Even if they do parse Javascript, they'd do all that processing on the client-side, which would not overwhelm the node server where dspace-angular runs.

Describe the solution you'd like

While I don't have an exact design in mind, we may wish to consider if a list of "trusted bots" is best kept on the backend or fronted.
* If kept on the frontend, it'd likely need to be a large JSON file which can be parsed at runtime (and rebuilt on demand from configs or similar).
* If kept on the backend, we'd have more flexibility of keeping it in the database or configuration (or a bit of both). However, we'd still need to pass the raw data to the frontend via JSON.

Describe alternatives or workarounds you've considered

DSpace doesn't currently have a solution that lines up with this idea:

• DSpace can support only enabling SSR for some pages. But, that's a global config and doesn't allow you to change the settings per IP
• DSpace can support rateLimiting as well, but again it's a global config. No way to have different settings for different IP ranges
• DSpace also supports basic cached of SSR with a bot-specific configuration. But, that bot-specific configuration uses the isbot library which tries to identify bots based on user-agent (which is less reliable and may not work at all for highly aggressive bots)

Additional information

This issue is related to #4565 and comes out of discussions with the Google Scholar team on Sept 8, 2026.