fix(SetupChecks): allow compatible X-Robots-Tag directives

[joshtrichards]

• Resolves: [Bug]: Setup check for X-Robot-Tag expects exact match, resulting in a warning #37409

Summary

Builds on #64571 -

Preserve the required noindex,nofollow directives while allowing compatible additions such as noarchive, nosnippet, noimageindex, and notranslate. Still disallow incompatible combinations.

Includes updated tests.

TODO

Checklist

• Code is properly formatted
• Sign-off message is added to all commits
• Tests (unit, integration, api and/or acceptance) are included
• Screenshots before/after for front-end changes
• Documentation (manuals or wiki) has been updated or is not required
• Backports requested where applicable (ex: critical bugfixes)
• Labels added where applicable (ex: bug/enhancement, 3. to review, feature component)
• Milestone added for target branch/version (ex: 32.x for stable32)

AI (if applicable)

• The content of this PR was partly or fully generated using AI

[come-nc]

I do not know enough of the sysadmin part to approve this one.
I’m also not sure why parseDirective is a callable and not inlined since it’s only used once?

[come-nc]

Should use the same trick as above, why did you implement it different? You can strlower the whole thing.

Suggested change

	$directives = array_map(
	static fn (string $directive): string => strtolower(trim($directive)),
	explode(',', $value)
	);
	$directives = array_map('trim', explode(',', strtolower($value)));

[MichaIng]

From sysadmin side it makes sense: If noindex, and nofollow are there, everything else is redundant, but does not hurt.