[TESTERS NEEDED] rsx/vk: Rewrite how blending works

[kd-11]

Introduces a ton of changes to how blending works in the emulator. Most of the implementation is Vulkan-only for now.

1. Programmable blending is now supported for some edge cases where PS3 behavior diverges from PC
2. Fixes ROP operation ordering. The correct order is sRGB -> Round8 -> AlphaTest -> Blend
3. Fix WCB behavior for some formats especially the 16-bit formats like RGB565 under Vulkan
4. Fix decoding of 16-bit blend constants. The previous assumptions have been completely disproven by hardware tests.
5. Rework of the ROP export code greatly reduces emitted code for games when PB is not in use. Most trivial shaders are now only a few instructions long. This builds upon previous work done to shrink fragment shaders.

Closes #19269
Relates to #15061 (not closed because OpenGL impl is missing).

OpenGL implementation is currently not provided, it will come later. The amount of changes is already too large so it makes sense to split the work.

I have no idea how many games are affected here, the blast radius is massive. There are a few issues left out that will come later:

1. The hardware tests need to be added to CI. The work here is verified across around 700 unique tests, we need to formalize them.
2. There are optimization opportunities that I decided to ignore for the moment since the blend decode mechanism needs a refactor. This will be done before OpenGL implementation is attempted.
3. The OpenGL implementation itself.

EDIT: Hardware tests attached:
rsx-blending-tests.zip

---

List of fixed issues:

• LittleBigPlanet Karting: Fixed incorrect lighting
• inFamous 1/2: Fixed refracting effects
• Jak 1/2: Fixed refracting effects
• Odin Sphere Leifthrasir: Fixed menu rendering

Fixes #19333
Fixes #19290
Fixes #11149
Partial fix #9150

[AniLeo]

Fixes #19333

[ShadowH69]

Fixes #19290

edit Also mercury menu is fixed again

[MarioSonic2987]

Fixes broken refraction effect in inFamous.

Before:

After:

[AniLeo]

Fixes broken refraction effect in inFamous.

Partially fixes #11149, Jax collection also needs to be verified

[Virtua64]

Jak 1 looks fixed:

jak.mp4

[ShadowH69]

Partially fixes #9150

Menu correctly loads and displays
as for in-game , those lines are still visible when pausing the game along with the player ghosting issue and the bird is another example

[woj1993]

Funny enought while testing this branch I had a graphical issue in Asuras Wrath but repeating the scene it played correctly. I thing I found a some kind of FIFO iassue but from my memory it arelady happened before so Asura Wrath tested and I didn't saw any difference

[kubacAkaGoomba]

Can confirm, this fixes the lighting bugs in LittleBigPlanet Karting:

Master:

PR:

On Master, it gets really bad on fully dark stages. Master:

PR makes some community levels finally playable. PR:

[ShadowH69]

I mean this was quite obvious it was still an issue but Dragon's crown is still broken just like the other Vanillaware game #4133

[sergus2008]

Regression: SoulCalibur IV (BLES00296) freezes within ~3 minutes on PR build.

Master 0.0.42-19931 works fine, 60 FPS, no freezes. PR: game hangs within 3 minutes — usually on character select screen or shortly after entering a mode. Tried Arcade, VS — same result in all modes.

Settings: WCB On, WDB On, Strict Texture Flushing On, Strict Rendering Mode Off, RCB Off. Vulkan, GTX 1070, i7-7700.

Log attached.
RPCS3.log

[sergus2008]

NFS ProStreet (BLES00177) — same hang as SC4

Game runs fine on master. On PR build, it hangs/freezes shortly after launch — same behavior as SoulCalibur 4. Log attached.

RPCS3.log

Infamous fix confirmed

No changes:
#11828
#18772

[Triticum0]

No changes on
#18194
#17628
#13749