Jit: Use LazyMemoryRegion-backed bit sets

[JosJuice]

I was hoping this would help with NBA Live 2005's performance, but unfortunately it doesn't seem to have helped.

I'll keep this open for a few days as a draft in case anyone wants to try it, but then I'll close it unless we find out it is in fact faster in some circumstance.

New description:

JitCache has been using a class called ValidBlockBitSet. This PR replaces it with a new class called LazyMemoryRegionBitSet, which works similarly but is backed by a LazyMemoryRegion, letting us skip allocating memory for parts of the bit set that correspond to emulated memory regions where the game isn't storing any code. I have also added optimized functions for setting or clearing long runs of bits.

Additionally, this PR replaces JitBase's three std::unordered_sets that are keeping track of instructions with LazyMemoryRegionBitSets. Here the benefit is more speed than memory usage. This improves NBA Live 2005's menu performance by 1% or so in SuperSamus's testing.

[SuperSamus]

Testing on my end, I do see a difference.
Granted, my testing isn't the most scientific: on two instances side-by-side (the JIT only operates on a single core, so that should be fine), I load a savestate of the NBA's loading screen before the match intro cutscene, and see which one finishes the cutscene first. PR is consistently faster.
(It's more noticeable when this is cherry-picked on top of the data oriented regcache branch, since it removes some other bottlenecks that dampen the improvements otherwise. And the big bottleneck of m_back_patch_info is still there.)
And profiling shows that erase() function to be gone.

Also, there is another use case for this PR, since it can replace the class ValidBlockBitSet, where a certain comment says:

dolphin/Source/Core/Core/PowerPC/JitCommon/JitCache.h

Lines 122 to 125 in cd3ea81

	// ValidBlockBitSet covers the whole 32-bit address-space in 32-byte
	// chunks.
	// FIXME: Maybe we can get away with less? There isn't any actual
	// RAM in most of this space.

So, about that...
The data in LazyMemoryRegion, is currently u8, while for ValidBlockBitSet is u32. I also made a branch where it's u64. Which one to do?

[JosJuice]

That testing methodology sounds as least as good as mine. I've going into "Play Now" and noting down what the lowest observed FPS value is. Do you have a number for how much this PR improves performance in your scenario?

What would the advantage of going above u8 be? In my implementation, I guess if you're calling SetBits/ClearBits with a moderate number of bits (let's say somewhere between 8 and 32), using something bigger than u8 could be faster because you either entirely skip the memset or the memset gets passed a size of 0, which lets it exit early... But I'm not sure if that's such a big advantage, and I also don't know if that's a common case. It looks like the JIT either calls ClearBits for a whole block (which probably has a lot of instructions?) or it calls SetBit/ClearBit for a single bit and there's no advantage to going above u8. If you're saying it's not showing up in the profiler, I would say let's not worry about it.

[SuperSamus]

It's not easy to measure, since the difference is not consistent, but it's something like "the intro uncapped is around 29 seconds, PR is around 0.5 seconds faster".
But just to hammer the inconsistency, sometime master wins the race (though it's rare).

With the uint size I was mostly referring to this PR also replacing ValidBlockBitSet, because the JIT is hardcoded to assume that the BitSet is an u32, so it must be changed accordingly.

[JosJuice]

It's not easy to measure, since the difference is not consistent, but it's something like "the intro uncapped is around 29 seconds, PR is around 0.5 seconds faster".
But just to hammer the inconsistency, sometime master wins the race (though it's rare).

Okay, so something like up to 2% faster. Thanks, that gives me a rough idea.

With the uint size I was mostly referring to this PR also replacing ValidBlockBitSet, because the JIT is hardcoded to assume that the BitSet is an u32, so it must be changed accordingly.

It's totally fine for the C++ code to use u8 and the assembly code to use u32, or whatever combination you want. Or we could change the JIT to use u8. But after thinking about it, I realized that using u32 or u64 unlocks the microoptimization of not having to mask the shift amount by 7, because both x86-64 and AArch64 mask the shift amount by 31 or 63 for free when you use a 32-bit or 64-bit shift instruction respectively. I'll go with u32.

[SuperSamus]

It's totally fine for the C++ code to use u8 and the assembly code to use u32, or whatever combination you want

Wouldn't that be true only in a big-endian system? (Which neither x86 or ARM are.)

[JosJuice]

Isn't it the other way around? It should work on little endian only.

I'll swap to u32 either way though.

[dolphin-ci]

FifoCI detected that this change impacts graphical rendering. Here are the behavior differences detected by the system:

Detected differences

	mtl-osx-m1	mvk-osx-m1
DKCR-Char	❌ fail	❌ fail
DKCR-fast-depth	❌ fail	❌ fail
MaS-LOG-wiimote	❌ fail	❌ fail
ab11-homebrew	❌ fail	❌ fail
aeon-charge-attack	❌ fail	❌ fail
bk-tev	❌ fail	❌ fail
burnout2-vehicletextures	❌ fail	❌ fail
chibi-robo-fastdepth	❌ fail	❌ fail
chibi-robo-zfighting	❌ fail	❌ fail
custom-brawl-char	❌ fail	❌ fail
dbz-depth	❌ fail	❌ fail
default-depth-range	❌ fail	❌ fail
djfny-menu	❌ fail	❌ fail
djhero2-blend	❌ fail	❌ fail
ea-pink	❌ fail	❌ fail
ea-vp6	❌ fail	❌ fail
ed-updated	❌ fail	❌ fail
et-vid	❌ fail	❌ fail
f-zero-rain	❌ fail	❌ fail
fifa-street	❌ fail	❌ fail
find-mii	❌ fail	❌ fail
fishing-resort-map	❌ fail	❌ fail
fog-adj	❌ fail	❌ fail
fortune-street	❌ fail	❌ fail
fortune-street-fog	❌ fail	❌ fail
fortune-street-white-box	❌ fail	❌ fail
fsa-layers	❌ fail	❌ fail
goldeneye-depth	❌ fail	❌ fail
gormiti	❌ fail	❌ fail
gx-points	❌ fail	❌ fail
hb-discgolf	❌ fail	❌ fail
hotwheels-shadows	❌ fail	❌ fail
inverted-depth-range	❌ fail	❌ fail
jb-shadow	❌ fail	❌ fail
jd2-fmv	❌ fail	❌ fail
jj-awae-mirrored	❌ fail	❌ fail
kirby-logicop	❌ fail	❌ fail
kirby-shadows	❌ fail	❌ fail
last-story-shadows	❌ fail	❌ fail
lego-star-wars-crane-shadow	❌ fail	❌ fail
lesson08	❌ fail	❌ fail
line-width-test	❌ fail	❌ fail
lit-cached-normal	❌ fail	❌ fail
lm-mario-portrait	❌ fail	❌ fail
luigi-shadows	❌ fail	❌ fail
major-minor	❌ fail	❌ fail
mario-baseball-shadows	❌ fail	❌ fail
mario-golf-oob	❌ fail	❌ fail
mario-golf-vertex-expansion	❌ fail	❌ fail
mario-sluggers-bar	❌ fail	❌ fail
mario-tennis-menu	❌ fail	❌ fail
megaman-heat	❌ fail	❌ fail
melee-depth	❌ fail	❌ fail
melee-lighting	❌ fail	❌ fail
metroid-visor	❌ fail	❌ fail
mii-channel	❌ fail	❌ fail
milotic-texture	❌ fail	❌ fail
mini-ninjas	❌ fail	❌ fail
mkdd-babypark	❌ fail	❌ fail
mkdd-efb	❌ fail	❌ fail
mkw-bridge	❌ fail	❌ fail
mkw-flags	❌ fail	❌ fail
mkwii-bluebox	❌ fail	❌ fail
mmx-light	❌ fail	❌ fail
monkeyball-fuse	❌ fail	❌ fail
mp2-scanner	❌ fail	❌ fail
mp3-bloom	❌ fail	❌ fail
mp4-vertexcache	❌ fail	❌ fail
mp7-text	❌ fail	❌ fail
mp8-widescreen	❌ fail	❌ fail
mtennis-zfreeze	❌ fail	❌ fail
my-word-coach	❌ fail	❌ fail
nddemo-bumpmapping	❌ fail	❌ fail
nddemo-lighting	❌ fail	❌ fail
nes-vc	❌ fail	❌ fail
nfsu-purplerect	❌ fail	❌ fail
nfsu-reflections	❌ fail	❌ fail
nhl-slap	❌ fail	❌ fail
nintendo-channel	❌ fail	❌ fail
nsmbw-coins	❌ fail	❌ fail
nsmbw-intro	❌ fail	❌ fail
oversized-depth-range	❌ fail	❌ fail
pbr-sfx	❌ fail	❌ fail
pm-hc-jp	❌ fail	❌ fail
pokemon-channel-tv	❌ fail	❌ fail
puzzle-collection	❌ fail	❌ fail
pw-black-bars	❌ fail	❌ fail
quake-gx	❌ fail	❌ fail
rs2-bumpmapping	❌ fail	❌ fail
rs2-glass	❌ fail	❌ fail
rs2-skybox	❌ fail	❌ fail
rs2-zfreeze	❌ fail	❌ fail
rs3-bumpmapping	❌ fail	❌ fail
rs3-skybox2	❌ fail	❌ fail
sadx-ui	❌ fail	❌ fail
sf-assault-flashing	❌ fail	❌ fail
sfa-shadows	❌ fail	❌ fail
shadow-eyes	❌ fail	❌ fail
simpsons-game	❌ fail	❌ fail
smb-mirror	❌ fail	❌ fail
smg-marioeyes	❌ fail	❌ fail
smg-mmg	❌ fail	❌ fail
smg-roar	❌ fail	❌ fail
smg2-fog	❌ fail	❌ fail
sms-bubbles	❌ fail	❌ fail
sms-coconut-drinks	❌ fail	❌ fail
sms-gc	❌ fail	❌ fail
sms-water	❌ fail	❌ fail
soa-black	❌ fail	❌ fail
sonic-riders-blur	❌ fail	❌ fail
sonic-riders-zg-4p	❌ fail	❌ fail
soniccolors-mm	❌ fail	❌ fail
sonicriderszg-gb	❌ fail	❌ fail
spider-man-inverted-depth	❌ fail	❌ fail
spyro-bloom	❌ fail	❌ fail
spyro-depth	❌ fail	❌ fail
ss-map	❌ fail	❌ fail
ssbb-mod-lloyd	❌ fail	❌ fail
ssbm-pointsize	❌ fail	❌ fail
super-sluggers-white-out	❌ fail	❌ fail
sw3-dt	❌ fail	❌ fail
taiko-depth	❌ fail	❌ fail
thps3-earlyz	❌ fail	❌ fail
thps4-shadow	❌ fail	❌ fail
tla-menu	❌ fail	❌ fail
tos-invis-char	❌ fail	❌ fail
tp-skin	❌ fail	❌ fail
tsp3-pinkgrass	❌ fail	❌ fail
vegas-party-depth	❌ fail	❌ fail
viewitful-joe-distortion	❌ fail	❌ fail
ww-song-of-passing	❌ fail	❌ fail
xblade-bloom	❌ fail	❌ fail
xenoblade-menu	❌ fail	❌ fail
zbuff-test	❌ fail	❌ fail
zelda1-vc	❌ fail	❌ fail
ztp-grass	❌ fail	❌ fail
zww-armos	❌ fail	❌ fail
zww-water	❌ fail	❌ fail
zww-waves	❌ fail	❌ fail

_{^{automated-fifoci-reporter}}