[WIP] Add more tests to GPU CI pipelines

[EZoni]

Summary

This pull request expands GPU test coverage and improves test selection for both the NVIDIA-H100 and AMD-MI300 CI pipelines in .gitlab/ci.yaml.

• Added support for new simulation dimensions by setting -DWarpX_DIMS="1;2;3;RZ;RCYLINDER;RSPHERE" in both the NVIDIA-H100 and AMD-MI300 build configurations, enabling tests for 1D, 2D, 3D, RZ, RCYLINDER, and RSPHERE geometries.

• Refactored GPU test selection to use a GPU_TESTS array and dynamically construct a regex (GPU_TESTS_REGEX) that matches all relevant test stages, making it easier to add or remove tests in the future.

• Updated the test execution commands to use the new regex, ensuring all specified GPU tests (and their stages, e.g., .run, .analysis) are included, rather than just a single hard-coded test.

[EZoni]

Interesting things happening with the test test_rz_embedded_boundary_diffraction on GPU:

      Start 137: test_rz_embedded_boundary_diffraction.checksum
 6/18 Test #137: test_rz_embedded_boundary_diffraction.checksum ...***Failed    1.88 sec
ERROR: Benchmark and output file checksum have different value for key [lev=0,Br_0_real]
Benchmark: [lev=0,Br_0_real] 6.734053747573620e-21
Test file: [lev=0,Br_0_real] 2.475321597000812e-12
Absolute error: 2.48e-12
Relative error: 3.68e+08
ERROR: Benchmark and output file checksum have different value for key [lev=0,Br_1_real]
Benchmark: [lev=0,Br_1_real] 6.790835689542519e-19
Test file: [lev=0,Br_1_real] 4.815061201766250e-12
Absolute error: 4.82e-12
Relative error: 7.09e+06
ERROR: Benchmark and output file checksum have different value for key [lev=0,Bt_0_real]
Benchmark: [lev=0,Bt_0_real] 4.224130526556551e-19
Test file: [lev=0,Bt_0_real] 2.050644543061253e-09
Absolute error: 2.05e-09
Relative error: 4.85e+09
ERROR: Benchmark and output file checksum have different value for key [lev=0,Bt_1_imag]
Benchmark: [lev=0,Bt_1_imag] 3.011041235672114e-18
Test file: [lev=0,Bt_1_imag] 4.837346348896552e-12
Absolute error: 4.84e-12
Relative error: 1.61e+06
ERROR: Benchmark and output file checksum have different value for key [lev=0,Bz_0_real]
Benchmark: [lev=0,Bz_0_real] 1.150599094570013e-20
Test file: [lev=0,Bz_0_real] 3.532782519213157e-13
Absolute error: 3.53e-13
Relative error: 3.07e+07
ERROR: Benchmark and output file checksum have different value for key [lev=0,Bz_1_real]
Benchmark: [lev=0,Bz_1_real] 2.353290703108424e-19
Test file: [lev=0,Bz_1_real] 7.370340026762080e-13
Absolute error: 7.37e-13
Relative error: 3.13e+06
ERROR: Benchmark and output file checksum have different value for key [lev=0,Er_0_real]
Benchmark: [lev=0,Er_0_real] 8.531992004330063e-11
Test file: [lev=0,Er_0_real] 4.599758982658386e-01
Absolute error: 4.60e-01
Relative error: 5.39e+09
ERROR: Benchmark and output file checksum have different value for key [lev=0,Er_1_imag]
Benchmark: [lev=0,Er_1_imag] 5.994872141130550e-10
Test file: [lev=0,Er_1_imag] 1.457765236677335e-03
Absolute error: 1.46e-03
Relative error: 2.43e+06
ERROR: Benchmark and output file checksum have different value for key [lev=0,Et_0_real]
Benchmark: [lev=0,Et_0_real] 3.480736443382604e-12
Test file: [lev=0,Et_0_real] 7.588022854179144e-04
Absolute error: 7.59e-04
Relative error: 2.18e+08
ERROR: Benchmark and output file checksum have different value for key [lev=0,Et_1_real]
Benchmark: [lev=0,Et_1_real] 1.525102870441420e-10
Test file: [lev=0,Et_1_real] 1.451718090066750e-03
Absolute error: 1.45e-03
Relative error: 9.52e+06
ERROR: Benchmark and output file checksum have different value for key [lev=0,Ez_0_real]
Benchmark: [lev=0,Ez_0_real] 4.271922350366116e-11
Test file: [lev=0,Ez_0_real] 2.336871176958084e-01
Absolute error: 2.34e-01
Relative error: 5.47e+09
ERROR: Benchmark and output file checksum have different value for key [lev=0,Ez_1_imag]
Benchmark: [lev=0,Ez_1_imag] 2.993715574333774e-10
Test file: [lev=0,Ez_1_imag] 1.821476040248589e-04
Absolute error: 1.82e-04
Relative error: 6.08e+05
Maximum relative error: 5.47e+09

May be a single-precision bug.

[EZoni]

Also, with the extra builds and tests, I think we may need to increase the time limit set for the pipelines. Tried it in 17a7bc8, let's see if it works.

[EZoni]

After switching to double precision, we might be able to tighten the checksum tolerance set in #7055:

warpx/Examples/analysis_default_regression.py

Lines 46 to 48 in fea9891

	if compute_backend in {"CUDA", "HIP", "SYCL"}:
	# GPU checksums
	default_tolerance = 1e-1

Testing in 980972b.

[EZoni]

Something else from the job logs we might need to take care of:

Job's log exceeded limit of 4194304 bytes.
Job execution will continue but no more output will be collected.

Suggested fix in 430eb79. The normal CMake build output still reports progress, warnings, and errors without printing every enormous GPU compiler command.

[EZoni]

New build error found after taking care of the side issues:

[ 99%] Linking CXX executable bin/warpx.rz.MPI.HIP.DP.PDP.OPMD.FFT.EB.QED
clang++: warning: argument unused during compilation: '-Xoffload-linker --whole-archive' [-Wunused-command-line-argument]
ld.lld: error: undefined symbol: blas::Queue::Queue(int, void*&)
>>> referenced by HankelTransform.cpp
>>>               HankelTransform.cpp.o:(HankelTransform::HankelTransform(int, int, int, double)) in archive lib/libwarpx.rz.MPI.HIP.DP.PDP.OPMD.FFT.EB.QED.a
ld.lld: error: undefined symbol: blas::Queue::set_stream(void*&)
>>> referenced by HankelTransform.cpp
>>>               HankelTransform.cpp.o:(HankelTransform::HankelForwardTransform(amrex::FArrayBox const&, int, amrex::FArrayBox&, int)) in archive lib/libwarpx.rz.MPI.HIP.DP.PDP.OPMD.FFT.EB.QED.a
>>> referenced by HankelTransform.cpp
>>>               HankelTransform.cpp.o:(HankelTransform::HankelInverseTransform(amrex::FArrayBox const&, int, amrex::FArrayBox&, int)) in archive lib/libwarpx.rz.MPI.HIP.DP.PDP.OPMD.FFT.EB.QED.a
clang++: error: linker command failed with exit code 1 (use -v to see invocation)
gmake[2]: *** [CMakeFiles/app_rz.dir/build.make:122: bin/warpx.rz.MPI.HIP.DP.PDP.OPMD.FFT.EB.QED] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:4441: CMakeFiles/app_rz.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....