[WIP] Set Eold solver vec to zero on nonzero dirichlet boundaries.

[JustinRayAngus]

This PR works around an issue in the ThetaImplicitEM solver when the curl-curl preconditioner is used with nonzero Dirichlet boundary conditions for the electric field. On the development branch, GMRES diverges with this setup.

The solver converges as expected when the components of m_Eold that are nodal in the boundary normal direction are set to zero on the nonzero Dirichlet boundaries before being passed to the nonlinear solver.

Recall that the nonlinear solver solves F(U) = U - b - R(U) = 0, where b = m_Eold is a constant.

I describe this as a workaround because it addresses the observed divergence, but may not
resolve its underlying cause.

TODO

• Add a CI test using curl curl PC with nonzero Dirichlet BCs
• Figure out if this is the best solution

CodeX has the following to say:
The more general root fix is for the nonlinear-solver interface to expose and apply a constrained-DOF projection. Then Eold can retain its physical nonzero boundary values, while the residual and correction space explicitly excludes those boundary degrees of freedom.

From what I understand, we should make the nonlinear solvers ignore degrees of freedom on any Dirichlet boundaries. This is effectively done without modification for zero-value Dirichlet, but not for nonzero-value Dirichlet.

@debog On a related note, PETSc's SNES nonlinear solver currently only works in WarpX when using periodic boundary conditions.

[JustinRayAngus]

Note: I also found that I needed to zero-out m_E on nonzero Dirichlet boundaries for things to behave properly on a restart. This makes no sense to me.