F #466: Ensure one-deploy-validation playbooks always finish with full error reporting and always clean the system

[rpetrescu-one]

Summary

Reworks the validation suite around three guarantees:

1. Always clean — every resource the suite creates (VMs, vnets, images,
   templates, OneFlow services, files, packages, processes, bridges/IPs) is
   registered in a run manifest and removed by an unconditional final sweep,
   regardless of test outcomes. What cannot be cleaned (e.g. a node that is
   powered off) is reported honestly as a LEFTOVER with an operator runbook.
2. Independent tests — any test can be enabled/disabled without breaking
   the others: template export and test-vnet handling moved into shared
   validation_common services (ensure_test_template / ensure_test_vnet)
   with created-vs-reused disposition driving cleanup.
3. Run to completion — any_errors_fatal is gone; every test runs in a
   block/rescue/always harness, records ok / failed / skipped — <reason>,
   and the suite always reaches the sweep and the report.

Highlights

• Run manifest + sweep: odv_manifest fact with a /tmp/odv_manifest.json
  mirror; final sweep (name-verified deletion, id fallback only when the name
  still matches) plus a pre-run preclean that removes the previous run's
  leftovers. validation.cleanup.keep_exported_images opts out of image/template
  deletion. Second net catches strays by the odv-* naming convention.
• Report: "Execution & Cleanup Summary" audit section (every test row +
  every resource's fate), PASS/INFO/WARN/FAIL semantics (config-driven skips
  render INFO), partial (-t <tag>) runs are labeled PARTIAL RUN,
  worst-of aggregation per test.
• Per-test tags (static import_role where needed so tags actually reach
  role tasks) and an induced-failure hook (-e odv_fail_in=<slug>, 12 slugs)
  to exercise the harness.
• VM HA overhaul (validated end-to-end on a live 7.4.1 EE HA lab):
  RAFT-leader-aware hook-log verification, configurable deploy/recovery
  windows, a wedge-guard for recreates stuck in CLEANUP_RESUBMIT (applies
  onevm recover --failure once and reports WARNING — never a silent pass),
  fenced host set OFFLINE (stops the host_error hook re-fire loop), and
  optional fully automated node recovery via validation.vm_ha.node_power_on_cmd
  (power-on counterpart of the fence driver; $ODV_FENCED_HOST /
  $ODV_FENCED_HOST_ID contract). VM HA runs last; OneFlow runs before the
  core-services checks (HA-leader oned restarts destabilize flow for minutes).
• Core services: host-cycle waits for hosts back to on; the always-restore
  re-enables only genuinely disabled hosts (a blind enable resets healthy
  hosts' monitoring to INIT and starves the next test's scheduler).
• Docs: root README test catalog (flag / default / tag / link) plus
  per-role operator docs in docs/<role>.md (all options actually read by the
  code, defaults from code, examples, report keys, cleanup behavior) and a
  docs/README.md index. Option tables audited doc↔code both directions.
• Bug fixes found along the way: test-vnet template rendered its NAME from
  a variable that was never set (multi-cluster create_vnet created an
  unsuffixed vnet the sweep then missed) and now tolerates omitted
  bridge/phydev; ldap_auth login-endpoint normalization default aligned
  with role defaults.
• Removed: cleanup_test_vm and validation_frontend roles (superseded by
  the shared services, the sweep, and the report play).

Behavior changes to review

• Cleanup now deletes by default everything the run created, including
  exported marketplace images/templates (keep_exported_images: false).
• run_conn_matrix gate defaults to false in the play (reference inventory
  already disables it; enabling requires OneGate).
• Test order changed: OneFlow → core services → … → VM HA last.

Testing

• make test — offline regression harness for the sweep plan/dedup/report
  logic (63 assertions) + shell lint; ansible-playbook --syntax-check green.
• Functionally validated on a real OpenNebula 7.4.1 EE FE-HA lab
  (3 front-ends, RAFT, 2 KVM nodes) across ~25 runs: full suite green
  including VM HA end-to-end (error → hook → fence → recreate → RUNNING on the
  surviving host → automated node recovery), induced-failure runs clean up and
  report honestly, independence verified with run_test_vm: false, partial
  tag runs labeled correctly, zero unexpected leftovers — lab returned to its
  exact pre-run baseline.