Repository navigation
Commit 9563240
authored
fix(server): make HA sandbox create and HA e2e tests reliable (NVIDIA#3635)
* fix(server): retry runtime identity persistence on sandbox create
With multiple gateway replicas, another replica can update a new sandbox
record between the create path's read and its compare-and-swap write of the
runtime identity. The create path made one attempt, so the conflict failed
the request and deleted the backend sandbox. Persist the identity through
the same retrying helper that start and startup recovery use, which rereads
the record and retries while the sandbox stays in the same generation and a
Provisioning or Ready phase.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
* ci(e2e): run Kubernetes HA tests one at a time
The two HA tests scale and roll the shared gateway Deployment. Their
in-process lock does not apply under nextest, which runs each test in its
own process, so one test could delete a gateway pod while the other was
executing through it. Put both tests in a nextest test group limited to one
thread in the e2e-kubernetes profile. The override matches test names
because a binary() filter fails in workspaces that lack the HA test binary.
Signed-off-by: Kris Hicks <khicks@nvidia.com>
---------
Signed-off-by: Kris Hicks <khicks@nvidia.com>1 parent a408f5d commit 9563240
2 files changed
Lines changed: 81 additions & 18 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
20 | 30 | | |
21 | 31 | | |
22 | 32 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1128 | 1128 | | |
1129 | 1129 | | |
1130 | 1130 | | |
1131 | | - | |
1132 | 1131 | | |
1133 | | - | |
1134 | | - | |
1135 | | - | |
1136 | | - | |
1137 | | - | |
1138 | | - | |
1139 | | - | |
1140 | | - | |
1141 | | - | |
1142 | | - | |
1143 | | - | |
1144 | | - | |
1145 | | - | |
| 1132 | + | |
| 1133 | + | |
| 1134 | + | |
| 1135 | + | |
| 1136 | + | |
| 1137 | + | |
| 1138 | + | |
1146 | 1139 | | |
1147 | 1140 | | |
1148 | 1141 | | |
| |||
1790 | 1783 | | |
1791 | 1784 | | |
1792 | 1785 | | |
1793 | | - | |
| 1786 | + | |
1794 | 1787 | | |
1795 | 1788 | | |
1796 | 1789 | | |
| |||
1841 | 1834 | | |
1842 | 1835 | | |
1843 | 1836 | | |
1844 | | - | |
| 1837 | + | |
1845 | 1838 | | |
1846 | 1839 | | |
1847 | 1840 | | |
| |||
1900 | 1893 | | |
1901 | 1894 | | |
1902 | 1895 | | |
1903 | | - | |
| 1896 | + | |
1904 | 1897 | | |
1905 | 1898 | | |
1906 | 1899 | | |
| |||
3072 | 3065 | | |
3073 | 3066 | | |
3074 | 3067 | | |
3075 | | - | |
| 3068 | + | |
3076 | 3069 | | |
3077 | 3070 | | |
3078 | 3071 | | |
| |||
7528 | 7521 | | |
7529 | 7522 | | |
7530 | 7523 | | |
| 7524 | + | |
| 7525 | + | |
| 7526 | + | |
| 7527 | + | |
| 7528 | + | |
| 7529 | + | |
| 7530 | + | |
| 7531 | + | |
| 7532 | + | |
| 7533 | + | |
| 7534 | + | |
| 7535 | + | |
| 7536 | + | |
| 7537 | + | |
| 7538 | + | |
| 7539 | + | |
| 7540 | + | |
| 7541 | + | |
| 7542 | + | |
| 7543 | + | |
| 7544 | + | |
| 7545 | + | |
| 7546 | + | |
| 7547 | + | |
| 7548 | + | |
| 7549 | + | |
| 7550 | + | |
| 7551 | + | |
| 7552 | + | |
| 7553 | + | |
| 7554 | + | |
| 7555 | + | |
| 7556 | + | |
| 7557 | + | |
| 7558 | + | |
| 7559 | + | |
| 7560 | + | |
| 7561 | + | |
| 7562 | + | |
| 7563 | + | |
| 7564 | + | |
| 7565 | + | |
| 7566 | + | |
| 7567 | + | |
| 7568 | + | |
| 7569 | + | |
| 7570 | + | |
| 7571 | + | |
| 7572 | + | |
| 7573 | + | |
| 7574 | + | |
| 7575 | + | |
| 7576 | + | |
| 7577 | + | |
| 7578 | + | |
| 7579 | + | |
| 7580 | + | |
| 7581 | + | |
| 7582 | + | |
| 7583 | + | |
7531 | 7584 | | |
7532 | 7585 | | |
7533 | 7586 | | |
| |||
0 commit comments