Do not read the catalog in is_age_extension_exists() outside a live transaction (#2581)

With age in shared_preload_libraries, post_parse_analyze runs for every
statement, the ROLLBACK of a failed transaction included. When the caches of
that backend were reset (the shared invalidation queue overflowed while it
was not reading), the cached answer of is_age_extension_exists() is gone and
get_extension_oid() misses the catcache: pg_extension and pg_class are opened
in TBLOCK_ABORT, after the resource owner of the transaction was released.
The locks are granted on the fast path and ResourceOwnerEnlarge() then fails
("called after release started"). Nothing releases those locks any more; the
backend exits with them in its PGPROC, and every later backend on that PGPROC
fails at connect with "failed to re-find shared lock object" until the
server restarts. With --enable-cassert the same path trips
Assert(IsTransactionState()) in AssertCouldGetRelation().

Return false outside a live transaction and cache nothing, so the next live
transaction asks the catalog again. Only the statements that leave a failed
transaction reach the parser there, and AGE has nothing to analyse in them.

Fixes 2580.

Co-authored-by: Konstantin Serikov <constse@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

Author: 3 people

src/backend/catalog/ag_catalog.c

@@ -77,6 +77,15 @@ is_age_extension_exists(void)
 {
     static bool callback_registered = false;
 
+    /*
+     * Outside a live transaction (the ROLLBACK of a failed one) the catalog
+     * must not be read: the resource owner is already released, and the locks
+     * a catalog scan takes here would leak into the PGPROC. Nothing is cached,
+     * so the next live transaction asks the catalog again.
+     */
+    if (!IsTransactionState())
+        return false;
+
     if (extension_cache_is_valid)
         return age_extension_exists;