You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit cbfbc06
Browse filesBrowse the repository at this point in the historyBrowse files
# Amazon Aurora PostgreSQL only (e.g. RepoDb.AuroraDb.PostgreSql). A plain PostgreSQL with PostGIS stands in for an Aurora PostgreSQL cluster.
207
+
# Amazon Aurora PostgreSQL only (e.g. RepoDb.AuroraDb.PostgreSql, RepoDb.AuroraDb.PostgreSql.BulkOperations). A plain PostgreSQL with PostGIS stands in for an Aurora PostgreSQL cluster.
description: "Package version to publish (e.g. 1.14.0)"
8
+
required: true
9
+
type: string
10
+
dependency-overrides:
11
+
description: "Semicolon-separated pins for dependencies not released at the same version this run, e.g. 'RepoDbCoreVersion=1.14.0;RepoDbAuroraDbPostgreSqlVersion=1.14.0'"
-[Bulk Insert and Merge Identity Correlation Relies on RETURNING Order](#bulk-insert-and-merge-identity-correlation-relies-on-returning-order-2)
123
+
-[Bulk COPY Errors Are Not Wrapped in AuroraDbException](#bulk-copy-errors-are-not-wrapped-in-auroradbexception)
124
+
-[Bulk Operations Ignore a Completed Transaction](#bulk-operations-ignore-a-completed-transaction)
122
125
-[Verification Status](#verification-status-10)
123
126
124
127
## Core
@@ -1618,7 +1621,7 @@ This could not be tested, because the staging tables these statements read from
1618
1621
1619
1622
## AuroraDB (PostgreSQL)
1620
1623
1621
-
These limitations are specific to the `RepoDb.AuroraDb.PostgreSql` package, on top of the [Core](#core) limitations above. Amazon Aurora PostgreSQL is PostgreSQL-compatible, so the package generates the same SQL as `RepoDb.PostgreSql`: `"` quoting, `RETURNING` for identities, `INSERT ... ON CONFLICT` for merges, `LIMIT`/`OFFSET` for paging, and `TRUNCATE ... RESTART IDENTITY`. It is built on [`RepoDb.Connector.AuroraDb.Npgsql`](https://www.nuget.org/packages/RepoDb.Connector.AuroraDb.Npgsql). That connector's `AuroraDbConnection` wraps Npgsql in the [AWS Advanced .NET Data Provider Wrapper](https://github.com/aws/aws-advanced-dotnet-data-provider-wrapper) (`AwsWrapperConnection<NpgsqlConnection>`). Unless stated otherwise, each behavior below was observed by running the integration test suite against the `postgis/postgis:latest` container (PostgreSQL `17.5`, PostGIS `3.5.2`). That container is a plain PostgreSQL server standing in for an Aurora cluster. None of it has been run against a real Aurora cluster.
1624
+
These limitations are specific to the `RepoDb.AuroraDb.PostgreSql` and `RepoDb.AuroraDb.PostgreSql.BulkOperations` packages, on top of the [Core](#core) limitations above. Amazon Aurora PostgreSQL is PostgreSQL-compatible, so the package generates the same SQL as `RepoDb.PostgreSql`: `"` quoting, `RETURNING` for identities, `INSERT ... ON CONFLICT` for merges, `LIMIT`/`OFFSET` for paging, and `TRUNCATE ... RESTART IDENTITY`. Both are built on [`RepoDb.Connector.AuroraDb.Npgsql`](https://www.nuget.org/packages/RepoDb.Connector.AuroraDb.Npgsql), and the bulk operations write rows through its `AuroraDbBulkCopy` (Npgsql binary `COPY`). That connector's `AuroraDbConnection` wraps Npgsql in the [AWS Advanced .NET Data Provider Wrapper](https://github.com/aws/aws-advanced-dotnet-data-provider-wrapper) (`AwsWrapperConnection<NpgsqlConnection>`). Unless stated otherwise, each behavior below was observed by running the integration test suite against the `postgis/postgis:latest` container (PostgreSQL `17.5`, PostGIS `3.5.2`). That container is a plain PostgreSQL server standing in for an Aurora cluster. None of it has been run against a real Aurora cluster.
1622
1625
1623
1626
### RepoDb.Connector.AuroraDb.Npgsql Is a Prerelease Package
1624
1627
@@ -1658,7 +1661,7 @@ None within RepoDB. The probes come from the AWS wrapper inside the connector. O
1658
1661
1659
1662
Inside a `TransactionScope`, the connection is enlisted in the ambient transaction when it opens. The failed probe from the [previous section](#the-aws-wrapper-probes-for-aurora-on-every-connection-open) then runs inside that transaction and aborts it. The first RepoDB operation fails with an `AuroraDbException` with `SqlState``25P02` (`current transaction is aborted, commands ignored until end of transaction block`), and nothing is committed. Explicit transactions (`connection.BeginTransaction()`) are not affected and pass their tests.
1660
1663
1661
-
On an Aurora cluster the probe succeeds, so `TransactionScope` is expected to work there, but this has not been verified. In the integration suite, the six `TransactionScope` tests are reported as inconclusive (skipped) when the server is not an Aurora cluster. Two negative tests pin the `25P02` failure, so a fix in the connector or the AWS wrapper will show up as a test failure.
1664
+
On an Aurora cluster the probe succeeds, so `TransactionScope` is expected to work there, but this has not been verified. In the integration suite, the six `TransactionScope` tests are reported as inconclusive (skipped) when the server is not an Aurora cluster. Two negative tests pin the `25P02` failure, so a fix in the connector or the AWS wrapper will show up as a test failure. The bulk operations behave the same way, and their suite has one inconclusive and one negative test of its own.
1662
1665
1663
1666
**Alternative Solution**
1664
1667
@@ -1685,14 +1688,44 @@ Aurora PostgreSQL supports PostGIS, and `AuroraDbType` has `Geometry` and `Geogr
1685
1688
1686
1689
Exchange the values as Well-Known Text through the PostGIS functions: `ST_GeomFromText(@Wkt, 4326)`/`ST_GeogFromText(@Wkt)` when writing, and `ST_AsText("Column")` when reading, with `ExecuteQuery`/`ExecuteScalar`. The integration suite's `SpatialTests` cover this. Registering `Npgsql.NetTopologySuite` through the global type mapper has not been tested.
1687
1690
1688
-
### No Bulk Operations Package Yet
1691
+
### Bulk Operations Staging Table
1692
+
1693
+
`BulkMerge`, `BulkUpdate`, `BulkDelete`, `BulkDeleteByKey`, and `BulkInsert` with `identityBehavior: ReturnIdentity` all stage rows into a pseudo table before applying them to the target table. Only a plain `BulkInsert` (without `ReturnIdentity`) writes straight into the target table through `AuroraDbBulkCopy`. The staging SQL is the same as [EnterpriseDB](#bulk-operations-staging-table-7)'s, and runs unchanged on PostgreSQL:
1694
+
1695
+
-**Staging table creation.**`CREATE [TEMP] TABLE <pseudo> AS SELECT ... WHERE (1 = 0)`, then `ALTER TABLE <pseudo> ADD COLUMN "__RepoDbBulkRowOrder__" BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY`, then `TRUNCATE TABLE <pseudo> RESTART IDENTITY`. These are DDL statements, so they need the `CREATE` privilege on the schema (for `Physical` staging) or the `TEMPORARY` privilege on the database (for `Memory` staging).
1696
+
-**Staging table type.**`Memory` uses a session-private `TEMP` table. `Physical` uses an ordinary table in the target schema, and `Auto` (the default) switches from `Memory` to `Physical` at 5,000 rows. A staging table is dropped once the operation finishes, including when it runs inside a transaction that is later rolled back.
1697
+
-**Name collisions.** Staging tables are named deterministically from `{pseudoTableType}{tableName}{Operation}` (for example `PhysicalPersonMerge`), the same scheme as the other bulk packages in this document. Two concurrent callers using `Physical` staging (or `Auto` at 5,000 rows or more) against the same target table share one staging table and can corrupt each other's staged rows.
1698
+
1699
+
**Alternative Solution**
1689
1700
1690
-
There is no `RepoDb.AuroraDb.PostgreSql.BulkOperations` package yet (`src/Providers/RepoDb.AuroraDb.PostgreSql.BulkOperations` only holds a placeholder README). The connector already ships `AuroraDbBulkCopy` (Npgsql binary `COPY`) for a future add-on.
1701
+
Use `Memory` staging for concurrent callers, and make sure the connecting user has the privileges above. On an Aurora cluster with read replicas, send bulk operations to the writer (cluster) endpoint, since they run DDL and write statements.
1702
+
1703
+
### Bulk Insert and Merge Identity Correlation Relies on RETURNING Order
1704
+
1705
+
`BulkInsert`/`BulkMerge` with `identityBehavior: ReturnIdentity` read identity values back with a single `INSERT ... SELECT ... ORDER BY <row-order column> ... RETURNING <identity>` statement. Each returned value is then assigned to the source entity or `DataRow` in order, which assumes `RETURNING` emits rows in the same order as the ordered `SELECT`. This is the same assumption described for [EnterpriseDB](#bulk-insert-and-merge-identity-correlation-relies-on-returning-order). PostgreSQL doesn't document any guarantee about `RETURNING` row order.
1691
1706
1692
1707
**Alternative Solution**
1693
1708
1694
-
Use the batched `InsertAll`/`MergeAll`/`UpdateAll` operations, or `AuroraDbBulkCopy` from `RepoDb.Connector.AuroraDb.Npgsql` directly.
1709
+
On a single PostgreSQL server, the integration suite checks the correlation for 1,000 rows (sync `BulkInsert` and async `BulkMerge`) by comparing each returned identity with the `RowGuid` of its row, and it holds. It has not been checked on an Aurora cluster. For critical data, verify the correlation against your cluster, or use a natural key instead of `ReturnIdentity`.
1710
+
1711
+
### Bulk COPY Errors Are Not Wrapped in AuroraDbException
1712
+
1713
+
`AuroraDbBulkCopy` runs the binary `COPY` on the Npgsql connection it unwraps from the AWS wrapper, so a server error raised during the `COPY` itself surfaces as an `Npgsql.PostgresException`. Errors raised by the staged SQL statements (which go through `AuroraDbCommand`) surface as an `AuroraDbException`, and so do the errors of `RepoDb.AuroraDb.PostgreSql`. For example, a plain `BulkInsert` that violates a primary key (`23505`), a `NOT NULL` constraint (`23502`) or a `VARCHAR` length (`22001`) throws `Npgsql.PostgresException`. A `catch (AuroraDbException)` block does not catch it.
1714
+
1715
+
**Alternative Solution**
1716
+
1717
+
Catch `System.Data.Common.DbException`, which both exception types derive from, and read its `SqlState`. The integration suite asserts the errors this way. A lasting fix belongs in the connector: wrap the `COPY` errors of `AuroraDbBulkCopy` the same way as the command errors.
1718
+
1719
+
### Bulk Operations Ignore a Completed Transaction
1720
+
1721
+
When a transaction that has already been committed or rolled back is passed as the `transaction` argument, the bulk operations do not reject it. The rows are written outside of any transaction and are auto-committed, so they cannot be rolled back. This was observed for both a plain `BulkInsert` (binary `COPY`, which takes no transaction object at all) and a staged `BulkMerge`. `AuroraDbTransaction` does not expose whether it has completed, so the bulk operations cannot detect it.
1722
+
1723
+
**Alternative Solution**
1724
+
1725
+
Only pass a transaction that is still active, typically by keeping the bulk call inside the `using` block of `BeginTransaction()` and before `Commit()`. A lasting fix needs the connector to expose the completion state of `AuroraDbTransaction` (for example an `IsCompleted` property), so that the bulk operations can throw an `InvalidOperationException`. Two tests in the bulk suite pin the current behavior.
1695
1726
1696
1727
### Verification Status
1697
1728
1698
1729
`RepoDb.AuroraDb.PostgreSql.UnitTests` has 194 test methods (288 test cases once the data rows are counted), and all of them pass. `RepoDb.AuroraDb.PostgreSql.IntegrationTests` has 760 test methods, 118 of which are negative (`ThrowException...`). The unit tests were ported from `RepoDb.CockroachDb`. The integration tests were ported from `RepoDb.PostgreSql`, plus the `AuroraDbType` attribute tests from `RepoDb.CockroachDb` and new `SpatialTests`. For comparison, the original `RepoDb.PostgreSql` integration suite passes 746/746 against the same container. Against the `postgis/postgis:latest` container, 1,042 of the 1,048 test cases pass on each of `net8.0`, `net9.0`, and `net10.0`; the other 6 are the inconclusive `TransactionScope` tests described above. The `build-auroradb-postgresql`/`build-pr-auroradb-postgresql` CI workflows run both suites against a `postgis/postgis:latest` service container. That image tag is not pinned, so a new PostgreSQL or PostGIS release can change CI results without any code change.
1730
+
1731
+
`RepoDb.AuroraDb.PostgreSql.BulkOperations` was ported from `RepoDb.CockroachDb.BulkOperations`, with its CockroachDB-specific staging SQL reverted to the PostgreSQL SQL of `RepoDb.EnterpriseDb.BulkOperations`. Its integration suite (`RepoDb.AuroraDb.PostgreSql.BulkOperations.IntegrationTests`, in a `RepoDbBulk` database) has 629 test cases: the 590 carried over from `RepoDb.CockroachDb.BulkOperations` (441 positive, 149 negative), plus 39 scenario cases. The scenario cases cover transactions, the pseudo-table types (including the 5,000-row `Auto` threshold), identity correlation, batch sizes, partial matches, nulls, Unicode, type coercion, the repositories, `TransactionScope`, server errors and cancellation. Against the `postgis/postgis:latest` container, 628 of the 629 cases pass on each of `net8.0`, `net9.0`, and `net10.0`; the other one is the `TransactionScope` test that needs an Aurora cluster. There is no unit test project for the `AuroraDbText` SQL generation. The `build-auroradb-postgresql-bulk`/`build-pr-auroradb-postgresql-bulk` CI workflows run the suite against the same `postgis/postgis:latest` service container.
0 commit comments