feat(datagrid): reorder table columns by dragging on every engine that can, and say why on the ones that cannot (#2586)

* feat(datagrid): offer the column reorder drag only where the engine can reorder, and add ClickHouse and Oracle

* feat(plugins): reorder columns by rebuilding the table on PostgreSQL, SQLite, libSQL, Turso and Cloudflare D1

* fix(plugins): keep a generated column generated when it is moved, and copy an identity column into the rebuilt table

* fix(datagrid): withhold the column drag on a rearranged list, and refuse to rebuild a virtual table

* fix(plugin-postgresql): name the views that will stop a column rebuild before it runs

* fix(plugins): run a column reorder on the tab's own scope, authorize it once, and stop the rebuild losing table state

Author: datlechin

CHANGELOG.md

@@ -17,6 +17,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Tab rows in Settings > General > Tabs, wrapping the strip instead of scrolling it. (#2438)
 - Autoscrolling while dragging a tab, so a tab can be moved past the run currently on screen. (#2438)
 - Move Tab to New Window on a tab's right-click menu, and by dragging a tab out of the strip. (#2438)
+- Column reorder by dragging on ClickHouse and Oracle. (#2479)
+- Column reorder on PostgreSQL, SQLite, libSQL, Turso and Cloudflare D1, through a table rebuild shown before anything runs. (#2479)
 - Recognition of SQLite and DuckDB databases by their contents, whatever they are named. (#2476)
 - `.parquet` files in Finder's Open With, read through DuckDB. (#2476)
 - Prompt to install the driver a file needs, before the file opens. (#2476)
@@ -27,6 +29,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - MongoDB editor diagnostics report JavaScript syntax errors rather than unsupported method names.
 - Editor tab presses handled by AppKit rather than SwiftUI gestures. (#2438)
 - Connection-first labels with the database or schema on a second line in the connections strip. (#2550)
+- Column reorder withheld, with the reason on the row number, where the engine cannot change column order. (#2479)
 - File > Open File… as an app command over every file TablePro reads. (#2476)
 
 ### Fixed

Plugins/ClickHouseDriverPlugin/ClickHousePlugin.swift

@@ -827,6 +827,52 @@ final class ClickHousePluginDriver: PluginDatabaseDriver, @unchecked Sendable {
         "ALTER TABLE \(quoteIdentifier(table)) DROP COLUMN \(quoteIdentifier(columnName))"
     }
 
+    /// `ALTER TABLE … MODIFY COLUMN name type FIRST | AFTER other`.
+    ///
+    /// Measured against 26.8: the type is mandatory (`MODIFY COLUMN c AFTER b` is a syntax error),
+    /// and naming it alone is enough. `MODIFY COLUMN` changes only the properties the statement
+    /// spells out, so the default, comment, codec, TTL and the MATERIALIZED, ALIAS and EPHEMERAL
+    /// kinds all survive a move, and the statement rewrites metadata without starting a mutation.
+    /// Restating the full definition instead would rewrite a MATERIALIZED column as a DEFAULT one
+    /// and requote every expression default.
+    ///
+    /// The type comes from the server rather than from the caller's definition, because it has to
+    /// be the stored type down to the `Nullable(…)` wrapper and a round trip is cheaper than a
+    /// column that comes back with a different type than it went in with.
+    func generateColumnReorderPlan(
+        table: String,
+        schema: String?,
+        columns: [PluginColumnDefinition],
+        desiredOrder: [String]
+    ) async throws -> PluginColumnReorderPlan? {
+        let storedTypes = try await fetchStoredColumnTypes(table: table)
+        let currentOrder = storedTypes.map(\.name)
+        let statements = PluginColumnReorderPlanner
+            .moves(from: currentOrder, to: desiredOrder)
+            .compactMap { move -> String? in
+                guard let type = storedTypes.first(where: { $0.name == move.column })?.type else { return nil }
+                let position = move.afterColumn.map { "AFTER \(quoteIdentifier($0))" } ?? "FIRST"
+                return "ALTER TABLE \(quoteIdentifier(table)) "
+                    + "MODIFY COLUMN \(quoteIdentifier(move.column)) \(type) \(position)"
+            }
+        guard !statements.isEmpty else { return nil }
+        return PluginColumnReorderPlan(statements: statements, cost: .metadataOnly)
+    }
+
+    private func fetchStoredColumnTypes(table: String) async throws -> [(name: String, type: String)] {
+        let escapedTable = table.replacingOccurrences(of: "'", with: "''")
+        let result = try await execute(query: """
+            SELECT name, type
+            FROM system.columns
+            WHERE database = currentDatabase() AND table = '\(escapedTable)'
+            ORDER BY position
+            """)
+        return result.rows.compactMap { row in
+            guard let name = row[safe: 0]?.asText, let type = row[safe: 1]?.asText else { return nil }
+            return (name, type)
+        }
+    }
+
     func generateAddIndexSQL(table: String, index: PluginIndexDefinition) -> String? {
         let cols = index.columns.map { quoteIdentifier($0) }.joined(separator: ", ")
         let indexType = index.indexType ?? "minmax"

Plugins/CloudflareD1DriverPlugin/CloudflareD1PluginDriver.swift

@@ -714,6 +714,32 @@ final class CloudflareD1PluginDriver: PluginDatabaseDriver, @unchecked Sendable
         "ALTER TABLE \(quoteIdentifier(table)) DROP COLUMN \(quoteIdentifier(columnName))"
     }
 
+    /// SQLite has no positional `ALTER`, so the order changes by rebuilding the table, using the
+    /// shared recipe every SQLite-derived driver follows.
+    ///
+    /// Never run by TablePro. D1 answers each statement over its own HTTP request, so nothing can
+    /// hold the rebuild's transaction open across them, and a half-applied rebuild is data loss.
+    func generateColumnReorderPlan(
+        table: String,
+        schema: String?,
+        columns: [PluginColumnDefinition],
+        desiredOrder: [String]
+    ) async throws -> PluginColumnReorderPlan? {
+        try await SQLiteColumnReorderPlanner.plan(
+            tableName: table,
+            desiredOrder: desiredOrder,
+            isRunnable: false,
+            execute: { try await self.execute(query: $0) }
+        )
+    }
+
+    func columnReorderSchemaFingerprint(table: String, schema: String?) async throws -> String? {
+        try await SQLiteColumnReorderPlanner.schemaFingerprint(
+            tableName: table,
+            execute: { try await self.execute(query: $0) }
+        )
+    }
+
     func generateAddIndexSQL(table: String, index: PluginIndexDefinition) -> String? {
         let uniqueStr = index.isUnique ? "UNIQUE " : ""
         let cols = index.columns.map { quoteIdentifier($0) }.joined(separator: ", ")

Plugins/LibSQLDriverPlugin/LibSQLPluginDriver.swift

@@ -730,6 +730,33 @@ final class LibSQLPluginDriver: PluginDatabaseDriver, @unchecked Sendable {
         "ALTER TABLE \(quoteIdentifier(table)) DROP COLUMN \(quoteIdentifier(columnName))"
     }
 
+    /// SQLite has no positional `ALTER`, so the order changes by rebuilding the table, using the
+    /// shared recipe every SQLite-derived driver follows.
+    ///
+    /// Runnable only in local mode. A local database is a real SQLite handle that holds a
+    /// transaction across statements, which is what makes the rebuild atomic; over HTTP each
+    /// statement is its own request and the script has to be handed to the user instead.
+    func generateColumnReorderPlan(
+        table: String,
+        schema: String?,
+        columns: [PluginColumnDefinition],
+        desiredOrder: [String]
+    ) async throws -> PluginColumnReorderPlan? {
+        try await SQLiteColumnReorderPlanner.plan(
+            tableName: table,
+            desiredOrder: desiredOrder,
+            isRunnable: isLocalMode,
+            execute: { try await self.execute(query: $0) }
+        )
+    }
+
+    func columnReorderSchemaFingerprint(table: String, schema: String?) async throws -> String? {
+        try await SQLiteColumnReorderPlanner.schemaFingerprint(
+            tableName: table,
+            execute: { try await self.execute(query: $0) }
+        )
+    }
+
     func generateAddIndexSQL(table: String, index: PluginIndexDefinition) -> String? {
         let uniqueStr = index.isUnique ? "UNIQUE " : ""
         let cols = index.columns.map { quoteIdentifier($0) }.joined(separator: ", ")

Plugins/MySQLDriverPlugin/MySQLColumnDefinitionSQL.swift

@@ -88,8 +88,18 @@ internal func mysqlColumnDefinitionSQL(_ column: PluginColumnDefinition) -> Stri
     // A generated column takes the expression in place of the ordinary default and auto-increment
     // attributes, and MySQL rejects most of them alongside it. The keyword is spelled out because
     // both MySQL and MariaDB default to VIRTUAL.
+    // Charset and collation belong to the type, so they come before the expression. Leaving them
+    // out reset a generated string column to the table defaults, because MODIFY COLUMN replaces
+    // the whole definition and a reorder restates it.
     let kind = (column.generationKind ?? .virtual).rawValue
-    var definition = "\(name) \(column.dataType) GENERATED ALWAYS AS (\(expression)) \(kind)"
+    var definition = "\(name) \(column.dataType)"
+    if let charset = column.charset, !charset.isEmpty {
+        definition += " CHARACTER SET \(charset)"
+    }
+    if let collation = column.collation, !collation.isEmpty {
+        definition += " COLLATE \(collation)"
+    }
+    definition += " GENERATED ALWAYS AS (\(expression)) \(kind)"
     if !column.isNullable { definition += " NOT NULL" }
     if let comment = column.comment, !comment.isEmpty {
         definition += " COMMENT '\(mysqlEscapeStringLiteral(comment))'"

Plugins/MySQLDriverPlugin/MySQLPluginDriver.swift

@@ -944,18 +944,31 @@ final class MySQLPluginDriver: PluginDatabaseDriver, @unchecked Sendable {
 
     func generateMoveColumnSQL(table: String, column: PluginColumnDefinition, afterColumn: String?) -> String? {
         let tableName = quoteIdentifier(table)
-        let colName = quoteIdentifier(column.name)
-
-        let def = "\(column.dataType)" + mysqlColumnAttributesSQL(column)
-
-        let position: String
-        if let afterCol = afterColumn {
-            position = "AFTER \(quoteIdentifier(afterCol))"
-        } else {
-            position = "FIRST"
-        }
-
-        return "ALTER TABLE \(tableName) MODIFY COLUMN \(colName) \(def) \(position)"
+        let position = afterColumn.map { "AFTER \(quoteIdentifier($0))" } ?? "FIRST"
+        /// The same builder `ADD COLUMN` uses, rather than the attribute list alone. `MODIFY`
+        /// replaces the whole definition, and the attribute list does not carry
+        /// `GENERATED ALWAYS AS`, so moving a generated column with it dropped the expression and
+        /// left a plain column of stored defaults behind.
+        return "ALTER TABLE \(tableName) MODIFY COLUMN \(buildColumnDefinitionSQL(column)) \(position)"
+    }
+
+    /// `MODIFY COLUMN` replaces the whole definition, so every move restates the column in full.
+    /// Restating only the type is what drops charset, collation and `ON UPDATE`.
+    func generateColumnReorderPlan(
+        table: String,
+        schema: String?,
+        columns: [PluginColumnDefinition],
+        desiredOrder: [String]
+    ) async throws -> PluginColumnReorderPlan? {
+        let byName = Dictionary(columns.map { ($0.name, $0) }, uniquingKeysWith: { first, _ in first })
+        let statements = PluginColumnReorderPlanner
+            .moves(from: columns.map(\.name), to: desiredOrder)
+            .compactMap { move -> String? in
+                guard let column = byName[move.column] else { return nil }
+                return generateMoveColumnSQL(table: table, column: column, afterColumn: move.afterColumn)
+            }
+        guard !statements.isEmpty else { return nil }
+        return PluginColumnReorderPlan(statements: statements, cost: .metadataOnly)
     }
 
     // MARK: - View Templates

Plugins/OracleDriverPlugin/OraclePlugin.swift

@@ -1034,6 +1034,48 @@ final class OraclePluginDriver: PluginDatabaseDriver, @unchecked Sendable {
         "ALTER TABLE \(oracleQualifiedTable(table)) DROP COLUMN \(quoteIdentifier(columnName))"
     }
 
+    /// Oracle has no positional clause, but making a column invisible and visible again moves it to
+    /// the end of the visible order, so any order is reachable by appending the right suffix.
+    ///
+    /// Measured against Oracle Free 23: the cycle works on the primary key, on an identity column
+    /// and on a virtual column; the rows, the default, the NOT NULL, the comment, the identity
+    /// sequence, the constraints, the indexes and the foreign keys pointing at the table all
+    /// survive it, and no data is read or written. Needs 12.1, where invisible columns arrived; an
+    /// older server rejects the statement and the error is reported as it is.
+    ///
+    /// The two halves of a cycle are separate statements because Oracle commits each DDL on its
+    /// own, so a column is invisible for the width of one statement. Cycling one column at a time
+    /// keeps that window as small as it can be.
+    func generateColumnReorderPlan(
+        table: String,
+        schema: String?,
+        columns: [PluginColumnDefinition],
+        desiredOrder: [String]
+    ) async throws -> PluginColumnReorderPlan? {
+        let qt = oracleQualifiedTable(table)
+        let currentOrder = try await fetchColumns(table: table, schema: schema).map(\.name)
+        let cycled = PluginColumnReorderPlanner.appendCycle(from: currentOrder, to: desiredOrder)
+        let statements = cycled.flatMap { column -> [String] in
+            let quoted = quoteIdentifier(column)
+            return [
+                "ALTER TABLE \(qt) MODIFY (\(quoted) INVISIBLE)",
+                "ALTER TABLE \(qt) MODIFY (\(quoted) VISIBLE)"
+            ]
+        }
+        guard !statements.isEmpty else { return nil }
+
+        /// Oracle commits each DDL statement on its own, so there is no transaction to roll back:
+        /// a cycle whose `VISIBLE` half fails, on a dropped connection or a server error, leaves
+        /// that column hidden for good. Every cycled column gets a compensating `VISIBLE` that the
+        /// executor runs on any mid-plan failure, which is idempotent on a column that is already
+        /// visible and puts back the one that is not.
+        return PluginColumnReorderPlan(
+            statements: statements,
+            compensation: cycled.map { "ALTER TABLE \(qt) MODIFY (\(quoteIdentifier($0)) VISIBLE)" },
+            cost: .metadataOnly
+        )
+    }
+
     func generateAddIndexSQL(table: String, index: PluginIndexDefinition) -> String? {
         oracleIndexDefinition(index, qualifiedTable: oracleQualifiedTable(table))
     }