From a946e808092e7950fd7d4a0a57c54d02c1965854 Mon Sep 17 00:00:00 2001 From: Claude Date: Wed, 21 Jan 2026 15:18:24 +0000 Subject: [PATCH] Remove TRANSFORM_FK_INVESTIGATION.md --- TRANSFORM_FK_INVESTIGATION.md | 200 ---------------------------------- 1 file changed, 200 deletions(-) delete mode 100644 TRANSFORM_FK_INVESTIGATION.md diff --git a/TRANSFORM_FK_INVESTIGATION.md b/TRANSFORM_FK_INVESTIGATION.md deleted file mode 100644 index b6cf2d4..0000000 --- a/TRANSFORM_FK_INVESTIGATION.md +++ /dev/null @@ -1,200 +0,0 @@ -# Investigation: Foreign Key Constraints During transform() - -This document summarizes the investigation into how the `transform()` method handles -foreign key constraints that **reference** the table being transformed (incoming FKs). - -## Background - -The `transform()` method in sqlite-utils performs the following steps: -1. Create a new temporary table with the desired schema -2. Copy data from the old table to the new table -3. Drop the old table -4. Rename the new table to the original table name -5. Recreate indexes - -This raises the question: what happens to foreign key constraints in **other tables** -that reference the table being transformed? - -## Key Finding: SQLite's ALTER TABLE RENAME Behavior - -SQLite's `ALTER TABLE ... RENAME TO` command automatically updates all foreign key -references in the schema. This is documented in the SQLite documentation: - -> "The RENAME command renames the table, and also updates all references to the -> table within the schema" - https://www.sqlite.org/lang_altertable.html - -This means: -- When `authors_new_xxx` is renamed to `authors`, all FK constraints that reference - `authors` continue to work correctly -- The FK references are stored by table **name**, and the rename updates them - -## Test Scenarios and Results - -### Scenario 1: Simple transform (no column rename) -``` -Setup: books.author_id REFERENCES authors(id) -Action: db["authors"].transform(types={"name": str}) -Result: FK constraints survive intact (both with FK ON and OFF) -``` - -### Scenario 2: Rename a non-referenced column -``` -Setup: books.author_id REFERENCES authors(id) -Action: db["authors"].transform(rename={"name": "author_name"}) -Result: FK constraints survive intact (both with FK ON and OFF) -``` - -### Scenario 3: Rename the referenced column (FK ON) -``` -Setup: books.author_id REFERENCES authors(id) -Action: db["authors"].transform(rename={"id": "author_pk"}) -Result: FAILS - "foreign key mismatch" error, transaction rolled back -``` -The transform correctly detects the FK violation via `PRAGMA foreign_key_check` -and rolls back the transaction, preserving the original schema. - -### Scenario 4: Rename the referenced column (FK OFF) -``` -Setup: books.author_id REFERENCES authors(id) -Action: db["authors"].transform(rename={"id": "author_pk"}) -Result: Transform succeeds, but FK constraint is now BROKEN -``` -The FK in `books` still references `authors(id)` but that column no longer exists. -Running `PRAGMA foreign_key_check` produces a "foreign key mismatch" error. - -### Scenario 5: Self-referential FK -``` -Setup: employees.manager_id REFERENCES employees(id) -Action: db["employees"].transform(types={"name": str}) -Result: FK constraint survives intact -``` - -### Scenario 6: Multiple tables referencing the transformed table -``` -Setup: books.author_id, articles.writer_id, quotes.speaker_id all REFERENCE authors(id) -Action: db["authors"].transform(types={"name": str}) -Result: All FK constraints survive intact -``` - -## How transform() Ensures FK Safety - -The `transform()` method (db.py lines 1853-1917) implements the following safety measures: - -1. **Saves FK enforcement state**: Checks `PRAGMA foreign_keys` before starting -2. **Disables FK enforcement**: Sets `PRAGMA foreign_keys=0` during the transform -3. **Executes transform SQL**: Within a transaction (`with self.db.conn:`) -4. **Validates FK integrity**: Runs `PRAGMA foreign_key_check` after the transform -5. **Rolls back on failure**: If FK check fails, the transaction is rolled back -6. **Restores FK state**: Re-enables FK enforcement if it was originally on - -## Summary Table - -| Scenario | FK ON | FK OFF | -|-------------------------------------|--------------------|--------------------| -| Simple transform | Works, FKs intact | Works, FKs intact | -| Rename non-referenced column | Works, FKs intact | Works, FKs intact | -| Rename referenced column | FAILS (rollback) | Works, FKs BROKEN! | -| Drop referenced column | FAILS (rollback) | Works, FKs BROKEN! | -| Change PK away from referenced col | FAILS (rollback) | Works, FKs BROKEN! | -| Self-referential FK | Works, FKs intact | Works, FKs intact | -| Multiple tables with FKs | Works, FKs intact | Works, FKs intact | - -## Cases That Break Incoming FKs - -There are **three** scenarios that break incoming FK constraints: - -1. **Rename referenced column**: The FK references `authors(id)` but the column is - renamed to `author_pk` - the FK now references a non-existent column. - -2. **Drop referenced column**: The FK references `authors(id)` but the column is - dropped entirely - the FK now references a non-existent column. - -3. **Remove PK/UNIQUE from referenced column**: SQLite requires that FK targets be - either PRIMARY KEY or UNIQUE. If you change `id` from `INTEGER PRIMARY KEY` to - just `INTEGER`, the FK becomes invalid ("foreign key mismatch") even though the - column still exists. - -## Known Issue: Leftover Temp Table on Failure - -When `transform()` fails (e.g., due to FK check failure), there may be a leftover -temporary table (e.g., `authors_new_xxx`). This appears to be because the error -occurs after some statements have executed but before the transaction fully commits. - -The original table remains intact, so this is a minor cosmetic issue rather than -a data integrity problem. - -## Recommendations - -1. **Always use FK enforcement** (`PRAGMA foreign_keys=ON`) when working with - relational data to ensure transform() catches FK violations early - -2. **Be cautious when renaming columns**: If a column is referenced by FKs from - other tables, you'll need to update those FKs as well. Consider: - - First transforming the referencing tables to update their FK constraints - - Then transforming the referenced table to rename the column - -3. **Use `foreign_key_check`** after bulk operations with FK enforcement off to - verify data integrity - -## Automatic Detection and Fixing - -### Detection (Easy) - -To detect if a transform will break incoming FKs: - -```python -def get_incoming_fks(db, table_name): - """Find all FKs from other tables that reference this table.""" - incoming = [] - for other_table in db.table_names(): - if other_table == table_name: - continue - for fk in db[other_table].foreign_keys: - if fk.other_table == table_name: - incoming.append({ - "from_table": fk.table, - "from_column": fk.column, - "to_column": fk.other_column, - }) - return incoming -``` - -Then check if any `to_column` values are in the `rename` or `drop` sets, or if -they're losing their PK/UNIQUE status. - -### Automatic Fixing for Column Renames (Moderate Complexity) - -For column renames, it's possible to automatically update the referencing tables: - -1. Before transforming table A, find all tables with FKs to A -2. For each table B with `FK(col -> A.old_col)`: - - Transform B with `foreign_keys=` parameter to update to `FK(col -> A.new_col)` -3. Then transform A with the rename - -### Challenges - -- **Circular references**: A -> B -> A requires careful ordering -- **Chain reactions**: Transforming B might affect table C -- **Transaction safety**: All transforms should succeed or all fail - -### Proposed API Enhancement - -```python -# Option 1: Auto-update with a flag -db["authors"].transform( - rename={"id": "author_pk"}, - update_incoming_fks=True # Automatically update FKs in other tables -) - -# Option 2: Better error message -# "Cannot rename 'id': books.author_id references this column. -# First transform 'books' to update its foreign key, or use -# update_incoming_fks=True to do this automatically." -``` - -## Code References - -- `transform()` method: sqlite_utils/db.py:1853-1917 -- `transform_sql()` method: sqlite_utils/db.py:1919-2127 -- FK handling in transform_sql: sqlite_utils/db.py:1957-1993 -- Related tests: tests/test_transform.py:301-500