mirror of
https://github.com/simonw/sqlite-utils.git
synced 2026-07-24 09:54:31 +02:00
Remove TRANSFORM_FK_INVESTIGATION.md
This commit is contained in:
Claude
parent
c727ceab07
commit
a946e80809
1 changed files with 0 additions and 200 deletions
|
|
@ -1,200 +0,0 @@
|
|||
# Investigation: Foreign Key Constraints During transform()
|
||||
|
||||
This document summarizes the investigation into how the `transform()` method handles
|
||||
foreign key constraints that **reference** the table being transformed (incoming FKs).
|
||||
|
||||
## Background
|
||||
|
||||
The `transform()` method in sqlite-utils performs the following steps:
|
||||
1. Create a new temporary table with the desired schema
|
||||
2. Copy data from the old table to the new table
|
||||
3. Drop the old table
|
||||
4. Rename the new table to the original table name
|
||||
5. Recreate indexes
|
||||
|
||||
This raises the question: what happens to foreign key constraints in **other tables**
|
||||
that reference the table being transformed?
|
||||
|
||||
## Key Finding: SQLite's ALTER TABLE RENAME Behavior
|
||||
|
||||
SQLite's `ALTER TABLE ... RENAME TO` command automatically updates all foreign key
|
||||
references in the schema. This is documented in the SQLite documentation:
|
||||
|
||||
> "The RENAME command renames the table, and also updates all references to the
|
||||
> table within the schema" - https://www.sqlite.org/lang_altertable.html
|
||||
|
||||
This means:
|
||||
- When `authors_new_xxx` is renamed to `authors`, all FK constraints that reference
|
||||
`authors` continue to work correctly
|
||||
- The FK references are stored by table **name**, and the rename updates them
|
||||
|
||||
## Test Scenarios and Results
|
||||
|
||||
### Scenario 1: Simple transform (no column rename)
|
||||
```
|
||||
Setup: books.author_id REFERENCES authors(id)
|
||||
Action: db["authors"].transform(types={"name": str})
|
||||
Result: FK constraints survive intact (both with FK ON and OFF)
|
||||
```
|
||||
|
||||
### Scenario 2: Rename a non-referenced column
|
||||
```
|
||||
Setup: books.author_id REFERENCES authors(id)
|
||||
Action: db["authors"].transform(rename={"name": "author_name"})
|
||||
Result: FK constraints survive intact (both with FK ON and OFF)
|
||||
```
|
||||
|
||||
### Scenario 3: Rename the referenced column (FK ON)
|
||||
```
|
||||
Setup: books.author_id REFERENCES authors(id)
|
||||
Action: db["authors"].transform(rename={"id": "author_pk"})
|
||||
Result: FAILS - "foreign key mismatch" error, transaction rolled back
|
||||
```
|
||||
The transform correctly detects the FK violation via `PRAGMA foreign_key_check`
|
||||
and rolls back the transaction, preserving the original schema.
|
||||
|
||||
### Scenario 4: Rename the referenced column (FK OFF)
|
||||
```
|
||||
Setup: books.author_id REFERENCES authors(id)
|
||||
Action: db["authors"].transform(rename={"id": "author_pk"})
|
||||
Result: Transform succeeds, but FK constraint is now BROKEN
|
||||
```
|
||||
The FK in `books` still references `authors(id)` but that column no longer exists.
|
||||
Running `PRAGMA foreign_key_check` produces a "foreign key mismatch" error.
|
||||
|
||||
### Scenario 5: Self-referential FK
|
||||
```
|
||||
Setup: employees.manager_id REFERENCES employees(id)
|
||||
Action: db["employees"].transform(types={"name": str})
|
||||
Result: FK constraint survives intact
|
||||
```
|
||||
|
||||
### Scenario 6: Multiple tables referencing the transformed table
|
||||
```
|
||||
Setup: books.author_id, articles.writer_id, quotes.speaker_id all REFERENCE authors(id)
|
||||
Action: db["authors"].transform(types={"name": str})
|
||||
Result: All FK constraints survive intact
|
||||
```
|
||||
|
||||
## How transform() Ensures FK Safety
|
||||
|
||||
The `transform()` method (db.py lines 1853-1917) implements the following safety measures:
|
||||
|
||||
1. **Saves FK enforcement state**: Checks `PRAGMA foreign_keys` before starting
|
||||
2. **Disables FK enforcement**: Sets `PRAGMA foreign_keys=0` during the transform
|
||||
3. **Executes transform SQL**: Within a transaction (`with self.db.conn:`)
|
||||
4. **Validates FK integrity**: Runs `PRAGMA foreign_key_check` after the transform
|
||||
5. **Rolls back on failure**: If FK check fails, the transaction is rolled back
|
||||
6. **Restores FK state**: Re-enables FK enforcement if it was originally on
|
||||
|
||||
## Summary Table
|
||||
|
||||
| Scenario | FK ON | FK OFF |
|
||||
|-------------------------------------|--------------------|--------------------|
|
||||
| Simple transform | Works, FKs intact | Works, FKs intact |
|
||||
| Rename non-referenced column | Works, FKs intact | Works, FKs intact |
|
||||
| Rename referenced column | FAILS (rollback) | Works, FKs BROKEN! |
|
||||
| Drop referenced column | FAILS (rollback) | Works, FKs BROKEN! |
|
||||
| Change PK away from referenced col | FAILS (rollback) | Works, FKs BROKEN! |
|
||||
| Self-referential FK | Works, FKs intact | Works, FKs intact |
|
||||
| Multiple tables with FKs | Works, FKs intact | Works, FKs intact |
|
||||
|
||||
## Cases That Break Incoming FKs
|
||||
|
||||
There are **three** scenarios that break incoming FK constraints:
|
||||
|
||||
1. **Rename referenced column**: The FK references `authors(id)` but the column is
|
||||
renamed to `author_pk` - the FK now references a non-existent column.
|
||||
|
||||
2. **Drop referenced column**: The FK references `authors(id)` but the column is
|
||||
dropped entirely - the FK now references a non-existent column.
|
||||
|
||||
3. **Remove PK/UNIQUE from referenced column**: SQLite requires that FK targets be
|
||||
either PRIMARY KEY or UNIQUE. If you change `id` from `INTEGER PRIMARY KEY` to
|
||||
just `INTEGER`, the FK becomes invalid ("foreign key mismatch") even though the
|
||||
column still exists.
|
||||
|
||||
## Known Issue: Leftover Temp Table on Failure
|
||||
|
||||
When `transform()` fails (e.g., due to FK check failure), there may be a leftover
|
||||
temporary table (e.g., `authors_new_xxx`). This appears to be because the error
|
||||
occurs after some statements have executed but before the transaction fully commits.
|
||||
|
||||
The original table remains intact, so this is a minor cosmetic issue rather than
|
||||
a data integrity problem.
|
||||
|
||||
## Recommendations
|
||||
|
||||
1. **Always use FK enforcement** (`PRAGMA foreign_keys=ON`) when working with
|
||||
relational data to ensure transform() catches FK violations early
|
||||
|
||||
2. **Be cautious when renaming columns**: If a column is referenced by FKs from
|
||||
other tables, you'll need to update those FKs as well. Consider:
|
||||
- First transforming the referencing tables to update their FK constraints
|
||||
- Then transforming the referenced table to rename the column
|
||||
|
||||
3. **Use `foreign_key_check`** after bulk operations with FK enforcement off to
|
||||
verify data integrity
|
||||
|
||||
## Automatic Detection and Fixing
|
||||
|
||||
### Detection (Easy)
|
||||
|
||||
To detect if a transform will break incoming FKs:
|
||||
|
||||
```python
|
||||
def get_incoming_fks(db, table_name):
|
||||
"""Find all FKs from other tables that reference this table."""
|
||||
incoming = []
|
||||
for other_table in db.table_names():
|
||||
if other_table == table_name:
|
||||
continue
|
||||
for fk in db[other_table].foreign_keys:
|
||||
if fk.other_table == table_name:
|
||||
incoming.append({
|
||||
"from_table": fk.table,
|
||||
"from_column": fk.column,
|
||||
"to_column": fk.other_column,
|
||||
})
|
||||
return incoming
|
||||
```
|
||||
|
||||
Then check if any `to_column` values are in the `rename` or `drop` sets, or if
|
||||
they're losing their PK/UNIQUE status.
|
||||
|
||||
### Automatic Fixing for Column Renames (Moderate Complexity)
|
||||
|
||||
For column renames, it's possible to automatically update the referencing tables:
|
||||
|
||||
1. Before transforming table A, find all tables with FKs to A
|
||||
2. For each table B with `FK(col -> A.old_col)`:
|
||||
- Transform B with `foreign_keys=` parameter to update to `FK(col -> A.new_col)`
|
||||
3. Then transform A with the rename
|
||||
|
||||
### Challenges
|
||||
|
||||
- **Circular references**: A -> B -> A requires careful ordering
|
||||
- **Chain reactions**: Transforming B might affect table C
|
||||
- **Transaction safety**: All transforms should succeed or all fail
|
||||
|
||||
### Proposed API Enhancement
|
||||
|
||||
```python
|
||||
# Option 1: Auto-update with a flag
|
||||
db["authors"].transform(
|
||||
rename={"id": "author_pk"},
|
||||
update_incoming_fks=True # Automatically update FKs in other tables
|
||||
)
|
||||
|
||||
# Option 2: Better error message
|
||||
# "Cannot rename 'id': books.author_id references this column.
|
||||
# First transform 'books' to update its foreign key, or use
|
||||
# update_incoming_fks=True to do this automatically."
|
||||
```
|
||||
|
||||
## Code References
|
||||
|
||||
- `transform()` method: sqlite_utils/db.py:1853-1917
|
||||
- `transform_sql()` method: sqlite_utils/db.py:1919-2127
|
||||
- FK handling in transform_sql: sqlite_utils/db.py:1957-1993
|
||||
- Related tests: tests/test_transform.py:301-500
|
||||
Loading…
Add table
Add a link
Reference in a new issue