{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "27ae18ec",
   "metadata": {},
   "source": [
    "# The sqlite-utils tutorial\n",
    "\n",
    "[sqlite-utils](https://sqlite-utils.datasette.io/en/stable/python-api.html) is a Python library (and [command-line tool](https://sqlite-utils.datasette.io/en/stable/cli.html) for quickly creating and manipulating SQLite database files.\n",
    "\n",
    "This tutorial will show you how to use the Python library to manipulate data.\n",
    "\n",
    "## Installation\n",
    "\n",
    "To install the library, run:\n",
    "\n",
    "    pip install sqlite-utils\n",
    "\n",
    "You can run this in a Jupyter notebook cell by executing:\n",
    "\n",
    "    %pip install sqlite-utils\n",
    "    \n",
    "Or use `pip install -U sqlite-utils` to ensure you have upgraded to the most recent version."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "bddee0d2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: sqlite_utils in /usr/local/Cellar/jupyterlab/3.0.16_1/libexec/lib/python3.9/site-packages (3.14)\n",
      "Requirement already satisfied: click-default-group in /usr/local/lib/python3.9/site-packages (from sqlite_utils) (1.2.2)\n",
      "Requirement already satisfied: sqlite-fts4 in /usr/local/lib/python3.9/site-packages (from sqlite_utils) (1.0.1)\n",
      "Requirement already satisfied: click in /Users/simon/Library/Python/3.9/lib/python/site-packages (from sqlite_utils) (7.1.2)\n",
      "Requirement already satisfied: tabulate in /usr/local/lib/python3.9/site-packages (from sqlite_utils) (0.8.7)\n",
      "Requirement already satisfied: dateutils in /usr/local/Cellar/jupyterlab/3.0.16_1/libexec/lib/python3.9/site-packages (from sqlite_utils) (0.6.12)\n",
      "Requirement already satisfied: python-dateutil in /usr/local/Cellar/jupyterlab/3.0.16_1/libexec/lib/python3.9/site-packages (from dateutils->sqlite_utils) (2.8.1)\n",
      "Requirement already satisfied: pytz in /usr/local/Cellar/jupyterlab/3.0.16_1/libexec/lib/python3.9/site-packages (from dateutils->sqlite_utils) (2021.1)\n",
      "Requirement already satisfied: six>=1.5 in /usr/local/Cellar/jupyterlab/3.0.16_1/libexec/lib/python3.9/site-packages (from python-dateutil->dateutils->sqlite_utils) (1.16.0)\n",
      "\u001b[33mWARNING: You are using pip version 21.1.1; however, version 21.2.2 is available.\n",
      "You should consider upgrading via the '/usr/local/Cellar/jupyterlab/3.0.16_1/libexec/bin/python3.9 -m pip install --upgrade pip' command.\u001b[0m\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install -U sqlite_utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "050e85a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sqlite_utils"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "348bcbfc",
   "metadata": {},
   "source": [
    "You can use the library with a database file on disk by running:\n",
    "\n",
    "    db = sqlite_utils.Database(\"path/to/my/database.db\")\n",
    "\n",
    "In this tutorial we will use an in-memory database. This is a quick way to try out new things, though you should note that when you close the notebook the data store in the in-memory database will be lost."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "4b2aee7e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Database <sqlite3.Connection object at 0x139a16300>>"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db = sqlite_utils.Database(memory=True)\n",
    "db"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1598ab43",
   "metadata": {},
   "source": [
    "## Creating a table\n",
    "\n",
    "We are going to create a new table in our database called `creatures` by passing in a Python list of dictionaries.\n",
    "\n",
    "`db[name_of_table]` will access a database table object with that name.\n",
    "\n",
    "Inserting data into that table will create it if it does not already exist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4a0ac420",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Table creatures (name, species, age)>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db[\"creatures\"].insert_all([{\n",
    "    \"name\": \"Cleo\",\n",
    "    \"species\": \"dog\",\n",
    "    \"age\": 6\n",
    "}, {\n",
    "    \"name\": \"Lila\",\n",
    "    \"species\": \"chicken\",\n",
    "    \"age\": 0.8,\n",
    "}, {\n",
    "    \"name\": \"Bants\",\n",
    "    \"species\": \"chicken\",\n",
    "    \"age\": 0.8,\n",
    "}])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "049d110b",
   "metadata": {},
   "source": [
    "Let's grab a `table` reference to the new creatures table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "8d84ad9c",
   "metadata": {},
   "outputs": [],
   "source": [
    "table = db[\"creatures\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffe45750",
   "metadata": {},
   "source": [
    "`sqlite-utils` automatically creates a table schema that matches the keys and data types of the dictionaries that were passed to `.insert_all()`.\n",
    "\n",
    "We can see that schema using `table.schema`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "136cee1e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CREATE TABLE [creatures] (\n",
      "   [name] TEXT,\n",
      "   [species] TEXT,\n",
      "   [age] FLOAT\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "print(table.schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e5c3ae9",
   "metadata": {},
   "source": [
    "## Accessing data\n",
    "\n",
    "The `table.rows` property lets us loop through the rows in the table, returning each one as a Python dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f812914d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'name': 'Cleo', 'species': 'dog', 'age': 6.0}\n",
      "{'name': 'Lila', 'species': 'chicken', 'age': 0.8}\n",
      "{'name': 'Bants', 'species': 'chicken', 'age': 0.8}\n"
     ]
    }
   ],
   "source": [
    "for row in table.rows:\n",
    "    print(row)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60bc6b2c",
   "metadata": {},
   "source": [
    "The `db.query(sql)` method can be used to execute SQL queries and return the results as dictionaries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "eaadd85f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'name': 'Cleo', 'species': 'dog', 'age': 6.0},\n",
       " {'name': 'Lila', 'species': 'chicken', 'age': 0.8},\n",
       " {'name': 'Bants', 'species': 'chicken', 'age': 0.8}]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(db.query(\"select * from creatures\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6614467b",
   "metadata": {},
   "source": [
    "Or in a loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "88fdd52e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cleo is a dog\n",
      "Lila is a chicken\n",
      "Bants is a chicken\n"
     ]
    }
   ],
   "source": [
    "for row in db.query(\"select name, species from creatures\"):\n",
    "    print(f'{row[\"name\"]} is a {row[\"species\"]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b81c031c",
   "metadata": {},
   "source": [
    "### SQL parameters\n",
    "\n",
    "You can run a parameterized query using `?` as placeholders and passing a list of variables. The variables you pass will be correctly quoted, protecting your code from SQL injection vulnerabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "267035d9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'name': 'Cleo', 'species': 'dog', 'age': 6.0}]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(db.query(\"select * from creatures where age > ?\", [1.0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87cb301b",
   "metadata": {},
   "source": [
    "As an alternative to question marks we can use `:name` parameters and feed in the values using a dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "83be9a80",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'name': 'Lila', 'species': 'chicken', 'age': 0.8},\n",
       " {'name': 'Bants', 'species': 'chicken', 'age': 0.8}]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(db.query(\"select * from creatures where species = :species\", {\"species\": \"chicken\"}))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e5179cc",
   "metadata": {},
   "source": [
    "### Primary keys\n",
    "\n",
    "When we created this table we did not specify a primary key. SQLite automatically creates a primary key called `rowid` if no other primary key is defined.\n",
    "\n",
    "We can run `select rowid, * from creatures` to see this hidden primary key:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c9d963df",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'rowid': 1, 'name': 'Cleo', 'species': 'dog', 'age': 6.0},\n",
       " {'rowid': 2, 'name': 'Lila', 'species': 'chicken', 'age': 0.8},\n",
       " {'rowid': 3, 'name': 'Bants', 'species': 'chicken', 'age': 0.8}]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(db.query(\"select rowid, * from creatures\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f87cdfb",
   "metadata": {},
   "source": [
    "We can also see that using `table.pks_and_rows_where()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "d365e405",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 {'rowid': 1, 'name': 'Cleo', 'species': 'dog', 'age': 6.0}\n",
      "2 {'rowid': 2, 'name': 'Lila', 'species': 'chicken', 'age': 0.8}\n",
      "3 {'rowid': 3, 'name': 'Bants', 'species': 'chicken', 'age': 0.8}\n"
     ]
    }
   ],
   "source": [
    "for pk, row in table.pks_and_rows_where():\n",
    "    print(pk, row)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b0e9b74",
   "metadata": {},
   "source": [
    "Let's recreate the table with our own primary key, which we will call `id`.\n",
    "\n",
    "`table.drop()` drops the table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "568a0e29",
   "metadata": {},
   "outputs": [],
   "source": [
    "table.drop()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "13ebd3ab",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Table creatures (does not exist yet)>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "522aa6d0",
   "metadata": {},
   "source": [
    "We can see a list of tables in the database using `db.tables`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "f3e62678",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db.tables"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b80d523",
   "metadata": {},
   "source": [
    "We'll create the table again, this time with an `id` column.\n",
    "\n",
    "We use `pk=\"id\"` to specify that the `id` column should be treated as the primary key for the table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "c9ee8b9f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Table creatures (id, name, species, age)>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db[\"creatures\"].insert_all([{\n",
    "    \"id\": 1,\n",
    "    \"name\": \"Cleo\",\n",
    "    \"species\": \"dog\",\n",
    "    \"age\": 6\n",
    "}, {\n",
    "    \"id\": 2,\n",
    "    \"name\": \"Lila\",\n",
    "    \"species\": \"chicken\",\n",
    "    \"age\": 0.8,\n",
    "}, {\n",
    "    \"id\": 3,\n",
    "    \"name\": \"Bants\",\n",
    "    \"species\": \"chicken\",\n",
    "    \"age\": 0.8,\n",
    "}], pk=\"id\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "523e01ab",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CREATE TABLE [creatures] (\n",
      "   [id] INTEGER PRIMARY KEY,\n",
      "   [name] TEXT,\n",
      "   [species] TEXT,\n",
      "   [age] FLOAT\n",
      ")\n"
     ]
    }
   ],
   "source": [
    "print(table.schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "811bea70",
   "metadata": {},
   "source": [
    "## Inserting more records\n",
    "\n",
    "We can call `.insert_all()` again to insert more records. Let's add two more chickens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "716df161",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Table creatures (id, name, species, age)>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table.insert_all([{\n",
    "    \"id\": 4,\n",
    "    \"name\": \"Azi\",\n",
    "    \"species\": \"chicken\",\n",
    "    \"age\": 0.8,\n",
    "}, {\n",
    "    \"id\": 5,\n",
    "    \"name\": \"Snowy\",\n",
    "    \"species\": \"chicken\",\n",
    "    \"age\": 0.9,\n",
    "}], pk=\"id\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "4b1b2476",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'id': 1, 'name': 'Cleo', 'species': 'dog', 'age': 6.0},\n",
       " {'id': 2, 'name': 'Lila', 'species': 'chicken', 'age': 0.8},\n",
       " {'id': 3, 'name': 'Bants', 'species': 'chicken', 'age': 0.8},\n",
       " {'id': 4, 'name': 'Azi', 'species': 'chicken', 'age': 0.8},\n",
       " {'id': 5, 'name': 'Snowy', 'species': 'chicken', 'age': 0.9}]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(table.rows)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2af4ae75",
   "metadata": {},
   "source": [
    "Since the `id` column is an integer primary key, we can insert a record without specifying an ID and one will be automatically added.\n",
    "\n",
    "Since we are only adding one record we will use `.insert()` instead of `.insert_all()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "246c6dd5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Table creatures (id, name, species, age)>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table.insert({\"name\": \"Blue\", \"species\": \"chicken\", \"age\": 0.9})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7c28e4d",
   "metadata": {},
   "source": [
    "We can use `table.last_pk` to see the ID of the record we just added."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "de012e1e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table.last_pk"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c38edaf4",
   "metadata": {},
   "source": [
    "Here's the full list of rows again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "7c27075e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'id': 1, 'name': 'Cleo', 'species': 'dog', 'age': 6.0},\n",
       " {'id': 2, 'name': 'Lila', 'species': 'chicken', 'age': 0.8},\n",
       " {'id': 3, 'name': 'Bants', 'species': 'chicken', 'age': 0.8},\n",
       " {'id': 4, 'name': 'Azi', 'species': 'chicken', 'age': 0.8},\n",
       " {'id': 5, 'name': 'Snowy', 'species': 'chicken', 'age': 0.9},\n",
       " {'id': 6, 'name': 'Blue', 'species': 'chicken', 'age': 0.9}]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(table.rows)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64931bd0",
   "metadata": {},
   "source": [
    "If you try to add a new record with an existing ID, you will get an `IntegrityError`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "36327794",
   "metadata": {},
   "outputs": [
    {
     "ename": "IntegrityError",
     "evalue": "UNIQUE constraint failed: creatures.id",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mIntegrityError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-24-4222c6abc759>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mtable\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minsert\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m{\u001b[0m\u001b[0;34m\"id\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;36m6\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"name\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m\"Red\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"species\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m\"chicken\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"age\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;36m0.9\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/usr/local/Cellar/jupyterlab/3.0.16_1/libexec/lib/python3.9/site-packages/sqlite_utils/db.py\u001b[0m in \u001b[0;36minsert\u001b[0;34m(self, record, pk, foreign_keys, column_order, not_null, defaults, hash_id, alter, ignore, replace, extracts, conversions, columns)\u001b[0m\n\u001b[1;32m   2027\u001b[0m         \u001b[0mcolumns\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mDEFAULT\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2028\u001b[0m     ):\n\u001b[0;32m-> 2029\u001b[0;31m         return self.insert_all(\n\u001b[0m\u001b[1;32m   2030\u001b[0m             \u001b[0;34m[\u001b[0m\u001b[0mrecord\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2031\u001b[0m             \u001b[0mpk\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mpk\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/Cellar/jupyterlab/3.0.16_1/libexec/lib/python3.9/site-packages/sqlite_utils/db.py\u001b[0m in \u001b[0;36minsert_all\u001b[0;34m(self, records, pk, foreign_keys, column_order, not_null, defaults, batch_size, hash_id, alter, ignore, replace, truncate, extracts, conversions, columns, upsert)\u001b[0m\n\u001b[1;32m   2143\u001b[0m             \u001b[0mfirst\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2144\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2145\u001b[0;31m             self.insert_chunk(\n\u001b[0m\u001b[1;32m   2146\u001b[0m                 \u001b[0malter\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2147\u001b[0m                 \u001b[0mextracts\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/Cellar/jupyterlab/3.0.16_1/libexec/lib/python3.9/site-packages/sqlite_utils/db.py\u001b[0m in \u001b[0;36minsert_chunk\u001b[0;34m(self, alter, extracts, chunk, all_columns, hash_id, upsert, pk, conversions, num_records_processed, replace, ignore)\u001b[0m\n\u001b[1;32m   1955\u001b[0m             \u001b[0;32mfor\u001b[0m \u001b[0mquery\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparams\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mqueries_and_params\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1956\u001b[0m                 \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1957\u001b[0;31m                     \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdb\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexecute\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mquery\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparams\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1958\u001b[0m                 \u001b[0;32mexcept\u001b[0m \u001b[0mOperationalError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1959\u001b[0m                     \u001b[0;32mif\u001b[0m \u001b[0malter\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;34m\" column\"\u001b[0m \u001b[0;32min\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/Cellar/jupyterlab/3.0.16_1/libexec/lib/python3.9/site-packages/sqlite_utils/db.py\u001b[0m in \u001b[0;36mexecute\u001b[0;34m(self, sql, parameters)\u001b[0m\n\u001b[1;32m    255\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_tracer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msql\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparameters\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    256\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mparameters\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 257\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexecute\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msql\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparameters\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    258\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    259\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexecute\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msql\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mIntegrityError\u001b[0m: UNIQUE constraint failed: creatures.id"
     ]
    }
   ],
   "source": [
    "table.insert({\"id\": 6, \"name\": \"Red\", \"species\": \"chicken\", \"age\": 0.9})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2e00692f",
   "metadata": {},
   "source": [
    "You can use `replace=True` to replace the matching record with a new one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "2be75589",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Table creatures (id, name, species, age)>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table.insert({\"id\": 6, \"name\": \"Red\", \"species\": \"chicken\", \"age\": 0.9}, replace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "83281675",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'id': 1, 'name': 'Cleo', 'species': 'dog', 'age': 6.0},\n",
       " {'id': 2, 'name': 'Lila', 'species': 'chicken', 'age': 0.8},\n",
       " {'id': 3, 'name': 'Bants', 'species': 'chicken', 'age': 0.8},\n",
       " {'id': 4, 'name': 'Azi', 'species': 'chicken', 'age': 0.8},\n",
       " {'id': 5, 'name': 'Snowy', 'species': 'chicken', 'age': 0.9},\n",
       " {'id': 6, 'name': 'Red', 'species': 'chicken', 'age': 0.9}]"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(table.rows)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7122b76",
   "metadata": {},
   "source": [
    "## Updating a record\n",
    "\n",
    "We will rename that row back to `Blue`, this time using the `table.update(pk, updates)` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "43df156d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Table creatures (id, name, species, age)>"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table.update(6, {\"name\": \"Blue\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "0b8f8422",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'id': 6, 'name': 'Blue', 'species': 'chicken', 'age': 0.9}]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(db.query(\"select * from creatures where id = ?\", [6]))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58142b86",
   "metadata": {},
   "source": [
    "## Extracting one of the columns into another table\n",
    "\n",
    "Our current table has a `species` column with a string in it - let's pull that out into a separate table.\n",
    "\n",
    "We can do that using the [table.extract() method](https://sqlite-utils.datasette.io/en/stable/python-api.html#extracting-columns-into-a-separate-table)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "6ab69111",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Table creatures (id, name, species_id, age)>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table.extract(\"species\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dca327b2",
   "metadata": {},
   "source": [
    "We now have a new table called `species`, which we can see using the `db.tables` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "76e95b36",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<Table species (id, species)>, <Table creatures (id, name, species_id, age)>]"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "db.tables"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ea43bf5",
   "metadata": {},
   "source": [
    "Our creatures table has been modified - instead of a `species` column it now has `species_id` which is a foreign key to the new table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "c0438bff",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CREATE TABLE \"creatures\" (\n",
      "   [id] INTEGER PRIMARY KEY,\n",
      "   [name] TEXT,\n",
      "   [species_id] INTEGER,\n",
      "   [age] FLOAT,\n",
      "   FOREIGN KEY([species_id]) REFERENCES [species]([id])\n",
      ")\n",
      "[{'id': 1, 'name': 'Cleo', 'species_id': 1, 'age': 6.0}, {'id': 2, 'name': 'Lila', 'species_id': 2, 'age': 0.8}, {'id': 3, 'name': 'Bants', 'species_id': 2, 'age': 0.8}, {'id': 4, 'name': 'Azi', 'species_id': 2, 'age': 0.8}, {'id': 5, 'name': 'Snowy', 'species_id': 2, 'age': 0.9}, {'id': 6, 'name': 'Blue', 'species_id': 2, 'age': 0.9}]\n"
     ]
    }
   ],
   "source": [
    "print(db[\"creatures\"].schema)\n",
    "print(list(db[\"creatures\"].rows))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0452c201",
   "metadata": {},
   "source": [
    "The new `species` table has been created and populated too:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "5d38c3a8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CREATE TABLE [species] (\n",
      "   [id] INTEGER PRIMARY KEY,\n",
      "   [species] TEXT\n",
      ")\n",
      "[{'id': 1, 'species': 'dog'}, {'id': 2, 'species': 'chicken'}]\n"
     ]
    }
   ],
   "source": [
    "print(db[\"species\"].schema)\n",
    "print(list(db[\"species\"].rows))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a0312d1e",
   "metadata": {},
   "source": [
    "We can use a join SQL query to combine data from these two tables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "6734ed5d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'id': 1, 'name': 'Cleo', 'age': 6.0, 'species_id': 1, 'species': 'dog'},\n",
       " {'id': 2, 'name': 'Lila', 'age': 0.8, 'species_id': 2, 'species': 'chicken'},\n",
       " {'id': 3, 'name': 'Bants', 'age': 0.8, 'species_id': 2, 'species': 'chicken'},\n",
       " {'id': 4, 'name': 'Azi', 'age': 0.8, 'species_id': 2, 'species': 'chicken'},\n",
       " {'id': 5, 'name': 'Snowy', 'age': 0.9, 'species_id': 2, 'species': 'chicken'},\n",
       " {'id': 6, 'name': 'Blue', 'age': 0.9, 'species_id': 2, 'species': 'chicken'}]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(db.query(\"\"\"\n",
    "    select\n",
    "      creatures.id,\n",
    "      creatures.name,\n",
    "      creatures.age,\n",
    "      species.id as species_id,\n",
    "      species.species\n",
    "    from creatures\n",
    "      join species on creatures.species_id = species.id\n",
    "\"\"\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c4802ac",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}