Simon Willison 31b21f5c5e Moved all SQLite queries to threads ... SQLite operations are blocking, but we're running everything in Sanic, an asyncio web framework, so blocking operations are bad - a long-running DB operation could hold up the entire server. Instead, I've moved all SQLite operations into threads. These are managed by a concurrent.futures ThreadPoolExecutor. This means I can run up to X queries in parallel, and I can continue to queue up additional incoming HTTP traffic while the threadpool is busy. Each thread is responsible for managing its own SQLite connections - one per database. These are cached in a threadlocal. Since we are working with immutable, read-only SQLite databases it should be safe to share SQLite objects across threads. On this assumption I'm using the check_same_thread=False option. Opening a database connection looks like this: conn = sqlite3.connect( 'file:filename.db?immutable=1', uri=True, check_same_thread=False, ) The following articles were helpful in figuring this out: * https://pymotw.com/3/asyncio/executors.html * https://marlinux.wordpress.com/2017/05/19/python-3-6-asyncio-sqlalchemy/ Closes #45. Refs #38.		2017-11-04 19:21:44 -07:00
..
templates	Started work on cli, which also meant adding setup.py	2017-10-27 00:08:24 -07:00
__init__.py	Started work on cli, which also meant adding setup.py	2017-10-27 00:08:24 -07:00
app.py	Moved all SQLite queries to threads	2017-11-04 19:21:44 -07:00
cli.py	Refactored everything into a factory function	2017-11-04 19:13:44 -07:00