SQLite operations are blocking, but we're running everything in Sanic, an
asyncio web framework, so blocking operations are bad - a long-running DB
operation could hold up the entire server.
Instead, I've moved all SQLite operations into threads. These are managed by a
concurrent.futures ThreadPoolExecutor. This means I can run up to X queries in
parallel, and I can continue to queue up additional incoming HTTP traffic
while the threadpool is busy.
Each thread is responsible for managing its own SQLite connections - one per
database. These are cached in a threadlocal.
Since we are working with immutable, read-only SQLite databases it should be
safe to share SQLite objects across threads. On this assumption I'm using the
check_same_thread=False option. Opening a database connection looks like this:
conn = sqlite3.connect(
'file:filename.db?immutable=1',
uri=True,
check_same_thread=False,
)
The following articles were helpful in figuring this out:
* https://pymotw.com/3/asyncio/executors.html
* https://marlinux.wordpress.com/2017/05/19/python-3-6-asyncio-sqlalchemy/Closes#45. Refs #38.
I now call a factory function to construct the Sanic app:
app = app_factory(files)
This allows me to pass additional arguments to it, e.g. the files to serve.
Also refactored my class-based views to accept jinja as an argument, e.g:
app.add_route(
TableView.as_view(jinja),
'/<db_name:[^/]+>/<table:[^/]+?><as_json:(.jsono?)?$>'
)
I'm using click, and click recommends using a setup.py - so I've added one of
those. I also refactored code into a new datasite package. It's not quite
deploying to now properly at the moment though - I seem to have messed up the
path handling a bit.
Also snuck in a new template for the "Row" view.
Refs #40
