mirror of
https://github.com/simonw/datasette.git
synced 2025-12-10 16:51:24 +01:00
An open source multi-tool for exploring and publishing data
https://datasette.io
- Python 88.4%
- HTML 7.5%
- JavaScript 2.4%
- CSS 1.2%
- Shell 0.3%
- Other 0.1%
31b21f5c5e
SQLite operations are blocking, but we're running everything in Sanic, an
asyncio web framework, so blocking operations are bad - a long-running DB
operation could hold up the entire server.
Instead, I've moved all SQLite operations into threads. These are managed by a
concurrent.futures ThreadPoolExecutor. This means I can run up to X queries in
parallel, and I can continue to queue up additional incoming HTTP traffic
while the threadpool is busy.
Each thread is responsible for managing its own SQLite connections - one per
database. These are cached in a threadlocal.
Since we are working with immutable, read-only SQLite databases it should be
safe to share SQLite objects across threads. On this assumption I'm using the
check_same_thread=False option. Opening a database connection looks like this:
conn = sqlite3.connect(
'file:filename.db?immutable=1',
uri=True,
check_same_thread=False,
)
The following articles were helpful in figuring this out:
* https://pymotw.com/3/asyncio/executors.html
* https://marlinux.wordpress.com/2017/05/19/python-3-6-asyncio-sqlalchemy/
Closes #45. Refs #38.
|
||
|---|---|---|
| datasite | ||
| .gitignore | ||
| .travis.yml | ||
| Dockerfile | ||
| LICENSE | ||
| setup.cfg | ||
| setup.py | ||
| test_helpers.py | ||