github/datasette
1
0
Fork 0
mirror of https://github.com/simonw/datasette.git synced 2025-12-10 16:51:24 +01:00
Code Issues Projects Releases Packages Wiki Activity
490 commits 143 branches 163 tags 44 MiB
Commit graph

144 commits

Author SHA1 Message Date
Simon Willison
97ae66ccab
 		404s ending in slash redirect to remove that slash, closes #309 2018-06-21 08:13:07 -07:00
Simon Willison
3b53eea382
 		Fixed incorrect display of compound primary keys with foreign key references 
Closes #319
 2018-06-21 07:56:28 -07:00
Simon Willison
3683a6b626
 		Docs + example of canned SQL query using || concatenation 
Closes #321
 2018-06-20 21:54:36 -07:00
Simon Willison
aeeb50f61b
 		Correctly display facets with value of 0 - fixes #318 
Also added comprehensive unit test for facet display HTML.
 2018-06-20 21:30:13 -07:00
Simon Willison
83f4ef7ec7
 		Improved UI for CSV/JSON export, closes #266 2018-06-17 23:05:18 -07:00
Simon Willison
fc3660cfad
 		Streaming mode for downloading all rows as a CSV (#315) 
* table.csv?_stream=1 to download all rows - refs #266

This option causes Datasette to serve ALL rows in the table, by internally
following the _next= pagination links and serving everything out as a stream.

Also added new config option, allow_csv_stream, which can be used to disable
this feature.

* New config option max_csv_mb limiting size of CSV export
 2018-06-17 20:21:02 -07:00
Simon Willison
0d7ba1ba67
 		Default to _labels=on on JSON/CSV links with foreign keys, refs #266 2018-06-17 15:56:55 -07:00
Simon Willison
db1e6bc182
 		--version-note for datasette, datasette publish and datasette package 
This is a relatively obscure new command-line argument that helps solve the
problem of showing accurate version information in deployed instances of
Datasette even if they were deployed directly from source code.

You can pass --version-note to datasette publish and package and it will then
in turn be passed to datasette when it starts:

    datasette --version-note=hello fixtures.db

Now if you visit /-/versions.json you will see this:

    {
        "datasette": {
            "note": "hello",
            "version": "0+unknown"
        },
        "python": {
            "full": "3.6.5 (default, Jun  6 2018, 19:19:24) \n[GCC 6.3.0 20170516]",
            "version": "3.6.5"
        },
        ...
    }

I plan to use this in some Travis CI configuration, refs #313
 2018-06-17 14:19:39 -07:00
Simon Willison
6a32df334d
 		Improved fixtures to support publication 
The fixtures database created by our unit tests makes for a good "live" demo
of Datasette in action.

I've improved the metadata it ships with to better support this use-case.

I've also improved the mechanism for writing out fixtures: you can do this:

	python tests/fixtures.py fixtures.db

To get just the fixtures database written out... or you can do this:

	python tests/fixtures.py fixtures.db fixtures.json

To get metadata which you can then serve like so:

	datasette fixtures.db -m fixtures.json

Refs #313
 2018-06-17 12:16:04 -07:00
Simon Willison
0357774c94
 		Renamed test_tables.db to fixtures.db in unit tests 2018-06-17 11:34:16 -07:00
Simon Willison
ed631e690b
 		?_labels= and ?_label=COL to expand foreign keys in JSON/CSV 
These new querystring arguments can be used to request expanded foreign keys
in both JSON and CSV formats.

?_labels=on turns on expansions for ALL foreign key columns

?_label=COLUMN1&_label=COLUMN2 can be used to pick specific columns to expand

e.g. `Street_Tree_List.json?_label=qSpecies&_label=qLegalStatus`

    {
        "rowid": 233,
        "TreeID": 121240,
        "qLegalStatus": {
            "value" 2,
            "label": "Private"
        }
        "qSpecies": {
            "value": 16,
            "label": "Sycamore"
        }
        "qAddress": "91 Commonwealth Ave",
        ...
    }

The labels option also works for the HTML and CSV views.

HTML defaults to `?_labels=on`, so if you pass `?_labels=off` you can disable
foreign key expansion entirely - or you can use `?_label=COLUMN` to request
just specific columns.

If you expand labels on CSV you get additional columns in the output:

`/Street_Tree_List.csv?_label=qLegalStatus`

    rowid,TreeID,qLegalStatus,qLegalStatus_label...
    1,141565,1,Permitted Site...
    2,232565,2,Undocumented...

I also refactored the existing foreign key expansion code.

Closes #233. Refs #266.
 2018-06-16 15:18:57 -07:00
Simon Willison
5bda4a477c
 		Fixed CSV tests - Python 3.6.5 and 3.6.3 apparently differ 
The test used to expect CSV to come back like this:

	hello
	world
	""

With the final blank value encoded in quotes.

Judging by Travis failures, this behaviour changed between Python 3.6.3 and
3.6.5:

	https://travis-ci.org/simonw/datasette/jobs/392586661
 2018-06-15 00:01:48 -07:00
Simon Willison
3a79ad98ea
 		Basic CSV export, refs #266 
Tables and custom SQL query results can now be exported as CSV.

The easiest way to do this is to use the .csv extension, e.g.

	/test_tables/facet_cities.csv

By default this is served as Content-Type: text/plain so you can see it in
your browser. If you want to download the file (using text/csv and with an
appropriate Content-Disposition: attachment header) you can do so like this:

	/test_tables/facet_cities.csv?_dl=1

We link to the CSV and downloadable CSV URLs from the table and query pages.

The links use ?_size=max and so by default will return 1,000 rows.

Also fixes #303 - table names ending in .json or .csv are now detected and
URLs are generated that look like this instead:

	/test_tables/table%2Fwith%2Fslashes.csv?_format=csv

The ?_format= option is available for everything else too, but we link to the
.csv / .json versions in most cases because they are aesthetically pleasing.
 2018-06-14 23:51:23 -07:00
Simon Willison
864328e074
 		Test client wrapper removing need for gather_request - refs #272 
As part of decoupling from Sanic, this will make it easier to run tests
against ASGI instead.
 2018-06-04 20:53:12 -07:00
Simon Willison
29edbe90ff
 		New cache_size_kb config for SQLite, closes #304 2018-06-04 09:02:07 -07:00
Simon Willison
9277c6236a
 		Fixed tests I broke in b18e45158 2018-05-31 07:47:22 -07:00
Simon Willison
969771770f
 		Upgrade pytest to 3.6.0 
https://github.com/pytest-dev/pytest/issues/1875 made it impossible to declare
a function as a fixture multiple times, which we were doing across different
modules. The fix was to move our @pytest.fixture calls into decorators in the
tests/fixtures.py module.
 2018-05-31 06:42:37 -07:00
Simon Willison
1a8b0213e0
 		Fixed broken test introduced in b0a95da96 2018-05-29 19:38:21 -07:00
Simon Willison
b0a95da963
 		Show more useful error message for SQL interrupted, closes #142 2018-05-28 14:24:19 -07:00
Simon Willison
98c8f0e728
 		Moved plugin HTML tests from test_html to test_plugins 2018-05-28 14:23:48 -07:00
Simon Willison
27c10f6482
 		Missing half of last commit fefb0db8ae 2018-05-28 13:42:48 -07:00
Simon Willison
fefb0db8ae
 		Unit test for 02870e57, closes #291 2018-05-28 13:41:53 -07:00
Simon Willison
a4d6acc239
 		Use scope='session' for all fixtures 
This means they will only be executed once which makes sense
since the database they create is immutable.
 2018-05-28 13:41:06 -07:00
Simon Willison
76d11eb768
 		New ?_json=colname argument for returning unescaped JSON 
Also extracted docs for special JSON arguments into a new section.

Closes #31
 2018-05-28 11:08:39 -07:00
Simon Willison
7944a8b0de
 		Added num_sql_threads config option, closes #285 2018-05-26 17:43:22 -07:00
Simon Willison
276913b748
 		?_shape=arrayfirst, closes #287 2018-05-26 17:32:15 -07:00
Simon Willison
b463f60158
 		?_ttl= parameter and default_cache_ttl config 
Refs #285, Closes #289
 2018-05-26 15:17:33 -07:00
Simon Willison
a6afc21aba
 		Test that ensures all config options are documented 2018-05-26 15:11:16 -07:00
Simon Willison
f722b0a730
 		allow_sql config option to disable custom SQL, closes #284 2018-05-24 22:50:50 -07:00
Simon Willison
50920cfe3d
 		allow_facet, allow_download, suggest_facets boolean --config 
Refs #284
 2018-05-24 18:12:27 -07:00
Simon Willison
2bfd111d65
 		Faceting no longer breaks pagination, fixes #282 2018-05-23 06:41:14 -07:00
Simon Willison
f6183ff5fa
 		Renamed --limit to --config, added --help-config, closes #274 
Removed the --page_size= argument to datasette serve in favour of:

    datasette serve --config default_page_size:50 mydb.db

Added new help section:

    $ datasette --help-config
    Config options:
      default_page_size            Default page size for the table view
                                   (default=100)
      max_returned_rows            Maximum rows that can be returned from a table
                                   or custom query (default=1000)
      sql_time_limit_ms            Time limit for a SQL query in milliseconds
                                   (default=1000)
      default_facet_size           Number of values to return for requested facets
                                   (default=30)
      facet_time_limit_ms          Time limit for calculating a requested facet
                                   (default=200)
      facet_suggest_time_limit_ms  Time limit for calculating a suggested facet
                                   (default=50)
 2018-05-20 10:01:49 -07:00
Simon Willison
39426ff0e4
 		Added /-/limits and /-/limits.json, closes #270 2018-05-17 23:16:28 -07:00
Simon Willison
cef9a9a870
 		--limit= mechanism plus new limits for facets 
Replaced the --max_returned_rows and --sql_time_limit_ms options to
"datasette serve" with a new --limit option, which supports a larger
list of limits.

Example usage:

	datasette serve --limit max_returned_rows:1000 \
		--limit sql_time_limit_ms:2500 \
		--limit default_facet_size:50 \
		--limit facet_time_limit_ms:1000 \
		--limit facet_suggest_time_limit_ms:500

New docs: https://datasette.readthedocs.io/en/latest/limits.html

Closes #270
Closes #264
 2018-05-17 22:08:26 -07:00
Simon Willison
b263da78e3 Reliable sort order for facets in Python 3.5, fixing test - refs #255 2018-05-16 08:27:24 -07:00
Simon Willison
6d12580ed7 Foreign key facets are now expanded to labels, refs #255 2018-05-16 08:27:24 -07:00
Simon Willison
f4943ca89b _facet selections persist through table form, refs #255 2018-05-16 08:27:24 -07:00
Simon Willison
1dc94f6eaa Facets can now be toggled off again, refs #255 2018-05-16 08:27:24 -07:00
Simon Willison
142a550a99 Facet results now have "truncated" field 
To indicate if there was more than 20 distinct values. Refs #255
 2018-05-16 08:27:24 -07:00
Simon Willison
a82175276c _sort/_next links now use new path_with_replaced_args method 2018-05-16 08:27:24 -07:00
Simon Willison
8a4ed052a5 path_with_added_args now works with multiple existing args 2018-05-16 08:27:24 -07:00
Simon Willison
f36c9d4b4c Facet toggling now works for integer columns, refs #255 2018-05-16 08:27:24 -07:00
Simon Willison
de05cf21aa Facet "selected" key and toggle_url now toggles, refs #255 2018-05-16 08:27:24 -07:00
Simon Willison
2b79f2bdeb path_with_added_args now preserves order in Python 3.5 2018-05-14 00:04:23 -03:00
Simon Willison
4301a8f3ac
 		Case insensitive querystring comparison, fix Python 3.5 2018-05-12 19:49:37 -03:00
Simon Willison
eaaa3ea149
 		Initial implementation of facets, plus tests and docs 
Refs #255
 2018-05-12 19:29:41 -03:00
Simon Willison
70ff615f1b
 		utils.path_with_added_args() improvements 
* Now covered by unit tests
* Preserves original order
* Can handle multiple args of the same name, e.g. ?bar=1&bar=2
 2018-05-12 18:44:09 -03:00
Simon Willison
04d21ccd08
 		/-/versions now includes SQLite fts_versions, closes #252 2018-05-11 10:19:25 -03:00
Simon Willison
801381b765
 		Default tests to using a longer timelimit 
Every now and then a test will fail in Travis CI on Python 3.5 because it hit
the default 20ms SQL time limit.

Test fixtures now default to a 200ms time limit, and we only use the 20ms time
limit for the specific test that tests query interruption. This should make
our tests on Python 3.5 in Travis much more stable.
 2018-05-05 19:44:32 -03:00
Simon Willison
1259b8ac0b
 		Support _search_COLUMN=text searches, closes #237 2018-05-05 19:33:08 -03:00