Some metadata values cause problems when empty. For example, a markdown file
containing a Slug: line with no additional text causing Pelican to produce a
file named ".html" instead of generating a proper file name. Others, like
those created by a PATH_METADATA regex, must be preserved even if empty,
so things like PAGE_URL="filename{customvalue}.html" will always work.
Essentially, we want to discard empty metadata that we know will be useless
or problematic. This is better than raising an exception because (a) it
allows users to deliberately keep empty metadata in their source files for
filling in later, and (b) users shouldn't be forced to fix empty metadata
created by blog migration tools (see #1398).
The metadata processors are the ideal place to do this, because they know
the type of data they are handling and whether an empty value is wanted.
Unfortunately, they can't discard items, and neither can process_metadata(),
because their return values are always saved by calling code. We can't
safely change the calling code, because some of it lives in custom reader
classes out in the field, and we don't want to break those working systems.
Discarding empty values at the time of use isn't good enough, because that
still allows useless empty values in a source file to override configured
defaults.
My solution:
- When processing a list of values, a metadata processor will omit any
unwanted empty ones from the list it returns.
- When processing an entirely unwanted value, it will return something easily
identifiable that will pass through the reader code.
- When collecting the processed metadata, read_file() will filter out items
identified as unwanted.
These metadata are affected by this change:
author, authors, category, slug, status, tags.
I also removed a bit of now-superfluous code from generators.py that was
discarding empty authors at the time of use.
The old code was naively comparing the strings in PAGE_EXCLUDES to the
subdirectory names produced by os.walk(). (Same with ARTICLE_EXCLUDES.)
This had two surprising effects:
Setting PAGE_EXCLUDES=['foo'] would exclude all directories named foo,
regardless of whether they were in the top-level content directory or
nested deep within a directory whose contents should not be excluded.
Setting PAGE_EXCLUDES=['subdir/foo'] would never exclude any directories.
In other words, there is no way to exclude a subdirectory without risking
the accidental exclusion of other directories with the same name elsewhere
in the file system.
This change fixes the problem, so 'subdir/foo' and 'foo' will be distinct
and both work as expected. If anyone out there is depending on the old
behavior, they will have to update their settings. I don't expect it to
affect most users yet, since Pelican doesn't yet make nested directory
structures very useful. When it does, this fix will become important to
more people.
This change partially addresses issue #1019, by teaching Pelican to distinguish
between static files and content source files. A user can now safely add the
same directory to both STATIC_PATHS and PAGE_PATHS (or ARTICLE_PATHS). Pelican
will then process the content source files in that directory normally, and
treat the remaining files as static, without copying the raw content source
files to the output directory. (The OUTPUT_SOURCES setting still works.)
In other words, images and markdown/reST files can now safely live together.
To keep those files together in the generated site, STATIC_SAVE_AS and
PAGE_SAVE_AS (or ARTICLE_SAVE_AS) should point to the same output directory.
There are two new configuration settings:
STATIC_EXCLUDES=[] # This works just like PAGE_EXCLUDES and ARTICLE_EXCLUDES.
STATIC_EXCLUDE_SOURCES=True # Set this to False to get the old behavior.
Two small but noteworthy internal changes:
StaticGenerator now runs after all the other generators. This allows it to see
which files are meant to be processed by other generators, and avoid them.
Generators now include files that they fail to process (e.g. those with missing
mandatory metadata) along with all the other paths in context['filenames'].
This allows such files to be excluded from StaticGenerator's file list, so they
won't end up accidentally published. Since these files have no Content object,
their value in context['filenames'] is None. The code that uses that dict has
been updated accordingly.
Old system was using manual string formatting for log messages.
This caused issues with common operations like exception logging
because often they need to be handled differently for Py2/Py3
compatibility. In order to unify the effort:
- All logging is changed to `logging.level(msg, arg1, arg2)` style.
- A `SafeLogger` is implemented to auto-decode exceptions properly
in the args (ref #1403).
- Custom formatters were overriding useful logging functionality
like traceback outputing (ref #1402). They are refactored to be
more transparent. Traceback information is provided in `--debug`
mode for `read_file` errors in generators.
- Formatters will now auto-format multiline log messages in order
to make them look related. Similarly, traceback will be formatted in
the same fashion.
- `pelican.log.LimitFilter` was (ab)using logging message which
would result in awkward syntax for argumented logging style. This
functionality is moved to `extra` keyword argument.
- Levels for errors that would result skipping a file (`read_file`)
changed from `warning` to `error` in order to make them stand out
among other logs.
- Small consistency changes to log messages (i.e. changing all
to start with an uppercase letter) and quality-of-life improvements
(some log messages were dumping raw object information).
reverts getpelican/pelican@ddcccfeaa9
If one used a locale that made use of unicode characters (like fr_FR.UTF-8)
the files on disk would be in correct locale while links would be to C.
Uses a SafeDatetime class that works with unicode format strigns
by using custom strftime to prevent ascii decoding errors with Python2.
Also added unicode decoding for the calendar module to fix period
archives.
Instead of one path a list can be given. This is due to popular request.
Should help people not wanting to use Pelican for blogging.
Maintain backward compatibility though.
Thanks to @ingwinlu for pointing out the change in StaticGenerator.
This is a reworked and improved version of content caching.
Notable changes:
- by default only raw content and metadata returned by readers are
cached which should prevent conficts with plugins, the speed benefit
of content objects caching is not very big with a simple setup
- renamed --full-rebuild to --ignore-cache
- added more elaborate logging to caching code
Under python 2, with non-ascii locales, u"{:%b}".format(date) can raise UnicodeDecodeError
because u"{:%b}".format(date) will call date.__format__(u"%b"), which will return a byte string
and not a unicode string.
eg:
locale.setlocale(locale.LC_ALL, 'ja_JP.utf8')
date.__format__(u"%b") == '12\xe6\x9c\x88' # True
This commit catches UnicodeDecodeError and calls date.__format__() with byte strings instead
of characters, since it to work with character strings
invoked before categories and tags lists are created
useful when e.g. modifying the list of articles to be generated
so that removed articles are not leaked in categories or tags
* Adds period tuple of (year, month, day) matching the time
period of the current archive. Note that this is done
to the archive context if period_archives.html doesn't exist.
* Adds tests to verify this.
* Adds documentation in themes.rst about period in period_archives.html.
Previously if you tried to mark an article as a draft by using a different
casing (for example, draft) you would get a warning when building:
`Unknown status Draft for file foo.md, skipping it.` This uses a
case-insensitive comparison when looking at article status instead. I
believe this behavior is a little easier for new Pelican users.
Add a `Readers` class which contains a dict of file extensions / `Reader`
instances. This dict can be overwritten with a `READERS` settings, for instance
to avoid processing *.html files:
READERS = {'html': None}
Or to add a custom reader for the `foo` extension:
READERS = {'foo': FooReader}
This dict is no storing the Reader classes as it was done before with
`EXTENSIONS`. It stores the instances of the Reader classes to avoid instancing
for each file reading.
Make deliberate overriding (*) works with overwrites detection.
(*) first introduced by d0e9c52410
The following are decided to be deliberate override:
- articles using the `save_as` metadata
- pages using the `save_as` metadata
- template pages (always)
Pelican now exits in the following 2 cases:
- at least 2 not deliberate writes to the same file name (behaviour introduced
by the overwrite detection feature ff7410ce2a)
- at least 2 deliberate writes to the same file name (new behaviour)
Also added info logging when deliberate overrides are performed.
Switched to StandardError instead of IOError, thanks to @ametaireau and
@russkel.
