Beastie is an experiment in parsing and processing BibTeX files.
Home page: https://purl.org/nxg/dist/beastie
Beastie is:
.bib files, amongst others..bst BibTeX Style Language, using that framework.The goals of this project are:
.bib files which aren't exclusively based on .bst;The first and second of those points aim to get away from BibTeX being a single-implementation system, where the only thing that can reliably process a .bib file is the bibtex program. This isn't quite the case at present, but combined with the third documentation point, it's quite hard to process BibTeX input in a confident way.
The .bst language counts as rather niche. It has substantial amusement value, but expertise could hardly be said to be widespread (of course, one could complain that Scheme has some niche aspects, but it also has its numerous compensating virtues).
The implementation of the .bst language is intended to be fairly complete, in the sense that it is nearly a drop-in replacement for the bibtex program (as described in the ‘Writing BibTeX styles’ document, ‘btxhak.pdf’, on the bibtex package page). Some of the differences from BibTeX will be bugs, but others are deliberate mild deviations, as discussed below.
Beastie also provides a framework for writing alternative bibliography tools. The ‘experimental’ status means that it is not yet fully-featured in that respect. The fact that it implements the .bst language confirms its scope, and acts as some indicator of ‘correctness’, and the examples/ directory in the distribution illustrates how one would go about writing a from-scratch style file, but there isn't (yet) a drop-in Scheme-based alternative to, for example, plain.bst.
It is a goal of Beastie to go slightly beyond BibTeX, and to make it a little easier to use .bib files to generate HTML or other formats, or for the output of beastie to be ingested into other processing pipelines which are not based on (La)TeX. This has informed some of the choices made.
Another limitation of BibTeX, and its ecosystem, is that, although it provides fairly comprehensive support for STEM-style bibliographies, it starts to struggle with citations much outside this area. For humanities or legal bibliographies, the most active support currently comes from BibLaTeX, possibly because the intricacy of the implied support requires a more approachable language than .bst.
It's also a goal of this project to produce a v1.0 at some point.
.bst BibTeX style language..bib, .bst, .aux. and Markdown parser/converters (and .json because why not?)..bst language..bib files promptly expands a selection of {\"u}-style escapes into the corresponding characters (this makes it easier to use this tool for generating HTML and other non-LaTeX bibliographies).There are one or two edge-cases where the btxhak document doesn't make it fully clear what counts as ‘correct behaviour’. There are also some places where the implementation deliberately extends or mildly deviates from BibTeX behaviour – see below.
The .bib parsing doesn't attempt to round-trip the file contents: inter-entry text is discarded when parsing, and {\"u} escapes are expanded promptly. This isn't so much a limitation, as a restriction of scope.
The Markdown support is fairly basic – it's a side-project here. It implements most of the Gruber spec, but not all, and in particular it doesn't know anything about HTML content.
Although the project's goal is to make it easier to write bibliography processors, the provision within Beastie is still at the level of adding parsing primitives, and not much at a higher level. That is, the distributed example scripts contain much that would probably become ‘library’ code in a mature state.
As far as the .bst language is concerned, it aims to support it quite fully, in the sense that it will produce the same results as BibTeX, for the standard styles. However it doesn't support it slavishly, and there are some deviations noted below. It also aims to slightly improve .bst error reporting, and adds a couple of extra .bst builtin functions, in both cases with the aim of being useful for debugging or developing .bst files (eg, show.stack$$ and printf$$).
Some of these points are discussed more extensively below.
Beastie is still a development project (though there is a firm determination to get to a version 1.0 before too long!). Some acknowledged gaps are:
Beastie contains support for Unicode both using the Unicode consortium's ICU library, and also using a local library for parsing and serialising Unicode. The main limitation of the latter is that is has only a very naive collation algorithm. This is unlikely to change, however, since it's mostly regarded as a fallback. The support for locales has not been thoroughly tested; bug reports are welcome.
The documentation is still somewhat ragged. There's quite a lot of it, but it's grown rather organically and could do with being rationalised.
Longer-term developments (that is, things which might wait until after a v1.0):
plain.bst in Scheme, as a proof-of-concept?Most of this is implemented in an embedded copy of s7, from ccrma.stanford.edu. Thanks to Bill Schottstaedt for creating s7, and for advice on some of its intricacies.
Thanks to Karl Berry and Paulo Ney for comments, and for functionality suggestions.
There is a manpage.
.aux files using .bstIf you call beastie with
% beastie foo
then it will parse the aux file foo.aux, looking for \citation, \bibdata and \bibstyle commands, just like BibTeX, and process the bibliography with the indicated .bst program. That is, this is (still tentatively) intended to be a drop-in alternative to bibtex, though I say this with some diffidence, since it hasn't really been pushed around much. There are some deviations from the authoritative BibTeX behaviour, noted below.
If you call beastie with
% beastie foo.scm [args...]
then the program foo.scm is interpreted as Scheme, extended with the functions described below. My assertion is that this is a more enjoyable way of generating bibliographies than the standard .bst language.
That is documented elsewhere.
Any args... are available in the program in the global variable *command-line*; there is a getopt function which may be useful if you are writing a script.
If you invoke beastie with the -r option, or if you make repl and invoke the program repl, then you're dropped into a basic REPL.
While beastie still has a version number less than 1.0, everything here should be regarded as provisional.
If invoked with the name of a .bib file,
% beastie foo.bib
then the given file will be parsed and re-written as a .bib file. If you supply the -O option, then you can select an output format; if this format is bib, json, or sexp, the output will be in the corresponding format.
You can do the same thing with beastie -I bib <foo.bib.
The program also contains a basic Markdown parser. This was because it was sort-of easy to do, and convenient for documenting the program. Its ambitions are modest – it is intended to implement the core of the format, but not to be a comprehensive implementation.
You can experiment with the Markdown support with
% beastie foo.md
The default output is in the form of ‘x-expressions’ – XML represented as scheme s-expressions, which is easy to wrangle (see the xexpr-* functions below). The output options are:
-O sxml (default) : x-expressions.-O xml and -O xhtml : output the result as XML – in the former case, this is a single (div ...) expression; in the latter this is wrapped so that it forms a complete XHTML document (the distinction in the naming of these two outputs is somewhat arbitrary).-O python : something easily ingestible with Python (specifically listxml).You can do the same thing with beastie -I md <foo.md.
The implementation covers a decent fraction of the Gruber spec; the only significant omission is that it currently knows nothing about inline HTML.
There is experimental built-in support for references (perhaps ‘illustrative’ support would be better), in the style of RMarkdown. Specifically, if a Markdown document contains references marked up like [@foo], then the multiple values returned from parse-markdown-file/metadata will be a parse-tree and a collection of ‘metadata’. From that metadata, you can extract (with (metadata/type 'citation)) a list of citations and the place in the parse-tree where they appear. The latter are the (cite "foo") elements in the parse-tree, which can therefore be edited retrospectively. A way of using this is illustrated in examples/bibliography-in-markdown.scm.
There is an alternative illustration of how one might support citations in examples/bibliography-in-markdown-alt.scm; see the comments at the top of the file for details.
Similarly, (metadata/type 'annotation) will produce a list of ("key" "value") lists containing annotations from the input Markdown file. The annotations are of the form
--- key: value
(ie, something resembling a YAML hash; this is the only YAML which is recognised). It happens that
Text --- key: value
will also be recognised as metadata, rather than a section heading. So don't write that.
Beastie is not intended to be a complete and exact replacement for BibTeX, and certainly not ‘bug-compatible’. It should be broadly compatible with BibTeX in ‘normal’ use (for most values of ‘normal’), but may treat edge cases differently.
These differences are discussed on the dislocations page.
There is a discussion of BibTeX (and some lamentations) in
@article{beebe93,
author = {Nelson Beebe},
title = {Bibliography Prettyprinting and Syntax Checking},
journal = {TUGboat},
year = {1993},
volume = {14},
number = {4},
pages = {395--419},
month = dec,
url = {https://tug.org/TUGboat/tb14-4/tb41beebe.pdf}
}I agree with most of the observations in Beebe's paper, and have not aimed to duplicate his collection of BibTeX formatting and validation bibliography tools. Although beastie can effectively do some normalisation of an input .bib file (with beastie -O bib foo.bib), this is intended to be a convenience, rather than a cleaning operation, as such, or a prettyprinting one. I would expect that beastie would have no problem parsing anything bibclean emits, and failing to do so would be regarded as a bug in beastie.
This article also contains useful (and otherwise very hard to find) information on the ‘Scribe’ format, which BibTeX effectively inherits. I have not attempted to support any part of the Scribe format which goes beyond what I have seen as ‘typical’ of BibTeX format files, and have felt free to be rather opinionated about what counts as ‘typical’ there. Beebe notes that both the lexical content, and the grammar, of .bib files are under-specified (to put it politely), and makes some suggestions.
He suggests [A-Za-z][-A-Za-zO-9:.+/']* as a pattern for names (keys and field names). I have seen _ and & in .bib files, and ADS routinely has keys starting with years, so these seem reasonable additions, to me. I'm willing to be slightly bolder than Beebe, and forbid the single-quote, ', as a name (it's just asking for trouble). I've therefore gone with [0-9]*[A-Za-z][-A-Za-z0-9:.+/_&]* as the name pattern in beastie.
Beastie supports %-comments: characters from % to the end of line are regarded as whitespace (the document suggests a slightly more complicated rule, including gobbling leading whitespace on the folowing line, but I don't fully understand the suggestion there, and believe that what I have implemented here implies the rule there, since beastie doesn't aim to make its parse round-trippable).
And beastie supports the suggested @include form.
There are a number of tests in the bibclean documentation, some of which have been imported, in adapted form, into the tests in test/. But a large fraction of these are testing the various normalisations that bibclean performs, which beastie doesn't attempt.
There are a number of miscellaneous BibTeX utilities at CTAN. Some of these overlap with Beastie.
bibtex8 is a version of bibtex which is 8-bit clean and has larger internal capacities (and which is sometimes the actual program invoked asbibtex). This can use a range of national character sets.bibtexu, is a version of bibtex8 which is Unicode-aware, in that it can do Unicode-informed case-changing and sorting..bib databases, and a binary which uses the library.biblatex relies on either BibTeX or biber to parse the bibliography, and uses TeX as the formatting language for bibliographies. Separately from the implementation details, BibLaTeX defines a substantially enlarged repertoire of .bib entry types and fields: such an expansion in expressivity is valuable, but few of them have been implemented in .bst style files.btparse) for parsing .bib files, along with a discussion of their syntax..bib parser in Python;.bib parser, which also includes a reference grammar for .bib files;.bib file, and supports an attractively compact templating language for describing bibliograpy formats.