Beastie is not intended to be a complete and exact replacement for BibTeX, and certainly not ‘bug-compatible’. It should be broadly compatible with BibTeX in ‘normal’ use (for most values of ‘normal’), but may treat edge cases differently.
In particular, there is a (2007) list of ‘BibTeX Tips and FAQ’ at the CTAN package page, which describes a fairly arcane set of tricks and workarounds to deal with the wide variation in names across the world. Some of these are very specific to the way that bibtex-the-program implements its (sometimes rather ill-defined) rules for parsing names, and few of them are situations authors are likely to encounter frequently. If Beastie behaves differently from BibTeX in similar cases, I do not intend to immediately class that as a bug.
.bib filesAll the files that Beastie reads are expected to be UTF-8 encoded (supporting UTF-16 and UTF-32 may come in future; there is already some support for the former internally).
beastie supports only a subset of the Scribe syntax. Following the Beebe document cited on the main page, the only concession to Scribe is supporting (...) as the entry delimiters.
Names – that is, the names of field names and string references – have a slightly more restricted syntax than BibTeX permits. See below.
That said, the contents of entry keys are are broader than BibTeX suggests (and broader than Beebe recommends). Myself, I think it's probably not a great idea to go far beyond Beebe's pattern for citation keys, as this seems likely to be storing up problems for downstream processing. But bibtex8 – the 8-bit version of BibTeX which is the one distributed in practice – is happy to accommodate a large range of bytes in citation keys, so this tells us (a) that this seems not to cause actual downstream problems in LaTeX workflows, and (b) that Beastie should do something similar.
Thus Beastie will (currently) accept entry keys containing ‘word characters’, digits, symbols or characters in the set [:./_&-] (specifically, it excludes ASCII characters below and including space (0x20), plus 0x7f, and the characters ["%,={}]; if you're somehow using any of these in a citation key, then... I have absolutely no sympathy: stop it!). The definition of a ‘word character’ here matches the set of characters one might expect to see in a ‘word’, defined in a script-independent way.
There is no case-folding for entry keys.
We accept within field names word characters, digits and a few punctuation characters, namely [-._:]. The selection is pragmatic rather than particularly principled, and is, again, intended to be sympathetic to downstream processors.
Beastie permits multiple crossref redirections, but still demands that the cross-references refer only to later entries. In consequence it will (like BibTeX) cope with circular cross-references by reporting a backward-pointing reference as ‘missing’. I may change this in future, to support backward references.
BibTeX handles such constructions such as {\"o} specially, in that it regards such a brace-level-1 construction as constituting a single character for counting purposes. If Beastie encounters such a construction, and if the replacement of the command is a single character within the braces (in this case ‘ö’), then the braced sequence is replaced by that character at parse time. Sequences Beastie doesn't recognise, or braced sequences containing more than one character, are left untouched.
Beastie accepts a larger set of such characters than (I think) BibTeX does, and also accepts some characters such as {\ss}. Specifically (and for the sake of a goal) we support the TeX commands for all of the characters in ISO-8859-1, and a few in ISO-8859-2, plus all of the letter characters in Unicode Latin-1 Supplement and most of the letters in the Latin Extended-A range, with a few exceptions where the LaTeX command is hard to find, which are therefore unlikely to be used in practice. The internal table of replacements is fixed, and is not currently customisable.
Btxdoc says these command sequences are recognised only at brace-level one, but beastie recognises them at all brace levels. This happens during the parsing of the .bib file.
We accept Unicode in field values, with some limitations. Specifically, we accept only UTF-8 input, and we can't recognise BibTeX ‘von’ particles which start with a letter which isn't ASCII-lowercase (but I've never seen that in a bib file, in practice).
We recognise %-comments, which ignore everything from the % to the end of line. These have always been permitted (and are occasionally seen) between entries in BibTeX files, but their content is ignored because BibTeX ignores all content between entries, rather than treating these characters specially. In beastie, such comments can appear anywhere whitespace can, thus within entries as well as outwith them, and, unlike BibTeX, a %-comment will hide a @entry on the same line.
Beastie supports the @include{...} form proposed by Beebe. The content of the braces is the name of a BibTeX database, including the .bib extension, which is looked up using kpsewhich. The indicated file is read at the point where this form appears, as if by direct inclusion (though it must be a valid .bib file as it stands, and cannot break an entry across different files). Thus, any @string definitions in a .bib file are visible in any file it subsequently includes, and are visible, after the @include form, in any file which includes it. Included files can include other files in turn, and beastie should detect if there is an implied inclusion loop. An alternative design is an @import form, which imports bibliographic databases while carefully isolating @string definitions to a single file: this is tidy, but since a reasonable use of this @include mechainism is to pull in sets of @string definitions, this seems likely to be less useful in practice.
If the argument of @include is a key which has been defined with @string, then the key is expanded to its value.
Beastie has a slightly narrower definition of legal entry and citation keys, than BibTeX has. BibTeX appears to accept just about any characters (or even any bytes) here that aren't syntactically special to that program. That's agreeably laid-back, but not perhaps very friendly to parser-writers. It also meant that, if people used non-ASCII characters here, the keys would potentially have different values depending on the assumed coding of the .bib file. Beastie restricts the keys here a little more than Nelson Beebe's proposal: the goal is to accept most of what I've seen in the wild, as it were, while ruling out characters that seem to be asking for trouble. These prescriptions explicitly include sets of Unicode codepoints.
N, overlaps with ASCII [0-9])S, overlaps with ASCII [$+<=>^|])[:./_&-] (which Unicode regards as ‘punctuation’)-The ‘word characters’ here are those characters that may conventionally be regarded as part of a word. There isn't a formal definition of this set in Unicode, but a conventional definition is Unicode ‘alphabetic’ characters, plus diacritics, extender characters and join-control characters (characters, that is, which possess the Unicode properties Alphabetic (which overlaps with ASCII [A-Za-z]), Diacritic, Extender and Join_Control – see the Unicode Character Database (UCD) for intricate details). The Beastie uchar-alphabetic? and uchar-word-character? predicates test for these.
Beastie currently restricts each of these predicates to the Base Multiligual Plane (BMP) only (ie, codepoints up to U+FFFF); non-BMP characters are fine in strings.
input: opt_interentry_text list_of_stanzas opt_interentry_text
list_of_stanzas: stanza
| list_of_stanzas opt_interentry_text stanza
opt_interentry_text: /* empty */
| opt_interentry_text INTERENTRYTEXT
stanza: entry
| atpreamble | atstring
| atcomment | atinclude
entry: ENTRYTYPE '{' NAME \
',' list_of_fields '}'
| ENTRYTYPE '{' NAME '}'
| ENTRYTYPE '{' NAME ',' '}'
/* at most one trailing comma */
list_of_fields: field
| list_of_fields ',' field
| list_of_fields ',' /* ditto */
field: NAME '=' string
atpreamble: ATPREAMBLE '{' string '}'
atstring: ATSTRING '{' NAME '=' string '}'
atinclude: ATINCLUDE '{' NAME '}'
atcomment: ATCOMMENT string
string: STRINGVALUE
| NAME
| string '#' STRINGVALUE
| string '#' NAMEThe terminals in this grammar are ENTRYTYPE, representing @article and the like; terminals representing the standard @preamble, @string and @comment forms, and a further terminal representing the extension @include form (all case-insensitive); NAME represents a entry key or field, STRINGVALUE a field value "..." or {...}, and INTERENTRYTEXT any other text between entries.
In fact, Beastie uses a slightly different (non-yacc) grammar framework to do its work, but it should be equivalent to this one. The lexer produces terminals ',', '=', '#', '{' and '}' after having stripped surrounding whitespace, and the latter two can be produced by the corresponding characters, or by '(' or ')', in a modest concession to Scribe's continuing legacy in extant .bib files.
The ‘standard’ here is the btxhak.pdf document at CTAN. Unfortunately, that document is a little vague in places. Nicolas Markey's Tame the BeaST document is an account of the whole BibTeX ecosystem, including a description of the .bst language.
BibTeX, with format-string {f.~}{ll} turns ‘Jean-Paul Sartre’ into J.-P. Sartre, and so does beastie. But BibTeX puts a non-default separator in here, too, so that {f{.}} turns ‘Jean-Paul Sartre’ into J.P, which, losing the hyphen, is surely wrong (or, if not wrong in some absolute sense, I'd hope to have an argument with a copy-editor about it, rather than Oren Patashnik). So I'm not going to follow BibTeX there. Should I completely ignore a non-default separator here, or perhaps re-insert the hyphen (so for example {f{x}} would produce Jx-P)?
The only text covering this in btxdoc is Sect.2.1, item 5, which illustrates this by saying ‘and if you’re using the abbrv style [which uses "{f.~}{vv}{ll}"], then the result is “J.-P. Sartre”’, but doesn't say what's supposed to happen if you're using a different f-format.
If substring$ is given a value for the start parameter, the modulus of which is greater than the length of the string, then substring$ returns "" without error. This isn't mentioned in the documentation.
Trivia: The prime-number-theorem bib entry in the torture test at https://mirror.ctan.org/biblio/bibtex/base/btxdoc.bib cites Charles Louis Xavier Joseph de la Vall{\'e}e Poussin, but (a) if Wikipedia is to be believed, it appears that he's Joseph-Xavier rather than the other way around, and (b) it's possible that he's being confused with his son, Charles-Jean, the mathematician responsible for the prime number theorem.
Beastie parses ‘Ruiz y Picasso, Pablo’ to get the surname ‘Ruiz y Picasso’. BibTeX takes the Last name to be ‘Picasso’ with von-part ‘Ruiz y’, which I think is (linguistically) wrong (and not specified in btxdoc). In any case, the Correct Behaviour here is probably highly language-sensitive, so either we should worry about it a lot, or not worry about it at all.
The BibTeX documentation notes that multi-word surnames – quoting the example of Per Brinch Hansen – need to be made explicit using the comma-form. Though it doesn't say so, it seems obvious that Per Brinch~Hansen should be parsed in a compatible way (here, as a single-token surname).
ADS formats Lidia van Driel-Gesztelyi's name as "{Driel-Gesztelyi}, Lidia van". I'm not sure this is entirely correct, but we should aim not to fail in this case, and should probably produce the same result as BibTeX, which parses this as surname ‘Driel-Gesztelyi’ and forenames ‘Lidia van’. This is a separate question from what the analysis should be here, in language terms rather than BibTeX ones, since the surname here is a hyphenated combination of a Dutch name with a von-particle, and a Hungarian name without. This may also be an attempt to respect different languages' name-ordering practices, but that's a bog we should probably avoid wandering into at this stage.
‘Myles na gCopaleen’ is hard, not least because I don't know what the correct analysis should be, in language terms. I understand that 'na' is a genitive particle, therefore similar to ‘von’, and to unassimilated 'mac/nic' in gaelic names, and the small ‘g’ (an eclipsis!) is a required spelling change to the noun ‘copaleen’ (though this particular name is... complicated, and its pseudonymous possessor spelled it more than one way). In BibTeX terms, however (which are here the only ones that matter), bibtex parses this name as having ‘surname’ ‘gCopaleen’, and ‘Myles na gCopaleen Smith’ as having surname ‘Smith’ and von-parts ‘na gCopaleen’. The key thing from BibTeX's point of view, it seems, is that there is always a Last name, even if it looks like a von-particle, starting with a lowercase letter.
This also means that, if someone styles their name as ‘Fred ffoulkes’, we shouldn't second-guess them.
There are few cases where the nominal case of a macro matters, and certainly none mentioned in the btxdoc documentation, but I feel that something like Finn \mc Cool (where the writer is perhaps equivocating between Mac, Mc or mac), or even \fionn \mc Cool (where the writer should really make their mind up), though probably ill-advised, should nonetheless not fail to parse to something.
Beyond the individual puzzles above, there is also a problem, easy to miss but hard to ignore, that BibTeX's conventions for structuring a name into first-von-last-junior parts, though elegant, may be insufficient for a fully international bibliographic solution, and the wide range of ways in which real names must be sliced up and reassembled.
One systematic approach to this is the section of the Unicode CLDR devoted to person names. This describes a structure for recording names which doesn't have to go very far beyond BibTeX's four-part division. It can't accommodate everything (arabic names, for example, have notably intricate structure, discussed illuminatingly in a LanguageHat blogpost and in A.F.L Beeston’s ‘Arabic Nomenclature’, and the W3C document mentions some of the variation possible), but it can accommodate the practical basics of naming structures in a wide variety of locales. The OASIS ‘Extensible Name Language’, OASIS-xNL, is broadly similar to the CLDR one, as an internationalised schema for describing names. And the Citation Style Language includes a structure for names very similar to BibTeX's, but with five parts rather than four (it distinguishes von-parts which do and don't disappear when only a surname is shown). The W3C, and the Dublin Core consortium, have published helpfully wide-ranging discussions of the names problem: their advice to keep things as simple as possible is practical in general, but too simple for our bibliographic problem.
The CLDR's framework seems both a feasible increment on BibTeX's framework, and as comprehensive as a BibTeX successor needs to be.
Possible other sources: The Chicago Manual of Style, chapter 8, gives some detailed and explicit suggestions in this area; I'm sure they're entertainingly disputed wherever and whenever copy-editors are wont to gather for recreation purposes (though they are quite extensive, they don't seem to resolve the ‘J.P’ vs. ‘J.-P.’ question above; the CLDR document suggests the correct practice here might be locale-dependent). Butcher's copy-editing (CUP, 2012) includes much useful general information, but doesn't really address name-parsing. ‘BS 5605:1990 – Recommendations for citing and referencing published material’ might be of interest, but it gives only general suggestions about formatting of bibliographies.
The library community seems less helpful here than one might initially guess, since their concern with names is both more specific (locating objects by author name) and more general (doing so through all of recorded history): the librarians' MARC, and associated standards, are more concerned to identify canonical versions of names (the process of ‘authority control’), than to analyse them. The Anglo-American Cataloguing Rules, though now superseded, provide a link to successor standards in this particular area.
Summary: names are hard (quoth falsehoods, ob-xkcd, and lots more).
Deviations from bibtex behaviour:
btxhak says that the function type$ ‘pushes the null string if the type is either unknown or undefined’, in the sense that there's no type defined inside a default.type function. That is, to me, an odd and rather unhelpful default, and beastie does define type$ in this case.
The btxhak document says that ‘If any popped literal has an incorrect type, BibTeX complains and pushes the integer 0 or the null string, depending on whether the function was supposed to push an integer or string.’ Beastie doesn't do that, but raises an exception in this case. I could probably be persuaded to change my mind about this.
Beastie tries somewhat harder than BibTeX to indicate where an empty stack, or wrong-type error happened (as opposed to reporting only the line-number of the ITERATE command).
If you include an entry-type twice in a ENTRY declaration, within a .bst file, then BibTeX gets very upset. Beastie warns about this but presses on.
The Beastie implementation of the format-string syntax may not precisely match the documentation. Partly, this is because there is some deliberate ambiguity (BibTeX is documented to output ties ‘if it thinks there’s a need for one’). It seems worth mentioning that bibtex-the-program does occasionally produce output which varies from what the document says should happen. Bibliographic minutiae is an area where it is hard to be dogmatic (well, to be honest, it's an area where it's probably quite easy to be ‘dogmatic’, but hard to get anyone to agree with your personal dogma).
The BibTeX documentation (and its variants in the LaTeX Companion, for example) states that the separator between names is and. Somewhat to my surprise, however, BibTeX matches this case-insensitively, so that ‘Andrew And and John Smith’ is interpreted by BibTeX as one person with surname ‘Andrew’, a missing author, and a ‘John Smith’. That seems sub-optimal to me: there are people in the world who labour technologically with the surname ‘Null’: we should attempt not to copy their troubles to people with surname ‘And’. Beastie currently does not match BibTeX here: the name-separator string must be only and, case-sensitively. The same is true for others, the ‘et al.’ marker.
This seems to be an opportunity to add a couple of minor enhancements to the .bst language, in the form of additional built-in functions. These are documented in the bst module documentation, and include:
show.stack$$ : similarly to BibTeX stack$, the function show.stack$$ displays the stack in the same way, but leaves it unchanged.
printf$$ : this pops a string format, and then pops as many further objects as there are ~a or ~s format specifiers in the string. It then formats and outputs the format string and arguments, leaving the result on the stack.
printf.push$$ and printf.pop$$ : by default, printf$$ sends its output to the same destination as write$, but this can be adjusted. The printf.push$$ function pops one argument from the .bst stack, and leaves none behind. The printf.pop$$ function pops nothing from the .bst stack, and pushes one item. The output can be redirected to a file or to a string.
The excellent ‘Tame the BeaST’ document describes, and implements a number of useful extension functions. Several of these would be excellent candidates for implementation as new primitives in Beastie. It can be usefully supplemented by the observations and suggestions in Shell and Hoadley's BibTeX FAQ. The BibTeXu documentation describes some additional and some extended primitive functions. Would anyone be interested in regexps in .bst?
.bst filesFor completeness, the grammar I have used for .bst files is as follows.
input: list.of.commands
list.of.commands:
command
| list.of.commands command
command:
CMD_ENTRY braced.list.of.tokens braced.list.of.tokens braced.list.of.tokens
| CMD_EXECUTE '{' TOKEN '}'
| CMD_FUNCTION '{' TOKEN '}' function.body
| CMD_INTEGERS braced.list.of.tokens
| CMD_ITERATE '{' TOKEN '}'
| CMD_MACRO '{' TOKEN '}' '{' STRING '}'
| CMD_READ
| CMD_REVERSE '{' TOKEN '}'
| CMD_SORT
| CMD_STRINGS braced.list.of.tokens
function.body: braced.list.of.tokens
braced.list.of.tokens:
'{' '}'
| '{' list.of.tokens '}'
list.of.tokens:
token
| braced.list.of.tokens
| list.of.tokens token
| list.of.tokens braced.list.of.tokens
token:
TOKEN
| NUMBER
| STRING
| QUOTE TOKEN
| ':='
| '>'
| '<'
| '='
| '+'
| '-'
| '*'The terminals are:
TOKEN : a function or variable name must start with a letter. The btxhak document says ‘Variable and function names may not begin with a numeral and may not contain any of the ten restricted characters on page 143 of the LaTeX book, but may otherwise contain any printing characters. Also, BibTeX considers upper- and lower-case equivalents to be the same.
For concreteness, I've restricted this to a subset of the ‘printing characters’, namely [A-Za-z][A-Za-z0-9.$_-]*, which seems to be compatible with the .bst files I've seen in practice.
The tokens are coerced to lowercase.
NUMBER : matches the regexp #-?[0-9]+
STRING : any characters between double-quotes, "...". This includes Unicode characters.
QUOTE : the usual ' single-quote character.
The CMD_... commands are the ten corresponding command-names, entry, execute, and so on, matched case-insensitively.
White space is permitted between all of the lexical elements, apart from after the QUOTE terminal (Beastie warns if it finds a space here, but accepts it).
The Beastie lexer discards everything from a % to the end of line, regarding it as whitespace.
The btxhak document, in its final section, implies a grammar for the format-strings used by format.name$. I believe it is equivalent to the following:
input: list.of.specs
list.of.specs:
string.or.piece
| list.of.specs string.or.piece
/* a 'piece' is a brace-level-1 element like {f}, which must include a FMTSYM */
piece: '{' list.nonalpha.or.tie fmt list.nonalpha.or.tie '}'
list.nonalpha.or.tie:
/* empty */
| nonalpha.or.tie
| list.nonalpha.or.tie nonalpha.or.tie
fmt:
FMTSYM
| FMTSYM '{' maybe.string '}'
string.or.piece: STRING | piece
nonalpha.or.tie: NONALPHA | '~' | '~~'
maybe.string:
/* empty */
| STRINGTerminals:
NONALPHA : a string of non-alphabetic letters.STRING : a sequence of any characters other than { or }.FMTSYM : one of the name-part tokens f, ff, and so on.The grammar here requires that the underlying lexer is sensitive to the brace-level, and will produce tokens FMTSYM, NONALPHA, ~ and ~~ only within brace-level 1, and that it will additionally object to any letters other than [fvlj] at brace-level 1.