---- title: Dislocations between Beastie and BibTeX SPDX-FileCopyrightText: 2025 Norman Gray SPDX-License-Identifier: BSD-2-Clause Beastie is not intended to be a complete and exact replacement for BibTeX, and certainly not ‘bug-compatible’. It should be broadly compatible with BibTeX in ‘normal’ use (for most values of ‘normal’), but may treat edge cases differently. In particular, there is a (2007) list of ‘BibTeX Tips and FAQ’ at the [CTAN package page](https://ctan.org/pkg/bibtex), which describes a fairly arcane set of tricks and workarounds to deal with the wide variation in names across the world. Some of these are very specific to the way that `bibtex`-the-program implements its (sometimes rather ill-defined) rules for parsing names, and few of them are situations authors are likely to encounter frequently. If Beastie behaves differently from BibTeX in similar cases, I do not intend to immediately class that as a bug. Deviations from BibTeX behaviour, and other anomalies ===================================================== Parsing `.bib` files -------------------- * All the files that Beastie reads are expected to be UTF-8 encoded (supporting UTF-16 and UTF-32 may come in future; there is already some support for the former internally). * `beastie` supports only a subset of the Scribe syntax. Following the Beebe document cited on the [main page](beastie.xhtml), the only concession to Scribe is supporting `(...)` as the entry delimiters. * Names – that is, the names of field names and string references – have a slightly more restricted syntax than BibTeX permits. See below. * That said, the contents of _entry keys_ are are broader than BibTeX suggests (and broader than Beebe recommends). Myself, I think it's probably not a great idea to go far beyond Beebe's pattern for citation keys, as this seems likely to be storing up problems for downstream processing. But `bibtex8` – the 8-bit version of BibTeX which is the one distributed in practice – is happy to accommodate a large range of bytes in citation keys, so this tells us (a) that this seems not to cause actual downstream problems in LaTeX workflows, and (b) that Beastie should do something similar. Thus Beastie will (currently) accept entry keys containing ‘word characters’, digits, symbols or characters in the set `[:./_&-]` (specifically, it excludes ASCII characters below and including `space` (0x20), plus 0x7f, and the characters `["%,={}]`; if you're somehow using any of these in a citation key, then... I have absolutely no sympathy: stop it!). The definition of a ‘word character’ here matches the set of characters one might expect to see in a ‘word’, defined in a script-independent way. There is no case-folding for entry keys. * We accept within field names word characters, digits and a few punctuation characters, namely `[-._:]`. The selection is pragmatic rather than particularly principled, and is, again, intended to be sympathetic to downstream processors. * Beastie permits multiple `crossref` redirections, but still demands that the cross-references refer only to later entries. In consequence it will (like BibTeX) cope with circular cross-references by reporting a backward-pointing reference as ‘missing’. I may change this in future, to support backward references. * BibTeX handles such constructions such as `{\"o}` specially, in that it regards such a brace-level-1 construction as constituting a single character for counting purposes. If Beastie encounters such a construction, and if the replacement of the command is a single character within the braces (in this case ‘ö’), then the braced sequence is replaced by that character at parse time. Sequences Beastie doesn't recognise, or braced sequences containing more than one character, are left untouched. Beastie accepts a larger set of such characters than (I think) BibTeX does, and also accepts some characters such as `{\ss}`. Specifically (and for the sake of a goal) we support the TeX commands for all of the characters in ISO-8859-1, and a few in ISO-8859-2, plus all of the letter characters in Unicode [Latin-1 Supplement](https://www.unicode.org/charts/PDF/U0080.pdf) and most of the letters in the [Latin Extended-A](https://www.unicode.org/charts/PDF/U0100.pdf) range, with a few exceptions where the LaTeX command is hard to find, which are therefore unlikely to be used in practice. The internal table of replacements is fixed, and is not currently customisable. Btxdoc says these command sequences are recognised only at brace-level one, but beastie recognises them at all brace levels. This happens during the parsing of the `.bib` file. * We accept Unicode in field values, with some limitations. Specifically, we accept only UTF-8 input, and we can't recognise BibTeX ‘von’ particles which start with a letter which isn't ASCII-lowercase (but I've never seen that in a bib file, in practice). * We recognise `%`-comments, which ignore everything from the `%` to the end of line. These have always been permitted (and are occasionally seen) between entries in BibTeX files, but their content is ignored because BibTeX ignores _all_ content between entries, rather than treating these characters specially. In `beastie`, such comments can appear anywhere whitespace can, thus within entries as well as outwith them, and, unlike BibTeX, a `%`-comment will hide a `@entry` on the same line. * Beastie supports the `@include{...}` form proposed by Beebe. The content of the braces is the name of a BibTeX database, including the `.bib` extension, which is looked up using `kpsewhich`. The indicated file is read at the point where this form appears, as if by direct inclusion (though it must be a valid `.bib` file as it stands, and cannot break an entry across different files). Thus, any `@string` definitions in a `.bib` file are visible in any file it subsequently includes, and are visible, after the `@include` form, in any file which includes it. Included files can include other files in turn, and beastie should detect if there is an implied inclusion loop. An alternative design is an `@import` form, which imports bibliographic databases while carefully isolating `@string` definitions to a single file: this is tidy, but since a reasonable use of this `@include` mechainism is to pull in sets of `@string` definitions, this seems likely to be less useful in practice. If the argument of `@include` is a key which has been defined with `@string`, then the key is expanded to its value. Citation keys and field names ------------------------- Beastie has a slightly narrower definition of legal entry and citation keys, than BibTeX has. BibTeX appears to accept just about any characters (or even any bytes) here that aren't syntactically special to that program. That's agreeably laid-back, but not perhaps very friendly to parser-writers. It also meant that, if people used non-ASCII characters here, the keys would potentially have different values depending on the assumed coding of the `.bib` file. Beastie restricts the keys here a little more than Nelson Beebe's proposal: the goal is to accept most of what I've seen in the wild, as it were, while ruling out characters that seem to be asking for trouble. These prescriptions explicitly include sets of Unicode codepoints. * entry key: sequence of - ‘word characters’ - digits (Unicode character category `N`, overlaps with ASCII `[0-9]`) - symbols (Unicode `S`, overlaps with ASCII `[$+<=>^|``]`) - or `[:./_&-]` (which Unicode regards as ‘punctuation’) - handled case-sensitively. * field names (and non-standard @type names): sequence of - word characters - digits (as above) - or `-` - handled case-insensitively. The ‘word characters’ here are those characters that may conventionally be regarded as part of a word. There isn't a formal definition of this set in Unicode, but a conventional definition is Unicode ‘alphabetic’ characters, plus diacritics, extender characters and join-control characters (characters, that is, which possess the Unicode properties `Alphabetic` (which overlaps with ASCII `[A-Za-z]`), `Diacritic`, `Extender` and `Join_Control` – see the Unicode Character Database ([UCD][]) for intricate details). The Beastie `uchar-alphabetic?` and `uchar-word-character?` predicates test for these. Beastie currently restricts each of these predicates to the Base Multiligual Plane (BMP) only (ie, codepoints up to U+FFFF); non-BMP characters are fine in strings. [Unicode]: https://www.unicode.org/versions/latest/ [UCD]: https://www.unicode.org/reports/tr44/ A grammar for .bib files ------------------------ input: opt_interentry_text list_of_stanzas opt_interentry_text list_of_stanzas: stanza | list_of_stanzas opt_interentry_text stanza opt_interentry_text: /* empty */ | opt_interentry_text INTERENTRYTEXT stanza: entry | atpreamble | atstring | atcomment | atinclude entry: ENTRYTYPE '{' NAME \ ',' list_of_fields '}' | ENTRYTYPE '{' NAME '}' | ENTRYTYPE '{' NAME ',' '}' /* at most one trailing comma */ list_of_fields: field | list_of_fields ',' field | list_of_fields ',' /* ditto */ field: NAME '=' string atpreamble: ATPREAMBLE '{' string '}' atstring: ATSTRING '{' NAME '=' string '}' atinclude: ATINCLUDE '{' NAME '}' atcomment: ATCOMMENT string string: STRINGVALUE | NAME | string '#' STRINGVALUE | string '#' NAME The terminals in this grammar are `ENTRYTYPE`, representing `@article` and the like; terminals representing the standard `@preamble`, `@string` and `@comment` forms, and a further terminal representing the extension `@include` form (all case-insensitive); `NAME` represents a entry key or field, `STRINGVALUE` a field value `"..."` or `{...}`, and `INTERENTRYTEXT` any other text between entries. In fact, Beastie uses a slightly different (non-yacc) grammar framework to do its work, but it should be equivalent to this one. The lexer produces terminals `','`, `'='`, `'#'`, `'{'` and `'}'` after having stripped surrounding whitespace, and the latter two can be produced by the corresponding characters, or by `'('` or `')'`, in a modest concession to Scribe's continuing legacy in extant `.bib` files. .bst and btxhak =============== The ‘standard’ here is the `btxhak.pdf` document at [CTAN](https://ctan.org/pkg/bibtex). Unfortunately, that document is a little vague in places. Nicolas Markey's [Tame the BeaST][tamethebeast] document is an account of the whole BibTeX ecosystem, including a description of the `.bst` language. btx anomalies ------------- * BibTeX, with format-string `{f.~}{ll}` turns ‘Jean-Paul Sartre’ into `J.-P. Sartre`, and so does beastie. But BibTeX puts a non-default separator in here, too, so that `{f{.}}` turns ‘Jean-Paul Sartre’ into `J.P`, which, losing the hyphen, is surely wrong (or, if not _wrong_ in some absolute sense, I'd hope to have an argument with a copy-editor about it, rather than Oren Patashnik). So I'm not going to follow BibTeX there. Should I completely ignore a non-default separator here, or perhaps re-insert the hyphen (so for example `{f{x}}` would produce `Jx-P`)? The only text covering this in btxdoc is Sect.2.1, item 5, which _illustrates_ this by saying ‘and if you’re using the abbrv style [which uses "{f.~}{vv}{ll}"], then the result is “J.-P. Sartre”’, but doesn't say what's supposed to happen if you're using a different f-format. * If `substring$` is given a value for the start parameter, the modulus of which is greater than the length of the string, then `substring$` returns "" without error. This isn't mentioned in the documentation. * Trivia: The `prime-number-theorem` bib entry in the torture test at cites `Charles Louis Xavier Joseph de la Vall{\'e}e Poussin`, but (a) if Wikipedia is to be believed, it appears that he's Joseph-Xavier rather than the other way around, and (b) it's possible that he's being confused with his son, Charles-Jean, the mathematician responsible for the prime number theorem. Parsing names ------------- * Beastie parses ‘Ruiz y Picasso, Pablo’ to get the surname ‘Ruiz y Picasso’. BibTeX takes the Last name to be ‘Picasso’ with von-part ‘Ruiz y’, which I think is (linguistically) wrong (and not specified in btxdoc). In any case, the Correct Behaviour here is probably highly language-sensitive, so either we should worry about it _a lot_, or not worry about it _at all_. * The BibTeX documentation notes that multi-word surnames – quoting the example of Per Brinch Hansen – need to be made explicit using the comma-form. Though it doesn't say so, it seems obvious that `Per Brinch~Hansen` should be parsed in a compatible way (here, as a single-token surname). * ADS formats Lidia van Driel-Gesztelyi's name as `"{Driel-Gesztelyi}, Lidia van"`. I'm not sure this is entirely correct, but we should aim not to fail in this case, and should probably produce the same result as BibTeX, which parses this as surname ‘Driel-Gesztelyi’ and _forenames_ ‘Lidia van’. This is a separate question from what the analysis _should_ be here, in language terms rather than BibTeX ones, since the surname here is a hyphenated combination of a Dutch name with a von-particle, and a Hungarian name without. This may also be an attempt to respect different languages' name-ordering practices, but that's a bog we should probably avoid wandering into at this stage. * ‘Myles na gCopaleen’ is hard, not least because I don't know what the correct analysis _should_ be, in language terms. I understand that 'na' is a genitive particle, therefore similar to ‘von’, and to unassimilated 'mac/nic' in gaelic names, and the small ‘g’ (an [eclipsis](https://en.wikipedia.org/wiki/Irish_initial_mutations#Eclipsis)!) is a required spelling change to the noun ‘copaleen’ (though this particular name is... [complicated](https://en.wikipedia.org/wiki/Flann_O%27Brien), and its pseudonymous possessor spelled it more than one way). In BibTeX terms, however (which are here the only ones that matter), `bibtex` parses this name as having ‘surname’ ‘gCopaleen’, and ‘Myles na gCopaleen Smith’ as having surname ‘Smith’ and von-parts ‘na gCopaleen’. The key thing from BibTeX's point of view, it seems, is that there is _always_ a Last name, even if it looks like a von-particle, starting with a lowercase letter. * This also means that, if someone styles their name as ‘Fred ffoulkes’, we shouldn't second-guess them. * There are few cases where the nominal case of a macro matters, and certainly none mentioned in the btxdoc documentation, but I feel that something like `Finn \mc Cool` (where the writer is perhaps equivocating between `Mac`, `Mc` or `mac`), or even `\fionn \mc Cool` (where the writer should _really_ make their mind up), though probably ill-advised, should nonetheless not fail to parse to _something_. Beyond the individual puzzles above, there is also a problem, easy to miss but hard to ignore, that BibTeX's conventions for structuring a name into first-von-last-junior parts, though elegant, may be insufficient for a fully international bibliographic solution, and the wide range of ways in which real names must be sliced up and reassembled. One systematic approach to this is the section of the Unicode [CLDR][] devoted to [person names][LDML-names]. This describes a structure for recording names which doesn't have to go very far beyond BibTeX's four-part division. It can't accommodate everything (arabic names, for example, have notably intricate structure, discussed illuminatingly in a LanguageHat [blogpost][arabicnames] and in [A.F.L Beeston’s ‘Arabic Nomenclature’][beeston], and the [W3C][] document mentions some of the variation possible), but it can accommodate the practical _basics_ of naming structures in a wide variety of locales. The OASIS ‘Extensible Name Language’, [OASIS-xNL][], is broadly similar to the CLDR one, as an internationalised schema for describing names. And the Citation Style Language includes a [structure for names][CSL-names] very similar to BibTeX's, but with five parts rather than four (it distinguishes von-parts which do and don't disappear when only a surname is shown). The [W3C][], and the [Dublin Core consortium](https://www.dublincore.org/specifications/dublin-core/name-representation/), have published helpfully wide-ranging discussions of the names problem: their advice to keep things as simple as possible is practical in general, but too simple for our bibliographic problem. The CLDR's framework seems both a feasible increment on BibTeX's framework, and as comprehensive as a BibTeX successor needs to be. Possible other sources: The [Chicago Manual of Style][], chapter 8, gives some detailed and explicit suggestions in this area; I'm sure they're entertainingly disputed wherever and whenever copy-editors are wont to gather for recreation purposes (though they are quite extensive, they don't seem to resolve the ‘J.P’ vs. ‘J.-P.’ question above; the [CLDR][LDML-names] document suggests the correct practice here might be locale-dependent). [Butcher's copy-editing (CUP, 2012)](https://doi.org/10.1017/CBO9780511482106) includes much useful general information, but doesn't really address name-parsing. ‘BS 5605:1990 – Recommendations for citing and referencing published material’ _might_ be of interest, but it gives only general suggestions about formatting of bibliographies. The library community seems less helpful here than one might initially guess, since their concern with names is both more specific (locating objects by author name) and more general (doing so through all of recorded history): the librarians' MARC, and associated standards, are more concerned to identify canonical versions of names (the process of ‘authority control’), than to analyse them. The [Anglo-American Cataloguing Rules](https://en.wikipedia.org/wiki/Anglo-American_Cataloguing_Rules), though now superseded, provide a link to successor standards in this particular area. Summary: names are hard (quoth [falsehoods](https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/), [ob-xkcd](https://xkcd.com/327/), and lots more). Enhancements and changes to the .bst language --------------------------------------------- Deviations from `bibtex` behaviour: * btxhak says that the function `type$` ‘pushes the null string if the type is either unknown or undefined’, in the sense that there's no type defined inside a `default.type` function. That is, to me, an odd and rather unhelpful default, and `beastie` does define `type$` in this case. * The btxhak document says that ‘If any popped literal has an incorrect type, BibTeX complains and pushes the integer 0 or the null string, depending on whether the function was supposed to push an integer or string.’ Beastie doesn't do that, but raises an exception in this case. I could probably be persuaded to change my mind about this. * Beastie tries somewhat harder than BibTeX to indicate where an empty stack, or wrong-type error happened (as opposed to reporting only the line-number of the `ITERATE` command). * If you include an entry-type twice in a `ENTRY` declaration, within a `.bst` file, then BibTeX gets very upset. Beastie warns about this but presses on. * The Beastie implementation of the format-string syntax may not precisely match the documentation. Partly, this is because there is some deliberate ambiguity (BibTeX is documented to output ties ‘if it thinks there’s a need for one’). It seems worth mentioning that bibtex-the-program does occasionally produce output which varies from what the document says should happen. Bibliographic minutiae is an area where it is hard to be dogmatic (well, to be honest, it's an area where it's probably quite _easy_ to be ‘dogmatic’, but hard to get anyone to agree with your personal dogma). * The BibTeX documentation (and its variants in the LaTeX Companion, for example) states that the separator between names is `and`. Somewhat to my surprise, however, BibTeX matches this _case-insensitively_, so that ‘Andrew And and John Smith’ is interpreted by BibTeX as one person with surname ‘Andrew’, a missing author, and a ‘John Smith’. That seems sub-optimal to me: there are people in the world who labour technologically with the surname ‘Null’: we should attempt not to copy their troubles to people with surname ‘And’. Beastie currently _does not match BibTeX here_: the name-separator string must be only `and`, case-sensitively. The same is true for `others`, the ‘et al.’ marker. This seems to be an opportunity to add a couple of minor enhancements to the `.bst` language, in the form of additional built-in functions. These are documented in the [bst module documentation](bst.xhtml), and include: * `show.stack$$` : similarly to BibTeX `stack$`, the function `show.stack$$` displays the stack in the same way, but leaves it unchanged. * `printf$$` : this pops a string format, and then pops as many further objects as there are `~a` or `~s` format specifiers in the string. It then formats and outputs the format string and arguments, leaving the result on the stack. * `printf.push$$` and `printf.pop$$` : by default, `printf$$` sends its output to the same destination as `write$`, but this can be adjusted. The `printf.push$$` function pops one argument from the `.bst` stack, and leaves none behind. The `printf.pop$$` function pops nothing from the `.bst` stack, and pushes one item. The output can be redirected to a file or to a string. The excellent [‘Tame the BeaST’][tamethebeast] document describes, and implements a number of useful extension functions. Several of these would be excellent candidates for implementation as new primitives in Beastie. It can be usefully supplemented by the observations and suggestions in Shell and Hoadley's [BibTeX FAQ][]. The [BibTeXu][] documentation describes some additional and some extended primitive functions. Would anyone be interested in regexps in `.bst`? Grammars ======== A grammar for `.bst` files -------------------------- For completeness, the grammar I have used for `.bst` files is as follows. input: list.of.commands list.of.commands: command | list.of.commands command command: CMD_ENTRY braced.list.of.tokens braced.list.of.tokens braced.list.of.tokens | CMD_EXECUTE '{' TOKEN '}' | CMD_FUNCTION '{' TOKEN '}' function.body | CMD_INTEGERS braced.list.of.tokens | CMD_ITERATE '{' TOKEN '}' | CMD_MACRO '{' TOKEN '}' '{' STRING '}' | CMD_READ | CMD_REVERSE '{' TOKEN '}' | CMD_SORT | CMD_STRINGS braced.list.of.tokens function.body: braced.list.of.tokens braced.list.of.tokens: '{' '}' | '{' list.of.tokens '}' list.of.tokens: token | braced.list.of.tokens | list.of.tokens token | list.of.tokens braced.list.of.tokens token: TOKEN | NUMBER | STRING | QUOTE TOKEN | ':=' | '>' | '<' | '=' | '+' | '-' | '*' The terminals are: * `TOKEN` : a function or variable name must start with a letter. The btxhak document says ‘Variable and function names may not begin with a numeral and may not contain any of the ten restricted characters on page 143 of the LaTeX book, but may otherwise contain any printing characters. Also, BibTeX considers upper- and lower-case equivalents to be the same. For concreteness, I've _restricted_ this to a subset of the ‘printing characters’, namely `[A-Za-z][A-Za-z0-9.$_-]*`, which seems to be compatible with the `.bst` files I've seen in practice. The tokens are coerced to lowercase. * `NUMBER` : matches the regexp `#-?[0-9]+` * `STRING` : any characters between double-quotes, `"..."`. This includes Unicode characters. * `QUOTE` : the usual `'` single-quote character. * The `CMD_...` commands are the ten corresponding command-names, `entry`, `execute`, and so on, matched case-insensitively. White space is permitted between all of the lexical elements, apart from after the `QUOTE` terminal (Beastie warns if it finds a space here, but accepts it). The Beastie lexer discards everything from a `%` to the end of line, regarding it as whitespace. A grammar for author-format-strings ----------------------------------- The btxhak document, in its final section, implies a grammar for the format-strings used by `format.name$`. I believe it is equivalent to the following: input: list.of.specs list.of.specs: string.or.piece | list.of.specs string.or.piece /* a 'piece' is a brace-level-1 element like {f}, which must include a FMTSYM */ piece: '{' list.nonalpha.or.tie fmt list.nonalpha.or.tie '}' list.nonalpha.or.tie: /* empty */ | nonalpha.or.tie | list.nonalpha.or.tie nonalpha.or.tie fmt: FMTSYM | FMTSYM '{' maybe.string '}' string.or.piece: STRING | piece nonalpha.or.tie: NONALPHA | '~' | '~~' maybe.string: /* empty */ | STRING Terminals: * `NONALPHA` : a string of non-alphabetic letters. * `STRING` : a sequence of any characters other than `{` or `}`. * `FMTSYM` : one of the name-part tokens `f`, `ff`, and so on. The grammar here requires that the underlying lexer is sensitive to the brace-level, and will produce tokens `FMTSYM`, `NONALPHA`, `~` and `~~` only within brace-level 1, and that it will additionally object to any letters other than `[fvlj]` at brace-level 1. [BibTeX FAQ]: https://ctan.org/pkg/bibtex [tamethebeast]: https://ctan.org/pkg/tamethebeast [Chicago Manual of Style]: https://www.chicagomanualofstyle.org/ [bibtexu]: https://ctan.org/pkg/bibtexu [LDML-names]: https://www.unicode.org/reports/tr35/tr35-personNames.html [CLDR]: https://cldr.unicode.org [arabicnames]: https://languagehat.com/arabic-names/ [beeston]: https://ia903407.us.archive.org/8/items/history-arabs/Arabic%20Nomenclature%20%28Beeston%2C%201971%29.pdf [OASIS-xNL]: https://www.oasis-open.org/committees/ciq/ciq.html [CSL-names]: https://docs.citationstyles.org/en/stable/specification.html#names [W3C]: https://www.w3.org/International/questions/qa-personal-names