Expanding simple TeX commands
Most of the (La)TeX commands found in BibTeX files are there to produce accented characters of one type or another. When Beastie parses such a file, it expands an assortment of such commands into the corresponding Unicode characters.
For example, the field
author = {Ku\v rt G\"{o}del and Ren{\'e}e R{\o}mer}will be processed as if it were
author = {Kuřt Gödel and Renée Rømer}Note the special case of {\'e} and {\o} – if this expansion process results in a single character appearing in braces, then the braces are discarded. This corresponds to a related, but distinct, special case in bibtex’s parsing of ‘accented characters’: bibtex treats such sequences, at brace-level one, as being a single character for the purposes of counting characters and generating labels. In Beastie, we support this case at any brace level, and go further by replacing the input sequences with the corresponding character.
Beastie recognises a fairly extensive set of such commands, and in particular should be able to manage all of the characters in ISO 8859-1 and the Unicode Latin-1 range, and most of the letters in the Unicode Latin Extended-A range, with a few exceptions where the LaTeX command is hard to find, which are therefore unlikely to be used in practice.
The set of supported commands is currently fixed, but may become expandable in future versions of Beastie.
Beastie also natively recognises Unicode characters in .bib files, so one could use either of the examples above in such a file, in this or any other script covered by Unicode.
Other commands in .bib files, such as \emph{foo} or \LaTeX, are not recognised by default, but you can add support for these within the Scheme layers (not within .bst) using the user-char-command function.
Beastie manipulates these internally using a ‘bstring’ type (for ‘braced string’). You will typically not manipulate bstrings directly, except if you use the combination of ustring-iterator/bstrings and list->bstring to iterate through a ustring in a bstring-aware way. Functions such as uppercase-string/bst from the 'bst module use this to uppercase strings while leaving brace-level-1 content alone, in the way that .bst files require.
You may have a use for function ustring->ustring, which converts a ustring to a ustring in a way which allows you to control how braces and no-break spaces, or tildes in TeX terms, are managed. This is logically part of this module because it does this via a bstring. The function untexify-string-or-list wraps this in a way which might be convenient if you are generating xexpr structures, on the way to generating (X)HTML.
Index:
(bstring->ustring ts [:write write/display/#t/#f/] [:nbsp string]) : format a bstring?.
If the :write argument is 'display or #f (the default), then produce a string for display; otherwise it should be 'write or #t to produce a string that should be reparseable by string->bstring.
If, instead of 'display, the selector is 'display/braces or 'display/without-braces, then the output ustring is or is not displayed surrounded by braces, respectively. If it is just :display then the outermost braces are skipped, but inner ones are included. This may seem a slightly fussy distinction, but recall that a .bib-file value like "foo {bar}" would be parsed as the bstring #{foo {bar}}, but the most natural way of printing this should end up with ‘foo {bar}’. I may add a settable global default (FIXME: cf subtex-load-hook code above).
The argument can alternatively be a list of bstring-content? objects.
Note that this default, preferring display over write since that is more common, is opposite to the internal object->string function.
(bstring? x) : return #t if the argument is a bstring.
A bstring? is a 'TeX string', returned by string->bstring. The bstring can contain objects of type bstring-content?.
(list->bstring l) : given a list of ustring?, integer?, bstring?, or 'nbsp, create a new bstring? object.
(parse-subtex str [:on-error #f]) : parse a ‘subtex’ string or ustring, str, producing a ustring? object as result.
Given:
One \emph{sharp} ep{\'e}e {here}The function recognises both the TeX-style command \emph, and the TeX-style braces {...}, and and the ‘accent’ {\'e}, which it expands to ‘é’. The result is an opaque ustring? object.
The input string should be a ustring, a valid UTF-8 string of characters, or a list of integer codepoints.
On error:
#f (the default), then raise a beastie-error with subtag 'subtex;#t return STR;That said, the procedure as currently implemented doesn't throw any errors on parse failures -- that is, we aim not to object to any input. The only errors thrown are for internal coding errors, which shouldn't be caught here. I may remove this error-handling in future.
Although this function is exposed for use in end-user programs, it is not much needed, as the expansion in question has generally been done when data is read from a .bib file.
The type is called a 'TeX'-string because it primarily expands TeX ‘accents’ such as \'e, but it expands a few other (La)TeX control sequences as well. The list is unspecified, and not (currently) modifiable, but is intended to cover the control sequences often occurring in BibTeX databases.
In the special case of braces (any level) enclosing a single accented character, for example n{\'e}e, the character is expanded and the braces discarded, resulting in the bstring ‘née’; this tidies a common case often found in .bib files, and illustrated in the BibTeX documentation. This special case does not extend to ASCII-range characters (ie, n{e}e is parsed as ‘n{e}e’), but it does extend to single Unicode characters written directly (ie, n{é}e is parsed as ‘née’); this is slightly inconsistent, and the behaviour here may change if it becomes clear that an alternative can be clearly defined. The special case is strictly restricted to single characters, thus n{\'ee} becomes ‘n{ée}’.
A tilde in the input (~) is interpreted as a non-breaking space, in the usual TeX way. An actual no-break space character (Unicode U+00A0) is equivalent.
(untexify-string-or-list x) : untexify a string or list. This removes braces and "~" from an input string, or a list containing strings. This wraps the function ustring->ustring.
(user-char-command key value) : add a mapping for TeX-style \\cmd sequences. The key must be a string? or ustring?. The value may be a string?, ustring? or integer? codepoint, or a procedure of zero, one or two (currently no more) arguments, which evaluates to one of those types.
Thus, after
(user-char-command "pounds" #"£")
(user-char-command "poundstring" "lb")
(user-char-command "poundtstring" #{lb})
(user-char-command "poundnum" #xa3)
(user-char-command "poundsproc" (λ () #"£££"))
(user-char-command "poundsproci" (λ () "££"))
(user-char-command "poundsprocii" (λ () #xa3))
(user-char-command "emph" (λ (a) (make-ustring (sprintf "**~a**" a))))
(user-char-command "concat" (λ (a b)
(sprintf "(~a/~a)" a b)))we might evaluate an input string as follows
(ustring->string
(ustring->ustring "ab\\pounds c \\emph{d} and \\concat e{f}"))
-> "#"ab£c **d** and (e/f)""In an input string, any unrecognised commands are turned into a string version of the command, without error. The only pre-defined commands are the single-character ones discussed in parse-subtex, below.
If you are interested in expanding commands within .bib files, then these commands must be defined before the .bib file is read.
(ustring->ustring str [:braces? #t/#f] [:nbsp "?"]) : rewrite a string? or ustring? into a ustring?, respecting ‘subtex’ conventions, and controlling the display.
This is very similar in effect to the combination of parse-subtex (qv) and bstring->ustring, but lets you control the resulting ustring somewhat.
If :braces? is #t (the default), then {...} braces in the input appear in the output. If it is #f, then they are suppressed (which is useful when the output result is not intended to be processed by TeX).
If :nbsp str is present, then that string is used when displaying a non-breaking space in the input. The default is "~" (ie, the usual TeX character), but setting this to a string containing a Unicode no-break space is an alternative. Thus, given the string "a~é {\~n}{c}" (with a no-break space in the middle), (ustring->ustring s) would evaluate to #"a~é~ñ{c}", but (ustring->ustring s :braces? #f :nbsp "+") would evaluate to #"a+é+ñc".
(ustring-iterator/bstrings us) : returns an iterator over the ustring, which will return single codepoint-integers on each call, just like the usual ustring iterator, except that it will return {braced content} as a single bstring? object, and non-breaking spaces (ie, from "~") as a symbol 'nbsp.
This, along with list->bstring, allows you to work through a ustring in a TeX/bst-compatible fashion, by preserving no-break spaces, and identifying all content outside brace-level 0. It means that, given a ustring #"ab~c{d}", the operation
(list->bstring (map values (ustring-iterator/bstrings #"ab~c{d}")))is effectively a no-op.