Basic Unicode support
Beastie does not have complicated Unicode-related requirements, but it does need to be able to read and write Unicode files, and to do language-sensitive sorting and case-switching.
The character class functions, in the utils module, are partially Unicode aware, in the sense that they will report the classes of characters in the BMP. These functions are char-alnum?, char-alpha?, char-upper?, char-lower?, char-other-letter?, char-digit?, char-space?, char-punct?, char-cntrl?, char-symbol? and char-mark?. There are also case-changing functions uchar-upcase, uchar-downcase, uchar-titlecase, and classification functions such as uchar-alphabetic?. See the documentation in the utils module for details of their behaviour, and see the Unicode Character Database (and its file collection) for detailed discussion of the properties in question.
Various functions within beastie produce and handle ‘ustrings’, which should generally be equivalent to normal strings in use, but which have various other unicode-related operations defined. These principally consist of the unicode-related functions described just below.
Construct an empty ustring with (make-ustring), or add content with (make-ustring codepoint/string/ustring ...). The various ustring-* functions below are intended to broadly match the corresponding ordinary string ones. Since ustring-append creates a new ustring, that is an alternative way of creating a ustring from scratch.
Ustrings have write-form #"abç", and the reader will parse this back into a ustring. Within the quotes, the only escapes recognised are \\, \", and \n, with the last denoting a newline. Any other escaped characters are ignored, with a warning.
A ustring is an ‘applicable object’ in s7 terms: if us is a ustring, then (us 2) is the same as (ustring-ref us 2), evaluating to the codepoint of us at index 2. Similarly map or for-each applied to a ustring will call the mapped function for each codepoint in the string.
A unicode-reader? is an object which turns a string or file into a sequence of Unicode codepoints. Associated functions are
A unicode-reader? object is a lexeme-source? in the sense of the klipspringer module.
It may occasionally be useful to note that two unicode-reader? objects are deemed to be equivalent? if they refer to the same underlying object, string or file; they are equal? only if they are additionally at the same offset into the string or file.
Separately, we can encode lists of codepoints to UTF-8, and decode a UTF-8 string.
Beastie aims to be Unicode-aware by design rather than by accident. That is, input is parsed and handled as Unicode codepoints, rather than being bundled into UTF-8 strings and hoping for the best.
This means, for example, that the rules for allowed strings in .bib key and field names are expressed in terms of Unicode alphabetic characters (thus नाम is a valid key or field name, since it's a list of letters, but a¶ isn't, since it includes a punctuation character). This obviously isn't a key beastie use-case, but it serves as a check on the ‘unicodeness’ of the code.
The conformance to Unicode is reasonably extensive, but not complete. There are two cases here.
If Beastie is built using the ICU library, then it has (some) locale support, and in particular it has language-sensitive sorting.
If Beastie is build without the ICU library, either because it is not available or because it was suppressed at build time, then there is still Unicode support, but less sophisticated.
Further notes, applicable to the with- and without-ICU cases:
We properly support only the Basic Multilingual Plane (BMP) – characters outside of that, in the ‘astral planes’, are perfectly welcome in strings, but are not regarded as being included in any character class (so yes, you can happily have emojis in field values, but no, you may not have them in database keys).
There are various edge-cases, such as around title casing, that the code avoids (yes, the eszett!).
It's doubtless possible to confuse the string-classification algorithm by using one or other Unicode normal forms.
Index:
(make-unicode-reader/file fn ascii-characters?) : make a unicode reader from a file fn.
If ascii-characters? is true, then the reader will return codepoints below 0x80 as characters rather than integers.
If the fn is #f, then this will read from stdin.
(make-unicode-reader/string fn ascii-characters?) : make a unicode reader from a string str.
If ascii-characters? is true, then the reader will return codepoints below 0x80 as characters rather than integers.
(make-ustring [codepoint/string/ustring ...]) : make a ustring, optionally adding content.
The list of arguments is the same as that for ustring-append, qv.
(symbol->ustring sym) : returns the symbol sym converted to a new ustring
(unicode-decode/utf8 s) : decode the UTF8 string (an s7 string, regarded as a sequence of bytes encoded in UTF-8) into a ustring
(unicode-decode1/port/utf8 [p]) : decode a single unicode character from the input port, which should be UTF-8 encoded. If no argument is supplied, it defaults to (current-input-port). This function isn't (currently) particularly robust against malformed UTF-8 input. It shouldn't collapse, but may skip characters when recovering.
Returns the next codepoint from the input, or #<eof>; returns Unicode replacement character on malformed input.
(unicode-encode/utf8 l) : given a list or iterator of Unicode codepoints, convert it to a UTF8 string (an s7 string, regarded as a sequence of bytes)
(unicode-encode1/utf8 i) : given a single character or integer, convert it to a UTF8 string (an s7 string, regarded as a sequence of bytes)
(unicode-get-locale [locale-name] [info]) : get information about the given locale.
With no arguments, returns an alist for the currently preferred locale. With one locale (string) argument, returns an alist of keys and values describing the locale. With two arguments, returns the (symbol) key information about the (string) locale, returning #f if the information is not available.
If the locale is given as #f, rather than a string, this is equivalent to the no-args case.
There is more information about Unicode locales in the ICU user guide.
(unicode-get-locales) : return a list of the available locales.
(unicode-reader-location rdr) : show the current location within the reader's source.
(unicode-reader-read unicode-reader) : return a single codepoint from the stream. Returns #<eof> at the end of the stream.
The unicode-reader object can also be called directly, as a no-argument function, to return the codepoint, and it can be called with argument 'location to give the current location, as an alternative to unicode-reader-location.
(unicode-reader-source rdr) : show the source from which the reader draws its content, either as a filename, or as a "string", enclosed in quotes. The output is intended to be printable.
(unicode-reader? x) : return #t if x is a unicode-reader
(unicode-set-locale! "locale") : set the preferred locale for string comparisons. Passing a locale of #f resets this to the default. If the environment variable $BEASTIE_LOCALE is set, then that is the initially preferred value. Returns the name of the previously-set locale, or #f if the locale initialisation isn't aligned with a locale (for example if it is based on (default) rules).
(ustring->hash u) : return a hash integer for the ustring
(ustring->list us) : convert a ustring to a list of codepoints
(ustring->string ustr [:display/:write/:readable/#t/#f]) : returns the ustring str converted to a (normal) UTF-8 string. If the second argument is :display or #f, then the string is suitable for display rather than for subsequent reading; otherwise, it is in a form which can be re-read to produce an equivalent object.
(ustring->symbol ustr) : returns the ustring str converted to a symbol.
(ustring-append character|codepoint|string|ustring|list ...) : append the arguments to the ustring. The procedure can take characters, codepoints, strings, ustrings, or a list of these. The result is a new ustring (cf ustring-append!).
(ustring-append! ustring char|codepoint|... ...) : append the arguments to the ustring. The replacement happens in place – the initial ustring argument is modified (cf ustring-append). The list of possible argument types is the same as for ustring-append.
(ustring-car us) : return the first codepoint in the string, as an integer.
(ustring-index us cp [:start 0] [:end #f]) : Return the index of the first character in the ustring? us which is equal to cp, where cp is an integer codepoint or a character, or a (integer? -> boolean?) procedure [similar to string-index, but with a more restricted form for cp.].
The :start and :end keyword arguments delimit the scan, and default to the start and end of the string; these arguments are indexes into the string, with end indicating the index one past the last character to be considered; end may be #f to indicate the end of the string.
Returns #f if the character is not present.
(ustring-join ustrings sep) : join a list of ustrings, by connecting them with the (ustring or string) separator.
(ustring-length us) : return the number of characters (ie, codepoints) in the ustring
(ustring-lowercase us) : return a new ustring with the contents converted to lowercase
(ustring-lowercase! us) : convert the ustring contents to lowercase, in place, and return the argument
(ustring-ref ustring idx) : return the codepoint (as an integer) at index idx (zero offset). Throws an out-of-range error if appropriate.
(ustring-split s c) : Split an argument, which must produce an integer iterator, at a given character. The argument will most typically be a ustring?, but anything which iterates to produce integers, such as a list of integers, is acceptable. Returns a list of ustrings. Argument c must be a string?, char? or procedure?. See the procedure string-split, to which this should be closely analogous.
(ustring-substring ustring start [end]) : return a copy of a substring of ustring, starting at index start and not including index end. Both start and end are zero-offset indexes of codepoints within the string. If end is absent or #f, then copy to the end of the string.
(ustring-titlecase us) : return a new ustring with the contents converted to titlecase
(ustring-titlecase! us) : convert the ustring contents to titlecase, in place, and return the argument
(ustring-tokenize us) : split the ustring us into a list of substrings, where each substring is a maximal non-empty contiguous sequence of characters separated by whitespace.
The ‘whitespace’ here is anything which matches char-wordbreak?.
Returns a list of ustring?.
The argument is typically a ustring?, but can be anything which has an iterator which produces integers.
(ustring-uppercase us) : return a new ustring with the contents converted to uppercase
(ustring-uppercase! us) : convert the ustring contents to uppercase, in place, and return the argument
(ustring<? u1 u2) : return #t if both arguments are of type ustring?, and u1 should be ordered before u2.
This is sensitive to the current locale/language.
(ustring=? s1 s2 [flags...]) : return true if the two arguments are ustrings or strings, and equal. Arguments of type string? are converted to ustring? before comparison.
There are no flags defined at present.
(ustring? x) : return #t if x is a ustring