utf8
Note: This is taken from the Chicken Wiki, where a more recent version could be available.
Unicode support
Alex Shinn
To make your code Unicode aware, just do the following:
(require-extension utf8)
then all core, extra and regex string operations will be Unicode aware. string-length will return the number of codepoints, not the number of bytes, string-ref will index by codepoints and return a char with an integer value up to 2^21, regular expressions will match single codepoints rather than bytes and understand Unicode character classes, etc.
Strings are still native strings and may be passed to external libraries (either Scheme or foreign) perfectly safely. Libraries that do parsing invariably do so on ASCII character boundaries and are thus guaranteed to be compatible. Libraries that reference strings by index would need to be modified with a UTF-8 version. Currently all existing eggs are UTF-8 safe to my knowledge.
To make a compiled library optionally Unicode aware, so that it will honor utf8 semantics if and only if the utf8 egg has been loaded into the top-level or by some other extension, you need to
(declare (not usual-integrations))
To use Unicode-aware SRFI-13 and SRFI-14 using UTF-8 semantics:
(require-extension utf8-srfi-13) (require-extension utf8-srfi-14)
The utf8 egg provides a 'utf8-strings feature which can be used to conditionally support such utf8 extensions:
(cond-expand (utf8-strings (use utf8-srfi-13 utf8-srfi-14)) (else (use srfi-13 srfi-14)))
The SRFI-14 module provides an alternative to the standard Chicken SRFI-14. As a pure superset which handles arbitrary-sized characters it should be usable as a drop-in replacement. The only aspect related to UTF-8 is STRING→CHAR-SET assumes the string is UTF-8 encoded.
The default SRFI-14 char-sets are defined using ASCII-only characters, since this is both useful and lighter-weight. To obtain full Unicode char-set definitions, use the unicode-char-sets unit:
(require-extension unicode-char-sets)
[Note this is the only extension in this egg with a unicode- prefix, because the char-set handling only depends on individual characters and is independent of the character encoding used in strings.]
The following char-sets are provided based on the Unicode properties:
char-set:alphabetic char-set:arabic char-set:armenian char-set:ascii-hex-digit char-set:bengali char-set:bidi-control char-set:bopomofo char-set:braille char-set:buhid char-set:canadian-aboriginal char-set:cherokee char-set:common char-set:cypriot char-set:cyrillic char-set:dash char-set:default-ignorable-code-point char-set:deprecated char-set:deseret char-set:devanagari char-set:diacritic char-set:ethiopic char-set:extender char-set:georgian char-set:gothic char-set:grapheme-base char-set:grapheme-extend char-set:grapheme-link char-set:greek char-set:gujarati char-set:gurmukhi char-set:han char-set:hangul char-set:hanunoo char-set:hebrew char-set:hex-digit char-set:hiragana char-set:hyphen char-set:id-continue char-set:id-start char-set:ideographic char-set:ids-binary-operator char-set:ids-trinary-operator char-set:inherited char-set:join-control char-set:kannada char-set:katakana char-set:katakana-or-hiragana char-set:khmer char-set:lao char-set:latin char-set:limbu char-set:linear-b char-set:logical-order-exception char-set:lowercase char-set:malayalam char-set:math char-set:mongolian char-set:myanmar char-set:noncharacter-code-point char-set:ogham char-set:old-italic char-set:oriya char-set:osmanya char-set:quotation-mark char-set:radical char-set:runic char-set:shavian char-set:sinhala char-set:soft-dotted char-set:sterm char-set:syriac char-set:tagalog char-set:tagbanwa char-set:tai-le char-set:tamil char-set:telugu char-set:terminal-punctuation char-set:thaana char-set:thai char-set:tibetan char-set:ugaritic char-set:unified-ideograph char-set:uppercase char-set:variation-selector char-set:white-space char-set:xid-continue char-set:xid-start char-set:yi
The SRFI-13 case-mapping procedures (string-upcase, etc.) are defined using only ASCII case-mappings, since this is both useful and lighter-weight. To get full Unicode aware case-mappings, do
(require-extension utf8-case-map)
which provides the upcase, downcase, and titlecase procedures. These take a first argument of either a string or port, and an optional second argument of locale (as a string), returning the appropriate locale-aware case-mapped string.
Sometimes you may need access to the original string primitives so you can directly access bytes, such as if you were implementing your own regex library or text buffer and wanted optimal performance. For these cases we have renamed the original primitives by replacing string with byte-string. Thus byte-string-length is the length in bytes, not characters, of the strings (the equivalent of Gauche's string-size). byte-string-set! can corrupt the UTF-8 encoding and should be used sparingly if at all.
Direct manipulation of the utf8 encoding is factored away in the utf8-lolevel unit. This includes an abstract string-pointer API, and an analogous string-pointer implementation for ASCII strings in the string-pointer unit, however as the API is not fixed you use these at your own risk.
peek-char currently does not have Unicode semantics (i.e. it peeks only a single byte) to avoid problems with port buffering.
char-sets are not interchangeable between the existing srfi-14 code and Unicode code (i.e. do not pass a Unicode char-set to an external library that directly uses the old srfi-14).
Attempting to mutate literal strings will result in an error if the mutated size does not occupy the same number of bytes as the original. This is standards compliant, since the programmer is not supposed to attempt to mutate literal values, but it may be a little confusing since the error is inconsistent.
string-length, string-ref and string-set! are all O(n) operations as opposed to the usual O(1) since UTF-8 is a variable width encoding. Use of these should be discouraged - it is much cleaner to use the high-level SRFI-13 procedures and string ports. For examples of how to do common idioms without these procedures look at any string-based code in Gauche.
Furthermore, string-set! and other procedures that modify strings in place may invoke gc if the mutated result does not fit within the same UTF-8 encoding size as the original string. If only mutating 7-bit ASCII strings (or only mutating within fixed encoding sizes such as Cyrillic→Cyrillic) then no gc will occur.
string?, string=?, string-append, all R5RS string comparisons, and read-line are unmodified.
Regular expression matching will be just as fast except in the case of Unicode character classes (which were not possible before anyway).
All other procedures incur zero to minor overhead, but keep the same asymptotic performance.
There are two ways to add Unicode string support to an existing language: redefine the strings themselves (i.e. add a new string type), or redefine the operations on the strings. The former causes a schism in your string libraries, dividing them between Unicode-aware and not, either doubling your library implementations or limiting them to one type or the other. You can't freely pass strings to other libraries without keeping track of their types and converting when needed. It becomes slow and unwieldy. C and Perl are the only language I know of who seriously tried this. In Perl the modules which worked with Unicode strings were minimal, frequent type conversions were needed, a general mess ensued, and Perl very quickly switched to the latter approach. In C as well, the libraries supporting wchar are still minimal, while most libraries still only support char.
UTF-8 is ideal for the in-place sort of extension because it is backwards compatible with ASCII. Any ASCII (7-bit) byte found within a UTF-8 string is guaranteed to be that character, not part of a multibyte character, so parsing libraries that work on ASCII characters work unmodified. This includes most existing text formats and network protocols. The EUC (Extended Unix Code) encodings also have this feature so a similar module could be implemented allowing users to (require 'euc-jp) for example and work in Japanese EUC rather than Unicode. Other encodings such as Shift_JIS satisfy the requirement that an ASCII string has the same meaning in the encoding, but multibyte characters in the encoding may include ASCII bytes, breaking the rule we need for safe ASCII parsing. A few encodings like UTF-16 and UTF-32 are completely incompatible. UTF-16 is primarily only used these days by Java, a victim of the unfortunate fact that at first UTF-16 was fixed with but is no longer with the advent of surrogate pairs. Note that even without this module you can write source code in Chicken in any ASCII compatible encoding like ISO-8859-* or UTF-8 and define symbols with that encoding (letting you replace lambda with syntax for a real greek lambda, for example).
Other languages that use UTF-8 include Perl, Python, TCL. XML and increasingly more and more network standards are using UTF-8 by default, and major databases all support UTF-8. Libraries with UTF-8 support include Gtk, SDL, and freetype.
Copyright (c) 2004-2008, Alex Shinn All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
Redistributions of source code must retain the above copyright notice, this list of conditions and the following
disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following
disclaimer in the documentation and/or other materials provided with the distribution.
Neither the name of the author nor the names of its contributors may be used to endorse or promote
products derived from this software without specific prior written permission.THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.