| TOC |
|
By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79.
Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet-Drafts.
Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as “work in progress.”
The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt.
The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html.
This Internet-Draft will expire on November 14, 2005.
Copyright © The Internet Society (2005).
This document describes different mechanisms for comparing and matching the tags for the identification of languages defined by [RFC 3066bis] (Phillips, A., Ed. and M. Davis, Ed., “Tags for the Identification of Languages (Internet-Draft),” February 2005.)[1]. Possible algorithms for language negotiation and content selection are described. Portions of this document obsolete [RFC 3066] (Alvestrand, H., “Tags for the Identification of Languages,” January 2001.)[19].
1.
Introduction
2.
The Language Range
2.1
Basic Language Range
2.1.1
Matching
2.1.2
Lookup
2.2
Extended Language Range
2.2.1
Extended Range Matching
2.2.2
Extended Range Lookup
2.2.3
Scored Matching
2.3
Meaning of Language Tags and Ranges
2.4
Choosing Between Alternate Matching Schemes
2.5
Considerations for Private Use Subtags
3.
IANA Considerations
4.
Changes
5.
Security Considerations
6.
Character Set Considerations
7.
References
7.1
Normative References
7.2
Informative References
§
Authors' Addresses
A.
Acknowledgements
§
Intellectual Property and Copyright Statements
| TOC |
Human beings on our planet have, past and present, used a number of languages. There are many reasons why one would want to identify the language used when presenting or requesting information.
Information about a user's language preferences commonly needs to be identified so that appropriate processing can be applied. For example, the user's language preferences in a browser can be used to select web pages appropriately. A choice of language preference can also be used to select among tools (such as dictionaries) to assist in the processing or understanding of content in different languages.
Given a set of language identifiers, such as those defined in RFC3066bis, various mechanisms can be envisioned for performing language negotiation and tag matching. The suitability of a particular mechanism to a particular application depends on the needs of that application.
This document defines language ranges and syntax for specifying user preferences in a request for language content. It also specifies a default algorithm for matching language ranges to content (language tags), as well as alternate mechanisms suitable for certain applications.
The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in [RFC 2119] (Bradner, S., “Key words for use in RFCs to Indicate Requirement Levels,” March 1997.)[5].
| TOC |
Language Tags are used to identify the language of some information item or content. Applications that use language tags are often faced with the problem of identifying sets of content that share certain language attributes. For example, HTTP 1.1 (Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P., and T. Berners-Lee, “Hypertext Transfer Protocol -- HTTP/1.1,” June 1999.)[10] describes language ranges in its discussion of the Accept-Language header (Section 14.4), which is used for selecting content from servers based on the language of that content.
When selecting content according to its language, it is useful to have a mechanism for identifying sets of language tags that share specific attributes. This allows users to select or filter content based on specific requirements. Such an identifier is called a "Language Range".
A basic language range (such as described in RFC 3066 (Alvestrand, H., “Tags for the Identification of Languages,” January 2001.)[19] and HTTP 1.1 (Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P., and T. Berners-Lee, “Hypertext Transfer Protocol -- HTTP/1.1,” June 1999.)[10]) is a set of languages whose tags all begin with the same sequence of subtags. A basic language range can be represented by a 'language-range' tag, by using the definition from HTTP/1.1 (Fielding, R., Gettys, J., Mogul, J., Frystyk, H., Masinter, L., Leach, P., and T. Berners-Lee, “Hypertext Transfer Protocol -- HTTP/1.1,” June 1999.)[10] :
language-range = language-tag / "*"
That is, a language-range has the same syntax as a language-tag or is the single character "*". This definition of language-range implies that there is a semantic relationship between tags that share the same prefix.
In particular, the set of language tags that match a specific language-range may not all be mutually intelligible. The use of a prefix when matching tags to language ranges does not imply that language tags are assigned to languages in such a way that it is always true that if a user understands a language with a certain tag, then this user will also understand all languages with tags for which this tag is a prefix. The prefix rule simply allows the use of prefix tags if this is the case.
When working with tags and ranges you should also note the following:
There are two matching schemes that are commonly associated with basic language ranges: matching and lookup.
Language tag matching is used to select all content that matches a given prefix. In matching, the language range represents the least specific tag which is an acceptable match and every piece of content that matches is returned.
For example, if an application is applying a style to all content in a web page in a particular language, it might use language tag matching to perform the matching.
A language-range matches a language-tag if it exactly equals the tag, or if it exactly equals a prefix of the tag such that the first character following the prefix is "-". (That is, the language-range "en-de" matches the language tag "en-DE-boont", but not the language tag "en-Deva".)
The special range "*" matches any tag. A protocol which uses language ranges may specify additional rules about the semantics of "*"; for instance, HTTP/1.1 specifies that the range "*" matches only languages not matched by any other range within an "Accept-Language:" header.
Content lookup is used to select the single information item that best matches the language range for a given request. In lookup, the language range represents the most specific tag which is an acceptable match and only the closest matching item is returned.
For example, if an application inserts some dynamic content into a web page, returning an empty string if there is no exact match is not an option. Instead, the application "falls back".
When performing lookup, the language range is
progressively truncated from the end until a matching piece of content is located. For example,
starting with the range "zh-Hant-CN-x-wadegile", the lookup would progressively search for content as shown below:
Range to match: zh-Hant-CN-x-wadegile 1. zh-Hant-CN-x-wadegile 2. zh-Hant-CN 3. zh-Hant 4. zh 5. (default content or the empty tag)
| Figure 2: Default Fallback Pattern Example |
Prefix matching using a Basic Language Range, as described above, is not always the most appropriate way to access the information contained in language tags when selecting or filtering content. Some applications may wish to define a more granular matching scheme and such a matching scheme requires the ability to specify the various attributes of a language tag in the language range. An extended language range can be represented by the following ABNF:
extended-language-range = grandfathered / privateuse / range
range = ( lang [ "-" script ] [ "-" region ] *( "-" variant )
[ "-" privateuse ] )
lang = ( 2*8ALPHA *[ *( "-" extlang ] ) ) / "*"
extlang = 3ALPHA / "*"
script = 4ALPHA / "*"
region = 2ALPHA / 3DIGIT / "*"
variant = 5*8alphanum / ( DIGIT 3alphanum ) / "*"
privateuse = ( "x" / "X" ) 1*( "-" ( 1*8alphanum ) )
grandfathered = 1*3ALPHA 1*2( "-" ( 2*8alphanum ) )
alphanum = ( ALPHA / DIGIT )
In an extended language range, the identifier takes the form of a series of subtags which must consist of well-formed subtags or the special subtag "*". For example, the language range "en-*-US" specifies a primary language of 'en', followed by any script subtag, followed by the region subtag 'US'.
A field not present in the middle of an extended language range MAY be treated as if the field contained a "*". For example, the range "en-US" MAY be considered to be equivalent to the range "en-*-US".
There are several matching algorithms or schemes which may be applied when matching extended language ranges to language tags.
In extended range matching, the subtags in a language tag are compared to the corresponding subtags in the extended language range. A subtag is considered to match if it exactly matches the corresponding subtag in the range or the range contains a subtag with the value "*" (which matches all subtags, including the empty subtag). Extended Range Matching is an extension of basic matching (Section 2.1.1 (Matching)): the language range represents the least specific tag which is an acceptable match.
By default all extensions and their subtags are ignored for extended language range matching.
Private use subtags may be specified in the language range and MUST NOT be ignored when matching.
Subtags not specified, included those at the end of the language range, are assigned the value "*". This makes each range into a prefix much like that used in basic language range matching. For example, the extended language range "zh-*-CN" matches all of the following tags because the unspecified variant field is expanded to "*":
zh-Hant-CN
zh-CN
zh-Hans-CN
zh-CN-x-wadegile
zh-Latn-CN-boont
In extended range lookup, the subtags in a language tag are compared to the corresponding subtags in the extended language range. The subtag is considered to match if it exactly matches the corresponding subtag in the range or the range contains a subtag with the value "*" (which matches all subtags, including the empty subtag). Extended language range lookup is an extension of basic lookup (Section 2.1.2 (Lookup)): the language range represents the most specific tag which will form an acceptable match.
Subtags not specified are assigned the value "*" prior to performing tag matching. Unlike in extended range matching, however, fields at the end of the range MUST NOT be expanded in this manner. For example, "en-US" must not be considered to be the same as the range "en-US-*". This allows ranges to be specific. The "*" wildcard MUST be used at the end of the range to indicate that all tags with the range as a prefix are allowable matches. That is, the range "zh-*" matches the tags "zh-Hant" and "zh-Hant-CN", while the range "zh" matches neither of those tags.
The wildcard "*" at the end of a range SHOULD be considered to match any private use subtag sequences (making extended language range lookup function exactly like extended range matching Section 2.2.1 (Extended Range Matching)).
By default all extensions and their subtags SHOULD be ignored for extended language range lookup. Private use subtags may be specified in the language range and MUST NOT be ignored when performing lookup. The wildcard "*" at the end of a range SHOULD be considered to match any private use subtag sequences in addition to variants.
For example, the range "*-US" matches all of the following tags:
en-US
en-Latn-US
en-US-r-extends (extensions are ignored)
fr-US
For example, the range "en-*-US" matches none of the following tags:
fr-US
en (missing region US)
en-Latn (missing region US)
en-Latn-US-scouse (variant field is present)
For example, the range "en-*" matches all of the following tags:
en-Latn
en-Latn-US
en-Latn-US-scouse
en-US
en-scouse
It should be noted that the ability to be specific in extended range lookup may make this matching scheme a more appropriate replacement for basic matching than the extended range matching scheme.
In the "scored matching" scheme, the extended language range and the language tags are pre-normalized by mapping grandfathered and obsolete tags into modern equivalents.
The language range and the language tags are normalized into quadruples of the form (language, script, country, variant), where extended language is considered part of language and x-private-codes are considered part of the language if they are initial and part of the variant if not initial. Missing components are set to "*". An "*" pattern becomes the quadruple ("*", "*", "*", "*").
Each language tag being matched or filtered is assigned a "quality value" such that higher values indicate better matches and lower values indicate worse ones. If the language matches, add 8 to the quality value. If the script matches, add 4 to the quality value. If the region matches, add 2 to the quality value. If the variant matches, add 1 to the quality value. Elements of the quadruples are considered to match if they are the same or if one of them is "*".
A value of 15 is a perfect match; 0 is no match at all. Different values may be more or less appropriate for different applications and implementations should probably allow users to choose the most appropriate selection value.
A language tag defines a language as spoken (or written, signed or otherwise signaled) by human beings for communication of information to other human beings.
If a language tag B contains language tag A as a prefix, then B is typically "narrower" or "more specific" than A. For example, "zh-Hant-TW" is more specific than "zh-Hant".
This relationship is not guaranteed in all cases: specifically, languages that begin with the same sequence of subtags are NOT guaranteed to be mutually intelligible, although they may be. For example, the tag "az" shares a prefix with both "az-Latn" (Azerbaijani written using the Latin script) and "az-Cyrl" (Azerbaijani written using the Cyrillic script). A person fluent in one script may not be able to read the other, even though the text might be otherwise identical. Content tagged as "az" most probably is written in just one script and thus might not be intelligible to a reader familiar with the other script.
The relationship between the tag and the information it relates to is defined by the standard describing the context in which it appears. Accordingly, this section can only give possible examples of its usage.
Implementations MAY choose to implement different styles of matching for different kinds of processing. For example, an implementation could treat an absent script subtag as a "wildcard" field; thus "az-AZ" would match "az-AZ", "az-Cyrl-AZ", "az-Latn-AZ", etc. but not "az" (this is extended range lookup). If one item is to be chosen, the implementation could pick among those matches based on other information, such as the most likely script used in the language/region in question or the script used by other content selected.
Because the primary language subtag cannot be absent in a language tag, the 'UND' subtag may sometimes be used as a 'wildcard' in basic matching. For example, in a query where you want to select all language tags that contain 'Latn' as the script code and 'AZ' as the region code, you could use the range "und-Latn-AZ". This requires an implementation to examine the actual values of the subtags, though. The matching schemes described elsewhere in this document do not require implementations to examine the values supplied and, except for scored matching, they do not require access to the Language Subtag Registry nor the use of valid subtags in language tags or ranges. This has great benefit for speed and simplicity of implementation.
Implementations may also wish to use semantic information external to the langauge tags when performing fallback. For example, the primary language subtags 'nn' (Nynorsk Norwegian) and 'nb' (Bokmal Norwegian) might both be usefully matched to the more general subtag 'no' (Norwegian). Or an application might infer that content labeled "zh-CN" is morely likely to match the range "zh-Hans" than equivalent content labeled "zh-TW".
Private-use subtags require private agreement between the parties that intend to use or exchange language tags that use them and great caution should be used in employing them in content or protocols intended for general use. Private-use subtags are simply useless for information exchange without prior arrangement.
The value and semantic meaning of private-use tags and of the subtags used within such a language tag are not defined. Matching private use tags using language ranges or extended language ranges may result in unpredictable content being returned.
| TOC |
This document presents no new or existing considerations for IANA.
| TOC |
This is the first version of this document. Changes from the reference work (draft-phillips-matching-00) are too numerious to record.
| TOC |
The only security issue that has been raised with language tags since the publication of RFC 1766, which stated that "Security issues are believed to be irrelevant to this memo", is a concern with language ranges used in content negotiation - that they may be used to infer the nationality of the sender, and thus identify potential targets for surveillance.
This is a special case of the general problem that anything you send is visible to the receiving party. It is useful to be aware that such concerns can exist in some cases.
The evaluation of the exact magnitude of the threat, and any possible countermeasures, is left to each application protocol.
Although the specification of valid subtags for an extension MUST be available over the Internet, implementations SHOULD NOT mechanically depend on it being always accessible, to prevent denial-of-service attacks.
| TOC |
The syntax in this document requires that language ranges use only the characters A-Z, a-z, 0-9, and HYPHEN-MINUS legal in language tags. These characters are present in most character sets, so presentation of language tags should not have any character set issues.
Rendering of characters based on the content of a language tag is not addressed in this memo. Historically, some languages have relied on the use of specific character sets or other information in order to infer how a specific character should be rendered (notably this applies to language and culture specific variations of Han ideographs as used in Japanese, Chinese, and Korean). When language tags are applied to spans of text, rendering engines may use that information in deciding which font to use in the absence of other information, particularly where languages with distinct writing traditions use the same characters.
| TOC |
| TOC |
| TOC |
| [13] | International Organization for Standardization, “ISO 639-1:2002, Codes for the representation of names of languages -- Part 1: Alpha-2 code,” ISO Standard 639, 2002. |
| [14] | International Organization for Standardization, “ISO 639-2:1998 - Codes for the representation of names of languages -- Part 2: Alpha-3 code - edition 1,” August 1988. |
| [15] | ISO TC46/WG3, “ISO 15924:2003 (E/F) - Codes for the representation of names of scripts,” January 2004. |
| [16] | International Organization for Standardization, “Codes for the representation of names of countries, 3rd edition,” ISO Standard 3166, August 1988. |
| [17] | Statistical Division, United Nations, “Standard Country or Area Codes for Statistical Use,” UN Standard Country or Area Codes for Statistical Use, Revision 4 (United Nations publication, Sales No. 98.XVII.9, June 1999. |
| [18] | Alvestrand, H., “Tags for the Identification of Languages,” RFC 1766, March 1995. |
| [19] | Alvestrand, H., “Tags for the Identification of Languages,” BCP 47, RFC 3066, January 2001. |
| [20] | Klyne, G. and C. Newman, “Date and Time on the Internet: Timestamps,” RFC 3339, July 2002. |
| TOC |
| Addison Phillips (editor) | |
| Quest Software | |
| Email: | addison dot phillips at quest dot com |
| Mark Davis (editor) | |
| IBM | |
| Email: | mark dot davis at ibm dot com |
| TOC |
Any list of contributors is bound to be incomplete; please regard the following as only a selection from the group of people who have contributed to make this document what it is today.
The contributors to RFC 3066 and RFC 1766, the precursors of this document, made enormous contributions directly or indirectly to this document and are generally responsible for the success of language tags.
The following people (in alphabetical order) contributed to this document or to RFCs 1766 and 3066:
Glenn Adams, Harald Tveit Alvestrand, Tim Berners-Lee, Marc Blanchet, Nathaniel Borenstein, Eric Brunner, Sean M. Burke, Jeremy Carroll, John Clews, Jim Conklin, Peter Constable, John Cowan, Mark Crispin, Dave Crocker, Martin Duerst, Michael Everson, Doug Ewell, Ned Freed, Tim Goodwin, Dirk-Willem van Gulik, Marion Gunn, Joel Halpren, Elliotte Rusty Harold, Paul Hoffman, Richard Ishida, Olle Jarnefors, Kent Karlsson, John Klensin, Alain LaBonte, Eric Mader, Keith Moore, Chris Newman, Masataka Ohta, George Rhoten, Markus Scherer, Keld Jorn Simonsen, Thierry Sourbier, Otto Stolz, Tex Texin, Andrea Vine, Rhys Weatherley, Misha Wolf, Francois Yergeau and many, many others.
Very special thanks must go to Harald Tveit Alvestrand, who originated RFCs 1766 and 3066, and without whom this document would not have been possible. Special thanks must go to Michael Everson, who has served as language tag reviewer for almost the complete period since the publication of RFC 1766. Special thanks to Doug Ewell, for his production of the first complete subtag registry, and his work in producing a test parser for verifying language tags.
For this particular document, John Cowan originated the scheme described in Section 2.2.3 (Scored Matching). Mark Davis originated the scheme described in the Section 2.1.2 (Lookup).
| TOC |
The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org.
This document and the information contained herein are provided on an “AS IS” basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Copyright © The Internet Society (2005). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights.
Funding for the RFC Editor function is currently provided by the Internet Society.