Internationalization (I18N) and Localization (L10N)

Many web application built with PHP will not have internationalization in mind when it was first written. It may be that it was not intended for use in languages and cultures. Internationalization is an important aspect due to the increase adoption of the Internet in many non-English speaking countries. The process of internationalization and localization will contain difficulties. Below are some general guidelines to internationalize an existing application.

Separate culture/locale sensitive data

Identify and separate data that varies with culture. The most obvious are text/string/message. Other type of data should also be considered. The following list categorize some examples of culture sensitive data

  • Strings, Messages, Text, in relatively small units (e.g. phrases, sentences, paragraphs, but not the full text of a book).
  • Labels on buttons.
  • Help files, large units of text, static text.
  • Sounds.
  • Colors.
  • Graphics,Icons.
  • Dates, Times.
  • Numbers, Currency, Measurements.
  • Phone numbers.
  • Honorific and personal titles.
  • Postal address.
  • Page layout.

If possible all manner of text should be isolated and store in a persistence format. These text include, application error messages, hard coded strings in PHP files, emails, static HTML text, and text on form elements (e.g. buttons).

Configuration

To enable the localization features in PRADO, you need to add a few configuration options in your application configuration. First you need to include the System.I18N.* namespace to your paths.

<paths>
    <using namespace="System.I18N.*" />
</paths>

Then, if you wish to translate some text in your application, you need to add one translation message data source.

<module id="globalization" class="TGlobalization">
    <translation type="XLIFF"
        source="MyApp.messages"
        marker="@@"
        autosave="true" cache="true" />
</module>

Where source in translation is the dot path to a directory where you are going to store your translate message catalogue. The autosave attribute if enabled, saves untranslated messages back into the message catalogue. With cache enabled, translated messages are saved in the application runtime/i18n directory. The marker value is used to surround any untranslated text.

With the configuration complete, we can now start to localize your application. If you have autosave enabled, after running your application with some localization activity (i.e. translating some text), you will see a directory and a messages.xml created within your source directory.

What to do with messages.xml?

The translation message catalogue file, if using type="XLIFF", is a standardized translation message interchange XML format. You can edit the XML file using any UTF-8 aware editor. The format of the XML is something like the following.

<?xml version="1.0"?>
<xliff version="1.0">
    <file original="I18N Example IndexPage"
          source-language="EN"
          datatype="plaintext"
          date="2005-01-24T11:07:53Z">
        <body>

<trans-unit>
<source>Hello world.</source>
<target>Hi World!!!</target>
</trans-unit>

        </body>
    </file>
</xliff>
Each translation message is wrapped within a trans-unit tag, where source is the original message, and target is the translated message. Editors such as Heartsome XLIFF Translation Editor can help in editing these XML files.

Using a Database for translation

Available from Prado versions 3.1.3 onwards.

Since version 3.1.3 the messages can also be stored in a database using the connection id from an existing TDataSourceConfig. You have to create two tables in your database: catalogue and trans_unit. The catalogue table needs an entry for each catalogue you want to use. Example schemas for different databases can be found in the framework's I18N/schema directory. To configure translation with a database use:

<module id="db1" class="System.Data.TDataSourceConfig">
    <database ConnectionString="mysql:host=localhost;dbname=demodb" Username="demo" Password="demo" />
</module>

<module id="globalization" class="TGlobalization">
    <translation
        type="Database"
        autosave="true"
        cache="false"
        source="db1" />
</module>

The translation messages will be stored in the trans_unit table. Add your translation in the target field of that table. You should make sure that you are working on the right catalogue by comparing the message's cat_id with that from the catalogue table.

Setting and Changing Culture

Once globalization is enabled, you can access the globalization settings, such as, Culture, Charset, etc, using

$globalization = $this->getApplication()->getGlobalization();
echo $globalization->Culture;
$globalization->Charset= "GB-2312"; //change the charset

You also change the way the culture is determined by changing the class attribute in the module configuration. For example, to set the culture that depends on the browser settings, you can use the TGlobalizationAutoDetect class.

<module id="globalization" class="TGlobalizationAutoDetect">
   ...
</module>

You may also provide your own globalization class to change how the application culture is set. Lastly, you can change the globalization settings on page by page basis using template control tags. For example, changing the Culture to "zh".

<%@ Application.Globalization.Culture="zh" %>

Localizing your PRADO application

There are two areas in your application that may need message or string localization, in PHP code and in the templates. To localize strings within PHP, use the localize function detailed below. To localize text in the template, use the TTranslate component.

Using localize function to translate text within PHP

The localize function searches for a translated string that matches original from your translation source. First, you need to locate all the hard coded text in PHP that are displayed or sent to the end user. The following example localizes the text of the $sender (assuming, say, the sender is a button). The original code before localization is as follows.

function clickMe($sender,$param)
{
  $sender->Text="Hello, world!";
}

The hard coded message "Hello, world!" is to be localized using the localize function.

function clickMe($sender,$param)
{
  $sender->Text=Prado::localize("Hello, world!");
}

Compound Messages

Compound messages can contain variable data. For example, in the message "There are 12 users online.", the integer 12 may change depending on some data in your application. This is difficult to translate because the position of the variable data may be difference for different languages. In addition, different languages have their own rules for plurals (if any) and/or quantifiers. The following example can not be easily translated, because the sentence structure is fixed by hard coding the variable data within message.

$num_users = 12;
$message = "There are " . $num_users . " users online.";
This problem can be solved using the localize function with string substitution. For example, the $message string above can be constructed as follows.
$num_users = 12;
$message = Prado::localize("There are {num_users} users online.", array('num_users'=>$num_users));

Where the second parameter in localize takes an associative array with the key as the substitution to find in the text and replaced it with the associated value. The localize function does not solve the problem of localizing languages that have plural forms, the solution is to use TChoiceFormat.

The following sample demonstrates the basics of localization in PRADO.

I18N Components

TTranslate

Messages and strings can be localized in PHP or in templates. To translate a message or string in the template, use TTranslate.

<com:TTranslate>Hello World</com:TTranslate>
<com:TTranslate Text="Goodbye" />

TTranslate can also perform string substitution. The Parameters property can be use to add name values pairs for substitution. Substrings in the translation enclosed with "{" and "}" are consider as the parameter names during substitution lookup. The following example will substitute the substring "{time}" with the value of the parameter attribute "Parameters.time=<%= time() %>".

<com:TTranslate Parameters.time=<%= time() %> >
The time is {time}.
</com:TTranslate>

A short for TTranslate is also provided using the following syntax.

<%[string]%>

where string will be translated to different languages according to the end-user's language preference. This syntax can be used with attribute values as well.

<com:TLabel Text="<%[ Hello World! ]%>" />

TDateFormat

Formatting localized date and time is straight forward.

<com:TDateFormat Value="12/01/2005" />

The Pattern property accepts 4 predefined localized date patterns and 4 predefined localized time patterns.

  • fulldate
  • longdate
  • mediumdate
  • shortdate
  • fulltime
  • longtime
  • mediumtime
  • shorttime

The predefined can be used in any combination. If using a combined predefined pattern, the first pattern must be the date, followed by a space, and lastly the time pattern. For example, full date pattern with short time pattern. The actual ordering of the date-time and the actual pattern will be determine automatically from locale data specified by the Culture property.

<com:TDateFormat Pattern="fulldate shorttime" />

You can also specify a custom pattern using the following sub-patterns. The date/time format is specified by means of a string time pattern. In this pattern, all ASCII letters are reserved as pattern letters, which are defined as the following:

Symbol   Meaning                 Presentation        Example
 ------   -------                 ------------        -------
 G        era designator          (Text)              AD
 y        year                    (Number)            1996
 M        month in year           (Text & Number)     July & 07
 d        day in month            (Number)            10
 h        hour in am/pm (1~12)    (Number)            12
 H        hour in day (0~23)      (Number)            0
 m        minute in hour          (Number)            30
 s        second in minute        (Number)            55
 E        day of week             (Text)              Tuesday
 D        day in year             (Number)            189
 F        day of week in month    (Number)            2 (2nd Wed in July)
 w        week in year            (Number)            27
 W        week in month           (Number)            2
 a        am/pm marker            (Text)              PM
 k        hour in day (1~24)      (Number)            24
 K        hour in am/pm (0~11)    (Number)            0
 z        time zone               (Time)              Pacific Standard Time
 '        escape for text         (Delimiter)         'Date='
 ''       single quote            (Literal)           'o''clock'

The count of pattern letters determine the format.

(Text): 4 letters uses full form, less than 4, use short or abbreviated form if it exists. (e.g., "EEEE" produces "Monday", "EEE" produces "Mon")

(Number): the minimum number of digits. Shorter numbers are zero-padded to this amount (e.g. if "m" produces "6", "mm" produces "06"). Year is handled specially; that is, if the count of 'y' is 2, the Year will be truncated to 2 digits. (e.g., if "yyyy" produces "1997", "yy" produces "97".) Unlike other fields, fractional seconds are padded on the right with zero.

(Text and Number): 3 or over, use text, otherwise use number. (e.g., "M" produces "1", "MM" produces "01", "MMM" produces "Jan", and "MMMM" produces "January".)

Any characters in the pattern that are not in the ranges of ['a'..'z'] and ['A'..'Z'] will be treated as quoted text. For instance, characters like ':', '.', ' ', and '@' will appear in the resulting time text even they are not embraced within single quotes.

Examples using the US locale:

Format Pattern                         Result
--------------                         -------
"yyyy.MM.dd G 'at' HH:mm:ss"      ->>  1996.07.10 AD at 15:08:56
"EEE, MMM d, ''yy"                ->>  Wed, Jul 10, '96
"h:mm a"                          ->>  12:08 PM
"hh 'o''clock' a, z"              ->>  12 o'clock PM, Pacific Daylight Time
"K:mm a"                          ->>  0:00 PM
"yyyy.MMMM.dd G hh:mm a"          ->>  1996.July.10 AD 12:08 PM

If the Value property is not specified, the current date and time is used.

TNumberFormat

PRADO's Internationalization framework provide localized currency formatting and number formatting. Please note that the TNumberFormat component provides formatting only, it does not perform current conversion or exchange.

Numbers can be formatted as currency, percentage, decimal or scientific numbers by specifying the Type attribute. The valid types are:

  • currency
  • percentage
  • decimal
  • scientific
<com:TNumberFormat Type="currency" Value="100" />

Culture and Currency properties may be specified to format locale specific numbers.

If someone from US want to see sales figures from a store in Germany (say using the EURO currency), formatted using the german currency, you would need to use the attribute Culture="de_DE" to get the currency right, e.g. 100,00$. The decimal and grouping separator is then also from the de_DE locale. This may lead to some confusion because people from US uses the "," (comma) as thousand separator. Therefore a Currency attribute is available, so that the output from the following example results in $100.00

<com:TNumberFormat Type="currency"
          Culture="en_US" Currency="EUR" Value="100" />

The Pattern property determines the number of digits, thousand grouping positions, the number of decimal points and the decimal position. The actual characters that are used to represent the decimal points and thousand points are culture specific and will change automatically according to the Culture property. The valid Pattern characters are:

  • # (hash) - represents the optional digits
  • 0 (zero) - represents the mandatory digits, zero left filled
  • . (full stop) - the position of the decimal point (only 1 decimal point is allowed)
  • , (comma) - thousand point separation (up to 2 commas are allowed)

For example, consider the Value="1234567.12345" and with Culture="en_US" (which uses "," for thousand point separator and "." for decimal separators).

Pattern                      Output
-------                      ------
##,###.00               ->>  1,234,567.12
##,###.##               ->>  1,234,567.12345
##,##.0000              ->>  1,23,45,67.1235
##,###,##.0             ->>  12,345,67.1
000,000,000.0           ->>  001,234,567.1

TTranslateParameter

Compound messages, i.e., string substitution, can be accomplished with TTranslateParameter. In the following example, the strings "{greeting}" and "{name}" will be replace with the values of "Hello" and "World", respectively.The substitution string must be enclose with "{" and "}". The parameters can be further translated by using TTranslate.

<com:TTranslate>
  {greeting} {name}!
  <com:TTranslateParameter Key="name">World</com:TTranslateParameter>
  <com:TTranslateParameter Key="greeting">Hello</com:TTranslateParameter>
</com:TTranslate>

TChoiceFormat

Using the localize function or TTranslate component to translate messages does not inform the translator the cardinality of the data required to determine the correct plural structure to use. It only informs them that there is a variable data, the data could be anything. Thus, the translator will be unable to determine with respect to the substitution data the correct plural, language structure or phrase to use . E.g. in English, to translate the sentence, "There are {number} of apples.", the resulting translation should be different depending on the number of apples.

The TChoiceFormat component performs message/string choice translation. The following example demonstrated a simple 2 choice message translation.

<com:TChoiceFormat Value="1"/>[1] One Apple. |[2] Two Apples</com:TChoiceFormat>

In the above example, the Value "1" (one), thus the translated string is "One Apple". If the Value was "2", then it will show "Two Apples".

The message/string choices are separated by the pipe "|" followed by a set notation of the form.

  • [1,2] -- accepts values between 1 and 2, inclusive.
  • (1,2) -- accepts values between 1 and 2, excluding 1 and 2.
  • {1,2,3,4} -- only values defined in the set are accepted.
  • [-Inf,0) -- accepts value greater or equal to negative infinity and strictly less than 0

Any non-empty combinations of the delimiters of square and round brackets are acceptable. The string chosen for display depends on the Value property. The Value is evaluated for each set until the Value is found to belong to a particular set.

Available from Prado versions 3.1.1 onwards.

Since version 3.1.1 the following set notation is also possible.

  • {n: n % 10 > 1 && n % 10 < 5} -- matches numbers like 2, 3, 4, 22, 23, 24

Where set is defined by the expression after n:. In particular, the expression accepts the following mathematical/logical operators to form a set of logical conditions on the value given by n:

  • < -- less than.
  • <= -- less than equals.
  • > -- greater than.
  • >= -- greater than equals.
  • == -- of equal value.
  • % -- modulo, e.g., 1 % 10 equals 1, 11 % 10 equals 1.
  • - -- minus, negative.
  • + -- addition.
  • & -- conditional AND.
  • && -- condition AND with short circuit.
  • | -- conditional OR.
  • || -- conditional OR with short circuit.
  • ! -- negation.

Additional round brackets can also be used to perform grouping. The following example represents ordinal values in English such as: "0th", "1st", "2nd", "3rd", "4th", "11th", "21st", "22nd", etc.

<com:TChoiceFormat Value="21">
 {n: n > 0 && n % 10 == 1 && n % 100 != 11} {Value}st 
|{n: n > 0 && n % 10 == 2 && n % 100 != 12} {Value}nd 
|{n: n > 0 && n % 10 == 3 && n % 100 != 13} {Value}rd 
|{n: n > -1 } {Value}th
|(-Inf, 0) {Value}
</com:TChoiceFormat>