I am the author and maintainer of utf8rewind, a low-level library written in C that aims to add support for UTF-8 encoded text. It has functions for converting to and from UTF-8, seeking in UTF-8 encoded text and case mapping UTF-8 encoded text. One of the core tenets of the library is safety. To ensure the safety of the library, I have added almost 3000 unit, integration, property and performance tests. And on Windows 10, this one was failing:

These problems have been fixed with the release of utf8rewind 1.4.1, but I want to talk about what went wrong here.

Why I care about Azeri

The Azeri (or Azerbaijanis) language is spoken by a Turkic ethnic group living mainly in Iranian Azerbaijan and the independent Republic of Azerbaijan. I haven’t met any of them, but I hope they use my software! The Latin version of the language is named specifically in the Unicode Consortium’s SpecialCasing.txt, which I have blogged about before. The file specifies exceptions for specific languages when case mapping, including Turkish or Azeri text. Like Turkish, Azeri keeps the dot in the lowercase i when uppercasing (i → İ) and when lowercasing (İ → i).

Now, the way I check for whether we’re dealing with Azeri text is with the system locale. Unfortunately, there isn’t a single cross-platform solution for retrieving this information, but we can use “_get_current_locale()” on Windows and “setlocale(LC_ALL, 0)” on POSIX systems, which both return information about the locale and the current code page.

The POSIX implementation is simple, it only accepts two values if you want the Azeri (Latin) codec:

1
2
3
4
5
6
7

    EXPECT_STREQ("az_AZ", setlocale(LC_ALL, "az_AZ"));
    EXPECT_LOCALE_EQ(CASEMAPPING_LOCALE_TURKISH_OR_AZERI_LATIN, casemapping_locale());
    RESET_LOCALE();
 
    EXPECT_STREQ("az_AZ.utf8", setlocale(LC_ALL, "az_AZ.utf8"));
    EXPECT_LOCALE_EQ(CASEMAPPING_LOCALE_TURKISH_OR_AZERI_LATIN, casemapping_locale());
    RESET_LOCALE();

The Windows version accepts a lot more values:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

    EXPECT_STREQ("az", setlocale(LC_ALL, "az"));
    EXPECT_LOCALE_EQ(CASEMAPPING_LOCALE_TURKISH_OR_AZERI_LATIN, casemapping_locale());
    RESET_LOCALE();
 
    EXPECT_STREQ("az-Cyrl-AZ", setlocale(LC_ALL, "az-Cyrl-AZ"));
    EXPECT_LOCALE_EQ(CASEMAPPING_LOCALE_DEFAULT, casemapping_locale());
    RESET_LOCALE();
 
    EXPECT_STREQ("az-Latn-AZ", setlocale(LC_ALL, "az-Latn-AZ"));
    EXPECT_LOCALE_EQ(CASEMAPPING_LOCALE_TURKISH_OR_AZERI_LATIN, casemapping_locale());
    RESET_LOCALE();
 
    EXPECT_STREQ("Azeri_Azerbaijan.1254", setlocale(LC_ALL, "azeri"));
    EXPECT_LOCALE_EQ(CASEMAPPING_LOCALE_TURKISH_OR_AZERI_LATIN, casemapping_locale());
    RESET_LOCALE();
 
    EXPECT_STREQ("Azeri_Azerbaijan.1254", setlocale(LC_ALL, "Azeri_Azerbaijan.1254"));
    EXPECT_LOCALE_EQ(CASEMAPPING_LOCALE_TURKISH_OR_AZERI_LATIN, casemapping_locale());
    RESET_LOCALE();
 
    EXPECT_STREQ("Azeri_Azerbaijan.1254", setlocale(LC_ALL, "Azeri_Azerbaijan.ACP"));
    EXPECT_LOCALE_EQ(CASEMAPPING_LOCALE_TURKISH_OR_AZERI_LATIN, casemapping_locale());
    RESET_LOCALE();
 
    EXPECT_STREQ("Azeri_Azerbaijan.857", setlocale(LC_ALL, "Azeri_Azerbaijan.857"));
    EXPECT_LOCALE_EQ(CASEMAPPING_LOCALE_TURKISH_OR_AZERI_LATIN, casemapping_locale());
    RESET_LOCALE();
 
    EXPECT_STREQ("Azeri_Azerbaijan.857", setlocale(LC_ALL, "Azeri_Azerbaijan.OCP"));
    EXPECT_LOCALE_EQ(CASEMAPPING_LOCALE_TURKISH_OR_AZERI_LATIN, casemapping_locale());
    RESET_LOCALE();

These values come directly from the horse’s mouth, the strings are in the format “<Language>_<Country>.<CodePage>”, where the codepage can be an ANSI codepage (ACP) or OEM codepage (OCP). So far, so good. The problem was that the tests for this specific format were failing. Something was broken on Windows 10.

Finding the Culprit

To figure out what went wrong, we have to dive into the implementation of “setlocale”. Microsoft has been surprisingly generous in this regard, providing source that allows us to step into the function itself, up to the point where it starts calling CRT functions. It turns out the “setlocale” is actually a wrapper for “GetLocaleNameFromLangCountry”, which enumerates the installed locales using “EnumSystemLocalesEx”:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

/***
*void GetLocaleNameFromLangCountry - get locale names from language and country strings
*
*Purpose:
*   Match the best locale names to the language and country string given.
*   After global variables are initialized, the LangCountryEnumProcEx
*   routine is registered as an EnumSystemLocalesEx callback to actually
*   perform the matching as the locale names are enumerated.
*
*Entry:
*   pchLanguage     - language string
*   bAbbrevLanguage - language string is a three-letter abbreviation
*   pchCountry      - country string
*   bAbbrevCountry  - country string ia a three-letter abbreviation
*   iPrimaryLen     - length of language string with primary name
*
*Exit:
*   localeName - locale name of given language and country
*
*Exceptions:
*
*******************************************************************************/
static void GetLocaleNameFromLangCountry (_psetloc_struct _psetloc_data)
{
    //  initialize static variables for callback use
    _psetloc_data->bAbbrevLanguage = wcslen(_psetloc_data->pchLanguage) == 3;
    _psetloc_data->bAbbrevCountry = wcslen(_psetloc_data->pchCountry) == 3;
 
    _psetloc_data->iPrimaryLen = _psetloc_data->bAbbrevLanguage ?
                             2 : GetPrimaryLen(_psetloc_data->pchLanguage);
 
    // Enumerate all locales that come with the operating system,
    // including replacement locales, but excluding alternate sorts.
    __crtEnumSystemLocalesEx(LangCountryEnumProcEx, LOCALE_WINDOWS | LOCALE_SUPPLEMENTAL, (LPARAM) NULL);
 
    //  locale value is invalid if the language was not installed or the language
    //  was not available for the country specified
    if (!(_psetloc_data->iLocState & __LOC_LANGUAGE) ||
        !(_psetloc_data->iLocState & __LOC_EXISTS) ||
        !(_psetloc_data->iLocState & (__LOC_FULL |
                                    __LOC_PRIMARY |
                                    __LOC_DEFAULT)))
        _psetloc_data->iLocState = 0;
}

Stepping over the callback, I found out what went wrong: Microsoft changed the locale name from “Azeri” to “Azerbaijani”.

Regardless of whether that was the right call to make on a cultural level, this breaks my code. Windows 10 does not transform “Azeri_Azerbaijan.1254” into “Azerbaijani_Azerbaijan.1254” when you call setlocale, the function will simply return NULL.

The Reality of an Imperfect World

Okay, fine, this isn’t the first platform-specific bug I have to work around and it definitely isn’t the last. The bug won’t affect users, because they should be using the string “az-Latn-AZ” anyway. But it does break a unit test, so let’s detect whether we’re running Windows 10 and use different strings on that platform.

The standard way of checking what Windows version you’re running is by calling GetVersionEx and checking the “dwMajorVersion” and “dwMinorVersion” members of the resulting struct. Unfortunately, this method has been deprecated, with good reason. The short of it is that well-intentioned, but naive, developers wrote code like this:

1
2
3
4

if (verMajor >= 5 && verMinor >= 1) {
    // oh, hello, you must be Windows XP or later
    // (clearly I didn’t test this code on Windows 6.0)
}

What happens when the major version is bumped to 10? This code will disable certain features of the application, because it thinks the Windows version is too old. As a result of this improper use of the API, GetVersionEx is now forever doomed to return “Windows 8.1”. Which puts me into a bit of a pickle, because I specifically want to test for Windows 10.

Luckily, there’s an alternative! There’s a “VersionsHelpers.h”, which provides this wonderful function:

1
2
3
4
5

VERSIONHELPERAPI
IsWindows10OrGreater()
{
    return IsWindowsVersionOrGreater(HIBYTE(_WIN32_WINNT_WINTHRESHOLD), LOBYTE(_WIN32_WINNT_WINTHRESHOLD), 0);
}

It worked on my development machine, but it wouldn’t compile on my laptop. The function is part of the Windows SDK, which is not a dependency I want to include for a low-level system library. So I rolled my own:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

    bool CheckWindowsVersion(DWORD versionIdentifier)
    {
        DWORDLONG conditionMask = 
            ::VerSetConditionMask(
                ::VerSetConditionMask(
                    0, VER_MAJORVERSION, VER_GREATER_EQUAL),
                VER_MINORVERSION, VER_GREATER_EQUAL);
 
        ::OSVERSIONINFOEXW osvi = { 0 };
        osvi.dwOSVersionInfoSize = sizeof(osvi);
        osvi.dwMajorVersion = HIBYTE(versionIdentifier);
        osvi.dwMinorVersion = LOBYTE(versionIdentifier);
 
        return ::VerifyVersionInfoW(&osvi, VER_MAJORVERSION | VER_MINORVERSION, conditionMask) != FALSE;
    }

And it didn’t work.

Down the Rabbit Hole

When you call this function with “_WIN32_WINNT_WIN10”, it returns false, but calling it with “_WIN32_WINNT_WINBLUE” returns true. It looks like “VerifyVersionInfo” function has the same problem as “GetVersionEx”: it claims Windows 10 is actually Windows 8.1. What is going on here? Luckily, there’s a handy articled on MSDN titled “Targeting your application for Windows”, which helps explain the steps you need to take in order for VerifyVersionInfo to return the actual version of Windows.

All you have to do is create a manifest file which adds compatibility info for your executable, which is basically a form you have to sign that states “I hereby acknowledge the existence of Windows versions beyond 8.1.”

Then you have to add that manifest file to your linker settings and you’re good to go!

This is a pretty mind-boggling solution on several levels:

• Deprecating GetVersionEx is fine and probably the right choice. But why then cripple the new function with the same problems of the old version?
• Why is this in the manifest file? Couldn’t this be a checkbox in Visual Studio?
• Better yet, why this isn’t this behavior a flag of the hypothetical “GetRealVersionEx” function?
• How am I ever going to explain this to my users? Link to this blogpost?

Summary

Windows 10 broke my code, deprecated the ability to check for Windows versions, crippled the new way to check for versions and added several unnecessary hoops for me to jump to in order to work around its broken behavior.

I hope y’all are proud of yourselves.

For utf8rewind 1.2.0, one of my goals was to add Unicode case mapping to the library. For the most part I’ve succeeded; it’s now possible to convert any kind of UTF-8 encoded text to upper-, lower- and titlecase using functions in the library. But there were a couple of exceptions to the case mapping algorithm that I glossed over.

Implementing case mapping

When you’re looking to implement Unicode-compliant case mapping, the first resources you should grab are the ones from the Unicode Consortium itself. These are freely available from their FTP server. The most important resource is UnicodeData.txt, which is the database for every code point encoded by this version of the Unicode standard. The database is a pure ASCII-encoded text file where every line is a record and each record is split into fields with the ‘;’ character (U+003B SEMICOLON, to be exact). A record in the database looks like this:

1

0051;LATIN CAPITAL LETTER Q;Lu;0;L;;;;;N;;;;0071;

Each record has fifteen fields, but for case mapping we’re only interested in a couple of them:

• The first field, which denotes the code point’s value (in this case U+0051).
• The second field, which is the official name for the code point (LATIN CAPITAL LETTER Q)
• The thirteenth field, which is the uppercase mapping (because it’s empty, the value is the same as the code point, so U+0051)
• The fourteenth field, which is the lowercase mapping (U+0071 LATIN SMALL LETTER Q)
• The fifteenth field, which is the titlecase mapping (U+0051 again)

Because Unicode is backwards-compatible with ASCII, the first 127 code points are exactly the same in both standards. This is how Unicode can get away with describing the first 127 characters of its standard, which it needs to describe to standard itself!

Case mapping has not posed much of a problem so far. Each code point may or may not have case mapping and the converted code point is always the same length as the source code point. Or is it?

The Eszett problem

When we go beyond Basic Latin and venture into the land of Latin-1, we encounter this peculiar fellow:

1

00DF;LATIN SMALL LETTER SHARP S;Ll;0;L;;;;;N;;;;;

You might think that this means that LATIN SMALL LETTER SHARP S does not have an uppercase version, but you’d be mistaken. You would end up disappointing a lot of German, Austrian, Danish and even Belgian users!

One of the limitations of the Unicode database is that it must be backwards-compatible with itself. An older versions of Unicode specified that this code point must be converted to U+1E9E LATIN CAPITAL LETTER SHARP when uppercasing. This was to ensure that each code point would only be converted to a single other code point when doing case conversion. Unfortunately, this is wrong according to German grammar rules. They state that the uppercase version of Eszett, as they call it, should be written as SS when uppercased. In effect, there is no uppercase version of the Eszett character, but Unicode did provide one! This discrepancy was corrected in a newer version of Unicode, but it meant the addition of a special text file, called SpecialCasing.txt.

Here, we find an updated record for the LATIN SMALL LETTER SHARP S code point:

1
2
3
4

# The German es-zed is special--the normal mapping is to SS.
# Note: the titlecase should never occur in practice. It is equal to titlecase(uppercase())
 
00DF; 00DF; 0053 0073; 0053 0053; # LATIN SMALL LETTER SHARP S

Combining this data with the record in the Unicode database gives us the complete story:

• The value for the code point is U+00DF
• Its official name is LATIN SMALL LETTER SHARP S
• The lowercase mapping is U+00DF
• The titlecase mapping is U+0053 U+0073 (Ss)
• The uppercase mapping is U+0053 U+0053 (SS)

Good, we got that sorted. Most code points map to a single code point, but some don’t. These exceptions can be found in SpecialCasing.txt.

However, when we scroll down, we get to a section ominously titled “Conditional Mappings”.

Conditional mappings

While the code points specified by the Unicode standard are language- and locale-independent, some of their behavior, unfortunately, isn’t. This is where the conditional mappings come in. When I implemented case mapping for utf8rewind 1.2.0, I glossed over this section entirely. The language used was a bit too vague for my tastes and I would rather ship imperfect case mapping now than perfect case mapping later. There wasn’t an immediate backlash, but I did receive a pull request to fix my Turkish uppercase strings. This made me decide to take a closer look at these conditional mappings. Luckily, by now I had received my copy of Unicode Demystified, which explains the case mapping algorithm in greater detail.

The Sigma Exception

Which brings us to GREEK CAPITAL LETTER SIGMA. Here’s what the book has to say about this code point:

Here’s a classic example of the use of the special conditions:

1 2	03A3; 03C2; 03A3; 03A3; Final_Sigma; # GREEK CAPITAL LETTER SIGMA # 03A3; 03C3; 03A3; 03A3; # GREEK CAPITAL LETTER SIGMA

These two entries are for the Greek capital letter sigma (Σ, U+03A3). This letter has two lowercase forms: one (ς, U+03C3) that appears only at the ends of words and another (σ, U+03C3) that appears at the beginning or in the middle of a word. The FINAL_SIGMA at the end of the first entry indicates that this mapping applies only at the end of a word (and when the letter isn’t the only letter in the word). The second entry, which maps to the initial/medial form of the lowercase sigma, is commented out (begins with a #) because this mapping is in the UnicodeData.txt file.

(Source: Richard Gillam, “Character Properties and the Unicode Character Database” in “Unicode Demystified: A Practical Programmer’s Guide to the Encoding Standard”, pp. 169.)

Now the mapping makes sense! Although the exception is relatively easy to program against, it is independent of the language used for the text. This means that if a text is in Dutch (and the system locale is set to Dutch), but it includes a capital letter sigma at the end of a word, it should still be lowercased to ς instead of σ! So now my lowercase implementation looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51

    while (state.src_size > 0)
    {
        size_t converted;
 
        if (!casemapping_readcodepoint(&state))
        {
            UTF8_SET_ERROR(INVALID_DATA);
 
            return bytes_written;
        }
 
        if (state.last_code_point == 0x03A3)
        {
            uint8_t should_convert;
 
            state.last_code_point_size = codepoint_read(
                state.src,
                state.src_size,
                &state.last_code_point);
 
            should_convert = state.last_code_point_size == 0;
 
            if (!should_convert)
            {
                state.last_general_category = database_queryproperty(
                    state.last_code_point,
                    UnicodeProperty_GeneralCategory);
 
                should_convert =
                    (state.last_general_category & GeneralCategory_Letter) == 0;
            }
 
            converted = codepoint_write(
                should_convert ? 0x03C2 : 0x03C3,
                &state.dst,
                &state.dst_size);
        }
        else
        {
            converted = casemapping_execute2(&state);
        }
        
        if (!converted)
        {
            UTF8_SET_ERROR(NOT_ENOUGH_SPACE);
 
            return bytes_written;
        }
 
        bytes_written += converted;
    }

And branch prediction algorithms in processors everywhere wept.

These changes are currently part of the local-independent-case-mapping branch of utf8rewind and won’t appear in a stable release until utf8rewind 1.3.0, the next minor version.

Suppose you’re working on a space action game called “Actioneroids”, which sounds a bit like something your doctor would prescribe a cream for. You started from scratch and got something on the screen as fast as possible. You wrote some code in C++ to create a window, loaded some ship graphics and now you want to add player input.

For a first test, the player should be able to rotate the ship using the left and right arrow keys and accelerate using the up and down arrow keys.

Your first pass will probably look very similar to this:

void Player::Player(Keyboard* a_Keyboard)
	: m_Keyboard(a_Keyboard)
	, m_Position(glm::vec2(0.0f, 0.0f))
	, m_Velocity(glm::vec2(0.0f, 0.0f))
	, m_Angle(0.0f)
	, m_Speed(0.0f)
	, m_TimeCooldown(0.0f)
{
}

void Player::Tick(float a_DeltaTime)
{
	if (m_Keyboard->IsKeyPressed(VK_LEFT))
	{
		m_Angle += 3.0f * a_DeltaTime;
	}
	if (m_Keyboard->IsKeyPressed(VK_RIGHT))
	{
		m_Angle -= 3.0f * a_DeltaTime;
	}
	if (m_Keyboard->IsKeyPresed(VK_UP))
	{
		m_Speed = glm::clamp(m_Speed + (1.5f * a_DeltaTime), 0.0f, 4.5f);
	}
	if (m_Keyboard->IsKeyPresed(VK_DOWN))
	{
		m_Speed = glm::clamp(m_Speed - (1.5f * a_DeltaTime), 0.0f, 4.5f);
	}

	m_Velocity = glm::vec2(glm::cos(m_Angle), glm::sin(m_Angle)) * m_Speed;
	m_Position += m_Velocity * a_DeltaTime;
	
	if (m_TimeCooldown > 0.0f)
	{
		m_TimeCooldown -= 0.1f * a_DeltaTime;
	}
	if (m_Keyboard->IsKeyPressed(VK_SPACE))
	{
		ShootBullet(m_Position, m_Velocity);
		
		m_TimeCooldown += 3.0f;
	}
}

After running the game, it appears everything works as intended. The player can rotate and move the ship using the arrow keys and fire with the spacebar. You relax in the knowledge of a job well done.

Rebinding keys

One of those pesky designers is at your desk. He says that the player input thus far is fine, but some players prefer using “WASD”. Alright, let’s add that as well:

void Player::Tick(float a_DeltaTime)
{
	if (m_Keyboard->IsKeyPressed(VK_LEFT) || m_Keyboard->IsKeyPressed('A'))
	{
		m_Angle += 3.0f * a_DeltaTime;
	}
	if (m_Keyboard->IsKeyPressed(VK_RIGHT) || m_Keyboard->IsKeyPressed('D'))
	{
		m_Angle -= 3.0f * a_DeltaTime;
	}
	if (m_Keyboard->IsKeyPresed(VK_UP) || m_Keyboard->IsKeyPressed('W'))
	{
		m_Speed = glm::clamp(m_Speed + (1.5f * a_DeltaTime), 0.0f, 4.5f);
	}
	if (m_Keyboard->IsKeyPresed(VK_DOWN) || m_Keyboard->IsKeyPressed('S'))
	{
		m_Speed = glm::clamp(m_Speed - (1.5f * a_DeltaTime), 0.0f, 4.5f);
	}

	m_Velocity = glm::vec2(glm::cos(m_Angle), glm::sin(m_Angle)) * m_Speed;
	m_Position += m_Velocity * a_DeltaTime;
	
	if (m_TimeCooldown > 0.0f)
	{
		m_TimeCooldown -= 0.1f * a_DeltaTime;
	}
	if (m_Keyboard->IsKeyPressed(VK_SPACE))
	{
		ShootBullet(m_Position, m_Velocity);
		
		m_TimeCooldown += 3.0f;
	}
}

Oh, but some players are using weird keyboard layouts like AZERTY and DVORAK, so what we really want is the ability to remap the keys.

Alright, it looks like we have to be a bit more invasive in our refactorings. Before we begin, let’s make a list of the requirements so far:

• Player input is done using the keyboard.
• The player ship can be moved using the arrow keys.
• The player ship can be moved using another combination of keys.
• All keys should be configurable.

If we spell out the requirements like that, it becomes a bit more obvious what should be done. First, we’ll make a struct that we can use for storing key bindings.

struct KeyBinding
{
	int key_first;
	int key_second;
};

Next, we’ll define a number of these structs for each of our key bindings: left, right, up, down and shoot.

void Player::Player(Keyboard* a_Keyboard)
	: m_Keyboard(a_Keyboard)
	, m_Position(glm::vec2(0.0f, 0.0f))
	, m_Velocity(glm::vec2(0.0f, 0.0f))
	, m_Angle(0.0f)
	, m_Speed(0.0f)
	, m_TimeCooldown(0.0f)
{
	LoadDefaultKeyBindings();
}

void Player::LoadDefaultKeyBindings()
{
	m_BindingLeft.key_first = VK_LEFT;
	m_BindingLeft.key_second = 'A';
	
	m_BindingRight.key_first = VK_RIGHT;
	m_BindingRight.key_second = 'D';
	
	m_BindingUp.key_first = VK_UP;
	m_BindingUp.key_second = 'W';
	
	m_BindingDown.key_first = VK_DOWN;
	m_BindingDown.key_second = 'S';
	
	m_BindingShoot.key_first = VK_SPACE;
	m_BindingShoot.key_second = -1;
}

With a few small changes, our Player class can now support any key binding the players can think of.

void Player::Tick(float a_DeltaTime)
{
	if (m_Keyboard->IsKeyPressed(m_BindingLeft.key_first) || m_Keyboard->IsKeyPressed(m_BindingLeft.key_second))
	{
		m_Angle += 3.0f * a_DeltaTime;
	}
	if (m_Keyboard->IsKeyPressed(m_BindingRight.key_first) || m_Keyboard->IsKeyPressed(m_BindingRight.key_second))
	{
		m_Angle -= 3.0f * a_DeltaTime;
	}
	if (m_Keyboard->IsKeyPressed(m_BindingUp.key_first) || m_Keyboard->IsKeyPressed(m_BindingUp.key_second))
	{
		m_Speed = glm::clamp(m_Speed + (1.5f * a_DeltaTime), 0.0f, 4.5f);
	}
	if (m_Keyboard->IsKeyPressed(m_BindingDown.key_first) || m_Keyboard->IsKeyPressed(m_BindingDown.key_second))
	{
		m_Speed = glm::clamp(m_Speed - (1.5f * a_DeltaTime), 0.0f, 4.5f);
	}
	
	m_Velocity = glm::vec2(glm::cos(m_Angle), glm::sin(m_Angle)) * m_Speed;
	m_Position += m_Velocity * a_DeltaTime;
	
	if (m_TimeCooldown > 0.0f)
	{
		m_TimeCooldown -= 0.1f * a_DeltaTime;
	}
	if (m_Keyboard->IsKeyPressed(m_BindingShoot.key_first) || m_Keyboard->IsKeyPressed(m_BindingShoot.key_second))
	{
		ShootBullet(m_Position, m_Velocity);
		
		m_TimeCooldown += 3.0f;
	}
}

Different input methods

Marketing is at your desk this time. They say that “Actioneroids” is shaping up to be a blockbuster hit, but they’d like to release simultaneously for PC, Xbox 360 and iPhone. So the game will need to support keyboard, controller and touchscreen input methods.

Oh, and the controls will still be bindable, right?

You sputter and fluster. The game wasn’t designed for those input methods! It’s going to be a maintenance nightmare! And the keys have to be bindable?!

But you’ll give it your best shot anyway.

Let’s modify the constructor. We’ll make a new class, an InputHandler, that has handles to all input methods. We’ll pass this class to the Player class.

Player::Player(InputHandler* a_InputHandler)
	: m_InputHandler(a_InputHandler)
	, m_Position(glm::vec2(0.0f, 0.0f))
	, m_Velocity(glm::vec2(0.0f, 0.0f))
	, m_Angle(0.0f)
	, m_Speed(0.0f)
	, m_TimeCooldown(0.0f)
{
}

The Tick method now has to account for all these different input methods, running on all these platforms. It’s… not going to be pretty.

void Player::Tick(float a_DeltaTime)
{
	Keyboard* kb = m_InputHandler->GetKeyboard();
	Gamepad* gp = m_InputHandler->GetGamepad();
	Touchscreen* ts = m_InputHandler->GetTouchscreen();

#if PLATFORM_PC
	if (kb->IsKeyPressed(m_BindingLeft.key_first) || kb->IsKeyPressed(m_BindingLeft.key_second))
#elif PLATFORM_XBOX
	if (gp->IsButtonPressed(GAMEPAD_BUTTON_DPAD_LEFT))
#elif PLATFORM_IPHONE
	if (ts->IsAreaTouched(glm::vec2(20.0f, 20.0f), glm::vec2(120.0f, 120.0f))
#endif
	{
		m_Angle += 3.0f * a_DeltaTime;
	}
	
	// snipped for sanity
}

You feel dirty, but it works. On all platforms at that. Marketing loves it! And the designers too. But they are wondering if you could maybe add controller support for PC as well…?

Action mapping to the rescue

All we’ve done so far is query the state of the different input devices and react to their output. But that assumes the actions are bound in the same way. For example, if you’re using a controller to rotate the ship, you use values between 0 and 1. This allows fine-grained control of the ship’s movement. But on a keyboard, you don’t get a percentage value for a key. You get 0 (nothing) or 1 (maximum). When you don’t take this into account, you can end up with a solution that works great with a controller, but feels awful when using a keyboard and mouse.

So what is action mapping and how does it help?

When using action mapping, you check for an action, but don’t care about the input. In the example, we already have four actions: rotate left, rotate right, accelerate and decelerate. The action mapper takes the name of an action and returns a normalized value as a float. Internally, it queries the different input methods and converts their values to the expected output.

For our action names, we will use strings. But don’t feel constrained. You can use incrementing integers, hashed strings or anything else, as long as it is unique for that action.

static const std::string g_Action_Player_RotateLeft = "Action_Player_RotateLeft";
static const std::string g_Action_Player_RotateRight = "Action_Player_RotateRight";
static const std::string g_Action_Player_Accelerate = "Action_Player_Accelerate";
static const std::string g_Action_Player_Decelerate = "Action_Player_Decelerate";
static const std::string g_Action_Player_Shoot = "Action_Player_Shoot";

Note that some of the actions can be combined. A rotation to the left is negative, while a rotation to the right is positive. So a rotation can be mapped to -1…0…1. The same is true for acceleration.

Now we only need three actions:

static const std::string g_Action_Player_Rotation = "Action_Player_Rotation";
static const std::string g_Action_Player_Acceleration = "Action_Player_Acceleration";
static const std::string g_Action_Player_Shoot = "Action_Player_Shoot";

We’ll put these action names in a header called “ActionNames.h”.

Without looking at the implementation for the action mapper just yet, what will the Player class look like now? A lot simpler:

void Player::Player(ActionMapper* a_ActionMapper)
	: m_ActionMapper(a_ActionMapper)
	, m_Position(glm::vec2(0.0f, 0.0f))
	, m_Velocity(glm::vec2(0.0f, 0.0f))
	, m_Angle(0.0f)
	, m_Speed(0.0f)
	, m_TimeCooldown(0.0f)
{
}

void Player::Tick(float a_DeltaTime)
{
	m_Angle += 3.0f * m_ActionMapper->GetAction(g_Action_Player_Rotation) * a_DeltaTime;
	
	m_Speed += 1.5f * m_ActionMapper->GetAction(g_Action_Player_Acceleration) * a_DeltaTime;
	m_Speed = glm::clamp(m_Speed, 0.0f, 4.5f);
	
	m_Velocity = glm::vec2(glm::cos(m_Angle), glm::sin(m_Angle)) * m_Speed;
	m_Position += m_Velocity * a_DeltaTime;
	
	if (m_TimeCooldown > 0.0f)
	{
		m_TimeCooldown -= 0.1f * a_DeltaTime;
	}
	if (m_ActionMapper->GetAction(g_Action_Player_Shoot) > 0.0f)
	{
		ShootBullet(m_Position);
		
		m_TimeCooldown += 3.0f;
	}
}

Internally, our action mapper will ask each of its handlers: do you recognize this action? If so, what value is it? Only one of the handlers gets to decide the output value, so the order is important.

float ActionMapper::GetAction(const std::string& a_Name) const
{
	float value = 0.0f;
	
	for (std::vector<IInputHandler*>::const_iterator handler_it = m_InputHandlers.begin(); handler_it != m_InputHandlers.end(); ++handler_it)
	{
		IInputHandler* handler = *handler_it;
		
		if (handler->GetAction(a_Name, &value))
		{
			break;
		}
	}
	
	return value;
}

Let’s look at the handler for the keyboard, because it was the first one we added. The implementation for the virtual GetAction method should compare the name of the action to the ones it knows. Some actions may still be platform or input-method specific.

bool KeyboardHandler::GetAction(const std::string& a_Name, float& a_Value)
{
	if (a_Name == g_Action_Player_Rotation)
	{
		if (m_Keyboard->IsKeyPressed(m_BindingLeft.key_first) || m_Keyboard->IsKeyPressed(m_BindingLeft.key_second))
		{
			a_Value = -1.0f;
		}
		else if (m_Keyboard->IsKeyPressed(m_BindingRight.key_first) || m_Keyboard->IsKeyPressed(m_BindingRight.key_second))
		{
			a_Value = 1.0f;
		}
		
		return true;
	}
	else if (a_Name == g_Action_Player_Acceleration)
	{
		if (m_Keyboard->IsKeyPressed(m_BindingUp.key_first) || m_Keyboard->IsKeyPressed(m_BindingUp.key_second))
		{
			a_Value = 1.0f;
		}
		else if (m_Keyboard->IsKeyPressed(m_BindingDown.key_first) || m_Keyboard->IsKeyPressed(m_BindingDown.key_second))
		{
			a_Value = -1.0f;
		}
		
		return true;
	}
	else if (a_Name == g_Action_Player_Shoot)
	{
		if (m_Keyboard->IsKeyPressed(m_BindingShoot.key_first) || m_Keyboard->IsKeyPressed(m_BindingShoot.key_second))
		{
			a_Value = 1.0f;
		}
		
		return true;
	}
	else
	{
		return false;
	}
}

It looks very similar to our earlier incarnation, doesn’t it? Note that even if a button is not pressed, the method returns true. That’s because the return value indicates “hey I know this action!” instead of “the user is doing this action”.

The major advantage is that it is now extremely easy to add a new input method. Simply build a new InputHandler and add it to the action mapper.

It’s not a silver bullet

You know how these posts go. This is a typical “I found a hammer, now everything can be treated as a nail!” post. I’m here to tell you that that is not true. There are distinct and clear disadvantages you must consider before implementing action mapping.

It’s a performance hit

You can’t expect to get the same performance when you’re comparing a string (an action) every frame instead of looking up a boolean (key pressed). It can be mitigated by comparing unique identifiers instead of strings, but you’ll still have to evaluate every incoming action request.

It’s more work

Games have been built and shipped with direct input mapping. It’s not a huge sin to use it. If you only plan to support keyboard and mouse for instance, it’s a lot of wasted effort to abstract that away behind a tree of interfaces.

It’s harder to debug

When you’re doing direct input mapping, it’s easy to set a breakpoint and inspect the value of an input. Was button A pressed? Yes, so says the KeyboardHandler. But when you use action mapping, it’s a lot harder to find your action in a sea of unrelated ones. The best approach is divide-and-conquer: split the GetAction method into multiple submethods, which only expect a small range of actions.

Not all input can be mapped in the same manner

In our game, we could have a guided missile. With the controller, you guide the missile using the right stick. When using mouse and keyboard, the missile homes in on the cursor position. Obviously, these actions cannot be mapped in the same manner. The controller uses a velocity for the missile to steer it, while the mouse sets the position to home in directly.

For these situations, it is often best to have two sets of actions, where each set is implemented by one input method, but ignored by the other.

Conclusion

Even with these downsides, I hope I’ve shown with a clear and concrete example what the benefits are: it’s easier to add new types of input, which means it’s easier to port to other platforms.

I promised I’d post more often and I guess I’m keeping my promise by posting… twice a month!

For the past week-and-a-half I’ve been focusing on the interface of my game, I have been doing research and searching other gaming websites to get more clues on the needs of the game, take a look at the interface of this casino games, they are wildly popular. Because I’m targeting such an old platform, I pretty much have to implement everything myself. But that’s okay, because it’s fun! Although I can’t use it directly, I’m using the Gameplay engine as inspiration and also blatantly steal the bits I like.

The reason I’m putting the focus on my interface is because it looks like this:

I can render text and some buttons, but that’s pretty much it. Moving on, I’ll have to put in some more engine work in order to support all the features I want. For example, I want to be able to load my entire interface from a text file and connect the events (button clicked, text entered, etc.) to my game logic. To learn more about modern games, visit https://farmingless.com/why-are-dota-2-skins-so-expensive/.

Additionally, I want the ability to specify a margin, border and padding for my controls. This is known as the box model and it comes from HTML. It’s really easy to explain too:

• Everything element on the screen is put into a box.
• Boxes are placed together as closely as possible without overlapping.
• Boxes can be placed inside other boxes.
• The distance between boxes is known as the margin.
• The inside of the box is known as the content.
• The edges around the box is known as the border.
• The distance between the border and the content is known as the padding

Using these simple rules, you can build pretty much any interface you want!

The next thing on my wishlist was the nine-patch. This is a box with a border in which the corners don’t deform when you stretch the content area.

The most difficult part was figuring out how I was going to store all this information. Ultimately I looked at how gameplay does it and copied that. The result:

Terrible programming art aside, it works! Next up, I attacked the text rendering. It’s a small change, but I can now center the text horizontally and vertically:

I also cleaned up the rendering in general. I am now able to draw text with multiple lines (!) and the code is general enough to fit in the base “Engine” project instead of the Windows Mobile 6-specific renderer. Unfortunately, working for yourself means you have to be tough. So the old text rendering code is still in there, but will be removed “eventually”.

This post has gotten too damn long again. I’m doing so much exciting stuff, I need to talk about it more!

On January the 24th I made the first commit. I had decided to finally build a game on my favorite crazy platform: Windows Mobile 6.1. Go take a look at this gaming website.

Windows Mobile is exactly as the name applies: Windows for your phone. And it programs like that too. You create a window with “CreateWindow”, you can render using GDI, DirectDraw or Direct3D and sound is done using MMSYSTEM or DirectSound. Programming for this platform is a lot like programming for Windows.

However, it is a mobile platform so people can play on the go as they do with geeksscan online slots. And it is old. My development device is a HP iPAQ PDA. Remember those? You’d lug them around to keep track of your day-by-day planning and they had wifi so you could check your e-mail. Eventually they would get replaced, first by Blackberries and finally by smartphones in general.

The device has, I estimate, a 300 MHz processor and 64 MB of RAM. It has a 240 x 320 pixel screen without multi-touch. You need to use a supplied pen to get it to actually register a click. And although it doesn’t sound like a beast of a machine, it’s enough to run Age of Empires II and a crappy version of Call of Duty.

So why pick this platform? Because it’s a challenge. I’ve been meaning to make something for it ever since I first got it. I have the skills, all I need now is persistence. Every day, before work, I launch Visual Studio and try to get something done. Last week I worked on level loading. I was using a hardcoded level definition that just didn’t cut it anymore. So I turned to my favorite serialization library: Protobuf.

It was a bit of a pain to setup, but it works! I can use protobuf-lite on Windows Mobile!

So naturally, after finishing the level loader, I converted my font loading to Protobuf too. And my sprites as well…

I got a bit carried away is all I’m sayin’.

My plan is to keep you posted by posting small updates every few days. I have a tendency to write gargantuan blog posts and those take a loooooong time to write. So by keeping it short I hope to increase frequency in posting. ;)