Windows 10 Broke My Unit Test
I am the author and maintainer of utf8rewind, a low-level library written in C that aims to add support for UTF-8 encoded text. It has functions for converting to and from UTF-8, seeking in UTF-8 encoded text and case mapping UTF-8 encoded text. One of the core tenets of the library is safety. To ensure the safety of the library, I have added almost 3000 unit, integration, property and performance tests. And on Windows 10, this one was failing:
These problems have been fixed with the release of utf8rewind 1.4.1, but I want to talk about what went wrong here.
Why I care about Azeri
The Azeri (or Azerbaijanis) language is spoken by a Turkic ethnic group living mainly in Iranian Azerbaijan and the independent Republic of Azerbaijan. I haven’t met any of them, but I hope they use my software! The Latin version of the language is named specifically in the Unicode Consortium’s SpecialCasing.txt, which I have blogged about before. The file specifies exceptions for specific languages when case mapping, including Turkish or Azeri text. Like Turkish, Azeri keeps the dot in the lowercase i when uppercasing (i → İ) and when lowercasing (İ → i).
Now, the way I check for whether we’re dealing with Azeri text is with the system locale. Unfortunately, there isn’t a single cross-platform solution for retrieving this information, but we can use “_get_current_locale()” on Windows and “setlocale(LC_ALL, 0)” on POSIX systems, which both return information about the locale and the current code page.
The POSIX implementation is simple, it only accepts two values if you want the Azeri (Latin) codec:
1 2 3 4 5 6 7 | EXPECT_STREQ("az_AZ", setlocale(LC_ALL, "az_AZ")); EXPECT_LOCALE_EQ(CASEMAPPING_LOCALE_TURKISH_OR_AZERI_LATIN, casemapping_locale()); RESET_LOCALE(); EXPECT_STREQ("az_AZ.utf8", setlocale(LC_ALL, "az_AZ.utf8")); EXPECT_LOCALE_EQ(CASEMAPPING_LOCALE_TURKISH_OR_AZERI_LATIN, casemapping_locale()); RESET_LOCALE(); |
The Windows version accepts a lot more values:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | EXPECT_STREQ("az", setlocale(LC_ALL, "az")); EXPECT_LOCALE_EQ(CASEMAPPING_LOCALE_TURKISH_OR_AZERI_LATIN, casemapping_locale()); RESET_LOCALE(); EXPECT_STREQ("az-Cyrl-AZ", setlocale(LC_ALL, "az-Cyrl-AZ")); EXPECT_LOCALE_EQ(CASEMAPPING_LOCALE_DEFAULT, casemapping_locale()); RESET_LOCALE(); EXPECT_STREQ("az-Latn-AZ", setlocale(LC_ALL, "az-Latn-AZ")); EXPECT_LOCALE_EQ(CASEMAPPING_LOCALE_TURKISH_OR_AZERI_LATIN, casemapping_locale()); RESET_LOCALE(); EXPECT_STREQ("Azeri_Azerbaijan.1254", setlocale(LC_ALL, "azeri")); EXPECT_LOCALE_EQ(CASEMAPPING_LOCALE_TURKISH_OR_AZERI_LATIN, casemapping_locale()); RESET_LOCALE(); EXPECT_STREQ("Azeri_Azerbaijan.1254", setlocale(LC_ALL, "Azeri_Azerbaijan.1254")); EXPECT_LOCALE_EQ(CASEMAPPING_LOCALE_TURKISH_OR_AZERI_LATIN, casemapping_locale()); RESET_LOCALE(); EXPECT_STREQ("Azeri_Azerbaijan.1254", setlocale(LC_ALL, "Azeri_Azerbaijan.ACP")); EXPECT_LOCALE_EQ(CASEMAPPING_LOCALE_TURKISH_OR_AZERI_LATIN, casemapping_locale()); RESET_LOCALE(); EXPECT_STREQ("Azeri_Azerbaijan.857", setlocale(LC_ALL, "Azeri_Azerbaijan.857")); EXPECT_LOCALE_EQ(CASEMAPPING_LOCALE_TURKISH_OR_AZERI_LATIN, casemapping_locale()); RESET_LOCALE(); EXPECT_STREQ("Azeri_Azerbaijan.857", setlocale(LC_ALL, "Azeri_Azerbaijan.OCP")); EXPECT_LOCALE_EQ(CASEMAPPING_LOCALE_TURKISH_OR_AZERI_LATIN, casemapping_locale()); RESET_LOCALE(); |
These values come directly from the horse’s mouth, the strings are in the format “<Language>_<Country>.<CodePage>”, where the codepage can be an ANSI codepage (ACP) or OEM codepage (OCP). So far, so good. The problem was that the tests for this specific format were failing. Something was broken on Windows 10.
Finding the Culprit
To figure out what went wrong, we have to dive into the implementation of “setlocale”. Microsoft has been surprisingly generous in this regard, providing source that allows us to step into the function itself, up to the point where it starts calling CRT functions. It turns out the “setlocale” is actually a wrapper for “GetLocaleNameFromLangCountry”, which enumerates the installed locales using “EnumSystemLocalesEx”:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 | /*** *void GetLocaleNameFromLangCountry - get locale names from language and country strings * *Purpose: * Match the best locale names to the language and country string given. * After global variables are initialized, the LangCountryEnumProcEx * routine is registered as an EnumSystemLocalesEx callback to actually * perform the matching as the locale names are enumerated. * *Entry: * pchLanguage - language string * bAbbrevLanguage - language string is a three-letter abbreviation * pchCountry - country string * bAbbrevCountry - country string ia a three-letter abbreviation * iPrimaryLen - length of language string with primary name * *Exit: * localeName - locale name of given language and country * *Exceptions: * *******************************************************************************/ static void GetLocaleNameFromLangCountry (_psetloc_struct _psetloc_data) { // initialize static variables for callback use _psetloc_data->bAbbrevLanguage = wcslen(_psetloc_data->pchLanguage) == 3; _psetloc_data->bAbbrevCountry = wcslen(_psetloc_data->pchCountry) == 3; _psetloc_data->iPrimaryLen = _psetloc_data->bAbbrevLanguage ? 2 : GetPrimaryLen(_psetloc_data->pchLanguage); // Enumerate all locales that come with the operating system, // including replacement locales, but excluding alternate sorts. __crtEnumSystemLocalesEx(LangCountryEnumProcEx, LOCALE_WINDOWS | LOCALE_SUPPLEMENTAL, (LPARAM) NULL); // locale value is invalid if the language was not installed or the language // was not available for the country specified if (!(_psetloc_data->iLocState & __LOC_LANGUAGE) || !(_psetloc_data->iLocState & __LOC_EXISTS) || !(_psetloc_data->iLocState & (__LOC_FULL | __LOC_PRIMARY | __LOC_DEFAULT))) _psetloc_data->iLocState = 0; } |
Stepping over the callback, I found out what went wrong: Microsoft changed the locale name from “Azeri” to “Azerbaijani”.
Regardless of whether that was the right call to make on a cultural level, this breaks my code. Windows 10 does not transform “Azeri_Azerbaijan.1254” into “Azerbaijani_Azerbaijan.1254” when you call setlocale, the function will simply return NULL.
The Reality of an Imperfect World
Okay, fine, this isn’t the first platform-specific bug I have to work around and it definitely isn’t the last. The bug won’t affect users, because they should be using the string “az-Latn-AZ” anyway. But it does break a unit test, so let’s detect whether we’re running Windows 10 and use different strings on that platform.
The standard way of checking what Windows version you’re running is by calling GetVersionEx and checking the “dwMajorVersion” and “dwMinorVersion” members of the resulting struct. Unfortunately, this method has been deprecated, with good reason. The short of it is that well-intentioned, but naive, developers wrote code like this:
1 2 3 4 | if (verMajor >= 5 && verMinor >= 1) { // oh, hello, you must be Windows XP or later // (clearly I didn’t test this code on Windows 6.0) } |
What happens when the major version is bumped to 10? This code will disable certain features of the application, because it thinks the Windows version is too old. As a result of this improper use of the API, GetVersionEx is now forever doomed to return “Windows 8.1”. Which puts me into a bit of a pickle, because I specifically want to test for Windows 10.
Luckily, there’s an alternative! There’s a “VersionsHelpers.h”, which provides this wonderful function:
1 2 3 4 5 | VERSIONHELPERAPI IsWindows10OrGreater() { return IsWindowsVersionOrGreater(HIBYTE(_WIN32_WINNT_WINTHRESHOLD), LOBYTE(_WIN32_WINNT_WINTHRESHOLD), 0); } |
It worked on my development machine, but it wouldn’t compile on my laptop. The function is part of the Windows SDK, which is not a dependency I want to include for a low-level system library. So I rolled my own:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 | bool CheckWindowsVersion(DWORD versionIdentifier) { DWORDLONG conditionMask = ::VerSetConditionMask( ::VerSetConditionMask( 0, VER_MAJORVERSION, VER_GREATER_EQUAL), VER_MINORVERSION, VER_GREATER_EQUAL); ::OSVERSIONINFOEXW osvi = { 0 }; osvi.dwOSVersionInfoSize = sizeof(osvi); osvi.dwMajorVersion = HIBYTE(versionIdentifier); osvi.dwMinorVersion = LOBYTE(versionIdentifier); return ::VerifyVersionInfoW(&osvi, VER_MAJORVERSION | VER_MINORVERSION, conditionMask) != FALSE; } |
And it didn’t work.
Down the Rabbit Hole
When you call this function with “_WIN32_WINNT_WIN10”, it returns false, but calling it with “_WIN32_WINNT_WINBLUE” returns true. It looks like “VerifyVersionInfo” function has the same problem as “GetVersionEx”: it claims Windows 10 is actually Windows 8.1. What is going on here? Luckily, there’s a handy articled on MSDN titled “Targeting your application for Windows”, which helps explain the steps you need to take in order for VerifyVersionInfo to return the actual version of Windows.
All you have to do is create a manifest file which adds compatibility info for your executable, which is basically a form you have to sign that states “I hereby acknowledge the existence of Windows versions beyond 8.1.”
Then you have to add that manifest file to your linker settings and you’re good to go!
This is a pretty mind-boggling solution on several levels:
- Deprecating GetVersionEx is fine and probably the right choice. But why then cripple the new function with the same problems of the old version?
- Why is this in the manifest file? Couldn’t this be a checkbox in Visual Studio?
- Better yet, why this isn’t this behavior a flag of the hypothetical “GetRealVersionEx” function?
- How am I ever going to explain this to my users? Link to this blogpost?
Summary
Windows 10 broke my code, deprecated the ability to check for Windows versions, crippled the new way to check for versions and added several unnecessary hoops for me to jump to in order to work around its broken behavior.
I hope y’all are proud of yourselves.