Bug in CryptStringToBinary function


When I was writing the Find tool, I decided to use for conversion hexadecimal strings to binary data the CryptStringToBinary function instead of my own function. According to MSDN (if the second parameter is zero, then the first parameter is a pointer to a null-terminated string) I wrote the following code:

if (CryptStringToBinary(String, 0, CRYPT_STRING_HEX, (PBYTE)Base, (DWORD *) NumberOfBytes, NULL, NULL)) {
     
     return Base;
}

ran the tool on Windows 7 SP1, with the 'FFFF' string to search for and got the result. But on Windows XP SP3 and Windows Server 2003 R2 SP1 function failed to convert the string. To find the cause of the failure, I ran the program under WinDbg, Step over CryptStringToBinary and then used the !gle command:

0:000> !gle 
LastErrorValue: (Win32) 0xd (13) - The data is invalid.

Obviously something has gone wrong under the hood because the hexadecimal string 'FFFF' is valid. To figure out who returns the error code I used the wt command with –or parameter.

0:000> wt -or
   24     0 [  0] CRYPT32!CryptStringToBinaryW
    1     0 [  1]   CRYPT32!wcslen
   34     0 [  1]   msvcrt!wcslen eax = 4
...
   25    62 [  5]           CRYPT32!_DigToHex eax = d
  118   931 [  4]         CRYPT32!_HexDecodeSimple eax = d
   12  1049 [  3]       CRYPT32!HexDecode eax = d
   36  1061 [  2]     CRYPT32!_DecodeCertSub eax = d
   55  1097 [  1]   CRYPT32!CryptStringToBinaryA
    1     0 [  2]     ntdll!RtlRestoreLastWin32Error
    7     0 [  2]     ntdll!RtlSetLastWin32Error eax = 7ffdf000
   63  1105 [  1]   CRYPT32!CryptStringToBinaryA eax = 0
   67  1566 [  0] CRYPT32!CryptStringToBinaryW
...

It took a minute to find out that CRYPT32!_DigToHex converts character to hexadecimal value, and returns an error code only if the input parameter was not a character of a particular representation of a hexadecimal digit. Apparently this function is not a cause of the issue and the next step was to debug CRYPT32!_HexDecodeSimple. The following excerpt shows that EAX takes a pointer to hexadecimal string and ECX the number of characters in the string. We can see that ECX takes the wrong number because the length of our string is 4 characters and we have seen above that the wcslen returns 4 characters too. That is why CRYPT32!_DigToHex returns error, it cannot convert the terminating NULL character.

0:000> u CRYPT32!_HexDecodeSimple
CRYPT32!_HexDecodeSimple:
77aa4bda 8bff            mov     edi,edi
77aa4bdc 55              push    ebp
77aa4bdd 8bec            mov     ebp,esp
77aa4bdf 83ec0c          sub     esp,0Ch
77aa4be2 8b4508          mov     eax,dword ptr [ebp+8]
77aa4be5 33c9            xor     ecx,ecx
77aa4be7 53              push    ebx
77aa4be8 56              push    esi
0:000> u
CRYPT32!_HexDecodeSimple+0xf:
77aa4be9 894dfc          mov     dword ptr [ebp-4],ecx
77aa4bec 894df8          mov     dword ptr [ebp-8],ecx
77aa4bef 894df4          mov     dword ptr [ebp-0Ch],ecx
77aa4bf2 8b4d0c          mov     ecx,dword ptr [ebp+0Ch]
77aa4bf5 57              push    edi
77aa4bf6 8d3c08          lea     edi,[eax+ecx]
77aa4bf9 3bc7            cmp     eax,edi
77aa4bfb 8bf0            mov     esi,eax
0:000> r
eax=00165578 ebx=00000000 ecx=00000005 edx=00000003 esi=0012eb20 edi=0016557d
eip=77aa4bfb esp=0012ea84 ebp=0012ea9c iopl=0         nv up ei ng nz ac po cy
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000293
CRYPT32!_HexDecodeSimple+0x21:
77aa4bfb 8bf0            mov     esi,eax
0:000> da @eax
00165578  "FFFF"

After a few minutes of digging around the root cause of failure was found. If the second parameter is zero (like in our case) the CryptStringToBinaryW calculates length of a string calling wcslen and then erroneously increments it.

0:000> u CRYPT32!CryptStringToBinaryW
CRYPT32!CryptStringToBinaryW:
77aa461d 8bff            mov     edi,edi
77aa461f 55              push    ebp
77aa4620 8bec            mov     ebp,esp
77aa4622 51              push    ecx
77aa4623 51              push    ecx
77aa4624 53              push    ebx
77aa4625 8b5d08          mov     ebx,dword ptr [ebp+8]
77aa4628 56              push    esi
0:000> u
CRYPT32!CryptStringToBinaryW+0xc:
77aa4629 57              push    edi
77aa462a 33ff            xor     edi,edi
77aa462c 3bdf            cmp     ebx,edi
77aa462e 897df8          mov     dword ptr [ebp-8],edi
77aa4631 897dfc          mov     dword ptr [ebp-4],edi
77aa4634 0f8483000000    je      CRYPT32!CryptStringToBinaryW+0xa0 (77aa46bd)
77aa463a 397d18          cmp     dword ptr [ebp+18h],edi
77aa463d 747e            je      CRYPT32!CryptStringToBinaryW+0xa0 (77aa46bd)
0:000> 
CRYPT32!CryptStringToBinaryW+0x22:
77aa463f 837d1002        cmp     dword ptr [ebp+10h],2
77aa4643 8b750c          mov     esi,dword ptr [ebp+0Ch]
77aa4646 7504            jne     CRYPT32!CryptStringToBinaryW+0x2f (77aa464c)
77aa4648 8bfb            mov     edi,ebx
77aa464a eb56            jmp     CRYPT32!CryptStringToBinaryW+0x85 (77aa46a2)
77aa464c 85f6            test    esi,esi
77aa464e 750c            jne     CRYPT32!CryptStringToBinaryW+0x3f (77aa465c)
77aa4650 53              push    ebx
0:000> 
CRYPT32!CryptStringToBinaryW+0x34:
77aa4651 e885bf0400      call    CRYPT32!wcslen (77af05db)
77aa4656 8bf0            mov     esi,eax
77aa4658 46              inc     esi
77aa4659 59              pop     ecx
77aa465a 85f6            test    esi,esi

Conclusion

To make the function work correctly in earlier versions of Windows you must calculate the length of the string by yourself.