When I was writing the Find tool, I decided to use for conversion hexadecimal strings to binary data the CryptStringToBinary function instead of my own function. According to MSDN (if the second parameter is zero, then the first parameter is a pointer to a null-terminated string) I wrote the following code:
if (CryptStringToBinary(String, 0, CRYPT_STRING_HEX, (PBYTE)Base, (DWORD *) NumberOfBytes, NULL, NULL)) {
return Base;
}
ran the tool on Windows 7 SP1, with the 'FFFF' string to search for and got the result. But on Windows XP SP3 and Windows Server 2003 R2 SP1 function failed to convert the string. To find the cause of the failure, I ran the program under WinDbg, Step over CryptStringToBinary and then used the !gle command:
0:000> !gle LastErrorValue: (Win32) 0xd (13) - The data is invalid.
Obviously something has gone wrong under the hood because the hexadecimal string 'FFFF' is valid. To figure out who returns the error code I used the wt command with –or parameter.
0:000> wt -or
24 0 [ 0] CRYPT32!CryptStringToBinaryW
1 0 [ 1] CRYPT32!wcslen
34 0 [ 1] msvcrt!wcslen eax = 4
...
25 62 [ 5] CRYPT32!_DigToHex eax = d
118 931 [ 4] CRYPT32!_HexDecodeSimple eax = d
12 1049 [ 3] CRYPT32!HexDecode eax = d
36 1061 [ 2] CRYPT32!_DecodeCertSub eax = d
55 1097 [ 1] CRYPT32!CryptStringToBinaryA
1 0 [ 2] ntdll!RtlRestoreLastWin32Error
7 0 [ 2] ntdll!RtlSetLastWin32Error eax = 7ffdf000
63 1105 [ 1] CRYPT32!CryptStringToBinaryA eax = 0
67 1566 [ 0] CRYPT32!CryptStringToBinaryW
...
It took a minute to find out that CRYPT32!_DigToHex converts character to hexadecimal value, and returns an error code only if the input parameter was not a character of a particular representation of a hexadecimal digit. Apparently this function is not a cause of the issue and the next step was to debug CRYPT32!_HexDecodeSimple. The following excerpt shows that EAX takes a pointer to hexadecimal string and ECX the number of characters in the string. We can see that ECX takes the wrong number because the length of our string is 4 characters and we have seen above that the wcslen returns 4 characters too. That is why CRYPT32!_DigToHex returns error, it cannot convert the terminating NULL character.
0:000> u CRYPT32!_HexDecodeSimple CRYPT32!_HexDecodeSimple: 77aa4bda 8bff mov edi,edi 77aa4bdc 55 push ebp 77aa4bdd 8bec mov ebp,esp 77aa4bdf 83ec0c sub esp,0Ch 77aa4be2 8b4508 mov eax,dword ptr [ebp+8] 77aa4be5 33c9 xor ecx,ecx 77aa4be7 53 push ebx 77aa4be8 56 push esi 0:000> u CRYPT32!_HexDecodeSimple+0xf: 77aa4be9 894dfc mov dword ptr [ebp-4],ecx 77aa4bec 894df8 mov dword ptr [ebp-8],ecx 77aa4bef 894df4 mov dword ptr [ebp-0Ch],ecx 77aa4bf2 8b4d0c mov ecx,dword ptr [ebp+0Ch] 77aa4bf5 57 push edi 77aa4bf6 8d3c08 lea edi,[eax+ecx] 77aa4bf9 3bc7 cmp eax,edi 77aa4bfb 8bf0 mov esi,eax 0:000> r eax=00165578 ebx=00000000 ecx=00000005 edx=00000003 esi=0012eb20 edi=0016557d eip=77aa4bfb esp=0012ea84 ebp=0012ea9c iopl=0 nv up ei ng nz ac po cy cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000293 CRYPT32!_HexDecodeSimple+0x21: 77aa4bfb 8bf0 mov esi,eax 0:000> da @eax 00165578 "FFFF"
After a few minutes of digging around the root cause of failure was found. If the second parameter is zero (like in our case) the CryptStringToBinaryW calculates length of a string calling wcslen and then erroneously increments it.
0:000> u CRYPT32!CryptStringToBinaryW CRYPT32!CryptStringToBinaryW: 77aa461d 8bff mov edi,edi 77aa461f 55 push ebp 77aa4620 8bec mov ebp,esp 77aa4622 51 push ecx 77aa4623 51 push ecx 77aa4624 53 push ebx 77aa4625 8b5d08 mov ebx,dword ptr [ebp+8] 77aa4628 56 push esi 0:000> u CRYPT32!CryptStringToBinaryW+0xc: 77aa4629 57 push edi 77aa462a 33ff xor edi,edi 77aa462c 3bdf cmp ebx,edi 77aa462e 897df8 mov dword ptr [ebp-8],edi 77aa4631 897dfc mov dword ptr [ebp-4],edi 77aa4634 0f8483000000 je CRYPT32!CryptStringToBinaryW+0xa0 (77aa46bd) 77aa463a 397d18 cmp dword ptr [ebp+18h],edi 77aa463d 747e je CRYPT32!CryptStringToBinaryW+0xa0 (77aa46bd) 0:000> CRYPT32!CryptStringToBinaryW+0x22: 77aa463f 837d1002 cmp dword ptr [ebp+10h],2 77aa4643 8b750c mov esi,dword ptr [ebp+0Ch] 77aa4646 7504 jne CRYPT32!CryptStringToBinaryW+0x2f (77aa464c) 77aa4648 8bfb mov edi,ebx 77aa464a eb56 jmp CRYPT32!CryptStringToBinaryW+0x85 (77aa46a2) 77aa464c 85f6 test esi,esi 77aa464e 750c jne CRYPT32!CryptStringToBinaryW+0x3f (77aa465c) 77aa4650 53 push ebx 0:000> CRYPT32!CryptStringToBinaryW+0x34: 77aa4651 e885bf0400 call CRYPT32!wcslen (77af05db) 77aa4656 8bf0 mov esi,eax 77aa4658 46 inc esi 77aa4659 59 pop ecx 77aa465a 85f6 test esi,esi
Conclusion
To make the function work correctly in earlier versions of Windows you must calculate the length of the string by yourself.