\n\nThis page has been machine-translated from the original page.
\n
This time I wanted to implement the RC4 cipher in C and reverse-engineer it with WinDbg.
\nAt Harekaze mini CTF 2021, held on 2021/12/24, there was a Rev challenge that required identifying RC4 usage from static analysis alone. Unfortunately, I was unable to identify it from the decompiler output at the time.
\nReference: harekaze-mini-ctf-2021 Pack
\nAs a review exercise, I decided to implement RC4 myself and reverse-engineer it.
\n\nRC4 is a type of stream cipher that was used in protocols such as WEP and SSL.
\nIt is now widely known to have vulnerabilities and its use is not recommended.
\nReference: RC4 - Wikipedia
\nA stream cipher is a type of symmetric cipher (one that uses the same key for both encryption and decryption) that encrypts data sequentially, bit by bit or byte by byte.
\nBlock ciphers are another form of symmetric cipher and are frequently compared with stream ciphers.
\nReference: Practical Guide to Applied Cryptography
\nBecause RC4 is a stream cipher, it encrypts plaintext by XORing it with a generated pseudorandom sequence.
\nThe encryption flow with RC4 is roughly as follows:
\nIn RC4, a predefined array S is initialized using the KSA (Key Scheduling Algorithm).
\nThe implementation follows the Wikipedia sample.
\nReference: RC4 - Wikipedia
\n#define N 256\n\nvoid swap(unsigned char *a, unsigned char *b)\n{\n int tmp = *a;\n *a = *b;\n *b = tmp;\n return;\n}\n\nint KSA(char *key, unsigned char *S)\n{\n\n // Initialize array S (S[0]=0, S[1]=1...S[255]=255)\n for (int i = 0; i < N; i++) S[i] = i;\n \n // Generate the initial stream with KSA\n int j = 0;\n int len = strlen(key);\n for (int i = 0; i < N; i++)\n {\n j = (j + S[i] + key[i % len]) % N;\n swap(&S[i], &S[j]);\n }\n\n return 0;\n}As shown above, the initial stream is generated by KSA based on the input key.
\nThe steps from PRGA to encryption are implemented as follows:
\ni = (i + 1) % 256j = (j + S[i]) % 256S[(S[i] + S[j]) % 256] and XOR it with the plaintextThe implementation is very straightforward.
\nint PRGA(unsigned char *S, char *text, unsigned char *encrypted_text)\n{\n\n int i = 0;\n int j = 0;\n for (size_t n = 0, len = strlen(text); n < len; n++)\n {\n i = (i + 1) % N;\n j = (j + S[i]) % N;\n swap(&S[i], &S[j]);\n int K = S[(S[i] + S[j]) % N];\n encrypted_text[n] = K ^ text[n];\n }\n\n return 0;\n}This completes the RC4 encryption implementation.
\nRC4 encrypts data using XOR with a pseudorandom sequence.
\nBecause the pseudorandom generation algorithm is deterministic, the same key always produces the same pseudorandom sequence.
\nTherefore, you can decrypt ciphertext by XORing it again with the pseudorandom sequence generated from the same key that was used for encryption.
\nThe full source code is shown below.
\nIt is also available in the following repository.
\nReference: Try2WinDbg/rc4_encrypt.c
\n#include <stdio.h>\n#include <string.h>\n#include <stdlib.h>\n\n#define N 256\n\nvoid swap(unsigned char *a, unsigned char *b)\n{\n int tmp = *a;\n *a = *b;\n *b = tmp;\n return;\n}\n\nint KSA(char *key, unsigned char *S)\n{\n\n // Initialize array S (S[0]=0, S[1]=1...S[255]=255)\n for (int i = 0; i < N; i++) S[i] = i;\n \n // Generate the initial stream with KSA\n int j = 0;\n int len = strlen(key);\n for (int i = 0; i < N; i++)\n {\n j = (j + S[i] + key[i % len]) % N;\n swap(&S[i], &S[j]);\n }\n\n return 0;\n}\n\nint PRGA(unsigned char *S, char *text, unsigned char *encrypted_text)\n{\n\n int i = 0;\n int j = 0;\n for (size_t n = 0, len = strlen(text); n < len; n++)\n {\n i = (i + 1) % N;\n j = (j + S[i]) % N;\n swap(&S[i], &S[j]);\n int K = S[(S[i] + S[j]) % N];\n encrypted_text[n] = K ^ text[n];\n }\n\n return 0;\n}\n\nint RC4(char *key, char *text, unsigned char *encrypted_text)\n{\n\n unsigned char S[N];\n\n KSA(key, S);\n PRGA(S, text, encrypted_text);\n return 0;\n}\n\nint main(int argc, char *argv[])\n{\n printf(\"RC4 module\\n\");\n\n // Note: a short RC4 key is easily guessable; exercise caution in production use\n char *key = \"testkey\";\n char *text = \"this is test.\";\n\n unsigned char *encrypted_text = malloc(sizeof(int) * strlen(text));\n RC4(key, text, encrypted_text);\n\n printf(\"This is encrypted text\\n\");\n printf(\"==> \");\n for (size_t i = 0, len = strlen(text); i < len; i++)\n {\n printf(\"%02hhX\", encrypted_text[i]);\n }\n printf(\"\\n\");\n\n unsigned char *decrypted_text = malloc(sizeof(int) * strlen(encrypted_text));\n RC4(key, encrypted_text, decrypted_text);\n\n printf(\"This is decrypted text\\n\");\n printf(\"==> \");\n for (size_t i = 0, len = strlen(text); i < len; i++)\n {\n printf(\"%c\", decrypted_text[i]);\n }\n\n return 0;\n}Finally, let’s reverse-engineer this program.
\nFirst, I loaded the compiled binary into Ghidra for analysis.
\nI will skip over the process of locating the main function.
\nLet’s start by looking at the RC4 calling function.
\nIn terms of the source code, this corresponds to the following section:
\nint RC4(char *key, char *text, unsigned char *encrypted_text)\n{\n unsigned char S[N];\n KSA(key, S);\n PRGA(S, text, encrypted_text);\n return 0;\n}Looking at the decompiled output, something seems off:
\nvoid __cdecl RC4(char *param_1,char *param_2,int param_3)\n{\n uint uVar1;\n undefined extraout_DL;\n undefined in_stack_fffffef8;\n \n uVar1 = DAT_00471090 ^ (uint)&stack0xfffffffc;\n thunk_FUN_004071b0(param_1,(int)&stack0xfffffef8);\n thunk_FUN_00407260((int)&stack0xfffffef8,param_2,param_3);\n thunk_FUN_00407648(uVar1 ^ (uint)&stack0xfffffffc,extraout_DL,in_stack_fffffef8);\n return;\n}I renamed the functions and variables to make analysis easier:
\nvoid __cdecl RC4(char *key,char *text,int encrypted_text)\n{\n uint uVar1;\n undefined extraout_DL;\n undefined S;\n \n uVar1 = DAT_00471090 ^ (uint)&stack0xfffffffc;\n KSA(key,(int)&stack0xfffffef8);\n PRGA((int)&stack0xfffffef8,text,encrypted_text);\n thunk_FUN_00407648(uVar1 ^ (uint)&stack0xfffffffc,extraout_DL,S);\n return;\n}The following two lines do not appear in the source code and it’s unclear what they’re doing:
\nuVar1 = DAT_00471090 ^ (uint)&stack0xfffffffc;\n/* ... (omitted) ... */\nthunk_FUN_00407648(uVar1 ^ (uint)&stack0xfffffffc,extraout_DL,S);So I analyzed this section using WinDbg’s TTD trace.
\nI captured a TTD trace as usual.
\nThe trace file is available here:
\nReference: Try2WinDbg/rc4_encrypt.zip
\nI loaded the trace file and set a breakpoint at the address corresponding to the uVar1 = DAT_00471090 ^ (uint)&stack0xfffffffc; instruction.
__security_cookie is defined in the data section and is used to detect buffer overflows.
Reference: ida - _securitycookie for function pointers in Windows 10
\nIt turns out those two mysterious lines implement the stack-cookie mechanism for detecting buffer overflows:
\nuVar1 = DAT_00471090 ^ (uint)&stack0xfffffffc;\n/* ... (omitted) ... */\nthunk_FUN_00407648(uVar1 ^ (uint)&stack0xfffffffc,extraout_DL,S);That clears things up a bit. Let’s move on to analyzing KSA and PRGA.
\nHere is the result of decompiling the KSA function in Ghidra:
\nundefined4 __cdecl KSA(char *param_1,int param_2)\n{\n size_t sVar1;\n uint local_10;\n int local_c;\n int local_8;\n \n for (local_c = 0; local_c < 0x100; local_c = local_c + 1) {\n *(undefined *)(param_2 + local_c) = (undefined)local_c;\n }\n local_10 = 0;\n sVar1 = _strlen(param_1);\n for (local_8 = 0; local_8 < 0x100; local_8 = local_8 + 1) {\n local_10 = *(byte *)(param_2 + local_8) + local_10 + (int)param_1[local_8 % (int)sVar1] &\n 0x800000ff;\n if ((int)local_10 < 0) {\n local_10 = (local_10 - 1 | 0xffffff00) + 1;\n }\n thunk_FUN_00867180((undefined *)(param_2 + local_8),(undefined *)(param_2 + local_10));\n }\n return 0;\n}This closely resembles the decompiled output of the binary from the Harekaze mini CTF 2021 Rev challenge.
\nReference: harekaze-mini-ctf-2021 Pack
\nIt’s frustrating to think that if I had caught this right away, I could have solved the challenge.
\nHere is the decompiled output:
\nundefined4 __cdecl PRGA(int param_1,char *param_2,int param_3)\n{\n size_t sVar1;\n uint local_10;\n uint local_c;\n uint local_8;\n \n local_8 = 0;\n local_10 = 0;\n local_c = 0;\n sVar1 = _strlen(param_2);\n for (; local_c < sVar1; local_c = local_c + 1) {\n local_8 = local_8 + 1 & 0x800000ff;\n if ((int)local_8 < 0) {\n local_8 = (local_8 - 1 | 0xffffff00) + 1;\n }\n local_10 = *(byte *)(param_1 + local_8) + local_10 & 0x800000ff;\n if ((int)local_10 < 0) {\n local_10 = (local_10 - 1 | 0xffffff00) + 1;\n }\n thunk_FUN_00867180((undefined *)(param_1 + local_8),(undefined *)(param_1 + local_10));\n *(byte *)(param_3 + local_c) =\n param_2[local_c] ^\n *(byte *)(param_1 +\n ((uint)*(byte *)(param_1 + local_8) + (uint)*(byte *)(param_1 + local_10) &\n 0x800000ff));\n }\n return 0;\n}One interesting point is how (i + 1) % 256 in the source code appears differently in the decompiled output:
Note: In C, + and & have the same operator precedence, so i + 1 is evaluated first.
i = i + 1 & 0x800000ff;\nif ((int)i < 0) {\n\ti = (i - 1 | 0xffffff00) + 1;\n}I wrote a small script to verify, and confirmed that this expression produces exactly the same result as % 256:
def mod256(n):\n n = n & 0x800000ff\n if n < 0:\n n = (n - 1 | 0xffffff00)\n\n return n\n\ndef main():\n for i in range(250, 260):\n print(mod256(i), end=\" \")\n print(\"\")\n\n return\n\nif __name__ == \"__main__\":\n main()That’s an interesting find.
\nIn future reversing work, when I see this kind of decompiled pattern I can recognize it as a modulo operation.
\nI initially assumed that all mod 256 code would produce the same output as shown above, but it turns out the output varies considerably depending on the compiler. Someone kindly pointed this out in a Twitter comment.
Source: original comment
\nThis time, as a review of Harekaze mini CTF 2021, I implemented RC4 encryption and reverse-engineered it.
\nI made some new discoveries along the way, so it was well worth trying.
\n