RSSIL 2011 - RCE encrypted file

by @Jonathan Salwan - 2011-05-30

In this challenge, the staff gives a binary which is used to encrypt a file. We need to reverse it for uncrypt the file and read the flag. The binary was compiled on Linux x86 64 bits.

[jonathan@pentest RSSIL]$ file ex1 
ex1: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked 
(uses shared libs), for GNU/Linux 2.6.4, not stripped

Below, the file cyphered.

[jonathan@pentest RSSIL]$ hexdump -C file_ex1 
00000000  38 15 4e 7f 52 29 12 3d  12 6e 0d 2f 18 4c 25 7d  |8.N.R).=.n./.L%}|
00000010  26 53 52 81 28 40 71 72  58 1d 45 36 51 26 4f 28  |&SR.(@qrX.E6Q&O(|
00000020  4b 40 2e                                          |K@.|
00000023

Let's see it with gdb.

[...]
   0x0000000000400a99 <+271>:   callq  0x4006f0 <fread@plt>     |
   0x0000000000400a9e <+276>:   mov    -0x28(%rbp),%rax         | Read the file and save it 
   0x0000000000400aa2 <+280>:   mov    %rax,%rdi                | 
   0x0000000000400aa5 <+283>:   callq  0x400710 <fclose@plt>    |
   0x0000000000400aaa <+288>:   mov    -0x14(%rbp),%edx         |
   0x0000000000400aad <+291>:   mov    -0x20(%rbp),%rax         |

   0x0000000000400ab1 <+295>:   mov    %edx,%esi
   0x0000000000400ab3 <+297>:   mov    %rax,%rdi
   0x0000000000400ab6 <+300>:   callq  0x4008e6 <xor>           <= calls xor()
   0x0000000000400abb <+305>:   mov    -0x14(%rbp),%edx
   0x0000000000400abe <+308>:   mov    -0x20(%rbp),%rax
   0x0000000000400ac2 <+312>:   mov    %edx,%esi
   0x0000000000400ac4 <+314>:   mov    %rax,%rdi
   0x0000000000400ac7 <+317>:   callq  0x400876 <reverse>       <= calls reverse()
   0x0000000000400acc <+322>:   mov    -0x14(%rbp),%edx
   0x0000000000400acf <+325>:   mov    -0x20(%rbp),%rax
   0x0000000000400ad3 <+329>:   mov    %edx,%esi
   0x0000000000400ad5 <+331>:   mov    %rax,%rdi
   0x0000000000400ad8 <+334>:   callq  0x400834 <add_pos>       <= calls add_pos()
   0x0000000000400add <+339>:   mov    $0x400cc6,%edx
   0x0000000000400ae2 <+344>:   mov    -0xd0(%rbp),%rax
   0x0000000000400ae9 <+351>:   add    $0x10,%rax
   0x0000000000400aed <+355>:   mov    (%rax),%rax
   0x0000000000400af0 <+358>:   mov    %rdx,%rsi
   0x0000000000400af3 <+361>:   mov    %rax,%rdi

   0x0000000000400af6 <+364>:   callq  0x4006a0 <fopen@plt>     <= And saves the string
[...]

First, we create a new file with some NULL bytes and put a breakpoint on xor() function to know which key is used.

(gdb) b *0x0000000000400ac2
Breakpoint 1 at 0x400ac2
(gdb) shell echo -e "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00" > test
(gdb) r ./test ./test2
Starting program: /home/jonathan/RSSIL/ex1 ./test ./test2

Breakpoint 1, 0x0000000000400ac2 in main ()
(gdb) i r $rax
rax            0x603010 6303760
(gdb) x/s $rax
0x603010:    "OpEnChAnGe424EgNaHcNePoOpEnChAd"
(gdb)

We can see as the key is OpEnChAnGe424EgNaHcNePo. Now put a breakpoint in reverse() function.

(gdb) b *0x0000000000400ad3
Breakpoint 2 at 0x400ad3
(gdb) c
Continuing.

Breakpoint 2, 0x0000000000400ad3 in main ()
(gdb) i r $rax
rax            0x603010 6303760
(gdb) x/s 0x603010
0x603010:    "dAhCnEpOoPeNcHaNgE424eGnAhCnEpO"
(gdb)

Ok, our string was just reversed. Now, put a breakpoint after add_pos() function.

(gdb) b *0x0000000000400ae9
Breakpoint 3 at 0x400ae9
(gdb) c
Continuing.

Breakpoint 3, 0x0000000000400ae9 in main ()
(gdb) x/s $rdi
0x603010:    "dBjFrJvVwYoYoUo]wVFEHz]\205Y\201]\211a\215m"
(gdb)

After the add_pos() function, we can see all characters was incremented.

So, to resolve this challenge we need to :

  1. Do a sub_pos() function on our string.
  2. Do a reverse() function
  3. Do a xor() function with the key : OpEnChAnGe424EgNaHcNePo

Our cyphered string is :

"\x38\x15\x4e\x7f\x52\x29\x12\x3d\x12\x6e\x0d\x2f\x18\x4c"
"\x25\x7d\x26\x53\x52\x81\x28\x40\x71\x72\x58\x1d\x45\x36"
"\x51\x26\x4f\x28\x4b\x40\x2e"

After the sub_pos() function, our string is :

"\x38\x14\x4c\x7c\x4e\x24\x0c\x36\x0a\x65\x03\x24\x0c\x3f"
"\x17\x6e\x16\x42\x40\x6e\x14\x2b\x5b\x5b\x40\x04\x2b\x1b"
"\x35\x09\x31\x09\x2b\x1f\x0c"

After the reverse() function, our string is :

"\x0c\x1f\x2b\x09\x31\x09\x35\x1b\x2b\x04\x40\x5b\x5b\x2b"
"\x14\x6e\x40\x42\x16\x6e\x17\x3f\x0c\x24\x03\x65\x0a\x36"
"\x0c\x24\x4e\x7c\x4c\x14\x38"

And after the xor() function, our string is :

"\x43\x6f\x6e\x67\x72\x61\x74\x75\x6c\x61\x74\x69\x6f\x6e"
"\x73\x20\x21\x0a\x75\x20\x72\x6f\x63\x6b\x73\x20\x64\x75"
"\x64\x65\x20\x3b\x29\x20\x0a"

Now, we can resolve this challenge.

jonathan@ArchLinux [tmp] $ echo -e "\x43\x6f\x6e\x67\x72\x61\x74\x75\x6c\x61\x74\x69\x6f\x6e\x73\x20\x21\x0a\x75\x20\x72\x6f\x63\x6b\x73\x20\x64\x75\x64\x65\x20\x3b\x29\x20\x0a"
Congratulations !
u rocks dude ;)

jonathan@ArchLinux [tmp] $