THC SSL Renegotiation DoS Tool for ESXi authd (port 902)

I had written about the Client-initiated SSL renegotiation DoS tool by THC and how to exploit SMTP STARTTLS mail servers with some modifications some time ago. At the time I’ve also noticed that to my surprise, Client-initiated SSL renegotiation is enabled by default on various vSphere/ESXi components and can be exploited with the THC Tool.

# openssl s_client -connect myesxi.local:443 -state -quiet -no_ign_eof <<< 'R' SSL_connect:before/connect initialization SSL_connect:SSLv2/v3 write client hello A SSL_connect:SSLv3 read server hello A verify error:num=19:self signed certificate in certificate chain verify return:0 SSL_connect:SSLv3 read server certificate A SSL_connect:SSLv3 read server done A SSL_connect:SSLv3 write client key exchange A SSL_connect:SSLv3 write change cipher spec A SSL_connect:SSLv3 write finished A SSL_connect:SSLv3 flush data SSL_connect:SSLv3 read finished A RENEGOTIATING SSL_connect:SSL renegotiate ciphers SSL_connect:SSLv3 write client hello A SSL_connect:SSLv3 read server hello A verify error:num=19:self signed certificate in certificate chain verify return:0 SSL_connect:SSLv3 read server certificate A SSL_connect:SSLv3 read server done A SSL_connect:SSLv3 write client key exchange A SSL_connect:SSLv3 write change cipher spec A SSL_connect:SSLv3 write finished A SSL_connect:SSLv3 flush data SSL_connect:SSLv3 read finished A DONE

The CIM-SSL port 5989 for hardware status monitoring is is also renegotiating. This affects all currently known versions of ESXi, including the most latest 6.0 U2 and 5.5 U3releases. vCenter on at least ports 443, 7444, 9443 is also affected.

ESXi also runs an important service on port 902 where the client-initiated renegotiation can be exploited most efficiently, commonly called the authd service. For example, this port is used for VM console connections. These connections are also SSL/TLS encrypted, but with a slight unusual touch:

ESXi responds with an initial plaintext banner message after the TCP connection has been established (similar to FTP or SMTP) and only then the SSL/TLS handshake can take place:

# telnet myesxi.local 902 Trying 10.110.5.33... Connected to myesxi.local Escape character is '^]'. 220 VMware Authentication Daemon Version 1.10: SSL Required, ServerDaemonProtocol:SOAP, MKSDisplayProtocol:VNC , VMXARGS supported, NFCSSL supported/t

Now this is a problem because a standard SSL/TLS client expects the first payload in the connection to be its own HELLO handshake message, and not some plaintext garbage sent by the server. Naturally, this leads to broken connections:

# openssl s_client -connect myesxi.local:902 -state CONNECTED(00000003) SSL_connect:before/connect initialization SSL_connect:SSLv2/v3 write client hello A SSL_connect:error in SSLv2/v3 read server hello A 3076646536:error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol:s23_clnt.c:794: --- no peer certificate available --- No client certificate CA names sent --- SSL handshake has read 7 bytes and written 361 bytes --- New, (NONE), Cipher is (NONE) Secure Renegotiation IS NOT supported Compression: NONE Expansion: NONE No ALPN negotiated ---

The THC DoS tool won’t work with that either. Basically, we first need to read the banner message and throw it down the bit bucket, and then initiate the actual SSL handshake. Since I had written an SMTP STARTTLS modification for the THC SSL renegotiation DoS tool already, I wrote a small patch to cope with this as well.

I found that on port 902 ESXi was more susceptible to reach higher CPU utilization than on port 443 which doesn’t require modification.

I had informed the VMware Security Response Team about it and supplied the patched code, after some months of no activity from their side whatsoever, they reported they could confirm a partial DoS and were “working on it”. More time passed, and after a whole year and several requests for status updates I never heard back from them again.

I don’t know what keeps them from disabling Client-initiated SSL renegotiation since it has virtually no valid use cases and surely isn’t even relevant for vSphere, and is disabled by default on pretty much all SSL/TLS implementations.

Running the DoS Tool

So what happens when you actually run it against an ESXi host? In my test case I use a Proliant DL 360 gen8 ESXi host with 16 physical cores/32 threads with HT. The host has some VM but is mostly idling at 3-6% CPU usage.

Note: I ran this from a VM with just a single vCPU which was getting maxed, so you will get better results if you run it from a slightly faster system.

Against port 443 you will see that the rhttpproxy service CPU usage spikes, using up 12 threads at once! (100% in esxtop equals the capacity of one physical thread)

Now I’m running it with my patched code against port 902, and you’ll notice multiple authd processes spawning, one for every TCP socket, each using up CPU resources as the SSL connection is being re-negotiated:

Of course we can also combine both ways and run the two in parallel for more efficiency and causing more load on the host:

Remember, this was all done from a slow system with just a single vCPU that was getting maxed. With this however we can burn a large part of the physical CPU resources of a modern 16 cores/32 threads host.

It continues to be interesting when I noticed that the authd seems to have a hard limit of the maximum number of TCP connections (even completely idle) ones.

Once this limit is hit, new authd connections will fail. This means for example you can’t open new VM consoles anymore, and the authd daemons spews an internal library error to the TCP socket… yikes!

$ nc -v myesxihost.local 902 citysvs0033.city.net.ffm [10.110.5.33] 902 (?) open authd: error while loading shared libraries: libvmkctl.so: failed to map segment from shared object: Error 28

Technically this also means you can never have more than ~30 active VM console connections per host. Maybe a parameter to add to the vSphere maximum config guidelines?

The limit seems to be around 30 in my case can be triggered with a script as simple as this, just opening idle TCP connections:

while [ $COUNTER -lt 30 ]; do nc -v citysvs0033.city.net.ffm 902 & let COUNTER=COUNTER+1 done The code

Here’s the patched code you need to apply to the original thc SSL DoS sources in order to use the port 902 authd banner option. This code is strictly intended for educational purposes only.

Note: You should run the tool with a limited number of concurrent connections using the -l option. I found 25-30 to be the sweet spot where connections on port 902 didn’t break, but your mileage may vary.

-