Debugging SSL communications
Introduction
The SSL and TLS protocols have become the de facto standard for securing network communications. These protocols provide confidentiality, authentication and message integrity, but add additional complexity to client server communications. This situation is most evident when application maintainers and system administrators need to debug application-layer protocols protected by SSL. This article will discuss two utilities (ssldump and openssl) that can help debug applications utilizing SSL.
How SSL Works
The SSL and TLS protocols define the rules SSL clients and servers use to communicate with each other. These rules specify the order in which messages are sent, the format of each message, and the way cryptographic algorithms are applied to network communications. The SSL and TLS protocols use a layered communications stack, and define several message types. The bottom layer of this communication stack is called the SSL record layer. This layer accepts protocol messages and application data from higher level protocols, adds SSL specific headers, and hands these messages ( often referred to as SSL record layer messages ) to TCP to be transmitted.
The SSL and TLS connection setup process consists of four stages. The first stage allows the client and server to negotiate security capabilities, such as the public-key algorithm, the symmetric key algorithm, and compression algorithms. The second stage allows the server to transmit digital certificates and key information to the client, allowing the client to validate the identity of the server. The third stage allows the client to exchange key information with the server, and optionally authenticate itself with a digital certificate. The final stage allows the client and server to use the negotiated parameters. The openssl utility can be used to connect to an SSL-enabled service, and print each stage described above:
$ openssl s_client -connect mail.prefetch.net:443 -state -nbio 2>&1 | grep "^SSL"
SSL_connect:before/connect initialization
SSL_connect:SSLv2/v3 write client hello A
SSL_connect:error in SSLv2/v3 read server hello A
SSL_connect:SSLv3 read server hello A
SSL_connect:SSLv3 read server certificate A
SSL_connect:SSLv3 read server key exchange A
SSL_connect:SSLv3 read server done A
SSL_connect:SSLv3 write client key exchange A
SSL_connect:SSLv3 write change cipher spec A
SSL_connect:SSLv3 write finished A
SSL_connect:SSLv3 flush data
SSL_connect:error in SSLv3 read finished A
SSL_connect:error in SSLv3 read finished A
SSL_connect:SSLv3 read finished A
SSL handshake has read 1584 bytes and written 346 bytes
SSL-Session:
For a detailed analysis of each state, please see the references.
Testing SSL Servers
When debugging SSL client server communications, it is often beneficial to watch the SSL connection setup process, and if this completes successfully, pass application data to the server. The openssl utility provides the s_client option which can be used to setup an SSL connection, and pass arbitrary data (eg., POP3 commands, IMAP commands, HTTP methods) to a server process. The following example lists the available messages on a POP3 server utilizing SSL:
$ openssl s_client -connect mail.prefetch.net:995
CONNECTED(00000003)
[ ..... ]
user ME
+OK Name is a valid mailbox
pass SOMETHING_OR_ANOTHER
+OK Mailbox locked and ready
list
+OK scan listing follows
1 1225
2 1863
3 1668
4 963
5 1969
6 1437
7 2357
8 3316
9 3517
10 8952
11 9616
quit
+OK
The following example shows how to request a specific URI from an HTTP server utilizing SSL:
$ openssl s_client -connect mail.prefetch.net:443
CONNECTED(00000003)
[ .... ]
GET / HTTP/1.0
HTTP/1.1 403 Forbidden
Date: Sat, 04 Sep 2004 16:54:01 GMT
Server: Apache/2.0.50
Content-Length: 202
Connection: close
Content-Type: text/html; charset=iso-8859-1
<html><head>
<title>403 Forbidden</title>
</head><body>>
Forbidden
You don't have permission to access / on this server.
</body></html>
closed
Printing Encrypted Application Layer Data
When the SSL and TLS protocols are used to protect network communications, the application layer data is obfuscated, and unavailable to network analysis tools (eg., snoop, tcpdump). This limits an administrators ability to debug application problems, and view application-layer headers and data. The ssldump utility can be used in these situations, and allows SSL messages and application data to be displayed in realtime. The following example shows how ssldump can be used to print the SSL handshake messages:
$ ssldump -a -A -H -i en0
New TCP connection #1: winnie.matty.com(32866) <-> 192.168.1.8(8389)
1 1 0.0043 (0.0043) C>S SSLv2 compatible client hello
1 2 0.0057 (0.0014) S>C Handshake
ServerHello
1 3 0.0057 (0.0000) S>C Handshake
Certificate
1 4 0.0057 (0.0000) S>C Handshake
ServerHelloDone
1 5 0.0182 (0.0125) C>S Handshake
ClientKeyExchange
1 6 0.0182 (0.0000) C>S ChangeCipherSpec
1 7 0.0182 (0.0000) C>S Handshake
1 8 0.0367 (0.0184) S>C ChangeCipherSpec
1 9 0.0367 (0.0000) S>C Handshake
1 10 3.2154 (3.1786) C>S application_data
1 11 3.2154 (0.0000) C>S application_data
1 12 3.4370 (0.2216) C>S application_data
1 13 3.4370 (0.0000) C>S application_data
1 14 3.4681 (0.0311) S>C application_data
1 15 3.4681 (0.0000) S>C application_data
2 3.4307 (3.4307) S>C TCP FIN
1 16 3.5172 (0.0491) S>C Alert
1 17 3.5178 (0.0006) C>S Alert
1 3.5180 (0.0001) C>S TCP FIN
2 3.4815 (0.0508) C>S TCP FIN
1 3.5194 (0.0013) S>C TCP FIN
The "-A" and "-H" options tell ssldump to print all of the SSL record layer headers. The "-a" option will provide TCP connection state (eg., SYN, SYN/ACK, ACK, FIN, etc) information. Communications originating from the client are marked with "C>S," and messages originating from the server are marked with "S>C." In the previous example, ssldump was not configured to decrypt communications, so the words "application_data" were printed instead of the actual application data.
To decrypt communications and dump application data, ssldump will need a copy of the private key from the server you wish to debug. ssldump will use this key to derive the session key that is negotiated between the client and the server, and used to encrypt all network communications. The following example uses the private key stored in the file rsa.key to decrypt communications, and display the application layer data:
$ ssldump -a -A -H -k rsa.key -i en0
< connection setup removed >
3.6155 (3.6155) C>S
---------------------------------------------------------------
GET / HTTP/1.0
---------------------------------------------------------------
2 12 3.8862 (0.2310) C>SV3.1(32) application_data
2 13 3.8862 (0.0000) C>SV3.1(32) application_data
3.8466 (0.2311) C>S
---------------------------------------------------------------
---------------------------------------------------------------
3.8777 (0.0310) S>C
---------------------------------------------------------------
HTTP/1.1 403 Forbidden
Date: Sun, 25 Jul 2004 04:27:16 GMT
Server: Apache/2.0.50
Content-Length: 1
Connection: close
Content-Type: text/html; charset=iso-8859-1
As you can see from this example, ssldump displays the HTTP requests sent to the prefetch.net web server. ssldump supports Berkeley Packet Filter style filters, allowing you to grab and decode specific communications. The following example shows how to capture SSL communications destined for host fred on TCP port 443:
$ ssldump -a -A -H -k rsa.key -i en0 host fred and port 443
Conclusion
This article provided a brief introduction to the SSL and TLS protocols, and showed how ssldump and openssl can be used to debug network communications. Both commands provide several facilities to help troubleshoot SSL and TLS communication problems. For additional information on the internal workings of the SSL state engine and handshake process, please see the references.
References
The following references were used while writing this article:
- TLS 1.0 RFC: http://www.ietf.org/html.charters/tls-charter.html
- SSL & TLS Essentials by Stephen Thomas
- OpenSSL Website: http://www.openssl.org
- SSL Dump Website: http://www.rtfm.com/ssldump/
Acknowledgements
Ryan would like to thank the ssldump and openssl developers, and all the folks who have put time and energy into the design and implementation of the TLS and SSL protocols.
Originally published in the December '04 issue of SysAdmin Magazine
Originally posted on Wed Jul 11, 2007
Comments