An Analysis of the OpenSSL SSL Handshake Error State Security Bypass (CVE-2017-3737)

OpenSSL is a widely used library for SSL and TLS protocol implementation that secures data using encryption and decryption based on cryptographic functions. However, a Security Bypass vulnerability – recently addressed in a patch by the OpenSSL Project –can be exploited to make vulnerable SSL clients or remote SSL servers send clean application data without encryption. This Security Bypass vulnerability (CVE-2017-3737) is caused by an error when the SSL_read or SSL_write function handles an "error state" during an SSL handshake. In this paper the FortiGuard Labs team examines the root cause of this vulnerability. The "error state" mechanism was introduced in OpenSSL beginning with version 1.0.2b, It is used to make OpenSSL move into an error state whenever a fatal error occurs during the SSL handshake that would fail if the SSL handshake continued. If SSL_read or SSL_write function is called directly, it checks the SSL handshake state and performs a new SSL handshake automatically if no handshake has been initiated. If a fatal error occurs during the SSL handshake, OpenSSL moves into the error state and returns an error message to the caller. However, the problem occurs if the caller doesn't check the error state and simply calls the SSL_read or SSL_write function again, because it then sends application data without encryption. The following code snippet was taken from OpenSSL 1.0.2m. (Comments added by me have been highlighted.) `ssl/s3_pkt.c`: ``` 638 int ssl3_write_bytes(SSL *s, int type, const void *buf_, int len) 639 { 640 const unsigned char *buf = buf_; 641 int tot; 642 unsigned int n, nw; 643 #if !defined(OPENSSL_NO_MULTIBLOCK) && EVP_CIPH_FLAG_TLS1_1_MULTIBLOCK 644 unsigned int max_send_fragment; 645 #endif 646 SSL3_BUFFER *wb = &(s->s3->wbuf); 647 int i; 648 649 s->rwstate = SSL_NOTHING; 650 OPENSSL_assert(s->s3->wnum <= INT_MAX); 651 tot = s->s3->wnum; 652 s->s3->wnum = 0; 653 654 if (SSL_in_init(s) && !s->in_handshake) { //checks to see if the SSL handshake state is initiated. The state will be SSL_ST_INIT the first time. 655 i = s->handshake_func(s); //performs a new SSL handshake if no handshake has been initiated. 656 if (i < 0) 657 return (i); 658 if (i == 0) { 659 SSLerr(SSL_F_SSL3_WRITE_BYTES, SSL_R_SSL_HANDSHAKE_FAILURE); 660 return -1; 661 } 662 } ``` ssl3_write_bytes() is called by the SSL_write function to send the application data. It checks the SSL handshake state and performs the SSL handshake if needed. `ssl/s3_clnt.c`: ``` 898 int ssl3_get_server_hello(SSL *s) 899 { 900 STACK_OF(SSL_CIPHER) *sk; 901 const SSL_CIPHER *c; 902 CERT *ct = s->cert; 903 unsigned char *p, *d; 904 int i, al = SSL_AD_INTERNAL_ERROR, ok; .... 1077 if (i < 0) { 1078 /* we did not say we would use this cipher */ 1079 al = SSL_AD_ILLEGAL_PARAMETER; 1080 SSLerr(SSL_F_SSL3_GET_SERVER_HELLO, SSL_R_WRONG_CIPHER_RETURNED); 1081 goto f_err; //a fatal error occurs 1082 } .... 1170 f_err: 1171 ssl3_send_alert(s, SSL3_AL_FATAL, al); //sends an SSL alert packet 1172 err: 1173 s->state = SSL_ST_ERR; //moves into an error state 1174 return (-1); 1175 } ``` We used a vulnerable SSL client as a target during the test. During the SSL handshake it received a malformed server hello message from the SSL server controlled by the attacker. ssl3_get_server_hello() is called to handle this server hello message and a fatal error occurs, causing OpenSSL to move into an error state by setting s->state from SSL_ST_INIT to SSL_ST_ERR. `ssl/s3_pkt.c`: ``` 638 int ssl3_write_bytes(SSL *s, int type, const void *buf_, int len) 639 { 640 const unsigned char *buf = buf_; 641 int tot; 642 unsigned int n, nw; 643 #if !defined(OPENSSL_NO_MULTIBLOCK) && EVP_CIPH_FLAG_TLS1_1_MULTIBLOCK 644 unsigned int max_send_fragment; 645 #endif 646 SSL3_BUFFER *wb = &(s->s3->wbuf); 647 int i; 648 649 s->rwstate = SSL_NOTHING; 650 OPENSSL_assert(s->s3->wnum <= INT_MAX); 651 tot = s->s3->wnum; 652 s->s3->wnum = 0; 653 654 if (SSL_in_init(s) && !s->in_handshake) { // SSL_in_init is called again to check the state ``` If the vulnerable SSL client doesn't check the error state and call SSL_write function to send application data again, ssl3_write_bytes() is called and uses SSL_in_init() to check the handshake state again. `include/openssl/ssl.h`: ``` 1749 # define SSL_in_init(a) (SSL_state(a)&SSL_ST_INIT) //s->state is now SSL_ST_ERR, and check returns are false. ``` This time the check fails, the SSL handshake is bypassed and the application data will be sent without encryption. The following traffic dump shows how the clean application data is sent:  During this attack, the attacker entices the vulnerable SSL client to connect to a malicious SSL server. The SSL client may bypass the handshake process and send the application data without encryption. The SSL server may also have the same vulnerability if SSL_read or SSL_write function is called directly. NOTE: authentication is NOT required to exploit this vulnerability. ##### IPS Signature FortiGuard released IPS signature OpenSSL.Handshake.Error.State.Security.Bypass to address this vulnerability.