IBM Domino KeyView PDF Filter Encrypted Stream Code Execution Vulnerability(CVE-2016-0277)

### Summary A stack overflow vulnerability present in the PDF filter of KeyView as used by Domino can lead to process crash and possible arbitrary code execution. ### Tested Versions KeyView 10.16 as used by IBM Domino 9.0.1 ### Product URLs http://www-03.ibm.com/software/products/en/ibmdomino ### Details While parsing a specially crafted PDF file, a user controlled length field is used in a write loop with fixed destination size leading to a stack based buffer overflow. The vulnerability is triggered while parsing the PDF file that specifies an encrypted stream. As per the PDF specification, the Length value specifies the key `length` and is at most 16 bytes long. In the vulnerable function a stack buffer 16 bytes in length is reserved, but unchecked `Length` value is used during the copy operation which allows adjacent stack data to be overwritten, including the return address. The minimized test case that triggers the vulnerability is as follows: ``` %PDF-1.4 trailer <</Size 9 /Root 1 0 R /Encrypt 8 0 R 8 0 obj<< /Length 768 /Filter/Standard /Type/Catalog /O (41414141414141414141414141414141) /U (42424242424242424242424242424242) /P 0 /R 3 >> >> obj<< >> endobj %%EOF ``` In the above test case, the PDF `trailer` specifies that object 8 is encrypted. Further, object 8 specifies that it is using a standard filter for encryption (`/Filter/Standard`) and is using a revision 3 (`/R 3`) of the algorithm. Owner password (`/O`) and user password ('/U'), as well as object type don't play a significant role in this test case. While parsing the supplied test case, the `CPDFConvertToUserPassword` function in `pdfsr.so` will be called. This function implements the algorithm for deriving the decryption key. The overflow happens in the following code (image base being 0xB79BA00): ``` .text:B79E97A5 loc_B79E97A5: .text:B79E97A5 movzx eax, [ebp+edx+var_40] [1] .text:B79E97AA xor eax, esi [2] .text:B79E97AC mov [ebp+edx+var_20], al [3] .text:B79E97B0 add edx, 1 .text:B79E97B3 .text:B79E97B3 loc_B79E97B3: .text:B79E97B3 mov ecx, [ebp+var_3E0] .text:B79E97B9 mov eax, [ecx+13CCh] .text:B79E97BF cmp edx, eax [4] .text:B79E97C1 jl short loc_B79E97A5 ``` In the above code, `edx` serves as a counter. At [1], a byte is zero extended from a stack based buffer, is xored with `esi` at [2] and written to a stack buffer at [3]. The value of `esi` comes from an outer loop counter, starts at 19 and is decreased untill 0. At [4], the counter in `edx` is compared against a maximum value in `eax` which comes straight from the `Length` value divided by 8. To reiterate, the PDF specification states that `Length` will be at most 128 bits, so the maximum value in `eax` should be 16. Appropriate ly, 16 bytes are allocated for `var_20` buffer. If the value of `Length` is more, a buffer overflow will occur, overwriting the adjacent stack memory. The supplied test case triggers the vulnerability and leads to a crash as the buffer overflow overwrites the return address as well as the stack cookie present on the stack. There are two mitigating factors that lower the chance of successful exploitation of this vulnerability. First, the function is protected with a stack cookie making a straight forward return address overwrite difficult. And second, the bytes that end up overflowing the buffer are constant. To elaborate on the second point, a shortened pseudo code of the algorithm follows: ``` if Revision == 3: if len(UserPassword) > 0: if len(UserPassword) < 32: #add padding to UserPassword else: #UserPassword = padding UserPassword = md5(UserPassword) if Revision == 3: for i in range(50): UserPassword = md5(UserPassword) for esi in range(13): for edx in range(Length/8): key[edx] = UserPassword[edx] ^ esi #here is the overflow initialize_arc4_key(key) ``` As can be seen from the pseudocode above, algorithm revision must be set to 3. Also, in examined use cases of this function, the `UserPassword` will always be blank, length 0, meaning that the UserPassword will be initialized to the fixed value of padding which is equal to magic value "28bf4e5e4e758a4164004e56fffa01082e2e00b6d0683e802f0ca9fe6453697a" that comes from PDF specification. This means that the attacker has limited control over overflowing bytes as it always depends on this fixed string (51 iterations of md5 of it, to be precise) and past contents of the stack. By controlling the size of the overwrite, data past the stack cookie and return address can be overwritten potentially leading to further abuse in certain circumstances. Detection of PDF files specifically crafted to trigger this vulnerability can be based on the presence of objects encrypted with revision 3 of the encryption algorithm (the exact algorithm is specified in PDF specification version 1.4) with abnormally, illegally, large `Length` value. ### Exploit Proof-of-Concept The vulnerability can be triggered with the supplied test case in the `filter` standalone KeyView binary shipped with IBM Domino, or by sending it as an attachment with an email to a Domino mail server. Timeline * 2016-02-09 - Vendor Notification * 2016-06-08 – Public Disclosure