IBM Domino KeyView PDF Filter Stream Length Code Execution Vulnerability(CVE-2016-0278)

### Description An integer overflow vulnerability present in the PDF filter of KeyView as used by Domino can lead to process crash and possible arbitrary code execution. ### Tested Versions KeyView 10.16 as used by IBM Domino 9.0.1 ### Product URLs http://www-03.ibm.com/software/products/en/ibmdomino ### Details An improper check on `Length` parameter of a compressed PDF stream can result in an integer overflow leading to an unbounded `memcpy` call. Shortened test case triggering the vulnerability can be summarized as follows: ``` %PDF-1.6 47 0 obj << /Filter/ASCIIHexDecode /Length 2147483647/Root 41 0 R/Size 60/Type/XRef >>stream 414141414141 endstream >> endobj 16 %%EOF ``` In the above test case the length value is specified to be exactly 2147483647 or 0x7fffffff in hex, which is the biggest positive value of a 32bit integer. String value of length is converted into an integer by calling `strtol` function (base address of pdfsr.so being 0xB79BA000): ``` .text:B79F3343 mov dword ptr [esp+0Ch], 0 ; group .text:B79F334B mov dword ptr [esp+8], 0Ah ; base .text:B79F3353 mov dword ptr [esp+4], 0 ; endptr .text:B79F335B mov edx, [ebp+var_948] .text:B79F3361 mov [esp], edx ; nptr .text:B79F3364 call ___strtol_internal .text:B79F3369 test eax, eax .text:B79F336B js loc_B79F464A ``` If a string representing an integer supplied to `strtol` is equal to 2147483647 or bigger, `strtol` will return 0x7fffffff. The integer overflow happens later in the code, when the parser specifically checks if the destination buffer for faulting `memcpy` call is bug enough to hold the source buffer: ``` .text:B79F4314 mov ecx, [ebp+n] ; n is the value returned by strtol .text:B79F431A add ecx, 1 ; here's where the integer overflow happens .text:B79F431D mov [ebp+var_920], ecx .text:B79F4323 mov edi, [ebp+var_93C] .text:B79F4329 cmp ecx, [edi+0Ch] ; edi+0xC is size of the destination buffer, by default 0x2000 .text:B79F432C jl short loc_B79F436C ``` An integer overflow happens above when 1 is added to the length value, the result being 0x80000000. A signed comparison is made with 0x2000 and the jump will be successful. Parser concludes that the destination buffer is big enough and proceeds to call memcpy with the original length value (0x7fffffff): ``` .text:B79F436C loc_B79F436C: .text:B79F436C mov edi, [ebp+n] .text:B79F4372 mov ecx, [ebp+var_93C] .text:B79F4378 mov [ecx+8], edi .text:B79F437B mov eax, [ecx+4] .text:B79F437E mov [esp+8], edi ; n .text:B79F4382 mov edx, [ebp+src] ; n gets set at B79F3371 .text:B79F4388 mov [esp+4], edx ; src .text:B79F438C mov [esp], eax ; dest .text:B79F438F call _memcpy .text:B79F4394 mov ecx, [ebp+var_984] .text:B79F439A cmp dword ptr [ecx+11B0h], 0 .text:B79F43A1 jz short loc_B79F ``` The unbounded `memcpy` call will result in a process crash when it hits invalid memory. Detection of PDF files specifically crafted to trigger this vulnerability can be based on the abnormally large stream `/Length` value in the PDF file. The vulnerability can be triggered with the supplied test case in the `filter` standalone KeyView binary shipped with IBM Domino, or by sending it as an attachment with an email to a Domino mail server. ### Timeline * 2016-02-09 - Vendor Notification * 2016-06-08 – Public Disclosure