Neat VNC: Buffer overflow due to oversized RSA public keys
Neat VNC is a VNC server library. Prior to 0.9.6, a pre-authentication stack buffer overflow exists in neatvnc in the RSA-AES security type handler. An unauthenticated remote attacker who can reach the VNC listening socket can send a crafted security type 5 (RSA-AES) or security type 129 (RSA-AES-256) handshake with an oversized client RSA public key, causing rsa_aes_send_challenge in src/auth/rsa-aes.c to overflow a 1024-byte on-stack buffer when encrypting the server challenge. This results in at least a denial of service via server crash. This vulnerability is fixed in 0.9.6.