New adversarial attacks GUI models face hit 75% success, study reveals

A new study reveals that the very way artificial intelligence models click on your screen could be turned against them. Researchers have found a previously overlooked weakness in how GUI visual grounding models — the AI systems that translate on-screen instructions into actual clicks — generate coordinates, and they’ve built a working exploit called MissClick to prove it. The findings, published by researcher Yu Ran, expose a new category of adversarial attacks GUI models face as they become more embedded in automating tasks on desktop, web, and mobile platforms.

Key takeaways

• GUI visual grounding models generate screen click coordinates as sequences of digit tokens, not as raw numbers.
• A single-digit change in a high place value, like the hundreds place, can shift a click by 100 units on screen.
• MissClick is a white-box attack with two variants: MissClick-U for untargeted disruption and MissClick-T for targeted hijacking.
• MissClick-U reached untargeted success rates of 75.07% on OS-Atlas and 72.93% on UGround.
• MissClick-T reached targeted success rates of 44.86% on OS-Atlas and 62.67% on UGround.

How GUI Visual Grounding Models Generate Coordinates

GUI visual grounding models predict where to click by producing digit-by-digit outputs rather than direct numerical coordinates, and that seemingly small design choice turns out to carry real security consequences. According to the research, these models were never built with adversarial resistance to digit-level manipulation in mind, which is exactly the gap MissClick exploits.

… Continue reading the full article at the original source below.