A Human-Centered XAI System for Phishing Detection

Abstract

AI models are widely used in cybersecurity to identify digital content that could be malicious. This is also the case with phishing emails: when a defense system detects a suspicious message, users are typically presented with warning dialogs that inform them of the risks. However, warnings are often designed without adequate consideration of the end user, leading to incorrect decisions that lower the benefit of the AI models. To overcome the limitations identified in the literature, this paper presents an XAI phishing detection system built following a human-centered design approach. It classifies phishing emails and generates polymorphic warning dialogs that explain to the user why the email might be a scam, with the ultimate goal of supporting a more informed decision on whether or not to open suspicious content.