The role of artificial intelligence in detecting and preventing fraud in online payment systems

Abstract

The rapid expansion of digital payment systems has been accompanied by increasingly sophisticated financial fraud, including synthetic identity fraud and account takeover, while rule-based detection remains largely reactive and poorly suited to non-linear, high-dimensional transaction data. This study evaluates and compares the performance of supervised machine learning models for fraud detection in online payment systems. A quantitative, experimental design was applied to the publicly available Credit Card Fraud Detection dataset released by the Machine Learning Group of the Université Libre de Bruxelles in collaboration with Worldline, comprising 284,807 European card transactions recorded over two days in September 2013, of which 492 (0.172%) are fraudulent. The feature set consists of twenty-eight variables obtained by principal component analysis, together with transaction time and amount. The data were partitioned 70:30 with stratification, and the Synthetic Minority Over-sampling Technique was applied to the training partition only. Random Forest, XGBoost and a feed-forward Neural Network were trained under identical preprocessing conditions and evaluated using precision, recall, F1-score, accuracy and AUC-ROC. XGBoost produced the strongest results, with precision of 0.92, recall of 0.88, F1-score of 0.90 and AUC-ROC of 0.99, followed by the Neural Network and Random Forest. The findings indicate that ensemble gradient boosting offers the most favourable balance between fraud capture and false-alarm cost on severely imbalanced payment data. Inference latency and a live rule-based baseline were not measured and remain necessary steps before operational deployment.