Fraud: When Domain Rules Outsmart Neural Networks

The integration of domain rules into neural networks for fraud detection initially seemed promising. By adding simple rules to the loss function, the results on imbalanced data appeared spectacular at first. However, after fixing a subtle bug related to the threshold and testing the entire setup on five different random seeds, the initial enthusiasm faded.

The experiment revealed that in the context of fraud detection, where rare events are common, the way success is measured can be misleading. Thresholds, random seeds, and metrics play a crucial role and can influence the results more than the model itself.

Although the addition of domain rules slightly improved rankings, as observed in the ROC-AUC, the actual gains proved to be modest and fragile. The complete story includes details on bugs, variance, and lessons learned, highlighting the importance of rigorous model evaluation to avoid performance illusions and ensure their effectiveness in critical applications like fraud detection.