Efficient and Interpretable Transformer for Counterfactual Fairness

Code Available at: https://github.com/PanyiDong/FairnessAwareAttention

The Problem: The Tension Between Fairness and Performance

The rapid adoption of ML in heavily regulated domains, such as insurance and finance, has created a critical tension between predictive power, model interpretability, and strict fairness regulations. Existing fairness-aware methods for tabular data typically rely on group-level metrics that conflict with risk-based pricing, or they require rigid, hard-to-validate causal graphs to ensure counterfactual fairness. Meanwhile, standard attention-heavy transformers project simple tabular data into complex, high-dimensional spaces. This obscures direct feature dependencies and makes it incredibly difficult to enforce fairness at the attention level without severely degrading model performance. We wanted to bridge this gap by designing a transparent architecture where fair attention naturally translates into fair predictions.

Our Methodology: FCorrTransformer and CAR

To solve these structural limitations, we developed the FCorrTransformer, an attention-light architecture tailored specifically for tabular data. Instead of using dense, high-dimensional embeddings, our model relies on one-dimensional embeddings. This design ensures that the attention matrix directly represents pairwise statistical dependencies between features. Leveraging this transparent structure, we introduced CAR. CAR operates by evaluating counterfactual permutations of sensitive features via an efficient input augmentation strategy, explicitly penalizing and suppressing biased dependencies directly within the attention matrix. Furthermore, to mitigate indirect discrimination, we introduced Domain Adaptation-based CAR (DACAR), which aligns non-sensitive features using CORAL mapping during training to account for correlated proxy biases.

Architecture Design of FCorrTransformer

The Results: Interpretable, and Fair AI

We rigorously evaluated our framework against standard baseline models using highly imbalanced, real-world financial and proprietary commercial insurance datasets. The FCorrTransformer paired with CAR successfully achieved strong counterfactual fairness while maintaining highly competitive predictive accuracy. Beyond fairness, our architecture vastly outperformed standard transformers in computational efficiency. Because of its attention-light design, our model drastically reduced parameter counts and utilized only a fraction of the GPU memory required by baseline models. Ultimately, this framework provides a highly interpretable, and practical pathway for deploying responsible AI in regulatory-sensitive environments.

Heatmap of Pre-SoftMax Attention Weights on BAF Data

Recommended citation: Dong, P., Quan, Z. (2026). Efficient and Interpretable Transformer for Counterfactual Fairness.
Download Paper