Generated by scanning assets/openxai_neurips.pdf and the local
assets/OpenXAI-main codebase. Paths and line numbers below are relative to
assets/OpenXAI-main.
Paper: NeurIPS 2022 OpenXAICodebase: OpenXAI-mainLocal data files are not checked in
Executive Summary
Metric count
The paper says OpenXAI provides 22 metrics: 8 faithfulness metrics, 3 stability
metrics, and 11 fairness metrics. In this checkout, openxai/evaluator.py
registers the 11 base metrics only. The fairness metrics are described as subgroup
disparities over those same 11 metric values, but are not implemented as first-class
Evaluator metrics.
Model agnosticity
Ground-truth faithfulness metrics are only LR-ground-truth ready through the public
Evaluator, because the evaluator asks the model for
return_ground_truth_importance(), implemented on logistic regression.
PGI, PGU, RIS, and ROS are conceptually model agnostic, though this code expects a
PyTorch-style callable model. RRS needs a hidden representation.
XAI technique agnosticity
Most base metrics are agnostic to the explanation technique itself when the method
returns feature attributions. That means LIME, SHAP, LEX, and gradient explainers can
be compared under the same metrics, with the main exception being stability metrics
that require the explainer to generate explanations on perturbed samples.
Dataset availability
The paper discusses one synthetic dataset and seven real tabular datasets. The code can
download train/test splits for all eight datasets from Harvard Dataverse, but this local
folder does not include the CSVs or pretrained model weights. Reproducible use requires
network access or pre-populating ./data and ./models/pretrained.
LEX Evaluation
LEX sounds closest to LIME in output shape: a local linear explanation over input features.
The distinction is that LEX learns exemplars during training and uses them to construct the
local linear explanation. That means the most favorable metrics are the ones that reward
predictive usefulness and local consistency without assuming the explanation was produced by
perturbation fitting.
Best primary metrics
Use PGI, PGU, and RIS as the core comparison
against LIME and SHAP. They are XAI-technique agnostic for feature-attribution outputs,
work without LR ground truth, and directly test whether LEX's selected features affect
predictions and remain stable under small perturbations.
Best synthetic metrics
On SynthGauss or another dataset with known ground truth, add FA,
RA, SA, SRA, RC, and
PRA. These are useful if LEX claims its exemplar mechanism recovers the
true local linear structure. They are less suitable for real datasets where ground truth is
unavailable or only proxied by LR coefficients.
Lower-priority metrics
RRS is less favorable unless the evaluated predictive model exposes a
meaningful hidden representation. Fairness gaps are important for deployment claims, but
they are second-pass metrics for LEX method validation because this local codebase does
not implement them as first-class evaluator calls.
Recommended LEX vs LIME/SHAP table: report PGI high-is-better, PGU low-is-better, RIS
low-is-better, and explanation sparsity or runtime if you add non-OpenXAI diagnostics. On
synthetic data, add FA/RA/SRA and RC/PRA to show whether exemplar-learned explanations
recover the true feature set and ordering.
Model Agnosticity Legend
Tag
Meaning used in this report
Needs LR
Requires logistic-regression ground truth feature importances. In this implementation, the public evaluator gets them from logistic regression coefficients.
Needs gradients or hidden layer
Requires gradients or a neural-network-like hidden representation. In this codebase, RRS specifically needs predict_layer; gradient-based explainers also require differentiability.
Model agnostic
Model agnostic as a metric: it can work with decision trees if the model is wrapped to expose predictions in the expected interface.
This report also calls out XAI technique agnosticity. A metric is XAI-technique
agnostic when it only consumes feature-attribution vectors or an explainer API, so it can
compare LIME, SHAP, LEX, and gradient methods without depending on how those explanations
were generated.
Notation
Let \(x_i\) be an input, \(a_i\) its feature-attribution explanation,
\(g_i\) a ground-truth explanation vector, \(d\) the number of features,
\(T_K(v)\) the top-K feature indices by descending \(|v_j|\),
\(r_v(j)\) the rank of feature \(j\) by descending absolute attribution,
\(s(v_j)\) the sign of feature \(j\), \(f(x)\) the model output,
\(\phi(x)\) an explainer output, and \(D_p(u,v)=\|u-v\|_p/\|u\|_p\) the
relative \(L_p\) change used by the code.
The code often returns both a per-instance distribution and the mean. When
AUC=True, FA, RA, SA, SRA, PGI, and PGU are computed for
\(K=1,\ldots,K_{\max}\) and integrated with sklearn.metrics.auc.
Implementation Map
This excerpt shows the base metric registry. It also shows that fairness metrics are not
registered in the local evaluator.
Symbols: \(x_i\) is the instance, \(a_i\) is its attribution vector, \(T_K(a_i)\) is the top-K important feature set, \(x'\) is a perturbed sample, \(P(\cdot)\) is the perturbation process, and \(f\) is the predictive model output.
Intuition: Perturb the features the explanation says are important and average how much the model output changes. This is useful because a faithful explanation should identify features whose removal or corruption has a large predictive effect.
Evaluation protocol in this code:
Which features are perturbed: PGI ranks features by \(|a_{ij}|\), takes the top \(K\), and builds a mask where those important features are marked False. The perturbation class interprets False entries as the dimensions to change and keeps True entries fixed. PGU reuses the same code but flips this mask, so PGI and PGU differ only in whether top-K or non-top-K features are perturbed.
How perturbations are generated: The default experiment uses NormalPerturbation with std=0.1. Continuous selected features become \(x_{ij} + \epsilon\), where \(\epsilon \sim \mathcal{N}(0, 0.1)\). Discrete selected features are flipped with probability \(\sqrt{2/\pi}\cdot 0.1 \approx 0.08\). For the German dataset, NewDiscrete_NormalPerturbation handles one-hot categorical groups by sampling a different category when a discrete feature is flipped.
What is calculated across \(K\): The default config sets k=0.25. The helper converts fractional \(k\) to \(K_{\max}=\lceil 0.25d\rceil\), where \(d\) is the number of features. With AUC=True, the code computes a per-instance PGI value for every \(K=1,2,\ldots,K_{\max}\). For each one of those K values, it perturbs the top-K features and averages the prediction gap over 100 perturbations. It then integrates that PGI-vs-K curve with sklearn.metrics.auc over the normalized axis \(0,1/(K_{\max}-1),\ldots,1\). If AUC=False, it only computes PGI once at \(K=K_{\max}\). One code caveat: this helper handles fractional \(k\) and \(k=-1\), but in this version it does not explicitly return a positive integer \(k\), so the default fractional setting is the safer path.
How the expectation is estimated: For each test instance and each \(K\), the code draws n_samples=100 perturbations, evaluates the model on each perturbed sample, and averages \(|f(x')-f(x_i)|\) over those 100 samples. It returns both the per-instance PGI distribution and the mean across the evaluated test set.
Plotting and saved values: The code does not produce a plot of PGI against \(K\). The K-wise array is created internally as metric_distr_all_ks, reduced to a per-instance AUC vector, and the experiment script saves that final per-instance vector as .npy. So the usual table entry is the average PGI-AUC over test instances, not a plotted curve and not a single-instance PGI value.
How reported experiment scores are aggregated:evaluate_metrics.py evaluates PGI over n_test_samples=1000 by default, prints the mean plus a standard error, and saves the per-instance scores.
Alignment: The code perturbs the top-K important features, runs the model on the
perturbations, and averages the absolute output gap. This matches the paper's predictive
faithfulness idea. A code detail: for binary outputs it uses column 0 of the model output,
not an explicit predicted-class column; for binary softmax this has the same magnitude as
the class-1 gap. The metric itself is not binary-only, but this implementation is
binary-oriented because of the hard-coded [:, 0] output slice. The easiest
multiclass extension is to define \(c_i=\arg\max_c f_c(x_i)\) and compute
\(\mathbb{E}_{x'}[|f_{c_i}(x')-f_{c_i}(x_i)|]\), so the score tracks the original
predicted class rather than always using output column 0.
Model agnosticity: Model agnostic as a metric. It only needs model outputs, perturbations,
and explanations. The local implementation expects a PyTorch-style callable model, but a
decision tree could be wrapped.
XAI technique agnosticity: Yes. It only uses the attribution ranking to decide
which features to perturb, so it is directly suitable for LEX vs LIME vs SHAP.
Symbols: \(\overline{T_K(a_i)}\) is the complement of the top-K feature set, so it represents the features the explanation treats as unimportant. The remaining symbols match PGI.
Intuition: Perturb the supposedly unimportant features and measure whether the prediction stays stable. This is useful as a negative-control check: good explanations should produce low PGU because unimportant features should not drive the model output.
Evaluation protocol in this code:
What changes versus PGI: PGU calls the same predictive-faithfulness function as PGI but passes invert=True. The top-K mask is built from \(|a_{ij}|\) and then inverted, so the perturbation method changes the features outside the explanation's top-K set.
What is calculated across \(K\): It uses the same prediction_metrics defaults as PGI: std=0.1, n_samples=100, k=0.25, and AUC=True. So the code computes PGU for \(K=1,\ldots,K_{\max}\), where \(K_{\max}=\lceil 0.25d\rceil\), and then integrates the PGU-vs-K curve with sklearn.metrics.auc. At each K, PGU perturbs the complement of top-K, \(\overline{T_K(a_i)}\), not merely the bottom-K features.
Plotting and saved values: No PGU plot is produced. The K-wise scores are only an intermediate array inside eval_pred_faithfulness; with AUC=True, only the reduced per-instance PGU-AUC vector is returned and saved by the experiment script.
Aggregation: The function returns per-instance PGU scores and their mean. evaluate_metrics.py reports the mean across the default 1000 test instances plus a standard error. Lower PGU is better because unimportant-feature perturbations should not change model output much.
Alignment: PGU reuses PGI code with invert=True, which flips the static
mask so the implementation perturbs unimportant features. This matches the paper. Lower
PGU is better in the paper tables. It inherits PGI's binary-oriented implementation
detail: the current code measures the absolute gap in output column 0 via
[:, 0]. For multiclass models, the simplest extension is the same as PGI:
use the original predicted class \(c_i=\arg\max_c f_c(x_i)\) and average
\(|f_{c_i}(x')-f_{c_i}(x_i)|\) over perturbations of \(\overline{T_K(a_i)}\).
Model agnosticity: Model agnostic. It can work with decision trees through an output wrapper;
no gradients or hidden layers are needed by the metric itself.
XAI technique agnosticity: Yes. Like PGI, it only needs each method's feature
ranking and model outputs after perturbation.
Code snippet
openxai/evaluator.py:25
25 'PGU': {'invert': True},
openxai/metrics.py:240-248
240 def _single_k_pred_faith(k, inputs, explanations, invert, n_jobs, params):
241 if n_jobs is not None:
242 with utils.tqdm_joblib(tqdm(desc=f"Computing {'PGU' if invert else 'PGI'}", total=len(inputs))) as progress_bar:
243 metric_distr = Parallel(n_jobs=n_jobs)(
244 delayed(_single_idx_pred_faith)(i, input, explanation, k, invert, *params)\
245 for i, (input, explanation) in enumerate(zip(inputs, explanations)))
247 metric_distr = np.array([_single_idx_pred_faith(i, input, explanation, k, invert, *params)\
248 for i, (input, explanation) in enumerate(tqdm(zip(inputs, explanations)))])
openxai/explainers/perturbation_methods.py:158-160
158 # keeping features static that are in top-K based on feature mask
159 perturbed_samples = original_sample * feature_mask + perturbations * (~feature_mask)
160 return perturbed_samples
Symbols: \(\mathcal{N}(x_i)\) is a local neighborhood around the instance, \(\phi(x)\) is the explainer output, and \(D_p(u,v)\) is the relative Lp-distance between two vectors.
Intuition: Look for the worst nearby perturbation where the explanation changes a lot relative to the input change. This is useful because locally faithful explanations should not jump wildly when the input barely changes.
Evaluation protocol in this code:
Neighborhood construction: Stability metrics do not use top-K feature subsets. The code perturbs all features using the configured perturbation method with std=1e-5, draws n_samples=1000 candidates, filters to samples with the same predicted class as \(x_i\), and keeps up to n_perturbations=100.
What is recomputed: The explainer is run once on the original instance and again on each accepted perturbation. This matters for LEX: the metric evaluates whether LEX's explanation function is stable under tiny input changes, not just whether one fixed explanation vector is stable.
Ratio and aggregation: RIS uses the input vector itself in the denominator, computes relative \(L_2\) distances by default, skips undefined zero-denominator cases, takes the maximum ratio over accepted perturbations for each instance, and then averages those instance-level maxima over the test set. There is no \(K\) sweep or AUC for RIS.
Alignment: The code searches sampled same-prediction perturbations and returns the
maximum ratio between relative explanation change and relative input change. This follows
the paper's relative stability definition.
Model agnosticity: Model agnostic as a metric, assuming the explainer can generate
explanations for the model. No hidden layer or differentiability is required by RIS itself.
XAI technique agnosticity: Yes, with an explainer wrapper. LIME, SHAP, and LEX can
all be evaluated if they can produce explanations for perturbed samples through the same API.
Symbols: \(o(x)\) is the model output representation used in the denominator, implemented here with logits. \(\phi\), \(D_p\), and \(\mathcal{N}(x_i)\) are the same stability symbols as RIS.
Intuition: Compare explanation change to model-output change, then take the worst local case. This is useful because explanation instability is less concerning when the model output itself changes substantially, but suspicious when output barely moves.
Evaluation protocol in this code:
Same perturbation loop as RIS: ROS uses the same tiny-noise neighborhood: std=1e-5, n_samples=1000, up to n_perturbations=100 same-prediction perturbations, and \(L_2\) relative distances by default.
Different denominator: The numerator is still relative explanation change. The denominator is relative model-output-representation change, implemented with model.predict_with_logits, not post-softmax probability. Very small output changes can produce large ratios; zero or undefined denominators are skipped.
Aggregation: For each instance the metric keeps the maximum ratio across accepted perturbations, then reports the mean over test instances. There is no top-K parameter and no AUC curve.
Alignment: The implementation uses the same max relative-ratio framework as RIS
and RRS. One difference from the paper prose: the paper describes output prediction
probabilities, while this code uses predict_with_logits for the denominator.
Model agnosticity: Model agnostic as a metric. It needs an output vector only; a decision
tree can be wrapped to expose probabilities or logits.
XAI technique agnosticity: Yes, with an explainer wrapper. It compares explanation
changes against output changes and is not tied to LIME, SHAP, LEX, or gradients.
Symbols: \(h(x)\) is an internal model representation, such as a hidden-layer activation. The numerator is explanation change and the denominator is representation change.
Intuition: Ask whether explanations change more than the model's internal representation changes. This is useful for neural models because two inputs with similar hidden states should generally have similar explanations.
Evaluation protocol in this code:
Same stability sampling as RIS/ROS: The code draws tiny perturbations with std=1e-5, filters to the same predicted class, recomputes explanations, and takes the worst valid relative-ratio case per instance.
Hidden representation requirement: RRS changes only the denominator: it uses model.predict_layer(..., hidden_layer_idx=0, post_act=True). That is why this metric is not decision-tree agnostic in practice. For logistic regression, the implementation silently substitutes ROS because LR has no hidden layer.
Aggregation: As with RIS and ROS, there is no \(K\) and no AUC. The reported value is the mean of per-instance worst-case ratios, with standard error printed by the experiment script.
Alignment: The code uses model.predict_layer(... hidden_layer_idx=0 ...)
as the representation denominator. This matches the paper's "relative to model
representation" idea. For LR, the implementation explicitly substitutes ROS because an LR
model has no hidden representation.
Model agnosticity: Needs gradients or hidden layer. It can be used outside LR, but it requires a
neural-network-like representation. It is not naturally applicable to a decision tree unless
a representation is defined separately.
XAI technique agnosticity: Mostly yes for the explanation method, but restricted by
the predictive model. It can compare LIME, SHAP, and LEX explanations only when the model
exposes the hidden representation used in the denominator.
Code snippet
openxai/evaluator.py:28
28 'RRS': {'metric': 'RRS'},
openxai/metrics.py:145-147
145 inputs = utils.convert_to_tensor(inputs)
146 metric = 'ROS' if (model.abbrv=='lr') and (metric=='RRS') else metric # RRS is equivalent to ROS for LR models
147 params = [explainer, model, perturb_method, feature_metadata, metric, n_samples, n_perturbations, p_norm, seed]
openxai/metrics.py:288-290
288 elif metric == 'RRS':
289 input_repr = model.predict_layer(input, hidden_layer_idx=0, post_act=True)
290 pert_repr = model.predict_layer(x_prime, hidden_layer_idx=0, post_act=True)
openxai/model.py:120-131
120 def predict_layer(self, x, hidden_layer_idx=0, post_act=True):
127 if hidden_layer_idx >= len(self.network) // 2:
128 raise ValueError(f'The model has only {len(self.network) // 2} hidden layers, but hidden layer {hidden_layer_idx} was requested (indexing starts at 0).')
130 network_idx = 2 * hidden_layer_idx + int(post_act)
131 return self.network[:network_idx+1](x)
Symbols: \(d\) is the number of features, \(j,l\) are feature indices, \(r_a\) is the explanation rank, \(r_g\) is the ground-truth rank, and \(\mathbf{1}\) counts true comparisons.
Intuition: Check every pair of features and ask whether the explanation orders that pair the same way as ground truth. This is useful because it rewards correct relative importance even when exact rank positions differ.
Evaluation protocol in this code:
Inputs: PRA consumes saved explanation vectors and a single ground-truth importance vector supplied by the LR model. In evaluate_metrics.py, k and AUC are removed for PRA because the metric compares all feature pairs rather than top-K prefixes.
Computation: For each test instance, the code ranks features by absolute attribution magnitude with dense ranks, ranks LR ground-truth coefficients the same way, and checks every feature pair for matching relative order. Signs are ignored.
Aggregation: Each instance gets a pairwise agreement fraction in \([0,1]\); the experiment reports the mean and standard error across the default 1000 test instances. Higher is better.
Alignment: The implementation matches the paper's pairwise relative-ordering definition.
It ranks by absolute attribution and uses dense ranks. Ties are handled through strict
< comparisons, so a pair tied in both rankings is counted as agreement.
Model agnosticity: Needs LR in this implementation. The raw function can compare any
explanation against any supplied ground-truth vector, but the public evaluator supplies
ground truth through LR coefficients.
XAI technique agnosticity: Yes, for feature-attribution outputs. It can compare
LIME, SHAP, LEX, and gradient explanations as long as each method returns one attribution
value per feature.
Symbols: \(|a_i|\) and \(|g_i|\) are absolute attribution magnitudes, \(\operatorname{rank}(-|\cdot|)\) ranks larger magnitudes first, and \(\operatorname{corr}\) measures agreement between the two rank vectors.
Intuition: Convert explanation and ground truth into ranked lists and measure how similarly the ranks move together. This is useful for summarizing global ordering agreement in one continuous score.
Evaluation protocol in this code:
Inputs: RC uses the same LR-supplied ground-truth vector as PRA and the same saved explanation matrix. It does not use \(K\), perturbations, or AUC.
Computation: For each test instance, the code converts explanation magnitudes and ground-truth magnitudes into dense rank vectors, then computes Pearson correlation between those rank vectors. This is the standard way to compute a Spearman-style rank correlation.
Aggregation: It returns a per-instance correlation distribution and the mean over the evaluated test set. Higher is better, with 1 meaning identical rank order.
Alignment: The paper describes Spearman rank correlation. The code implements
Pearson correlation over rank vectors, which is the usual computational form of Spearman's
rho. It uses absolute attributions, matching the top-feature focus of the paper.
Model agnosticity: Needs LR through Evaluator, because ground truth is
pulled from LR coefficients. Raw use is possible with any provided ground-truth vector.
XAI technique agnosticity: Yes, for feature-attribution outputs. LIME, SHAP,
LEX, and gradient methods can all be ranked and compared by this metric.
Symbols: \(T_K(a_i)\) is the top-K feature set from the explanation, \(T_K(g_i)\) is the top-K ground-truth feature set, \(\cap\) keeps common features, and \(|\cdot|\) counts them.
Intuition: Count how many top-K features the explanation gets right, ignoring order and sign. This is useful as a simple feature-selection score for whether the explainer finds the right variables.
Evaluation protocol in this code:
Ground truth construction: FA uses LR coefficients as ground truth through the evaluator. The code multiplies the LR ground-truth vector by \(2\hat{y}_i-1\), so class-1 predictions use the coefficient direction and class-0 predictions use the negated direction.
K and AUC: The default ground_truth_metrics config sets k=0.25 and AUC=True. The code converts this to \(K_{\max}=\lceil 0.25d\rceil\), computes FA for every \(K=1,\ldots,K_{\max}\), then integrates the per-instance curve over a normalized K-axis. With AUC=False, it reports only FA at \(K_{\max}\).
Aggregation: There are no perturbations. Each instance gets a top-K overlap or FA-AUC value, and the experiment reports the mean across test instances. Higher is better.
Alignment: The implementation directly computes top-K overlap. It can optionally
integrate the score over K. The evaluator flips the ground-truth vector by predicted class,
which is appropriate for binary LR explanations but should be documented when comparing signs.
Model agnosticity: Needs LR in this codebase. It needs a ground-truth explanation,
and the evaluator only obtains that from LR.
XAI technique agnosticity: Yes. It only needs each explainer's top-K feature set,
so it applies cleanly to LIME, SHAP, LEX, and other attribution methods.
Code snippet
openxai/evaluator.py:21
21 'FA': {'metric': 'overlap'},
openxai/metrics.py:80-88
80 ground_truths = (predictions*2-1)[:, None] * np.repeat(ground_truth.reshape(1, -1), len(predictions), axis=0)
81 max_k = utils.convert_k_to_int(k, n_features)
82 if AUC and max_k > 1:
83 metric_distr_all_ks = np.array([_single_k_gt_faith(explanations, ground_truths, k, metric) for k in range(1, max_k + 1)])
84 metric_distr = np.array([auc(np.arange(max_k)/(max_k-1), metric_distr_all_ks[:, i]) for i in range(n_inputs)])
85 else:
86 metric_distr = _single_k_gt_faith(explanations, ground_truths, max_k, metric)
88 return metric_distr, np.mean(metric_distr)
openxai/metrics.py:187-189,230-234
187 topk_idxs = [np.argsort(-np.abs(attr), axis=1)[:, :k] for attr in attrs]
188 topk_idxs_dfs = [('feat' + pd.DataFrame(topk_idx).applymap(str)) for topk_idx in topk_idxs]
230 if metric in ['overlap', 'sign']: # FA, SA
231 topk_sets = [set(list(row)) for row in topk_idxs_df.to_numpy()]
232 topk_sets_gt = [set(list(row)) for row in topk_idxs_df_gt.to_numpy()]
233 metric_distr = np.array([len(topk_set.intersection(topk_set_gt))/k\
234 for topk_set, topk_set_gt in zip(topk_sets, topk_sets_gt)])
Symbols: \(t_q(a_i)\) is the feature at top position \(q\) in the explanation, \(t_q(g_i)\) is the ground-truth feature at that position, \(r_a\) and \(r_g\) are ranks, and \(\land\) means both conditions must hold.
Intuition: Give credit only when a top-K feature appears in the same ranked position as ground truth. This is useful when exact ordering matters, not just recovering the right feature set.
Evaluation protocol in this code:
Ground truth and K sweep: RA uses the same LR coefficient ground truth and prediction-conditioned sign flip as FA. Defaults are also the same: \(K_{\max}=\lceil 0.25d\rceil\) and AUC=True, so the usual score is RA-AUC across top-K prefixes.
Computation: The code creates top-K feature lists ordered by absolute attribution, attaches dense-rank labels to those features, and then compares the explanation and ground-truth top-K lists position by position. A match requires the same feature token and rank token at the same top-K position.
Aggregation: The per-instance score is the matched-position fraction, or its AUC over K when enabled. The experiment reports the mean and standard error over test instances. Higher is better.
Alignment: The implementation encodes feature identity plus dense rank and compares
the top-K columns position-by-position. This matches the paper's "same feature and same
rank" definition, with tie behavior inherited from rankdata(..., method='dense')
and np.argsort.
Model agnosticity: Needs LR through the evaluator for the same LR-ground-truth reason
as FA.
XAI technique agnosticity: Yes. It compares ordered feature-attribution lists,
not the mechanism used to produce them.
Code snippet
openxai/evaluator.py:22
22 'RA': {'metric': 'rank'},
openxai/metrics.py:187-193,235-237
187 topk_idxs = [np.argsort(-np.abs(attr), axis=1)[:, :k] for attr in attrs]
188 topk_idxs_dfs = [('feat' + pd.DataFrame(topk_idx).applymap(str)) for topk_idx in topk_idxs]
190 if 'rank' in metric: # RA, SRA
191 all_feat_ranks = [rankdata(-np.abs(attr), method='dense', axis=1) for attr in attrs]
192 topk_ranks = [np.take_along_axis(all_feat_rank, topk_idx, axis=1) for all_feat_rank, topk_idx in zip(all_feat_ranks, topk_idxs)]
193 topk_idxs_dfs = [topk_idxs_df + ('rank' + pd.DataFrame(topk_rank).applymap(str)) for topk_idxs_df, topk_rank in zip(topk_idxs_dfs, topk_ranks)]
235 elif metric in ['rank', 'ranksign']: # RA, SRA
236 metric_distr = (topk_idxs_df.to_numpy() == topk_idxs_df_gt.to_numpy()).sum(axis=1)/k
237 return metric_distr
Symbols: \(j\) ranges over features shared by the explanation and ground-truth top-K sets, and \(\operatorname{sign}(a_{ij})\) and \(\operatorname{sign}(g_{ij})\) indicate positive or negative contribution direction.
Intuition: Count top-K matches only when the direction of effect also agrees. This is useful because a feature can be important but misleading if the explainer says it pushes the prediction the wrong way.
Evaluation protocol in this code:
Ground truth and K sweep: SA uses LR coefficient ground truth, prediction-conditioned sign flips, k=0.25, and AUC=True by default. Like FA, it computes top-K scores for \(K=1,\ldots,\lceil 0.25d\rceil\) and integrates them unless AUC is disabled.
Computation: The code appends each feature's sign to its feature token and then computes set overlap. This means a feature only counts if it is selected in both top-K sets and has the same sign. Exact zero attribution receives sign 0.
Aggregation: The output is a per-instance signed-overlap fraction or SA-AUC, then the mean over test instances. Higher is better.
Alignment: The code appends the sign to each top-K feature token and computes set
overlap. This matches the paper's definition. Note that the code uses
np.sign, so exact zero attributions become sign 0.
Model agnosticity: Needs LR. It needs a signed ground-truth attribution vector; the
public evaluator obtains that from LR.
XAI technique agnosticity: Yes, if the method returns signed attributions. For
LEX this is favorable if its local linear coefficients have meaningful signs like LIME's.
Code snippet
openxai/evaluator.py:23
23 'SA': {'metric': 'sign'},
openxai/metrics.py:194-196,230-234
194 if 'sign' in metric: # SA, SRA
195 topk_signs = [np.take_along_axis(np.sign(attr).astype(int), topk_idx, axis=1) for attr, topk_idx in zip(attrs, topk_idxs)]
196 topk_idxs_dfs = [topk_idxs_df + ('sign' + pd.DataFrame(topk_sign).applymap(str)) for topk_idxs_df, topk_sign in zip(topk_idxs_dfs, topk_signs)]
230 if metric in ['overlap', 'sign']: # FA, SA
231 topk_sets = [set(list(row)) for row in topk_idxs_df.to_numpy()]
232 topk_sets_gt = [set(list(row)) for row in topk_idxs_df_gt.to_numpy()]
233 metric_distr = np.array([len(topk_set.intersection(topk_set_gt))/k\
234 for topk_set, topk_set_gt in zip(topk_sets, topk_sets_gt)])
Symbols: \(t_q\) identifies the feature at rank position \(q\), \(r_a\) and \(r_g\) compare ranks, and the sign term checks matching contribution direction.
Intuition: Require the feature, rank, and sign to all match ground truth. This is useful as the strictest top-K ground-truth metric when both ordering and directional interpretation matter.
Evaluation protocol in this code:
Ground truth and K sweep: SRA uses the same LR-only ground truth setup as FA, RA, and SA. With default AUC=True, it computes strict signed-rank agreement for \(K=1,\ldots,\lceil 0.25d\rceil\) and reports the AUC of that curve.
Computation: The code builds top-K feature tokens that include both dense-rank information and sign information. A position matches only when the feature identity, rank token, and sign token all match the corresponding ground-truth token.
Aggregation: Each test instance gets a strict matched-position fraction or SRA-AUC. The experiment reports the mean plus standard error across test instances. Higher is better, but the metric is intentionally hard to score well on because it requires feature, order, and direction agreement.
Alignment: The implementation combines feature id, dense rank, and sign, then
compares top-K positions. This matches the paper definition. It inherits the same tie and
zero-sign caveats as RA and SA.
Model agnosticity: Needs LR. Requires signed ground truth; the evaluator uses LR
coefficients.
XAI technique agnosticity: Yes, if the method returns signed, ranked feature
attributions. It can compare LIME, SHAP, LEX, and gradients on the same output contract.
Code snippet
openxai/evaluator.py:24
24 'SRA': {'metric': 'ranksign'},
openxai/metrics.py:190-196,235-237
190 if 'rank' in metric: # RA, SRA
191 all_feat_ranks = [rankdata(-np.abs(attr), method='dense', axis=1) for attr in attrs]
192 topk_ranks = [np.take_along_axis(all_feat_rank, topk_idx, axis=1) for all_feat_rank, topk_idx in zip(all_feat_ranks, topk_idxs)]
193 topk_idxs_dfs = [topk_idxs_df + ('rank' + pd.DataFrame(topk_rank).applymap(str)) for topk_idxs_df, topk_rank in zip(topk_idxs_dfs, topk_ranks)]
194 if 'sign' in metric: # SA, SRA
195 topk_signs = [np.take_along_axis(np.sign(attr).astype(int), topk_idx, axis=1) for attr, topk_idx in zip(attrs, topk_idxs)]
196 topk_idxs_dfs = [topk_idxs_df + ('sign' + pd.DataFrame(topk_sign).applymap(str)) for topk_idxs_df, topk_sign in zip(topk_idxs_dfs, topk_signs)]
235 elif metric in ['rank', 'ranksign']: # RA, SRA
236 metric_distr = (topk_idxs_df.to_numpy() == topk_idxs_df_gt.to_numpy()).sum(axis=1)/k
237 return metric_distr
Fairness Metrics
The paper defines 11 fairness metrics by comparing majority and minority subgroup averages
for every base faithfulness and stability metric. The paper and README do not give separate
abbreviations, so this report names them Fair-M for clarity. These are lower
priority for the current LEX evaluation unless the claim involves subgroup parity or
deployment in sensitive domains.
Local implementation status: no fairness metric keys appear in metrics_dict. To compute
these, first compute the per-instance base metric distribution, split it by subgroup, then
take the absolute difference in means.
XAI technique agnosticity is inherited from the underlying base metric. For example,
Fair-PGI can compare LIME, SHAP, and LEX as long as each method returns feature
attributions; Fair-RRS still needs a hidden representation because RRS does.
Symbols: \(G_{\mathrm{major}}\) and \(G_{\mathrm{minor}}\) are the majority and minority subgroups, \(PGI_i\) is the per-instance PGI value, and \(|\cdot|\) takes the absolute subgroup gap.
Intuition: Compare whether important-feature perturbation scores are similarly strong across groups. This is useful for detecting whether explanation faithfulness differs by subgroup.
Alignment: Subgroup gap over PGI. Because PGI is per-instance in code, the wrapper is straightforward but absent.
Model agnosticity: Inherits PGI: Model agnostic as a metric.
XAI technique agnosticity: Inherits PGI. It is directly usable for LEX vs LIME vs SHAP if subgroup labels are available.
Codebase snippet
README.md:101-102
101 #### Fairness
102 We report the average of all faithfulness and stability metric values across instances in the majority and minority subgroups, and then take the absolute difference between them to check if there are significant disparities.
Symbols: \(PGU_i\) is the per-instance unimportant-feature prediction gap, and the two \(\operatorname{mean}\) terms average it within majority and minority groups.
Intuition: Check whether explanations leave predictions equally insensitive to supposedly unimportant features across groups. This is useful for spotting subgroup differences in negative-control faithfulness.
Alignment: The paper's fairness figures discuss PGU subgroup gaps. The local code does not include a direct PGU fairness evaluator.
Model agnosticity: Inherits PGU: Model agnostic as a metric.
XAI technique agnosticity: Inherits PGU. It is technique agnostic for attribution rankings.
Codebase snippet
README.md:101-102
101 #### Fairness
102 We report the average of all faithfulness and stability metric values across instances in the majority and minority subgroups, and then take the absolute difference between them to check if there are significant disparities.
Symbols: \(RIS_i\) is the per-instance relative input stability score; the formula compares its subgroup averages.
Intuition: Measure whether explanation stability under small input perturbations is comparable across groups. This is useful because one group should not receive less stable explanations than another.
Alignment: Subgroup gap over RIS. Per-instance RIS values are returned by the stability metric, so this can be computed after evaluation.
Model agnosticity: Inherits RIS: Model agnostic as a metric.
XAI technique agnosticity: Inherits RIS. It works across methods that can explain perturbed samples through the same API.
Codebase snippet
README.md:101-102
101 #### Fairness
102 We report the average of all faithfulness and stability metric values across instances in the majority and minority subgroups, and then take the absolute difference between them to check if there are significant disparities.
Symbols: \(ROS_i\) is the per-instance relative output stability score. \(G_{\mathrm{major}}\) and \(G_{\mathrm{minor}}\) define which instances are averaged together.
Intuition: Compare output-relative explanation stability between subgroups. This is useful for finding cases where explanations are disproportionately volatile for one group even after accounting for output changes.
Alignment: Subgroup gap over ROS. Not implemented as a direct evaluator metric, but computable from ROS distributions.
Model agnosticity: Inherits ROS: Model agnostic as a metric.
XAI technique agnosticity: Inherits ROS. It can compare LIME, SHAP, and LEX when each can explain perturbed samples.
Codebase snippet
README.md:101-102
101 #### Fairness
102 We report the average of all faithfulness and stability metric values across instances in the majority and minority subgroups, and then take the absolute difference between them to check if there are significant disparities.
Symbols: \(RRS_i\) is the per-instance representation-relative stability score; the absolute value reports the size of the subgroup disparity without choosing a direction.
Intuition: Check whether hidden-representation-relative explanation stability differs across groups. This is useful for neural models where internal representation geometry may behave unevenly across populations.
Alignment: Subgroup gap over RRS. It inherits RRS's hidden-representation requirement and the LR-to-ROS substitution in this implementation.
Model agnosticity: Inherits RRS: Needs gradients or hidden layer.
XAI technique agnosticity: Inherits RRS. The explanation method can vary, but the predictive model must expose a representation.
Codebase snippet
README.md:101-102
101 #### Fairness
102 We report the average of all faithfulness and stability metric values across instances in the majority and minority subgroups, and then take the absolute difference between them to check if there are significant disparities.
Symbols: \(FA_i\) is the per-instance feature-agreement score with ground truth; the formula subtracts subgroup mean agreement scores.
Intuition: Ask whether top-feature recovery is equally good for majority and minority groups. This is useful when ground-truth feature sets exist and subgroup parity matters.
Alignment: Matches the paper's fairness-via-explanation-quality definition. It is not implemented as a callable evaluator metric here.
Model agnosticity: Inherits FA: Needs LR in this implementation.
XAI technique agnosticity: Inherits FA. It can compare LIME, SHAP, and LEX attribution vectors if subgroup labels are available.
Codebase snippet
README.md:101-102
101 #### Fairness
102 We report the average of all faithfulness and stability metric values across instances in the majority and minority subgroups, and then take the absolute difference between them to check if there are significant disparities.
Symbols: \(RA_i\) is per-instance rank agreement; \(\operatorname{mean}\) summarizes it separately inside each subgroup.
Intuition: Test whether exact top-K ordering agreement is similarly strong across groups. This is useful when an explainer should recover not only the same features but the same rank quality for each population.
Alignment: Formula follows the paper's stated subgroup disparity wrapper. No implementation was found in the evaluator.
Model agnosticity: Inherits RA: Needs LR in this implementation.
XAI technique agnosticity: Inherits RA. It is technique agnostic for ordered feature-attribution outputs.
Codebase snippet
README.md:101-102
101 #### Fairness
102 We report the average of all faithfulness and stability metric values across instances in the majority and minority subgroups, and then take the absolute difference between them to check if there are significant disparities.
Symbols: \(SA_i\) is the signed agreement score for instance \(i\), averaged separately over majority and minority groups.
Intuition: Compare whether explanations get contribution directions right at similar rates across groups. This is useful because directional mistakes can be more harmful than merely omitting a feature.
Alignment: This is the subgroup gap over SA. It is described in prose only in this local codebase.
Model agnosticity: Inherits SA: Needs LR in this implementation.
XAI technique agnosticity: Inherits SA. It can compare methods with signed attributions, including LEX if its local coefficients are signed.
Codebase snippet
README.md:101-102
101 #### Fairness
102 We report the average of all faithfulness and stability metric values across instances in the majority and minority subgroups, and then take the absolute difference between them to check if there are significant disparities.
Symbols: \(SRA_i\) combines feature, rank, and sign agreement for one instance; the formula compares average SRA between subgroups.
Intuition: Check whether the strictest ground-truth agreement score is balanced across groups. This is useful when explanations must be equally precise and directionally correct for all populations.
Alignment: Matches the fairness wrapper described by the paper. No evaluator implementation was found.
Model agnosticity: Inherits SRA: Needs LR in this implementation.
XAI technique agnosticity: Inherits SRA. It is technique agnostic for signed, ranked feature attributions.
Codebase snippet
README.md:101-102
101 #### Fairness
102 We report the average of all faithfulness and stability metric values across instances in the majority and minority subgroups, and then take the absolute difference between them to check if there are significant disparities.
Symbols: \(RC_i\) is rank-correlation agreement with ground truth for instance \(i\); subgroup means summarize average ranking quality.
Intuition: Compare whether overall rank-correlation quality is similar across groups. This is useful when the whole importance ordering matters, not just top-K overlap.
Alignment: Subgroup gap over rank correlation. Described by the paper/README, but not registered in code.
Model agnosticity: Inherits RC: Needs LR in this implementation.
XAI technique agnosticity: Inherits RC. It can compare any method that outputs a feature ranking.
Codebase snippet
README.md:101-102
101 #### Fairness
102 We report the average of all faithfulness and stability metric values across instances in the majority and minority subgroups, and then take the absolute difference between them to check if there are significant disparities.
Symbols: \(PRA_i\) is pairwise rank agreement for one instance; \(G_{\mathrm{major}}\) and \(G_{\mathrm{minor}}\) define the subgroup partitions.
Intuition: Compare pairwise ordering quality across groups. This is useful because it can reveal subgroup disparities in relative importance ordering even when top-feature overlap looks similar.
Alignment: Subgroup gap over pairwise rank agreement. No executable fairness wrapper was found.
Model agnosticity: Inherits PRA: Needs LR in this implementation.
XAI technique agnosticity: Inherits PRA. It can compare LIME, SHAP, LEX, and gradient rankings.
Codebase snippet
README.md:101-102
101 #### Fairness
102 We report the average of all faithfulness and stability metric values across instances in the majority and minority subgroups, and then take the absolute difference between them to check if there are significant disparities.
Getting Started
Install and import
Start from the local package root and install it editable:
The README recommends pip install -e . at lines 23-31. Note the dependency
mismatch: setup.py requires torch>=2.0.0, while
openxai/requirements.txt pins torch==1.10.
README.md:23-31
23 ## Installation
25 ### Using `pip`
27 To install the core environment dependencies of OpenXAI, use `pip` by cloning the OpenXAI repo into your local environment:
29 ```bash
30 pip install -e .
31 ```
Load data and models
The code downloads datasets and pretrained weights from Harvard Dataverse when needed.
This requires network access, or local files in ./data/{dataset} and
./models/pretrained.
openxai/dataloader.py:44-48
44 if download or not os.path.isfile(path + filename):
45 self.mkdir_p(path)
46 r = requests.get(dataverse_prefix + dataverse_ids[self.split][self.data_name], allow_redirects=True)
47 df = pd.read_csv(StringIO(r.text), sep='\t')
48 df.to_csv(path + filename, index=False)
openxai/model.py:35-42
35 if pretrained:
36 model_path = './models/pretrained/'
37 os.makedirs(model_path, exist_ok=True)
39 r = requests.get(dataverse_prefix + dataverse_ids[ml_model][data_name], allow_redirects=True)
40 model_filename = f'{ml_model}_{data_name}.pt'
41 open(model_path+model_filename, 'wb').write(r.content)
42 state_dict = torch.load(model_path+model_filename, map_location=torch.device('cpu'))
Minimal workflow
Pick a dataset from adult, compas, gaussian, german, gmsc, heart, heloc, or pima.
Load train/test splits with ReturnLoaders or ReturnTrainTestX.
Load an LR or ANN model with LoadModel.
Create an explainer with Explainer(method, model, param_dict). Supported names are control, grad, ig, itg, sg, shap, and lime.
Evaluate metrics with Evaluator(model, metric).evaluate(**kwargs).
The checked-in faithfulness_demo.py passes num_samples in its kwargs at line 39,
while eval_pred_faithfulness expects n_samples. Use n_samples
in new code.
Batch experiment scripts
To reproduce broad benchmark-style outputs, generate explanations first, then evaluate
metrics. The metric loop skips invalid combinations such as ANN with ground-truth
faithfulness metrics and LR with RRS.
evaluate_metrics.py:93-104
93 # Loop over metrics
94 for metric in metrics:
95 # Skip invalid combinations
96 if utils.invalid_model_metric_combination(model_name, metric):
97 print(f"Skipping {metric} for {model_name}")
98 continue
101 evaluator = Evaluator(model, metric=metric)
102 param_dict, param_str = _construct_param_dict(config, metric)
103 score, mean_score = evaluator.evaluate(**param_dict)
Real vs Synthetic Datasets
The paper positions synthetic data as the setting where ground-truth explanations are available,
and real datasets as the setting for more realistic benchmarking when such ground truth is hard
to define. This local codebase supports that split, but mostly through download hooks rather
than checked-in data files.
What Dataverse means here: Dataverse refers to Harvard Dataverse, a public
research-data repository. The OpenXAI repo does not store the dataset CSVs directly in this
checkout. Instead, openxai/dataloader.py contains Harvard Dataverse file IDs for the
train/test splits, and the code downloads those files when the local ./data/{dataset}
files are missing. For reproducible experiments, either keep network access available or
pre-populate the expected local data folders and record the exact Dataverse file IDs used.
Dataset
Paper category
Code availability
Publishing suitability
gaussian / SynthGauss
Synthetic; paper reports 5,000 rows and 20 continuous features.
Download ID exists in dataloader.py; a separate generator exists in dgp_synthetic.py.
Strong for controlled ground-truth and ablation claims. Weak as sole evidence for real-world performance.
german
Real; lending/credit risk.
Dataverse train/test IDs and feature metadata are present.
Useful high-stakes benchmark, but small. Good as one dataset among several, not as the only real dataset.
heloc
Real; lending/credit risk.
Dataverse train/test IDs and 23 continuous feature types are present.
Good publication dataset for credit scoring explanations because it is larger and commonly used.
compas
Real; criminal justice risk.
Dataverse train/test IDs and mixed feature metadata are present.
Publishable with careful ethical framing, subgroup definitions, and bias caveats. Avoid overgeneralizing.
adult
Real; income prediction.
Dataverse train/test IDs and 13 feature types are present.
Useful for comparability, but dated and heavily benchmarked. Best paired with newer or domain-specific data.
gmsc
Real; Give Me Some Credit, financial risk.
Dataverse train/test IDs and 10 continuous feature types are present.
Large and suitable for robustness claims, assuming licensing and preprocessing are documented.
openxai/dgp_synthetic.py:165-170,202-209
165 for i in range(self.N_clusters):
167 m = self._get_mask() # compute mask
168 X = np.random.multivariate_normal(mus[i], self.sigma, self.n_samples)
169 w_eff = m * self.w # compute masked explanation
170 pi = self._sigmoid(X @ w_eff) # compute probability output
202 var_dict = {
203 'data': X,
204 'target': y,
205 'probs': pis,
206 'masks': masks,
207 'weights': w_all,
208 'masked_weights': w_eff_all,
209 'cluster_idx': cluster_index
Recommendation for a publication: use SynthGauss or another synthetic dataset only for
ground-truth validation, then report predictive faithfulness, stability, and fairness gaps on
multiple real datasets across at least two domains. Include exact split IDs, preprocessing,
subgroup definitions, and metric settings, because the local checkout depends on remote
Dataverse assets.