Generated by scanning assets/openxai_neurips.pdf and the local
assets/OpenXAI-main codebase. Paths and line numbers below are relative to
assets/OpenXAI-main.
Paper: NeurIPS 2022 OpenXAICodebase: OpenXAI-mainLocal data files are not checked in
Executive Summary
Metric count
The paper says OpenXAI provides 22 metrics: 8 faithfulness metrics, 3 stability
metrics, and 11 fairness metrics. In this checkout, openxai/evaluator.py
registers the 11 base metrics only. The fairness metrics are described as subgroup
disparities over those same 11 metric values, but are not implemented as first-class
Evaluator metrics.
Model agnosticity
Ground-truth faithfulness metrics are only LR-ground-truth ready through the public
Evaluator, because the evaluator asks the model for
return_ground_truth_importance(), implemented on logistic regression.
PGI, PGU, RIS, and ROS are conceptually model agnostic, though this code expects a
PyTorch-style callable model. RRS needs a hidden representation.
XAI technique agnosticity
Most base metrics are agnostic to the explanation technique itself when the method
returns feature attributions. Comparisons require the same predictor, explained output,
feature coordinates, and a documented attribution convention. Stability metrics also
require explanations on perturbed samples. Equal vector dimensions alone do not
establish that two explanation methods measure the same quantity.
Dataset availability
The paper discusses one synthetic dataset and seven real tabular datasets. The code can
download train/test splits for all eight datasets from Harvard Dataverse, but this local
folder does not include the CSVs or pretrained model weights. Reproducible use requires
local dataset files or network access. The model loader always downloads supported
pretrained weights; offline use additionally requires an explicit local loading path.
LEX Evaluation
LEX constructs local linear explanations from learned exemplars. First specify whether
LEX, LIME, and SHAP explain the same fixed predictor, or whether LEX is evaluated as a
predictor with its own explanations. Across different predictors, PGI/PGU also reflect
differences in model sensitivity and cannot isolate explanation quality. Choose metrics
appropriate to the stated claim and define the comparison contract first.
Primary comparison metrics
Provisionally use PGI, PGU, and RIS for
a fixed predictor. Match the explained output, preprocessing, feature ordering, K range,
perturbations, and evaluation inputs. Document whether vectors contain local coefficients,
input-weighted contributions, or baseline-relative contributions; these quantities are
not interchangeable. Report sensitivity to perturbation scale and randomness. Low RIS
can also reward a constant, uninformative explanation, so pair it with faithfulness.
Two kinds of ground truth
FA, RA, SA, SRA, RC, and PRA compare explanations with a declared
reference. The standard evaluator uses fitted LR weights, including on real datasets;
this tests model-coefficient agreement. Recovering a synthetic generating rule is a
separate claim. SynthGauss masks and per-cluster weights are not automatically used by
the evaluator. Retain per-instance references, align units and signs, and adapt or call
the metric functions per instance. A binary mask supports feature recovery but does
not define signed weight ordering.
Additional claim-dependent metrics
Use RRS when the predictor exposes a meaningful internal representation,
and ROS when the output representation is defined consistently.
Include fairness gaps for subgroup-parity claims, irrespective of wrapper availability.
For LEX as a predictor, also report predictive error and local surrogate fidelity;
feature-ranking metrics alone do not assess those properties.
LEX integration: The local LexRegressor.py
(repository-root path, outside OpenXAI) returns scalar regression predictions and local
coefficients including an intercept. Exclude the intercept from feature attributions.
PGI/PGU require a scalar-output adapter or implementation change; RIS/ROS currently use
class labels to select perturbations and explanation targets, so regression requires an
explicitly defined neighborhood and explainer interface. Recommended result columns are
PGI ↑, PGU ↓, RIS ↓, valid sample counts, and optional sparsity/runtime. Label model-reference
agreement separately from generating-rule recovery.
Model Agnosticity Legend
Tag
Meaning used in this report
LR reference in this checkout
Requires a reference attribution vector. The built-in provider returns the fitted binary LR weight difference; the metric concept does not require LR, and the evaluator accepts a custom model exposing the same method.
Needs an internal representation
RRS requires access to an internal representation through predict_layer. It does not require gradients. Differentiability is a separate requirement of some explainers.
Model agnostic
Model agnostic as a metric: it can work with decision trees if the model is wrapped to expose predictions in the expected interface.
This report also calls out XAI technique agnosticity. A metric is XAI-technique
agnostic when it only consumes feature-attribution vectors or an explainer API, so it can
compare LIME, SHAP, LEX, and gradient methods without depending on how those explanations
were generated.
Notation
Let \(x_i\in\mathbb R^d\) be an input, \(a_i=\phi(x_i)\in\mathbb R^d\) its
attribution vector for a fixed explained target, and \(g_i\in\mathbb R^d\) its reference.
For any attribution vector \(v\in\mathbb R^d\), such as \(a_i\) or \(g_i\),
write \(t_q(v)\) for the feature index at position \(q\) after sorting features by
descending \(|v_j|\), and \(T_K(v)=\{t_1(v),\ldots,t_K(v)\}\), where \(1\le K\le d\).
Sorting breaks ties to select exactly K features; tie order depends on the sorting routine.
Distinguish this position from the code's dense rank
\(r_v(j)=1+|\{|v_\ell|:|v_\ell|>|v_j|\}|\), which counts distinct larger magnitudes.
Use \(\operatorname{sgn}(v_j)\in\{-1,0,1\}\) for sign.
Let \(F(x)\) be the vector of class probabilities and \(c_i=\arg\max_c F_c(x_i)\).
In PGI/PGU, \(f_i(x)=F_0(x)\) describes the current implementation;
\(f_i(x)=F_{c_i}(x)\) is a proposed multiclass convention with \(c_i\) held fixed.
Let \(o(x)\) denote logits in ROS and \(h(x)\) the selected hidden activation in RRS.
Define \(D_p(u,v)=\|u-v\|_p/\|u\|_p\) for \(\|u\|_p>0\).
In stability formulas, \(\widehat{\mathcal N}_i\) is the finite set of accepted sampled
perturbations with unchanged predicted class, not the full mathematical neighborhood.
Ratios with undefined relative changes or zero denominator are excluded.
Ground truth and aggregation: For FA/RA/SA/SRA the standard LR reference is
\(g_i=(2c_i-1)(w_1-w_0)\); PRA/RC use \(w_1-w_0\) and ignore signs. This is model-coefficient
ground truth, not automatically the synthetic generating rule (paper p. 4;
openxai/model.py:74-75, openxai/metrics.py:80).
For FA/RA/SA/SRA/PGI/PGU, AUC=True integrates \(M_i(K)\) at
\(u_K=(K-1)/(K_{\max}-1)\), \(K=1,\ldots,K_{\max}\), with the trapezoidal rule.
If \(K_{\max}=1\), the code returns \(M_i(1)\) directly.
\(K_{\max}=\lceil kd\rceil\) for fractional \(0<k<1\).
Stability means exclude removed NaN instances; RC uses a plain mean and may propagate NaNs.
Implementation Map
Evidence and scope: Static review of the supplied PDF and local source,
plus small numerical checks of ranking behavior; installation and the full benchmark
have not been reproduced. Paper references use printed page numbers.
The supplied PDF refers to an appendix for detailed equations and experimental settings,
but that appendix is absent from this file. Displayed formulas describe the local
implementation; alignment statements compare it with the available paper prose, subject
to the stated caveats. LEX and publication recommendations are reviewer judgments.
Paper definitions: pp. 6–7
(ground-truth faithfulness), p. 7
(predictive faithfulness, stability, and fairness).
Reviewed source fingerprints (SHA-256; review date 2026-09-12)
These identify the supplied PDF and key implementation files, rather than an inferred upstream release.
Pages 8–9, Tables 2–5 compare six explanation
methods and a random baseline. On the displayed LR benchmarks, gradient methods score
strongly on unsigned ranking agreement, while LIME scores strongly on signed agreement.
Stability rankings vary across datasets; the fairness analysis illustrates PGU subgroup
disparities. These findings motivate complementary metrics and do not establish a
universally best explainer or validate LEX.
The paper's stability tables include negative values, whereas the implemented raw ratios
are nonnegative. The script also prints log(mean_score) and
log(std_err); the latter is not a standard error for the former.
Do not equate raw and transformed results or claim table reproduction without resolving
the reporting convention (evaluate_metrics.py:108-110).
This excerpt shows the base metric registry. It also shows that fairness metrics are not
registered in the local evaluator.
Symbols: \(x_i\) is the instance, \(a_i\) is its attribution vector, \(T_K(a_i)\) is the top-K important feature set, \(x'\) is a perturbed sample, \(P(\cdot)\) is the perturbation process, and \(f_i\) is the scalar model output defined in the notation section.
Intuition: Perturb the features the explanation says are important and average how much the model output changes. This is useful because a faithful explanation should identify features whose removal or corruption has a large predictive effect.
Evaluation protocol in this code:
Which features are perturbed: PGI ranks features by \(|a_{ij}|\), takes the top \(K\), and builds a mask where those important features are marked False. The perturbation class interprets False entries as the dimensions to change and keeps True entries fixed. PGU reuses the same code but flips this mask, so PGI and PGU differ only in whether top-K or non-top-K features are perturbed.
How perturbations are generated: The default experiment uses NormalPerturbation with std=0.1. Continuous selected features become \(x_{ij} + \epsilon\), where \(\epsilon \sim \mathcal{N}(0, 0.1^2)\). Discrete selected features are flipped with probability \(\sqrt{2/\pi}\cdot 0.1 \approx 0.08\). For the German dataset, NewDiscrete_NormalPerturbation handles one-hot categorical groups by sampling a different category when a discrete feature is flipped.
What is calculated across \(K\): The default config sets k=0.25. The helper converts fractional \(k\) to \(K_{\max}=\lceil 0.25d\rceil\), where \(d\) is the number of features. With AUC=True, the code computes a per-instance PGI value for every \(K=1,2,\ldots,K_{\max}\). For each one of those K values, it perturbs the top-K features and averages the prediction gap over 100 perturbations. It then integrates that PGI-vs-K curve with sklearn.metrics.auc over the normalized axis \(0,1/(K_{\max}-1),\ldots,1\). If AUC=False, it only computes PGI once at \(K=K_{\max}\). One code caveat: this helper handles fractional \(k\) and \(k=-1\), but in this version it does not explicitly return a positive integer \(k\), so the default fractional setting is the safer path.
How the expectation is estimated: For each test instance and each K, the code draws n_samples=100 perturbations, evaluates the model, and averages \(|f_i(x')-f_i(x_i)|\). It returns the per-instance PGI distribution and its mean.
Plotting and saved values: The code does not produce a plot of PGI against \(K\). The K-wise array is created internally as metric_distr_all_ks, reduced to a per-instance AUC vector, and the experiment script saves that final per-instance vector as .npy. So the usual table entry is the average PGI-AUC over test instances, not a plotted curve and not a single-instance PGI value.
How reported experiment scores are aggregated:evaluate_metrics.py evaluates PGI over up to n_test_samples=1000 by default, prints the mean plus a standard error, and saves the per-instance scores.
Paper–code comparison: The code perturbs the top-K important features, runs the model on the
perturbations, and averages the absolute output gap. This matches the paper's predictive
faithfulness idea. A code detail: for binary outputs it uses column 0 of the model output,
not an explicit predicted-class column; for binary softmax this has the same magnitude as
the class-1 gap. The metric itself is not binary-only, but this implementation is
binary-oriented because of the hard-coded [:, 0] output slice. The easiest
multiclass extension is to define \(c_i=\arg\max_c f_c(x_i)\) and compute
\(\mathbb{E}_{x'}[|f_{c_i}(x')-f_{c_i}(x_i)|]\), so the score tracks the original
predicted class rather than always using output column 0.
Model agnosticity: Model agnostic as a metric. It only needs model outputs, perturbations,
and explanations. The local implementation expects a PyTorch-style callable model, but a
decision tree could be wrapped.
XAI technique agnosticity: Yes. It only uses the attribution ranking to decide
which features to perturb, so it supports LEX vs LIME vs SHAP under the comparison contract above.
Symbols: \(\overline{T_K(a_i)}\) is the complement of the top-K feature set, so it represents the features the explanation treats as unimportant. The remaining symbols match PGI.
Intuition: Perturb the supposedly unimportant features and measure whether the prediction stays stable. This is useful as a negative-control check: good explanations should produce low PGU because unimportant features should not drive the model output.
Evaluation protocol in this code:
What changes versus PGI: PGU calls the same predictive-faithfulness function as PGI but passes invert=True. The top-K mask is built from \(|a_{ij}|\) and then inverted, so the perturbation method changes the features outside the explanation's top-K set.
What is calculated across \(K\): It uses the same prediction_metrics defaults as PGI: std=0.1, n_samples=100, k=0.25, and AUC=True. So the code computes PGU for \(K=1,\ldots,K_{\max}\), where \(K_{\max}=\lceil 0.25d\rceil\), and then integrates the PGU-vs-K curve with sklearn.metrics.auc. At each K, PGU perturbs the complement of top-K, \(\overline{T_K(a_i)}\), not merely the bottom-K features.
Plotting and saved values: No PGU plot is produced. The K-wise scores are only an intermediate array inside eval_pred_faithfulness; with AUC=True, only the reduced per-instance PGU-AUC vector is returned and saved by the experiment script.
Aggregation: The function returns per-instance PGU scores and their mean. evaluate_metrics.py reports the mean across up to 1000 test instances plus a standard error. Lower PGU is better because unimportant-feature perturbations should not change model output much.
Paper–code comparison: PGU reuses PGI code with invert=True, which flips the static
mask so the implementation perturbs unimportant features. This matches the paper. Lower
PGU is better in the paper tables. It inherits PGI's binary-oriented implementation
detail: the current code measures the absolute gap in output column 0 via
[:, 0]. For multiclass models, the simplest extension is the same as PGI:
use the original predicted class \(c_i=\arg\max_c f_c(x_i)\) and average
\(|f_{c_i}(x')-f_{c_i}(x_i)|\) over perturbations of \(\overline{T_K(a_i)}\).
Model agnosticity: Model agnostic. It can work with decision trees through an output wrapper;
no gradients or hidden layers are needed by the metric itself.
XAI technique agnosticity: Yes. Like PGI, it only needs each method's feature
ranking and model outputs after perturbation.
Code snippet
openxai/evaluator.py:25
25 'PGU': {'invert': True},
openxai/metrics.py:240-248
240 def _single_k_pred_faith(k, inputs, explanations, invert, n_jobs, params):
241 if n_jobs is not None:
242 with utils.tqdm_joblib(tqdm(desc=f"Computing {'PGU' if invert else 'PGI'}", total=len(inputs))) as progress_bar:
243 metric_distr = Parallel(n_jobs=n_jobs)(
244 delayed(_single_idx_pred_faith)(i, input, explanation, k, invert, *params)\
245 for i, (input, explanation) in enumerate(zip(inputs, explanations)))
247 metric_distr = np.array([_single_idx_pred_faith(i, input, explanation, k, invert, *params)\
248 for i, (input, explanation) in enumerate(tqdm(zip(inputs, explanations)))])
openxai/explainers/perturbation_methods.py:158-160
158 # keeping features static that are in top-K based on feature mask
159 perturbed_samples = original_sample * feature_mask + perturbations * (~feature_mask)
160 return perturbed_samples
Symbols: \(\widehat{\mathcal N}_i\) is the accepted sampled neighborhood for instance \(i\), \(\phi(x)\) is the explainer output, and \(D_p(u,v)\) is the relative Lp-distance between two vectors.
Intuition: Look for the worst nearby perturbation where the explanation changes a lot relative to the input change. This is useful because locally faithful explanations should not jump wildly when the input barely changes.
Evaluation protocol in this code:
Neighborhood construction: Stability metrics do not use top-K feature subsets. The code perturbs all features using the configured perturbation method with std=1e-5, draws n_samples=1000 candidates, filters to samples with the same predicted class as \(x_i\), and keeps up to n_perturbations=100.
What is recomputed: The explainer is run once on the original instance and again on each accepted perturbation. This matters for LEX: the metric evaluates whether LEX's explanation function is stable under tiny input changes, not just whether one fixed explanation vector is stable.
Ratio and aggregation: RIS uses the input denominator and relative \(L_2\) distances by default. It skips undefined relative changes and zero denominators, then takes each instance's largest valid sampled ratio. An instance with no valid ratio returns NaN; the outer function drops it before averaging and does not return its index. Report valid counts and exclusion rates, and preserve identities for subgroup analysis. There is no K sweep or AUC.
Paper–code comparison: The code searches sampled same-prediction perturbations and returns the
maximum ratio between relative explanation change and relative input change. This follows
the paper's relative-stability idea (p. 7); the displayed formula describes the sampled code calculation.
Model agnosticity: Model agnostic as a metric, assuming the explainer can generate
explanations for the model. No hidden layer or differentiability is required by RIS itself.
XAI technique agnosticity: Yes, with an explainer wrapper. LIME, SHAP, and LEX can
all be evaluated if they can produce explanations for perturbed samples through the same API.
Symbols: \(o(x)\) is the model output representation used in the denominator, implemented here with logits. \(\phi\), \(D_p\), and \(\widehat{\mathcal N}_i\) are the same stability symbols as RIS.
Intuition: Compare explanation change to model-output change, then take the worst local case. This is useful because explanation instability is less concerning when the model output itself changes substantially, but suspicious when output barely moves.
Evaluation protocol in this code:
Same perturbation loop as RIS: ROS uses the same tiny-noise neighborhood: std=1e-5, n_samples=1000, up to n_perturbations=100 same-prediction perturbations, and \(L_2\) relative distances by default.
Different denominator: The numerator is still relative explanation change. The denominator is relative model-output-representation change, implemented with model.predict_with_logits, not post-softmax probability. Very small output changes can produce large ratios; zero or undefined denominators are skipped.
Aggregation: The metric keeps the maximum valid sampled ratio per instance, drops instances with no valid ratio, and averages the retained scores. Report valid counts and preserve input identities as for RIS. There is no top-K parameter or AUC curve.
Paper–code comparison: The implementation uses the same max relative-ratio framework as RIS
and RRS. One difference from the paper prose: the paper describes output prediction
probabilities, while this code uses predict_with_logits for the denominator.
Model agnosticity: Model agnostic in concept. The local code requires
predict_with_logits, predictions, and the model's abbrv
attribute. A probability-based tree wrapper changes the denominator convention.
Even equivalent classifiers can have different logit representations, so specify the
representation when comparing ROS.
XAI technique agnosticity: Yes, with an explainer wrapper. It compares explanation
changes against output changes and is not tied to LIME, SHAP, LEX, or gradients.
Symbols: \(h(x)\) is an internal model representation, such as a hidden-layer activation. The numerator is explanation change and the denominator is representation change.
Intuition: Ask whether explanations change more than the model's internal representation changes. This is useful for neural models because two inputs with similar hidden states should generally have similar explanations.
Evaluation protocol in this code:
Same stability sampling as RIS/ROS: The code draws tiny perturbations with std=1e-5, filters to the same predicted class, recomputes explanations, and takes the worst valid relative-ratio case per instance.
Hidden representation requirement: RRS changes only the denominator: it uses model.predict_layer(..., hidden_layer_idx=0, post_act=True). That is why this metric is not decision-tree agnostic in practice. For logistic regression, the implementation silently substitutes ROS because LR has no hidden layer.
Aggregation: There is no K sweep or AUC. The result is the mean of valid per-instance sampled maxima after NaN instances are removed. Report retained counts and exclusion rates; the script's standard error covers the retained scores only.
Paper–code comparison: The code uses model.predict_layer(... hidden_layer_idx=0 ...)
as the representation denominator. This matches the paper's "relative to model
representation" idea. For LR, the implementation explicitly substitutes ROS because an LR
model has no hidden representation.
Model agnosticity: Needs an internal representation, but no gradients.
The local implementation uses a neural hidden activation. A decision tree would need
a separately defined representation and a compatible interface.
XAI technique agnosticity: Mostly yes for the explanation method, but restricted by
the predictive model. It can compare LIME, SHAP, and LEX explanations only when the model
exposes the hidden representation used in the denominator.
Code snippet
openxai/evaluator.py:28
28 'RRS': {'metric': 'RRS'},
openxai/metrics.py:145-147
145 inputs = utils.convert_to_tensor(inputs)
146 metric = 'ROS' if (model.abbrv=='lr') and (metric=='RRS') else metric # RRS is equivalent to ROS for LR models
147 params = [explainer, model, perturb_method, feature_metadata, metric, n_samples, n_perturbations, p_norm, seed]
openxai/metrics.py:288-290
288 elif metric == 'RRS':
289 input_repr = model.predict_layer(input, hidden_layer_idx=0, post_act=True)
290 pert_repr = model.predict_layer(x_prime, hidden_layer_idx=0, post_act=True)
openxai/model.py:120-131
120 def predict_layer(self, x, hidden_layer_idx=0, post_act=True):
127 if hidden_layer_idx >= len(self.network) // 2:
128 raise ValueError(f'The model has only {len(self.network) // 2} hidden layers, but hidden layer {hidden_layer_idx} was requested (indexing starts at 0).')
130 network_idx = 2 * hidden_layer_idx + int(post_act)
131 return self.network[:network_idx+1](x)
6. Pairwise Rank Agreement (PRA)
LR reference in this checkoutGround-truth faithfulness
Symbols: \(d\ge2\) is the number of features; \(j,l\) are feature indices. \(r_{a_i}\) and \(r_{g_i}\) are dense magnitude ranks, and \(\mathbf{1}[B]\) is 1 when condition B holds and 0 otherwise.
Intuition: Check every pair of features and ask whether the explanation orders that pair the same way as ground truth. This is useful because it rewards correct relative importance even when exact rank positions differ.
Evaluation protocol in this code:
Inputs: PRA consumes saved explanation vectors and a single ground-truth importance vector supplied by the LR model. In evaluate_metrics.py, k and AUC are removed for PRA because the metric compares all feature pairs rather than top-K prefixes.
Computation: For each test instance, the code ranks features by absolute attribution magnitude with dense ranks, ranks LR ground-truth coefficients the same way, and checks every feature pair for matching relative order. Signs are ignored.
Aggregation: Each instance gets a pairwise agreement fraction in \([0,1]\); the experiment reports the mean and standard error across up to 1000 test instances. Higher is better.
Paper–code comparison: The implementation agrees on strictly ordered inputs,
but ties expose a substantive mismatch.
The paper describes matching pairwise ordering (p. 7). The code compares strict
< tests on dense ranks of absolute attributions. A pair tied in both
vectors counts as agreement, but a tie versus a strict ordering can also count as
agreement because both tests can be false. Thus agreement can depend on column order.
For \(a=(1,1)\), \(g=(1,2)\), the score is 1; reversing both vectors gives 0.
This numerical check exposes a tie-handling defect, especially relevant to sparse
explanations. The displayed formula documents the code, not a corrected tie policy.
Model agnosticity: Needs LR in this implementation. The raw function can compare any
explanation against any supplied ground-truth vector, but the public evaluator supplies
ground truth through LR coefficients.
XAI technique agnosticity: Yes, for feature-attribution outputs. It can compare
LIME, SHAP, LEX, and gradient explanations as long as each method returns one attribution
value per feature.
Symbols: \(r_{a_i}(j)\) and \(r_{g_i}(j)\) are dense ranks of feature j by attribution magnitude. Pearson correlation is taken across the d features, separately for each instance.
Intuition: Convert explanation and ground truth into ranked lists and measure how similarly the ranks move together. This is useful for summarizing global ordering agreement in one continuous score.
Evaluation protocol in this code:
Inputs: RC uses the same LR-supplied ground-truth vector as PRA and the same saved explanation matrix. It does not use \(K\), perturbations, or AUC.
Computation: The code computes Pearson correlation between dense ranks of attribution magnitudes. Standard Spearman uses average ranks for ties; the calculations agree without ties but can differ with ties. Constant rank vectors yield undefined correlation, and the plain mean does not exclude resulting NaNs.
Aggregation: It returns a per-instance correlation distribution and the mean over the evaluated test set. Higher is better, with 1 meaning identical rank order.
Paper–code comparison: The paper specifies Spearman correlation (p. 7);
this checkout uses dense-rank Pearson correlation over magnitudes. For
\(a=(4,3,3,3,2,1)\), \(g=(6,5,4,3,2,1)\), numerical checks give 0.93159
for this implementation and 0.94112 for standard Spearman. Label results
“dense-rank Pearson (OpenXAI RC)” and document ties; the definitions are not
unconditionally equivalent.
Model agnosticity: Needs LR through Evaluator, because ground truth is
pulled from LR coefficients. Raw use is possible with any provided ground-truth vector.
XAI technique agnosticity: Yes, for feature-attribution outputs. LIME, SHAP,
LEX, and gradient methods can all be ranked and compared by this metric.
Symbols: \(T_K(a_i)\) is the top-K feature set from the explanation, \(T_K(g_i)\) is the top-K ground-truth feature set, \(\cap\) keeps common features, and \(|\cdot|\) counts them.
Intuition: Count how many top-K features the explanation gets right, ignoring order and sign. This is useful as a simple feature-selection score for whether the explainer finds the right variables.
Evaluation protocol in this code:
Ground truth construction: FA uses LR coefficients as ground truth through the evaluator. The code multiplies the LR ground-truth vector by \(2\hat{y}_i-1\), so class-1 predictions use the coefficient direction and class-0 predictions use the negated direction.
K and AUC: The default ground_truth_metrics config sets k=0.25 and AUC=True. The code converts this to \(K_{\max}=\lceil 0.25d\rceil\), computes FA for every \(K=1,\ldots,K_{\max}\), then integrates the per-instance curve over a normalized K-axis. With AUC=False, it reports only FA at \(K_{\max}\).
Aggregation: There are no perturbations. Each instance gets a top-K overlap or FA-AUC value, and the experiment reports the mean across test instances. Higher is better.
Paper–code comparison: The implementation directly computes top-K overlap. It can optionally
integrate the score over K. The metric helper flips the ground-truth vector by predicted class,
which is appropriate for binary LR explanations but should be documented when comparing signs.
Model agnosticity: Needs LR in this codebase. It needs a ground-truth explanation,
and the evaluator only obtains that from LR.
XAI technique agnosticity: Yes. It only needs each explainer's top-K feature set,
so it can compare LIME, SHAP, LEX, and other methods with an aligned attribution convention.
Code snippet
openxai/evaluator.py:21
21 'FA': {'metric': 'overlap'},
openxai/metrics.py:80-88
80 ground_truths = (predictions*2-1)[:, None] * np.repeat(ground_truth.reshape(1, -1), len(predictions), axis=0)
81 max_k = utils.convert_k_to_int(k, n_features)
82 if AUC and max_k > 1:
83 metric_distr_all_ks = np.array([_single_k_gt_faith(explanations, ground_truths, k, metric) for k in range(1, max_k + 1)])
84 metric_distr = np.array([auc(np.arange(max_k)/(max_k-1), metric_distr_all_ks[:, i]) for i in range(n_inputs)])
85 else:
86 metric_distr = _single_k_gt_faith(explanations, ground_truths, max_k, metric)
88 return metric_distr, np.mean(metric_distr)
openxai/metrics.py:187-189,230-234
187 topk_idxs = [np.argsort(-np.abs(attr), axis=1)[:, :k] for attr in attrs]
188 topk_idxs_dfs = [('feat' + pd.DataFrame(topk_idx).applymap(str)) for topk_idx in topk_idxs]
230 if metric in ['overlap', 'sign']: # FA, SA
231 topk_sets = [set(list(row)) for row in topk_idxs_df.to_numpy()]
232 topk_sets_gt = [set(list(row)) for row in topk_idxs_df_gt.to_numpy()]
233 metric_distr = np.array([len(topk_set.intersection(topk_set_gt))/k\
234 for topk_set, topk_set_gt in zip(topk_sets, topk_sets_gt)])
9. Rank Agreement (RA)
LR reference in this checkoutGround-truth faithfulness
Symbols: \(t_q(a_i)\) and \(t_q(g_i)\) are feature indices at sorted position q. \(r_{a_i}\) and \(r_{g_i}\) give dense magnitude ranks. Both feature identity and dense rank must match.
Intuition: Give credit only when a top-K feature appears in the same ranked position as ground truth. This is useful when exact ordering matters, not just recovering the right feature set.
Evaluation protocol in this code:
Ground truth and K sweep: RA uses the same LR coefficient ground truth and prediction-conditioned sign flip as FA. Defaults are also the same: \(K_{\max}=\lceil 0.25d\rceil\) and AUC=True, so the usual score is RA-AUC across top-K prefixes.
Computation: The code creates top-K feature lists ordered by absolute attribution, attaches dense-rank labels to those features, and then compares the explanation and ground-truth top-K lists position by position. A match requires the same feature token and rank token at the same top-K position.
Aggregation: The per-instance score is the matched-position fraction, or its AUC over K when enabled. The experiment reports the mean and standard error over test instances. Higher is better.
Paper–code comparison: The paper describes matching features at the same
rank position (pp. 6–7). The code additionally compares dense-rank tokens at sorted
positions. These coincide without ties; with ties, sorted positions and dense ranks
differ, and tie-breaking can affect the result. The formula records both code conditions,
rather than assuming a uniquely specified tie policy in the paper.
Model agnosticity: Needs LR through the evaluator for the same LR-ground-truth reason
as FA.
XAI technique agnosticity: Yes. It compares ordered feature-attribution lists,
not the mechanism used to produce them.
Code snippet
openxai/evaluator.py:22
22 'RA': {'metric': 'rank'},
openxai/metrics.py:187-193,235-237
187 topk_idxs = [np.argsort(-np.abs(attr), axis=1)[:, :k] for attr in attrs]
188 topk_idxs_dfs = [('feat' + pd.DataFrame(topk_idx).applymap(str)) for topk_idx in topk_idxs]
190 if 'rank' in metric: # RA, SRA
191 all_feat_ranks = [rankdata(-np.abs(attr), method='dense', axis=1) for attr in attrs]
192 topk_ranks = [np.take_along_axis(all_feat_rank, topk_idx, axis=1) for all_feat_rank, topk_idx in zip(all_feat_ranks, topk_idxs)]
193 topk_idxs_dfs = [topk_idxs_df + ('rank' + pd.DataFrame(topk_rank).applymap(str)) for topk_idxs_df, topk_rank in zip(topk_idxs_dfs, topk_ranks)]
235 elif metric in ['rank', 'ranksign']: # RA, SRA
236 metric_distr = (topk_idxs_df.to_numpy() == topk_idxs_df_gt.to_numpy()).sum(axis=1)/k
237 return metric_distr
10. Sign Agreement (SA)
LR reference in this checkoutGround-truth faithfulness
Symbols: \(j\) ranges over features shared by the explanation and ground-truth top-K sets, and \(\operatorname{sgn}(a_{ij})\) and \(\operatorname{sgn}(g_{ij})\) indicate positive or negative contribution direction.
Intuition: Count top-K matches only when the direction of effect also agrees. This is useful because a feature can be important but misleading if the explainer says it pushes the prediction the wrong way.
Evaluation protocol in this code:
Ground truth and K sweep: SA uses LR coefficient ground truth, prediction-conditioned sign flips, k=0.25, and AUC=True by default. Like FA, it computes top-K scores for \(K=1,\ldots,\lceil 0.25d\rceil\) and integrates them unless AUC is disabled.
Computation: The code appends each feature's sign to its feature token and then computes set overlap. This means a feature only counts if it is selected in both top-K sets and has the same sign. Exact zero attribution receives sign 0.
Aggregation: The output is a per-instance signed-overlap fraction or SA-AUC, then the mean over test instances. Higher is better.
Paper–code comparison: The code appends the sign to each top-K feature token and computes set
overlap. This matches the paper's definition. Note that the code uses
np.sign, so exact zero attributions become sign 0.
Model agnosticity: Uses the built-in LR reference. It needs a signed ground-truth attribution vector; the
public evaluator obtains that from LR.
XAI technique agnosticity: Yes, if the method returns signed attributions. For
LEX, align the coefficient sign convention with the explained target and reference.
Code snippet
openxai/evaluator.py:23
23 'SA': {'metric': 'sign'},
openxai/metrics.py:194-196,230-234
194 if 'sign' in metric: # SA, SRA
195 topk_signs = [np.take_along_axis(np.sign(attr).astype(int), topk_idx, axis=1) for attr, topk_idx in zip(attrs, topk_idxs)]
196 topk_idxs_dfs = [topk_idxs_df + ('sign' + pd.DataFrame(topk_sign).applymap(str)) for topk_idxs_df, topk_sign in zip(topk_idxs_dfs, topk_signs)]
230 if metric in ['overlap', 'sign']: # FA, SA
231 topk_sets = [set(list(row)) for row in topk_idxs_df.to_numpy()]
232 topk_sets_gt = [set(list(row)) for row in topk_idxs_df_gt.to_numpy()]
233 metric_distr = np.array([len(topk_set.intersection(topk_set_gt))/k\
234 for topk_set, topk_set_gt in zip(topk_sets, topk_sets_gt)])
11. Signed Rank Agreement (SRA)
LR reference in this checkoutGround-truth faithfulness
Symbols: \(t_q(v)\) identifies the feature at sorted position q, \(r_v\) gives its dense magnitude rank, and \(\operatorname{sgn}\) checks the signed attribution for that feature. All three conditions must hold.
Intuition: Require the feature, rank, and sign to all match ground truth. This is useful as the strictest top-K ground-truth metric when both ordering and directional interpretation matter.
Evaluation protocol in this code:
Ground truth and K sweep: SRA uses the same LR-only ground truth setup as FA, RA, and SA. With default AUC=True, it computes strict signed-rank agreement for \(K=1,\ldots,\lceil 0.25d\rceil\) and reports the AUC of that curve.
Computation: The code builds top-K feature tokens that include both dense-rank information and sign information. A position matches only when the feature identity, rank token, and sign token all match the corresponding ground-truth token.
Aggregation: Each test instance gets a strict matched-position fraction or SRA-AUC. The experiment reports the mean plus standard error across test instances. Higher is better, but the metric is intentionally hard to score well on because it requires feature, order, and direction agreement.
Paper–code comparison: The code combines feature identity, dense rank,
and sign at each sorted position. This follows the paper's signed-rank idea (p. 7),
subject to RA's distinction between dense ranks and positions under ties, and SA's
treatment of exact zero signs.
Model agnosticity: Uses the built-in LR reference. Requires signed ground truth; the evaluator uses LR
coefficients.
XAI technique agnosticity: Yes, if the method returns signed, ranked feature
attributions. It can compare LIME, SHAP, LEX, and gradients on the same output contract.
Code snippet
openxai/evaluator.py:24
24 'SRA': {'metric': 'ranksign'},
openxai/metrics.py:190-196,235-237
190 if 'rank' in metric: # RA, SRA
191 all_feat_ranks = [rankdata(-np.abs(attr), method='dense', axis=1) for attr in attrs]
192 topk_ranks = [np.take_along_axis(all_feat_rank, topk_idx, axis=1) for all_feat_rank, topk_idx in zip(all_feat_ranks, topk_idxs)]
193 topk_idxs_dfs = [topk_idxs_df + ('rank' + pd.DataFrame(topk_rank).applymap(str)) for topk_idxs_df, topk_rank in zip(topk_idxs_dfs, topk_ranks)]
194 if 'sign' in metric: # SA, SRA
195 topk_signs = [np.take_along_axis(np.sign(attr).astype(int), topk_idx, axis=1) for attr, topk_idx in zip(attrs, topk_idxs)]
196 topk_idxs_dfs = [topk_idxs_df + ('sign' + pd.DataFrame(topk_sign).applymap(str)) for topk_idxs_df, topk_sign in zip(topk_idxs_dfs, topk_signs)]
235 elif metric in ['rank', 'ranksign']: # RA, SRA
236 metric_distr = (topk_idxs_df.to_numpy() == topk_idxs_df_gt.to_numpy()).sum(axis=1)/k
237 return metric_distr
Fairness Metrics
The paper defines 11 fairness metrics by comparing majority and minority subgroup averages
for every base faithfulness and stability metric. The paper and README do not give separate
abbreviations, so this report names them Fair-M for clarity. Their relevance
follows the evaluation claim; lack of a built-in wrapper is not a reason to omit them
from a subgroup-parity study. The panels below retain individual metric references;
the shared protocol applies to all eleven.
Local implementation status: No fairness keys appear in
metrics_dict. Compute subgroup means from scores with known input identities.
Stability evaluation removes NaNs and returns a shortened vector without input indices
(openxai/metrics.py:156-160); applying the original subgroup mask to that vector
is unsafe. Preserve indices or a validity mask in an adapted evaluator, or evaluate groups
separately. With separate calls, control per-instance seeds explicitly because the default
seed depends on the instance's position within the call.
Shared definition: For base metric \(M\), let \(V_M\) be the set of
instances with finite, defined scores and \(G\) a declared subgroup. Define
\(\bar M_G=|G\cap V_M|^{-1}\sum_{i\in G\cap V_M}M_i\) and
\(\mathrm{Fair}\text{-}M=|\bar M_{G_{\mathrm{major}}}-\bar M_{G_{\mathrm{minor}}}|\).
The gap is undefined if either valid group is empty. In the formulas below,
subgroup means use this valid-instance convention. Use the same fixed-K or AUC convention
for both groups. Report both group means, uncertainty, valid counts, and exclusion rates,
alongside the gap: zero disparity can mean equally poor explanations.
A gap alone is not a statistical significance test or a general fairness guarantee.
All model and attribution requirements are inherited from the base metric.
Symbols: \(G_{\mathrm{major}}\) and \(G_{\mathrm{minor}}\) are the majority and minority subgroups, \(PGI_i\) is the per-instance PGI value, and \(|\cdot|\) takes the absolute subgroup gap.
Intuition: Compare whether important-feature perturbation scores are similarly strong across groups. This is useful for detecting whether explanation faithfulness differs by subgroup.
Paper–code comparison: Subgroup gap over PGI. Because PGI is per-instance in code, a subgroup wrapper can be added using the shared protocol.
Model agnosticity: Inherits PGI: Model agnostic as a metric.
XAI technique agnosticity: Inherits PGI. It is directly usable for LEX vs LIME vs SHAP if subgroup labels are available.
Codebase snippet
README.md:101-102
101 #### Fairness
102 We report the average of all faithfulness and stability metric values across instances in the majority and minority subgroups, and then take the absolute difference between them to check if there are significant disparities.
Symbols: \(PGU_i\) is the per-instance unimportant-feature prediction gap, and the two \(\operatorname{mean}\) terms average it within majority and minority groups.
Intuition: Check whether explanations leave predictions equally insensitive to supposedly unimportant features across groups. This is useful for spotting subgroup differences in negative-control faithfulness.
Paper–code comparison: The paper's fairness figures discuss PGU subgroup gaps. The local code does not include a direct PGU fairness evaluator.
Model agnosticity: Inherits PGU: Model agnostic as a metric.
XAI technique agnosticity: Inherits PGU. It is technique agnostic for attribution rankings.
Codebase snippet
README.md:101-102
101 #### Fairness
102 We report the average of all faithfulness and stability metric values across instances in the majority and minority subgroups, and then take the absolute difference between them to check if there are significant disparities.
Symbols: \(RIS_i\) is the per-instance relative input stability score; the formula compares its subgroup averages.
Intuition: Measure whether explanation stability under small input perturbations is comparable across groups. This is useful because one group should not receive less stable explanations than another.
Paper–code comparison: Subgroup gap over RIS. RIS values are returned after NaN removal; retain input identities or evaluate groups separately.
Model agnosticity: Inherits RIS: Model agnostic as a metric.
XAI technique agnosticity: Inherits RIS. It works across methods that can explain perturbed samples through the same API.
Codebase snippet
README.md:101-102
101 #### Fairness
102 We report the average of all faithfulness and stability metric values across instances in the majority and minority subgroups, and then take the absolute difference between them to check if there are significant disparities.
Symbols: \(ROS_i\) is the per-instance relative output stability score. \(G_{\mathrm{major}}\) and \(G_{\mathrm{minor}}\) define which instances are averaged together.
Intuition: Compare output-relative explanation stability between subgroups. This is useful for finding cases where explanations are disproportionately volatile for one group even after accounting for output changes.
Paper–code comparison: Subgroup gap over ROS. No direct evaluator exists; use the shared protocol to account for NaN removal and input identities.
Model agnosticity: Inherits ROS: Model agnostic as a metric.
XAI technique agnosticity: Inherits ROS. It can compare LIME, SHAP, and LEX when each can explain perturbed samples.
Codebase snippet
README.md:101-102
101 #### Fairness
102 We report the average of all faithfulness and stability metric values across instances in the majority and minority subgroups, and then take the absolute difference between them to check if there are significant disparities.
Symbols: \(RRS_i\) is the per-instance representation-relative stability score; the absolute value reports the size of the subgroup disparity without choosing a direction.
Intuition: Check whether hidden-representation-relative explanation stability differs across groups. This is useful for neural models where internal representation geometry may behave unevenly across populations.
Paper–code comparison: Subgroup gap over RRS. It inherits RRS's hidden-representation requirement and the LR-to-ROS substitution in this implementation.
Model agnosticity: Inherits RRS: Needs an internal representation.
XAI technique agnosticity: Inherits RRS. The explanation method can vary, but the predictive model must expose a representation.
Codebase snippet
README.md:101-102
101 #### Fairness
102 We report the average of all faithfulness and stability metric values across instances in the majority and minority subgroups, and then take the absolute difference between them to check if there are significant disparities.
Symbols: \(FA_i\) is the per-instance feature-agreement score with ground truth; the formula subtracts subgroup mean agreement scores.
Intuition: Ask whether top-feature recovery is equally good for majority and minority groups. This is useful when ground-truth feature sets exist and subgroup parity matters.
Paper–code comparison: Matches the paper's fairness-via-explanation-quality definition. It is not implemented as a callable evaluator metric here.
Model agnosticity: Inherits FA's reference-vector requirement; the built-in provider is LR.
XAI technique agnosticity: Inherits FA. It can compare LIME, SHAP, and LEX attribution vectors if subgroup labels are available.
Codebase snippet
README.md:101-102
101 #### Fairness
102 We report the average of all faithfulness and stability metric values across instances in the majority and minority subgroups, and then take the absolute difference between them to check if there are significant disparities.
Symbols: \(RA_i\) is per-instance rank agreement; \(\operatorname{mean}\) summarizes it separately inside each subgroup.
Intuition: Test whether exact top-K ordering agreement is similarly strong across groups. This is useful when an explainer should recover not only the same features but the same rank quality for each population.
Paper–code comparison: Formula follows the paper's stated subgroup disparity wrapper. No implementation was found in the evaluator.
Model agnosticity: Inherits RA's reference-vector requirement; the built-in provider is LR.
XAI technique agnosticity: Inherits RA. It is technique agnostic for ordered feature-attribution outputs.
Codebase snippet
README.md:101-102
101 #### Fairness
102 We report the average of all faithfulness and stability metric values across instances in the majority and minority subgroups, and then take the absolute difference between them to check if there are significant disparities.
Symbols: \(SA_i\) is the signed agreement score for instance \(i\), averaged separately over majority and minority groups.
Intuition: Compare whether explanations get contribution directions right at similar rates across groups. This is useful because directional mistakes can be more harmful than merely omitting a feature.
Paper–code comparison: This is the subgroup gap over SA. It is described in prose only in this local codebase.
Model agnosticity: Inherits SA's reference-vector requirement; the built-in provider is LR.
XAI technique agnosticity: Inherits SA. It can compare methods with signed attributions, including LEX if its local coefficients are signed.
Codebase snippet
README.md:101-102
101 #### Fairness
102 We report the average of all faithfulness and stability metric values across instances in the majority and minority subgroups, and then take the absolute difference between them to check if there are significant disparities.
Symbols: \(SRA_i\) combines feature, rank, and sign agreement for one instance; the formula compares average SRA between subgroups.
Intuition: Check whether the strictest ground-truth agreement score is balanced across groups. This is useful when explanations must be equally precise and directionally correct for all populations.
Paper–code comparison: Matches the fairness wrapper described by the paper. No evaluator implementation was found.
Model agnosticity: Inherits SRA's reference-vector requirement; the built-in provider is LR.
XAI technique agnosticity: Inherits SRA. It is technique agnostic for signed, ranked feature attributions.
Codebase snippet
README.md:101-102
101 #### Fairness
102 We report the average of all faithfulness and stability metric values across instances in the majority and minority subgroups, and then take the absolute difference between them to check if there are significant disparities.
Symbols: \(RC_i\) is rank-correlation agreement with ground truth for instance \(i\); subgroup means summarize average ranking quality.
Intuition: Compare whether overall rank-correlation quality is similar across groups. This is useful when the whole importance ordering matters, not just top-K overlap.
Paper–code comparison: Subgroup gap over rank correlation. Described by the paper/README, but not registered in code.
Model agnosticity: Inherits RC's reference-vector requirement and dense-rank caveats.
XAI technique agnosticity: Inherits RC. It can compare any method that outputs a feature ranking.
Codebase snippet
README.md:101-102
101 #### Fairness
102 We report the average of all faithfulness and stability metric values across instances in the majority and minority subgroups, and then take the absolute difference between them to check if there are significant disparities.
Symbols: \(PRA_i\) is pairwise rank agreement for one instance; \(G_{\mathrm{major}}\) and \(G_{\mathrm{minor}}\) define the subgroup partitions.
Intuition: Compare pairwise ordering quality across groups. This is useful because it can reveal subgroup disparities in relative importance ordering even when top-feature overlap looks similar.
Paper–code comparison: Subgroup gap over pairwise rank agreement. No executable fairness wrapper was found.
Model agnosticity: Inherits PRA's reference-vector requirement and tie-handling defect.
XAI technique agnosticity: Inherits PRA. It can compare LIME, SHAP, LEX, and gradient rankings.
Codebase snippet
README.md:101-102
101 #### Fairness
102 We report the average of all faithfulness and stability metric values across instances in the majority and minority subgroups, and then take the absolute difference between them to check if there are significant disparities.
Getting Started
Install and import
Start from the local package root and install it editable:
The README recommends pip install -e . at lines 23-31. Note the dependency
mismatch: setup.py requires torch>=2.0.0, while
openxai/requirements.txt pins torch==1.10.
README.md:23-31
23 ## Installation
25 ### Using `pip`
27 To install the core environment dependencies of OpenXAI, use `pip` by cloning the OpenXAI repo into your local environment:
29 ```bash
30 pip install -e .
31 ```
Load data and models
Dataset files are downloaded if missing or explicitly requested. In contrast,
LoadModel(..., pretrained=True) always requests and overwrites the weights
for a supported dataset. Pre-populating ./models/pretrained does not make
that call work offline. Offline use requires a separate local-loading path that
instantiates the matching architecture and loads its state dictionary. The workflow
below assumes network access for model loading.
openxai/dataloader.py:44-48
44 if download or not os.path.isfile(path + filename):
45 self.mkdir_p(path)
46 r = requests.get(dataverse_prefix + dataverse_ids[self.split][self.data_name], allow_redirects=True)
47 df = pd.read_csv(StringIO(r.text), sep='\t')
48 df.to_csv(path + filename, index=False)
openxai/model.py:35-42
35 if pretrained:
36 model_path = './models/pretrained/'
37 os.makedirs(model_path, exist_ok=True)
39 r = requests.get(dataverse_prefix + dataverse_ids[ml_model][data_name], allow_redirects=True)
40 model_filename = f'{ml_model}_{data_name}.pt'
41 open(model_path+model_filename, 'wb').write(r.content)
42 state_dict = torch.load(model_path+model_filename, map_location=torch.device('cpu'))
Minimal workflow
Pick a dataset from adult, compas, gaussian, german, gmsc, heart, heloc, or pima.
Load train/test splits with ReturnLoaders or ReturnTrainTestX.
Load an LR or ANN model with LoadModel.
Create an explainer with Explainer(method, model, param_dict). Supported names are control, grad, ig, itg, sg, shap, and lime.
Evaluate metrics with Evaluator(model, metric).evaluate(**kwargs).
The checked-in faithfulness_demo.py passes num_samples in its kwargs at line 39,
while eval_pred_faithfulness expects n_samples. Use n_samples
in new code. Also, convert_k_to_int falls through for positive integer K;
use a supported fractional K or fix the helper before relying on integer settings.
Preprocessing affects reproduction: The default loader constructs train
and test datasets separately, fitting a separate min-max scaler to each
(openxai/dataloader.py:78-80,124-128). This can put test inputs in coordinates
inconsistent with the predictor's training coordinates. For a corrected experiment, fit
preprocessing on training data and reuse it on test data; establish compatibility with
the weights, retraining if necessary. Label that corrected protocol separately from
reproducing this checkout. Record the scaler and feature encoding because Gaussian noise
operates in transformed coordinates and is not clipped to the data range. The default
test subset is the first up to 1,000 rows, not a newly randomized sample
(openxai/dataloader.py:148).
Batch experiment scripts
To reproduce broad benchmark-style outputs, generate explanations first, then evaluate
metrics. The metric loop skips invalid combinations such as ANN with ground-truth
faithfulness metrics and LR with RRS.
evaluate_metrics.py:93-104
93 # Loop over metrics
94 for metric in metrics:
95 # Skip invalid combinations
96 if utils.invalid_model_metric_combination(model_name, metric):
97 print(f"Skipping {metric} for {model_name}")
98 continue
101 evaluator = Evaluator(model, metric=metric)
102 param_dict, param_str = _construct_param_dict(config, metric)
103 score, mean_score = evaluator.evaluate(**param_dict)
Real vs Synthetic Datasets
The paper distinguishes synthetic generating-rule references from real datasets where
that rule is usually unknown (pp. 4–5). Model-coefficient references remain available for
fitted LR on either category. The standard evaluator does not retrieve synthetic per-cluster
references. Dataset files are available through download hooks rather than this checkout.
The final table column contains reviewer judgments about experimental coverage, not
publication guarantees or findings established by the source.
What Dataverse means here: Dataverse refers to Harvard Dataverse, a public
research-data repository. The OpenXAI repo does not store the dataset CSVs directly in this
checkout. Instead, openxai/dataloader.py contains Harvard Dataverse file IDs for the
train/test splits, and the code downloads those files when the local ./data/{dataset}
files are missing. For reproducible experiments, either keep network access available or
pre-populate the expected local data folders and record the exact Dataverse file IDs used.
Dataset
Paper category
Code availability
Reviewer assessment
gaussian / SynthGauss
Synthetic; paper reports 5,000 rows and 20 continuous features.
Download ID exists in dataloader.py; a separate generator exists in dgp_synthetic.py.
Strong for controlled ground-truth and ablation claims. Weak as sole evidence for real-world performance.
german
Real; lending/credit risk.
Dataverse train/test IDs and feature metadata are present.
Useful high-stakes benchmark, but small. Good as one dataset among several, not as the only real dataset.
heloc
Real; lending/credit risk.
Dataverse train/test IDs and 23 continuous feature types are present.
Good publication dataset for credit scoring explanations because it is larger and commonly used.
compas
Real; criminal justice risk.
Dataverse train/test IDs and mixed feature metadata are present.
Publishable with careful ethical framing, subgroup definitions, and bias caveats. Avoid overgeneralizing.
adult
Real; income prediction.
Dataverse train/test IDs and 13 feature types are present.
Useful for comparability, but dated and heavily benchmarked. Best paired with newer or domain-specific data.
gmsc
Real; Give Me Some Credit, financial risk.
Dataverse train/test IDs and 10 continuous feature types are present.
Large and suitable for robustness claims, assuming licensing and preprocessing are documented.
openxai/dgp_synthetic.py:165-170,202-209
165 for i in range(self.N_clusters):
167 m = self._get_mask() # compute mask
168 X = np.random.multivariate_normal(mus[i], self.sigma, self.n_samples)
169 w_eff = m * self.w # compute masked explanation
170 pi = self._sigmoid(X @ w_eff) # compute probability output
202 var_dict = {
203 'data': X,
204 'target': y,
205 'probs': pis,
206 'masks': masks,
207 'weights': w_all,
208 'masked_weights': w_eff_all,
209 'cluster_idx': cluster_index
Reviewer recommendation: use synthetic data for controlled recovery and ablation studies,
explicitly declaring whether the reference is the generating rule or the fitted model.
Add predictive faithfulness and stability on several real datasets to support broader
claims, and subgroup gaps where relevant. Record split IDs, preprocessing, model and
explainer settings, subgroup definitions, seeds, and valid sample counts. Preserve downloaded
weights with an explicit local loader when offline reproduction is required.