SuRE Evaluation: A Supplementary Material

This repository contains supplementary material regarding the evaluations presented in the paper Visual Exploration of Machine Learning Model Behavior with Hierarchical Surrogate Rule Sets.

The evaluation consists of three main parts:

(1) Evaluation of the rule generation algorithm (based on 10 benchmark datasets);

(2) A usability study aimed at evaluating the effectiveness of the Feature-Aligned Tree visualization;

(3) An observational study aimed at evaluating the usefulness of the interactive workflow (with 7 domain experts).

The material is organized around three main folders you can find in the repository:

• /algo_experiment

This folder contains the code used to train the models and to perform the parameter sensitivity analysis and the comparison with decision trees.

• /usability_study

This folder contains the slides used in the study to instruct the participants, the anonymous responses we collected and the notebook we used to analyze the data.

• /observational_study

This folder contains the slides we used in the interview to introduce the visualizations, the system, and the study procedure. We also include the list of tasks we collected and the task categorization we created.