Dataset for the Research2Clinics @ NeurIPS 2021 Paper: What Do You See in this Patient? Behavioral Testing of Clinical NLP Models

Last update: Sep 20, 2022

Overview

Behavioral Testing of Clinical NLP Models

This repository contains code for testing the behavior of clinical prediction models based on patient letters. For a detailed description of the testing framework see our paper What Do You See in this Patient? Behavioral Testing of Clinical NLP Models.

Usage

Install requirements: pip install -r requirements.txt

Run main.py, e.g. for diagnosis prediction test on gender, age and ethnicity:

python main.py 
    --test_set_path ./path_to_test_set
    --model_path bvanaken/CORe-clinical-diagnosis-prediction
    --task diagnosis
    --shift_keys gender,age,ethnicity
    --save_dir ./results
    --gpu False

Parameter	Description
test_set_path	Path to original test set file
model_path	Path to model or Huggingface model hub checkpoint
task	Current options: diagnosis, mortality
shift_keys	Which patient characteristics to test. Current options: age, gender, ethnicity, weight, intersectional (gender + ethnicity)
save_dir	Directory to save results, default: "./results"
gpu	Whether to use a gpu during inference or not, default: False

Using Non-Transformer models

The framework currently focuses on testing Transformer-based models. However, it is easy to extend it to any other prediction model. To do so, simply create a new class implementing the Predictor interface and add it to the TASK_MAP in main.py.

Cite

@inproceedings{vanAken2021,
  author    = {Betty van Aken and
               Sebastian Herrmann and
               Alexander Löser},
  title     = {What Do You See in this Patient? Behavioral Testing of Clinical NLP Models},
  booktitle = {Bridging the Gap: From Machine Learning Research to Clinical Practice, 
               Research2Clinics Workshop @ NeurIPS 2021},
  year      = {2021}
}

Dataset for the Research2Clinics @ NeurIPS 2021 Paper: What Do You See in this Patient? Behavioral Testing of Clinical NLP Models

Related tags

Overview

Behavioral Testing of Clinical NLP Models

Usage

Using Non-Transformer models

Cite

Owner

Betty van Aken

Code for paper "Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs"

IndoNLI: A Natural Language Inference Dataset for Indonesian

Implementation of a Transformer, but completely in Triton

Semi-supervised Semantic Segmentation with Directional Context-aware Consistency (CVPR 2021)

Code for the CVPR2021 paper "Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for Place Recognition"

Benchmark datasets, data loaders, and evaluators for graph machine learning

Code of the lileonardo team for the 2021 Emotion and Theme Recognition in Music task of MediaEval 2021

Barlow Twins and HSIC

Official Pytorch implementation of "DivCo: Diverse Conditional Image Synthesis via Contrastive Generative Adversarial Network" (CVPR'21)

PyTorch implementation of PNASNet-5 on ImageNet

Official implementation for paper Render In-between: Motion Guided Video Synthesis for Action Interpolation

UltraGCN: An Ultra Simplification of Graph Convolutional Networks for Recommendation

The Pytorch implementation for "Video-Text Pre-training with Learned Regions"

Style transfer, deep learning, feature transform

A Graph Neural Network Tool for Recovering Dense Sub-graphs in Random Dense Graphs.

This is the code for the paper "Contrastive Clustering" (AAAI 2021)

Inhomogeneous Social Recommendation with Hypergraph Convolutional Networks

Pre-trained model, code, and materials from the paper "Impact of Adversarial Examples on Deep Learning Models for Biomedical Image Segmentation" (MICCAI 2019).

Repository sharing code and the model for the paper "Rescoring Sequence-to-Sequence Models for Text Line Recognition with CTC-Prefixes"

Unpaired Caricature Generation with Multiple Exaggerations