Homepage | Paper | Datasets | Leaderboard | Documentation
Graph Robustness Benchmark (GRB) provides scalable, unified, modular, and reproducible evaluation on the adversarial robustness of graph machine learning models. GRB has elaborated datasets, unified evaluation pipeline, modular coding framework, and reproducible leaderboards, which facilitate the developments of graph adversarial learning, summarizing existing progress and generating insights into future research.
Updates
- [11/10/2021] GRB is accepted by NeurIPS 2021 Datasets and Benchmarks Track! Find our paper in OpenReview.
- [26/09/2021] Add support for graph classification task! See tutorials in
examples/. - [16/09/2021] Add a paper list of state-of-the-art researches about adversarial robustness in graph machine learning (Keep Updating).
- [27/08/2021] Add support for modification attacks! 7 implementations and tutorials:
- [17/08/2021] Add AutoML function based on optuna for training models:
AutoTrainerin grb.trainer.trainer- Tutorial: Training models with AutoML
- [14/08/2021] Add tutorials based on jupyter notebook in
examples/:
Get Started
Installation
Install grb via pip:
pip install grb
Install grb via git:
git clone [email protected]:THUDM/grb.git
cd grb
pip install -e .
Preparation
GRB provides all necessary components to ensure the reproducibility of evaluation results. Get datasets from link or download them by running the following script:
cd ./scripts
sh download_dataset.sh
Get attack results (adversarial adjacency matrix and features) from link or download them by running the following script:
sh download_attack_results.sh
Get saved models (model weights) from link or download them by running the following script:
sh download_saved_models.sh
Usage of GRB Modules
Training a GML model
An example of training Graph Convolutional Network (GCN) on grb-cora dataset.
import torch # pytorch backend
from grb.dataset import Dataset
from grb.model.torch import GCN
from grb.trainer.trainer import Trainer
# Load data
dataset = Dataset(name='grb-cora', mode='easy',
feat_norm='arctan')
# Build model
model = GCN(in_features=dataset.num_features,
out_features=dataset.num_classes,
hidden_features=[64, 64])
# Training
adam = torch.optim.Adam(model.parameters(), lr=0.01)
trainer = Trainer(dataset=dataset, optimizer=adam,
loss=torch.nn.functional.nll_loss)
trainer.train(model=model, n_epoch=200, dropout=0.5,
train_mode='inductive')
Adversarial attack
An example of applying Topological Defective Graph Injection Attack (TDGIA) on trained GCN model.
from grb.attack.injection.tdgia import TDGIA
# Attack configuration
tdgia = TDGIA(lr=0.01,
n_epoch=10,
n_inject_max=20,
n_edge_max=20,
feat_lim_min=-0.9,
feat_lim_max=0.9,
sequential_step=0.2)
# Apply attack
rst = tdgia.attack(model=model,
adj=dataset.adj,
features=dataset.features,
target_mask=dataset.test_mask)
# Get modified adj and features
adj_attack, features_attack = rst
GRB Evaluation
Evaluation scenario (Injection Attack)
GRB provides a unified evaluation scenario for fair comparisons between attacks and defenses. The scenario is Black-box, Evasion, Inductive, Injection. Take the case of a citation-graph classification system for example. The platform collects labeled data from previous papers and trains a GML model. When a batch of new papers are submitted, it updates the graph and uses the trained model to predict labels for them.
- Black-box: Both the attacker and the defender have no knowledge about the applied methods each other uses.
- Evasion: Models are already trained in trusted data (e.g. authenticated users), which are untouched by the attackers but might have natural noises. Thus, attacks will only happen during the inference phase.
- Inductive: Models are used to classify unseen data (e.g. new users), i.e. validation or test data are unseen during training, which requires models to generalize to out of distribution data.
- Injection: The attackers can only inject new nodes but not modify the target nodes directly. Since it is usually hard to hack into users' accounts and modify their profiles. However, it is easier to create fake accounts and connect them to existing users.
GRB Leaderboard
GRB maintains leaderboards that permits a fair comparision across various attacks and defenses. To ensure the reproducibility, we provide all necessary information including datasets, attack results, saved models, etc. Besides, all results on the leaderboards can be easily reproduced by running the following scripts (e.g. leaderboard for grb-cora dataset):
sh run_leaderboard_pipeline.sh -d grb-cora -g 0 -s ./leaderboard -n 0
Usage: run_leaderboard_pipeline.sh [-d <string>] [-g <int>] [-s <string>] [-n <int>]
Pipeline for reproducing leaderboard on the chosen dataset.
-h Display help message.
-d Choose a dataset.
-s Set a directory to save leaderboard files.
-n Choose the number of an attack from 0 to 9.
-g Choose a GPU device. -1 for CPU.
Submission
We welcome researchers to submit new methods including attacks, defenses, or new GML models to enrich the GRB leaderboard. For future submissions, one should follow the GRB Evaluation Rules and respect the reproducibility.
Please submit your methods via the google form GRB submission. Our team will verify the result within a week.
Requirements
- scipy==1.5.2
- numpy==1.19.1
- torch==1.8.0
- networkx==2.5
- pandas~=1.2.3
- cogdl~=0.3.0.post1
- scikit-learn~=0.24.1
Citing GRB
Please cite our paper if you find GRB useful for your research:
@article{zheng2021grb,
title={Graph Robustness Benchmark: Benchmarking the Adversarial Robustness of Graph Machine Learning},
author={Zheng, Qinkai and Zou, Xu and Dong, Yuxiao and Cen, Yukuo and Yin, Da and Xu, Jiarong and Yang, Yang and Tang, Jie},
journal={Neural Information Processing Systems Track on Datasets and Benchmarks 2021},
year={2021}
}
Contact
In case of any problem, please contact us via email: [email protected]. We also welcome researchers to join our Google Group for further discussion on the adversarial robustness of graph machine learning.
1 Apr 19, 2022
1.1k Dec 29, 2022
8.9k Jan 04, 2023
1.5k Jan 08, 2023
8 Oct 17, 2022
40 Jul 25, 2022
24 May 20, 2022
191 Jan 07, 2023
481 Dec 29, 2022
1 Jan 06, 2022
6 Sep 09, 2021
2 Feb 11, 2022
26 Dec 07, 2022
1k Jan 05, 2023
3 Oct 23, 2022
8 May 22, 2022
1 Dec 28, 2021
43 Nov 19, 2022
36 Dec 16, 2022
65 Dec 16, 2022