On Evaluation Metrics for Graph Generative Models

Authors: Rylee Thompson, Boris Knyazev, Elahe Ghalebi, Jungtaek Kim, Graham Taylor

This is the official repository for the paper On Evaluation Metrics for Graph Generative Models (hyperlink TBD). Our evaluation metrics enable the efficient computation of the distance between two sets of graphs regardless of domain. In addition, they are more expressive than previous metrics and easily incorporate continuous node and edge features in evaluation. If you're primarily interested in using our metrics in your work, please see evaluation/ for a more lightweight setup and installation and Evaluation_examples.ipynb for examples on how to utilize our code. The remainder of this README describes how to recreate our results which introduces additional dependencies.

Table of Contents

• Requirements and installation
• Reproducing main results
  • Permutation experiments
  • Pretraining GNNs
  • Generating graphs
• Visualization
• License
• Citation

Requirements and installation

The main requirements are:

• Python 3.7
• PyTorch 1.8.1
• DGL 0.6.1

pip install -r requirements.txt

Following that, install an appropriate version of DGL 0.6.1 for your system and download the proteins and ego datasets by running ./download_datasets.sh.

Reproducing main results

The arguments of our scripts are described in config.py.

Permutation experiments

Below, examples to run the scripts to run certain experiments are shown. In general, experiments can be run as:

python main.py --permutation_type={permutation type} --dataset={dataset}\
{feature_extractor} {feature_extractor_args}

For example, to run the mixing random graphs experiment on the proteins dataset using random-GNN-based metrics for a single random seed:

python main.py --permutation_type=mixing-random --dataset=proteins\
gnn

The hyperparameters of the GNN are set to our recommendations by default, however, they are easily changed by additional flags. To run the same experiment using the degree MMD metric:

python main.py --permutation_type=mixing-random --dataset=proteins\
mmd-structure --statistic=degree

Rank correlations are automatically computed and printed at the end of each experiment, and results are stored in experiment_results/. Recreating our results requires running variations of the above commands thousands of times. To generate these commands and store them in a bash script automatically, run python create_bash_script.py.

Pretraining GNNs

To pretrain a GNN for use in our permutation experiments, run python GIN_train.py, and see GIN_train.py for tweakable hyperparameters. Alternatively, the pretrained models used in our experiments can be downloaded by running ./download_pretrained_models.sh. Once you have a pretrained model, the permutation experiments can be ran using:

python main.py --permutation_type={permutation type} --dataset={dataset}\
gnn --use_pretrained {feature_extractor_args}

Generating graphs

Some of our experiments use graphs generated by GRAN. To find instructions on training and generating graphs using GRAN, please see the official GRAN repository. Alternatively, the graphs generated by GRAN used in our experiments can be downloaded by running ./download_gran_graphs.sh.

Visualization

All code for visualizing results and creating tables is found in data_visualization.ipynb.

License

We release our code under the MIT license.

Citation

@inproceedings{thompson2022evaluation,
  title={On Evaluation Metrics for Graph Generative Models},
  author={Thompson, Rylee, and Knyazev, Boris and Ghalebi, Elahe and Kim, Jungtaek, and Taylor, Graham W},
booktitle={International Conference on Learning Representations},
  year={2022}  
}