A PyTorch Implementation of "Watch Your Step: Learning Node Embeddings via Graph Attention" (NeurIPS 2018).

Last update: Dec 09, 2022

Overview

Attention Walk

⠀⠀

A PyTorch Implementation of Watch Your Step: Learning Node Embeddings via Graph Attention (NIPS 2018).

Abstract

Graph embedding methods represent nodes in a continuous vector space, preserving different types of relational information from the graph. There are many hyper-parameters to these methods (e.g. the length of a random walk) which have to be manually tuned for every graph. In this paper, we replace previously fixed hyper-parameters with trainable ones that we automatically learn via backpropagation. In particular, we propose a novel attention model on the power series of the transition matrix, which guides the random walk to optimize an upstream objective. Unlike previous approaches to attention models, the method that we propose utilizes attention parameters exclusively on the data itself (e.g. on the random walk), and are not used by the model for inference. We experiment on link prediction tasks, as we aim to produce embeddings that best-preserve the graph structure, generalizing to unseen information. We improve state-of-the-art results on a comprehensive suite of real-world graph datasets including social, collaboration, and biological networks, where we observe that our graph attention model can reduce the error by up to 20%-40%. We show that our automatically-learned attention parameters can vary significantly per graph, and correspond to the optimal choice of hyper-parameter if we manually tune existing methods.

This repository provides an implementation of Attention Walk as described in the paper:

Watch Your Step: Learning Node Embeddings via Graph Attention. Sami Abu-El-Haija, Bryan Perozzi, Rami Al-Rfou, Alexander A. Alemi. NIPS, 2018. [Paper]

The original Tensorflow implementation is available [here].

Requirements

The codebase is implemented in Python 3.5.2. package versions used for development are just below.

networkx          2.4
tqdm              4.28.1
numpy             1.15.4
pandas            0.23.4
texttable         1.5.0
scipy             1.1.0
argparse          1.1.0
torch             1.1.0
torchvision       0.3.0

Datasets

The code takes an input graph in a csv file. Every row indicates an edge between two nodes separated by a comma. The first row is a header. Nodes should be indexed starting with 0. Sample graphs for the `Twitch Brasilians` and `Wikipedia Chameleons` are included in the `input/` directory.

### Options

Learning of the embedding is handled by the src/main.py script which provides the following command line arguments.

Input and output options

  --edge-path         STR   Input graph path.     Default is `input/chameleon_edges.csv`.
  --embedding-path    STR   Embedding path.       Default is `output/chameleon_AW_embedding.csv`.
  --attention-path    STR   Attention path.       Default is `output/chameleon_AW_attention.csv`.

Model options

  --dimensions           INT       Number of embeding dimensions.        Default is 128.
  --epochs               INT       Number of training epochs.            Default is 200.
  --window-size          INT       Skip-gram window size.                Default is 5.
  --learning-rate        FLOAT     Learning rate value.                  Default is 0.01.
  --beta                 FLOAT     Attention regularization parameter.   Default is 0.5.
  --gamma                FLOAT     Embedding regularization parameter.   Default is 0.5.
  --num-of-walks         INT       Number of walks per source node.      Default is 80.

Examples

The following commands learn a graph embedding and write the embedding to disk. The node representations are ordered by the ID.

Creating an Attention Walk embedding of the default dataset with the standard hyperparameter settings. Saving this embedding at the default path.

``` python src/main.py ```

Creating an Attention Walk embedding of the default dataset with 256 dimensions.

python src/main.py --dimensions 256

Creating an Attention Walk embedding of the default dataset with a higher window size.

python src/main.py --window-size 20

Creating an embedding of another dataset the Twitch Brasilians. Saving the outputs under custom file names.

python src/main.py --edge-path input/ptbr_edges.csv --embedding-path output/ptbr_AW_embedding.csv --attention-path output/ptbr_AW_attention.csv

License

GNU License

Comments

Nan parameters

Thanks for your pytorch code. I found that my parameters become Nan during training. Nan parameters include model.left_factors, model.right_factors, model.attention. All the entries of them become Nan during training. And also the loss. I'm trying to find the reason. I would appreciate it if you could give me some help or hints.

opened by kkkkk001 9

Memory Error

I'm getting OOM errors even with small files. The attached file link_network.txt throws the following error:

Adjacency matrix powers: 100%|███████████████████████████████████████████████████████| 4/4 [00:00<00:00, 108.39it/s]
Traceback (most recent call last):
  File "src\main.py", line 79, in <module>
    main()
  File "src\main.py", line 74, in main
    model = AttentionWalkTrainer(args)
  File "E:\AttentionWalk\src\attentionwalk.py", line 70, in __init__
    self.initialize_model_and_features()
  File "E:\AttentionWalk\src\attentionwalk.py", line 76, in initialize_model_and_features
    self.target_tensor = feature_calculator(self.args, self.graph)
  File "E:\AttentionWalk\src\utils.py", line 53, in feature_calculator
    target_matrices = np.array(target_matrices)
MemoryError

I guess this is due to the large indices of the nodes. Any workarounds for this?

opened by davidlenz 2

modified normalized_adjacency_matrix calculation

As mentioned in this issue: https://github.com/benedekrozemberczki/AttentionWalk/issues/9

Added normalization into calculation, able to prevent unbalanced loss and prevent loss_on_mat to be extreme big while node count of data is big.

opened by neilctwu 1
miscalculations of normalized adjacency matrix
Thanks for sharing this awesome repo.

The issue is I found that loss_on_target will become extreme big while training from the original code, and I think is due to the miscalculation of normalized_adjacency_matrix.

From your original code, normalized_adjacency_matrix is been calculated by:

normalized_adjacency_matrix = degs.dot(adjacency_matrix)

However while the matrix hasn't been normalize but simply multiple by degree of nodes. I think the part of normalized_adjacency_matrix should be modified like its original definition:

normalized_adjacency_matrix = degs.power(-1/2)\ .dot(adjacency_matrix)\ .dot(degs.power(-1/2))

It'll turn out to be more reasonable loss shown below:

Am I understand it correctly?
opened by neilctwu 1
problem with being killed

Hi, I tried to train the model with new dataset which have about 60000 nodes, but I have a problem of getting Killed suddenly. Do you have any idea why? Thanks :)

opened by amy-hyunji 1
Directed weighted graphs

Is it possible to use the code with directed and weighted graphs? The paper states the attention walk framework for unweighted graphs only, but i'd like to use it for such types of networks. Thank you for your attention.

opened by federicoairoldi 1

Releases(v_00001)

v_00001(May 17, 2021)

Source code(tar.gz)
Source code(zip)

Owner

Benedek Rozemberczki

Machine Learning Engineer at AstraZeneca | PhD from The University of Edinburgh.

GitHub Repository

Pytorch implementation of Implicit Behavior Cloning.

Implicit Behavior Cloning - PyTorch (wip) Pytorch implementation of Implicit Behavior Cloning. Install conda create -n ibc python=3.8 pip install -r r

49 Dec 25, 2022

OOD Generalization and Detection (ACL 2020)

Pretrained Transformers Improve Out-of-Distribution Robustness How does pretraining affect out-of-distribution robustness? We create an OOD benchmark

57 Jan 09, 2023

Controlling Hill Climb Racing with Hand Tacking

Controlling Hill Climb Racing with Hand Tacking Opened Palm for Gas Closed Palm for Brake

3 Jan 18, 2022

Implementation based on Paper - Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

3 Jul 08, 2022

Towards End-to-end Video-based Eye Tracking

Towards End-to-end Video-based Eye Tracking The code accompanying our ECCV 2020 publication and dataset, EVE. Authors: Seonwook Park, Emre Aksan, Xuco

76 Dec 12, 2022

TOOD: Task-aligned One-stage Object Detection, ICCV2021 Oral

One-stage object detection is commonly implemented by optimizing two sub-tasks: object classification and localization, using heads with two parallel branches, which might lead to a certain level of

264 Jan 09, 2023

Code for "The Box Size Confidence Bias Harms Your Object Detector"

The Box Size Confidence Bias Harms Your Object Detector - Code Disclaimer: This repository is for research purposes only. It is designed to maintain r

24 Dec 07, 2022

Safe Model-Based Reinforcement Learning using Robust Control Barrier Functions

README Repository containing the code for the paper "Safe Model-Based Reinforcement Learning using Robust Control Barrier Functions". Specifically, an

13 Nov 24, 2022

Code for the paper: Sketch Your Own GAN

Sketch Your Own GAN Project | Paper | Youtube | Slides Our method takes in one or a few hand-drawn sketches and customizes an off-the-shelf GAN to mat

677 Dec 28, 2022

Companion code for the paper "Meta-Learning the Search Distribution of Black-Box Random Search Based Adversarial Attacks" by Yatsura et al.

META-RS This is the companion code for the paper "Meta-Learning the Search Distribution of Black-Box Random Search Based Adversarial Attacks" by Yatsu

7 Dec 09, 2022

A PyTorch Implementation of "Watch Your Step: Learning Node Embeddings via Graph Attention" (NeurIPS 2018).

Related tags

Overview

Attention Walk

Abstract

Requirements

Datasets

Input and output options

Model options

Examples

Comments

Nan parameters

Memory Error

modified normalized_adjacency_matrix calculation

miscalculations of normalized adjacency matrix

problem with being killed

Directed weighted graphs

Releases(v_00001)

v_00001(May 17, 2021)

Owner

Benedek Rozemberczki

Pytorch implementation of Implicit Behavior Cloning.

OOD Generalization and Detection (ACL 2020)

Controlling Hill Climb Racing with Hand Tacking

Implementation based on Paper - Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Towards End-to-end Video-based Eye Tracking

TOOD: Task-aligned One-stage Object Detection, ICCV2021 Oral

Code for "The Box Size Confidence Bias Harms Your Object Detector"

Safe Model-Based Reinforcement Learning using Robust Control Barrier Functions

Code for the paper: Sketch Your Own GAN

Companion code for the paper "Meta-Learning the Search Distribution of Black-Box Random Search Based Adversarial Attacks" by Yatsura et al.

Image morphing without reference points by applying warp maps and optimizing over them.

Decorators for maximizing memory utilization with PyTorch & CUDA

Dataset and codebase for NeurIPS 2021 paper: Exploring Forensic Dental Identification with Deep Learning

Brain tumor detection using CNN (InceptionResNetV2 Model)

DyNet: The Dynamic Neural Network Toolkit

Mesh Graphormer is a new transformer-based method for human pose and mesh reconsruction from an input image

This is the source code of the 1st place solution for segmentation task (with Dice 90.32%) in 2021 CCF BDCI challenge.

Official PyTorch implementation of Retrieve in Style: Unsupervised Facial Feature Transfer and Retrieval.

Our solution for SSN Invente 2021's Hackathon

Code for MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks