Exporter for Storage Area Network (SAN)

Related tags

Deep Learningsan_exporter
Overview

SAN Exporter

license CI Docker Pulls Code size

Prometheus exporter for Storage Area Network (SAN).

We all know that each SAN Storage vendor has their own glossary of terms, health/performance metrics and monitoring tool.

But from operator view,

  • We normally focus on some main metrics which are similar on different storage platform.
  • We are not only monitoring SAN storage but also other devices and services at multi-layer (application, virtual Machine, hypervisor, operating system and physical).

That's why we build this to have an unified monitoring/alerting solution with Prometheus and Alermanager.

Architecture overview

SAN exporter architecture

Features

There are some main features you might want to know, for others, please see example configuration.

  • Enable/disable optinal metrics for each backend
  • Enable/disable backend
  • Backend will automatically stop collecting data from SAN system after timeout seconds from last request of client. With this feature, we can deploy two instances as Active/Passive mode for high availability.

Note: Backend may not respond metrics in the first interval while collecting, calculating and caching metrics.

Quick start

  • Start a dummy driver with Docker
$ git clone [email protected]:vCloud-DFTBA/san_exporter.git
$ cd san_exporter/
$ cp examples/dummy_config.yml config.yml
# docker run --rm -p 8888:8888 -v $(pwd)/config.yml:/san-exporter/config.yml --name san-exporter daikk115/san-exporter:0.1.0

See the result at http://localhost:8888/dummy_backend

  • Start a dummy driver manually
$ git clone [email protected]:vCloud-DFTBA/san_exporter.git
$ cd san_exporter/
$ cp examples/dummy_config.yml config.yml
$ sudo apt-get install libxml2-dev libxslt1-dev python3.7-dev
$ pip3 install -r requirements.txt
$ python3.7 manage.py

See the result at http://localhost:8888/dummy_backend

Deployment

Create configuration file

# mkdir /root/san-exporter
# cp /path/to/san_exporter/examples/config.yml.sample /root/san-exporter/config.yml

Update /root/san-exporter/config.yml for corresponding to SAN storage

Run new container

# docker volume create san-exporter
# docker run -d -p 8888:8888 -v san-exporter:/var/log/ -v /root/san-exporter/config.yml:/san-exporter/config.yml --name san-exporter daikk115/san-exporter:latest

Supported Drivers

  • Matrix of driver's generic metrics
Capacity all Capacity pool IOPS/Throuhgput pool Latency pool IOPS/Throughput node Latency node CPU node RAM node IOPS/Throughput LUN Latency LUN IOPS/Throughput disk Latency disk IOPS/Throughput port Latency port Alert
HPMSA X X X X X X X X
DellUnity X X X X X X X X X X
HitachiG700 X X X
HPE3Par X X X X X X X X
NetApp X X X X X X
SC8000 X X X X X X X X X X X
V7k X X X X X X
  • Connection port requirements
    • For some SAN system, we collect metrics over SP API but some others, we collect metrics dirrectly from controller API.
    • In some special cases, we collect alerts over SSH.
SAN System Service Processor Connection Port
HPMSA NO 443
Dell Unity NO 443
Hitachi G700 YES 23451
IBM V7000 NO #TODO
IBM V5000 NO #TODO
HPE 3PAR YES #TODO
NetApp ONTAP NO 443
SC8000 NO 3033

Metrics

All metrics are prefixed with "san_" and has at least 2 labels: backend_name and san_ip

Info metrics:

Metrics name Type Help
san_storage_info gauge Basic information: serial, version, ...

Controller metrics:

Metrics name Type Help
san_totalNodes gauge Total nodes
san_masterNodes gauge Master nodes
san_onlineNodes gauge Online nodes
san_compress_support gauge Compress support, 1 = Yes, 0 = No
san_thin_provision_support gauge Thin provision support, 1 = Yes, 0 = No
san_system_reporter_support gauge System reporter support, 1 = Yes, 0 = No
san_qos_support gauge QoS support, 1 = Yes, 0 = No
san_totalCapacityMiB gauge Total system capacity in MiB
san_allocatedCapacityMiB gauge Total allocated capacity in MiB
san_freeCapacityMiB gauge Total free capacity in MiB
san_cpu_system_utilization gauge The average percentage of time that the processors on nodes are busy doing system I/O tasks
san_cpu_compression_utilization gauge The approximate percentage of time that the processor core was busy with data compression tasks
san_cpu_total gauge The cpus spent in each mode

Pool metrics:

Metrics name Type Help
san_pool_totalLUNs gauge Total LUNs (or Volumes)
san_pool_total_capacity_mib gauge Total capacity of pool in MiB
san_pool_free_capacity_mib gauge Free of pool in MiB
san_pool_provisioned_capacity_mib gauge Provisioned of pool in MiB
san_pool_number_read_io gauge Read I/O Rate - ops/s
san_pool_number_write_io gauge Write I/O Rate - ops/s
san_pool_read_cache_hit gauge Read Cache Hits - %
san_pool_write_cache_hit gauge Write Cache Hits - %
san_pool_read_kb gauge gauge Read Data Rate - KiB/s
san_pool_write_kb gauge Write Data Rate - KiB/s
san_pool_read_service_time_ms gauge Read Response Time - ms/op
san_pool_write_service_time_ms gauge Write Response Time - ms/op
san_pool_read_IOSize_kb gauge Read Transfer Size - KiB/op
san_pool_write_IOSize_kb gauge Write Transfer Size - KiB/op
san_pool_queue_length gauge Queue length of pool

Port metrics:

Metrics name Type Help
san_port_number_read_io gauge Port Read I/O Rate - ops/s
san_port_number_write_io gauge Port Write I/O Rate - ops/s
san_port_write_kb gauge Port Write Data Rate - KiB/s
san_port_read_kb gauge Port Read Data Rate - KiB/s
san_port_write_IOSize_kb gauge Port Write Transfer Size - KiB/op
san_port_read_IOSize_kb gauge Port Read Transfer Size - KiB/op
san_port_queue_length gauge Queue length of port

For more information about specific metrics of SANs, see Specific SAN Metrics

Integrate with Prometheus, Alertmanager and Grafana

Some grafana images:

SAN exporter dashboard overview

SAN exporter dashboard pool

SAN exporter dashboard port

You might also like...
Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.
Web mining module for Python, with tools for scraping, natural language processing, machine learning, network analysis and visualization.

Pattern Pattern is a web mining module for Python. It has tools for: Data Mining: web services (Google, Twitter, Wikipedia), web crawler, HTML DOM par

Computational Linguistics Research Group 8.4k Jan 3, 2023
Neurolab is a simple and powerful Neural Network Library for Python

Neurolab Neurolab is a simple and powerful Neural Network Library for Python. Contains based neural networks, train algorithms and flexible framework

null 152 Dec 6, 2022
A scikit-learn compatible neural network library that wraps PyTorch

A scikit-learn compatible neural network library that wraps PyTorch. Resources Documentation Source Code Examples To see more elaborate examples, look

null 4.9k Dec 31, 2022
Visualizer for neural network, deep learning, and machine learning models
Visualizer for neural network, deep learning, and machine learning models

Netron is a viewer for neural network, deep learning and machine learning models. Netron supports ONNX (.onnx, .pb, .pbtxt), Keras (.h5, .keras), Tens

Lutz Roeder 21k Jan 6, 2023
Graph neural network message passing reframed as a Transformer with local attention

Adjacent Attention Network An implementation of a simple transformer that is equivalent to graph neural network where the message passing is done with

Phil Wang 49 Dec 28, 2022
data/code repository of "C2F-FWN: Coarse-to-Fine Flow Warping Network for Spatial-Temporal Consistent Motion Transfer"

C2F-FWN data/code repository of "C2F-FWN: Coarse-to-Fine Flow Warping Network for Spatial-Temporal Consistent Motion Transfer" (https://arxiv.org/abs/

EKILI 46 Dec 14, 2022
Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network)
Simple command line tool for text to image generation using OpenAI's CLIP and Siren (Implicit neural representation network)

Deep Daze mist over green hills shattered plates on the grass cosmic love and attention a time traveler in the crowd life during the plague meditative

Phil Wang 4.4k Jan 3, 2023
End-to-End Object Detection with Fully Convolutional Network
End-to-End Object Detection with Fully Convolutional Network

This project provides an implementation for "End-to-End Object Detection with Fully Convolutional Network" on PyTorch.

null 472 Dec 22, 2022
TensorFlow-based neural network library
TensorFlow-based neural network library

Sonnet Documentation | Examples Sonnet is a library built on top of TensorFlow 2 designed to provide simple, composable abstractions for machine learn

DeepMind 9.5k Jan 7, 2023
Comments
  • Support purestorage please!

    Support purestorage please!

    Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

    Describe the solution you'd like A clear and concise description of what you want to happen.

    Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

    Additional context Add any other context or screenshots about the feature request here. Can you support purestorage?

    opened by wanbeepeto 0
Releases(v0.8.0)

  • v0.8.0(Aug 17, 2021)

    • Release notes:
      • Add Dell Unnity driver
      • Add Hitachi G700 driver
      • Add HPE 3PAR driver
      • Add HPMSA driver
      • Add NetApp ONTAP driver
      • Add Dell SC800 driver
      • Add IBM V7000 driver
    • Docker image: daikk115/san-exporter:0.8.0
    Source code(tar.gz)
    Source code(zip)

  • v0.1.0(Aug 15, 2021)

Owner
vCloud
Not Only vCloud - Don’t Forget To Be Awesome
vCloud
GitHub Repository
Feature board for ERPNext

ERPNext Feature Board Feature board for ERPNext Development Prerequisites k3d kubectl helm bench Install K3d Cluster # export K3D_FIX_CGROUPV2=1 # use

Revant Nandgaonkar 16 Nov 09, 2022
Gender Classification Machine Learning Model using Sk-learn in Python with 97%+ accuracy and deployment

Gender-classification This is a ML model to classify Male and Females using some physical characterstics Data. Python Libraries like Pandas,Numpy and

Aryan raj 11 Oct 16, 2022
A torch implementation of "Pixel-Level Domain Transfer"

Pixel Level Domain Transfer A torch implementation of "Pixel-Level Domain Transfer". based on dcgan.torch. Dataset The dataset used is "LookBook", fro

Fei Xia 260 Sep 02, 2022
NOD: Taking a Closer Look at Detection under Extreme Low-Light Conditions with Night Object Detection Dataset

NOD (Night Object Detection) Dataset NOD: Taking a Closer Look at Detection under Extreme Low-Light Conditions with Night Object Detection Dataset, BM

Igor Morawski 17 Nov 05, 2022
CAMoE + Dual SoftMax Loss (DSL): Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss

CAMoE + Dual SoftMax Loss (DSL): Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Loss This is official implement of "

程星 87 Dec 24, 2022
(Python, R, C/C++) Isolation Forest and variations such as SCiForest and EIF, with some additions (outlier detection + similarity + NA imputation)

IsoTree Fast and multi-threaded implementation of Extended Isolation Forest, Fair-Cut Forest, SCiForest (a.k.a. Split-Criterion iForest), and regular

141 Dec 29, 2022
[NeurIPS-2021] Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data

MosaicKD Code for NeurIPS-21 paper "Mosaicking to Distill: Knowledge Distillation from Out-of-Domain Data" 1. Motivation Natural images share common l

ZJU-VIPA 37 Nov 10, 2022
RMNA: A Neighbor Aggregation-Based Knowledge Graph Representation Learning Model Using Rule Mining

RMNA: A Neighbor Aggregation-Based Knowledge Graph Representation Learning Model Using Rule Mining Our code is based on Learning Attention-based Embed

宋朝都 4 Aug 07, 2022
Veri Setinizi Yolov5 Formatına Dönüştürün

Veri Setinizi Yolov5 Formatına Dönüştürün! Bu Repo da Neler Var? Xml Formatındaki Veri Setini .Txt Formatına Çevirme Xml Formatındaki Dosyaları Silme

Kadir Nar 4 Aug 22, 2022
This repository builds a basic vision transformer from scratch so that one beginner can understand the theory of vision transformer.

vision-transformer-from-scratch This repository includes several kinds of vision transformers from scratch so that one beginner can understand the the

1 Dec 24, 2021
DyStyle: Dynamic Neural Network for Multi-Attribute-Conditioned Style Editing

DyStyle: Dynamic Neural Network for Multi-Attribute-Conditioned Style Editing Figure: Joint multi-attribute edits using DyStyle model. Great diversity

74 Dec 03, 2022
Unofficial JAX implementations of Deep Learning models

JAX Models Table of Contents About The Project Getting Started Prerequisites Installation Usage Contributing License Contact About The Project The JAX

107 Jan 05, 2023
Using CNN to mimic the driver based on training data from Torcs

Behavioural-Cloning-in-autonomous-driving Using CNN to mimic the driver based on training data from Torcs. Approach First, the data was collected from

Sudharshan 2 Jan 05, 2022
[ICCV 2021] Amplitude-Phase Recombination: Rethinking Robustness of Convolutional Neural Networks in Frequency Domain

Amplitude-Phase Recombination (ICCV'21) Official PyTorch implementation of "Amplitude-Phase Recombination: Rethinking Robustness of Convolutional Neur

Guangyao Chen 53 Oct 05, 2022
Torch implementation of various types of GAN (e.g. DCGAN, ALI, Context-encoder, DiscoGAN, CycleGAN, EBGAN, LSGAN)

gans-collection.torch Torch implementation of various types of GANs (e.g. DCGAN, ALI, Context-encoder, DiscoGAN, CycleGAN, EBGAN). Note that EBGAN and

Minchul Shin 53 Jan 22, 2022
Deep Reinforcement Learning for mobile robot navigation in ROS Gazebo simulator

DRL-robot-navigation Deep Reinforcement Learning for mobile robot navigation in ROS Gazebo simulator. Using Twin Delayed Deep Deterministic Policy Gra

87 Jan 07, 2023
ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models (ICCV 2021 Oral)

ILVR + ADM This is the implementation of ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models (ICCV 2021 Oral). This repository is h

Jooyoung Choi 225 Dec 28, 2022
Benchmark for Answering Existential First Order Queries with Single Free Variable

EFO-1-QA Benchmark for First Order Query Estimation on Knowledge Graphs This repository contains an entire pipeline for the EFO-1-QA benchmark. EFO-1

HKUST-KnowComp 14 Oct 24, 2022
Notepy is a full-featured Notepad Python app

Notepy A full featured python text-editor Notable features Autocompletion for parenthesis and quote Auto identation Syntax highlighting Compile and ru

Mirko Rovere 11 Sep 28, 2022
Convolutional Neural Network for 3D meshes in PyTorch

MeshCNN in PyTorch SIGGRAPH 2019 [Paper] [Project Page] MeshCNN is a general-purpose deep neural network for 3D triangular meshes, which can be used f

Rana Hanocka 1.4k Jan 04, 2023