A Pythonic client for the official https://data.gov.gr API.

Overview

pydatagovgr

PyPI PyPI - Python Version codecov PyPI - Wheel Code style: black GitHub Workflow Status

An unofficial Pythonic client for the official data.gov.gr API. Aims to be an easy, intuitive and out-of-the-box way to:

  • find data published by central government, local authorities and public bodies of Greece
  • build related products and services.

while being robust, following best-practices and eliminating developer-induced bugs.

The aspiration for this library is to enable users of different backgrounds (academia, industry, students etc.) with an interest to programmatically explore and utilize the open data of data.gov.gr, to do so without having to write-debug-maintain trivial code or worry about that.

Install

The recommended installation is via pip:

pip install pydatagovgr

Quick Usage

You must have an account on data.gov.gr to use the API service. In order to register and request an API token, submit a request in the designated official form here. The procedure is very simple and takes less than 5 minutes.

from pydatagovgr import DataGovClient


gov_client = DataGovClient(token='xoxb-1234-1243')

# fetch the COVID-19 vaccination data
covid_data = gov_client.query('mdg_emvolio')

# fetch data on Greece's internet traffic
traffic_data = gov_client.query('internet_traffic')

# fetch a list of the forest fires
fire_data = gov_client.query('mcp_forest_fires')

Features

The pydatagovgr client supports out-of-the-box all the things you know (and love), such as:

  • authentication: properly handles the authentication to data.gov.gr - all you have to do is provide a valid token.
  • persistent session: making several requests to data.gov.gr reuses the same underlying connection.
  • timeout policy: the client will stop waiting for a response from data.gov.gr after some time. Defaults to 10 sec.
  • retry policy: to account for potential server failures of lossy network connections, client automatically retries with an exponential-backoff, to avoid harming the data.gov.gr. Defaults to a maximum of 3 retries.

Also, this library comes with extensive test coverage (100%) of the core functionality. The test suite will constantly improve towards the v1 version.

Not-So-Quick Usage

The data.gov.gr API is currently organized into endpoints called datasets, each available via the query endpoint.

The pydatagovgr client provides a corresponding query method, through which every available dataset can be obtained. You can also pass additional arguments to filter the results accordingly.

from pydatagovgr import DataGovClient


gov_client = DataGovClient(token='xoxb-1234-1243')

# fetch the COVID-19 vaccination data for the 2021
data = gov_client.query('mdg_emvolio', date_from='2021-01-01', date_to='2021-12-31')

You can also use Python objects as arguments:

import datetime


data = gov_client.query('mdg_emvolio', 
                        date_from=datetime.date(2021, 1, 1), 
                        date_to=datetime.date(1, 12, 31))

Apart from the authentication token, you can also configure the timeout and retry policies of your client. For example:

from pydatagovgr import DataGovClient


# this client will stop waiting for a response after 7 seconds 
gov_client = DataGovClient(token='xoxb-1234-1243', timeout=7)

# this client will retry at most 3 times, with an exponential-backoff
# (i.e. each retry waits exponentially longer before occurs: 1, 2, 4, 8, 16, 32, 64, ... seconds)
gov_client = DataGovClient(token='xoxb-1234-1243', max_retries=3)

# this client will respect both a timeout policy and a retry policy
gov_client = DataGovClient(token='xoxb-1234-1243', timeout=7, max_retries=3)

How to contribute

If you wish to contribute, this is a great place to start!

License

Distributed under the MIT License.

Acknowledgements

All rights are reserved by the official https://data.gov.gr site, its developers, its maintainers and the Hellenic Government.

Comments
  • support api/v1/search endpoint

    support api/v1/search endpoint

    Proposal

    The https://data.gov.gr API has a POST api/v1/search endpoint available, used for the "Search datasets" functionality under "Datasets" tab.

    It would be interesting to extend the client so as to support that search functionality. For example, one may wish to search what are the available datasets returned for a particular keyword (e.g. "covid") before actually requesting a dataset through the already-available DataGovClient.query action.

    Technical Notes

    This endpoint does not require the typical authorization process (as DataGovClient.query). One can receive a successful response (200) without adding any authorization in the request headers. For that very reason, it may not make sense to simply extend DataGovClient (instantiation requires token) with a new attribute called search. A design decision must be taken here, as to how a client user should interact with the library naturally while using both query and search endpoints.

    The basic accepted paramerer is the query, through which the search functionality occurs. Additional parameters are:

    • publisher: to facilitate the "Filter by > Publisher" functionality.
    • topic: to facilitate the "Filter by > Topic" functionality.
    • pageNo: the page number (pagination-related).
    • pageSize: number of results contained in each page (pagination-related).
    • sort: the sorting method (e.g. "updated", "best", "popular")

    Additional context

    search_endpoint

    search_endpoint_2

    search_endpoint_3

    enhancement good first issue 
    opened by ilias-ant 2
  • support for python 3.11

    support for python 3.11

    Is your feature request related to a problem? Please describe. python version support.

    Describe the solution you'd like pydatagovgr to also support python 3.11 (first released: 2022-10-24)

    Additional context https://www.python.org/downloads/release/python-3110/

    opened by ilias-ant 0
  • support Python 3.10

    support Python 3.10

    See:

    Changes:

    • check for potential codebase changes
    • CI step should also check against python 3.10
    enhancement good first issue CI 
    opened by ilias-ant 0
  • build and publish documentation

    build and publish documentation

    Proposal

    • use mkdocs library
    • find a way to not duplicate documentation
    • use Read the Docs as a documentation space
    • add the publish action in the CD step
    • add dry-run publish action in the CI step (if possible)
    documentation 
    opened by ilias-ant 0
  • builds against macOS, windows in CI step

    builds against macOS, windows in CI step

    Proposal

    The CI procedure should also test against macos-latest, windows-latest.

    ...
    
    jobs:
      ci:
        runs-on: ubuntu-latest
        strategy:
          matrix:
            python-version: [3.7, 3.8, 3.9]
            os: [ubuntu-latest, macos-latest, windows-latest]
    ...
    
    
    CI 
    opened by ilias-ant 0
  • new dev-dependency: isort

    new dev-dependency: isort

    Proposal

    • use isort as development dependency, to ensure that imports in pydatagovgr/ modules are properly formatted.
    • include isort check in CI step.
    • inherit the black profile.
    good first issue housekeeping 
    opened by ilias-ant 0
  • urllib3 sub-dependency: got an unexpected keyword argument 'allowed_methods'

    urllib3 sub-dependency: got an unexpected keyword argument 'allowed_methods'

    The following dependency-bug was brought to my attention by @sp1thas (much obliged!)

    Describe the bug pydatagovgr==0.1.3 uses the newly introduced param allowed methods instead of the soon-to-be-deprecated method_whitelist in the urllib3.util.retry.Retry class. However, at the same time it allows, through its poetry.lock configuration urllib3 versions < 1.26.0 which do not have the new param available.

    To Reproduce

    from pydatagovgr import DataGovClient
    import urllib3
    
    
    urllib3.__version__
    #  '1.25.10'
    
    gov = DataGovClient(token='xoxb-1234-1243')
    

    yields:

    Traceback (most recent call last):
      File "/home/.../.../lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3418, in run_code
        exec(code_obj, self.user_global_ns, self.user_ns)
      File "<ipython-input-4-a9f7822d9844>", line 1, in <module>
        gov = DataGovClient(token='xoxb-1234-1243')
      File "/home/.../.../lib/python3.7/site-packages/pydatagovgr/base_client.py", line 38, in __init__
        self.session = self._init_session()
      File "/home/.../.../lib/python3.7/site-packages/pydatagovgr/base_client.py", line 72, in _init_session
        allowed_methods=["GET"],
    TypeError: __init__() got an unexpected keyword argument 'allowed_methods'
    

    Expected behavior Initialization of the DataGovClient should not raise any exceptions.

    Desktop (please complete the following information):

    • OS: Ubuntu 18.04.5 LTS x86_64
    • Python: 3.7.9
    bug 
    opened by ilias-ant 0
Releases(v0.3.0)
  • v0.3.0(Nov 6, 2022)

    A minor (i.e. backwards-compatible) release, with:

    Features

    • support for the newly released Python v3.11.
    • minor version upgrades for the development dependencies of the project (see: [tool.poetry.dev-dependencies]).

    Security

    CI/CD

    • CI pipeline now includes builds against Python v3.11.
    • CI pipeline has migrated codecov/codecov-action Github Action from v1 to v3.
    • CI and CD pipelines explicitly pin poetry version to 1.2.2.

    For more technical details, see PR #14.

    Source code(tar.gz)
    Source code(zip)
  • v0.2.2(Jul 24, 2022)

    A patch release, addressing a broken documentation build.

    The broken build originated from error:

    ModuleNotFoundError: No module named 'mkdocstrings.handlers.python'
    

    In that respect, mkdocs-related dependencies were updated and pinned down in a way that guarantees a successful documentation build.

    Miscellaneous:

    • most of the dev dependencies of the project have been updated to their latest versions. No change in required dependencies however (i.e. the ones that are part of the PyPI package)
    • a new entry has been added in Related Projects (datagovgR)
    Source code(tar.gz)
    Source code(zip)
  • v0.2.1(Dec 25, 2021)

    This is an artificial patch release, in order to properly trigger the CD step through Github Actions - published package version v0.2.1 contains exactly the same sourcecode as v0.2.0.

    (my personal access token had expired and v0.2.0 deployment never triggered :facepalm:)

    Source code(tar.gz)
    Source code(zip)
  • v0.2.0(Dec 25, 2021)

    A minor (i.e. backwards-compatible) Christmas-y :christmas_tree: release, with lots of goodies :socks: :gift:. Ho Ho Ho!

    Features

    • support for the newly released Python v3.10 - PR #12
    • codebase is now isort-compliant - commit 76f7729

    Docs

    • build and publish documentation, with mkdocs & Read the Docs, available at https://pydatagovgr.readthedocs.io - PR #10
    • set up a security policy - PR #7

    CI/CD

    • CI step now includes:
      • builds against MacOS and Windows - PR #8
      • builds against Python v3.10 - PR #12
      • isort check (for ill-ordered imports) - commit 76f7729

    p.s. A big thank you to @sndoja for contributing on issue #11 ๐ŸŽ…


    image caption: the landing page of https://pydatagovgr.readthedocs.io - documentation is being updated live, with each commit to main

    Source code(tar.gz)
    Source code(zip)
  • v0.1.9(May 19, 2021)

    Documentation:

    • add new section Related Projects in README, where similar projects will be referenced (e.g. a Golang-based SDK for data.gov.gr)
    • improvements in README and CONTRIBUTING documents
    • fixed a typo in a README code block usage example
    Source code(tar.gz)
    Source code(zip)
  • v0.1.8(Apr 2, 2021)

    Miscellaneous:

    • increased default timeout to 60 sec
    • added detailed description of timeout parameter - note that this policy is something that should also be respected server-sider as well to have an effect i.e. this client informs the data.gov.gr about the maximum time it will wait, however data.gov.gr is the one who decides if this is to be the case

    Documentation:

    • simplified some README sections
    • improved README code snippets
    • tweaked the README badges a bit, to be more informative visually
    Source code(tar.gz)
    Source code(zip)
  • v0.1.7(Jan 31, 2021)

    CI/CD:

    • package releases to PyPI now install only the absolutely required dependencies and not the dev dependencies as well - this creates a minimal wheel and/or sdist to be used by other applications, simplifying their dependency graph etc.
    • builds against the main branch now also attempt to build the package, for completeness
    Source code(tar.gz)
    Source code(zip)
  • v0.1.6(Jan 30, 2021)

    Miscellaneous:

    • black is now available as dev-dependency
      • main branch builds and package release builds now fail if code is found not to be black-formatted
      • line length is enforce to be 88 characters
    • google-style docstrings are used throughout the codebase (but not strictly enforced)
    • package __version__ is now publicly available to the end-user - this can be included in bug reports etc. and enhance reproducibility down the road. Example usage:
    import pydatagovgr
    
    
    pydatagovgr.__version__
    '0.1.5'
    
    Source code(tar.gz)
    Source code(zip)
  • v0.1.5(Jan 20, 2021)

    This could easily be a v0.2.0 release but i categorized it as a patch release, even though no source code patches are introduced.

    Miscellaneous:

    • significantly enriched the contribution guile
    • added a Github action to automatically publish the package to PyPi when creating a new release via Github UI.
    • extra minor styling issue corrected on README
    Source code(tar.gz)
    Source code(zip)
  • v0.1.4(Jan 11, 2021)

    Bug Fixes:

    • urllib3 sub-dependency: got an unexpected keyword argument 'allowed_methods' (issue #1)

    Miscellaneous:

    • failed responses from data.gov.gr with 413 status are now included in the retry policy
    • fixed some code style issues originating from black reformatting. Will try to automate this soon.
    • some metadata configuration on pyproject.toml file
    Source code(tar.gz)
    Source code(zip)
  • v0.1.3(Jan 10, 2021)

  • v0.1.2(Jan 9, 2021)

  • v0.1.1(Jan 9, 2021)

  • v0.1.0(Jan 9, 2021)

    This inaugural release introduces the DataGovClient object, through which a user can access the open-data of data.gov.gr

    In addition, a respective test suite is introduced as well as documentation and packaging considerations.

    Source code(tar.gz)
    Source code(zip)
Owner
Ilias Antonopoulos
Machine Learning Engineer.
Ilias Antonopoulos
An API serving data on all creatures, monsters, materials, equipment, and treasure in The Legend of Zelda: Breath of the Wild

Hyrule Compendium API An API serving data on all creatures, monsters, materials, equipment, and treasure in The Legend of Zelda: Breath of the Wild. B

Aarav Borthakur 116 Dec 01, 2022
Simple VK API wrapper for Python

VK Admier: documentation VK Admier is simple VK API wrapper for community bot development. Authorization You should create bot object from Client clas

Egor Light 2 Nov 10, 2022
A Telegram Bot That Can Find Lyrics Of Song

Lyrics-Search-Bot A Telegram Bot That Can Find Lyrics Of Song A Simple Telegram Bot That Can Extract Lyrics Of Any Songs Deploy Commands start - To St

Muhammed Fazin 11 Oct 21, 2022
Project for QVault Hackathon which plays sounds based on the letters of a user's name

virtual_instrument Project for QVault Hackathon which plays sounds based on the letters of a user's name I created a virtual instrument using Python a

Paolo Sidera 2 Feb 11, 2022
Bavera is an extensive and extendable Python 3.x library for the Discord API

Bavera is an extensive and extendable Python 3.x library for the Discord API. Bavera boasts the following major features: Expressive, functiona

Bavera 1 Nov 17, 2021
A simple Python library to integrate with the Heron Data API

Heron Python This library provides easy access to the Heron Data API from applications written in Python. Documentation No language-specific docs are

Heron Data 11 Nov 11, 2022
็Ÿญไฟกๅ‘้€ Python ็จ‹ๅบ(ๅŒ…ๅซ1000+ๆœ‰ๆ•ˆๆŽฅๅฃ)

็Ÿญไฟก่ฝฐ็‚ธ Python ็จ‹ๅบ(ๅŒ…ๅซ1000+ๆœ‰ๆ•ˆๆŽฅๅฃ) ๅ‰่จ€ ่ฟ™ๆ˜ฏไธ€ไธช็ˆฌๅ–็ฝ‘็ปœไธŠๅœจ็บฟ่ฝฐ็‚ธ็š„ๆŽฅๅฃ๏ผŒๅŽ้€š่ฟ‡ Python ๅผ‚ๆญฅ ่ฏทๆฑ‚ๆŽฅๅฃไปฅ่พพๅˆฐ ๆ‰‹ๆœบ็Ÿญไฟก่ฝฐ็‚ธ ็š„็›ฎ็š„ใ€‚ ๆญคไธบๅผ€ๆบ้กน็›ฎ๏ผŒไป…ไพ›ๅจฑไนๅญฆไน ไฝฟ็”จ๏ผŒไฝฟ็”จ่€…ๆ‰€ๅธฆๆฅ็š„ไธ€ๅˆ‡ๅŽๆžœไธŽไฝœ่€…ๆ— ๅ…ณ๏ผŒไฝฟ็”จ่ฏท้ตๅฎˆ็›ธๅ…ณ็š„ๆณ•ๅพ‹ๆณ•่ง„๏ผŒๅˆ็†ไฝฟ็”จ๏ผŒ่ฏทๅ‹ฟๆปฅ็”จใ€‚ ้ฃŸ็”จๆ–นๆณ• 1. ็ˆฌๅ–ๆŽฅ

่“้ฒธ่ฝ 10.2k Jan 02, 2023
Bulk NFT uploader to OpenSea!

Bulk NFT Uploader Description Simple easy peasy python script which logins to opensea account using metamask and bulk uploads NFT to your default coll

Lakshya Khera 25 May 23, 2022
Vhook: A Discord webhook spammer / deleter open source coded by vesper

Vhook_Spammer Vhook is a advanced Discord webhook spammer / deleter with embeds,

Vesper 17 Nov 13, 2022
Python API wrapper around Trello's API

A wrapper around the Trello API written in Python. Each Trello object is represented by a corresponding Python object. The attributes of these objects

Richard Kolkovich 904 Jan 02, 2023
Very Sempil Bot Auto Filter bot

SAMANTHA_BOT Very Sempil Bot Auto Filter bot ##[ # ๐‚๐‹๐ˆ๐‚๐Š ๐๐„๐‹๐Ž๐– ๐ˆ๐Œ๐€๐†๐„ ๐“๐Ž ๐ƒ๐„๐๐‹๐Ž๐˜ ๐Ÿ‘‡ ๐Ÿ‘‡ ๐Ÿ‘‡ Auto Filter Manuel Filter IMDB Admin Co

DARK WEBLOAD 3 Jun 27, 2022
Discord-disnake - This package allows to use disnake without changing the discord namespace

This package is a shim This module allows to use disnake using discord namespace. This is not an independent library. Installing Python 3.8 or higher

5 Dec 13, 2022
A mood based crypto tracking application.

Crypto Bud - API A mood based crypto tracking application. The main repository is private. I am creating the API before I connect everything to the ma

Krishnasis Mandal 1 Oct 23, 2021
Python wrapper for Wikipedia

Wikipedia API Wikipedia-API is easy to use Python wrapper for Wikipedias' API. It supports extracting texts, sections, links, categories, translations

Martin Majlis 369 Dec 30, 2022
The python SDK for Eto, the AI focused data platform for teams bringing AI models to production

Eto Labs Python SDK This is the python SDK for Eto, the AI focused data platform for teams bringing AI models to production. The python SDK makes it e

5 Apr 21, 2022
Creating a Python API, for the MakeMyTrip Flight Schedules.

MakeMyTripAPI Creating a Python API, for the MakeMyTrip Flight Schedules. Source: MakeMyTrip is an Indian online travel company founded in 2000. Headq

Aman Priyanshu 0 Jan 12, 2022
Home Assistant Hilo Integration via HACS

BETA This is a beta release. There will be some bugs, issues, etc. Please bear with us and open issues in the repo. Hilo Hilo integration for Home Ass

66 Dec 23, 2022
A curated list of awesome Amazon Web Services (AWS) libraries, open source repos, guides, blogs, and other resources.

A curated list of awesome Amazon Web Services (AWS) libraries, open source repos, guides, blogs, and other resources. Featuring the Fiery Meter of AWSome.

Donne Martin 11.1k Jan 04, 2023
PyManGenerator is a token generator for discord, it joins servers using webbot to automate everything

PyManGenerator is a token generator for discord, it joins servers using webbot to automate everything. Captcha can be done by itself unless you used your current IP Address more than once.

5 Nov 27, 2021
Public repo of the bot

wiki-reddit-bot Public repo of u/wikipedia_answer_bot Tools Language: Python Libraries: praw (Reddit API) mediawikiapi (Wikipedia API) tenacity How it

TheBugYouCantFix 51 Dec 03, 2022