API to parse tibia.com content into python objects.

Related tags

Web Crawlingpythonpython3beautifulsouptibiawebcrawlingcrawling-python
Overview

Tibia.py

An API to parse Tibia.com content into object oriented data.

No fetching is done by this module, you must provide the html content.

GitHub Workflow Status codecov PyPI PyPI - Python Version PyPI - License

Features:

  • Converts data into well-structured Python objects.
  • Type consistent attributes.
  • All objects can be converted to JSON strings.
  • Can be used with any networking library.
  • Support for characters, guilds, houses and worlds, tournaments, forums, etc.

Installing

Install and update using pip

pip install tibia.py

Installing the latest version form GitHub

pip install git+https://github.com/Galarzaa90/tibia.py.git -U

Usage

This library is composed of two parts, parsers and an asynchronous request client.

The asynchronous client (tibiapy.Client) contains methods to obtain information from Tibia.com.

The parsing methods allow you to get Python objects given the html content of a page.

import tibiapy

# Asynchronously
import aiohttp

async def get_character(name):
    url = tibiapy.Character.get_url(name)

    async with aiohttp.ClientSession() as session:
        async with session.get(url) as resp:
            content = await resp.text()
    character = tibiapy.Character.from_content(content)
    return character

# Synchronously
import requests

def get_character_sync(name):
    url = tibiapy.Character.get_url(name)
    
    r = requests.get(url)
    content = r.text
    character = tibiapy.Character.from_content(content)
    return character

Documentation

https://tibiapy.readthedocs.io/

Comments
  • DisplayItems attribute name is returning None

    DisplayItems attribute name is returning None

    I was querying the items of a character at the bazaar and the DisplayItem attribute name is returning None

    Here is my code snippet:

    response : TibiaResponse = self.client.loop.run_until_complete(self.client.fetch_auction(character_auction_id, 
        fetch_items=True))
        
details : AuctionDetails = response.data

items : List[DisplayItem] = details.items.entries

print(items)

    The print of the returned list:

    [<DisplayItem name=None count=22 item_id=238>, <DisplayItem name=None count=347 item_id=239>, <DisplayItem name=None count=11 item_id=266>]

    Yesterday everything was working fine, the problem emerged this morning. Maybe the Tibia Bazaar website changed something in their HTML indexing?

    opened by guriosam 6
  • Problem with the comma when there are more than 1000 players online.

    Problem with the comma when there are more than 1000 players online.

    Hello! Cipsoft added a comma in the world list when there are more than 1000 players online.

    File "\site-packages\tibiapy\world.py", line 267, in from_content world._parse_world_info(tables.get("World Information", [])) File "\site-packages\tibiapy\world.py", line 344, in _parse_world_info self.online_count = int(world_info.pop("players_online")) ValueError: invalid literal for int() with base 10: '1,024'

    opened by Pixelao 2
  • tibiapy.HousesSection.from_content is broken

    tibiapy.HousesSection.from_content is broken

    Problem Method tibiapy.HousesSection.from_content(link to code) throws tibiapy.errors.InvalidContent error instead of returning a HousesSection object.

    The reason this unintended behavior happens is because we try to access a Nonetype object inside HouseSection._parse_filters method when we call it house_results._parse_filters(tables["House Search"]) in line 172.

    Quick fix Commenting line 172 solves the issue for me (at least it does for my use case).

    opened by aaronga97 1
  • v2.4.0

    v2.4.0

    Changes

    • Added support for multiple houses per character. Accessible on Character.houses field.
    • Character.house is now deprecated. It will contain the character's first house or None.
    opened by Galarzaa90 1
  • v1.1.0

    v1.1.0

    Changes

    • Parsing Highscores from Tibia.com and TibiaData.
    • Some strings from TibiaData had unpredictable trailing whitespaces, all leading and trailing whitespaces are removed.
    • Added type hints to many variables and methods.
    opened by Galarzaa90 1
  • Parse character position

    Parse character position

    Character's position is not being parsed, for example, Steve has CipSoft Member position. This seems to be the only available position.

    Not to be confused with the position field in Account Information.

    Also, these characters show Cipsoft Member in their status when listed as "Other Characters".

    enhancement 
    opened by Galarzaa90 1
  • Add HouseStatus and Sex enums

    Add HouseStatus and Sex enums

    Testing the waters to maybe improve code. Those do not seem to be used in many places, so they are both good testing subjects.

    Tests kept successful so it seems to be ok. This might have some limitations with json parsing constructors, so let's discuss what you think later :)

    opened by Tschis 1
  • Refactor guild.py and unit tests

    Refactor guild.py and unit tests

    • Refactor guild.py to split different sections of the parsing method. This makes it easier to test and analyze/maintain the code in the future;
    • Add new resource and unit tests for best case scenario on each section;
    opened by Tschis 1
  • Parse character titles and main character information

    Parse character titles and main character information

    Summer update 2019 introduced character titles, as well as a small icon that indicates which is the main character of the account in the other characters section.

    opened by Galarzaa90 0
  • v1.0.0

    v1.0.0

    Changes

    • Added support for TibiaData JSON parsing. To have interoperability between Tibia.com and TibiaData.
    • Added support for parsing Houses, House lists, World and World list
    • Added support for many missing attributes in Character and Guilds.
    • All objects are now serializable to JSON strings.
    opened by Galarzaa90 0
  • Add /tests as a package

    Add /tests as a package

    As seen here, this might help running the tests from the root folder.

    Also added some changes that might facilitate changes into the current test case or new test cases to be added in the future (relative resources path).

    opened by Tschis 0
Releases(v5.5.2)

  • v5.5.2(Sep 3, 2022)

    Changes

    • Fixed Houses section not parsing due to a change in the filters table.
    • Fixed status parameter not generating the correct URL in the houses section.
    Source code(tar.gz)
    Source code(zip)

  • v5.5.1(Aug 2, 2022)

    Changes

    • Adjusted parsing to support the changes related to mobile devices introduced on the day of this release. The following sections were affected:
      • Highscores
      • News
      • Forums
      • Spells
    Source code(tar.gz)
    Source code(zip)

  • v5.5.0(Jul 27, 2022)

  • v5.4.0(Jul 23, 2022)

  • v5.3.0(Jul 23, 2022)

  • v5.2.1(Mar 1, 2022)

  • v5.2.0(Dec 31, 2021)

  • v5.1.0(Sep 16, 2021)

    Changes

    • Added traded attribute to death killers, to indicate that the killer was traded after the death occurred.
    • Properly handle deaths caused by summons of traded characters.
    Source code(tar.gz)
    Source code(zip)

  • v5.0.1.post0(Aug 26, 2021)

  • v5.0.1(Aug 26, 2021)

    Changes

    • Fixed many sections not being parsed correctly due to changes to Tibia.com.
      • Houses list
      • News list
      • Spells section
      • Forums section
    • Fixed character's houses failing to parse due to a bug in the display in Tibia.com.
      • Temporarily disabling this attribute.
    Source code(tar.gz)
    Source code(zip)

  • v5.0.1a1(Aug 24, 2021)

    Prelease to address part of the Tibia.com changes.

    Changes

    • Fixed character's houses failing to parse due to a bug in the display in Tibia.com.
      • Temporarily disabling this attribute.
    • Fixed house list not parsing correctly due to Tibia.com changes
    • Fixed news list not parsing correctly due to Tibia.com changes

    Pending

    • Forums section
    • Spells section
    Source code(tar.gz)
    Source code(zip)

  • v5.0.0(Aug 7, 2021)

    Changes

    • Added parsing for Tibia Drome leaderboards, new Leaderboard class.
      • Auxiliary classes LeaderboardEntry and LeaderboardRotation were added as well.
      • New Client method: fetch_leaderboards.
    • Added parsing for Spells library, new SpellsSection class.
      • Auxiliary classes Spell and SpellEntry.
      • New Client methods: fetch_spell and fetch_spells
    • Fix last page of highscores having 0 as page value.
    • Using the Client class, you can now fetch and parse content from the test version of www.tibia.com when available.
      • Note that if the test website has changes, parsing might not be possible.
      • Internal URL attributes might still point to the regular website.
    • New HousesSection class, including the house filtering attributes.
    • New NewsArchive class, including the news filtering attributes.
    • New GuildsSection class, to replace the lists of GuildEntry
    • Many "ListedObject" classes were renamed to "ObjectEntry", for details check the breaking changes below.
    • Fixed bug with other characters not being parsed.
    • Added traded attribute to LastPost class.
    • Added thread_starter_traded attribute to ThreadEntry

    Breaking Changes:

    • Python 3.7 or higher is now required.
    • House classes no longer have get_list_url and list_from_content methods.
    • Client.fetch_world_houses now returns a HousesSection instance in its data attribute, instead of a list of ListedHouses.
    • ListedHouse.highest_bid attribute now may be None if the house's auction has not yet started.
    • ListedHouse class renamed to HouseEntry.
    • Removed deprecated property AuctionFilters.item.
    • Client.fetch_news_archive, Client.fetch_recent_news now returns an instance of NewsArchive in the data attribute.
    • ListedNews class renamed to NewsEntry.
    • News and NewsEntry no longer have a get_list_url method.
    • ListedBoard class renamed to BoardEntry.
    • ListedThread class renamed to ThreadEntry.
    • ListedAnnouncement class renamed to AnnouncementEntry.
    • ListedWorld class renamed to WorldEntry.
    • ListedAuction class renamed to AuctionEntry.
    • AuctionDetails class renamed to Auction.
    • ListedGuild class renamed to GuildEntry.
    • ListedTournament class renamed to TournamentEntry.
    • Creature class renamed to CreatureEntry.
    • CreatureDetail class renamed to Creature.
    • Guild and GuildEntry class no longer have a get_list_url method.
    • Renamed begin_date parameter to start_date in fetch_news_archive.
    • Renamed race attribute of CreatureEntry and Creature to identifier, method parameters renamed as well.
    • CreaturesSection.from_boosted_creature_header renamed to CreaturesSection.boosted_creature_from_header.
    Source code(tar.gz)
    Source code(zip)

  • v4.1.7(Jun 30, 2021)

  • v4.1.6(Jun 28, 2021)

    Changes

    • Fixed worlds not being parsed correctly again due to tournament worlds order changing. After this fix, the order should not matter anymore.
    Source code(tar.gz)
    Source code(zip)

  • v4.1.5(Jun 26, 2021)

  • v4.1.4(Jun 17, 2021)

  • v4.1.3(May 12, 2021)

  • v4.1.2(Apr 27, 2021)

  • v4.1.1(Apr 19, 2021)

  • v4.1.0(Mar 30, 2021)

    Changes

    • Added prey_wildcards attribute to AuctionDetails.
    • Added filters parameter to CharacterBazaar.get_auctions_history_url and Client.fetch_auction_history.
    Source code(tar.gz)
    Source code(zip)

  • v4.0.0(Mar 11, 2021)

    Changes

    • Breaking change: Removed BoostedCreature class, replaced by Creature class.
      • Attributes should be compatible, image_url is a property of Creature, calculated from its race attribute.
    • Added parsing and fetching for the Creature library section.
      • Added CreatureSection, Creature, and CreatureDetail classes.
    • Added traded attribute to ForumAuthor. Indicates if the author was a traded character.
      • Previously, it would mark the author as a deleted character and its name would include (traded).
    • Fixed a bug with ForumBoards not parsing due to the cookies dialog that was added.
    • Added battleye_type attribute to ListedWorld and World classes. Indicates the type of BattlEye protection the world has.
      • battleye_protected is now a property instead of an attribute.
    • Added YELLOW and GREEN aliases to all BattlEye related enums.
    • Fixed wrong timezone being used for forum related dates.
    Source code(tar.gz)
    Source code(zip)

  • v3.7.1(Feb 16, 2021)

  • v.3.7.0(Feb 9, 2021)

    Changes

    • Parse familiars from auctions
    • Updated the way tooltips in auctions are parsed, the format changed, resulting in the previous code not working anymore.
    • Results count in bazaar pages are now properly parsed when there are comma thousand separators.
    • Item amounts are now more accurate instead of being based from their indicator (which was grouping them in thousands)
    Source code(tar.gz)
    Source code(zip)

  • v3.6.5(Jan 28, 2021)

  • v3.6.4(Jan 26, 2021)

  • v3.6.3(Jan 15, 2021)

  • v3.6.2(Jan 2, 2021)

  • v3.6.1(Dec 28, 2020)

  • v3.6.0(Dec 12, 2020)

    Changes

    • Added support for the new filtering options in Highscores
      • Added battleye_filter and pvp_types_filter attributes.
    • Added get_page_url() instance method to Highscores class.
    • Added previous_page_url and next_page_url properties.
    Source code(tar.gz)
    Source code(zip)

  • v3.5.7(Dec 4, 2020)

Owner
Allan Galarza
Mechatronic Engineer
Allan Galarza
GitHub Repository https://tibiapy.readthedocs.io/
🥫 The simple, fast, and modern web scraping library

About gazpacho is a simple, fast, and modern web scraping library. The library is stable, actively maintained, and installed with zero dependencies. I

Max Humber 692 Dec 22, 2022
Scrape puzzle scrambles from csTimer.net

Scroodle Selenium script to scrape scrambles from csTimer.net csTimer runs locally in your browser, so this doesn't strain the servers any more than i

Jason Nguyen 1 Oct 29, 2021
Bulk download tool for the MyMedia platform

MyMedia Bulk Content Downloader This is a bulk download tool for the MyMedia platform. USE ONLY WHERE ALLOWED BY THE COPYRIGHT OWNER. NOT AFFILIATED W

Ege Feyzioglu 3 Oct 14, 2022
Web scraper build using python.

Web Scraper This project is made in pyhthon. It took some info. from website list then add them into data.json file. The dependencies used are: reques

Shashwat Harsh 2 Jul 22, 2022
Visual scraping for Scrapy

Portia Portia is a tool that allows you to visually scrape websites without any programming knowledge required. With Portia you can annotate a web pag

Scrapinghub 8.7k Jan 05, 2023
🤖 Threaded Scraper to get discord servers from disboard.org written in python3

Disboard-Scraper Threaded Scraper to get discord servers from disboard.org written in python3. Setup. One thread / tag If you whant to look for multip

Ѵιcнч 11 Nov 01, 2022
Lovely Scrapper

Lovely Scrapper

Tushar Gadhe 2 Jan 01, 2022
This is a web crawler that works on employ email data by gmane.org and visualizes it in different ways.

crawler_to_visual_gmane Analyzing an EMAIL Archive from gmane and vizualizing the data using the D3 JavaScript library. This is a set of tools that al

Saim Zafar 1 Dec 20, 2021
Extract gene TSS site form gencode/ensembl/gencode database GTF file and export bed format file.

GetTss python Package extract gene TSS site form gencode/ensembl/gencode database GTF file and export bed format file. Install $ pip install GetTss Us

laojunjun 6 Nov 21, 2022
对于有验证码的站点爆破，用于安全合法测试

使用方法 python3 main.py + 配置好的文件 python3 main.py Verify.json python3 main.py NoVerify.json 以上分别对应有验证码的demo和无验证码的demo Tips: 你可以以域名作为配置文件名字加载：python3 main

47 Nov 09, 2022
SmartScraper: 简单、自动、快捷的Python网络爬虫

SmartScraper: 简单、自动、快捷的Python网络爬虫 Note: The origin developer of SmartScraper is Alireza Mika， I only change a little code of AutoScraper. SmartScraper

DaDeng 9 Apr 16, 2022
A simple, configurable and expandable combined shop scraper to minimize the costs of ordering several items

combined-shop-scraper A simple, configurable and expandable combined shop scraper to minimize the costs of ordering several items. Features Define an

2 Dec 13, 2021
An arxiv spider

An Arxiv Spider 做为一个cser，杰出男孩深知内核对连接到计算机上的硬件设备进行管理的高效方式是中断而不是轮询。每当小伙伴发来一篇刚挂在arxiv上的”热乎“好文章时，杰出男孩都会感叹道：”师兄这是每天都挂在arxiv上呀，跑的好快~“。于是杰出男孩找了找 github，借鉴了一下其

Jie Liu 11 Sep 09, 2022
Proxy scraper. Format: IP | PORT | COUNTRY | TYPE

proxy scraper 🔎 Installation: git clone https://github.com/ebankoff/proxy_scraper Required pip libraries (pip install library name): lxml beautifulso

Eban'ko 19 Dec 07, 2022
Google Developer Profile Badge Scraper

Google Developer Profile Badge Scraper GDev Profile Badge Scraper is a Google Developer Profile Web Scraper which scrapes for specific badges in a use

Siddhant Lad 7 Jan 10, 2022
This is a web scraper, using Python framework Scrapy, built to extract data from the Deals of the Day section on Mercado Livre website.

Deals of the Day This is a web scraper, using the Python framework Scrapy, built to extract data such as price and product name from the Deals of the

David Souza 1 Jan 12, 2022
Simple tool to scrape and download cross country ski timings and results from live.skidor.com

LiveSkidorDownload Simple tool to scrape and download cross country ski timings

0 Jan 07, 2022
Amazon scraper using scrapy, a python framework for crawling websites.

#Amazon-web-scraper This is a python program, which use scrapy python framework to crawl all pages of the product and scrap products data. This progra

Akash Das 1 Dec 26, 2021
Kusonime scraper using python3

Features Scrap from url Scrap from recommendation Search by query Todo [+] Search by genre Example # Get download url from kusonime import Scrap

MhankBarBar 2 Jan 28, 2022
Universal Reddit Scraper - A comprehensive Reddit scraping command-line tool written in Python.

Universal Reddit Scraper - A comprehensive Reddit scraping command-line tool written in Python.

Joseph Lai 543 Jan 03, 2023