API to parse tibia.com content into python objects.

Related tags

Web Crawlingpythonpython3beautifulsouptibiawebcrawlingcrawling-python
Overview

Tibia.py

An API to parse Tibia.com content into object oriented data.

No fetching is done by this module, you must provide the html content.

GitHub Workflow Status codecov PyPI PyPI - Python Version PyPI - License

Features:

  • Converts data into well-structured Python objects.
  • Type consistent attributes.
  • All objects can be converted to JSON strings.
  • Can be used with any networking library.
  • Support for characters, guilds, houses and worlds, tournaments, forums, etc.

Installing

Install and update using pip

pip install tibia.py

Installing the latest version form GitHub

pip install git+https://github.com/Galarzaa90/tibia.py.git -U

Usage

This library is composed of two parts, parsers and an asynchronous request client.

The asynchronous client (tibiapy.Client) contains methods to obtain information from Tibia.com.

The parsing methods allow you to get Python objects given the html content of a page.

import tibiapy

# Asynchronously
import aiohttp

async def get_character(name):
    url = tibiapy.Character.get_url(name)

    async with aiohttp.ClientSession() as session:
        async with session.get(url) as resp:
            content = await resp.text()
    character = tibiapy.Character.from_content(content)
    return character

# Synchronously
import requests

def get_character_sync(name):
    url = tibiapy.Character.get_url(name)
    
    r = requests.get(url)
    content = r.text
    character = tibiapy.Character.from_content(content)
    return character

Documentation

https://tibiapy.readthedocs.io/

Comments
  • DisplayItems attribute name is returning None

    DisplayItems attribute name is returning None

    I was querying the items of a character at the bazaar and the DisplayItem attribute name is returning None

    Here is my code snippet:

    response : TibiaResponse = self.client.loop.run_until_complete(self.client.fetch_auction(character_auction_id, 
        fetch_items=True))
        
details : AuctionDetails = response.data

items : List[DisplayItem] = details.items.entries

print(items)

    The print of the returned list:

    [<DisplayItem name=None count=22 item_id=238>, <DisplayItem name=None count=347 item_id=239>, <DisplayItem name=None count=11 item_id=266>]

    Yesterday everything was working fine, the problem emerged this morning. Maybe the Tibia Bazaar website changed something in their HTML indexing?

    opened by guriosam 6
  • Problem with the comma when there are more than 1000 players online.

    Problem with the comma when there are more than 1000 players online.

    Hello! Cipsoft added a comma in the world list when there are more than 1000 players online.

    File "\site-packages\tibiapy\world.py", line 267, in from_content world._parse_world_info(tables.get("World Information", [])) File "\site-packages\tibiapy\world.py", line 344, in _parse_world_info self.online_count = int(world_info.pop("players_online")) ValueError: invalid literal for int() with base 10: '1,024'

    opened by Pixelao 2
  • tibiapy.HousesSection.from_content is broken

    tibiapy.HousesSection.from_content is broken

    Problem Method tibiapy.HousesSection.from_content(link to code) throws tibiapy.errors.InvalidContent error instead of returning a HousesSection object.

    The reason this unintended behavior happens is because we try to access a Nonetype object inside HouseSection._parse_filters method when we call it house_results._parse_filters(tables["House Search"]) in line 172.

    Quick fix Commenting line 172 solves the issue for me (at least it does for my use case).

    opened by aaronga97 1
  • v2.4.0

    v2.4.0

    Changes

    • Added support for multiple houses per character. Accessible on Character.houses field.
    • Character.house is now deprecated. It will contain the character's first house or None.
    opened by Galarzaa90 1
  • v1.1.0

    v1.1.0

    Changes

    • Parsing Highscores from Tibia.com and TibiaData.
    • Some strings from TibiaData had unpredictable trailing whitespaces, all leading and trailing whitespaces are removed.
    • Added type hints to many variables and methods.
    opened by Galarzaa90 1
  • Parse character position

    Parse character position

    Character's position is not being parsed, for example, Steve has CipSoft Member position. This seems to be the only available position.

    Not to be confused with the position field in Account Information.

    Also, these characters show Cipsoft Member in their status when listed as "Other Characters".

    enhancement 
    opened by Galarzaa90 1
  • Add HouseStatus and Sex enums

    Add HouseStatus and Sex enums

    Testing the waters to maybe improve code. Those do not seem to be used in many places, so they are both good testing subjects.

    Tests kept successful so it seems to be ok. This might have some limitations with json parsing constructors, so let's discuss what you think later :)

    opened by Tschis 1
  • Refactor guild.py and unit tests

    Refactor guild.py and unit tests

    • Refactor guild.py to split different sections of the parsing method. This makes it easier to test and analyze/maintain the code in the future;
    • Add new resource and unit tests for best case scenario on each section;
    opened by Tschis 1
  • Parse character titles and main character information

    Parse character titles and main character information

    Summer update 2019 introduced character titles, as well as a small icon that indicates which is the main character of the account in the other characters section.

    opened by Galarzaa90 0
  • v1.0.0

    v1.0.0

    Changes

    • Added support for TibiaData JSON parsing. To have interoperability between Tibia.com and TibiaData.
    • Added support for parsing Houses, House lists, World and World list
    • Added support for many missing attributes in Character and Guilds.
    • All objects are now serializable to JSON strings.
    opened by Galarzaa90 0
  • Add /tests as a package

    Add /tests as a package

    As seen here, this might help running the tests from the root folder.

    Also added some changes that might facilitate changes into the current test case or new test cases to be added in the future (relative resources path).

    opened by Tschis 0
Releases(v5.5.2)

  • v5.5.2(Sep 3, 2022)

    Changes

    • Fixed Houses section not parsing due to a change in the filters table.
    • Fixed status parameter not generating the correct URL in the houses section.
    Source code(tar.gz)
    Source code(zip)

  • v5.5.1(Aug 2, 2022)

    Changes

    • Adjusted parsing to support the changes related to mobile devices introduced on the day of this release. The following sections were affected:
      • Highscores
      • News
      • Forums
      • Spells
    Source code(tar.gz)
    Source code(zip)

  • v5.5.0(Jul 27, 2022)

  • v5.4.0(Jul 23, 2022)

  • v5.3.0(Jul 23, 2022)

  • v5.2.1(Mar 1, 2022)

  • v5.2.0(Dec 31, 2021)

  • v5.1.0(Sep 16, 2021)

    Changes

    • Added traded attribute to death killers, to indicate that the killer was traded after the death occurred.
    • Properly handle deaths caused by summons of traded characters.
    Source code(tar.gz)
    Source code(zip)

  • v5.0.1.post0(Aug 26, 2021)

  • v5.0.1(Aug 26, 2021)

    Changes

    • Fixed many sections not being parsed correctly due to changes to Tibia.com.
      • Houses list
      • News list
      • Spells section
      • Forums section
    • Fixed character's houses failing to parse due to a bug in the display in Tibia.com.
      • Temporarily disabling this attribute.
    Source code(tar.gz)
    Source code(zip)

  • v5.0.1a1(Aug 24, 2021)

    Prelease to address part of the Tibia.com changes.

    Changes

    • Fixed character's houses failing to parse due to a bug in the display in Tibia.com.
      • Temporarily disabling this attribute.
    • Fixed house list not parsing correctly due to Tibia.com changes
    • Fixed news list not parsing correctly due to Tibia.com changes

    Pending

    • Forums section
    • Spells section
    Source code(tar.gz)
    Source code(zip)

  • v5.0.0(Aug 7, 2021)

    Changes

    • Added parsing for Tibia Drome leaderboards, new Leaderboard class.
      • Auxiliary classes LeaderboardEntry and LeaderboardRotation were added as well.
      • New Client method: fetch_leaderboards.
    • Added parsing for Spells library, new SpellsSection class.
      • Auxiliary classes Spell and SpellEntry.
      • New Client methods: fetch_spell and fetch_spells
    • Fix last page of highscores having 0 as page value.
    • Using the Client class, you can now fetch and parse content from the test version of www.tibia.com when available.
      • Note that if the test website has changes, parsing might not be possible.
      • Internal URL attributes might still point to the regular website.
    • New HousesSection class, including the house filtering attributes.
    • New NewsArchive class, including the news filtering attributes.
    • New GuildsSection class, to replace the lists of GuildEntry
    • Many "ListedObject" classes were renamed to "ObjectEntry", for details check the breaking changes below.
    • Fixed bug with other characters not being parsed.
    • Added traded attribute to LastPost class.
    • Added thread_starter_traded attribute to ThreadEntry

    Breaking Changes:

    • Python 3.7 or higher is now required.
    • House classes no longer have get_list_url and list_from_content methods.
    • Client.fetch_world_houses now returns a HousesSection instance in its data attribute, instead of a list of ListedHouses.
    • ListedHouse.highest_bid attribute now may be None if the house's auction has not yet started.
    • ListedHouse class renamed to HouseEntry.
    • Removed deprecated property AuctionFilters.item.
    • Client.fetch_news_archive, Client.fetch_recent_news now returns an instance of NewsArchive in the data attribute.
    • ListedNews class renamed to NewsEntry.
    • News and NewsEntry no longer have a get_list_url method.
    • ListedBoard class renamed to BoardEntry.
    • ListedThread class renamed to ThreadEntry.
    • ListedAnnouncement class renamed to AnnouncementEntry.
    • ListedWorld class renamed to WorldEntry.
    • ListedAuction class renamed to AuctionEntry.
    • AuctionDetails class renamed to Auction.
    • ListedGuild class renamed to GuildEntry.
    • ListedTournament class renamed to TournamentEntry.
    • Creature class renamed to CreatureEntry.
    • CreatureDetail class renamed to Creature.
    • Guild and GuildEntry class no longer have a get_list_url method.
    • Renamed begin_date parameter to start_date in fetch_news_archive.
    • Renamed race attribute of CreatureEntry and Creature to identifier, method parameters renamed as well.
    • CreaturesSection.from_boosted_creature_header renamed to CreaturesSection.boosted_creature_from_header.
    Source code(tar.gz)
    Source code(zip)

  • v4.1.7(Jun 30, 2021)

  • v4.1.6(Jun 28, 2021)

    Changes

    • Fixed worlds not being parsed correctly again due to tournament worlds order changing. After this fix, the order should not matter anymore.
    Source code(tar.gz)
    Source code(zip)

  • v4.1.5(Jun 26, 2021)

  • v4.1.4(Jun 17, 2021)

  • v4.1.3(May 12, 2021)

  • v4.1.2(Apr 27, 2021)

  • v4.1.1(Apr 19, 2021)

  • v4.1.0(Mar 30, 2021)

    Changes

    • Added prey_wildcards attribute to AuctionDetails.
    • Added filters parameter to CharacterBazaar.get_auctions_history_url and Client.fetch_auction_history.
    Source code(tar.gz)
    Source code(zip)

  • v4.0.0(Mar 11, 2021)

    Changes

    • Breaking change: Removed BoostedCreature class, replaced by Creature class.
      • Attributes should be compatible, image_url is a property of Creature, calculated from its race attribute.
    • Added parsing and fetching for the Creature library section.
      • Added CreatureSection, Creature, and CreatureDetail classes.
    • Added traded attribute to ForumAuthor. Indicates if the author was a traded character.
      • Previously, it would mark the author as a deleted character and its name would include (traded).
    • Fixed a bug with ForumBoards not parsing due to the cookies dialog that was added.
    • Added battleye_type attribute to ListedWorld and World classes. Indicates the type of BattlEye protection the world has.
      • battleye_protected is now a property instead of an attribute.
    • Added YELLOW and GREEN aliases to all BattlEye related enums.
    • Fixed wrong timezone being used for forum related dates.
    Source code(tar.gz)
    Source code(zip)

  • v3.7.1(Feb 16, 2021)

  • v.3.7.0(Feb 9, 2021)

    Changes

    • Parse familiars from auctions
    • Updated the way tooltips in auctions are parsed, the format changed, resulting in the previous code not working anymore.
    • Results count in bazaar pages are now properly parsed when there are comma thousand separators.
    • Item amounts are now more accurate instead of being based from their indicator (which was grouping them in thousands)
    Source code(tar.gz)
    Source code(zip)

  • v3.6.5(Jan 28, 2021)

  • v3.6.4(Jan 26, 2021)

  • v3.6.3(Jan 15, 2021)

  • v3.6.2(Jan 2, 2021)

  • v3.6.1(Dec 28, 2020)

  • v3.6.0(Dec 12, 2020)

    Changes

    • Added support for the new filtering options in Highscores
      • Added battleye_filter and pvp_types_filter attributes.
    • Added get_page_url() instance method to Highscores class.
    • Added previous_page_url and next_page_url properties.
    Source code(tar.gz)
    Source code(zip)

  • v3.5.7(Dec 4, 2020)

Owner
Allan Galarza
Mechatronic Engineer
Allan Galarza
GitHub Repository https://tibiapy.readthedocs.io/
A web scraper which checks price of a product regularly and sends price alerts by email if price reduces.

Amazon-Web-Scarper Created a web scraper using simple functions to check price of a product on amazon (can be duplicated to check price at other marke

Swaroop Todankar 1 Jan 17, 2022
A universal package of scraper scripts for humans

Scrapera is a completely Chromedriver free package that provides access to a variety of scraper scripts for most commonly used machine learning and data science domains.

299 Dec 15, 2022
A scalable frontier for web crawlers

Frontera Overview Frontera is a web crawling framework consisting of crawl frontier, and distribution/scaling primitives, allowing to build a large sc

Scrapinghub 1.2k Jan 02, 2023
A module for CME that spiders hashes across the domain with a given hash.

hash_spider A module for CME that spiders hashes across the domain with a given hash. Installation Simply copy hash_spider.py to your CME module folde

37 Sep 08, 2022
Anonymously scrapes onlinesim.ru for new usable phone numbers.

phone-scraper Anonymously scrapes onlinesim.ru for new usable phone numbers. Usage Clone the repository $ git clone https://github.com/thomasgruebl/ph

16 Oct 08, 2022
京东云无线宝积分推送，支持查看多设备积分使用情况

JDRouterPush 项目简介 本项目调用京东云无线宝API,可每天定时推送积分收益情况,帮助你更好的观察主要信息 更新日志 2021-03-02: 查询绑定的京东账户 通知排版优化 脚本检测更新 支持Server酱Turbo版 2021-02-25: 实现多设备查询 查询今

雷疯 199 Dec 12, 2022
让中国用户使用git从github下载的速度提高1000倍!

序言 github上有很多好项目,但是国内用户连github却非常的慢.每次都要用插件或者其他工具来解决. 这次自己做一个小工具,输入github原地址后,就可以自动替换为代理地址,方便大家更快速的下载. 安装 pip install cit 主要功能与用法 主要功能 change 将目标地址转换为

35 Aug 29, 2022
WebScrapping Project - G1 Latest News

Web Scrapping com Python Esse projeto consiste em um código para o usuário buscar as últimas nóticias sobre um termo qualquer, no site G1. Para esse p

Eduardo Henrique 2 Feb 13, 2022
PyQuery-based scraping micro-framework.

demiurge PyQuery-based scraping micro-framework. Supports Python 2.x and 3.x. Documentation: http://demiurge.readthedocs.org Installing demiurge $ pip

Matias Bordese 109 Jul 20, 2022
Screen scraping and web crawling framework

Pomp Pomp is a screen scraping and web crawling framework. Pomp is inspired by and similar to Scrapy, but has a simpler implementation that lacks the

Evgeniy Tatarkin 61 Jun 21, 2021
Crawler do site Fundamentus.com com o uso do framework scrapy, tanto da aba detalhada como a de resumo.

Crawler do site Fundamentus.com com o uso do framework scrapy, tanto da aba detalhada como a de resumo. (Todas as infomações)

Guilherme Silva Uchoa 3 Oct 04, 2022
热搜榜-python爬虫+正则re+beautifulsoup+xpath

仓库简介 微博热搜榜, 参数wb 百度热搜榜, 参数bd 360热点榜, 参数360 csdn热榜接口, 下方查看 其他热搜待加入 如何使用? 注册vercel fork到你的仓库, 右上角 点击这里完成部署(一键部署) 请求参数 vercel配置好的地址+api?tit=+参数(仓库简介有参数信息

Harry 3 Jul 08, 2022
基于Github Action的定时HITsz疫情上报脚本，开箱即用

HITsz Daily Report 基于 GitHub Actions 的「HITsz 疫情系统」访问入口 定时自动上报脚本，开箱即用。 感谢 @JellyBeanXiewh 提供原始脚本和 idea。 感谢 @bugstop 对脚本进行重构并新增 Easy Connect 校内代理访问。

Ter 56 Nov 27, 2022
A simple proxy scraper that utilizes the requests module in python.

Proxy Scraper A simple proxy scraper that utilizes the requests module in python. Usage Depending on your python installation your commands may vary.

3 Sep 08, 2021
The core packages of security analyzer web crawler

Security Analyzer 🐍 A large scale web crawler (considered also as vulnerability scanner tool) to take an overview about security of Moroccan sites Cu

Security Analyzer 10 Jul 03, 2022
:arrow_double_down: Dumb downloader that scrapes the web

You-Get NOTICE: Read this if you are looking for the conventional "Issues" tab. You-Get is a tiny command-line utility to download media contents (vid

Mort Yao 46.4k Jan 03, 2023
SmartScraper: 简单、自动、快捷的Python网络爬虫

SmartScraper: 简单、自动、快捷的Python网络爬虫 Note: The origin developer of SmartScraper is Alireza Mika， I only change a little code of AutoScraper. SmartScraper

DaDeng 9 Apr 16, 2022
Web-Scrapper using Python and Flask

Web-Scrapper "[초급]Python으로 웹 스크래퍼 만들기" 코스 -NomadCoders 기초적인 Python 문법강의부터 시작하여 웹사이트의 html파일에서 원하는 내용을 Scrapping해서 출력, csv 파일로 저장, flask를 이용한 간단한 웹페이지

윤성도 1 Nov 10, 2021
Kusonime scraper using python3

Features Scrap from url Scrap from recommendation Search by query Todo [+] Search by genre Example # Get download url from kusonime import Scrap

MhankBarBar 2 Jan 28, 2022
Simply scrape / download all the media from an fansly account.

Simply scrape / download all the media from an fansly account. Providing updates as long as its continuously gaining popularity, so hit the ⭐ button!

Mika C. 334 Jan 01, 2023