Automatically detect changes made to the official Telegram sites.

Overview

🕷 Telegram Web Crawler

This project is developed to automatically detect changes made to the official Telegram sites. This is necessary for anticipating future updates and other things (new vacancies, API updates, etc).

Name Commits Status
Site updates tracker Commits Fetch new content of tracked links to files
Site links tracker Commits Generate or update list of tracked links
  • passing – new changes
  • failing – no changes

You should to subscribe to channel with alerts to stay updated. Copy of Telegram websites stored here.

GitHub pretty diff

How it works

  1. Link crawling runs as often as possible. Starts crawling from the home page of the site. Detects relative and absolute sub links and recursively repeats the operation. Writes a list of unique links for future content comparison. Additionally, there is the ability to add links by hand to help the script find more hidden (links to which no one refers) links. To manage exceptions, there is a system of rules for the link crawler.

  2. Content crawling is launched as often as possible and uses the existing list of links collected in step 1. Going through the base it gets contains and builds a system of subfolders and files. Removes all dynamic content from files.

  3. Using of GitHub Actions. Works without own servers. You can just fork this repository and own tracker system by yourself. Workflows launch scripts and commit changes. All file changes are tracked by the GIT and beautifully displayed on the GitHub. GitHub Actions should be built correctly only if there are changes on the Telegram website. Otherwise, the workflow should fail. If build was successful, we can send notifications to Telegram channel and so on.

FAQ

Q: How often is "as often as possible"?

A: TLTR: content update action runs every ~10 minutes. More info:

Q: Why there is 2 separated crawl scripts instead of one?

A: Because the previous idea was to update tracked links once at hour. It was so comfortably to use separated scripts and workflows. After Telegram 7.7 update, I realised that find new blog posts so slowly is bad idea.

Q: Why alert for sending alerts have while loop?

A: Because GitHub API doesn't return information about commit immediately after push to repository. Therefore, script are waiting for information to appear...

Q: Why are you using GitHab Personal Access Token in action/checkout workflow`s step?

A: To have ability to trigger other workflows by on push trigger. More info:

Q: Why are you using GitHab PAT in make_and_send_alert.py?

A: To increase limits of GitHub API.

TODO list

  • add storing history of content using hashes;
  • add storing hashes of image, svg, video.

Example of link crawler rules configuration

CRAWL_RULES = {
    # every rule is regex
    # empty string means match any url
    # allow rules with higher priority than deny
    'translations.telegram.org': {
        'allow': {
            r'^[^/]*$',  # root
            r'org/[^/]*/$',  # 1 lvl sub
            r'/en/[a-z_]+/$'  # 1 lvl after /en/
        },
        'deny': {
            '',  # all
        }
    },
    'bugs.telegram.org': {
        'deny': {
            '',    # deny all sub domain
        },
    },
}

Current hidden urls list

HIDDEN_URLS = {
    # 'corefork.telegram.org', # disabled

    'telegram.org/privacy/gmailbot',
    'telegram.org/tos',
    'telegram.org/tour',
    'telegram.org/evolution',

    'desktop.telegram.org/changelog',
}

License

Licensed under the MIT License.

Owner
Il'ya
Telegram: https://t.me/MarshalX
Il'ya
Want to play What Would Rather on your Server? Invite the bot now!😏

What is this Bot? 👀 What You Would Rather? is a Guessing game where you guess one thing. Long Description short Take this example: You typed r!rather

丂ㄚ么乙ツ 2 Nov 17, 2021
A simple Discord bot that notifies users of new Abitti versions

A simple Discord bot that notifies users of new Abitti versions. New features might be added later on. If you have good ideas, feel free to do a PR.

1 Feb 11, 2022
An unofficial python wrapper for the comdirect API for private consumers.

An unofficial python wrapper for the comdirect API for private consumers.

Alexander Knittel 18 Nov 30, 2022
Trabalho N1 para a materia Tecnicas de Progamação da Anhembi Morumbi

Projeto da Anhembi Morumbi - Tecnicas de Programação. RPG de Console (CMD) Trabalho proposto pelo professor André Santana, na materia Tecnicas de Prog

Leonardo Silva M de Barros 3 Sep 12, 2021
A Python app which retrieves the rank and players' equipped skins during a match

VALORANT rank yoinker About The Project Usage Contributing Contact Acknowledgements Disclaimer About The Project Their Queue Current Skin Current Rank

Isaac Kenyon 270 Jan 04, 2023
Framework to collect and process weather data from wttr.in.

Weathercrawler Automatic extraction and processing framework for weather data from wttr.in Installation tested with: Python 3.7.3 Python 3.9.4 git clo

Maurice Günder 0 Jul 26, 2021
The program for obtaining a horoscope in Python using API from rapidapi.com site.

Python horoscope The program allows you to get a horoscope for your zodiac sign and immediately translate it into almost any language. Step 1 The firs

Architect 0 Dec 25, 2021
A python API for BSCScan (Binance Smart Chain Explorer), available on PyPI.

bscscan-python A complete Python API for BscScan.com, available on PyPI. Powered by BscScan.com APIs. This is a gently modified fork of the etherscan-

Panagiotis Kotsias 246 Dec 31, 2022
A Telegram Bot to return Youtube Video Tags Using YoutubeTags API

YouTube-TagFind-Bot A Telegram Bot to return Youtube Video Tags Using YoutubeTags API YoutubeTags API Wrapper YoutubeTags is a python third-party api

Nuhman Pk 9 Aug 25, 2022
A multi purpose discord bot for python

Sypher The best multi purpose discord bot. Add Sypher right now Invite Me | Join

Johan Naizu 1 Dec 15, 2022
Telegram bot to stream videos in telegram voicechat for both groups and channels. Supports live strams, YouTube videos and telegram media.

Telegram VCVideoPlayBot An Telegram Bot By @ZauteKm To Stream Videos in Telegram Voice Chat. NOTE: Make sure you have started a VoiceChat in your Grou

Zaute 20 Oct 21, 2022
This is a free python bot program that crosses you to farm with auto click in space crypto NFT game, having fun :) Creator: Marlon Zanardi

🚀 Space Crypto auto click bot ready-to-use 🚀 This is a free python bot program that crosses you to farm with auto click in space crypto NFT game, ha

170 Dec 20, 2022
POC de uma AWS lambda que executa a consulta de preços de criptomoedas, e é implantada na AWS usando Github actions.

Cryptocurrency Prices Overview Instalação Repositório Configuração CI/CD Roadmap Testes Overview A ideia deste projeto é aplicar o conteúdo estudado s

Gustavo Santos 3 Aug 31, 2022
A library for demo trading | backtest and forward test simulation

Trade Engine a library for demo trading | backtest and forward test simulation Features Limit/Market orders: you can place a Limit or Market order in

Ali Moradi 7 Jul 02, 2022
The community bot for the Python Discord community

Python Utility Bot This project is a Discord bot specifically for use with the Python Discord server. It provides numerous utilities and other tools t

Python Discord 998 Jan 03, 2023
Yes, it's true :revolving_hearts: This repository has 301 stars.

Yes, it's true! Inspired by a similar repository from @RealPeha, but implemented using a webhook on AWS Lambda and API Gateway, so it's serv

510 Dec 28, 2022
IOGen - An Open source discord token generator

_____ ____ _____ |_ _/ __ \ / ____| | || | | | |

0xVichy#1234 85 Nov 03, 2022
A Telegram Bot Written In Python

TelegraphUploader A Telegram Bot Written In Python DEPLOY Local Machine Clone the repository Install requirements: pip3 install -r requirements.txt e

Wahyusaputra 2 Dec 29, 2021
A userbot made for telegram

𝚃𝙷𝙴 𝙼𝙰𝙵𝙸𝙰𝙱𝙾𝚃 This is a userbot made for telegram. I made this userbot with help of all other userbots available in telegram. All credits go

MafiaBotOP 8 Apr 08, 2022
Beyonic API Python official client library simplified examples using Flask, Django and Fast API.

Beyonic API Python official client library simplified examples using Flask, Django and Fast API.