Automatically detect changes made to the official Telegram sites.

Overview

๐Ÿ•ท Telegram Web Crawler

This project is developed to automatically detect changes made to the official Telegram sites. This is necessary for anticipating future updates and other things (new vacancies, API updates, etc).

Name Commits Status
Site updates tracker Commits Fetch new content of tracked links to files
Site links tracker Commits Generate or update list of tracked links
  • โœ… passing โ€“ new changes
  • โŒ failing โ€“ no changes

You should to subscribe to channel with alerts to stay updated. Copy of Telegram websites stored here.

GitHub pretty diff

How it works

  1. Link crawling runs as often as possible. Starts crawling from the home page of the site. Detects relative and absolute sub links and recursively repeats the operation. Writes a list of unique links for future content comparison. Additionally, there is the ability to add links by hand to help the script find more hidden (links to which no one refers) links. To manage exceptions, there is a system of rules for the link crawler.

  2. Content crawling is launched as often as possible and uses the existing list of links collected in step 1. Going through the base it gets contains and builds a system of subfolders and files. Removes all dynamic content from files.

  3. Using of GitHub Actions. Works without own servers. You can just fork this repository and own tracker system by yourself. Workflows launch scripts and commit changes. All file changes are tracked by the GIT and beautifully displayed on the GitHub. GitHub Actions should be built correctly only if there are changes on the Telegram website. Otherwise, the workflow should fail. If build was successful, we can send notifications to Telegram channel and so on.

FAQ

Q: How often is "as often as possible"?

A: TLTR: content update action runs every ~10 minutes. More info:

Q: Why there is 2 separated crawl scripts instead of one?

A: Because the previous idea was to update tracked links once at hour. It was so comfortably to use separated scripts and workflows. After Telegram 7.7 update, I realised that find new blog posts so slowly is bad idea.

Q: Why alert for sending alerts have while loop?

A: Because GitHub API doesn't return information about commit immediately after push to repository. Therefore, script are waiting for information to appear...

Q: Why are you using GitHab Personal Access Token in action/checkout workflow`s step?

A: To have ability to trigger other workflows by on push trigger. More info:

Q: Why are you using GitHab PAT in make_and_send_alert.py?

A: To increase limits of GitHub API.

TODO list

  • add storing history of content using hashes;
  • add storing hashes of image, svg, video.

Example of link crawler rules configuration

CRAWL_RULES = {
    # every rule is regex
    # empty string means match any url
    # allow rules with higher priority than deny
    'translations.telegram.org': {
        'allow': {
            r'^[^/]*$',  # root
            r'org/[^/]*/$',  # 1 lvl sub
            r'/en/[a-z_]+/$'  # 1 lvl after /en/
        },
        'deny': {
            '',  # all
        }
    },
    'bugs.telegram.org': {
        'deny': {
            '',    # deny all sub domain
        },
    },
}

Current hidden urls list

HIDDEN_URLS = {
    # 'corefork.telegram.org', # disabled

    'telegram.org/privacy/gmailbot',
    'telegram.org/tos',
    'telegram.org/tour',
    'telegram.org/evolution',

    'desktop.telegram.org/changelog',
}

License

Licensed under the MIT License.

Owner
Il'ya
Telegram: https://t.me/MarshalX
Il'ya
Wakatime Response In javascript and python

Project Title Wakatime Response In javascript and python Description just for refrence Getting Started Dependencies For Python: requests json For Java

Gjenius20 1 Dec 31, 2021
A simple discord bot based on python

A simple discord bot based on python

SENPAI LEGEND 2 Jul 24, 2022
TG-Streaming-bot - TG Simple Streaming bot

TG Simple Streaming bot telegram video straming bot ๐ŸŽš๏ธ Features Play youtube li

HyDrix 4 May 05, 2022
YouTube-Discord-Bot - Discord Bot to Search YouTube

YouTube Bot Info YouTube Bot is a discord bot where you can search for anything

Riceblades11 10 Mar 05, 2022
A Discord Server Cloner Which Can Clone Any Discord Server In Just Few Minutes

A Discord Server Cloner Which Can Clone Any Discord Server In Just Few Minutes.

samet 4 Jul 23, 2022
Python3 wrapper for the Sibyl System antispam API for telegram

SibylSystem-Py Python3 wrapper for the Sibyl System antispam API for telegram Installation pip install sibylsystem Usage from SibylSystem import

Kaizoku 6 Nov 04, 2022
Connect your Nintendo Switch playing status to Discord!

Disclaimer: Unfortunately, it appears that Nintendo has removed returning self-Presence in their API as of recently, making this project near obsolete

Deltaion Lee 145 Dec 30, 2022
An almost dependency-less, synchronous Discord gateway library meant for my personal use

An almost dependency-less, synchronous Discord gateway library meant for my personal use.

h0nda 4 Feb 05, 2022
Fast and multi-threaded script to automatically claim targeted username including 14 day bypass

Instagram Username Auto Claimer Fast and multi-threaded script to automatically claim targeted username. Click here to report bugs. Usage Download ZIP

265 Dec 28, 2022
A smooth and powerful Telegram Userbot made to make Telegram easier.

| Xแด‡ษดแด Bแดแด› Is One Of The Fastest & Smoothest Bot On Telegram Based on Telethon|

SimpleBoy 1 Dec 01, 2021
Posts locally saved videos to the desired subreddit

redditvideoposter posts locally saved videos to the desired subreddit ================================================================= STEPS: pip ins

Kyrus 2 Dec 01, 2021
Pycardano - A lightweight Cardano client in Python

PyCardano PyCardano is a standalone Cardano client written in Python. The librar

151 Dec 31, 2022
Instagram bot that upload images for you which scrape posts from 9gag meme website or other Instagram users , which is 24/7 Automated Runnable.

Autonicgram Automates your Instagram posts by taking images from sites like 9gag or other Instagram accounts and posting it onto your page. Features A

Mastermind 20 Sep 17, 2022
Tools for use in DeFi. Impermanent Loss calculations, staking and farming strategies, coingecko and pancakeswap API queries, liquidity pools and more

DeFi open source tools Get Started Instalation General Tools Impermanent Loss, simple calculation Compare Buy & Hold with Staking and Farming Complete

Juan Pablo Pisano 467 Jan 08, 2023
Maintained Fork of Jishaku For nextcord

Onami a debugging and utility extension for nextcord bots Read the documentation online. Fork Onami is a actively maintained fork of Jishaku for nextc

RPS 11 Dec 14, 2022
:lock: Python 2.7/3.X client for HashiCorp Vault

hvac HashiCorp Vault API client for Python 3.x Tested against the latest release, HEAD ref, and 3 previous minor versions (counting back from the late

hvac 1k Dec 29, 2022
A telegram bot that can send you high-quality audio ๐ŸŽง๐ŸŽง๐ŸŽง

Music downloader bot Still under development Please Report issues to improve this repo.I will try to fix bugs in next update Music downloader bot is a

Anish Gowda 36 Dec 06, 2022
Python package for Calendly API v2

PyCalendly Python package to use Calendly API-v2. Installation Install with pip $ pip install PyCalendly Usage Getting Started See Getting Started wi

Lakshmanan Meiyappan 20 Dec 05, 2022
Housing Price Prediction Using Machine Learning.

HOUSING PRICE PREDICTION USING MACHINE LEARNING DESCRIPTION Housing Price Prediction Using Machine Learning is to predict the data of housings. Here I

Shreya Shree Padhi 1 Aug 03, 2022
Minimal API for the COVID Booking System of the Offices at the UniPD Math Dep

Simple and easy to use python BOT for the COVID registration booking system of the math department @ unipd (torre archimede). This API creates an interface with the official website, with more useful

Guglielmo Camporese 4 Dec 24, 2021