Skip to content

faisal12101123/Website-Crawler-Python-

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

3 Commits
 
 
 
 

Repository files navigation

Website-Crawler-Python

This is a simple website crawler which asks for a website link from the user to crawl and find specific data from the given website address. After getting the website address, it asks for how much crawling depth the user wants in between the number of links has been found after providing the website address.

Website Crawler takes 3 inputs:

  1. A website address
  2. Integer value for the crawling depth
  3. A user specified regular expression to find user specific data

General tasks:

  1. Find all the Nowgegian mobile numbers and saves into a text file.
  2. Find all the sub-links inside the given website and saves into a text file.
  3. Saves the website's raw HTML code into a text file.
  4. Find all email addresses and save into a text file.
  5. Find all the comments used in the website and saves it into a text file.
  6. Find five most used words and print it into the terminal.

This is a Python based project and used some dependent libraries to execute the functionalities.

  1. RegEx
  2. Urllib3
  3. BeautifulSoup 4
  4. Counter in Collections

About

This is a simple website crawler which asks for a website link from the user to crawl and find specific data from the given website address.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages