A research into mail services used by different business sectors.

Overview

A Brief Research in Mail Service Used by Organisations

Author: Zeji Chen (email), student @ Lancaster University.


Table of Contents

Introduction

This project, or research, is conducted in supplyment of evidence for the author's essay assignment. This project uses Python as a crawler to verify MX Records (mail server record) for individual domains.

The Python crawler used is written by Ryan Zhao (email) - student at Manchester University, UK. Ryan has fully agreed and authorised the use of his script in this research.

The data, scripts and other files of this project is made public at https://github.com/ChenFocus/alexa-top-sites-mx. The code repository contains some files involved in the early stage of this project, that may not be relevant.

Aim of Research

This research attempts to find out the greater market share occupier between Gmail and Outlook, by looking at a large sample of organisations' domains in both technology and education sector. Please refer to Dataset Used.

Datasets Used

Alexa Top 1M Sites

Alexa is a web analytic & intelligence company owned by Amazon. The Alexa World Site Rank is often regarded as the most representive world site rank.

World Global University Dataset

Datasets Availability

Alexa Top 1M Sites Dataset: http://s3.amazonaws.com/alexa-static/top-1m.csv.zip

Please be aware that although this data is from Alexa officially, some said it's no longer updated.

Global University Dataset: https://raw.githubusercontent.com/Hipo/university-domains-list/master/world_universities_and_domains.json

Please be aware that as this data is from an open source project, which its content may change anytime in the future, if you would like to get the same dataset as this project, at the date of this research is conducted, please instead download the data from this repository, or at https://raw.githubusercontent.com/ChenFocus/university-domains-list/master/world_universities_and_domains.json.

Method

Through the dig commandline software, one is able to find the MX Record (mail server record) for a domain. By running this command:

dig +short mx example.com

If we tried to find, for example, what mail service is used by Twitter. Inc, we can run:

dig +short mx example

# response
10 aspmx.l.google.com.
30 aspmx3.googlemail.com.
20 alt1.aspmx.l.google.com.
20 alt2.aspmx.l.google.com.
30 aspmx2.googlemail.com.

Therefore we know that Twitter use googlemail, which is Gmail, for its domain twitter.com.

The Python crawler, can read a single-lined .csv (a data-sheet format), then run the dig command from a Python library (dependecy) dnspython, print the current progress of querying MX records and current stats gathered, eventually output the result as such:

Gmail:  305 
Outlook:  83 
Others:  524 
N/A:  88

Explanation:

Service name/type Description
Gmail: The count of domains in the dataset provided that use Gmail on their domain
Outlook: The count of domains in the dataset provided that use Outlook on their domain
Other: The count of domains in the dataset provided that use other email serviceson their domain
N/A: No relevant record detected; the domain could have not been configured to run email service

Accuracies/Error Analysis

As the Alexa dataset may contain asset domains (domains that are used to solely distribute static file/other recources), such as software-download.microsoft.com, which dig +short mx software-download.microsoft.com wouldn't return anything as there is no mail services running on the domain; or, no mail service is ran on the root domain. However, all calculations made from the result will exclude occassions like this.

It's known that The World Global University Dataset may not be complete and contains some inaccurate information (for example, the domain of Lancaster University in the dataset is lancs.ac.uk, instead of lancaster.ac.uk). This may affect the conclusion drawn from this dataset.

It's also known that different runs of the script may produce slightly different results, as one's DNS record may be failed to be fetched due to server/internet errors. Multiple runs of the script should eliminate this error partially.

Effort had been made to eliminate any error for any calculations made / conclusions drawn, at the best of personal knowledge, capabiltiy and computer/network limits.

Method

Getting the usage of mail service in Top 1k, 10k, 100k, 1M Alexa Ranked Sites

At a Linux computer:

git clone github.com/ChenFocus/alexa-top-sites-mx
cd alexa-top-sites-mx
cd python_method

# make sure Python 3 and Pip 3 is installed
pip install dnspython
python main.py 1000 | tee result_1000.txt # modify the `1000` to 10000 and more to get different results
# wait for completion of the script

A cloud computer (server) is used to calculate results for above targets.

As the Alexa rank list goes further, more irrevelant/personal domains may be recorded, therefore resulting in more inaccurate results.

Getting the usage of mail service in Global University Dataset

At a Linux computer:

git clone github.com/ChenFocus/alexa-top-sites-mx
cd alexa-top-sites-mx
cd python_method

# make sure Python 3 and Pip 3 is installed
pip install dnspython
python main.py 9773 | tee result.txt # Check whole dataset - usage of mail services
python us_uni.py 2730 | tee result_us_uni.txt # Check for US universities only
python uk_uni.py 161 | tee result_uk_uni.txt # Check for UK universities only
# wait for completion of the script

The conversion of the original .json data to .csv format is through a series of replace/regrex match/plugin operation conducted in Visual Studio Code.

Results

Alexa Dataset

Dataset Name/Variation Gmail Counts Outlook Counts Others Counts N/A Counts Total Counts Total Valid Counts Calculated Gmail Share Calculated Outlook Share Gmail Outcompete Outlook in Technology Sector By
Alexa Top 1M - 1k list 305 83 524 88 1000 912 33.4% 9.1% 73%
Alexa Top 1M - 10k list 2779 873 4961 1387 10000 8613 32.3% 10.1% 69%
Alexa Top 1M - 100k list 22527 8461 51942 17070 100000 82930 27.2% 10.2% 62%

Note that the result for the whole 1 million site is currently being calculated by a cloud server. The result of this may be updated in the future.

Global University Dataset

Dataset Name/Variation Gmail Counts Outlook Counts Others Counts N/A Counts Total Counts Total Valid Counts Calculated Gmail Share Calculated Outlook Share Outlook Outcompete Gmail in Education Sector By
World University list 1296 2158 4691 1628 9773 8145 15.9% 26.5% 40%
World University list - US only 298 923 876 273 2370 2097 14.2% 44.0% 68%
World University list - UK only 8 63 65 25 161 136 5.9% 46.3% 87%

Simple Conclusions

From the results, Gmail leads the competition in the technology sector by occupying 33.4% market share in the Top 1000 websites ranked by Alexa. This data is extended into 32.3%, 27.2% when 10k, 100k top sites are used as samples. From results available, Gmail beats Outlook in the technology sector with at least 60% more market share.

Outlook outcompetes Gmail in the education sector, occupying 44.0% and 46.3% market share in education section in US & UK respectively, outcompeting Gmail with 68% and 87% more market share.

Result Tables

Technology Sector - From Alexa Top 1M Dataset

Dataset Name/Variation Calculated Gmail Share Gmail Outcompete Outlook in Technology Sector By
Alexa Top 1M - 1k list 33.4% 73%
Alexa Top 1M - 10k list 32.3% 69%
Alexa Top 1M - 100k list 27.2% 62%

Education Sector - From Global University Dataset

Dataset Name/Variation Calculated Outlook Share Outlook Outcompete Gmail in Education Sector By
World University list 26.5% 40%
World University list - US only 44.0% 68%
World University list - UK only 46.3% 87%
Owner
Focus Chen
My Academic Research at Lancaster University
Focus Chen
Django module to easily send templated emails using django templates, or using a transactional mail provider (mailchimp, silverpop, etc.)

Django-Templated-Email Info: A Django oriented templated email sending class Author: Bradley Whittington (http://github.com/bradwhittington, http://tw

Vinta Software 659 Dec 27, 2022
A python script that helps you understand why your E-Mail ended up in Spam

decode-spam-headers.py Whether you are trying to understand why a specific e-mail ended up in SPAM/Junk for your daily Administrative duties or for yo

Mariusz Banach 316 Jan 05, 2023
faceFarm is an active yahoo email detection script that functions to take over the facebook account using email.

faceFarm – The simple Email Detector. Email Valid Detector for Facebook (Yahoo) faceFarm is an active yahoo email detection script that functions to t

Fadjrir Herlambang 2 Jan 18, 2022
Simple, powerfull and nonobstructive django email middleware.

djmail djmail is a BSD Licensed, simple and nonobstructive django email middleware. Why use djmail? Because it: Sends emails asynchronously without ad

David Barragán Merino 77 Aug 30, 2021
Kanmail - An email client that functions like a kanban board, for Mac/Windows/Docker

Kanmail - An email client that functions like a kanban board, for Mac/Windows/Docker

Oxygem 1.2k Dec 31, 2022
An API to send emails through python3's smtplib module.

An API to send emails through python3's smtplib module. Just configure your SMTP server credentials and you are ready to send a lot of emails through API, designed to be used as a newsletter service.

Adnan Ahmad 15 Nov 24, 2022
A Pythonic interface for Google Mail

GMail for Python A Pythonic interface to Google's GMail, with all the tools you'll need. Search, read and send multipart emails, archive, mark as read

Charlie Guo 1.7k Dec 29, 2022
A CLI client for sending text emails. (Currently only gmail supported)

emailCLI A CLI client for sending text emails. (Currently only gmail supported)

Amethist 3 Dec 17, 2021
Mail-Checker is a python script that lets you see your mails directly from the terminal without having to login each time.

Mail-Checker ##Mail-Checker is a python script that lets you see your mails directly from the terminal without having to login each time. ##Before you

Siddharth Pradeep 1 Jan 12, 2022
Envia-emails - A Python Program that creates emails

Envia-emails Os emails é algo muito importante e usado. Pensando nisso, eu criei

José Rodolfo 2 Mar 05, 2022
A light-weight, modular, message representation and mail delivery framework for Python.

Marrow Mailer A highly efficient and modular mail delivery framework for Python 2.6+ and 3.2+, formerly called TurboMail. © 2006-2019, Alice Bevan-McG

Marrow Open Source Collective 255 Dec 28, 2022
Django SMTP Protocol with Gmail

Django SMTP Protocol with Gmail This is the free service from gmail to send and receive emails. What we need for this things done, Python/pip install

Mehedi Hasan 3 Dec 13, 2022
This Tool Is For Sending Emails From A Terminal(Termux/Kali) etc.

This is a Basic python script to send emails from a Terminal(Termux/Kali) are the only tested currently.

AnonyVox 2 Apr 04, 2022
xxnx its a simple smtp tool for mails spaming

xxnx its a simple smtp tool for mails spaming what is smpt? Simple Mail Transfer Protocol or smtp service. The Simple Mail Transfer Protocol (SMTP) is

0xD4$H 3 Feb 27, 2022
This python script will generate passwords for your emails, With certain lengths, And saves them into plain text files.

How to use. Change the Default length of genereated password in default.length.txt Type the email for your account. Type the website that the email an

2 Dec 26, 2021
automatic mails sender with attachments

أزعجني لين تدربني Automatic mails sender with attachments. Note: You need to have gmail account & and you need to turn on "Less secure app access" set

6 Dec 30, 2022
利用阿里的云函数发送电子邮件

alifc_email 主要特性 利用阿里的云函数发送电子邮件 使用场景 hw中的钓鱼邮件发送,一些邮服会解析出邮件的来源ip(此来源ip并不是邮服的ip,而是从客户端发送邮件时,邮服自动带上的客户端ip),对于这些来源ip可能会做一些风控。 本项目利用云函数出口ip较多来绕过这些风控 使用方法 首

19 Dec 01, 2022
Read/sync your IMAP mailboxes (python2)

Upstream status (master branch): Upstream status (next branch): Financial contributors: Links: Official github code repository: offlineimap Website: w

OfflineIMAP 1.7k Dec 29, 2022
Send email in Python conveniently for gmail using yagmail

yagmail -- Yet Another GMAIL/SMTP client For the asynchronous asyncio version, look here: https://github.com/kootenpv/aioyagmail The goal here is to m

Pascal van Kooten 2.4k Dec 31, 2022
Yahoo Mail Validator For Python

Validator Validator helps to know if the mail is valid or not Installation Install The libraries pip install requests bs4 colorama Usage Create a new

Mr Python 3 Mar 12, 2022