A machine learning model for analyzing text for user sentiment and determine whether its a positive, neutral, or negative review.

Overview

Sentiment Analysis on Yelp's Dataset

Author: Roberto Sanchez, Talent Path: D1 Group

Docker Deployment:

Deployment of this application can be found here hosted on AWS

Running it locally:

docker pull rsanchez2892/sentiment_analysis_app

Overview

The scope of this capstone is centered around the data processing, exploratory data analysis, and training of a model to predict sentiment on user reviews.

End goal of the model

Business Goals

Create a model to be able to be used in generating sentiment on reviews or comments found in external / internal websites to give insights on how people feel about certain topics.

This could give the company insights not easily available on sites where ratings are required or for internal use to determine sentiment on blogs or comments.

Business Applications

By utilizing this model, the business can use it for the following purposes:

External:

  • Monitoring Brand and Reputation online
  • Product Research

Internal:

  • Customer Support
  • Customer Feedback
  • Employee Satisfaction

Currently method to achieving this is by using outside resources which come at a cost and increases risk for leaking sensitive data to the public. This product will bypass these outside resources and give the company the ability to do it in house.

Model Deployment

Link: Review Analyzer

After running multiple models and comparing accuracy, I found that the LinearSVC model is a viable candidate to be used in production for analyzing reviews of services or food.

Classification Report / Confusion Matrix:

Classification Report

Technology Stack

I have been using these technologies for this project:

  • Jupyter Notebook - Version 6.3.0
    • Used for most of the data processing, EDA, and model training.
  • Python - Version 3.8.8
    • The main language this project will be done in.
  • Scikit-learn - Version 0.24
    • Utilizing metrics reports and certain models.
  • Postgres - Version 13
    • Main database application used to store this data.
  • Flask - Version 1.1.2
    • Main backend technology to host a usable version of this project to the public.
  • GitHub
    • Versioning control and online documentation
  • Heroku
    • Online cloud platform to host this application for public use

Data Processing

This capstone uses the Yelp dataset found on Kaggle which comprises of multiple files:

  • Business Data
  • Check-in Data
  • Review Data
  • Tips Data
  • User Data

Stage 1 - Read in From JSON files into Postgres

Overview

  • Read in JSON files
  • General observations on the features found in each file
  • Modifying feature names to meet Postgres naming convention
  • Normalized the data to prepare for import to Postgres
  • Saved copies of each table as CSV file for backup incase Database goes down
  • Exported data into Postgres

As stated above, Kaggle provided several JSON files with a large amount of data that needed to be stored in a location for easy access and provide a quick way to query data on the fly. As the files were read in Jupyter notebook a general observation was made to the feature names and amount each file contained to see what data I was dealing with along with the types associated with them. The business data contained a strange number of attributes that had to be broken up into separate data frames to be normalized for Postgres.

Stage 2 - Pre-Processing Data

Overview

  • Read in data from Postgres
  • IDing Null Values
  • Removing Sparse features
  • Saved data frame as a pickle to be used in model training

This stage I performed elementary data analysis where I analyze any null values, see the distribution of my ratings and review lengths.

Stage 3 - Cleaning Up Data

Overview

  • Replace contractions with expanded versions
  • Lemmatized text
  • Removed special characters, dates, emails, and URLs
  • Removed stop words
  • Remove non-english text
  • Normalized text

Exploratory Data Analysis

Analyzing Null Values in Dataset

Below is a visualization of the data provided by Kaggle showing which features have "NaN " values. Its is clear that the review ratings (review_stars) and reviews (text) are fully populated. Some of the business attributes are sparse but have enough values to be useful for other things. Note several other features were dropped in the Data Processing since they did not provide any insights for the scope of this project.

Heatmap of several million rows of data.

Looking Closer at the Ratings (review_stars)

This is a sample of 2 million rows from the original 8 million in the dataset. This distribution of ratings has a left skew on it where most of the reviews are 4 to 5 stars.

A bar graph showing the distribution of ratings between 1 to 5. there is a significant amount of 5 stars compared to 1-3 combined.

I simplified the ratings to better categorize the sentiment of the review by grouping 1 and 2 star reviews as 'negative', 3 star review as 'neutral', and 4 and 5 star reviews as 'positive'.

Simplified Barchar showing just the negative, neutral, and positive ratings

Looking Closer at the Reviews (text)

To analyze the text, I've calculated the length of each review in the sample and plotted a distribution graph showing them the number of characters of each review. The statistics were that the median review was approx. 606 characters with a range of 0 through 5000 characters.

Showing a distribution chart of the length of the reviews. Clearly the distribution skews right with a median around 400 characters.

A closer inspection on the range 0 - 2000 we can see that most of the reviews are around this general area.

A zoomed in version of the same distribution chart now focusing on 0 - 2000 characters

In order to produce a viable word cloud, I've had to process all of the text in the sample to remove special characters and stop words from NLTK to produce a viable string to be used in word cloud. Below is a visualization of all of the key words found in the positive reviews.

Created a word cloud from the positive words after cleaning

As expected, words like "perfect", "great", "good", "great place", and "highly recommend" came out on top.

A word cloud showing all the words from the negative reviews

On the negative word cloud, words like "bad", "customer service", "never", "horrible", and "awful" are appearing on the word cloud.

Model Training

Model Selection

model selection flow chart

These four models were chosen to be trained with this data. Each of these models had a pipeline created with TfidfVectorizer.

Model Training

  • Run a StratifiedKFold with a 5 fold split and analyze the average scores and classification reports
    • Get an average accuracy of the model for comparison
  • Create a single model to generate a confusion matrix
  • Test out model on a handful of examples

Below is the average metrics after running 5 fold cross validation on LinearSVC

average metrics for linearSVC model

Testing Model

After the model was trained, I fed it some reviews I found online to test out whether or not the model can properly detect the right sentiment. The following reviews are ordered as "Negative", "Neutral", and "Positive":

new_test_data = [
    "This was the worst place I've ever eaten at. The staff was rude and did not take my order until after i pulled out my wallet.",
    "The food was alright, nothing special about this place. I would recommend going elsewhere.",
    "I had a pleasent time with kimberly at the granny shack. The food was amazing and very family friendly.",
]
res = model.prediction(new_test_data)

Below is the results of the prediction, notice that the neutral review has been labeled as negative. This makes sense since the model has a poor recall for neutral reviews as shown in the classification report.

Results from the prediction

End Notes

There are some improvements to be made such as the follow:

  • Balancing the data
    • This can be seen in the confusion matrix for the candidate models and other models created that the predictions come out more positive than negative or neutral.
    • While having poor scores in the neutral category, the most important features are found in the negative and positive predictions for business applications.
  • Hyper-parametrization improvement
    • Logistic Regression and Multinomial NB models produced models within a reasonable time frame while returning reasonable scores. Random Forrest Classifier and SVM took a significant amount of time to produce just one iteration. In order to produce results from this model StratifiedKFold was not used in these two models. Changing SVM to LinearSVC improved performance dramatically and replaced the SVM model and outperformed Logistic Regression which was the original candidate model.
Owner
Roberto Sanchez
Full Stack Web Developer / Data Science Startup
Roberto Sanchez
CCF BDCI 2020 房产行业聊天问答匹配赛道 A榜47/2985

CCF BDCI 2020 房产行业聊天问答匹配 A榜47/2985 赛题描述详见:https://www.datafountain.cn/competitions/474 文件说明 data: 存放训练数据和测试数据以及预处理代码 model_bert.py: 网络模型结构定义 adv_train

shuo 40 Sep 28, 2022
Telegram bot to auto post messages of one channel in another channel as soon as it is posted, without the forwarded tag.

Channel Auto-Post Bot This bot can send all new messages from one channel, directly to another channel (or group, just in case), without the forwarded

Aditya 128 Dec 29, 2022
基于GRU网络的句子判断程序/A program based on GRU network for judging sentences

SentencesJudger SentencesJudger 是一个基于GRU神经网络的句子判断程序,基本的功能是判断文章中的某一句话是否为一个优美的句子。 English 如何使用SentencesJudger 确认Python运行环境 安装pyTorch与LTP python3 -m pip

8 Mar 24, 2022
The (extremely) naive sentiment classification function based on NBSVM trained on wisesight_sentiment

thai_sentiment The naive sentiment classification function based on NBSVM trained on wisesight_sentiment วิธีติดตั้ง pip install thai_sentiment==0.1.3

Charin 7 Dec 08, 2022
Gathers machine learning and Tensorflow deep learning models for NLP problems, 1.13 < Tensorflow < 2.0

NLP-Models-Tensorflow, Gathers machine learning and tensorflow deep learning models for NLP problems, code simplify inside Jupyter Notebooks 100%. Tab

HUSEIN ZOLKEPLI 1.7k Dec 30, 2022
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context

Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context This repository contains the code in both PyTorch and TensorFlow for our paper

Zhilin Yang 3.3k Dec 28, 2022
This repo is to provide a list of literature regarding Deep Learning on Graphs for NLP

This repo is to provide a list of literature regarding Deep Learning on Graphs for NLP

Graph4AI 230 Nov 22, 2022
Search Git commits in natural language

NaLCoS - NAtural Language COmmit Search Search commit messages in your repository in natural language. NaLCoS (NAtural Language COmmit Search) is a co

Pushkar Patel 50 Mar 22, 2022
DiY Oxygen Concentrator based on the OxiKit

M19O2 DiY Oxygen Concentrator based on / inspired by the OxiKit, OpenOx, Marut, RepRap and Project Apollo platforms. About Read about the project on H

Maker's Asylum 62 Dec 22, 2022
This project uses unsupervised machine learning to identify correlations between daily inoculation rates in the USA and twitter sentiment in regards to COVID-19.

Twitter COVID-19 Sentiment Analysis Members: Christopher Bach | Khalid Hamid Fallous | Jay Hirpara | Jing Tang | Graham Thomas | David Wetherhold Pro

4 Oct 15, 2022
Tools, wrappers, etc... for data science with a concentration on text processing

Rosetta Tools for data science with a focus on text processing. Focuses on "medium data", i.e. data too big to fit into memory but too small to necess

207 Nov 22, 2022
Open-Source Toolkit for End-to-End Speech Recognition leveraging PyTorch-Lightning and Hydra.

OpenSpeech provides reference implementations of various ASR modeling papers and three languages recipe to perform tasks on automatic speech recogniti

Soohwan Kim 26 Dec 14, 2022
Signature remover is a NLP based solution which removes email signatures from the rest of the text.

Signature Remover Signature remover is a NLP based solution which removes email signatures from the rest of the text. It helps to enchance data conten

Forges Alterway 8 Jan 06, 2023
Easy, fast, effective, and automatic g-code compression!

Getting to the meat of g-code. Easy, fast, effective, and automatic g-code compression! MeatPack nearly doubles the effective data rate of a standard

Scott Mudge 97 Nov 21, 2022
The code from the whylogs workshop in DataTalks.Club on 29 March 2022

whylogs Workshop The code from the whylogs workshop in DataTalks.Club on 29 March 2022 whylogs - The open source standard for data logging (Don't forg

DataTalksClub 12 Sep 05, 2022
Python library for parsing resumes using natural language processing and machine learning

CVParser Python library for parsing resumes using natural language processing and machine learning. Setup Installation on Linux and Mac OS Follow the

nafiu 0 Jul 29, 2021
Abhijith Neil Abraham 2 Nov 05, 2021
Creating a Feed of MISP Events from ThreatFox (by abuse.ch)

ThreatFox2Misp Creating a Feed of MISP Events from ThreatFox (by abuse.ch) What will it do? This will fetch IOCs from ThreatFox by Abuse.ch, convert t

17 Nov 22, 2022
숭실대학교 컴퓨터학부 전공종합설계프로젝트

✨ 시각장애인을 위한 버스도착 알림 장치 ✨ 👀 개요 현대 사회에서 대중교통 위치 정보를 이용하여 사람들이 간단하게 이용할 대중교통의 정보를 얻고 쉽게 대중교통을 이용할 수 있다. 해당 정보는 각종 어플리케이션과 대중교통 이용시설에서 위치 정보를 제공하고 있지만 시각

taegyun 3 Jan 25, 2022
(ACL 2022) The source code for the paper "Towards Abstractive Grounded Summarization of Podcast Transcripts"

Towards Abstractive Grounded Summarization of Podcast Transcripts We provide the source code for the paper "Towards Abstractive Grounded Summarization

10 Jul 01, 2022