Realtime Financial Market Data Visualization and Analysis

Introduction

This repo shows my project about real-time stock data pipeline. All the code is written in PYTHON. In this project, I play with various Data Engineering frameworks to develop a financial data processing and visualization platform using Apache Kafka, Apache Cassandra, and Bokeh. I used Kafka for realtime stock price and market news streaming, Cassandra for historical and realtime stock data warehousing, and Bokeh for visualization on web browsers. I also wrote a web crawler to scrape companys' financial statements and basic information from Yahoo Finance, and played with various economy data APIs.

Architecture

There are currently 3 tabs in the webpage:

• Stock: Streaming & Fundamental
  • Single stock's candlestick plot, basic company & financial information;
  • Realtime S&P500 price during trading hours (fake date during non-trading hours)
• Stock: Comparison
  • 2 user-selected stocks' price, and their statstical summay and correlation
  • 5,10,30-day moving average of adjusted close price
• Economy
  • Geomap of various economy data by state
  • 4 economy indicators nationwide for comparison
  • The most recent market news

Here is the architecture of the platform.

How Stock Data is Streamed via Kafka to Cassandra:

Please check each tab's screenshot:

Tab 1:

Tab 2:

Tab 3: