Twitter data streaming using Twitter APIs

Streaming real-time Twitter Trending Hash-Tag data using twitter API, Kafka, Python.

1. Python
2. Twitter Developer Account
3. Kafka

https://www.python.org/downloads/

https://developer.twitter.com/en/portal/products/elevated

https://kafka.apache.org/quickstart

Steps to run the code successfully -

1. Stop previous running zookeeper(if zookeeper stops aburptly when you try to start) -

sudo service zookeeper stop

2. Start the zookeeper -

bin/zookeeper-server-start.sh config/zookeeper.properties

3. Start the kafka server in new terminal -

bin/kafka-server-start.sh config/server.properties

4. Create topics -

bin/kafka-topics.sh --create --partitions 1 --replication-factor 1 --topic trump --bootstrap-server localhost:9092

5. Run the python code in new terminal -

python kafka_producer_pandas.py

7. Start kafka consumer in new terminal-

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic trump --from-beginning

8. Run the kafka consumer and streaming code

python kafka_consumer_pandas_df.py

Streaming real-time Twitter data using twitter API, Kafka, Pyspark.

1. Python
2. Twitter Developer Account
3. Kafka
4. Spark

https://www.python.org/downloads/

https://developer.twitter.com/en/portal/products/elevated

https://kafka.apache.org/quickstart

https://spark.apache.org/downloads.html

Steps to run the code successfully -

1. Stop previous running zookeeper(if zookeeper stops aburptly when you try to start) -

sudo service zookeeper stop

2. Start the zookeeper -

bin/zookeeper-server-start.sh config/zookeeper.properties

3. Start the kafka server in new terminal -

bin/kafka-server-start.sh config/server.properties

4. Create topics -

bin/kafka-topics.sh --create --partitions 1 --replication-factor 1 --topic trump --bootstrap-server localhost:9092

5. Run the pyspark producer code in new terminal -

python Kafka_producer_pyspark.py

7. Start kafka consumer in new terminal-

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic trump --from-beginning

8. Now run the pyspark streaming code

python kafka-pyspark-twitter-streaming.py