Description
TL;DR
1. Do twitter streaming analysis using Spark Streaming
2. Count hashtags and special words
3. Submit your codes and report to Canvas
Abstract:
In this assignment, you will implement a streaming analysis process. The architecture is
as follows. A socket request data from twitter API and send data to spark streaming
process. Spark read real time data and do analysis. It also save temp streaming results
to Google Storage. After the streaming process terminate, it reads the final data from
Google Storage and save it to BigQuery, and then clean the data in Storage.
Fig 1. Twitter streaming architecture
You will be asked to do two analysis tasks based on the tweets you get. And you need
to figure out how to save temp steaming results to google storage.
Tasks:
1. Calculate the accumulated hashtags count sum for 600 seconds and sort it by
descending order of the count. Hashtag usually starts with “#” and followed by a
series of alphanumeric. [40%]
2. Filter the chosen 5 words and calculate the appearance frequency of them every
60 seconds (no overlap). Collect the data for 600 seconds. The words are: ‘data’,
‘spark’, ‘ai’, ‘movie’, ‘good’. [40%]
3. Save results to google BigQuery. You only have to write code to save temp
steaming results to google storage. [20%]
Some important notes:
1. Remember to start the socket first (run twitterHTTPClient.py) and then the
streaming process (run sparkStreaming.py).
2. You don’t need to restart the socket process every time you rerun the
streaming process. When a streaming process stops, the connection socket will
automatically close. And the listening socket will continue listening to the same IP
and Port and wait for the next connection. That is, you can leave
twitterHTTPClient.py running and submit sparkStreaming.py multiple times.
3. Remember to stop the socket program on cluster when you don’t want to stream
data from it.
4. You will get “port already in use” error when you run socket program multiple
times in a short time period, or you try to run multiple socket programs at the
same time.
5. For tasks (2), it is ok if you don’t see all of the words in your results every time.
6. For tasks (3), create a table first before saving results to it.
7. You can stop your cluster instance in Computer Engine page and keep your
cluster. If you stop your instance, you won’t pay too much for having that cluster.
8. You can use Jupyter Notebook, just copy and paste the code. But remember to
create two different notebooks and paste the code of two files in separate
notebooks. The steps to run the code is the same as you submit the file to
dataproc.
Steps:
Step1: Register on Twitter Apps (Please do this step ASAP)
1. Go to https://developer.twitter.com/en/apply-for-access.html and apply for a
twitter developer account.
2. Login at https://apps.twitter.com/
3. Click “Create New App”, fill out the form, and click “Create your Twitter
application”. For the Website URL, you can put any URL here. You only need to
fill in all the required places.
4. In the next page, click on “API keys” tab, and copy your “API key” and “API
secret”.
5. Scroll down and click “Create my access token”, copy your “Access token” and
“Access token secret”.
Step2: Create a cluster and get streaming data
1. Use the following command to create a cluster.
gcloud beta dataproc clusters create \
–optional-components=ANACONDA,JUPYTER \
–image-version=preview –enable-component-gateway \
–metadata ‘PIP_PACKAGES=requests_oauthlib google-cloud-bigquery
tweepy’ \
–metadata gcs-connector-version=1.9.16 \
–metadata bigquery-connector-version=0.13.16 \
–project \
–bucket \
–initialization-actions=gs://dataproc-initialization-actions/pyth
on/pip-install.sh,gs://dataproc-initialization-actions/connectors/
connectors.sh \
–single-node
2. Download start code from canvas.
3. Open twitterHTTPClient.py Replace the Twitter API credential with your own and
run on dataproc. Remember to stop the job after you finish streaming.
4. Open sparkStreaming.py. You can first comment code from line 152 (words=…)
to line 160 (wordCount.pprint()) and from line 178 (saveToBigQuery) to line 179
(saveToBigQuery). And then run it on dataproc.
5. To stream the data:
a. submit twitterHTTPClient.py. (Jupyter Notebook: run the related cells in
one notebook)
b. submit sparkStreaming.py. (Jupyter Notebook: run the related cells in
another notebook)
c. You can test sparkStreaming.py multiple times and leave
twitterHTTPClient.py running
d. Stop twitterHTTPClient.py. You can go to job page of cluster and cancel it.
Or you can use gcloud dataproc jobs kill JOB. (Jupyter Notebook:
stop the cell)
6. You are expected to see a tweet stream printed out like this. Ignore all warnings
while running.
Step3: Analyse Stream data and store data in BigQuery
1. Once you successfully get the stream, you can start writing your own code. The
analytics tasks are listed before. Tasks
2. You can first change the global variable “STREAMTIME” to a smaller value when
testing your algorithm. And then collect the data for 600 seconds.
3. You can first set your streaming window size to a smaller value (less than 60
seconds). After you make sure your code is correct and then you can change it
back to 60 seconds and start collecting the data.
4. To save the data, remember to:
a. Replace the bucket global variable in the code with your own bucket.
b. Create a BigQuery dataset first using
bq mk .
5. You are expected to see the following tables created in your dataset
Homework Submissions
1. A report includes:
a. Screenshot of your code to do all the tasks. Tasks
b. Screenshot of the preview of your data stored in BigQuery. You have to
include two tables: hashtags and wordcount.
2. Your code
Useful links:
https://spark.apache.org/docs/latest/streaming-programming-guide.html
https://docs.python.org/3/library/re.html
