Collection Chat Data from Twitch Streamers
Twitch chat is a rich and interesting source of text data for NLP projects, but it's not entirely obvious how to get text from their API.
Web scraping would be one option, but fortunately for us Twitch offers a way to stream chat through IRC, which we can easily connect to using Python sockets.
To stream messages from Twitch IRC you need to get a to token for authentication. To do that you need to:
Create a Twitch account
Go to https://twitchapps.com/tmi/ to request an auth token for your Twitch account. You'll need to click "Connect with Twitch" and "Authorize" to produce a token
Your token should look something like oauth:43rip6j6fgio8n5xly1oum1lph8ikl1 (fake for this tutorial).
Including your token, there's five constants we'll define for the connection to a Twitch channel's chat feed:
In [1]:
server = 'irc.chat.twitch.tv'
port = 6667
nickname = '<YOUR_USERNAME>'
token = '<YOUR_TOKEN>'
channel = '#ninja'
channel corresponds to the streamer's name and can be the name of any channel you're interested in. I chose Ninja because he is usually streaming every day for several hours and he has a lot of people watching him and chatting at once. So we rack up tons of text data quickly.
We'll stream one channel at a time to start, but towards the end of the article we'll create a class and command line arguments to watch multiple channels at once. Streaming multiple channels would provide us with some neat real-time data when we have a text processor in place.
Connecting to Twitch with sockets
To establish a connection to Twitch IRC we'll be using Python's socket library. First we need to instantiate a socket:
In [2]:
import socket
sock = socket.socket()
Next we'll connect this socket to Twitch by calling connect() with the server and port we defined above:
In [3]:
sock.connect((server, port))
Once connected, we need to send our token and nickname for authentication, and the channel to connect to over the socket.
With sockets, we need to send() these parameters as encoded strings:
In [4]:
sock.send(f"PASS {token}\n".encode('utf-8'))
sock.send(f"NICK {nickname}\n".encode('utf-8'))
sock.send(f"JOIN {channel}\n".encode('utf-8'))
Out[4]:
12
PASS carries our token, NICK carries our username, and JOIN carries the channel. These terms are actually common among many IRC connections, not just Twitch. So you should be able to use this for other IRC you wish to connect to, but with different values.
Note that we send encoded strings by calling .encode('utf-8'). This encodes the string into bytes which allows it to be sent over the socket.
Receiving channel messages
Now we have successfully connected and can receive responses from the channel we subscribed to. To get a single response we can call .recv() and then decode the message from bytes:
In [6]:
resp = sock.recv(2048).decode('utf-8')
resp
Out[6]:
':spappygram!spappygram@spappygram.tmi.twitch.tv PRIVMSG #ninja :Chat, let Ninja play solos if he wants. His friends can get in contact with him.\r\n'
Note: running this the first time will show a welcome message from Twitch. Run it again to show the first message from the channel.
The 2048 is the buffer size in bytes, or the amount of data to receive. The convention is to use small powers of 2, so 1024, 2048, 4096, etc. Rerunning the above will receive the next message that was pushed to the socket.
If we need to close and/or reopen the socket just use:
In [22]:
#sock.close()
Writing messages to a file
Right now, our socket is being inundated with responses from Twitch but we have two problems:
We need to continuously check for new messages
We want to log the messages as they come in
To fix, we'll use a loop to check for new messages while the socket is open and use Python's logging library to log messages to a file.
First, let's set up a basic logger in Python that will write messages to a file:
In [8]:
import logging
logging.basicConfig(level=logging.DEBUG,
format='%(asctime)s — %(message)s',
datefmt='%Y-%m-%d_%H:%M:%S',
handlers=[logging.FileHandler('chat.log', encoding='utf-8')])
We're setting the log level to DEBUG, which allows all levels of logging to be written to the file. The format is how we want each line to look, which will be the time we recorded the line and message from the channel separated by an em dash. The datefmt is how we want the time portion of the format to be recorded (example below).
Finally, we pass a FileHandler to handlers. We could give it multiple handlers to, for example we could add another handler that prints messages to the console. In this case, we're logging to chat.log, which will be created by the handler. Since we're passing a plain filename without a path, the handler will create this file in the current directory. Later on we'll make this filename dynamic to create separate logs for different channels.
Let's log the response we received earlier to test it out:
In [ ]:
logging.info(resp)
Opening chat.log we can see the first message:
2018-12-10_11:26:40 — :spappygram!spappygram@spappygram.tmi.twitch.tv PRIVMSG #ninja :Chat, let Ninja play solos if he wants. His friends can get in contact with him.
So we have the time the message was logged at the beginning, a double dash separator, and then the message. This format corresponds to the format argument we used in basicConfig.
Later, we'll be parsing these each message and use the time as a piece of data to explore.
Continuous message writing
Now on to continuously checking for new messages in a loop.
When we're connected to the socket, Twitch (and other IRC) will periodically send a keyword — "PING" — to check if you're still using the connection. We want to check for this keyword, and send an appropriate response — "PONG".
One other thing we'll do is parse emojis so they can be written to a file. To do this, we'll use the emoji library that will provide a mapping from emojis to their meaning in words. For example, if a 👍 shows up in a message it'll be converted to :thumbs_up:.
The following is a while loop that will continuously check for new messages from the socket, send a PONG if necessary, and log messages with parsed emojis:
In [ ]:
from emoji import demojize
while True:
resp = sock.recv(2048).decode('utf-8')
if resp.startswith('PING'):
sock.send("PONG\n".encode('utf-8'))
elif len(resp) > 0:
logging.info(demojize(resp))
This will keep running until you stop it. To see the messages in real-time open a new terminal, navigate to the log's location, and run tail -f chat.log.
Parsing logs
Our goal for this section is to parse the chat log into a pandas DataFrame to prepare for analysis
The columns we'd like to have for analysis are:
date and time
sender's username
and the message
We'll need to parse the information from each line, so let's look at an example line again:
In [17]:
msg = '2018-12-10_11:26:40 — :spappygram!spappygram@spappygram.tmi.twitch.tv PRIVMSG #ninja :Chat, let Ninja play solos'
We can see the date is easy to extract since we know the format and can use the datetime library. Let's split it off and parse it:
In [59]:
from datetime import datetime
time_logged = msg.split()[0].strip()
time_logged = datetime.strptime(time_logged, '%Y-%m-%d_%H:%M:%S')
time_logged
Out[59]:
datetime.datetime(2018, 12, 10, 11, 26, 40)
Great! We have a datetime. Let's parse the rest of the message.
Since using an em dash (—, or Right-ALT+0151 on Windows) is sometimes used in chat, we will need to split on it, skip the date, and rejoin with an em dash to ensure the message is the same:
In [61]:
username_message = msg.split('—')[1:]
username_message = '—'.join(username_message).strip()
username_message
Out[61]:
':spappygram!spappygram@spappygram.tmi.twitch.tv PRIVMSG #ninja :Chat, let Ninja play solos'
The message is structure with a username at the beginning, a '#' denoting the channel, and a colon to say where the message begins.
Regex is great for this kind of thing. We have three pieces of info we want to extract from a well-formatted string.
In the regex search below, each parentheses — (.*) — will capture that part of the string:
In [56]:
import re
username, channel, message = re.search(':(.*)\!.*@.*\.tmi\.twitch\.tv PRIVMSG #(.*) :(.*)', username_message).groups()
print(f"Channel: {channel} \nUsername: {username} \nMessage: {message}")
Channel: ninja
Username: spappygram
Message: Chat, let Ninja play solos
Excellent. Now we have each piece parsed. Let's loop through the entire chat log, parse each line like the example line, and create a DataFrame at the end. If you haven't used DataFrames that much, definitely check out our beginners guide to pandas.
Here's it all put together:
In [21]:
import pandas as pd
def get_chat_dataframe(file):
data = []
with open(file, 'r', encoding='utf-8') as f:
lines = f.read().split('\n\n\n')
for line in lines:
try:
time_logged = line.split('—')[0].strip()
time_logged = datetime.strptime(time_logged, '%Y-%m-%d_%H:%M:%S')
username_message = line.split('—')[1:]
username_message = '—'.join(username_message).strip()
username, channel, message = re.search(
':(.*)\!.*@.*\.tmi\.twitch\.tv PRIVMSG #(.*) :(.*)', username_message
).groups()
d = {
'dt': time_logged,
'channel': channel,
'username': username,
'message': message
}
data.append(d)
except Exception:
pass
return pd.DataFrame().from_records(data)
df = get_chat_dataframe('chat.log')
Let's quickly view what we have now:
In [23]:
df.set_index('dt', inplace=True)
print(df.shape)
df.head()
(10591, 3)
Out[23]:
channel message username
dt
2018-12-10 11:26:40 ninja Chat, let Ninja play solos if he wants. His fr... spappygram
2018-12-10 11:27:22 ninja !mouse c_4rn3ge
2018-12-10 11:27:23 ninja !song novaplexutube
2018-12-10 11:27:24 ninja https://www.shazam.com/ nightbot
2018-12-10 11:27:28 ninja Hm glmc12
Just from streaming messages over a couple of hours we have over 10,000 rows in our DataFrame.
Here's a few basic questions I'm particularly interested in:
Which user commented the most during this time period?
Which commands — words that start with ! — were used the most?
What are the most used emotes and emojis?
We'll use this dataset in the next article to explore these questions and more.
From here
We've create a basic script to monitor a single channel on Twitch and successfully parsed the data into a DataFrame.
There's still many improvements that can be made. For example:
Variable logging file for monitoring and comparing different channels
Use Twitch API to retrieve live and popular channels under certain games
Generalize script to stream chat from multiple channels at once
Plus we need to answer those questions mentioned above and more.
There's a ton of interesting things we could do with this data, and so if you have any ideas for interesting questions to ask, leave it in the comments below!
import pandas as pd
from datetime import datetime
import re
def get_chat_dataframe(file):
data = []
with open(file, 'r', encoding='utf-8') as f:
lines = f.read().split('\n\n\n')
for line in lines:
try:
time_logged = line.split('—')[0].strip()
time_logged = datetime.strptime(time_logged, '%Y-%m-%d_%H:%M:%S')
username_message = line.split('—')[1:]
username_message = '—'.join(username_message).strip()
username, channel, message = re.search(
':(.*)\!.*@.*\.tmi\.twitch\.tv PRIVMSG #(.*) :(.*)', username_message
).groups()
d = {
'dt': time_logged,
'channel': channel,
'username': username,
'message': message
}
data.append(d)
except Exception:
pass
return pd.DataFrame().from_records(data)
