# The Rap Network

Note: I originally published this on the zillablog:

Kanye West and Jay-Z, Dr. Dre and Snoop Dogg, Raekwon and Ghostface Killah, Lil Wayne and Drake: some rappers are best known for their  featured appearances and partnerships. After countless debates over the greatest rap duo of all time (still Raekwon and Ghostface, for the record), I was curious about how easily I could quantify the strength of rap affiliations. I also wanted to find out how interconnected the network of rap collaborations really was.

Who collaborates the most with other artists? Which pairs of artists have the most collaborations? And, most importantly, if we modeled the entire network of hip hop artists as an undirected graph, how would Lil Wayne’s PageRank stack up to Kanye West or RZA? These are fascinating questions. Somebody, somewhere woke up today wondering all of these things.

## Modeling the Data

In the “Hip Hop Graph,” artists (producers are considered artists) are represented as nodes, and edges between nodes represent collaborations between artists. It is straightforward to represent edge weights between two artists as the number of collaborations in the corpus of songs. For ease of use in calculations (i.e. shortest paths and minimum spanning trees), taking the inverse of the number of collaborations (1 / # collaborations) is a common technique. The graphs displayed consist of the connections of the 130 most prolific artists.

The final result (one layout of the minimum spanning tree) using networkx (the python package) and Gephi:

Another image of the minimum spanning tree, zoomed in. The size and color of the nodes is correlated with the degree of the node in the MST.

I chose to model the Hip Hop Graph as an undirected graph, in which one collaboration consists of any song in which both artists are featured (either as primary artist, featured artists, or producer). For example, the song “Winter Warz” by Ghostface Killah, featuring Cappadonna, Masta Killa, Raekwon, and U-God, produced by RZA has {6 C 2} = 12 combinations of collaborations.

For any song, there are three categories of artists. Each song has a “primary artist” to whom the track is attributed. Additionally, every song may have multiple featured artists and one or more producers. To generate the graph, I needed to produce a dataset of the specific format:

{String song, String artist, Array featuredArtists, Array producers }

Finding easily accessible music metadata that conforms to this format of attribution is not an easy task. However, certain websites such as Rap Genius maintain metadata on each song in this format. So the song corpus being analyzed is the set of songs registered for artists on Rap Genius.

I wrote a quick and dirty scraper in python for Rap Genius’ html using BeautifulSoup that served as my API. The first function returns the URLs for all the songs attributed to a particular primary artist:

``````from bs4 import BeautifulSoup
import requests
from urlparse import parse_qs, urlsplit

RAPGENIUS_URL = 'http://rap.genius.com'
RAPGENIUS_SEARCH_URL = 'http://genius.com/search'
RAPGENIUS_ARTIST_URL = 'http://genius.com/artists'

def getArtistSongs(url):
soup = BeautifulSoup(requests.get(url).text)
songs = []
for row in soup.findAll('ul', class_='song_list primary_list '):
try:
songs.append(row.find('a').get('href'))
continue
except:
print "couldnt find songs"

for r in soup('div', {'class':'pagination'}):
try:
for i in xrange(last):
i += 4
nurl = RAPGENIUS_URL + upart
print nurl
scheme, netloc, path, query_string, fragment = urlsplit(nurl)
query_params = parse_qs(query_string)
query_params['page'] = [str(i)]
r = requests.get(RAPGENIUS_URL + path, params=query_params)
r = r.text
page = BeautifulSoup(r)
for pageRow in page.find('ul', class_='song_list primary_list '):
if(type(pageRow.find('span'))!=int):
songs.append(pageRow.find('a').get('href'))
except:
print "error somewhere"

return songs``````

The second function extracts the primary artist, featured artists, and producers for any given song URL:

``````def setSong(url):
soup = BeautifulSoup(requests.get(url).text)
featured = []
producers = []
aName = ''

#Extract primary artist name
try:
art =  soup.find('span', class_= 'text_artist')
aName = art.contents[1].text
#print aName
except:
pass

#Extract featured artist names
try:
features = soup.find('span', class_='featured_artists').contents
for f in features:
try:
featured.append(f.text)
except:
pass
except:
pass

#Extract producers' names
try:
features = soup.find('span', class_='producer_artists').contents
for f in features:
try:
producers.append(f.text)
except:
pass
except:
pass

return (aName, featured, producers)``````

## Using Zillabyte to Gather Data

In order to get all the data I needed, I had to pipe these two functions together to get all the metadata for my entire list of songs. This required a number of http requests to Rap Genius. Zillabyte allows easy distributed crawling, using pipe programming to modify streams of tuples. Here are some basic Zillabyte apps that should get you familiar with pipe programming and stream processing. Using Zillabyte, I could fetch about 30,000 songs’ data in 45 minutes with a parallelism of 8.

I seeded my app with a CSV of hip hop artists I was interested in; these are the first 10 rows:

``````Eminem
Outkast
JayZ
WuTangClan
UGK
Drake
Raekwon
MethodMan
2Pac
KanyeWest``````

Here is the code for the Zillabyte app:

``````import zillabyte

#rapgenius.py is the module that houses the two functions above
from rapgenius import *
import csv

def prep(controller):
return

def getsongs(controller, tup):
artist = tup["artist"]
artistUrl = "http://genius.com/artists/" + artist
songs = getMostArtistSongs(artistUrl)
for song in songs:
print song
controller.emit({"song" : song, "artist" : artist})
return

def buildGraph(controller, tup):

song = tup["song"]
artist = tup["artist"]
songData = setSong(song)
print songData
controller.emit({"song" : song, "artist": songData[0], "featuredArtists": songData[1], "producers": songData[2]})

def nt(controller):
with open("rapperlist.csv") as rl:
for line in rl:
controller.emit({"artist" : line})
controller.end_cycle()

app = zillabyte.app(name = "pygenius")
app.source(name="raplist", next_tuple = nt, end_cycle_policy="explicit")\
.each(execute = getsongs)\
.each(execute = buildGraph)\
.sink(name = "zbrap", columns = [{"song":"string"}, {"artist":"string"}, {"featuredArtists":"array"}, {"producers":"array"}])``````

This app uses the python syntax with my seeded list of artists as the custom source. Zillabyte’s pipe programming paradigm applies a series of transformations to a stream of tuples. This app uses the following workflow:

source: list of artists => each: list of urls of all available songs for every artist => each: extract artist, features, and producers metadata for each song’s url => sink: emit to CSV

An example row of the sunk CSV:

“3070”,”http://rap.genius.com/Kanye-west-mercy-lyrics”,”Kanye West”,”["2 Chainz"\,"Big Sean"\,"Pusha T"]”,”["Hudson Mohawke"\,"Kanye West"\,"Lifted"\,"Mike Dean"\,"Mike WiLL Made It"]”,”2014-10-21 19:51:36”,”1”,””

## Visualization

The fully connected graph has so many strongly connected components that distinguishing between rendered edges becomes extremely difficult. An excellent and simple method to limit the number of rendered edges is to build a minimum spanning tree from the fully connected graph. Keeping my workflow in python, I used an extracting script and networkx on the output CSV to generate the graph. Check out the code on

Kanye West’s quadrant of the MST: