Source: DataValueTalk + Scott Adams

Lessons from Launching a Data Product

Over one month ago, we launched a product called the SocialRank Index. The Index tracks the Twitter activity of the world’s biggest brands, and our goal with it is to build a tool that monitors the “pulse” of how people are engaging with brands online.

Since the Index has launched, we’ve learned myriad lessons on what makes for a truly compelling and useful data product (hint: it’s a lot harder than just pulling some graphs together).

Here are three big things we’ve learned after getting feedback from users, marketers, and data scientists/statisticians.

1. Tweet Annotations: There’s a story to uncover in the data

index-engagement-oscars

When we got the first working version of the Index up and running, we were really excited to see the Engagement graph in action. This graph showed a particular industry’s hourly flow of retweets, mentions, and replies. Our first “Wow, this is really interesting” moment happened the morning after Obama’s State of the Union address. We took a look at the Tech Media Index (which consists of big hitters like the New York Times, Wall Street Journal, and BuzzFeed, among others):

index-State-of-the-union

Around 9pm on Tuesday night, there was an unexpected spike in activity in the Tech Media Index. After some deep deliberation, we concluded the spike was due to the State of the Union, which had begun … at 9pm.

Duh.

But what about situations where there is no immediately recognizable reason for a spike in Twitter activity? Why leave this kind of “aha!” to guesswork and inference? The data clearly is telling a story, and so we should do our best to uncover what that story is.

Thanks to some technical wizardry by our co-founder Michael, the Index now automatically locates and annotates the largest peaks in the engagement graph. Can you guess when the Oscars were?

index-oscars

If you hover over these annotations, you can see the “highest velocity Tweet” at that particular hour. This is our best guess at the Tweet that got shared/retweeted/faved most frequently within that given hour.

According to the Tech Media Index, BuzzFeed and Lady Gaga won the Oscars. This is the Tweet that we captured at the peak of the graph:

index-top-tweet-oscars

Data isn’t very useful without context (see Jen Lowe’s great talk “Data Needs Memory”). Being able to correctly identify which events contributed to an anomalous piece of the data is crucial. Continually looking deeper at the data and trying to articulate which stories are being told (or not told) makes the data itself more insightful and valuable.

We still have a lot more work to do in this regard: our current system isn’t foolproof, doesn’t identify every single peak, and doesn’t answer every relevant question we might have.

For example: looking at the graph above, I notice that the mini-peaks throughout the week tend to fall at around the same time (around 11am). What’s the insight from that? I could deduce from anecdotal evidence that this is the time many tech media outlets push out new stories in order to maximize attention time (when people are about to break for lunch or take a mid-morning break). But of course, that is me just guessing– I would love to have something more to support this inkling.

Lesson learned: keep asking what the data is really telling or not telling us.

2. List View: Most “Big Data problems” are actually “Display problems”

index-global-brands-list-top5

One of the first things you learn the hard way when shipping product is that not everything is as obvious as you think it is. One common piece of feedback we get on the Index is “Wait, what exactly am I looking at?” To us, that is obvious — the graphs and charts show you what the average company in an index looks like on Twitter. But it became very clear after early rounds of feedback that this wasn’t crystal.

We’ve focused a lot of our efforts on the specific metrics to track, the specific types of graphs to plot, and the specific brands and industries to monitor. But the overarching issue of usability remains a sore spot. Our “Big Data” problem is a display problem. It isn’t that we aren’t pulling in enough data. Rather, we aren’t being as clear as we should be with how we show all of this data.

While this is still a major work in progress for us, our feedback from users told us something important about display problems: they happen when people are required to jump through too many cognitive hoops to figure out what’s going on. Our friends would first ask us “What exactly am I looking at?” and then follow up with “Wait, so which companies are in this index? Why can’t I just see how Adidas is doing?”

So today we’re unveiling List View, which breaks down the stats for each and every brand in an index. Here is the List View for the Tech Startups Index, sorted by Total Daily Engagement:

index-list-global-brands-full-view

Now you can not only see which companies are in an index, but also what their specific stats are. We still need to tweak and retool the rest of the Index from a UX/UI standpoint to make everything more obvious. But we feel the “List View” will go a long way in helping people get a more intuitive understanding of what’s going on.

Lesson learned: it’s not that you don’t have enough data, it’s that you’re showing it all wrong.

3. Mean vs. Median: Hunt for a less misleading way to show data

index-media-median-mean
A quick look at the two numbers above should raise some eyebrows. The mean (or average) number of followers for companies in the Tech Media Index is over 1 million. The median number of followers for companies in this index is just under 245,000. The difference between the mean and the median is over 700,000 followers — that is a lot of followers.

When we first began building the Index, the way we processed data made it much more practical to calculate averages (or means), and in the spirit of shipping things fast, we settled on the mean for all of our stats.

But the Index is supposed to display data for the “average company” in an index. And the average company in the Tech Media Index definitely doesn’t have over a million followers. In fact, only 24 out of the 95 brands in this index have over 1 million followers. Due to outliers such as the New York Times and the Wall Street Journal, using the mean to represent what an average company looks like was terribly misleading.

Here is the distribution of followers for brands in the Tech Media Index:

index-media-distribution
@Medium has 1.09mm followers, which is right around what the mean was that we calculated. There are only 23 other companies in this index that have more followers than Medium. Meanwhile, @PCWorld has some 244,000 followers, which also happens to be the median here. Notice how much more representative PCWorld is of the companies in the Tech Media Index than Medium is.

There’s too much misleading and lazy data out there that goes viral and gets morphed into “truth.” And when certain numbers get repeated enough, the desire to check if they actually represent reality grows stale.

We don’t want to contribute to that.

So we’ve switched all of the numbers in the Index to median measurements. We could’ve opted for more advanced statistical maneuvers, but we highly value simplicity, and we also recognize the difference between accuracy and precision.

Obviously this “mean vs. median” discussion is very basic compared to some of the more challenging problems other analytics products might be struggling with. But the lesson holds all the way through, regardless of the type of data problem.

Things to further consider: the size of each index. Right now, each index has about 100 members, but maybe this arbitrarily determined total is skewing the data (example: does the Tech Media Index need 100 members? Or is looking at just the top 50 or top 25 most useful?)

Lesson learned: Be simple, be useful, don’t mislead.

4. Retail & Music Index Launch: You’re building this for customers, not yourself.

index-music-annotation

At first, our process for determining which indexes to launch was internally determined — which ones did we think were cool and awesome and completely relevant to marketers?

And so we launched with indexes for Global Brands, Tech Media, Tech Companies, and Tech Startups. Each of these have a high amount of pop culture value, and journalists for the most part loved seeing them.

But when we started showing these indexes to existing customers, they kept asking whether there would be an index coming out for music or for retail or for their specific industry. Which is when we realized that we should’ve asked the market before building the product in the first place. While our initial indexes displayed data on the world’s “hottest brands,” marketers and strategists are more interested in relevant brands in their own specific industries.

index-retail-list

Today, we are listening to our users and launching the Retail Index and the Music Index. The Retail Index consists of companies in the National Retail Federation’s annual top 100 list (think Amazon, Apple, Walmart, and the rest of the gang). The Music Index is comprised of artists from the Billboard 200. These indexes are equipped with all the updates listed above (List View, Median, and Annotations).

All of the lessons we’ve learned so far are some version of DJ Patil’s advice to “put the human back in the equation.” We’re excited to keep developing the Index into a place where marketers, brand strategists, and community managers can get real with data and start using it more meaningfully. If you are interested in what we are building, please don’t hesitate to shoot us an email at [email protected]

Tech Startups Index

We recently launched a product we’ve been working on for several months — the SocialRank Index. The Index is a tool that tracks the Twitter activity of the world’s biggest brands. Anyone can log in and, without paying a dime, look at how the average company in a particular industry is performing. So far, we have released three industry-specific indexes: the Global Brands Index, Tech Companies Index, and Tech Media Index.

techstartups1

Today, we’re releasing one a lot of people will be interested to look at: the Tech Startups Index. The types of companies that populate this index? Uber, Airbnb, Dropbox, Slack, and more.

This index is based on a list of the world’s most valuable startups, published by Fred Wilson, William Mougayar, and the Wall Street Journal. For simplicity’s sake, we filtered based on three criteria: 1) founded after 2006, 2) founded in the USA, and 3) still privately-owned (hasn’t been acquired or gone public yet).

techstartups3

The companies listed in this index were born into an era where there’s an assumption that building and maintaining a highly engaged online audience is crucial to business.

Time will tell whether a strong social presence is a good predictor of the long-term success of a startup.

techstartups2

We will continue to release more industry-specific indexes to add to the four we now have publicly available. If you are interested in what we are building, please reach out to us ([email protected]) or me ([email protected]) with any ideas of metrics to track or indexes to build.

Tech Media Index

Tech Media Index

Last week, we debuted the SocialRank Index, which tracks the Twitter activity of the world’s biggest brands. So far, we have released the Global Brands Index (Nike, Pepsi, etc) and the Tech Companies Index (Microsoft, IBM, etc). Our plan is to continue to release new industry-specific indexes over the following weeks and months.

techmedia1

Today, we’re releasing a particularly saucy one: the Tech Media Index. This index includes the likes of TechCrunch, WSJ, the New York Times, Buzzfeed, and every other major outlet that covers tech news. The Tech Media Index was inspired by Techmeme’s Leaderboard, which lists the 100 most frequently posted outlets on Techmeme.

techmedia3

These media outlets play a significant role in shaping the landscape of conversations online. So of course the data this index yields will be very interesting.

In a nutshell, tech media rules Twitter.

A few interesting highlights:

  • An American Bias: The average company in the Tech Media Index has over 60% of its followers in the U.S.
  • Tweet-Happy Followers: It is no surprise that the Tech Media Index is dominant on the engagement side. Tech media bests the Tech Companies and Global Brands Indexes from overall Engagement all the way down to each specific type of engagement (RT, @Replies, and Mentions). Nearly 60% of the average tech media outlet’s followers have tweeted something in the past 90 days. This is higher than the Tech Companies Index (53%) and Global Brands Index (54%).
  • High-Profile Fans: The average company in the Tech Media Index has more than 3x the amount of verified followers (2,869) than the average company in the Global Brands Index (941). Moreover, they are beating the other indexes by far in terms of number of followers who have 1k+ followers. The average tech media outlet has over 45,000 followers with over 1,000 followers (Global Brands: 32,000; Tech Companies: 8,400).

techmedia5

We won’t ruin all the fun for you. Log in, check out the Tech Media Index, and if you’re a blogger feeling especially daring, compare yourself to this Index via the Dashboard.

Next Steps

The SocialRank Index is part of our larger mission to build a more sophisticated analytics tool for marketers and brand managers. Instead of looking at wonky numbers like total followers and total “reach” (whatever that means), we want you to be able to really drill down to a granular level, as well as pull back up and view things at a high level.

If you are interested in what we are building, please reach out to us ([email protected] or [email protected]) with any ideas of metrics to track or indexes to build.

Tech Companies Index

A few days ago, we launched the SocialRank Index, which is a real-time look at the Twitter activity of the world’s biggest brands. Over the next few weeks, we will be releasing a slew of other industry-specific indexes.

Today, we begin this process with the release of the Tech Companies Index.

Tech Companies Index

techco1

This index was inspired by PwC’s Global Software Leaders report, which ranks tech companies based on software sales. Some of these are the behemoths that made their big rise in the ‘80s and ‘90s. We’re talking about the Adobes, Intuits, IBMs, Amazons, Googles, Microsofts, and Salesforces of the world.

The Data

techco3

The Tech Companies Index visualizes one big question: how gracefully have these mature tech companies managed their relationships online? Out of the 100 companies on PwC’s original list, 96% have public Twitter accounts. Moreover, 54% of the followers of the average company in this index have tweeted something in the past 90 days. Compare the same number to the Global Brands Index (53%).

There are many more insights to glean, so we invite you to go check out the index and see how these legacy tech companies are doing on Twitter. If you’re feeling more adventurous, visit the Dashboard to compare your own account to this index.

Next Steps

In professional baseball, we have seen the rise of “Moneyball” — great franchises now build winning teams through rigorous data analysis (OPS, PECOTA, etc), paying less attention to headline-grabbing stats like Home Runs or ERAs.

We want to bring that same ethos to brand management, where marketers and strategists work round the clock trying to crack the code on how to develop relationships online and build brand equity. We want the SocialRank Index to become Moneyball for Brands– great marketing teams building winning campaigns through a more sophisticated analysis of social data.

If you are interested in what we are building, please reach out to us ([email protected]) or me ([email protected]) with any ideas of metrics to track or indexes to build.