Tag Archives: dashboard

SAS and Twitter–how to harness SAS to grab data from Twitter in 2 easy steps

I recently published a post titled, “4 Key Tweeting Attributes of Guy Kawasaki in one Infographic.” I made extensive use of SAS to gather and manipulate the data from Twitter. Turns out, SAS is pretty awesome for this type of work. In this post I’m going to document how to use SAS to gather data from Twitter’s API. My next post on SAS and Twitter will build off of this one and teach you how to gather data about your subject’s followers, find ReTweets, and listen in on conversations. Click here to get that post delivered to your inbox as soon as it’s published.

First off, you might wonder, why do this? Well, successful analyzers of the future will be adept at analyzing all sorts of data, including data from social networks, like Twitter. Also, if you’re looking to market your analytical skills, what hiring manager wouldn’t be impressed with someone who gathered data from Twitter’s API with SAS, then mined, analyzed, and presented the data in a compelling way. Oh, almost forgot, because you’re analyzing a current event (it’s on Twitter, right?) and mentioning Twitter in your post, your analysis will be more search engine friendly, so you’ll likely get a wider and more targeted audience than if you analyzed something outside of the Twitterverse. Some smart analyzers have even been known to analyze Tweets about their target employer and use the analysis to help get themselves hired. On a larger scale, this is almost exactly what Seth Godin has done with Brands in Public.

Before we get started I have to tell you a little about Twitter’s rate limiting policy. Unfortunately, the search area of Twitter’s API doesn’t have a hard rate limit. Rather, Twitter says they allow a rate limit quite a bit higher than their standard 150 hits/hour, but they decline to say how much. Full documentation can be found here, about 1/2 down the page. I have run afoul of the limit before and guess that it’s around 600 hits per hour or more than 30 per minute. When you exceed the unpublished rate, you have to wait between 1-3 hours for your ip address to be allowed to his Twitter again. If you’re just searching for someone’s post, like we’re doing with Guy Kawasaki, you needn’t worry about getting anywhere near Twitter’s rate limit.

Ok, so now let’s get started.

Step 1:
After you figure out what you want to search for (this site is a good start to find trends, and they graph them out for you), you’ll need to plug your search term into the url string that your SAS program will use. If you’re searching for a person, like I did, your string will look like this:

http://search.twitter.com/search.atom?q=from%3Aguykawasaki&rpp=100

The ‘q=from’ tells Twitter that you’re searching for Tweets from a specific user. The ‘%3A’ is url encoding for a ‘:’. And the ‘&rpp’ tells Twitter to return the maximum (100) items per page. You can copy and paste that string into your browser right now and get back some nicely formatted xml representing Guy’s last 100 Tweets.

Step 2: Ok, you know what you’re searching for and how to format the url string to get your results. But Twitter returns a paltry 100 results at a time. You’re a SAS user, you don’t work with 100 record data sets! You want more, so you wrap your code in a macro, key off of Twitter’s page= parameter to get older results, and append the new results to your master dataset. Twitter will generally allow you to pull down 1 week’s worth of search results. The code to do this is located here.

That’s enough to get you started. You now have a SAS data set with lots of Twitter data, including text to mine, dates and times to trend out, and, hopefully, an interesting topic to help show showcase your analytical prowess to your audience.

You can access the full code here.

Don’t forget to come back in about 2 weeks to read my post on how to wrangle and append other data from Twitter to your search dataset. Or, better yet, click here and get all of my posts in your inbox as soon as they’re published.

Customer Insight Dashboard for debt collectors

In today’s economy your collectors need the best customer insights they can get. That means giving them the the right information at the right time in the right format. Forget working off of mainframe green screens, or bolted on front-ends–those tools aren’t made to provide maximum insights to your collectors.

Your collectors need a Customer Insight Dashboard like the one below*. The dashboard shows, in detail, information that your collectors need to maximize their debt collection efforts. Across the top of the dashboard is the customer’s financial trend information and pertinent scores about their risk level and ability to pay you back. Along the left-hand side of the dashboard we provide your collectors with the ability to listen to prior interactions with the customer as well as access information they might use to locate a customer who is avoiding your calls. In addition, your staff could locate customer’s nearby your target customer for aid in tracking them down.

Dashboard

Click image for high resolution version

On the bottom left-hand side of the dashboard your collectors have access to the customer’s most recent credit bureau data. This is a critical component to making sure you get paid first. We’ve parsed the information from the credit bureau to show your collectors which of your customer’s credit card lines they could use to balance transfer their bad debt off of your books and onto your competitors books.

If a picture’s worth a thousand words, an image of the customer’s house or business might be worth $10,000. We find this information very useful to collectors in helping them figure out what makes each customer tick. Your collectors can then use their skills of persuasion and apply the information to help them collect the debt that’s due to you.

Lastly, we show some recent transactions on the customer’s account. Seeing how they spent the money they owe you can also help your collectors be more persuasive in the collections efforts.

A dashboard like the one above could be implemented in your system in a few weeks. The dashboard itself is all done in Excel 2003 (2007 works too) with a $250 add in.

*The data presented in this dashboard are not real. They are provided for illustrated purposes only.

The Best Insights into U.S. unemployment, revealed in this Dashboard

At precisely 8:30am, on the first Friday of each month, the Bureau of Labor Statistics releases its Employment Situation report, the most anticipated report for stock, bond, and currency traders in the world. The report is analyzed by a wide variety of sources like CNN, WSJ, Bloomberg, NYTimes, Economy.com, AP, and MSNBC.

The Economic Situation report is critical because it covers the single most important factor in the world’s economy, employment in the U.S. Put simply, if U.S. consumers are losing their jobs, spending will decrease. And since household spending accounts for more than two-thirds of the U.S.’s economy, any change in spending will have an impact on the rest of the world’s economy.

The Economic Situation report is important for another reason. According to Bernard Baumohl, author of the book, The Secrets of Economic Indicators, “Experts have a difficult time trying to predict the unemployment figures because so little other information is out yet for that month.”

With so much riding on this one report, the Business Intelligence Guru thought it the perfect area to apply his information visualization and analytical skills. After all, the data released by the Bureau of Labor Statistics are pretty lifeless–just a bunch of numbers in twenty different data tables. Trying to identify trends in such raw form data is difficult and time consuming. When high quality info viz is properly applied to such data, however, the fog lifts and insights come shining through.

The BLS tables contain different looks at employment and unemployment like:

  • Employment status by sex and age
  • Employment status by race, sex, and age
  • Employment status by education level
  • Unemployment by reason for unemployment
  • Unemployment by duration of unemployment
  • Average weekly hours of work
  • Average earnings (hourly/weekly) by type of industry
  • Monthly changes in employment

The challenge and opportunity here is to provide a clear, consolidated, and insightful view of related and relevant data from the BLS. The Economic Situation report for July 2009 contains nearly 1,000 words. The data tables in the report add approximately 300 data points to the document. But neither the text nor web version of the report on BLS’ website contain a single graph. It doesn’t take a Business Intelligence Guru to know that this is a ripe opportunity for a well-designed dashboard to shed light on. And so, The Business Intelligence Guru presents you with the “Insights into Unemployment in the United States” dashboard for July 2009.

The Busines Intelligence Guru's Dashboard of U.S. Unemployment

The Business Intelligence Guru's Dashboard of U.S. Unemployment

I intend to update this dashboard the first Friday of each month, shortly after the BLS releases the report, so check back then for timely updates.

Lastly, I’m always on the lookout for ways to improve my work, so feel free to leave suggestions and criticism.

Thanks.

Reblog this post [with Zemanta]

The Best Insights into U.S. unemployment, revealed in this Award Winning Dashboard

At precisely 8:30am, on the first Friday of each month, the Bureau of Labor Statistics releases its Employment Situation report, the most anticipated report for stock, bond, and currency traders in the world. The report is analyzed by a wide variety of sources like CNN, WSJ, Bloomberg, NYTimes, Economy.com, AP, and MSNBC.

The Economic Situation report is critical because it covers the single most important factor in the world’s economy, employment in the U.S. Put simply, if U.S. consumers are losing their jobs, spending will decrease. And since household spending accounts for more than two-thirds of the U.S.’s economy, any change in spending will have an impact on the rest of the world’s economy.

The Economic Situation report is important for another reason. According to Bernard Baumohl, author of the book, The Secrets of Economic Indicators, “Experts have a difficult time trying to predict the unemployment figures because so little other information is out yet for that month.”

With so much riding on this one report, the Business Intelligence Guru thought it the perfect area to apply his information visualization and analytical skills. After all, the data released by the Bureau of Labor Statistics are pretty lifeless–just a bunch of numbers in twenty different data tables. Trying to identify trends in such raw form data is difficult and time consuming. When high quality info viz is properly applied to such data, however, the fog lifts and insights come shining through.

The BLS tables contain different looks at employment and unemployment like:

  • Employment status by sex and age
  • Employment status by race, sex, and age
  • Employment status by education level
  • Unemployment by reason for unemployment
  • Unemployment by duration of unemployment
  • Average weekly hours of work
  • Average earnings (hourly/weekly) by type of industry
  • Monthly changes in employment

The challenge and opportunity here is to provide a clear, consolidated, and insightful view of related and relevant data from the BLS. The Economic Situation report for July 2009 contains nearly 1,000 words. The data tables in the report add approximately 300 data points to the document. But neither the text nor web version of the report on BLS’ website contain a single graph. It doesn’t take a Business Intelligence Guru to know that this is a ripe opportunity for a well-designed dashboard to shed light on. And so, The Business Intelligence Guru presents you with the “Insights into Unemployment in the United States” dashboard for July 2009.

Clicking the image of the dashboard (below) will get you a high-resolution version of it.

Dashboard of U.S. Unemployment

Dashboard of U.S. Unemployment

Lastly, I’m always on the lookout for ways to improve my work, so feel free to leave suggestions and criticism.

Thanks.

–John

Reblog this post [with Zemanta]