If you are looking for the smartest data scientists to help you with a project, the go-to place is Kaggle.com, where they “make data science into a sport.”  More than 82,000 different people from 100 different countries all over the world have signed on, and many of them have submitted at least one entry to the more than 250 different contests held since it opened its doors back in 2010.

I have to say that I am a big fan (from afar) of Kaggle, mainly because of my training. One of my hardest but most fun classes when I was an engineering graduate student was a class in building mathematical models, which is what we called data science back in the day.

Each Kaggle problem set is run as a competition, with prizes, deadlines, and rules aplenty. Kaggle takes a percentage cut off the top to administer the contest. It has a blue-chip roster of customers who also conduct privately sponsored contests. “This is because some of their data is too sensitive to be public,” their CEO Anthony Goldbloom told me. Examples include Microsoft, who used Kaggle to improve gesture recognition on the Xbox; NASA, for better dark matter imaging tools; and GE, for more accurate airline arrival time estimation.

Kaggle offers “companies a cost-effective way to harness the cognitive surplus of the world’s best data scientists,” according to their website. “There are some pretty amazing people who compete,” says Goldbloom. “And some enter 80 or more times per contest, devoting a lot of their time.” Even Goldbloom has tried his hand on a few, although he isn’t highly ranked.

Kaggle has been so successful that other contest providers have come online, including India-based CrowdAnalytix.com, Innocentive.com for the life sciences and TunedIT.org mainly for education and research projects. But Kaggle has been around the longest and has the largest talent pool to draw on.

Here are five contests that are somewhat off the beaten path and illustrate the depth and breadth of their reach and influence.

1. Identify the best performing models to predict personality traits based on Twitter usage.

This awarded just $500 but almost 100 teams entered, showing that it isn’t always about the dough. One of the top entries was from Jason Karpeles, a marketing forecaster from Texas who is in the top ten overall of all Kagglers and has participated in 36 different contests. I spoke to him about his accomplishments. Karpeles isn’t your typical data scientist: he has economics degrees and a MBA from Duke and works in marketing. “I don’t know if it is impressive or pathetic the number of contests that I have entered,” he said. He signed up early in Kaggle’s history and admits that he is “obsessed with the site.” What is interesting is his total dollar winnings are miniscule, especially when you compare them to his total time spent on various contests. With one contest that had more than a thousand people entered, he spent many hours working on the problem.

Why enter so many contests? Mainly for his own self-education. “Being in a Kaggle contest is a lot like getting a post-graduate education,” he says. “It is also a good way to sharpen my skills, expand my knowledge and see how to manipulate particular data sets that I don’t often come into contact with,” he said. “I was afraid that I might fall behind in the marketplace because data science is moving so quickly.”

Karpeles also mentioned something that is very interesting. “I am very introverted, and I don’t market myself very well, so this has been a way for me to get out there. Kaggle has been great for me to see how I perform globally across industries.” He tells potential contestants to just “get out and start doing something, just to try it. Don’t be afraid of failure, or your ranking. Experience is the best teacher.”

2. Whale Detection Challenge

During World War II the science of operations research got its start when it was trying to track German submarine movements and keep Allied ships from getting torpedoed. So it is somewhat fitting that a current Kaggle contest, which ends in April, is doing something similar. Only this time instead of German subs they are looking at audio recordings of whales and trying to prevent them from hitting transatlantic ships. Cornell University’s Bioacoustic Research Program has extensive experience in identifying endangered whale species and has deployed a 24/7 buoy network to guide ships from colliding with the last 400 of a particular species of whale. The contest will pay out $10,000 to the best detection algorithm, and so far there are 137 teams hard at work on this contest, including two graduate students who have inevitably called their team Free Willyzx and another team named Herman Melville.

3. Solve the traveling salesman problem to help Santa Claus deliver his presents.

This one paid out $3000 to a Slovakian and was a bit of fun. “Santa needs help choosing the route he takes when delivering presents around the globe. Every year, Santa has to visit every boy and girl on his list.  It’s a tough challenge, and Santa admits he scored a B- on his combinatorial optimization final.” The winner had to find two shortest-distance paths through a route of chimneys.

4.Predict whether a comment posted during a public discussion is
considered insulting to one of the participants.

How many of us have been insulted from a comment posted online? What, are you stupid or something? Exactly. So this contest was to predict when something would be considered insulting to someone else. Or as the contest introduction states, “create a generalizable single-class classifier which could operate in a near real-time mode, scrubbing the filth of the Internet away in one pass.” It wasn’t all that altruistic. Security vendor Impermium sponsored the contest. They were looking to “identify new ways to defend against malicious language and social spam online, and help clean up the web by scrubbing away unwanted obscenities from user-generated content.” Not surprisingly, the competition found out that people tend to be most abusive between 9:00 pm and 10:00 pm.

This was big money, with a prize of $10,000 and had 50 entries. The winner was Vivek Sharma, who has entered numerous Kaggle contests. He and other top finishers were offered a job interview at the company along with the prize purse. While they ultimately did not hire anyone, “the Kaggle competition was useful and we were able to examine many interesting algorithms,” said their PR rep via email.  Their engineering team has a fresh perspective on this problem and “helped ensure against tunnel vision.”

5. Produce an automated scoring algorithm for high school essays.

This competition was held last year and sponsored by the William and Flora Hewlett Foundation, with the top prize of $60,000 going to a team called “SirGuessalot” who could match the average of two human teachers grading high school essays. The team submitted more than 140 different attempts before wining the top prize. “It almost sounds like science fiction,” says Goldbloom.

Maybe some of these will stimulate your imagination and get you to try your hand at one contest. Good luck!

If you want to see into the future of BI, then look no further than the nearest gaming development studio. It isn’t all fun and first-person-shooting. The game developers are the sentinels of a variety of advanced IT techniques and are usually out in front of the general IT population in terms of using big data, real time analytics, and cloud computing, among other areas.

We all know that computer games are big business, with last year’s worldwide sales north of $20 billion and even the subcategory of social games at over $2 billion: compare this to around $8 billion for the average annual US movie ticket box office.

We last looked at how Riot Games is using Hadoop and other NoSQL tools to track its players statistics and improve game play in December, but they are just one of many game studios that are taking technology to new heights.

Gamers have been ahead of the curve in three key areas: rapid changes in computing infrastructure, persistent and more personalized data connections from the cloud, and a long history of using graphical processors (GPUs) to support high performance computing. Let’s look at each of these items in more detail, and why the gamers get them.

Rapid on-demand computing changes

“From an infrastructure perspective, games have a high volume of data points due to user interactions and typically have a unique need for fast response. This makes them very tricky cloud data users,” says Robert Nelson, the CEO of Facebook and mobile game developers Broken Bulb Studios. The company makes use of SoftLayer for their cloud hosting and has several terabytes of data with peak transfer rates over 200 Mbps.

“SoftLayer’s platform has a unique combination of scalability and customizability, which supports the dynamic infrastructure of gaming companies. SoftLayer can provision cloud computing instances in minutes, allowing us to rapidly scale up or down as our needs change,” he said. For example, last year as part of a new game launch they saw 1.4 million players come to their website in a week, up from a few thousand beta users prior to the launch. Some of their games have required SoftLayer to double their infrastructure overnight because of heavy demand. This variation in demand is the wheelhouse of cloud computing, but games do seem to have a higher fluctuation than a traditional IT application.

Another gaming studio, Hothead Games, launched its Big Win series of sports games last year. They saw the number of servers rise from six to 60 on the SoftLayer hosting network. It was all handled with ease. “Our code makes hundreds of millions of database transactions a day. It’s critical to our business that every single one of those works reliably and is super fast,” said Joel DeYoung, director of technology at Hothead Games.

SoftLayer isn’t alone in recognizing this market. Peer 1 Hosting has also worked with some of the world’s largest game developers to deliver their games under wide fluctuations in demand. One launch saw traffic spike to more than a thousand servers, which were automatically provisioned by their managed hosting service. And Joyent has supported one of the largest e-learning games called Quizlet with their hosting services. Thanks to some careful analysis, Quizlet found too many PHP calls and was able to rewrite their code to speed up their operations. They have scaled up from a few hundred beta users several years ago to more than 60 million page views a month today.

Persistence and personalization

“Gaming is a more interesting target market than traditional B2C spaces,” says Brian Stone of Causata, a customer experience management company. “And online gaming is even more so, since it offers unparalleled opportunities for cross-selling and upselling. You are competing with your friends and constantly checking your play statistics, and very involved with your social network. Compare that to an online banking app: the game is a lot more engaging and personal.”

Providing the underlying analysis for this personalization means solid BI support, and games use a variety of tools. IsCool Entertainment can analyze more than a million of its online gamers’ activity and social behavior with Actian’s Vectorwise, a Hadoop analytics engine. This provides data for calculating rewards, generates leader boards and delivers virtual prizes, all to enhance customer engagement and retention.

“Games have a persistent connection with the user and as a result, we get so much more data,” said Reid Tatoris, the CEO of PlayThru.com. The company produces games that are used instead of the annoying Captcha Turing tests to verify that a human is signing up for a website. “Interacting with an app doesn’t give you the how. Is this person’s interaction human? How did you go about completing the task? That is where we try to help.” An example of their game is shown below

PlayThru has gotten lots of insights into personal preferences as a result of deploying their app across 20 million page views. “When you are playing a mobile game, you can get all sorts of information about what the user is doing, where they are located, and how they are interacting with the game in near real-time.” Try getting that kind of insight with a user creating a Word document. As a result of PlayThru’s games, they are seeing submission rates increase by 40 percent over the traditional text-based Captcha applications.

One game that is a champion of personalization is the site Fanhood.com, which connects sports fans with their favorite teams through Facebook. “There is so much content to navigate, we try to focus on what is relevant for a particular fan,” says the company’s CEO Brandon Ramsey. “What’s more, we try to structure it within your Facebook social graph so you can immediately tell which of your friends are fans of teams that your local team is playing this week.” Fanhood uses MongoDB and Cassandra to manage millions of rows of data for each team and fan to create its personal team updates.

Having all this data is a tremendous opportunity if managed properly. Causata is handling an online sports betting site, and can provide all sorts of specifics such as who is opening which emails and the path that a customer takes within the site. “We can then predict the number of bets made and their value, the average duration between bets, and the sports that each visitor is most interested in,” says Stone. Causata builds these models using R and Hadoop.

GPU computing

Finally, there is the notion of using graphics processors for boosting general computing tasks. While this concept isn’t new, even here the gaming industry has been ahead of the curve. Several years ago, a group of Swiss researchers put together a cluster of 200 PS3s to form a primitive supercomputer. While Sony disabled this ability soon afterwards, a number of hosting providers now offer on-demand GPU computing in the cloud, making use of Nvidia graphics processors and specialized Linux operating systems that can take advantage of this increased horsepower. The providers include Amazon Web Services, Peer 1 and SoftLayer, among others. This provides more intensive CPU cycles at lower cost too. One of the Amazon configurations was able to place in the top500 list of the most powerful supercomputers for several years running.

All of these BI tools and advanced computing techniques have brought about what Kimberly Chulis, the CEO of Core Analytics, calls “a new focus on advanced analytics and micro-segmentation to drive player monetization. Game developers and brands have an opportunity to apply these big data analytics techniques to capture rich and varied behavioral and multi-structured game and player data.”

In our story last month, we covered various crowdsourced community methods, looking at the combination of Kaggle contests and Greenplum analytics. There are other examples besides this collaboration where communities are leveraging their own people and data, and some of the most illustrative quite literally are what people are doing to build their own specialized maps.

A map is a powerful data visualization tool: at one glance, you can see trends, clusters of activities and track events. Edward Tufte, data visualization expert, explains how one doctor’s mapping of a Cholera outbreak in 1850’s London was able to track the cause of the epidemic – a bacterium that was transmitted through infected water – to a particular street corner water pump. The good doctor didn’t use Hadoop but shoe leather to figure out where people who were getting sick were getting their water supply. And what is interesting is that this is long before the actual bacterium was discovered in the 1880s.

Enough of the history lesson. Let’s see how crowdmapping and big data science are bringing new ways to visualize data in a more meaningful context.

As one example, last month I was visiting our nation’s capital and noticed that on many streets were racks of bicycles ready to be rented for a few dollars a day. These bikesharing programs are becoming popular in many cities – New York is set to roll its one out sometime soon. The DC program has been in place for about a year, with more than 1600 bikes spread across the city in 175 different locations. Now the operation, called Capital Bikeshare, wants to expand across the Potomac River into the Arlington, Virginia suburbs. So they decided to crowdsource where to put these new locations, and set up this site here to collect the suggestions. On the map you can see locations that the community has suggested and where county planners have recommended, along with of course the locations of the existing stations. You can also leave comments on others’ suggested locations. It is a great idea and one that wouldn’t have been possible just a few years ago, when these mapping tools were expensive or finicky to code up.

Some other successful crowdmaps can be found in some unusual locations. While we traditionally think that you need a lot of computing power and modern data collection methods, crowdmapping is also happening in places where there is little continuous electricity, let alone Internet access, in many places in the third world.

For example, outside of Nairobi Kenya a neighborhood called Kibera was a blank spot on most online maps until a few years ago. Then a bunch of residents decided to map their own community, using online open source mapping tools. It has grown into a complete interactive community project, and as you can see from the map below important locations such as running water and clinics are shown on the map quite accurately.

In another third-world crowdmap is one that is just as essential as the Kibera project. This effort, called Women Under Siege, has been documenting the history of sexual violence attacks in Syria. The site’s creators state on their home page, “We are relying on you to help us discover whether rape and sexual assault are widespread–such evidence can be used to aid the international community in grasping the urgency of what is happening in Syria, and can provide the base for potential future prosecutions. Our goal is to make these atrocities visible, and to gather evidence so that one day justice may be served.” You can filter the reports by the type of attack or neighborhood, and also add your own report to the map.

One of the first community mapping efforts was started by Adrian Holovaty in 2007 in Chicago, mapping city crime reports to the local police precincts. Since then the Everyblock.com site has been purchased by MSNBC and expanded to 18 other cities around the US, including Seattle, DC, and Miami. “Our goal is to help you be a better neighbor, by giving you frequently updated neighborhood news, plus tools to have meaningful conversations with neighbors,” the site’s About page states. You can set up a custom page with your particular neighborhood and get email alerts when crime reports and other hyperlocal news items are posted to the site. The site now pulls together a variety of information besides crime reports, including building permits, restaurant inspections, and local Flickr photos too. This shows the power of the map interface, making this kind of information come alive and meaningful to those who live near these events.

Another effort is called SeeClickFix, which has mobile apps that you can download to your smartphone where citizens who see a problem can report it to their local government and provide detailed information. It was most recently used by various communities that were hard hit by Superstorm Sandy in October, such as this collection of issues from Middletown Conn. area as seen below:

Google put together its own Sandy Crisis Map and displays open gas stations and other data points on it to help storm victims find shelter or resources.

Communities are what you define them, and not always from people living near each other but who share common interests. Our next map is from California’s Napa Valley, home to 900 or so wineries in a few miles juxtaposition. David Smith put this map together that shows you each winery and when they are open, whether appointments are required for tastings, and other information. Once he got this project started, Barry Rowlingson added on to it using R to help with the statistics. What makes this fascinating is this is just a couple of guys who are using open source APIs to build their maps and make it easier to navigate around Napa’s wineries.

Here is another great idea for mapping very perishable data. There are several cities that have implemented real-time transit maps that show you how long you have to wait for your next bus or streetcar. There are dozens of transit systems that are part of NextBus’s website, which mostly focuses on US-based locations. But there are plenty of others: Toronto’s map can be found here, and Helsinki’s transit map can be found here. You can mouse-over the icons on the map to get more details about the particular vehicle. The best thing about these sorts of sites are that they are very simple to use and encourage people to take transit, since they can see quite readily when their next bus or tram will arrive at their stop.

If I have stimulated your mapping appetite, know that there are lots of other crowdmap sites, including Crowdmap.com, ushahidi.com and
Openstreetmap.org, along with efforts from Google. They are all worthy projects, and combine a variety of geo-locating tools with wiki-style commenting features and interfaces to attach programs to extend their utility.

If you want learn more, here is a Web-based tutorial offered by Google’s Mapmaker blog that will show you the simple steps involved in creating your own crowd map and how to find the data to begin your explorations. Here is a similar tutorial for CrowdMap. Good luck with finding your own map to some interesting data relationships.