Slash BI: How B.I. and Data Make a More Efficient Farm

Business intelligence is not just for big-city businesses anymore. B.I. has come to agribusiness, with farmers and cattle ranchers using many of the same tools found in numerous corporate cubicles. Thanks to everything from sophisticated tractor instruments to automated milking machines, farmers can collect all sorts of operational data in order to improve efficiency and keep production levels high.

You can read the full story over at Slashdot here of several examples of what is going on down on the farm.

SlashBI:Do You Need a Chief Data Officer?

The city of Philadelphia recently hired Mark Headd as their Chief Data Officer, and we are beginning to see this title crop up more frequently since it first came into being several years ago. Why should your organization have one and what do they do that differs from the CIO, CFO or CTO?  

Headd is an interesting choice. He comes from Code for America and ran several open data projects in his career, being a big supporter of numerous civic hackathons and using technology to open up government data to the public. <a href=”http://www.newsworks.org/index.php/local//innovation/42657-meet-mark-headd-philadelphias-first-chief-data-officer”>He is “making sure agencies aren’t reinventing the wheel, running into the same problems,” Headd said to the NewsWorks blog here</a>. “You get someone who can take a holistic view across city government and be strategic about the city’s use of data,” he added.

And the title is taking hold. <a href=”http://www.information-management.com/news/data-steward-chief-data-officer-goldensource-10022664-1.html”>”Over 60 percent of firms surveyed are actively working towards creating specialized data stewards, and eventually Chief Data Officers, for their enterprise,” according to a recent survey by GoldenSource Corporation cited in Information Management Magazine here.</a>

In a presentation by Deepak Bhaskar, the Senior Data Governance Manager at Digital River, he shows that this isn’t so simple, and that managing data across the entire enterprise can take on many dimensions.

Some organizations have begun to even identify their CDOs on their websites, as the <a href=”http://www.fcc.gov/data/chief-data-officers“>Federal Communications Commission does here. </a> They claim to be the first federal agency with the title, and have CDOs in each bureau or specialized office such as wireless or wireline communications. It is a notable effort.

Consulting firm Cap Gemini, among others, says that ideally one person should be focused on the quality, management, governance and the availability of data. It is about treating data as a strategic asset. <a href=”http://www.analytics-magazine.org/septemberoctober-2011/401-chief-data-officer-new-seat-in-the-c-suite”>But most organizations don’t give data the same kind of attention as other corporate assets</a> such as people or expenses, write Rich Cohen and Ara Gopal for Analytics Magazine last year.

That is ironic, because as IT departments move towards bring-your-own-devices and cloud-based computing, they should be more focused on their data. As one corporate IT manager from a large manufacturing firm told me, “Nowadays we don’t own the devices, we don’t own the servers, we don’t own the networks that connect them, and we don’t own any of our apps that run on these devices. All that we have left to derive value from is our data.” How true.

Why consider a CDO now? Several reasons. “Enterprise data is no longer black and white,” says Cohen and Gopal, meaning that data can be found anywhere and everywhere. Indeed, in some cases, IT departments can’t even figure out where their data actually lives. Chris Wolf, an analyst at Gartner, spoke about one firm that he interviewed that made a recent upgrade of its Microsoft Office suite from 2003 to 2010 version. In that upgrade, they lost support for an aging Access 2003 database that they weren’t aware of, yet was used by dozens of employees for their mission-critical data by one department. Oops.

Just think about all your customer interactions today. You can have a collection of text messages, blog entries, social network posts, mobile applications, your own and customer-generated videos, Tweets, emails, and even Instant Messages. How can you track all of this data? Ultimately, one person needs to be in charge and see the entire data landscape.

Another reason is that “recent regulatory reforms have placed an even higher emphasis on data accuracy and the risks associated with the lack of end-to-end visibility,” say Cohen and Gopal.

What is the role of the CDO?

<a href=”http://smartdatacollective.com/brett-stupakevich/52217/top-3-reasons-you-need-chief-data-officer”>Brett Stupakevich, writing in a blog on Smart Data Collective</a>, thinks that CDOs should have three primary responsibilities: data stewardship, or being the chief owner of all enterprise data; data aggregation, or being responsible for “building bridges between business units and creating an enterprise focus for the data”; and communicate data schemas and eliminate any semantic differences among enterprise data. That is a nice way of thinking about it.

Cohen and Gopal add that the CDO should “develop capabilities to measure and predict risk and influence enterprise risk appetite at the executive table.” The CDO should also be watching the top line revenue numbers and bottom line as well.

“The one skill that helps me a lot in my current job is to interact well with people, so I lead my team efforts to better serve the business needs of my C-Level peers,” says Brazilan CDO Mario Faria on a LinkedIn forum. “Technical and business skills are necessary however not sufficient for this job.”

Whom should the CDO report to?

Cohen and Gopal ask a very good question: “Does the CIO report to the CEO or to the CFO; in other words, is the IT organization seen as an integral part of the corporate strategy or it is seen as a cost center that enables day-to-day business operations?” But there could be a larger issue, namely, does the entire C-suite buy into the CDO as another peer? Cohen and Gopal also say, “it can be difficult to secure executive backing unless the CDO initiative is seen as a direct response to a burning business problem.”

Depending on your own organization, a CIO with enough revenue authority – or a CFO with enough IT authority — might be called a CDO. But it isn’t so much the title but what the person is actually responsible for and how much visibility into a corporation’s data footprint they actually have.

<a href=”http://www.linkedin.com/groups/Can-present-CFO-CIO-play-1863121.S.110000183?qid=b6051d7e-21d2-481f-9107-a06726ab77cf&trk=group_most_popular-0-b-ttl&goback=.gmp_1863121″>But not so fast, according to many members of the CDO LinkedIn group who responded to this very question</a>. “Each role (CIO and CDO) has its purpose and everything can not be under control thanks to a single function. And even more in big company,” said one poster.

It seems that CDO has yet to find its way to corporations involved in Big Data. Chief Scientist for Bit.ly Hilary Mason told me “I don’t know that many people with the specific title of Chief Data Officer. It seems like that would be a title found at larger corporations, but I like that it implies that whoever controls the data actually reports directly to the CEO.”

SlashBI: Lessons Learned From Boeing’s Data Analytics Experiments

It wasn’t all that long ago that Boeing didn’t even have an IT department, let alone any processes in place to make use of the massive amount of data it collected to improve its aircraft manufacturing efforts.

But today’s aircraft couldn’t be manufactured without a significant amount of BI and data management. Boeing today moves 60 petabytes around its network, and the company is in the middle of several Big Data pilot projects as well. Let’s see what they’ve been cooking up.

You can read the full article here on Slashdot/BI.

Gartner Catalyst Conference: Tales of IT Daring-Do

I was fortunate enough to attend the Gartner annual Catalyst conference this past week, where I heard some interesting stories from IT managers about their innovative approaches. Here are links to them:

Slashdot: Public Data: Where to Test Your Next Big Data App

When building Big Data apps, you need to conduct a test run with someone else’s data before you put the software into production. Why? Because using an unfamiliar dataset can help illuminate any flaws in your code, perhaps making it easier to test and perfect your underlying algorithms. To that end, there are a number of public data sources freely available for use. Some of them are infamous, such as the Enron email archive used in court hearings about the malfeasance of that company. You can read more of my article that appeared today in Slashdot here.

SlashBI: B.I. and Big Data Can Play Together Nicely

Integrating a Big Data project with a traditional B.I. shop can take a lot of work, but a few suggestions could make the process easier. Here are a few suggestions from the Hadoop Summit conference from last week, including many from Abe Taha, vice president of engineering at KarmaSphere.

A second article in Slashdot about best practices for Hadoop in enterprise deployments can be found here. There are lots of efforts underway to make Hadoop more suitable for large-scale business deployments—including the addition of integral elements such as high availability, referential integrity, failovers, and the like. My story goes into some of the details, including the ability to deploy the MapR version under Amazon’s Web Services (above).

Tell your children to learn Hadoop

I spent some time last week with several vendors and users of Hadoop, the formless data repository that is the current favorite of many dot coms and the darling of the data nerds. It was instructive. Moms and Dads, tell your kids to start learning this technology now. The younger the better.

I still know relatively little about the Hadoop ecosystem, but it is a big tent and getting bigger. To grok it, you have to cast aside several long-held tech assumptions. First, that you know what you are looking for when you build your databases: Hadoop encourages pack rats to store every log entry, every Tweet, every Web transaction, and other Internet flotsam and jetsam. The hope is that one day some user will come with a question that can’t be answered in any way other than to comb through this morass. Who needs to spend months on requirements documents and data dictionaries when we can just shovel our data into a hard drive somewhere? Turns out, a lot of folks.

Think of Hadoop as the ultimate in agile software development: we don’t even know what we are developing at the start of the project, just that we are going to find that proverbial needle in all those zettabytes.

Hadoop also casts aside the notion that we in IT have even the slightest smidgen of control over our “mission critical” infrastructure. It also casts aside that we turn to open source code when we have reached a commodity product class that can support a rich collection of developers. That we need solid n.1 versions after the n.0 release has been debugged and straightened out. Versions which are offered by largish vendors who have inked deals with thousands of customers.

No, no, no and no. The IT crowd isn’t necessarily leading the Hadooping of our networks. Departmental analysts can get their own datasets up and running, although you really need skilled folks who have a handle on the dozen or so helper technologies to really make Hadoop truly useful. And Hadoop is anything but a commodity: there are at least eight different distributions with varying degrees of support and add-ons, including ones from its originators at Yahoo. And the current version? Try something like 0.2. Maybe this is an artifact of the open source movement which loves those decimal points in their release versions. Another company has released its 1.0 version last week, and they have been at it for several years.

And customers? Some of the major Hadoop purveyors have dozens, in some cases close to triple digits. Not exactly impressive, until you run down the list. Yahoo (which began the whole shebang as a way to help its now forlorn search engine) has the largest Hadoop cluster around at more than 42,000 nodes. And I met someone else who has a mere 30-node cluster: he was confident by this time next year he would be storing a petabyte on several hundred nodes. That’s a thousand terabytes, for those that aren’t used to thinking of that part of the metric system. Netflix already has a petabyte of data on their Hadoop cluster, which they run on Amazon’s Web Services. And Twitter, Facebook, eBay and other titans and dot com darlings have similarly large Hadoop installations.

Three years ago I would have told you to teach your kids WordPress, but that seems passé, even quaint now. Now even grade schoollers can set up their own blogs and websites without knowing much code at all, and those who are sufficiently motivated can learn Perl and PHP online. But Hadoop clearly has captured the zeitgeist, or at least a lot of our data, and it poised to gather more of it as time goes on. Lots of firms are hiring too, and the demand is only growing. (James Kobielus, now with IBM, goes into more detail here.)

Cloudera has some great resources to get you started from knowing nothing about it: they claim 12,000 people have watched or participated in their training sessions. You can start your engines here.

ITworld: NoSQL: Breaking free of structured data

As companies use the Web to build new applications, and as the amount of data generated by them increases, they are reaching the limits of traditional relational databases. A set of alternatives, grouped under the umbrella label NoSQL (for not only SQL), has become more popular and a number of notable use cases, including social networking giants Facebook and Twitter, are leading the way in this arena.

You can read my article over at ITworld here.

Using DataCore Sansymphony-V to manage your virtual storage

DataCore’s latest version of its storage networking management tool solves the biggest problem stalling server and desktop virtualization projects. It provides a powerful and graphical mechanism to set up storage pools and provide multipath and continuous data protection for a wide variety of SANs.

Pricing: DataCore-authorized solution providers offer packages starting under $10K for a two-node, high-availability environment.

Requirements: Windows Server 2008 R2

Management console runs on Windows desktop versions from XP SP3 to Windows 7.

You can watch my three minute screencast video review here.