From The Book: Open Data Now

From the book – Open Data Now:

In a series of studies led by Harvey Lewis, a research director in Deloitte’s Insight Team, the firm has identified five Open Data business “archetypes”:

  1. Suppliers publish their data as Open Data that can be easily used. While they don’t charge for the data—if they did, it wouldn’t be Open Data—they increase customer loyalty and enhance their reputations by releasing it.
  2. Aggregators collect Open Data, analyze it, and charge for their insights or make money from the data in other ways.
  3. Developers “design, build, and sell web-based, tablet, or smart-phone applications” using Open Data as a free resource.
  4. Enrichers are “typically large, established businesses” that use Open Data to “enhance their existing products and services,” for example by using demographic data to understand their customers better.
  5. Enablers charge companies to make it easier for them to use Open Data.

You can download a sample chapter free.

Think About Shifting Emphasis on Smart Data

From Key Digital Trends for 2014:

Don’t just focus on Big Data; think about shifting emphasis to “Smart Data”.



So what are the jobs that let us:

  • Find a variety of useful data sources and integrate them?
  • Analyze large volume of (unstructured) data?
  • Intelligence Monitoring?
  • Make sense of it all the “smart data”
  • Gain insight and ask critical business questions?

Something to think about.


Where Does the Traffic Come From

Once in a while, I go and check where my blog readers are coming from. I look for Referrers and Search terms. WordPress does a good job of giving me those stats.



The search terms is a bit funny. One of the most popular terms seems to be “Love”. I don’t really write about love at all. I think it got picked up from a title called “I love Python”. I am trying to imagine the reaction of people who type “love” in Google and land up on this blog post.


Tech Trends: Apache Hadoop, Big Data and Analysis

From 7 Ways Businesses are Leveraging Hadoop 

When you hear about Hadoop or Big Data, you probably think of large Internet companies, such as Google or Facebook. However, the use of Hadoop is not limited to large search engines or even Internet marketing.

Look at the companies like Cloudera who are working on helping businesses harness the power of data using Hadoop and other technologies. Look at their customers – the early adopters. Watch the applications being developed, deployed. Here is a hint at the upcoming opportunities.

It is not just big data. It is data of any size – big, medium, small. From I can’t believe you don’t use it

This year, we’re going to see data go from an opaque, untapped, and mystifying asset to a hyper competitive, I-can’t-believe-you-don’t-use-it weapon for businesses. I don’t mean big data; I mean data of any size: big, medium, and small. In fact, it’s not about the amount of data, it’s about the kind of data you have (and, of course, being smart enough to use it). This is all starting to happen because software is being built specifically to analyze lots of data – and it’s no longer cost-prohibitive to use this software, and the insights can fundamentally change the trajectory of your business.

It may be customer behavioral data, it may be support data or data from our market or industry (events, facts, theories, opinions, trends) pouring in. It is going to be the world of analyzers. These are not prediction machines. These are assistants that will fetch you something you can use to verify your hypotheses. They will slice, dice, analyze and offer you different views – in the blink of an eye.

So what happens to traditional data warehouse technologies? IDC survey results point out that Hadoop may be replacing a lot of those. 

Research firm IDC in a recent report shows that companies combine Hadoop with other databases to make big data analysis. A significant proportion of survey respondents said Hadoop is used to replace traditional data warehouse technologies. This ranges from the analysis of raw data, whether operational data, data from different machines or terminals or point-of-sale, or data on customer behavior collected by the e-commerce retail systems.

If you are in the traditional data warehousing, what does this mean to your business? What skills do you need to develop to keep up?

Look at the companies that are leveraging data analysis to run their businesses. These are the early adopters and leading indicators.

Companies like Amazon Web Services, Cloudera, Hortonworks, IBM, Intel, MapR Technologies, Pivotal Software, Twitter, Facebook and others are couching their big data message and providing insight into where the market is headed using Apache Hadoop technology.

Arthur C Clarke: No One Can See Into The Future

No one can see into the future. What I try to do is outline possible “futures” – Possible Futures by Arthur C Clarke.

No one can see into the future. What I try to do is outline possible “futures” – although totally expected inventions or events can render predictions absurd after only a few years. The classic example is the statement, made in the late 1940s, by the then chairman of IBM that the world market for computers was five. I have more than that in my own office.

Perhaps I am in no position to criticise: in 1971 I predicted the first Mars Landing in 1994; now we’ll be lucky if we make it by 2010. On the other hand, I thought I was being wildly optimistic in 1951 by suggesting a mission to the moon in 1978. Neil Armstrong and Buzz Aldrin beat me by almost a decade.

Still, I take pride in the fact that communications satellites are placed exactly where I suggested in 1945, and the name “Clarke Orbit” is often used (if only because it’s easier to say than “geostationary orbit”).

Some of the event listed here, particularly the space missions, are already scheduled. I believe all the other events could happen, although several, I hope, will not. Check me for accuracy – on December 31, 2100.

Somewhere in mid 60s I was reading his 1962 book which got me interested in Space. I always remember a few authors who thought deeply, wrote with passion about possible futures. While I may not believe every one of their predictions or take them seriously, they are triggers for thought. It is nice to think about future – not yours but of humanity and wonder if we live for another 100 years what we will see, experience or even comprehend.

What possible futures are you thinking of?

Stories Give Meaning to Events and Facts

A nice read – 8 Essential Truths About People

People think in stories. Ever since first humans sat around the first campfire, they’ve been enthralled by stories because stories give meaning to events and facts that otherwise would seem random.

Therefore, weave facts into a narrative in order to attract and hold more customers, employees and investors.

“stories give meaning to events and facts”. So why don’t we focus on telling stories and helping students develop those skills as well as part of early education?

Young Innovators

It was such a pleasure and a privilege,  to be part of #ymic (Young Minds Innovation Challenge) conducted by KCG and Enchanter.

Their energy was clearly in evidence in the exhibit hall.  The students were pitching to every one interested in listening and their enthusiasm was infectious.  One of my roles as a judge (along with two others) was to walk around and talk to these young innovators. It was probably two of the most enjoyable days I spent.

Some of them (mostly the school students) were bold and eager. Some of them (mostly college students) were shy. Some were super confident. They had this look – “we are going to change the world”. A kid from Chettinad Vidyashram, was telling me about his product Algenol. He is actually going to US to present his project.

Moms rock.  I saw one patient mom sitting next to her son, handing him things to show to visitors, getting him lunch during the break and encouraging him. She was there both the days of the event, untiring and always smiling. She was proud of her son and she had good reason to.

I think families make such a big difference. I am sure behind every one of these young innovators is a family member – a mom, a dad or a grand parent giving them encouragement and support.

The unsung heroes were the teachers. Those who were present –  from DAV, Chennai Public School were there throughout the event – dedicated, tireless, helping and guiding their students. These students were well organized with their models, their posters and were extremely articulate.

“Sir we added blue tooth support overnight”, said the 7th grader from Chennai Public School. They were demonstrating an Ironing Robot (to press clothes). These kids were amazing. They start their robotics training pretty early (at 4th grade) with Lego Mindstorms. All the kids I met were either 7th or 8th graders. One of the youngest programmer I have met from that school programs in “Embedded C’ and won programming competitions. 

The organizer of the event – Gautam is an innovator, himself. He is also a budding entrepreneur. He is a student from KCG College of Technology and worked on an innovation cell project funded by Department of Science and Technology. People like him are a source of inspiration for many of us who run these innovation programs.

We had to pick a few winners and give them awards. However, I feel that every one the participants, the parents who encouraged them, and the teachers and institutions that helped them are all winners. We just need to find a way to help them all.




Competitive Environment – Pieces of a Puzzle

If run any kind of business, this book  Competitive Intelligence Advantage: How to Minimize Risk, Avoid Surprises, and Grow Your Business in a Changing World is worth a read.

Here is a picture from the book that talks about Competitive Environment (with due acknowledgement) an amazingly clear picture that demystifies Competitive Intelligence.



I will provide a brief summary once I finish the book. Sheena demystifies a lot of confusing terms like Business Intelligence, Competitive Intelligence, Competitor Intelligence, Open Source Intelligence and a whole lot of others.

This book provides  a compelling argument about why every company should pay attention to competitive intelligence. Sheena provokes a lot of reflection about information, intelligence and a context in which you can think about them. .


Data Science Salaries in 2013

“Whether you call it Big Data, data science, or simply analytics, modern businesses see data as a gold mine.” This was in evidence in this salary survey results.

From O’Reilly Data Science Survey ( a free ebook on registration):

By a significant margin, more respondents used SQL than any other tool (71% of respondents, compared to 43% for the next highest ranked tool, R).

The open source tools R and Python, used by 43% and 40% of respondents, respectively, proved more widely used than Excel (used by 36% of respondents).

Salaries positively correlated with the number of tools used by respondents. The average respondent selected 10 tools and had a median income of $100k; those using 15 or more tools had a median salary of $130k.

Two clusters of correlating tool use: one consisting of open source tools (R, Python, Hadoop frameworks, and several scalable machine learning tools), the other consisting of commercial tools such as Excel, MSSQL, Tableau, Oracle RDB, and BusinessObjects.

Respondents who use more tools from the commercial cluster tend to use them in isolation, without many other tools.

Respondents selecting tools from the open source cluster had higher salaries than respondents selecting commercial tools. For example, respondents who selected 6 of the 19 open source tools had a median salary of $130k, while those using 5 of the 13 commercial cluster tools earned a median salary of $90k