Generating Value from Data

Education hasn’t kept up with development in Big Data according to Deng.

“We are facing a huge deficit in people to not only handle big data, but more importantly to have the knowledge and skills to generate value from data — dealing with the non-stop tsunami. How do you aggregate and filter data, how do you present the data, how do you analyze them to gain insights, how do you use the insights to aid decision-making, and then how do you integrate this from an industry point of view into your business process? The whole thing is hugely important for the future.”

Good questions. But education alone may not be answer. How about moving some of the data(base) experts and content analysis experts into this field?

The problem in solely focusing on teaching this stuff is that students do not have enough context about industry needs. While educational institutions teach theory, there is very little industry alignment or effort to understand industry needs. This problem cannot be solved in isolation. It has to be part of a bigger picture and spending a year or so in doing a variety of projects helping the industry.

Do You Believe That Developers Prefer Work to Being Idle?

Some times, a blog is not paragraphs of text. It is just one beautiful idea or a philosophy, you resonate with. When you see something this good, you want to share it and know what others think.

Great thoughts by Scott Chacon on Handy Tools for Optimizing Your Team.

I don’t think it applies just to developers. This may be the assumptions with which we want to welcome any new member into a team, community or organization. We can assume that they have these qualities unless proven wrong.


I have a confession to make. This is not how I always looked at people. It took me a long time to figure out even some of the basic ideas of how to work with people.

Python and Digital Asset Librarians

In one of my Infostreams alerts, there was an entry for a Librarian job. Out of curiosity I clicked on the link and was surprised to see these job skill  requirements (partial) for a Digial Asset’s Librarian.

knowledge of emerging trends in scholarly communications and library and information technologies.

MLS degree (or recognized equivalent) from an ALA-accredited program;
demonstrated large-scale project management expertise;
demonstrated experience with XML, applying metadata standards and schema, and controlled vocabularies;
demonstrated expertise with one or more metadata manipulation and scripting languages (e.g. XSLT, Perl, Python);
demonstrated applied web application development experience, including familiarity with development frameworks (e.g. Ruby on Rails, Django), and application programming language(s) such as Java, PHP, or others;
familiarity with semantic and linked data standards such as RDF and OWL;

Text/Data manipulation, scripting and meta data – are these  the new skills needed in science, engineering and knowledge work?

What Does It Take to Build a Product Eco-system in the Technology Space?

What does it take to build an ecosystem that fosters product development in the technology space? This question came up in our Chennai Open Coffee Club meeting in Jan.

Here are some thoughts.

  1. You need an entrepreneurial spirit. These are people who are willing to risk their careers on building some thing to solve a pressing problem. They should be willing to give up short term gains to pursue their dreams.
  2. You need a culture that reacts to failure well. This will take time to build.
  3. It will help to have a few successful entrepreneurs who did well. That is happening very slowly. It will also help to have people who have done well in their careers and are willing to try something new.
  4. It needs a community. Communities are starting in Chennai – TiE Chennai Startup SIG, Nasscom Emerge, Chennai Open Coffee Club, Chennai Geeks and several other tech groups.
  5. It requires great educational institutions. I don’t think we have enough.
  6. Our Universities have to turn into research engines. There are a handful of institutions that do this a bit. Not enough, though.
  7. We need to have proximity to great people who have built products. That is beginning to happen.
  8. We need a spirit of collaboration. That is beginning to happen too.
  9. We need capital infrastructure. This includes angel investors, venture capitalists, financial institutions and other supporting services. What we have now is nowhere near what we need.
  10. We have the first generation of incubators.  They provide space and infrastructure but not what entrepreneurs really need – guidance in building a business from customer validation to business development. There are a few experiments happening and hopefully their success will inspire others.
  11. A knowledge of the market would help. It is beginning to happen thanks to the Web.
  12. We need lots and lots of conversations on both successes and failures. This is beginning to happen.
  13. We need the spirit of bootstrapping. Out of necessity (due to the lack of funding sources), this is probably going to be the default model of building a business.
  14. We have good experience in building service companies. How can we leverage this to move into products?
  15. We have to find something unique and stuff that matters. This is much harder to come up with but not impossible.

There are encouraging signs. More and more product companies are showing up at the annual Nasscom product conclave every year. Incubators have a chance to play the role of catalysts. I think there is an opportunity here to find some unique problems and solve them.

What are your thoughts?

A Bit Shocking But Very Convincing

This is an amazing view. I kept saying ‘No’ while I was listening to Mark Pagel but some how it seemed very convincing.

A tiny number of ideas can go a long way, as we’ve seen. And the Internet makes that more and more likely. What’s happening is that we might, in fact, be at a time in our history where we’re being domesticated by these great big societal things, such as Facebook and the Internet. We’re being domesticated by them, because fewer and fewer and fewer of us have to be innovators to get by. And so, in the cold calculus of evolution by natural selection, at no greater time in history than ever before, copiers are probably doing better than innovators. Because innovation is extraordinarily hard. My worry is that we could be moving in that direction, towards becoming more and more sort of docile copiers.

I have difficulty in accepting this view. Not sure why, for I cannot argue against this possibility. Infiintely Stupidity? is a video worth watching. This half an hour journey is guaranteed to stimulate you and make you think.

LinkLog: Science’s Most Beautiful Theories

Favorite theories of the world’s most eminent thinkers are as eclectic as science itself. In this fascinating study, they cover:

  • Darwin’s Natural Selection
  • Einstein’s Theory of Relativity
  • The discovery that the conscious, deliberative mind is not the author of important decisions
  • Emergence
  • Pavlovian conditioning
  • Two brain systems
  • Chance shaping Personality
  • People become what they do
  • Group polarization

Link to the article

Meta: After some Googling, I found 2012 : “What is your favorite deep, Elegant or Beautiful Explanation” 

These 189 responses are going to take a while to digest 🙂


Making Learning Programming Fun

Does that sound like an oxymoron? Not as long as we have people who think like this:

The strange history of “Think Python”
(Allen B. Downey)
In January 1999 I was preparing to teach an introductory programming class in Java. I had
taught it three times and I was getting frustrated. The failure rate in the class was too high
and, even for students who succeeded, the overall level of achievement was too low.
One of the problems I saw was the books. They were too big, with too much unnecessary
detail about Java, and not enough high-level guidance about how to program. And they all
suffered from the trap door effect: they would start out easy, proceed gradually, and then
somewhere around Chapter 5 the bottom would fall out. The students would get too much
new material, too fast, and I would spend the rest of the semester picking up the pieces.
Two weeks before the first day of classes, I decided to write my own book. My goals were:
• Keep it short. It is better for students to read 10 pages than not read 50 pages.
• Be careful with vocabulary. I tried to minimize the jargon and define each term at
first use.
• Build gradually. To avoid trap doors, I took the most difficult topics and split them
into a series of small steps.
• Focus on programming, not the programming language. I included the minimum
useful subset of Java and left out the rest.
I needed a title, so on a whim I chose How to Think Like a Computer Scientist

Focus on programming, not the programming language. That is it. That may be the magic formula. But I would add a couple more. Start with some really interesting, engaging problems. It is more difficult than it looks. I like how Moran starts out teaching Java in his Stanford Course.  “Interesting and Engaging” can be different for different people.

For me, it is a tug of war between insulting the intelligence of my students (by being too basic and boring the good ones) and being too obscure to start with.


A Tweet Is…

I keep thinking about my compulsion to tweet. After a bit of reflection, I concluded that to me a tweet is:

  1. A shared (social?) bookmark – something I want to remember with some context
  2. A fleeting thought – something I want to capture but mull over later
  3. An opinion – an opinion fragment, really. It may be mine or something that I resonate with
  4. A question – I may not even state it as a question but sometimes it is really a question
  5. A quote – I don’t always put it in quotes but the link provides a context
  6. An idea – Something I want to throw out there and see what people think
  7. A signal – Mostly to myself but it is kind of public
  8. A note – Lots of times it is just a note (with no opinion or endorsement attached)
  9. Reflection – You know what I mean
  10. Just a way of thinking aloud – Think along with me, it says (sometimes).

What is it to you?

How Can Students Make Money?

FirstDot  powered by NEN, is India’s national mentoring and recognition platform for student startups. I was at FirstDot along with Kiruba, Pravin Sekhar, Narayanan, Soundarya, Karthikeyan Vijaykumar to do un-conference style sessions. Some notes from my session on Innovation and Entrepreneurship.

We started with – What, Why and How of Innovation. We discussed Ideas and Innovation. We covered a variety of topics on why students should do startups, how to do innovation in colleges, how students can make money and a few other orthogonal topics.

What is innovation?

Implementation of an idea and creating value.

How do you start innovation?

Starting with some ideas and building upon them.

Whare are good Ideas?

There are no good or bad ideas. All ideas are good starting points.

Why Innovate?

The urge to improve, to solve problems, and for self-development

Where do ideas come from?

Fulfilling needs, meeting challenges, solving problems, incrementally improvements

Why should students try a  startup?

To understand entrepreneurship by practicing it, for independence, for fun.

We finally ended up with the topic on how students can make money. Here are some ideas from students themselves:

  1.  Teaching other (younger, high school students)
  2. Doing projects for seniors (this was something I did not expect at all)
  3. Creating niche online magazines (example given was a car magazine)
  4. Selling customized T-shirts
  5. Managing Events
  6. Online data entry
  7. Writing documentation
  8. Note taking (for students by students)
  9. Blogging
  10. Helping companies work on promotional activities (marketing help)
  11. Amazon Mechanical Turk (my addition)
Some points from the previous year’s session on how students can make money (from memory)
  1. Create YouTube videos
  2. Create websites using WordPress
  3. Design screen casts
  4. Design services (logo, website templates and stuff)
  5. Write marketing content
I talked a bit about the Value Pyramid where at the base are low skill, low-income jobs moving up to higher income and requiring higher skills. We also talked a bit about Microproducts and Micropreneurship.
I think different moderators used different formats for the unconference sessions and had different levels of interactivity.
One disturbing comment was that many colleges have innovation cells (or equivalents) but they are dormant with literally no activity. Need to find out why.
I enjoyed the session. One thing I would change next time around is to have the students pick themes instead of pre-defining them in the true spirit of unconference.

InfoStream Alert – Python, Django, Pyramid News

Here are a set of links about Python and related products like Django, Pyramid and lots of libraries and tools.

  1. obspy.core 0.6.0o  – a Python framework for seismological observatories.
  2. opencl-for-python 0.4.0  Open CL Python bindings
  3. Clyther 0.3-beta  OpenCL Python integration
  4. selenium 2.16.0  Python bindings for Selenium
  5. bliss 0.1.17  A native Python implementation of the OGF SAGA standard (GFD.90).
  6. fandjango 4.0.1 Fandjango makes it stupidly easy to create Facebook applications with Django.
  7. pyramid_skins 1.0 Templating framework for Pyramid.
  8. stoqdrivers 0.9.14 Python fiscal printer (ECF) drivers
  9. FilterPype 0.3.1 FilterPype is a process-flow pipes-and-filters Python framework.
  10. Flask-DebugToolbar 0.6 A port of the Django debug toolbar to Flask
  11. Astropysics Astrophysics libraries for Python
  12. PyTeVCat 1.1.2 Python wrapper for TeVCat
  13. OpenSRS 0.1.1 Higher level Python interface to the OpenSRS XML API
  14. django-coverage 1.2.2 Django Test Coverage App.
  15. django-user-metrics 0.1 capture metrics for each user
  16. joblib 0.6.0a Lightweight pipelining: using Python functions as pipeline jobs.
  17. django-maintenancemode Django-maintenancemode allows you to temporary shutdown your site for maintenance work
  18. django-configglue 0.6.1 Django commands for managing configglue generated settings
  19. pyperry 1.2.8 Python library for querying and mapping data through generic interfaces (this is a port of the Ruby “perry” library)
  20. python-sld 1.0.6 A simple python library that enables dynamic SLD creation and manipulation.
  21. django-sld 1.0.4 A simple django library that generates SLD documents from geographic models.
  22. django-threaded-messages 0.1.26 User-to-user threaded messaging system (similar to facebook) for Django
  23. django-wysiwyg-forms 0.1.0 A What You See Is What You Get form editor for Django.
  24. sleekxmpp 1.0 SleekXMPP is an elegant Python library for XMPP (aka Jabber, Google Talk, etc).
  25. django-debug-toolbar 0.9.1 A configurable set of panels that display various debug information about the current request/response.
  26. django-envelope 0.4.1 A contact form app for Django
  27. crawler 0.1.2 python crawler.Do you want to receive these alerts in your email? Let us know by sending a message.