Machine Learning, Software Engineering and the (not so) Mysterious Relationship Between the Two

A few links on Machine Learning and Software Engineering. The first one talks about how to explain machine learning to a software engineer and why software professionals need to pay attention to ML. It is both a tool and a bit of a threat.

The second article compares the way we build software and how it differs from building ML applications.

How to Explain Machine Learning to a Software Engineer

Software engineering is about developing programs or tools to automate tasks. Instead of “doing things manually,” we write programs; a program is basically just a machine-readable set of instructions that can be executed by a computer.

Now, machine learning is all about automating automation! Instead of coming up with the rules to automate a task such as e-mail spam filtering ourselves, we feed data to a machine learning algorithm, which figures out these rules all by itself. In this context, “data” shall be representative sample of the problem we want to solve — for example, a set of spam and non-spam e-mails so that the machine learning algorithm can “learn from experience.”

Software Engineering vs Machine Learning Concepts

Not all core concepts from software engineering translate into the machine learning universe. Here are some differences I’ve noticed.

A few thoughts:

  • ML and Software development will co-evolve. Software will be used to build tools for building ML. ML will automate automation. Since software is the current tool for automation, ML will replace many of the software activities. Does this pose a threat to software profession?
  • Do we need a different mindset for building ML apps, compared to building software? What principles of software development can be reused while building ML apps?
  • Can ML help us build better software by improving the building process?
  • The software industry is one of the heaviest users of tools for automating their own work. Various low-level (assembly), high-level (Java, C++, C#) and very high-level (Python, Ruby) languages and their associated tool chains simplified building applications. Now we have tools for not only building software, but debugging, profiling, optimizing,  and managing it.  Is ML going to be another one of these tools? Will these new class of ML apps take software as input and produce better software as output?