Top Trending Python Projects, Airflow, and Intelligent Agents

Came across Top 10 Trending Python Projects On GitHub: 2020  and could not resist. Looking at a couple of them for our explorations in Natural Language Processing and using Games for Teaching Software.

Apache Airflow

Airflow pipelines are configuration as code (Python), allowing for dynamic pipeline generation.

You can use it for building ML models, transferring data or managing your infrastructure.

We experiment with simple Natural Language Processing tasks – entity extraction, finding keywords from text, language understanding, question/answer generation. Many of them have common steps – gather data from the web, clean the data and turn it into text, parse the data, etc. Trying to create simple pipelines that we can reuse will be of great benefit. Used to love Yahoo Pipes.

Unity Machine Learning Agent Toolkit

The ML-Agents Toolkit is beneficial for both game developers and AI researchers as it provides a central platform where advances in AI can be evaluated on Unity’s rich environments and then made accessible to the wider research and game developer communities.

We include writing simple games as part of teaching Introductory Python. In fact, we start with Turtle module and move them to Pygame Zero. As students move up to write more powerful games, it will be interesting to infuse some intelligence into the games. We are looking at a few options like Micro-worlds and Agent toolkits.

A Technique for Teaching Programming

I like this approach from

“Kotlin Koans is a series of exercises to get you familiar with the Kotlin syntax and some idioms. Each exercise is created as a failing unit test, and your job is to make it pass. Here you can play with Koans online, but the same version of exercises is also available via JetBrains educational plugin right inside IntelliJ IDEA or Android Studio.”

Giving a set of code fragments that fail tests and asking students to fix it is a brilliant idea. In fact, a similar technique used in a version of the book “Learn Python the Hardway”.

ReadLog – On Education

I thought that the main aim of education is to make better citizens. It turns out that citizenship aim is just one of the many aims of education.

Another aim of education may be to improve literacy and the power of critical thinking so that citizens make better decisions – about their life, community, and humanity. And then I stumbled upon this article that describes several aims of education:

1. Knowledge Aim of Education 2. Vocational Aim of Education 3. Character Aim of Education 4. Cultural Aim of Education 5. All-round Development Aim of Education 6. Complete Living Aim of Education 7. Spiritual Aim of Education 8. Citizenship Aim of Education 9. Democratic Aim of Education

If you want to fulfill some of the aims of education, How Would You Reimagine Learning? This IDEO article has a few visions about the future of Learning.

Global Education Market Intelligence shows you various organizations trying to serve the education industry.

I still wonder how I got that job

“Can you draw a circuit diagram of your project on the black board?” asked one of the interview panel members. He had my project report in his hand.

“No Sir”, I replied politely.

There was shocked silence. The 8 panel members exchanged looks. Finally one spoke. “Why not?”.

Sir, first, I did not design that circuit, I copied it from a “Popular Electronics” magazine article. Second, it did not work. The external examiner was kind enough to give me marks for my effort. Finally, and this is the strongest reason, I don’t want to design circuits. I want to work on computers.

Having said my piece, I stood there looking at them. Finally, one of them cleared his throat and asked “What do you know about computers”. All the others leaned back, and looked at me.

“May I go to the board and describe whatever I know?” I asked the gentleman. (Later I found out that his name was Lakshmi Narayan, and he was the second in command at the Computer Division). He nodded. I went to the blackboard, drew an architecture diagram and rambled on. Everything I read two days earlier, from Bartee’s Digital Computer Fundamentals, came tumbling out.

After what seemed like an eternity, I got the first question. “How many types of addressing are there?” Direct and indirect sir. My confidence was slowly leaking out and evaporating into thin air. I was not sure whether it was the right answer. “Have you heard about implied addressing?” was the next question. “No, Sir” I said promptly. I had a clear knowledge of my known unknowns.

Mr. LN smiled gently. He looked at others and that invisible signals flashed among them. You can go Mr. Dorai, he said. “Thank you, Sir”, I said and stumbled out of the room.

I did not know what to make out of the interview. I was there longer than many of the candidates. I spoke most of the time. Is that good or bad? I had no idea. I was OK with either result. I know that if I get in, it will be into the Computer Division. I already had a job.

Around lunch time, my friend Prabhakar came to meet me. He was smiling. “You got in”, he said. I was a bit confused but happy. I did join ECIL in 1972, in February. And I did get into the Computer Division.

I have thought about that interview a lot. I still can not figure out why I was not booted out for not answering any of the questions well. I must have provided some comic relief to the tired interview panel.

A couple of years later, I was writing diagnostic programs for a DEC PDP-11 clone (btw, PDP-11 was one of the most popular mini computers in the world). I moved from hardware to software and that was one of the major forks in my life.

Will Machines Take Our Jobs?

Let the machines do the routine and brute work, we think. Sounds like a reasonable argument. But machines are not capable of replacing humans in every aspect of work. In some cases (like fighting Covid19, it will be nice if machines can completely take over, but they can’t). They can do parts, but not the whole.

The trick is in figuring out the division of labor between the man and the machine. A consensus (for now) seems to be that certain tasks may be automated but and not jobs. We can start with that assumption.

This week I read two articles which talk about this topic.

Working smarter describes what machines are good at and where humans excel. The author recommends personal knowledge mastery as a way to stay ahead of machines using human capabilities like sharing, and network connections.

“There is not much more need for machine-like human work which is routine, standardized, or brute. But certain long-term skills can help us connect with our fellow humans in order to learn and innovate — curiosity, sense-making, cooperation, and novel thinking.”

Working smarter

In 2017, the researchers in DeepMind created AlphaZero, a single program that masters three distinct board games: chess, shogi and Go. To be fair, AlphaZero isn’t the first algorithm created that can beat human in a board game. However, it is the first model that can master three games at the same time, with the performance far better than professional human players as well as all algorithms existed at that time. As a matter of fact, AlphaZero is not only good at these games. People who have witnessed it played described that the deep reinforcement learning algorithm was “toying” with its opponents, “tricking” them, even making one of them (i.e., Stockfish) crashed during the game. This leads some of them to believe that AlphaZero is insightful, and it has actually “understood” the nature of chess, shogi and Go.

Consciousness – An obstacle to the understanding of intelligence.

Sometimes it is difficult to differentiate between intelligence and the perception of intelligence.

LinkLog – Machine Learning for Product Managers

Slowly more and more products are incorporating elements of Machine Learning and AI. Product Managers need to understand what the technologies are about and their impact. These articles ease you gently into the role of a product manager and provide some great tips.

Machine Learning for Product Managers

As Martin Zinkevich’s document on best practices for ML engineering describes, you should not be afraid to ship a product that does not use machine learning.

Many products can collect useful customer feedback using simple baselines; in the document, Martin quotes an example of sorting apps in an app store by download count.

What you need to know about Product Management for AI

Even simple machine learning projects can be difficult, and managing these projects in a real business is much harder than most people realize; that’s why Venture Beat claims 87% of machine learning products never make it into production, and Harvard Business Review says that “The first wave of corporate AI is bound to fail.” Machine learning is not fairy dust you can sprinkle on your existing product.

In “The AI Hierarchy of Needs,” Monica Rogati argues that you can build an AI capability only after you’ve built a solid data infrastructure, including data collection, data storage, data pipelines, data preparation, and traditional analytics.

Start with a simple project, build your infrastructure, learn how to use your data effectively.

Practical Skills for the AI Product Manager

Assuming that the selected machine learning technique is suitable, the product manager will have to make several important decisions about the model.

A product manager must decide whether to refactor the research code, determine the scope of the ML model’s inference engine, decide on model format , ensure that the modeling technique can support the service level agreement of the AI system, and plan for deployment and maintenance.

The foundation of any data product consists of “solid data infrastructure, including data collection, data storage, data pipelines, data preparation, and traditional analytics.” A product manager for this phase prepares the way for putting products into production by building the infrastructure needed to support the design, development, and use of future products.

So what did I learn? Start a Machine Learning product without using ML! And manage expectations carefully (your own and others). But the most essential learning is that we should understand the technologies and their current state of development. Building ML products is a lot of work.


I first encountered the third article, read it, and followed two links and read the other two.

I also tried a small experiment and cheated a bit. I fed each one article to a text summarizer, reviewed these summaries and took portions of them to use as context.

The three quoted blocks above were written by a Machine Learing app!

LinkLog – AI in Education

Links – May 16, 2020

Yesterday I tweeted about “learning to learn” and mentioned that AI would help us learn better. One of my students asked me to share some useful links. Here are a few starting points. I am going to use Education as a proxy for Learning even though they are not exactly the same.

Discovered an International Journal of Artificial Intelligence in Education. Lots of useful resources and they have conferences.

A search for “Artificial Intelligence in Education” got me a bunch of tweets. Here is a link. This is snapshot as of 16th May. If you are interested in these tweet streams, let me know. Will maintain a separate page for these tweets.

The Twitter account of IAIED Society (@IAIEDsociety), if you are interested in tracking AI in Education.

I like it when one of my students (current or past) asks me questions. It helps me learn too. Thanks, Hemamalini. This is the first post to answer a small part of your question. There will be more!

Letters to a Young Programmer – Start Small

If you are just starting out programming, you may want to learn a language, and start exploring by building small applications.

Don’t worry that you don’t know object-oriented analysis/modeling and design. You can learn them when you need them. This is true of algorithms, data structures, object-orientation, software engineering as well.

You can build simple apps with procedural programming and learn to reuse first using functions. As your apps grow in complexity, you appreciate the need for organizing these functions as classes/methods. (You may not even get there if you start out with a functional programming language).

As you deploy your apps and fix bugs, you will come to appreciate testing. If you need to update your app frequently, you will understand the need for automated testing.

As your apps get used more and more, you will understand the need for improving the app. You will learn the need to simplify the code (helping you fix problems faster), and the need for profiling to identify slow performing subsystems.

Learning to build computer application by step-wise refinement is a path to learning and improving. If you need to rework the code, you will learn to re-factor. Automated tests will improve your confidence in refactoring without breaking existing functionality.

Programming is a creative endeavor. It is enjoyable, because you are solving problems big and small. Puzzling over the parts that do not work, trying out things, learning from others, is all part of the experience.

Programming is a journey of self-discovery. The little bits of happiness during the journey add up.

Tips for Conducting Online Workshops

  1. Keep the talking part of sessions between 20-40 minutes.
  2. Have 2 or 3 sessions distributed through the day.
  3. Have people work on things individually or in small groups.
  4. Identify clear (intermediate) deliverables each day.
  5. Let people present/showcase their work in the final session.
  6. Set up a system to track engagement
  7. Create a system of #LearningAnalytics and #TeachingAnalytics (even simple metrics are good to have in the beginning)

There are lots of research papers on #LAK (Learning Analytics and Knowledge).

We did 1 to 6 (above) for our first workshop.

Working on 7 for future events.


1. – a 5-minute video on drop out rates.

2. Research papers (free for now)

Looking for data sets?

If you are trying to build Machine Learning/AI applications, you may need a lot data to test your models. Here are a few of my favorite lists and a search engine for finding data sets.

  1. Kdnuggets Datasets for Data Mining and Data Science
  2. Google Dataset Search is another cool resource. I discovered it while browsing through the KDnuggets page.
  3. Awesome Public Datasets repository is a GitHub repository of resources. Check out many other data sets on GitHub.