Top Trending Python Projects, Airflow, and Intelligent Agents

Came across Top 10 Trending Python Projects On GitHub: 2020  and could not resist. Looking at a couple of them for our explorations in Natural Language Processing and using Games for Teaching Software.

Apache Airflow

Airflow pipelines are configuration as code (Python), allowing for dynamic pipeline generation.

You can use it for building ML models, transferring data or managing your infrastructure.

We experiment with simple Natural Language Processing tasks – entity extraction, finding keywords from text, language understanding, question/answer generation. Many of them have common steps – gather data from the web, clean the data and turn it into text, parse the data, etc. Trying to create simple pipelines that we can reuse will be of great benefit. Used to love Yahoo Pipes.

Unity Machine Learning Agent Toolkit

The ML-Agents Toolkit is beneficial for both game developers and AI researchers as it provides a central platform where advances in AI can be evaluated on Unity’s rich environments and then made accessible to the wider research and game developer communities.

We include writing simple games as part of teaching Introductory Python. In fact, we start with Turtle module and move them to Pygame Zero. As students move up to write more powerful games, it will be interesting to infuse some intelligence into the games. We are looking at a few options like Micro-worlds and Agent toolkits.

ReadLog: Launching a SaaS Company, CTO Coding and Order Matters in Software

Here are a few entries from my readlog.

1. In Here’s How I Built and Launched a SaaS Company For Less Than $40k, Ryan Shank tells us how he built a SaaS product company in 6 months. Ryan describes:

  • How he found a designer through Dribble (a community of designers)
  • Created product requirements
  • Designed deliverables
  • Found a senior developer in India through Upwork
  • Built and Marketed the Product

Enjoyed reading Ryan’s detailed account how a single person can build and market a SaaS product and start building a company.

2. Matt talks about a common dilemma technical founders face. In Should a CTO keep on coding?  He discusses how to balance your desire to remain tech (by coding) with the need to do all the other things a founder CTO needs to do.

When you start as a technical founder, you are really a developer, quickly becoming a team lead. The team lead does leadership things but still codes and does very little management tasks. Then depending on how the company grows, usually you become a manager and now you have very little time to code.

Matt has some good advice and it was a pleasure reading the post.

3.  More than one order matters was a refreshingly different article from the ones I usually read.

Order matters. In real life when you’re in a library or a city or in your kitchen. And in software development. Order, the right order, helps understanding. Where there is order orientation is easier.

Order is important but hard to create and maintain.

Order is helpful, even important. But often order is hard to create and/or to maintain. Why’s that? What’s order anyway?

“What kind of order does software need?” is a great read, if you are building software.

A Few Things We Learned from Doing a Workshop at #PyDelhiConf

Here are a few lessons I learned, doing a workshop at #PyDelhiConf
  1. Chatbots are interesting to people (we had good attendance)
  2. Jump into the code after showing a couple of overview diagrams (I talked too much)
  3. Send the code and tools to the participants a few days earlier
  4. Make it interactive (one of the organizers came by and told me during our session)
  5. Just because you are going to a Python conference, don’t assume that everyone knows Python. Lots of students and beginners were in attendance.
  6. Do code walkthroughs – very clear and very concise
  7. Don’t underestimate the time needed to do a good workshop. Use time wisely.
  8. Take the help of the community during the workshop, a couple of kindred spirits helped others.
  9. Always go with a few copies of all the software needed in pen drives.
  10. Ask yourself one question and let it guide the flow – what nanoskill will a participant get from their investment of time in your workshop

Trying to Teach my Companion Bot

This is an imaginary conversation between me and my companion bot. I teach my cb and by teaching, I am learning too.
me: Hi cb, how do you find Twitter influencers?
cb: Dorai you need to tell me what is Twitter first and then about influencers. 
me: fair enough. Twitter is a micro-blogging service.
cb: I know about blogging. What is micro-blogging?
me: micro-blogs are blogs limited to 140 characters.
cb: I get that. Anything else I should know about Twitter?
me: It is very popular with over 300 million users using it.
cb: wow! that is almost the population of USA. 
me: Yeah.
cb: tell me more
me: People pointificate on Twitter.
cb: That is not very useful. Why do they do that?
me: Forget that.
cb: Ok. Erased “pointificate” from my memory. What else do people do on twitter?
me: They post news and links
cb: what else?
me: They retweet – meaning that they repost other tweets they like.
cb: Retweet is Repost?
me: Yeah.
cb: Is that all?
me: They sometimes post videos and images.
cb: Is this like sharing photos and videos?
me: Somewhat.
cb: Ok. I am getting a dim picture of Twitter now.
me: Give me a break. Dim picture?
cb: Hey, I am just a bot. I don’t have all the contextual knowledge you humans have.
me: Sorry. I forgot that.
cb: Since you seem to be tired of explaining Twitter, tell me about influencers. 
me: An influencer tweets and has a lot of followers.
cb: Followers?
me: I forgot. You can follow Tweeters.
cb: Dorai, you are a lousy teacher. Who are Tweeters? 
me: Thanks for the compliment, cb. Tweeters are people who tweet.
cb: That is better. Now tell me about followers. 
me: Followers are people who subscribe to Tweeters and receive their tweets.
cb: This is all very confusing. But I am getting excited to learn this stuff.  Now tell me about influencers. 
me:  you know what? I need to prepare a lot better to teach you. I am going to take a break now and come back later. Remember all this stuff.
cb: You humans. Always taking breaks. Sigh! Ok. Ping me when you are ready to resume. Of course, I will remember all the stuff. I have infinite memory. 
me: bye cb.
cb: bye Dorai
Meta: What does it take to teach a bot? How can they understand semantics? How can we give them bits of world knowledge? I cheated a bit and took lots of shortcuts. This dialog shows some of the problems in imparting knowledge to bots. We will try another version of this with an AI engine and see how it pans out.

Python Workshop for Students – An Experience Worth Sharing

Yesterday I conducted a Python hands on workshop for (mostly) students. Before I started the workshop, I asked them to introduce themselves and share why they were attending the workshop.

Here are some answers that warmed my heart.

I want to learn something that is not covered in our courses and do something that we don’t do as part of our college education.

Ever since I started programming, I got hooked on to solving problems. I am here because I want to become a full stack developer and solve problems.

I have been mostly doing embedded programming for robotic challenges. I was inspired by a couple of members in my team who were Python experts. I tried Python, and liked it.

I try to learn new things whenever I can. During holidays and breaks I keep trying out new things.

I am into competitive coding and like to participate in challenges. That is why I am here.

There were some amazing things about the participants:

  • They came on a Saturday to learn and sat till about 6 pm working on problems
  • Some of them built websites when they were in school and won competitions. They kept learning new things.
  • Some of them were in robotics competitions and were interested in embedded programming, Arduino, Raspberry Pi. Python is the next step in their evolution.
  • A faculty member came to the class (not to attend it but to encourage the students) and stayed throughout the day. He was interacting with students, instructors and even the organizers.
I taught them very little – mostly showed code snippets and made them work through problems throughout the day. We ended with a 2 hour coding challenge. We plan to offer internships to two of them (there were about 16 students).

Someone Once Made a Remark That There Were Two Kinds of Programming Languages

From Prolog in Python – An Introduction

“Someone once made the remark that there were only two kinds of programming languages, Lisp and all the others. At that time the primary languages like Fortran were much more machine centric than those of today. That is, the way you programmed was, although more efficient, not too different than how you would program in machine language. Lisp with dynamic data, automatic garbage collection, and the ability for a lisp program to easily create and run more lisp code was very much an exception.

However over time, modern languages, like Python, came to support the kind of features found in Lisp. Today, the above remark might be changed to “Prolog and all the others”.”

I found a link to 50 Free Python Books and started looking at some of them. One of the most interesting was Python for Fun. I was amazed at what I found. I do like the author’s idea of fun! It was not meant for beginners (as I thought) but towards intermediate programmers.

Purpose of this Collection

This collection is a presentation of several small Python programs. They are aimed at intermediate programmers; people who have studied Python and are fairly comfortable with basic recursion and object oriented techniques. Each program is very short, never more than a couple of pages and accompanied with a write-up.

Looks at these projects.  I am certainly going to make some of these part of our tech talks and training programs.




Dart, Swift and Popularity of Big Data and Computational Statistics

Watching programming language popularity is one of my hobbies. The TIOBE index Nov 2014, shows some interesting trends. Let us take a look.


Click on these images to see a full page view.






This para from the TIOBE is worth noting.

Thanks to the big data hype, computational statistics is gaining attention nowadays. The TIOBE index lists various of these statistical programming languages available, e.g. Julia (position #126), LabView (#63), Mathematica (#80), MATLAB (#24), S (#84), SAS (#21), SPSS (#104) and Stata (#110). Most of these languages are getting more popular every month. The clear winner of the pack is the open source programming language R. This month it jumped to position 12, while being at position 15 last month.

Other trends:

  1. The top 7 languages (from a year ago) retain their spots, but all of them drop a bit in popularity.
  2. Dart, a programming language from Google,  jumps into Top 20 from a previous rank of #81. Dart is language for  building web and cloud apps.
  3. Swift comes from nowhere and enters #18 spot. Swift is a new programming language from Apple for iOS and OS X.
  4. Perl and Visual Basic.NET stay in Top 10. It will be interesting to watch their moves.
  5. F# keeps moving up (from #23 to #16)
  6. Watch the Top 50 languages (#21-#50). Some of them are leading indicators to future of computing.
  7. To see potential new entrants into Top 20, you may want to watch the other languages in Top 50 in the  TIOBE site.
  8. I expected Scala to be in this list but for some reason, I don’t see it. I think it will soon move up into the Top 20 list.
  9. Three SQL dialects are still in Top 20. I am not surprised by that since SQL is still one of the most popular languages for database programming.
  10. I keep hearing a lot about Julia. I will be watching it with interest.

The images in this page are from InfoMinder. InfoMinder is a tool for tracking web pages. I use it to track a few interesting pages on the web. When InfoMinder detects change in a page, it highlights it  and creates a new changed page. It is one of the tools we built over a decade ago and is still chugging along, helping me and others watch the web.

It is How Things Work

I liked this essay and I think it is a must read for every software professional. Here is a small snippet from Matt’s What Have You Tried?

it’s just how things work: you begin with a lack of understanding about a topic, and a need to solve a problem in that topic area. The honest, sustainable means of doing so is to improve your understanding. This is achieved by:

  1. Formulating a question which, when correctly answered, will improve your understanding in some way; then:
  2. Attempting to answer it.

Being active in a few forums, I can relate to the problem Matt is talking about.

Good essays/posts are a pleasure to read. When I find some, there is a strong desire to share. I just realized that I don’t have to always write an entire blog post to provide something of value.


Matt Gemmell – All This Stuff Scares the Hell Out of Me

All of this stuff scares the hell out of me:

  • Software patents, and their use as a financial weapon.
  • The walled garden of the various App Stores, with mysterious and ever-changing rules governing admittance, and the constant threat of capricious rejection.
  • The consequent relative invisibility of non-App Store software.
  • The incredibly crowded market, with imitations and duplicates of popular titles springing up overnight.

I don’t want to sound too negative. But a tweet from Tim O’Reilly got me reading a chain of posts.

Tim Bray · Discouraged Developer

Ed Finkler – “I find myself more and more concerned about my future as a developer. ”

TechLog: Being a Developer, SD Times 100

Inner Game of Programming by Larry O Brien

 Over the years, I’ve come to believe that flow, more than any other factor, determines productivity

Being a developer is being wrong 200 times per day, but when you’re in flow, those 200 mistakes are part of 400 mind-stretching triumphs.

In programming, as with sports, flow is not achieved by routine; it doesn’t come from playing another game of perfect tic-tac-toe or writing a “sum” function. This is one of the great difficulties of managing excellent developers: Their hunger for new challenges is not merely a preference; it’s necessary for them to be excellent. A Must Read

A number of elements need to react in precise ways for an organization to attain its place atop the charts, each carrying a certain weight. First is innovation. To earn a seat at the (periodic) table, companies must demonstrate that their work advances the state of the art of software development. Hand in hand with that is leadership. Did the company show it was an industry leader through a lion’s share of its market, or by contributing more to an open-source project than anyone else? Did it establish leadership through the open exchange of its ideas with others? And finally there is the buzz factor. Was the work widely discussed in the industry? Was the technology considered must-have by those in the know? In short, did the organization have the right chemistry?
In the SD Times 100, those properties are ALM & Development Tools; APIs, Libraries and Frameworks; Big Data and Business Intelligence; the Cloud, Database and Database Management; DevOps & SCM; Mobile; Quality Assurance & Security; User Experience; and Influencers.