Tinygrad and Micrograd – Deep Neural Networks in Python

One of my favorite daily reads was a blog post called “A few short links”. Unfortunately, it was discontinued. It was great because it used to point to content that you don’t normally find in the tech pubs. Now I depend on hackernews and interesting tweets.

I came upon the book Shape Up, yesterday. Not sure of the source but it is a great addition to the other books from the BaseCamp team. It is a free book and I want to quote the part that drew my attention and a bit that kept it.

This is from the foreword by Jaon Fried.

You’ll often hear people say “execution is everything,” but that’s not quite right. In fact, it’s often quite wrong.

When it comes to project work, and specifically software development, executing something the wrong way can destroy morale, grind teams down, erode trust, crunch gears, and wreck the machinery of long-term progress. So yeah, it’s “done,” but at what cost? By doing, what have we done to ourselves? Do we really have to do that again, over and over month after month, year after year?

I have seen this happen in my own startups and in a few others. An extreme version is known as “Death March”.

I really liked this part.

Over the last few years, there’s been a heightened curiosity about how we work at Basecamp. People often ask us how we get so much done so quickly at such a high level of quality with such a small team. And how we keep our teams together for years and years.

For one, we’re not into waterfall or agile or scrum. For two, we don’t line walls with Post-it notes. For three, we don’t do daily stand ups, design sprints, development sprints, or anything remotely tied to a metaphor that includes being tired and worn out at the end. No backlogs, no Kanban, no velocity tracking, none of that.

You get the drift. There is more of this in this free book. Give it a try and let me know what you think. I already have some interesting reactions from my Twitter friends.

The second great find was a project called tinygrad. From their github page:

For something in between a pytorch and a karpathy/micrograd

This may not be the best deep learning framework, but it is a deep learning framework.

The sub 1000 line core of it is in tinygrad/

Due to its extreme simplicity, it aims to be the easiest framework to add new accelerators to, with support for both inference and training. Support the simple basic ops, and you get SOTA vision models/efficientnet.py and language models/transformer.py models.

See how one good thing leads to another? Now I have two projects to check out – Tinygrad and Micrograd.

Fuelling my recent obsession with everything GPT-3, here is a gem. I may actually end up using some of these ideas to teach kids.

Building games and apps entirely through natural language using OpenAI’s code-davinci model by @AndrewMayne

TL;DR: OpenAI has a new code generating model that’s improved in a number of ways and can handle nearly two times as much text (4,000 tokens.) I built several small games and applications without touching a single line of code. There are limitations, and coding purely by simple text instructions can stretch your imagination, but it’s a huge leap forward and a fun experiment. All the demos can be played with here: https://codepen.io/collection/qOqJqk

There are a few more of these gems on Twitter and HackerNews.