There are many good Go programming courses and books. But over the years, my go-to have been "Learn Go with TDD" course and Effective Go book.

Why is the documentation structured this way?

However hard you work on documentation, it won’t work for your software — unless you do it the right way.

There is a secret that needs to be understood in order to write good software documentation: there isn’t one thing called documentation, there are four.

They are: tutorials, how-to guides, explanation and technical reference.

Source: What nobody tells you about documentation

No Silver Bullet—Essence and Accident in Software Engineering — Fred Brooks

In the 1986 paper called “No Silver Bullet”, Fred Brooks observed the productivity of programmers.

Brooks argued that in software development, there were two artificial barriers: accidental complexity and essential complexity.

Accidental complexity is difficulties that software engineers introduce into the design unintentionally.

Essential complexity is the area of complexity that can’t be whittled down. Bottom line, essential complexity is unavoidable.

Let’s see how this complexity argument manifests for things I did at work.

The growing complexity of modern software systems#

Lately, I have been dealing with the challenges of managing complexity in “modern” software engineering.

It’s almost the end of 2019. Looking back, in retrospective as a software engineer, I see the software we created is extremely complex to build and maintain. Example, many websites or web apps are bloated (2 MB just to load a static website?), deployment used to be simple. I wonder how we ended up in this state.

Out of curiosity, I launched myself into research mode and the following is what I’ve learned.

Note that this is a non-exhaustive observation. I’m aware this is missing a lot of things, because I just want to put it down in writing.

Complexity of Web Front-end#

In a nutshell, UI is difficult to build. But, we’re getting better :-)

1. Rich Harris’s on “Rethinking Reactivity, the history and future of reactive programming”

• He delivered the presentation at “You Gotta Love Frontend” - Code Camp 2019 [slide deck]
• A refreshing approach to solve this problem space:
  • Reaching out to the “forgoten” power of compiler and language (yeah, wisdoms back in the good ol’ days and lessons from computer science).
• Rich created Svelte, a UI framework that compiles your UI components into optimal JavaScript at build time, instead of doing what most frameworks do which is lots of unnecessary work at run time.
• FRAMEWORKS ARE NOT TOOLS FOR ORGANISING YOUR CODE, THEY ARE TOOLS FOR ORGANISING YOUR MIND.

2. Pete Hunt’s React talk about “The Secrets of React’s Virtual DOM” at FutureJS 2014 conference

• Some inspiring quotes:
  • “The art of programming is the art of organizing complexity, of mastering multitude and avoiding its bastard chaos as effectively as possible.” — Edsger Dijkstra
  • “All non-trivial abstractions, to some degree, are leaky.” — Joel Spolsky
  • “Simplicity is prerequisite for reliability.” — Edsger Dijkstra
  • “What matters for simplicity is that there’s no interleaving.” - Rich Hickey on definition of simplicity
  • Simple is not equal to familiar
  • “Intellectuals solve problems. Geniuses prevent them.” — Albert Einstein, never used data binding <- :lol:
  • “We can create precisely the same programs we’re creating right now with drastically simpler tools” — Rich Hickey
  • “Programmers know the value of everything and the cost of nothing.” — Alan Perlis
  • “Simplicity is the ultimate sophistication.” — Leonardo da Vinci
• In computer science, the expressive power of a language is the breadth of ideas that can be represented and communicated in that language. The more expressive a language is, the greater the variety and quality of ideas it can be used to represent.
• Practical expressivity is a measure of ideas expressible concisely and readily in a language.
• “Don’t trade simplicity for familiarity”.

Complexity of Web Back-end#

• “Backends are needlessly complicated.” - Dark
  • According to their founders, “Dark is a holistic programming language, editor, and infrastructure for building backends without accidental complexity.”
  • My opinion: the idea is good but their execution (and implementation) is not, AFAICT (still waiting for my Beta access). Their idea was inspired by Bret Victor’s work. I get to know about Bret’s work when I discovered Light Table in 2013. My mind was blown away by it, but sadly, it doesn’t take-off.
• Decisions fatigue
  • Minimalistic library vs. batteries-included framework
  • Convention over configuration
  • Example: should we stick to Ruby on Rails/Django or use Express Node.js for building REST-ful API server?
• Configurations “hell” - solved by zero-config tools (i.e. CRA CLI)
• I hear about a new next generation database every other day (funnily, this used to happen a lot in front-end)

Before you go, let’s take a moment to reflect with these quotes:

“If there’s one rule in programming it’s this: there will always be trade-offs.”

“A balancing act”.

A place to keep some of my Go programming flashback.

Timeline#

• Nov 2009: I tweeted about Google releasing their own programming language, Go. ( while I was learning Ruby on Rails :D )

• May 2013: I learned about some Golang early successes in production and tweeted it.

• Oct 2014: I attended the GoSG meetup for the first time and tweeted about it.

• My first serious (and fun) foray into Go programming dated back to Sept 2015. I remember I was using version 1.5.1 for some weekend hacks (hobby projects).

  

These non-fiction books are on my reading list for a very long time.

Before, I don’t read books that much. But when I do, I want to read books that are worth reading.

Software Development#

• The Pragmatic Programmer: From Journeyman to Master - re-read and write summary.
• The Architecture of Open Source Applications
• Shape Up - Stop Running in Circles and Ship Work that Matters by Ryan Singer.

Programming Language#

• The Rust Programming Language (TRPL) book - currently re-reading chapter 15.
• Learn Rust With Entirely Too Many Linked Lists
• Rust 🦀 and WebAssembly 🕸 - plan to re-read (when SIMD and GC feature land?)

Deep Learning#

• fastai library v2 is currently (Sept 2019) in active development. Once it is done, there will be a O’Reilly book about it:

Deep Learning for Coders with fastai and PyTorch: AI Applications Without a PhD.

to be continued…

A new super(hu)man powers born!

I haz a new superman powers, TensorFlow. Lit!

At Singapore second A.I. Day 2019

TF ‘celebrity’, @DynamicWebPaige kicks off AI Day with style (those Keras unicorn) at Google Singapore. It’s full house!

Oh, luckily it’s Saturday. The audience is ready for more BERTology. Dr Martin Andrews talked about XLNet.

Who else had lotsa fun?

• Kotlin is the focus for Android development from now.
• Android Jetpack
  • CameraX
• Android Studio
  • Fixed over 400+ bugs
  • Instant Run is great but not reliable. They rebuilt the foundation of Instant Run.
  • 3.5 Beta launches today
  • New in AS 3.5
    • Memory usage
    • CPU performanc
    • Build speed
    • UI responsiveness
    • Apply changes
    • Project upgrades
    • Gradle sync
    • Deployment flow
    • https://g.co/androidstudio/preview
• Google Assistant
  • Content Creators
    • HowTo Template
  • App developers
    • App Actions (https://g.co/appactions)
      • Health & fitness
      • Finance
      • Ridesharing
      • Food ordering
    • Interactive Canvas (https://g.co/interactivecanvas)
      • Games
• Open web
  • Latest improvements to Chrome and their developer tools:
    • Faster experiences
    • Powerful capabilities
    • User trust and safety
  • Chrome 74 startup time - web page loads almost half the time.
    • This speed ups are partly due to V8 - 2X faster JS parsing, 20% less memory usage
  • Image lazy loading

    <img src="io2019ftw.png" alt="Google I/O 2019" loading="lazy" />
    

  • Lighthouse
    • Tool to audit websites
    • Performance budgets
  • PWA
  • User trust and safety
    • Private and secure cookies by default
    • Easy to use privacy controls
    • Anti-fingerprinting protection
  • Keep track the latest features and best practices of the web
    • New website called https://web.dev to help.
  • Chrome OS
    • Linux for Chromebooks.
    • Android Studio is now available with one-click install.
    • All Chromebooks launch this year will be Linux-ready right out of the box.
• Machine Learning
  • Google AI
    • Out of the box - ML Kit
    • Performance - Google Cloud
    • Flexibility - TensorFlow
    • ML Kit
      • Vision
        • Landmark detection
        • Image labeling
        • Barcode scanning
        • Face detection
      • Natural Language
        • Language identification
        • Smart Reply
      • Custom
        • Model Serving
      • New features
        • On-device translation.
        • Object detection and tracking API.
          • Pair with Google Cloud Product Search API.
        • You can get started using these new APIs today through Firebase (https://g.co/mlkit).
    • Google Cloud (https://g.co/cloud/ai)
      • For developer who needs more performance for large ML workloads, Google Cloud provides a complete set of AI tools and this includes Cloud AutoML.
      • Cloud AutoML Tables
      • Cloud AutoML Video Intelligence
      • Cloud TPU Pods beta
    • TensorFlow
      • TensorFlow 2.0 alpha (all about usability)
        • More intuitive APIs
        • Less code
        • Portability
      • TensorFlow.JS
        • Build, train and deploy custom models right in the browser and on the Node.js platform.
        • Build custom models
        • Train new models or retrain existing models
        • Deploy with JavaScript
      • TensorFlow Lite
        • For developers working with on-device platform such as mobile devices and IoT
        • Demo of “Dance Like”, an app by Google. All on device, simultaneously in real-time. The 5 on-device tasks running at the same time:
          • Running two body-part segmentation models
          • Matching the segmentation models
          • Running dynamic time warping
          • Playing a video
          • Encoding a video
        • It’s that cool!?
• Firebase
  • Build your app with fully managed backend:
    • Build better apps
    • Improve app quality
    • Engage your users
  • Firebase + ML Kit make it easy for you to bring ML to your app regardless of expertise
    • Today we are expanding ML Kit with AutoML Vision Edge
      • Demo
  • Firebase Performance Monitoring for Android, iOS and the web.
• Flutter for web technical preview available today
• 10 years of Google Developer community

The theme for this year is “Building a more helpful Google for everyone”.

• Google Lens
• Duplex on the web
• Google Assistant - 100 GB DL model to 0.5 GB
  • voice is faster than typing (tapping) your phone
• AI and bias, fair for everyone
  • Zebra model + TCAV
• Data Privacy & Security
  • Privacy
    • privacy controls
      • recent activities
      • auto delete control
    • 1-tap access to Google products
    • incognito mode in Maps (bringing to Chrome, YT, Search this year)
  • Security
    • Android phone as Security Key - launching today in Android 7.0+
• Federated Learning
  • Global Model
    • E.g: Gboard
• People with disabilities
  • Google Live Transcribe
    • Team: Dimitri Kanevsky & Chet Gnegy
  • Live Caption
    • TODO: checkout the Android sessions how they do it
  • Live Relay
  • Project Euphonia (https://g.co/euphonia)
    • Dimitri - speech stutter, ALS
    • The work of Shanqing Cai
• Android 10.0 (Android Q)
  • What’s coming next:
    • Innovation
      • Foldables
        • Screen continuity
      • 5G
      • On-device Machine Learning
        • Live Caption
      • Dark Theme
    • Security & Privacy
      • Protections
        • Android 9.0
          • Kernel control flow integrity
          • StrongBox
          • Protected confirmation
          • DNS-over-TLS
          • MAC address randomization
          • TLSv3
        • All versions
          • Further investment in hardware based security
          • Sandbox/API hardening
          • Anti-exploitation
      • Google Play Protect
        • Gardner report - highest rating in 26 of 30 security categories
      • Almost 50 features focused on security & privacy
      • Faster security updates
        • Android OS Framework
          • Compatibility, security and privacy (OS modules) updateable directly over-the-air
          • Now this can be updated individually as soon as there are available and without a reboot of the device
    • Digital Wellbeing
      • Last year, they launched:
        • Dashboard
        • App timer
        • Flip to Shhh
        • Wind Down
      • A new mode for Android and it’s called, Focus mode
        • Coming to devices on P and Q this fall
      • Family
        • Parental Controls
  • Q Beta 3 is available on 21 devices that is 12 OEMs
• Google continue to believe that the biggest breakthroughs are happening at the intersection of “AI + Software + Hardware”
  • Welcome to the helpful home.
  • Google Home Hub renamed to Nest Hub
    • Nest Hub Max
    • Smart home controller
    • Video calls with Google Duo
    • Kitchen TV with YouTubeTV
    • Digital photo frame
    • Indoor camera
  • Google Pixel
    • Introduce you to the newest members of the Pixel family, Google Pixel 3a and 3a XL designed to deliver premium features with high performance at the price people would love.
    • They start at just US$399, half the price of typical flagship phone.
    • Pixel 3a can take amazing photo in low light with Night Sight.
    • Portrait mode on both the front and rear cameras
    • Super Res Zoom
    • Access the Google Assistant with Active Edge
    • Call Screen using Google Speech Recognition and NLP to help you filtered out those unwanted calls.
    • Using AR on Google Maps
      • You’re going to see arrows in the real world to tell you where to turn next.
    • Battery life
      • Adaptive battery using Machine Learning to optimize based on how you use your phone.
      • You can get up to 30 hours on a single charge and 7 hours with 15 minutes of charge.
    • Strongest data protection
      • In a recent Gardner report, Pixel scored the highest for built-in security amongst smart phones.
    • Available starting today
      • You can also get it from the Google Store (https://googlestore.com)
      • In 13 markets - Australia, Germany, Italy, Spain, USA, Canada, India, Japan, Taiwan, France, Ireland, Singapore, UK.
• Google AI
  • Jeff Dean talk about research
    • Speech recognition, RNN, BERT
    • All these ML momentum wouldn’t be possible without platform innovation.
    • TensorFlow is the software infrastructure that underlies Google’s work in ML and AI.
    • AI for Social Good
      • Research & engineering
        • Flood forecasting project
      • Building the ecosystem
        • Google AI Impact Challenge

Recently, I often see developers sharing fairly complicated TypeScript code that I couldn’t wrap my mind around easily. I want to understand TypeScript better. So, this post will take a fairly complicated TypeScript example and try to break it down.

interface Array<T> {
  concat(...items: Array<T[] | T>): T[];
  reduce<H>(
    callback: (state: H, element: H, index: number, array: T[]) => H,
    firstState?: H): H;
  ···
}

This is an interface for an Array whose elements are of type T that we have to fill in whenever we use this interface:

• method .concat() has zero or more parameters (defined via the rest operator ...). Each of those parameters has the type T[]|T. That is, it is either an Array of T values or a single T value.
• method .reduce() introduces its own type variable, U. U expresses the fact that the following entities all have the same type (which you don’t need to specify, it is inferred automatically):
  • Parameter state of callback() (which is a function)
  • Result of callback()
  • Optional parameter firstState of .reduce()
  • Result of .reduce()

callback also gets a parameter element whose type has the same type T as the Array elements, a parameter index that is a number and a parameter array with T values.

The explanations above were written after I read and “understand TypeScript’s type notation”. It’s a good post by 2ality.

I also refer to a few learning resources below to help me demystified the previous code:

• The definitive guide to TypeScript and possibly the best TypeScript book. Free and Open Source. (this is the book where I used the second time when I picked up TypeScript seriously in Aug 2018. The first time was when TypeScript 2.0 was released in Sept 2016.)
• A cheatsheet for React users using TS with React for the first (or nth!) time

What is everyone’s go-to web stack today in 2019?

If you plan to quickly put together a simple web app or website with React.JS.

YMMV depending on what you’re doing, but the following is a good bet if you want to make the project accessible to other developer, and it doesn’t need to quickly scale.

Front-end#

Use React.JS with TypeScript.

Create React App now makes it dead easy. Just run this command:

create-react-app myapp --typescript

The general consensus is, do not use Redux until you know React well. You might not need it. If you do need it, use boilerplate/starter kit such as redux-starter-kit, offered by the core Redux team.

Back-end#

Use Node.js (if you are comfortable with JavaScript) or Rails (if you prefer Ruby or Django if you have Python skill).

If you go with Node.js:

• Express is a good starting point if you need a simple abstraction around Node HTTP/web server
• Sequelize for SQL ORM library
• Async/await has made things much easier on the Node.js side though. Express doesn’t still have async support by default, but there are middlewares and native support is coming with Express v5.

Database#

• Use relational databases (PostgreSQL) by default for all data (primary/important and secondary/meta-data)
• If possible, avoid NoSQL database such as MongoDB. I wouldn’t want to put my important transactional data such as subscription and e-commerce sales data there.

Caching#

Redis.

Message queue#

RabbitMQ or Kafka.

Authentication and Authorization#

• For authentication on the web, just use cookies (do not use JWT).
• Put nginx (app proxy) in front of Node.js and your static files (React, etc) from the same domain so that you do not have to use CORS, JWT, etc. So, an easy choice is ‘mydomain.com/static’ serves your React bundle while the cookies are on ‘mydomain.com’

Tips#

Tooling tips:

• Use Visual Studio (VS) Code if using JavaScript.
• Use Prettier (for JavaScript, CSS, HTML, and relevant VS Code extension) and Black ( for Python) for automated code formatting.
• Use Jest and VS Code’s Jest extension (Orta’s) for automated tests within the editor.
• For deployment, I roll my own using Docker container. As a one-man shop, I try to minimize sysadmin/DevOps works by off-loading as much as I can to the Cloud service providers by going the managed/zero-ops direction.
• CodeSandbox.io, REPL.it etc are amazing for testing out various stuff without downloading stuff. You can get a React/Vue/whatever environment within seconds that will give you a public url for your app.
• json-server in NPM will get you a mock REST API with GET/POST/etc support within seconds from a JSON file.

What did you learn the hard way?

Worse is better#

To put it a bit more optimistically—usable now is better than perfect later.

I have found that, if I disappear behind a curtain and spend a long time trying to make something really well-polished and feature rich, that just gives the user a lot of time to build up their expectations, and also to get frustrated by the delay.

By the time you ship, they will be actively looking for ways to find fault. When I YAGNI my way into a 80% or 90% solution and turn it around quickly, though, more often than not, they will initially just be impressed at how quickly I was able to help them. Requests for changes will come, but they are generally small, so it’s usually relatively easy to turn those around quickly as well.

Why business fail#

Why did your business fail and what did you learn?

1. Family run business

• Lesson 1 is, never use your cultural beliefs in business. Stick to contracts.
• Lesson 2 is, don’t just trust family.

2. I chased and successfully won a huge customer for my small and fledgling startup. I chased and successfully won a sole service contract for a key part of their business process. I allowed a credit situation with them to grow over the course of 3 months while I allowed them to have 60 day terms. And then, they went out of business and left me holding the bag with $100k in unpaid AR after I spent $90k generating that AR with them.

• The lesson is never trust the size of a company as sufficient reasoning that they can and will pay their bills.

3. Big-corp CEOs don’t make for good startup CEOs 99% of the time.

This is a note to myself.

1. Everything in software development has already been invented. People just keep rediscovering it and pretending they invented it. Whatever you think is so cool and new, was copied from Smalltalk, or HAKMEM, or Ivan Sutherland, or Douglas Engelbart, or early IBM, or maybe Bell Labs.

2. Don’t trust the compiler. Don’t trust the tools. Don’t trust the documentation. Don’t trust yourself.

3. We don’t need any more computer languages. Still, you will run right off and invent another one. Let me guess, your amazing new language uses IEEE-754 math and fixed-precision integers. Your amazing new language is broken.

4. Maintaining code is harder than writing it. Writing lots and lots of new code can be a mark of laziness.

5. You have been taught to program as though memory, processor time, and network bandwidth are all free and infinite. It isn’t, it isn’t, and it isn’t. Read the rest of Knuth’s paragraph about premature optimization.

6. You’re going to forget what your code does in a few months. Make it ridiculously easy to read.

7. Sometimes, all you need is a one-liner in sed. (KISS principle)

8. Beware of programmers who speak in absolutes, such as My Method Is Always Better Than Yours. Programming is an art, not a religion.

9. If you know you will do a fixed sequence of steps more than ten times, automate it.

10. Backing it up is one thing. Restoring it is another.

11. Just because it works on your machine does not mean there is not a bug. -Piers Sutton

12. Wait for the point-one release of development tools before installing them. Let other people be guinea pigs.

13. Good programmers write good code. Great programmers write no code. Zen programmers delete code.

14. No matter how many managers are screaming at you, the first step is to reliably replicate the bug.

15. Sooner or later, you will meet some older man who has been around for a while. At some point, this man will lecture you about the Laws of Programming. Ignore this man entirely.

Source: Quora

Related HN discussion.

By Sanyam Bhutani and Cedric Chee

(Part 1 of the blog series to document the creation of another “Not Hot Dog” App using PyTorch.)

This post serves as an introduction to Transfer Learning as well as a few key points that I’ve learnt are good for building a baseline for a Machine Learning model. I’ll also introduce a crazy idea of building and porting an idea to an app with PyTorch as the main framework. This will be a 3 part series to document and share our success or failure of a re-build of the “Not Hot Dog App”

This is Part 1 of the series where I’ll share how we’re (We refers to a group of students from the FastAI community) building a tiny ML app.

We’re replicating results by Tim Anglade, who had built the original app for the TV Series: Silicon Valley.

Fun fact: Tim is also a FastAI Student so we’re confident that we’d be able to achieve a good result.

Here is our three-step game plan:

• Build a good prototype model baseline. (Prototype)
• Port the baseline to a mobile-friendly architecture. (Production ready model)
• Port the architecture into an app. (Put the architecture to production)

For Step 1, we’re using FastAI to build a classifier baseline.

This is the easiest part since it’s just fine-tuning the model for a quick few steps and that should provide us with a solid baseline result.

So, the first question is why are we trying to re-build an app in a framework that’s not the best choice for Mobile deployment?

We’re actually trying to use this experiment as a testing ground for an “app idea”. Now, since FastAI is my favorite framework and PyTorch follows automatically, this experiment will help us and hopefully, you, understand how hard/easy/wise/stupid it is to try and put a PyTorch model into a mobile environment.

The other reason for doing it, what better way to kill a weekend than build a HotDog or NotHotDog App?

The Baseline#

Dataset#

For our little experiment, we’ve decided to use this dataset curated by Dan Becker, hosted at kaggle.

After basic inspection, the dataset looks like a good start and has 250 images per-label, which would allow us to perform transfer learning on these images.

Transfer Learning#

The best and quickest way to achieve a baseline here is to simply use a “pre-trained” network and then “fine-tune” it to our dataset. The images are derived or similar to ImageNet so “Fine-tuning” should work well.

What is a Pre-trained Network?

Let’s for the sake of explanation consider our “Model” to be a three-year-old kid’s brain.

We have a smart and curious kid-we’re teaching him how to recognize objects in images. The kid here is the model and the task is ImageNet Challenge. The expert refers to the research groups that train the model to perform well on the Leader board.

What is Fine Tuning?

Now we have our “educated kid”, who is good at ImageNet.

Now, We give him our simple task: Name if the image is a not hot dog image.

Fine tuning: The process of taking our “smart kid” or model that performs well at ImageNet Challenge and then re-training him or giving it a quick training to a new category of images that are similar to what he is good at.

Why Fine-Tuning?

• Faster: Its faster than training a Neural net from scratch.

  If some expert has spent their time to train the smart kid. We can just teach the smart kid a new task that he is already good at.

• Efficient:

  As previously mentioned, the kid is smart. Smart in the sense that he is good at the ImageNet Challenge. So obviously, he would do good on similar challenges. Or at least we’d hope so.

Transfer Learning in FastAI#

This section will just be a quick walkthrough of performing Transfer Learning in FastAI for our use-case.

This is just an attempt to explain what is happening here. For a much clear explanation, please check out our Guru’s (Jeremy Howard’s) fast.ai MOOC’s V3 which comes out in 2019.

For our “baseline”, we’re testing a kid named ResNet34

What is a baseline result?

When you’re working with a ML idea, it’s easy to get lost into the complications and keep building without having a good result until a long time.

The approach suggested by Jeremy, in the fast.ai ML MOOC: build a baseline as fast as possible, and then build on top of it.

The baseline result is the fastest result of an acceptable “accuracy” for our experiment.

Accuracy here refers to how accurately the “kid” (Model) recognises the given image as not being a hot dog.

NotHotDog Baseline

• Since the data is already separated into two folders, fast.ai supports this “ImageNet” like data and we can create our data model right away.

• We download our kid’s brain: ResNet34-Pretrained Weights.

• We let the model run for 38 seconds and finally we have a model with 87% accuracy.

• Why is this important?

  In under a day-we have an idea of what should be a good or possible accuracy given our problem.

• How good is 87%?

• The first and third images and final images are confusing obviously, so there are some faults in the data.

• We’ve decided that this is a good enough baseline and we can move onto step 2.

• For the next blog, I’ll share the steps required to port this model onto a mobile.

• Why do I think 87% is good?

  We actually have to use a “Mobile friendly” architecture so that we can run inference on the mobile phone, which means the training wheels and power of ResNet 34 won’t be there and if we’re using SqueezeNet or MobileNet-87% would be a good mark to hit.

What’s Next?#

Cedric Chee, who is another fast.ai International Fellow in our Asia virtual study group and fellow student in the fast.ai community has developed the major portions required for Step 2 and 3 of our pipeline. Please checkout the mobile app demo:

Jupyter notebooks that will walk you through every step:

cedrickchee/data-science-notebooks

We’d want to use SqueezeNet/MobileNet-whatever works better eventually and make it run on the mobile.

PS: Tim Anglade, please wish us luck. We’ll bring your Emmy home this time 😎

Originally published at Medium

Hey folks. I hope your day is going well.

Today, I am excited to share my fast.ai “Cutting Edge Deep Learning for Coders” complete course notes. This is my personal notes on the 2018 edition of fast.ai Deep Learning Part 2. The notes are mainly written text transcript of each video lesson and they are partially time-coded. Thanks to our fellow student, Hiromi Suenaga for manually (old-fashioned) transcribe the full set of videos.

Benefits of this:

• We can refer back to the transcripts without having to play the videos all the time.
• This will save us a ton of time and helping us learn more effectively.
  • The transcript files are extremely helpful for searching up contents quickly.
• For whom English is not their first language that a major impediment to understanding the content is the lack of written transcript or course notes.

If you are looking for the 2017 edition (Keras+TensorFlow version), the course notes are available as well here.

Wait, there’s even more in my knowledge base (wiki). But, currently, it’s mostly for fast.ai course notes.

These notes will continue to be updated and improved as I continue to study and review the course.

Till next time, happy learning!

All rights belong to their respective owners.

Summary: fastai v1 — the rewrite of the fast.ai deep learning library.

fastai_v1 is the codename for fastai deep learning library version 1.0. It’s the beginning of the new version of the library.

From the fast.ai forums:

We’re doing a rewrite of the fastai library, with the following goals:

• Support consistent API for classification, regression, localization, and generation, across all of: vision, NLP, tabular data, time series, and collaborative filtering
• Clear and complete documentation for both new and experienced users
• Well tested (both unit and integration tests)
• Better structured code
• Notebooks showing how and why the library is built as it is.

fast.ai has decided to create a new version of the library for the next course on October 2018. The current fastai v0.x will be maintained with the necessary fixes but fast.ai won’t make any major change.

fast.ai do plan to have a library used outside of the course. With the changes in PyTorch on one hand, and the new features added as time goes, Jeremy felt he had to start again from scratch to create something more intuitive that ties all the existing APIs together.

Current Status#

At the moment (July 19, 2018), they’ve only just started. At this stage (Aug 4, 2018), nothing is functional. If you’re interested in contributing, join the discussion at fast.ai dev forum. The development is happening in this GitHub repo.

UPDATE (2018-08-12): They’re in the process of incorporating all of the best practices used in the DAWNBench and “train ImageNet in 18 minutes” project directly into the fastai library, including automating the selection of hyper-parameters for fast and accurate training.

Why Follow Along?#

It’s early days, so I see them refactor the code every day. It would be nice to see the whole process, like for example, how and why certain decisions were made during development. I think we can learn a lot from that.

You can follow the dev commits in GitHub and see what’s happening, and fast.ai welcome you to ask questions in fastai-dev forum or make suggestions.

It’s cool to see something being written from the ground up.

I often train machine learning/deep learning models and it takes a very long time to finish. Even an epoch in a moderately complex model takes near to half an hour to train. So, I constantly need to check (baby sit) the training process.

To help reduce the pain, I need a way to notify me on the training metrics. The idea is, we will send the training metrics (messages) as notifications on mobile using PyTorch Callbacks.

I have written some Python code snippets that helps me send my training metrics log as mobile push notifications using Pushover service. They have a limit of 7500 requests per month per user—which is fine for my usecase.

Those who’d like to have something like this, you can grab those little hacky scripts.

Cool, now, I can make tea while training without being anxious :smile:

fastai_example.py:

from notification_callback import NotificationCallback

# An example of integrating PyTorch callback with fastai model training loop
learn = ConvLearner.from_model_data(md, data)
notif_cb = NotificationCallback()
learn.fit(lr, 1, wds=wd, cycle_len=2, use_clr=(10, 20), callbacks=[notif_cb])

notification_callback.py:

from send_notification import send_notification

class NotificationCallback(Callback):
    """
        PyTorch callback for model training
    """
    def on_train_begin(self):
        self.epoch = 0

    def on_epoch_end(self, metrics):
        val_loss, accuracy = metrics[0], metrics[1]
        message = "epoch: " + str(self.epoch) + " val loss: " + str(val_loss[0])[0:7] + " val acc: " + str(accuracy)[0:7]
        send_notification(message)
        self.epoch += 1

send_notification.py:

def send_notification(msg):
    """
        Send message to mobile using Pushover notifications.
        Calls Pushover API to do that.
        Pushover API docs: https://pushover.net/api
    """
    import requests
    from datetime import datetime

    url = "https://api.pushover.net/1/messages.json"
    data = {
        "user"  : "<<YOUR_USER>",
        "token" : "<<YOUR_TOKEN>>",
        "sound" : "magic"
    }
    data["message"] = msg
    data['message'] = data['message'] + "\n" + str(datetime.now())

    r = requests.post(url = url, data = data)

Discriminative Learning Rate#

This paper, Large Batch Training of Convolutional Networks by Boris Ginsburg et. al has discriminative learning rate algorithm known as Layer-wise Adaptive Rate Scaling (LARS).

It was used to train ImageNet with very very large batch sizes by looking at the ratio between the gradient and the mean at each layer and using that to change the learning rate of each layer automatically. They found that they could use much larger batch sizes.

Code Implementation#

A training algorithm based on LARS implemented as an optimizer in PyTorch follows:

from torch.optim.optimizer import Optimizer, required


class LARS(Optimizer):
    def __init__(
        self,
        params,
        lr=required,
        momentum=0,
        dampening=0,
        weight_decay=0,
        nesterov=False,
        eta=0.001,
    ):
        if lr is not required and lr < 0.0:
            raise ValueError("Invalid learning rate: {}".format(lr))
        if momentum < 0.0:
            raise ValueError("Invalid momentum value: {}".format(momentum))
        if weight_decay < 0.0:
            raise ValueError("Invalid weight_decay value: {}".format(weight_decay))

        defaults = dict(
            lr=lr,
            momentum=momentum,
            dampening=dampening,
            weight_decay=weight_decay,
            nesterov=nesterov,
            eta=eta,
        )

        if nesterov and (momentum <= 0 or dampening != 0):
            raise ValueError("Nesterov momentum requires a momentum and zero dampening")
        super().__init__(params, defaults)

    def __setstate__(self, state):
        super().__setstate__(state)

        for group in self.param_groups:
            group.setdefault("nesterov", False)

    def step(self, closure=None):
        """
        Performs a single optimization step.

        Arguments:
            closure (callable, optional): A closure that reevaluates the model and returns the loss.
        """
        loss = None

        if closure is not None:
            loss = closure()

        for group in self.param_groups:
            weight_decay = group["weight_decay"]
            momentum = group["momentum"]
            dampening = group["dampening"]
            nesterov = group["nesterov"]
            eta = group["eta"]

            for p in group["params"]:
                if p.grad is None:
                    continue

                d_p = p.grad.data
                d_pn = d_p.norm()

                if weight_decay != 0:
                    d_p.add_(weight_decay, p.data)
                    d_pn.add_(weight_decay, p.data.norm())

                if momentum != 0:
                    param_state = self.state[p]

                    if "momentum_buffer" not in param_state:
                        buf = param_state["momentum_buffer"] = torch.zeros_like(p.data)
                        buf.mul_(momentum).add_(d_p)
                    else:
                        buf = param_state["momentum_buffer"]
                        buf.mul_(momentum).add_(1 - dampening, d_p)

                    if nesterov:
                        d_p = d_p.add(momentum, buf)
                    else:
                        d_p = buf
                rho = eta * p.data.norm() / (1e-15 + d_pn)
                p.data.add_(-group["lr"] * rho, d_p)

        return loss

Integrate mindfulness into your everyday life.

1. Non-reactivity - body sensation
2. Non-reactivity - working with sound
3. Non-reactivity - thinking
4. Non-reactivity - emotion
5. The judging mind
6. Mental noise as addiction
7. External cues as mindfulness reminders
8. Beginners mind
9. One step at a time
10. Grasping and aversion
11. Riding the waves
12. Letting go
13. Balance and nourishment
14. Questioning our thoughts
15. Catching ourselves
16. Leaning in
17. Allowing
18. Impermanence
19. Loving kindness
20. Maintaining momentum
21. Practice in each moment

My notes on self-improvement plan.

Ways to change habits:

1. Get more sleep
2. Make time to exercise
3. Drink more water
4. Eat less sugar
5. Stay teachable
6. Read and write more
7. Remove clutter
8. More random acts of kindness
9. Don’t respond to negativity
10. Spend quality time with family
11. Show gratitude
12. Forgive first

My quick notes for the “Building the Software 2.0 Stack” talk by Andrej Karpathy at Train AI 2018 conference - machine learning for a human world.

Training Datasets#

The part around building and managing datasets is very interesting. We don’t get to hear about these problems often.

Software 2.0 Integrated Development Enviroments (IDEs)#

What IDEs including code editors will look like?

• Show a full inventory or statistics of the current dataset.
• Create or edit annotation layers for any datapoint.
• Flag, escalate & resolve discrepancies in multiple labels.
• Flag and escalate datapoints that are likely to be mislabeled.
• Display predictions on an arbirary set of test datapoints.
• Autosuggest datapoints that should be labeled.