Interesting papers from AI skeptics

Green AI

Neural networks winning by overfitting

The leaderboard and metrics problem appears again in these two articles on recommendation systems, and natural language processing (NLP).

Neural networks losing to statistics techniques

This brief comment on Hacker News is a few months old now, but I think of it often. It’s a warning to people like me who try running their data through AutoML platforms. AutoML was bested by feature engineers (but given the same inputs, AutoML did outperform XGBoost).

Natural Adversarial Examples

(in other words: real-world AI fuck-ups)

Conclusions

  1. It’s a lucky miracle that AIs can drive cars. I sometimes wonder: how can robots drive but not make pancakes yet? It turns out that cars don’t need to be so picky — they make 3D models of their surroundings and don’t bump into those things. They benefit from wide roadways where everyone has learned since childhood not to get in the way. In a catastrophic situation, if the AI stops the car safely, it did good.
    A fully autonomous car would have more problems to solve (reading signs, following directions of a flagger, etc.) but these jobs are made easier by repeatedly viewing the same intersections, or giving the wheel back to a human driver.
  2. Leaderboards are an interesting driver in showing us when AI advances a lot, but appear in all of these papers as a negative force. I wonder, is it possible for a leaderboard to be a moving target? Ideas:
    - organizers gradually add adversarial examples and new classes
    - an additional ‘none of the above’ class
  3. As someone working on AI/ML as a side project, I don’t know what to do on projects where a MegaCorp already has a team of people doing the work. A good example is map tracing for OpenStreetMap — it would be awesome for me to work on such a project, but there are actual research teams at Facebook making progress on this already. It’s a solid positive for the mapping world — similar to how I felt when data imports, drones, and cars started mapping — but whatever experiments that I tried in this space would be outpaced and out-computed fairly easily.
    I appreciated an interesting and encouraging thread on this subject back in January:

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Nick Doiron

Nick Doiron

Web->ML developer and mapmaker.