Who is your AI trying to beat?

Here’s a classic: 

Two men are all alone on the savannah with no guns or weapons. Suddenly a big hungry-looking lion jumps out of bushes. There’s no doubt what’s going to happen. While the lion is slowly moving closer to its prey, one of the men kneels down and starts tying his shoelaces. The other man looks confused and says: “Do you really think you can outrun a lion?”. “No”, says the first man. “I just have to outrun you”.

I think about this story a lot when working with AI. Sometimes we focus on beating the lion when setting the success criteria for an AI-solution. Instead of trying to beat the other man, which is actually achievable. Often the users or customers will be pushing to beat the lion. Generally there is a tendency to expect all-problem-solving results from new technology. Especially technology like AI that has a kind of magic aura to it. So how do we address this problem?

The Paperflow Lion

You have to spend some time finding the true benchmark. In the company I co-founded Paperflow we were late to the game here. Paperflow captures data off of invoices in pdf or picture format and delivers the structured data to e.g. ERP’s. Simple right? So what is the lion and the other man in this case? 

The benchmark we initially set in Paperflow was to be correct almost 100 percent of the time. That was the lion. As we got smarter, the benchmark shifted towards the average bookkeeper. But how good is the average bookkeeper? If you ask the bookkeepers they almost never make mistakes. If you look at the books there’s another story to be told. 

So in Paperflow the problem was not only that there was a lion (100% correct results) and another man(Average bookkeeper). There was also a perception bias. And that’s just how humans are.

The solution 

So how should we deal with that? The answer is communication and curiosity. 

By communicating a lot with the users about what really to expect is essential. It sounds obvious but I have made this mistake myself and I’ve seen a lot of other AI-projects go easy on managing expectations. So communication has to be stressed. The lion story can be a useful tool when communicating with users. Make them aware of what they are asking. It might be easier to find common ground when the user knows that they initially asked for a lion.

Curiosity is also key. Everybody that works with AI has asked themself about what level of domain knowledge they need to make a successful AI. For some it’s a cherry on top and for some it’s more a foundation to build on. As most people probably are, I’m somewhere in the middle. The trick for me is to be sincerely curious and non-judgmental in the research of the domain. Processes and behavior that might seem stupid at first are always there for a reason. So asking questions and putting your assumptions aside is the way to go here. 

So don’t try to be a lion king. You don’t have to.