bFloat16 for Neural Networks? You win some and you lose some

So you're probably wondering what's the hype with Nervana's processor and the push to adopt bFloat16 format for AI.

One of the main issues is that the extreme performance levels needed for large scale neural networks requires performance in the 50-200 TFLOP range.  Large scale being something like 8-20 billion neuron units.  So there are a few cards that can do such things like Nvidia's new Titan Rcx(?) something or other. But there's a catch. It's only at 16bit. go to full 32 bit precision and performance drops off.

Neural networks rarely need such large numbers. What they need more is precision. Yet bFloat16 only has 7 bits for the fraction, rather than standard 16 bit float which has 10. since each is a DOUBLING of range, those 2 bits mean a big difference in precision. What gives?

Remember that most neural networks store values between 0 and 1. The need for large numbers doesn't exist.

It has to do with the speed of moving huge pipelines of 32 bit precision data in traditional memory into and out of these systems. By removing 2 precision bits it allows conversion from 32 bit float to be much simpler as the exponent is identical. You basically just whack 16 bits off the end.

32 bit floating point

bFloat16

What is the trade off? Well, its the loss of precision in the small numbers. So if you are designing a re-entrant highly convolutional dimensional neural network precision might be much more important to you vs. a vision system which needs huge data pipes into the system.  Since neural network systems use very specialized memory (gddr6 or HMB) which is more expensive, its tough to get 16 or even 24 gigabytes of ram to support the processing. That limits the number of nodes you can express.

bFloat16 is the google pushed format, but is it right for everyone? No it's not. The loss of precision will probably hinder the kinds of problems you can solve with neural networks.  Personally, I'd rather go with precision.

The Importance of Broad Querying and Narrow Indexing

One of the issues we came across with developing our enterprise NLP search engine was the issue of precision. Rather than rank, if the precision level is not met documents simply fall out of query match.

Let's take the case of an object being a predicate object in the query vs being a noun subject in the sentence being indexed.  There would be no match.  So how do we handle this? Well, we can create a broad query - containing the object as both noun subject and predicate object. Now it will match sentences that have either case.

Wait didn't we lose precision? Yep, we sure did. So how does that pass the muster? Well it has to do with two things. The first is many other things are contributing to the match. And all those elements will result in partial scores that add up to a total. The end user experience of the results found will match their mental expectation - they think that queries will resolve symmetry of grammar structure, or as we call them inversions.  There can be models where prepositional clauses dangle from different parts of the sentence. Should that be a match? For enterprise search a large part of this process is tuning and using a "gist" a "gestalt" that things feel ok to the brain. There is no hard and set rule.

Another way to approach broadening is to handle cases where you didn't find the expected document and you realize it was due to a narrow definition.  While you can't simply broaden the case, you can run a test broadening it and see how it effects your regression test set of queries. And then do a bit of spot checking. 

Finally the last issue is that of the near miss. This often happens with the verb. Microsoft accquires Documentum.  Microsoft purchased Documentum.  This might result in a total miss. So how to handle this? The best technique is to run your docset through a processor which determines similarity and clustering. Then you can extend your query with additional terms if there are any within a specified distance.  Again, its a technique that can assist or blow up your query. It takes tuning and time to review many queries.

So if your latest search technology isn't producing the results you expected, remember that broadening the search vs the index is one technique to bring more potential matches into your query rankings.