Recurrent neural networks and why you should care

 

“21 Of The Most Life–Changing Food Magazine Moments Of 2013”

“Why Are The Kids On The Golf Team Changing The World?”

“John McCain Warns Supreme Court To Stand Up For Birth Control Reform”

 

Take a look at the above headlines. At first glance, they might seem completely typical, not unlike the stories on the front pages of more clickbait-y sites like BuzzFeed or Gawker.

In fact, those aren’t real headlines: a computer algorithm has generated them. The model was trained using 2 million real headlines from websites like the ones mentioned above. After three days, it was able to output grammatically correct, vaguely real sounding headlines like these. The model achieved this through the use of so-called recurrent neural networks (RNN).

Artificial neural networks have been used in machine learning for a while. They are a simple representation of the functioning of a human brain: a network of nodes (neurons), with connections between them (synapses). Each of the neurons takes its input and transforms it to an output, while each synapse is assigned weights. This neural network is then ‘trained’ to transform the initial input to your desired output, by adjusting the weights of the synapses.

If that makes little sense, consider the above diagram: The first layer of neurons simply looks for lines, corners, or colors in the input image. The next layer interprets those to look for shapes or components, like ears or a nose. This continues through the layers, until the top neurons are able to identify the image as a cat with 90% certainty. For a more detailed explanation, feel free to visit Google’s research blog.

A recurrent neural network improves on the basic concept by also feeding its state at a past timestep into the current timestep. Essentially, this makes the model very suited for operations on sequential data, like text sentences. This is exactly what happened in the experiment above. The theoretic foundations for RNNs were laid a long time ago, but recent advances in computing hardware have made it possible to actually put them to use.

Now while computers generating headlines for blogs is cool, it doesn’t add much value. Luckily, RNNs can be applied to more useful problems as well. For example, Google relies on RNNs for natural speech recognition to power their voice assistant, Google Now. The same goes for the feature in Google Photos that allows you to search for ‘sunset’ and returns only those pictures you took with a sunset in them, and nothing else. They can be used to automatically classify movie reviews by general sentiment (very negative, neutral, positive, etc.). They can be used to predict stock prices of the S&P500. And so on.

In short, recurrent neural networks are what happens behind the scenes of many IT applications that make you go ‘Wow’.