Google open-sources BERT, a state-of-the-art pretraining technique for natural language processing

Natural language processing – the subcategory of artificial intelligence that spans language translation, sentiment analysis, semantic search, and dozens of other linguistic tasks – is easier said than done.

One popular solution is pretraining, which refines general-purpose language models trained on unlabeled text to perform specific tasks.

Google this week open-sourced its cutting-edge take on the technique – Bidirectional Encoder Representations from Transformers, or BERT – which it claims enables developers to train a “State-of-the-art” NLP model in 30 minutes on a single Cloud TPU or a few hours on a single graphics processing unit.

As Jacob Devlin and Ming-Wei Chang, research scientists at Google AI, explained, BERT is unique in that it’s both bidirectional, allowing it to access context from both past and future directions, and unsupervised, meaning it can ingest data that’s neither classified nor labeled.

BERT learns to model relationships between sentences by pretraining on a task that can be generated from any corpus, Devlin and Chang wrote.

It builds on Google’s Transformer, an open source neural network architecture based on a self-attention mechanism that’s optimized for NLP. When tested on the Stanford Question Answering Dataset, a reading comprehension dataset comprising questions posed on a set of Wikipedia articles, BERT achieved 93.2 percent accuracy, besting the previous state-of-the-art and human-level scores of 91.6 percent and 91.2 percent, respectively.

The release of BERT follows on the heels of the debut of Google’s AdaNet, an open source tool for combining machine learning algorithms to achieve better predictive insights, and ActiveQA, a research project that investigates the use of reinforcement learning to train AI agents for question answering.

This article was summarized automatically.

Original link