Let’s play with Transformers!

Back in 2017, taming the transformer was no trivial task. Researchers had to wrangle with a very large perimeter space. Now with many trials blazed, you play with transformers on your own. It's easy and fun!

To begin, let’s open a google colab and create a new notebook (or if you have a really high end computer, a jupyter notebook will work). You’ll probably want to take advantage of Google’s free GPU or TPU hardware accelerators (runtime -> change runtime type).

First, let’s import our favorite package:

import tensorflow as tf

Second, we need a transformer model to play with. You could build one from the ground up with tensorflow, but transformers makes it easy:

!pip install transformers
import transformers
transformer = transformers.TFGPT2LMHeadModel.from_pretrained('gpt2')

We did it! We now have a gpt2 transformer in our very own python environment to work with. We can summarize articles, answer questions, continue conversations, finish sentences, and more. Before we order pizza though, we need a way to communicate with transformer . That’s because while natural language is expressed in words, transformer speaks in numbers. Any input words must first be tokenized or translated to an integer sequence representation. It works like this:

tokenizer = transformers.GPT2Tokenizer.from_pretrained('gpt2')
tokens = tokenizer.encode('Machine learning is awesome!')
print(tokens)