Measuring loss on new GPT tokens

Before fine-tuning

The initial answers from the T0pp model do not show an understanding of New York foods, transit, or accessibility. I could work on the prompt, but there seems to be minimal interest in NYC topics:

Measuring token-specific match during generative model fine-tuning

Most GPT-derived code either queries a pre-trained model from the command line, fine-tunes without collecting metrics, or calculates perplexity after training. We would like to track likelihood of correctly finding specific tokens during the fine-tuning process.

Measuring loss within added tokens

https://colab.research.google.com/drive/1gd_yzuUxKm28f3Ezc8LiHCnkzMz3PSIT

loss on new tokens
the ideal: a gradually decreasing metric. loss from each step in the GPT-2 model

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store