Measuring loss on new GPT tokens

Before fine-tuning

Measuring token-specific match during generative model fine-tuning

Measuring loss within added tokens

loss on new tokens
the ideal: a gradually decreasing metric. loss from each step in the GPT-2 model

--

--

--

Web->ML developer and mapmaker.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

What are all these terms … Data Science, Machine Learning, Artificial Intelligence? | AI MakerSpace

Image Style Transfer using Pre-trained ConvNets

Neural Networks: Everything you Wanted to Know

Real-Time Image Captioning on an Embedded System using a light-weight Deep Learning Model (Part-1)

Building RGB Color Classifier: Part 1

A light introduction to transformers for NLP — by Murilo Cunha

RNN image

Retrieval Augmented Language Model Pre-Training (REALM)

Statistical Language Models

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Nick Doiron

Nick Doiron

Web->ML developer and mapmaker.

More from Medium

New Discord Group for DAIR.AI

Multi-head Attention for global context summarisation in multimodal analysis

Sentence Transformer Fine-Tuning (SetFit): Outperforms GPT-3 on few-shot Text-Classification while…

The NLP Landscape from 1960 to 2020