When word and sub-word don’t do the job
A Thai BERT model that I’d adapted recently appeared in a paper on cross-language learning in biomedicine. This same model is in the spaCy docs. People use it on GitHub and Kaggle. As in most NLP models, I had used a tokenizer which splits text into a stream of words…
Reinflection experiments in Spanish and Arabic
Early this year, I uploaded a seq2seq model to generate gender counterfactuals in Spanish (el profesor viejo <-> la profesora vieja). Since then, I debugged some issues, made a general-purpose library, and created an initial seq2seq model for Arabic.
The goal is to see if passing data through process creates…
Comparing NLP’s text mutation tools
In image classifiers, a huge library of image augmentation tools crop, rotate, and otherwise edit input images to add variety to the training data and make classifiers more robust.
It’s reasonable to think that a similar technique could improve text data, but editing a sentence is more than moving pixels in…
The Supreme Court, and an Indonesian choose-your-adventure. I’ve started reading a book on the Russian Navy which will appear next month.
Wild Bill: The Legend and Life of William O. Douglas (Bruce Murphy, 2003)
This Supreme Court biography opens with a quote (“William O. Douglas was the oddest duck to ever serve on the United States Supreme Court”) and an anecdote which is…
Following up on Podcasts part 1
Asian Enough
https://podcasts.apple.com/us/podcast/asian-enough/id1501446978
Creators: Los Angeles Times
Concept: Interviews with famous/prominent Asian Americans about living their life, growing up, embracing cultures, and current events. Every episode is very different; it depends which celebrities you know and which issues you want to know more about.
Schedule: Weekly…
638 days ago, I went from a BrooklynJS meetup to a late night showing of Star Wars, and then took the 2:30am train back to Boston. We knew it was the final days of 2019, but we had no clue. …
I’m in the acknowledgments but it’s complicated
In December 2020, I stumbled on an early draft of the now widely-read FAccT conference paper ‘On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜’. After debating whether I would be taken seriously or blend in with the fray of political comments, I sent a Tweet…
Four days after being cancelled, ‘Republic of Sarah’ aired its final episode of its one season on The CW. It’s the story of a history teacher (Sarah) who rallies her town on the US-Canadian border to declare independence, to avoid destruction by mining interests. Is this the first micro-nation /…
Supreme Power: Franklin Roosevelt vs. the Supreme Court (Jeff Shesol, 2010)
A comprehensive timeline of Roosevelt’s bid for control of the Supreme Court, starting with grumblings in his first presidential campaign. The push and pull between New Deal reforms and judicial review was taught in my US History class in a few paragraphs, but I never got a sense of how…
I added ‘NYC words’ to the vocabulary. Was that necessary?
While I was building GPT-NYC, one of my focus areas for customization was tokens. Long story short, GPT-2 doesn’t come with individual tokens for bagels, strollers, or the Bronx. …
