Hugging Face Tokenizers (11.2)
Hugging Face hub provides several sub-word tokenizers that can take input text and break it down into common words and parts of words. This video shows how to use Hugging Face to tokenize your input text.
Code for This Video:
https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_11_02_tokenizers.ipynb
Course Homepage: https://sites.wustl.edu/jeffheaton/t81-558/
Follow Me/Subscribe:
https://www.youtube.com/user/HeatonResearch
https://github.com/jeffheaton
Tweets by jeffheaton
Support Me on Patreon: https://www.patreon.com/jeffheaton
