Hugging Face Tokenizers (11.2)

Hugging Face hub provides several sub-word tokenizers that can take input text and break it down into common words and parts of words. This video shows how to use Hugging Face to tokenize your input text.

Code for This Video:
https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_11_02_tokenizers.ipynb

Course Homepage: https://sites.wustl.edu/jeffheaton/t81-558/

Follow Me/Subscribe:
https://www.youtube.com/user/HeatonResearch
https://github.com/jeffheaton

Support Me on Patreon: https://www.patreon.com/jeffheaton

Source of this machine learning/AI Video

AI video(s) you might be interested in …