thinkall/tokenizer-play-ground

This is a tokenizer play ground implemented with python.

How to run

• Step by Step

pip install -r requirements.txt
cd src
python app.py

• Quick launch

Linux/Mac

bash run.sh

Windows

double click `run.bat`

Snapshots