Introduction to token

🛠 Tools and Programming 🤖Artificial Intelligence
1

Convert words/characters into numeric vectors
Cover as much text as possible with effective tokens

Algorithms

BPE
WordPiece

References

3分钟搞清楚大模型的Token是什么_哔哩哔哩_bilibili