`pip install bigcode-tokenizer==0.1.0`

Tool to tokenize source code

Source

Among fairly niche packages on PyPI.

Over 284 downloads in the last 90 days.

Version usage of `bigcode-tokenizer`

Proportion of downloaded versions in the last 3 months (only versions over 1%).

0.1.0

100.00%