tokenize — Expert Playground

Tokenize Python source code into individual tokens

Python Playground

# tokenize internals
import tokenize
import sys

mod = tokenize
print(f"Module: {mod.__name__}")
print(f"File: {getattr(mod, '__file__', 'built-in')}")
print(f"Package: {getattr(mod, '__package__', 'N/A')}")
print(f"Size in modules: {sys.getsizeof(mod)} bytes")

Output

Click "Run" to execute your code

Understanding the internals of tokenize helps write more efficient code.

Challenge

Try modifying the code above to explore different behaviors. Can you extend the example to handle a new use case?

View all Expert examples for tokenize