tokenize — Easy Playground

Tokenize Python source code into individual tokens

Python Playground

import tokenize

print(f"Module: tokenize")
print(f"Contents: {dir(tokenize)[:10]}")

Output

Click "Run" to execute your code

The tokenize module is part of Python's standard library. Tokenize Python source code into individual tokens.

Try modifying the code above to explore different behaviors. Can you extend the example to handle a new use case?