Tokenize(str,
whitespace=' \t\r\n ' ,
quotes=' " ' ,
escapes=' \\ ' )
| source code
|
String tokenizer
This function tokenizes a string while taking quotation and escaping
into account.
>>> import dhm.strtools
>>> dhm.strtools.Tokenize("this is a test")
['this', 'is', 'a', 'test']
>>> dhm.strtools.Tokenize("this "is a" test")
['this', 'is a', 'test']
>>> dhm.strtools.Tokenize("this \"is\" a test")
['this', '"is"', 'a', 'test']
>>> dhm.strtools.Tokenize("this "is a test")
Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "/usr/local/lib/python2.2/site-packages/dhm/strtools.py", line 80, in Tokenize
raise TokenizeError, "Unexpected end of string in quoted text"
dhm.strtools.TokenizeError: Unexecpted end of string in quoted text
- Parameters:
str (string) - string to tokenize
whitespace (string) - whitespace characters seperating tokens
quotes (string) - legal quoting characters
escapes (string) - characters which can escape quoting characters
- Returns: sequence of strings
- list of tokens
|