Tokenize

<tokenize>

Purpose/Function

If the whitespace attribute is true, calls the org.apache.common.lang.StringUtils.split method; else it calls a utility method which cycles through the chain (list) of delimiters until a delimiter is not found. If paired is true, it processes the delimiters in pairs, treating odd tokens as "start" tags and even ones as "end" tags.

Attributes

NAME	TYPE	DEFAULT VALUE	DESCRIPTION
string	`java.lang.String`	required	string to be tokenized
whitespace	`boolean`	false	tokenize on all whitespace characters
paired	`boolean`	false	process delimiters in pairs, returning token between them
includeDelimiter	`boolean`	false	include the delimiter with the token
asList	`boolean`	true	if `true`, return value as `java.util.ArrayList`; else as `java.lang.String[]`
chain	`java.util.List`	`null`	the chain of delimiters

Elements

TAG	TYPE	COUNT	DESCRIPTION
`<delimiter>`	`java.lang.String`	0:N (only if whitespace is `false` and chain is not set)	add a delimiter to the chain

In the case of a single delimiter, this can also be set as an attribute (cf. 5. Interchanging Attributes and Tags).

ReturnValues

DEFAULT NAME	TYPE	DESCRIPTION
tokens	either `java.util.ArrayList` or `java.lang.String[]`	the tokenized string

Examples

<ogrescript>
   <declare name="s0" string="abcdefghijklmnop	qrstuvwxyzab,cdefghijklm nopqrstuvwxyzlmnopqrstuvwxyz"/>
   <declare name="tokens"/>
   <tokenize string="${s0}" whitespace="true">
   	    <return-value defaultName="tokens"/>
   </tokenize>
   <echo message="${tokens}"/>
   <tokenize string="${s0}" delimiter="," asList="false">
   	    <return-value defaultName="tokens"/>
   </tokenize>
   <echo message="${tokens}"/>
   <tokenize string="${s0}" includeDelimiter="true">
	    <delimiter>ghijk</delimiter>
	    <delimiter> </delimiter>
	    <delimiter>vw</delimiter>
   	    <return-value defaultName="tokens"/>
   </tokenize>
   <echo message="${tokens}"/>
</ogrescript>

Paired tokens:

<ogrescript>
   <declare name="s0" string="[OgreScript:abcdefghijklmnop]NOTINCLUDED[OgreScript:qrstuvwxyzab]NOTINCLUDED[OgreScript:cdefghijklm]NOTINCLUDED[OgreScript:nopqrstuvwxyzlmnopqrstuvwxyz]"/>
   <declare name="tokens"/>
   <tokenize string="${s0}" paired="true" includeDelimiter="true">
	    <delimiter>[OgreScript:</delimiter>
	    <delimiter>]</delimiter>
   	    <return-value defaultName="tokens"/>
   </tokenize>
   <echo message="${tokens}"/>
</ogrescript>

Child pages