[bug-idutils] How to find tokens inside strings?

[sean]

Sorry for posting to the bug list.  Is there a support forum I could use
instead?

Anyway,
I'm looking for a way to have idutils identify tokens inside strings.  It
seems to work for strings with one word but not for strings with multiple. 
For example, in:

const char* str = "string";

xtokid finds the token "string";

But in:

const char* str = "string with multiple words";

xtokid finds nothing.  Is there a way to have it find each individual word
in that string?

Thanks.
Sean