The following (improbable) RPP presents a difficult question for characterization: 
:[ \t]
!([a-z]+) ([a-z]+)			\1 ~ \2
This preprocessor rewrites "I arrived." to "I ~ arrived.", and then tokenizes it a ["I", "~", "arrived."]. The correct characterization of the first and last tokens is pretty clear, but the correct characterization of the "~" token is not. There are at least 4 plausible answers:
  1. "~" corresponds to the space between the two input words.
  2. "~" corresponds to the zero-length string between "I" and " arrived."
  3. "~" corresponds to the zero-length string between "I " and "arrived."
  4. "~" does not correspond to any input character positions.
Unfortunately these interpretations each have shortcomings.
  1. A token should not correspond to a string matching the token separator expression.
  2. There is no reason to prefer this to option 3.
  3. There is no reason to prefer this to option 2.
  4. Every token should correspond to some input string segment.