UnicodeString::doExtract writes only one single trailing null always

The patch for this bug will allow non-byte based encodings, like UTF-16 and UTF-32, to be properly NULL terminated when the result is interpreted as a non-byte based encoding.

The change breaks user code, as seen in our test code (see ticket 5882). Need to either re-fix or revert.

The APIs did work as designed, even if that wasn't the best design.

I reverted the change.

svn merge -c -22476 svn+ssh://source.icu-project.org/repos/icu/icu/trunk
(fix copyrights to 2007)
svn commit -m "..."
Committed revision 22607.

This change is too risky this close to a release. It needs more discussion for whether we want the change at all, and if so, how it should work.

For backward compatibility, in every API that does not have a destCapacity parameter and that used to write exactly one 0x00 byte we must continue to write exactly one 0x00 byte. (This is what ticket #5882 is about.)

For backward compatibility, we must write at least one 0x00 if it fits, even if sizeOfNull bytes won't fit. r22476 wrote all or nothing. If we make this sort of change, we need to write either 0 (if capacity<=length) or 1 (if length<capacity<length+sizeOfNULL) or sizeOfNull (if length+sizeOfNULL<=capacity) bytes.

We need to consider when to set U_STRING_NOT_TERMINATED_WARNING. r22476 set it if there was not space for sizeOfNull bytes, even if there was space for one and the old code wrote the 0x00 and didn't set the warning. I think we need to keep the old behavior so that we don't mess with existing user code. We could discuss changing the semantics such that we set the warning if not all of the sizeOfNull bytes fit, but we should consider it carefully.

• If we change it (set the warning if not all of sizeOfNull fit) then we might break existing code.
• If we don't change it (set the warning only if not even one 0x00 fits) then the warning becomes meaningless for sizeOfNull>1, but at least we are not changing existing behavior.

For backward compatibility, we cannot change the input side of functions like ucnv_toUChars() (ticket #5870): If we changed them to look for a NUL with sizeOfNull bytes, then we would not find the single NUL that we used to find. Someone might depend on the old behavior. (The old behavior is useless for UTF-16 but works fine for old-style DBCSes that don't use 0x00 in double-byte characters.

If we change the output side (carefully) but cannot change the input side, then we might confuse as many users as now because of the mismatch. We could decide that that's ok though.

It would be preferrable to not change the signature of u_terminateChars() but to introduce a sibling function with the additional parameter, so that we need not change as many call sites.

Just in terms of review and comfort level, I would like to take the time and examine each API (or many) that indirectly calls u_terminateChars(), not just directly.

Reviewer: There should be no net change here since the change should have been backed out.