Conversion mapping table collection tool update

This is only a small part of the problem. See reply for more details.

• 08/28/02 20:40:13 grhoten changed notes2
• 08/28/02 20:40:13 grhoten changed notes
• 08/28/02 20:40:15 grhoten marked private
• 08/28/02 20:40:15 grhoten moved from incoming to config_build
• 08/28/02 21:02:59 grhoten sent reply 1
• 08/28/02 21:04:42 grhoten changed notes2
• 08/28/02 21:04:42 grhoten changed notes
• 08/29/02 15:16:26 hshih sent reply 2
• 08/30/02 16:02:43 hshih changed notes2
• 08/30/02 16:02:43 hshih changed notes
• 10/24/02 17:30:47 schererm moved from config_build to conversion
• 01/03/03 14:45:45 hshih changed notes2
• 01/03/03 14:45:45 hshih changed notes
• 01/24/03 18:58:29 grhoten changed notes2
• 01/24/03 18:58:29 grhoten changed notes
• 04/14/03 18:24:20 ram changed notes2
• 04/14/03 18:24:20 ram changed notes
• 04/15/03 18:10:14 schererm changed notes2
• 04/15/03 18:10:14 schererm changed notes
• 06/12/03 18:09:28 grhoten changed notes2
• 06/12/03 18:09:28 grhoten changed notes
• 06/23/03 14:57:01 hshih changed notes2
• 06/23/03 14:57:01 hshih changed notes
• 09/08/03 17:52:31 schererm changed notes2
• 09/08/03 17:52:32 schererm changed notes
• 02/01/04 00:24:11 grhoten changed notes2
• 02/01/04 00:24:11 grhoten changed notes
• 07/18/04 15:59:12 grhoten changed notes2
• 07/18/04 15:59:12 grhoten changed notes
• Tue Jul 12 16:46:58 2005 grhoten changed notes2: target: "3.2" to "3.6",
• Tue Jul 12 16:46:58 2005 grhoten changed notes
• Fri Sep 16 08:46:56 2005 grhoten changed notes2: assign: "grhoten, barry" to "grhoten",
• Fri Sep 16 08:46:56 2005 grhoten changed notes
• Thu Aug 10 20:09:32 2006 grhoten changed notes2: target: "3.6" to "3.8",
• Thu Aug 10 20:09:32 2006 grhoten changed notes
• Fri Oct 13 23:30:00 2006 grhoten changed notes2: target: "3.8" to "UNSCH",
• Fri Oct 13 23:30:00 2006 grhoten changed notes
• Mon Nov 27 17:26:31 2006 schererm changed notes2: summmary: "" to "RFE: Conversion mapping table collection tool update",
• Mon Nov 27 17:26:31 2006 schererm changed notes

this is basically an ongoing bug, file separate bugs for individual work items (markus 20030415)

All of the charset tools need to handle shift-in, shift-out and reverse fallbacks.

This needs an overview discussion too. There are many tools, and they don't interoperate with each other very well.

1) They all handle the validity tables differently, and only one of my current tools does a partial verification of the tables. This verification needs to be extended, and all of the .ucm files need proper validity tables. Many won't work with makeconv.

2) The generated .ucm files need to be equivalent to the output generated by makeconv\misc\canonucm.c and makeconv\misc\rptp2ucm.c

3) Handle shift-in, shift-out in collection tools, comparison tools and UCM->XML tools.

4) An XML->UCM might be nice to have, but no one has needed it yet.

5) The collection tools need to collect reverse fallbacks, but it's difficult to get them consistently. Right now they need manual intervention and the tool only works on Windows, which is not nice.

6) Not all of these tools are public. They should be made public eventually.

7) Documentation is needed for these tools.

8) More testing is needed when these tools are updated. I have occasionally run into bugs for these tools. I try to fix them when I run into them, and the tools are much better than before. These tools can still be greatly improved.

9) Various people have complained about the format and ease of use for the roundtripIndex.html. We should look into making this table easier to navigate. A table of contents may be nice so that I can look at the comparison really quick, and a link to the tables in CVS would be nice too. Once item 5 works, then the reverse fallbacks can be compared too.

10) and the list goes on...

additional note from Markus:

We should improve our charmap collection tools to gain greater confidence in and fidelity of our tables. Many people rely on our data, more if CDRA procedure changes go through.

- collect reverse fallbacks - add/fix MBCS state tables - collect ISO-8859-X standard charmaps (unicode.org?) - split collection code into common part and per-platform part for easier porting - collect EBCDIC_STATEFUL-type charmaps - collect ISO-2022-type charmaps?? (not sure this is feasible)