2 '\" Copyright (c) 1997-1998 Sun Microsystems, Inc.
4 '\" See the file "license.terms" for information on usage and redistribution
5 '\" of this file, and for a DISCLAIMER OF ALL WARRANTIES.
7 '\" RCS: @(#) $Id: Encoding.3,v 1.29 2007/12/13 15:22:31 dgp Exp $
9 .\" The -*- nroff -*- definitions below are for supplemental macros used
10 .\" in Tcl/Tk manual entries.
12 .\" .AP type name in/out ?indent?
13 .\" Start paragraph describing an argument to a library procedure.
14 .\" type is type of argument (int, etc.), in/out is either "in", "out",
15 .\" or "in/out" to describe whether procedure reads or modifies arg,
16 .\" and indent is equivalent to second arg of .IP (shouldn't ever be
17 .\" needed; use .AS below instead)
20 .\" Give maximum sizes of arguments for setting tab stops. Type and
21 .\" name are examples of largest possible arguments that will be passed
22 .\" to .AP later. If args are omitted, default tab stops are used.
25 .\" Start box enclosure. From here until next .BE, everything will be
26 .\" enclosed in one large box.
29 .\" End of box enclosure.
32 .\" Begin code excerpt.
37 .\" .VS ?version? ?br?
38 .\" Begin vertical sidebar, for use in marking newly-changed parts
39 .\" of man pages. The first argument is ignored and used for recording
40 .\" the version when the .VS was added, so that the sidebars can be
41 .\" found and removed when they reach a certain age. If another argument
42 .\" is present, then a line break is forced before starting the sidebar.
45 .\" End of vertical sidebar.
48 .\" Begin an indented unfilled display.
51 .\" End of indented unfilled display.
54 .\" Start of list of standard options for a Tk widget. The manpage
55 .\" argument defines where to look up the standard options; if
56 .\" omitted, defaults to "options". The options follow on successive
57 .\" lines, in three columns separated by tabs.
60 .\" End of list of standard options for a Tk widget.
62 .\" .OP cmdName dbName dbClass
63 .\" Start of description of a specific option. cmdName gives the
64 .\" option's name as specified in the class command, dbName gives
65 .\" the option's name in the option database, and dbClass gives
66 .\" the option's class in the option database.
69 .\" Print arg1 underlined, then print arg2 normally.
72 .\" Print arg1 in quotes, then arg2 normally (for trailing punctuation).
75 .\" Print an open parenthesis, arg1 in quotes, then arg2 normally
76 .\" (for trailing punctuation) and then a closing parenthesis.
78 .\" RCS: @(#) $Id: man.macros,v 1.9 2008/01/29 15:32:33 dkf Exp $
80 .\" # Set up traps and other miscellaneous stuff for Tcl/Tk man pages.
84 .\" # Start an argument description
88 . ie !"\\$2"" .TP \\n()Cu
93 \&\\$1 \\fI\\$2\\fP (\\$3)
106 .\" # define tabbing values for .AP
109 .if !"\\$1"" .nr )A \\w'\\$1'u+3n
112 .if !"\\$2"" .nr )B \\w'\\$2'u+\\n()Au+3n
113 .nr )C \\n()Bu+\\w'(in/out)'u+2n
115 .AS Tcl_Interp Tcl_CreateInterp in/out
116 .\" # BS - start boxed text
117 .\" # ^y = starting y location
125 .if n \l'\\n(.lu\(ul'
128 .\" # BE - end boxed text (draw box now)
133 .ie n \l'\\n(^lu\(ul'
135 .\" Draw four-sided box normally, but don't draw top of
136 .\" box if the box started on an earlier page.
138 \h'-1.5n'\L'|\\n(^yu-1v'\l'\\n(^lu+3n\(ul'\L'\\n(^tu+1v-\\n(^yu'\l'|0u-1.5n\(ul'
141 \h'-1.5n'\L'|\\n(^yu-1v'\h'\\n(^lu+3n'\L'\\n(^tu+1v-\\n(^yu'\l'|0u-1.5n\(ul'
148 .\" # VS - start vertical sidebar
149 .\" # ^Y = starting y location
150 .\" # ^v = 1 (for troff; for nroff this doesn't matter)
154 .ie n 'mc \s12\(br\s0
157 .\" # VE - end of vertical sidebar
165 \h'|\\n(^lu+3n'\L'|\\n(^Yu-1v\(bv'\v'\\n(^tu+1v-\\n(^Yu'\h'-|\\n(^lu+3n'
172 .\" # Special macro to handle page bottom: finish off current
173 .\" # box/sidebar if in box/sidebar mode, then invoked standard
174 .\" # page bottom macro.
181 .\" Draw three-sided box if this is the box's first page,
182 .\" draw two sides but no top otherwise.
183 .ie !\\n(^b-1 \h'-1.5n'\L'|\\n(^yu-1v'\l'\\n(^lu+3n\(ul'\L'\\n(^tu+1v-\\n(^yu'\h'|0u'\c
184 .el \h'-1.5n'\L'|\\n(^yu-1v'\h'\\n(^lu+3n'\L'\\n(^tu+1v-\\n(^yu'\h'|0u'\c
187 .nr ^x \\n(^tu+1v-\\n(^Yu
188 \kx\h'-\\nxu'\h'|\\n(^lu+3n'\ky\L'-\\n(^xu'\v'\\n(^xu'\h'|0u'\c
201 .\" # DS - begin display
207 .\" # DE - end display
213 .\" # SO - start of list of standard options
215 'ie '\\$1'' .ds So \\fBoptions\\fR
216 'el .ds So \\fB\\$1\\fR
217 .SH "STANDARD OPTIONS"
223 .\" # SE - end of list of standard options
228 See the \\*(So manual entry for details on the standard options.
230 .\" # OP - start of full description for a single option
235 Command-Line Name: \\fB\\$1\\fR
236 Database Name: \\fB\\$2\\fR
237 Database Class: \\fB\\$3\\fR
241 .\" # CS - begin code excerpt
247 .\" # CE - end code excerpt
252 .\" # UL - underline word
256 .\" # QW - apply quotation marks to word
258 .ie '\\*(lq'"' ``\\$1''\\$2
259 .\"" fix emacs highlighting
260 .el \\*(lq\\$1\\*(rq\\$2
262 .\" # PQ - apply parens and quotation marks to word
264 .ie '\\*(lq'"' (``\\$1''\\$2)\\$3
265 .\"" fix emacs highlighting
266 .el (\\*(lq\\$1\\*(rq\\$2)\\$3
268 .\" # QR - quoted range
270 .ie '\\*(lq'"' ``\\$1''\\-``\\$2''\\$3
271 .\"" fix emacs highlighting
272 .el \\*(lq\\$1\\*(rq\\-\\*(lq\\$2\\*(rq\\$3
274 .\" # MT - "empty" string
278 .TH Tcl_GetEncoding 3 "8.1" Tcl "Tcl Library Procedures"
281 Tcl_GetEncoding, Tcl_FreeEncoding, Tcl_GetEncodingFromObj, Tcl_ExternalToUtfDString, Tcl_ExternalToUtf, Tcl_UtfToExternalDString, Tcl_UtfToExternal, Tcl_WinTCharToUtf, Tcl_WinUtfToTChar, Tcl_GetEncodingName, Tcl_SetSystemEncoding, Tcl_GetEncodingNameFromEnvironment, Tcl_GetEncodingNames, Tcl_CreateEncoding, Tcl_GetEncodingSearchPath, Tcl_SetEncodingSearchPath, Tcl_GetDefaultEncodingDir, Tcl_SetDefaultEncodingDir \- procedures for creating and using encodings
284 \fB#include <tcl.h>\fR
287 \fBTcl_GetEncoding\fR(\fIinterp, name\fR)
290 \fBTcl_FreeEncoding\fR(\fIencoding\fR)
294 \fBTcl_GetEncodingFromObj\fR(\fIinterp, objPtr, encodingPtr\fR)
298 \fBTcl_ExternalToUtfDString\fR(\fIencoding, src, srcLen, dstPtr\fR)
301 \fBTcl_UtfToExternalDString\fR(\fIencoding, src, srcLen, dstPtr\fR)
304 \fBTcl_ExternalToUtf\fR(\fIinterp, encoding, src, srcLen, flags, statePtr,
305 dst, dstLen, srcReadPtr, dstWrotePtr, dstCharsPtr\fR)
308 \fBTcl_UtfToExternal\fR(\fIinterp, encoding, src, srcLen, flags, statePtr,
309 dst, dstLen, srcReadPtr, dstWrotePtr, dstCharsPtr\fR)
312 \fBTcl_WinTCharToUtf\fR(\fItsrc, srcLen, dstPtr\fR)
315 \fBTcl_WinUtfToTChar\fR(\fIsrc, srcLen, dstPtr\fR)
318 \fBTcl_GetEncodingName\fR(\fIencoding\fR)
321 \fBTcl_SetSystemEncoding\fR(\fIinterp, name\fR)
325 \fBTcl_GetEncodingNameFromEnvironment\fR(\fIbufPtr\fR)
329 \fBTcl_GetEncodingNames\fR(\fIinterp\fR)
332 \fBTcl_CreateEncoding\fR(\fItypePtr\fR)
336 \fBTcl_GetEncodingSearchPath\fR()
339 \fBTcl_SetEncodingSearchPath\fR(\fIsearchPath\fR)
343 \fBTcl_GetDefaultEncodingDir\fR(\fIvoid\fR)
346 \fBTcl_SetDefaultEncodingDir\fR(\fIpath\fR)
348 .AS "const Tcl_EncodingType" *dstWrotePtr in/out
349 .AP Tcl_Interp *interp in
350 Interpreter to use for error reporting, or NULL if no error reporting is
352 .AP "const char" *name in
353 Name of encoding to load.
354 .AP Tcl_Encoding encoding in
355 The encoding to query, free, or use for converting text. If \fIencoding\fR is
356 NULL, the current system encoding is used.
357 .AP Tcl_Obj *objPtr in
359 Name of encoding to get token for.
361 .AP Tcl_Encoding *encodingPtr out
363 Points to storage where encoding token is to be written.
365 .AP "const char" *src in
366 For the \fBTcl_ExternalToUtf\fR functions, an array of bytes in the
367 specified encoding that are to be converted to UTF-8. For the
368 \fBTcl_UtfToExternal\fR and \fBTcl_WinUtfToTChar\fR functions, an array of
369 UTF-8 characters to be converted to the specified encoding.
370 .AP "const TCHAR" *tsrc in
371 An array of Windows TCHAR characters to convert to UTF-8.
373 Length of \fIsrc\fR or \fItsrc\fR in bytes. If the length is negative, the
374 encoding-specific length of the string is used.
375 .AP Tcl_DString *dstPtr out
376 Pointer to an uninitialized or free \fBTcl_DString\fR in which the converted
377 result will be stored.
379 Various flag bits OR-ed together.
380 \fBTCL_ENCODING_START\fR signifies that the
381 source buffer is the first block in a (potentially multi-block) input
382 stream, telling the conversion routine to reset to an initial state and
383 perform any initialization that needs to occur before the first byte is
384 converted. \fBTCL_ENCODING_END\fR signifies that the source buffer is the last
385 block in a (potentially multi-block) input stream, telling the conversion
386 routine to perform any finalization that needs to occur after the last
387 byte is converted and then to reset to an initial state.
388 \fBTCL_ENCODING_STOPONERROR\fR signifies that the conversion routine should
389 return immediately upon reading a source character that does not exist in
390 the target encoding; otherwise a default fallback character will
391 automatically be substituted.
392 .AP Tcl_EncodingState *statePtr in/out
393 Used when converting a (generally long or indefinite length) byte stream
394 in a piece-by-piece fashion. The conversion routine stores its current
395 state in \fI*statePtr\fR after \fIsrc\fR (the buffer containing the
396 current piece) has been converted; that state information must be passed
397 back when converting the next piece of the stream so the conversion
398 routine knows what state it was in when it left off at the end of the
399 last piece. May be NULL, in which case the value specified for \fIflags\fR
400 is ignored and the source buffer is assumed to contain the complete string to
403 Buffer in which the converted result will be stored. No more than
404 \fIdstLen\fR bytes will be stored in \fIdst\fR.
406 The maximum length of the output buffer \fIdst\fR in bytes.
407 .AP int *srcReadPtr out
408 Filled with the number of bytes from \fIsrc\fR that were actually
409 converted. This may be less than the original source length if there was
410 a problem converting some source characters. May be NULL.
411 .AP int *dstWrotePtr out
412 Filled with the number of bytes that were actually stored in the output
413 buffer as a result of the conversion. May be NULL.
414 .AP int *dstCharsPtr out
415 Filled with the number of characters that correspond to the number of bytes
416 stored in the output buffer. May be NULL.
417 .AP Tcl_DString *bufPtr out
419 Storage for the prescribed system encoding name.
421 .AP "const Tcl_EncodingType" *typePtr in
422 Structure that defines a new type of encoding.
423 .AP Tcl_Obj *searchPath in
425 List of filesystem directories in which to search for encoding data files.
427 .AP "const char" *path in
428 A path to the location of the encoding file.
432 These routines convert between Tcl's internal character representation,
433 UTF-8, and character representations used by various operating systems or
434 file systems, such as Unicode, ASCII, or Shift-JIS. When operating on
435 strings, such as such as obtaining the names of files or displaying
436 characters using international fonts, the strings must be translated into
437 one or possibly multiple formats that the various system calls can use. For
438 instance, on a Japanese Unix workstation, a user might obtain a filename
439 represented in the EUC-JP file encoding and then translate the characters to
440 the jisx0208 font encoding in order to display the filename in a Tk widget.
441 The purpose of the encoding package is to help bridge the translation gap.
442 UTF-8 provides an intermediate staging ground for all the various
443 encodings. In the example above, text would be translated into UTF-8 from
444 whatever file encoding the operating system is using. Then it would be
445 translated from UTF-8 into whatever font encoding the display routines
448 Some basic encodings are compiled into Tcl. Others can be defined by the
449 user or dynamically loaded from encoding files in a
450 platform-independent manner.
453 \fBTcl_GetEncoding\fR finds an encoding given its \fIname\fR. The name may
454 refer to a built-in Tcl encoding, a user-defined encoding registered by
455 calling \fBTcl_CreateEncoding\fR, or a dynamically-loadable encoding
456 file. The return value is a token that represents the encoding and can be
457 used in subsequent calls to procedures such as \fBTcl_GetEncodingName\fR,
458 \fBTcl_FreeEncoding\fR, and \fBTcl_UtfToExternal\fR. If the name did not
459 refer to any known or loadable encoding, NULL is returned and an error
460 message is returned in \fIinterp\fR.
462 The encoding package maintains a database of all encodings currently in use.
463 The first time \fIname\fR is seen, \fBTcl_GetEncoding\fR returns an
464 encoding with a reference count of 1. If the same \fIname\fR is requested
465 further times, then the reference count for that encoding is incremented
466 without the overhead of allocating a new encoding and all its associated
469 When an \fIencoding\fR is no longer needed, \fBTcl_FreeEncoding\fR
470 should be called to release it. When an \fIencoding\fR is no longer in use
471 anywhere (i.e., it has been freed as many times as it has been gotten)
472 \fBTcl_FreeEncoding\fR will release all storage the encoding was using
473 and delete it from the database.
476 \fBTcl_GetEncodingFromObj\fR treats the string representation of
477 \fIobjPtr\fR as an encoding name, and finds an encoding with that
478 name, just as \fBTcl_GetEncoding\fR does. When an encoding is found,
479 it is cached within the \fBobjPtr\fR value for future reference, the
480 \fBTcl_Encoding\fR token is written to the storage pointed to by
481 \fIencodingPtr\fR, and the value \fBTCL_OK\fR is returned. If no such
482 encoding is found, the value \fBTCL_ERROR\fR is returned, and no
483 writing to \fB*\fR\fIencodingPtr\fR takes place. Just as with
484 \fBTcl_GetEncoding\fR, the caller should call \fBTcl_FreeEncoding\fR
485 on the resulting encoding token when that token will no longer be
489 \fBTcl_ExternalToUtfDString\fR converts a source buffer \fIsrc\fR from the
490 specified \fIencoding\fR into UTF-8. The converted bytes are stored in
491 \fIdstPtr\fR, which is then null-terminated. The caller should eventually
492 call \fBTcl_DStringFree\fR to free any information stored in \fIdstPtr\fR.
493 When converting, if any of the characters in the source buffer cannot be
494 represented in the target encoding, a default fallback character will be
495 used. The return value is a pointer to the value stored in the DString.
497 \fBTcl_ExternalToUtf\fR converts a source buffer \fIsrc\fR from the specified
498 \fIencoding\fR into UTF-8. Up to \fIsrcLen\fR bytes are converted from the
499 source buffer and up to \fIdstLen\fR converted bytes are stored in \fIdst\fR.
500 In all cases, \fI*srcReadPtr\fR is filled with the number of bytes that were
501 successfully converted from \fIsrc\fR and \fI*dstWrotePtr\fR is filled with
502 the corresponding number of bytes that were stored in \fIdst\fR. The return
503 value is one of the following:
506 All bytes of \fIsrc\fR were converted.
507 .IP \fBTCL_CONVERT_NOSPACE\fR 29
508 The destination buffer was not large enough for all of the converted data; as
509 many characters as could fit were converted though.
510 .IP \fBTCL_CONVERT_MULTIBYTE\fR 29
511 The last few bytes in the source buffer were the beginning of a multibyte
512 sequence, but more bytes were needed to complete this sequence. A
513 subsequent call to the conversion routine should pass a buffer containing
514 the unconverted bytes that remained in \fIsrc\fR plus some further bytes
515 from the source stream to properly convert the formerly split-up multibyte
517 .IP \fBTCL_CONVERT_SYNTAX\fR 29
518 The source buffer contained an invalid character sequence. This may occur
519 if the input stream has been damaged or if the input encoding method was
521 .IP \fBTCL_CONVERT_UNKNOWN\fR 29
522 The source buffer contained a character that could not be represented in
523 the target encoding and \fBTCL_ENCODING_STOPONERROR\fR was specified.
526 \fBTcl_UtfToExternalDString\fR converts a source buffer \fIsrc\fR from UTF-8
527 into the specified \fIencoding\fR. The converted bytes are stored in
528 \fIdstPtr\fR, which is then terminated with the appropriate encoding-specific
529 null. The caller should eventually call \fBTcl_DStringFree\fR to free any
530 information stored in \fIdstPtr\fR. When converting, if any of the
531 characters in the source buffer cannot be represented in the target
532 encoding, a default fallback character will be used. The return value is
533 a pointer to the value stored in the DString.
535 \fBTcl_UtfToExternal\fR converts a source buffer \fIsrc\fR from UTF-8 into
536 the specified \fIencoding\fR. Up to \fIsrcLen\fR bytes are converted from
537 the source buffer and up to \fIdstLen\fR converted bytes are stored in
538 \fIdst\fR. In all cases, \fI*srcReadPtr\fR is filled with the number of
539 bytes that were successfully converted from \fIsrc\fR and \fI*dstWrotePtr\fR
540 is filled with the corresponding number of bytes that were stored in
541 \fIdst\fR. The return values are the same as the return values for
542 \fBTcl_ExternalToUtf\fR.
544 \fBTcl_WinUtfToTChar\fR and \fBTcl_WinTCharToUtf\fR are
545 Windows-only convenience
546 functions for converting between UTF-8 and Windows strings. On Windows 95
547 (as with the Unix operating system),
548 all strings exchanged between Tcl and the operating system are
550 based. On Windows NT, some strings exchanged between Tcl and the
553 oriented while others are in Unicode. By
554 convention, in Windows a TCHAR is a character in the ANSI code page
555 on Windows 95 and a Unicode character on Windows NT.
557 If you planned to use the same
559 based interfaces on both Windows
560 95 and Windows NT, you could use \fBTcl_UtfToExternal\fR and
561 \fBTcl_ExternalToUtf\fR (or their \fBTcl_DString\fR equivalents) with an
562 encoding of NULL (the current system encoding). On the other hand,
563 if you planned to use the Unicode interface when running on Windows NT
566 interfaces when running on Windows 95, you would have
567 to perform the following type of test over and over in your program
568 (as represented in pseudo-code):
571 encoding <- Tcl_GetEncoding("unicode");
572 nativeBuffer <- Tcl_UtfToExternal(encoding, utfBuffer);
573 Tcl_FreeEncoding(encoding);
575 nativeBuffer <- Tcl_UtfToExternal(NULL, utfBuffer);
578 \fBTcl_WinUtfToTChar\fR and \fBTcl_WinTCharToUtf\fR automatically
579 handle this test and use the proper encoding based on the current
580 operating system. \fBTcl_WinUtfToTChar\fR returns a pointer to
581 a TCHAR string, and \fBTcl_WinTCharToUtf\fR expects a TCHAR string
582 pointer as the \fIsrc\fR string. Otherwise, these functions
583 behave identically to \fBTcl_UtfToExternalDString\fR and
584 \fBTcl_ExternalToUtfDString\fR.
586 \fBTcl_GetEncodingName\fR is roughly the inverse of \fBTcl_GetEncoding\fR.
587 Given an \fIencoding\fR, the return value is the \fIname\fR argument that
588 was used to create the encoding. The string returned by
589 \fBTcl_GetEncodingName\fR is only guaranteed to persist until the
590 \fIencoding\fR is deleted. The caller must not modify this string.
592 \fBTcl_SetSystemEncoding\fR sets the default encoding that should be used
593 whenever the user passes a NULL value for the \fIencoding\fR argument to
594 any of the other encoding functions. If \fIname\fR is NULL, the system
595 encoding is reset to the default system encoding, \fBbinary\fR. If the
596 name did not refer to any known or loadable encoding, \fBTCL_ERROR\fR is
597 returned and an error message is left in \fIinterp\fR. Otherwise, this
598 procedure increments the reference count of the new system encoding,
599 decrements the reference count of the old system encoding, and returns
603 \fBTcl_GetEncodingNameFromEnvironment\fR provides a means for the Tcl
604 library to report the encoding name it believes to be the correct one
605 to use as the system encoding, based on system calls and examination of
606 the environment suitable for the platform. It accepts \fIbufPtr\fR,
607 a pointer to an uninitialized or freed \fBTcl_DString\fR and writes
608 the encoding name to it. The \fBTcl_DStringValue\fR is returned.
611 \fBTcl_GetEncodingNames\fR sets the \fIinterp\fR result to a list
612 consisting of the names of all the encodings that are currently defined
613 or can be dynamically loaded, searching the encoding path specified by
614 \fBTcl_SetDefaultEncodingDir\fR. This procedure does not ensure that the
615 dynamically-loadable encoding files contain valid data, but merely that they
618 \fBTcl_CreateEncoding\fR defines a new encoding and registers the C
619 procedures that are called back to convert between the encoding and
620 UTF-8. Encodings created by \fBTcl_CreateEncoding\fR are thereafter
621 visible in the database used by \fBTcl_GetEncoding\fR. Just as with the
622 \fBTcl_GetEncoding\fR procedure, the return value is a token that
623 represents the encoding and can be used in subsequent calls to other
624 encoding functions. \fBTcl_CreateEncoding\fR returns an encoding with a
625 reference count of 1. If an encoding with the specified \fIname\fR
626 already exists, then its entry in the database is replaced with the new
627 encoding; the token for the old encoding will remain valid and continue
628 to behave as before, but users of the new token will now call the new
631 The \fItypePtr\fR argument to \fBTcl_CreateEncoding\fR contains information
632 about the name of the encoding and the procedures that will be called to
633 convert between this encoding and UTF-8. It is defined as follows:
636 typedef struct Tcl_EncodingType {
637 const char *\fIencodingName\fR;
638 Tcl_EncodingConvertProc *\fItoUtfProc\fR;
639 Tcl_EncodingConvertProc *\fIfromUtfProc\fR;
640 Tcl_EncodingFreeProc *\fIfreeProc\fR;
641 ClientData \fIclientData\fR;
646 The \fIencodingName\fR provides a string name for the encoding, by
647 which it can be referred in other procedures such as
648 \fBTcl_GetEncoding\fR. The \fItoUtfProc\fR refers to a callback
649 procedure to invoke to convert text from this encoding into UTF-8.
650 The \fIfromUtfProc\fR refers to a callback procedure to invoke to
651 convert text from UTF-8 into this encoding. The \fIfreeProc\fR refers
652 to a callback procedure to invoke when this encoding is deleted. The
653 \fIfreeProc\fR field may be NULL. The \fIclientData\fR contains an
654 arbitrary one-word value passed to \fItoUtfProc\fR, \fIfromUtfProc\fR,
655 and \fIfreeProc\fR whenever they are called. Typically, this is a
656 pointer to a data structure containing encoding-specific information
657 that can be used by the callback procedures. For instance, two very
658 similar encodings such as \fBascii\fR and \fBmacRoman\fR may use the
659 same callback procedure, but use different values of \fIclientData\fR
660 to control its behavior. The \fInullSize\fR specifies the number of
661 zero bytes that signify end-of-string in this encoding. It must be
662 \fB1\fR (for single-byte or multi-byte encodings like ASCII or
663 Shift-JIS) or \fB2\fR (for double-byte encodings like Unicode).
664 Constant-sized encodings with 3 or more bytes per character (such as
665 CNS11643) are not accepted.
667 The callback procedures \fItoUtfProc\fR and \fIfromUtfProc\fR should match the
668 type \fBTcl_EncodingConvertProc\fR:
671 typedef int Tcl_EncodingConvertProc(
672 ClientData \fIclientData\fR,
673 const char *\fIsrc\fR,
676 Tcl_EncodingState *\fIstatePtr\fR,
679 int *\fIsrcReadPtr\fR,
680 int *\fIdstWrotePtr\fR,
681 int *\fIdstCharsPtr\fR);
684 The \fItoUtfProc\fR and \fIfromUtfProc\fR procedures are called by the
685 \fBTcl_ExternalToUtf\fR or \fBTcl_UtfToExternal\fR family of functions to
686 perform the actual conversion. The \fIclientData\fR parameter to these
687 procedures is the same as the \fIclientData\fR field specified to
688 \fBTcl_CreateEncoding\fR when the encoding was created. The remaining
689 arguments to the callback procedures are the same as the arguments,
690 documented at the top, to \fBTcl_ExternalToUtf\fR or
691 \fBTcl_UtfToExternal\fR, with the following exceptions. If the
692 \fIsrcLen\fR argument to one of those high-level functions is negative,
693 the value passed to the callback procedure will be the appropriate
694 encoding-specific string length of \fIsrc\fR. If any of the \fIsrcReadPtr\fR,
695 \fIdstWrotePtr\fR, or \fIdstCharsPtr\fR arguments to one of the high-level
696 functions is NULL, the corresponding value passed to the callback
697 procedure will be a non-NULL location.
699 The callback procedure \fIfreeProc\fR, if non-NULL, should match the type
700 \fBTcl_EncodingFreeProc\fR:
702 typedef void Tcl_EncodingFreeProc(
703 ClientData \fIclientData\fR);
706 This \fIfreeProc\fR function is called when the encoding is deleted. The
707 \fIclientData\fR parameter is the same as the \fIclientData\fR field
708 specified to \fBTcl_CreateEncoding\fR when the encoding was created.
711 \fBTcl_GetEncodingSearchPath\fR and \fBTcl_SetEncodingSearchPath\fR
712 are called to access and set the list of filesystem directories searched
713 for encoding data files.
715 The value returned by \fBTcl_GetEncodingSearchPath\fR
716 is the value stored by the last successful call to
717 \fBTcl_SetEncodingSearchPath\fR. If no calls to
718 \fBTcl_SetEncodingSearchPath\fR have occurred, Tcl will compute an initial
719 value based on the environment. There is one encoding search path for the
720 entire process, shared by all threads in the process.
722 \fBTcl_SetEncodingSearchPath\fR stores \fIsearchPath\fR and returns
723 \fBTCL_OK\fR, unless \fIsearchPath\fR is not a valid Tcl list, which
724 causes \fBTCL_ERROR\fR to be returned. The elements of \fIsearchPath\fR
725 are not verified as existing readable filesystem directories. When
726 searching for encoding data files takes place, and non-existent or
727 non-readable filesystem directories on the \fIsearchPath\fR are silently
730 \fBTcl_GetDefaultEncodingDir\fR and \fBTcl_SetDefaultEncodingDir\fR
731 are obsolete interfaces best replaced with calls to
732 \fBTcl_GetEncodingSearchPath\fR and \fBTcl_SetEncodingSearchPath\fR.
733 They are called to access and set the first element of the \fIsearchPath\fR
734 list. Since Tcl searches \fIsearchPath\fR for encoding data files in
735 list order, these routines establish the
737 directory in which to find encoding data files.
740 Space would prohibit precompiling into Tcl every possible encoding
741 algorithm, so many encodings are stored on disk as dynamically-loadable
742 encoding files. This behavior also allows the user to create additional
743 encoding files that can be loaded using the same mechanism. These
744 encoding files contain information about the tables and/or escape
745 sequences used to map between an external encoding and Unicode. The
746 external encoding may consist of single-byte, multi-byte, or double-byte
749 Each dynamically-loadable encoding is represented as a text file. The
750 initial line of the file, beginning with a
753 that provides a human-readable description of the file. The next line
754 identifies the type of encoding file. It can be one of the following
757 A single-byte encoding, where one character is always one byte long in the
758 encoding. An example is \fBiso8859-1\fR, used by many European languages.
760 A double-byte encoding, where one character is always two bytes long in the
761 encoding. An example is \fBbig5\fR, used for Chinese text.
763 A multi-byte encoding, where one character may be either one or two bytes long.
764 Certain bytes are lead bytes, indicating that another byte must follow
765 and that together the two bytes represent one character. Other bytes are not
766 lead bytes and represent themselves. An example is \fBshiftjis\fR, used by
767 many Japanese computers.
769 An escape-sequence encoding, specifying that certain sequences of bytes
770 do not represent characters, but commands that describe how following bytes
771 should be interpreted.
773 The rest of the lines in the file depend on the type.
775 Cases [1], [2], and [3] are collectively referred to as table-based encoding
776 files. The lines in a table-based encoding file are in the same
777 format as this example taken from the \fBshiftjis\fR encoding (this is not
780 # Encoding file: shiftjis, multi-byte
784 0000000100020003000400050006000700080009000A000B000C000D000E000F
785 0010001100120013001400150016001700180019001A001B001C001D001E001F
786 0020002100220023002400250026002700280029002A002B002C002D002E002F
787 0030003100320033003400350036003700380039003A003B003C003D003E003F
788 0040004100420043004400450046004700480049004A004B004C004D004E004F
789 0050005100520053005400550056005700580059005A005B005C005D005E005F
790 0060006100620063006400650066006700680069006A006B006C006D006E006F
791 0070007100720073007400750076007700780079007A007B007C007D203E007F
792 0080000000000000000000000000000000000000000000000000000000000000
793 0000000000000000000000000000000000000000000000000000000000000000
794 0000FF61FF62FF63FF64FF65FF66FF67FF68FF69FF6AFF6BFF6CFF6DFF6EFF6F
795 FF70FF71FF72FF73FF74FF75FF76FF77FF78FF79FF7AFF7BFF7CFF7DFF7EFF7F
796 FF80FF81FF82FF83FF84FF85FF86FF87FF88FF89FF8AFF8BFF8CFF8DFF8EFF8F
797 FF90FF91FF92FF93FF94FF95FF96FF97FF98FF99FF9AFF9BFF9CFF9DFF9EFF9F
798 0000000000000000000000000000000000000000000000000000000000000000
799 0000000000000000000000000000000000000000000000000000000000000000
801 0000000000000000000000000000000000000000000000000000000000000000
802 0000000000000000000000000000000000000000000000000000000000000000
803 0000000000000000000000000000000000000000000000000000000000000000
804 0000000000000000000000000000000000000000000000000000000000000000
805 300030013002FF0CFF0E30FBFF1AFF1BFF1FFF01309B309C00B4FF4000A8FF3E
806 FFE3FF3F30FD30FE309D309E30034EDD30053006300730FC20152010FF0F005C
807 301C2016FF5C2026202520182019201C201DFF08FF0930143015FF3BFF3DFF5B
808 FF5D30083009300A300B300C300D300E300F30103011FF0B221200B100D70000
809 00F7FF1D2260FF1CFF1E22662267221E22342642264000B0203220332103FFE5
810 FF0400A200A3FF05FF03FF06FF0AFF2000A72606260525CB25CF25CE25C725C6
811 25A125A025B325B225BD25BC203B301221922190219121933013000000000000
812 000000000000000000000000000000002208220B2286228722822283222A2229
813 000000000000000000000000000000002227222800AC21D221D4220022030000
814 0000000000000000000000000000000000000000222022A52312220222072261
815 2252226A226B221A223D221D2235222B222C0000000000000000000000000000
816 212B2030266F266D266A2020202100B6000000000000000025EF000000000000
819 The third line of the file is three numbers. The first number is the
820 fallback character (in base 16) to use when converting from UTF-8 to this
821 encoding. The second number is a \fB1\fR if this file represents the
822 encoding for a symbol font, or \fB0\fR otherwise. The last number (in base
823 10) is how many pages of data follow.
825 Subsequent lines in the example above are pages that describe how to map
826 from the encoding into 2-byte Unicode. The first line in a page identifies
827 the page number. Following it are 256 double-byte numbers, arranged as 16
828 rows of 16 numbers. Given a character in the encoding, the high byte of
829 that character is used to select which page, and the low byte of that
830 character is used as an index to select one of the double-byte numbers in
831 that page \- the value obtained being the corresponding Unicode character.
832 By examination of the example above, one can see that the characters 0x7E
833 and 0x8163 in \fBshiftjis\fR map to 203E and 2026 in Unicode, respectively.
835 Following the first page will be all the other pages, each in the same
836 format as the first: one number identifying the page followed by 256
837 double-byte Unicode characters. If a character in the encoding maps to the
838 Unicode character 0000, it means that the character does not actually exist.
839 If all characters on a page would map to 0000, that page can be omitted.
841 Case [4] is the escape-sequence encoding file. The lines in an this type of
842 file are in the same format as this example taken from the \fBiso2022-jp\fR
846 # Encoding file: iso2022-jp, escape-driven
859 In the file, the first column represents an option and the second column
860 is the associated value. \fBinit\fR is a string to emit or expect before
861 the first character is converted, while \fBfinal\fR is a string to emit
862 or expect after the last character. All other options are names of
863 table-based encodings; the associated value is the escape-sequence that
864 marks that encoding. Tcl syntax is used for the values; in the above
865 example, for instance,
867 represents the empty string and
869 represents character 27.
871 When \fBTcl_GetEncoding\fR encounters an encoding \fIname\fR that has not
872 been loaded, it attempts to load an encoding file called \fIname\fB.enc\fR
873 from the \fBencoding\fR subdirectory of each directory that Tcl searches
874 for its script library. If the encoding file exists, but is
875 malformed, an error message will be left in \fIinterp\fR.
877 utf, encoding, convert
OSDN Git Service
