1 // Copyright (c) 2012-2018 Ugorji Nwoke. All rights reserved.
2 // Use of this source code is governed by a MIT license found in the LICENSE file.
5 Package codec provides a
6 High Performance, Feature-Rich Idiomatic Go 1.4+ codec/encoding library
7 for binc, msgpack, cbor, json.
9 Supported Serialization formats are:
11 - msgpack: https://github.com/msgpack/msgpack
12 - binc: http://github.com/ugorji/binc
13 - cbor: http://cbor.io http://tools.ietf.org/html/rfc7049
14 - json: http://json.org http://tools.ietf.org/html/rfc7159
19 go get github.com/ugorji/go/codec
21 This package will carefully use 'unsafe' for performance reasons in specific places.
22 You can build without unsafe use by passing the safe or appengine tag
23 i.e. 'go install -tags=safe ...'. Note that unsafe is only supported for the last 3
24 go sdk versions e.g. current go release is go 1.9, so we support unsafe use only from
25 go 1.7+ . This is because supporting unsafe requires knowledge of implementation details.
27 For detailed usage information, read the primer at http://ugorji.net/blog/go-codec-primer .
29 The idiomatic Go support is as seen in other encoding packages in
30 the standard library (ie json, xml, gob, etc).
32 Rich Feature Set includes:
34 - Simple but extremely powerful and feature-rich API
35 - Support for go1.4 and above, while selectively using newer APIs for later releases
36 - Excellent code coverage ( > 90% )
37 - Very High Performance.
38 Our extensive benchmarks show us outperforming Gob, Json, Bson, etc by 2-4X.
39 - Careful selected use of 'unsafe' for targeted performance gains.
40 100% mode exists where 'unsafe' is not used at all.
41 - Lock-free (sans mutex) concurrency for scaling to 100's of cores
42 - In-place updates during decode, with option to zero the value in maps and slices prior to decode
43 - Coerce types where appropriate
44 e.g. decode an int in the stream into a float, decode numbers from formatted strings, etc
46 Overflows, nil maps/slices, nil values in streams are handled correctly
47 - Standard field renaming via tags
48 - Support for omitting empty fields during an encoding
49 - Encoding from any value and decoding into pointer to any value
50 (struct, slice, map, primitives, pointers, interface{}, etc)
51 - Extensions to support efficient encoding/decoding of any named types
52 - Support encoding.(Binary|Text)(M|Unm)arshaler interfaces
53 - Support IsZero() bool to determine if a value is a zero value.
54 Analogous to time.Time.IsZero() bool.
55 - Decoding without a schema (into a interface{}).
56 Includes Options to configure what specific map or slice type to use
57 when decoding an encoded list or map into a nil interface{}
58 - Mapping a non-interface type to an interface, so we can decode appropriately
59 into any interface type with a correctly configured non-interface value.
60 - Encode a struct as an array, and decode struct from an array in the data stream
61 - Option to encode struct keys as numbers (instead of strings)
62 (to support structured streams with fields encoded as numeric codes)
63 - Comprehensive support for anonymous fields
64 - Fast (no-reflection) encoding/decoding of common maps and slices
65 - Code-generation for faster performance.
66 - Support binary (e.g. messagepack, cbor) and text (e.g. json) formats
67 - Support indefinite-length formats to enable true streaming
68 (for formats which support it e.g. json, cbor)
69 - Support canonical encoding, where a value is ALWAYS encoded as same sequence of bytes.
70 This mostly applies to maps, where iteration order is non-deterministic.
71 - NIL in data stream decoded as zero value
72 - Never silently skip data when decoding.
73 User decides whether to return an error or silently skip data when keys or indexes
74 in the data stream do not map to fields in the struct.
75 - Detect and error when encoding a cyclic reference (instead of stack overflow shutdown)
76 - Encode/Decode from/to chan types (for iterative streaming support)
77 - Drop-in replacement for encoding/json. `json:` key in struct tag supported.
78 - Provides a RPC Server and Client Codec for net/rpc communication protocol.
79 - Handle unique idiosyncrasies of codecs e.g.
80 - For messagepack, configure how ambiguities in handling raw bytes are resolved
81 - For messagepack, provide rpc server/client codec to support
82 msgpack-rpc protocol defined at:
83 https://github.com/msgpack-rpc/msgpack-rpc/blob/master/spec.md
87 Users can register a function to handle the encoding or decoding of
90 There are no restrictions on what the custom type can be. Some examples:
95 type MyStructWithUnexportedFields struct { a int; b bool; c []int; }
96 type GifImage struct { ... }
98 As an illustration, MyStructWithUnexportedFields would normally be
99 encoded as an empty map because it has no exported fields, while UUID
100 would be encoded as a string. However, with extension support, you can
101 encode any of these however you like.
103 Custom Encoding and Decoding
105 This package maintains symmetry in the encoding and decoding halfs.
106 We determine how to encode or decode by walking this decision tree
108 - is type a codec.Selfer?
109 - is there an extension registered for the type?
110 - is format binary, and is type a encoding.BinaryMarshaler and BinaryUnmarshaler?
111 - is format specifically json, and is type a encoding/json.Marshaler and Unmarshaler?
112 - is format text-based, and type an encoding.TextMarshaler?
113 - else we use a pair of functions based on the "kind" of the type e.g. map, slice, int64, etc
115 This symmetry is important to reduce chances of issues happening because the
116 encoding and decoding sides are out of sync e.g. decoded via very specific
117 encoding.TextUnmarshaler but encoded via kind-specific generalized mode.
119 Consequently, if a type only defines one-half of the symmetry
120 (e.g. it implements UnmarshalJSON() but not MarshalJSON() ),
121 then that type doesn't satisfy the check and we will continue walking down the
126 RPC Client and Server Codecs are implemented, so the codecs can be used
127 with the standard net/rpc package.
131 The Handle is SAFE for concurrent READ, but NOT SAFE for concurrent modification.
133 The Encoder and Decoder are NOT safe for concurrent use.
135 Consequently, the usage model is basically:
137 - Create and initialize the Handle before any use.
138 Once created, DO NOT modify it.
139 - Multiple Encoders or Decoders can now use the Handle concurrently.
140 They only read information off the Handle (never write).
141 - However, each Encoder or Decoder MUST not be used concurrently
142 - To re-use an Encoder/Decoder, call Reset(...) on it first.
143 This allows you use state maintained on the Encoder/Decoder.
147 // create and configure Handle
150 mh codec.MsgpackHandle
154 mh.MapType = reflect.TypeOf(map[string]interface{}(nil))
156 // configure extensions
157 // e.g. for msgpack, define functions and enable Time support for tag 1
158 // mh.SetExt(reflect.TypeOf(time.Time{}), 1, myExt)
160 // create and use decoder/encoder
165 h = &bh // or mh to use msgpack
168 dec = codec.NewDecoder(r, h)
169 dec = codec.NewDecoderBytes(b, h)
172 enc = codec.NewEncoder(w, h)
173 enc = codec.NewEncoderBytes(&b, h)
179 conn, err := listener.Accept()
180 rpcCodec := codec.GoRpc.ServerCodec(conn, h)
181 //OR rpcCodec := codec.MsgpackSpecRpc.ServerCodec(conn, h)
182 rpc.ServeCodec(rpcCodec)
186 //RPC Communication (client side)
187 conn, err = net.Dial("tcp", "localhost:5555")
188 rpcCodec := codec.GoRpc.ClientCodec(conn, h)
189 //OR rpcCodec := codec.MsgpackSpecRpc.ClientCodec(conn, h)
190 client := rpc.NewClientWithCodec(rpcCodec)
194 To run tests, use the following:
198 To run the full suite of tests, use the following:
200 go test -tags alltests -run Suite
202 You can run the tag 'safe' to run tests or build in safe mode. e.g.
204 go test -tags safe -run Json
205 go test -tags "alltests safe" -run Suite
209 Please see http://github.com/ugorji/go-codec-bench .
213 Struct fields matching the following are ignored during encoding and decoding
214 - struct tag value set to -
215 - func, complex numbers, unsafe pointers
216 - unexported and not embedded
217 - unexported and embedded and not struct kind
218 - unexported and embedded pointers (from go1.10)
220 Every other field in a struct will be encoded/decoded.
222 Embedded fields are encoded as if they exist in the top-level struct,
223 with some caveats. See Encode documentation.
229 // - For Go 1.11, when mid-stack inlining is enabled,
230 // we should use committed functions for writeXXX and readXXX calls.
231 // This involves uncommenting the methods for decReaderSwitch and encWriterSwitch
232 // and using those (decReaderSwitch and encWriterSwitch) in all handles
233 // instead of encWriter and decReader.
234 // The benefit is that, for the (En|De)coder over []byte, the encWriter/decReader
235 // will be inlined, giving a performance bump for that typical case.
236 // However, it will only be inlined if mid-stack inlining is enabled,
237 // as we call panic to raise errors, and panic currently prevents inlining.
240 // - To make Handle comparable, make extHandle in BasicHandle a non-embedded pointer,
241 // and use overlay methods on *BasicHandle to call through to extHandle after initializing
242 // the "xh *extHandle" to point to a real slice.
244 // BEFORE EACH RELEASE:
245 // - Look through and fix padding for each type, to eliminate false sharing
246 // - critical shared objects that are read many times
249 // decNaked, decNakedContainers, codecFner, typeInfoLoadArray,
250 // - small objects allocated independently, that we read/use much across threads:
252 // - Objects allocated independently and used a lot
254 // xxxHandle, xxxEncDriver, xxxDecDriver (xxx = json, msgpack, cbor, binc, simple)
255 // - In all above, arrange values modified together to be close to each other.
257 // For all of these, either ensure that they occupy full cache lines,
258 // or ensure that the things just past the cache line boundary are hardly read/written
259 // e.g. JsonHandle.RawBytesExt - which is copied into json(En|De)cDriver at init
261 // Occupying full cache lines means they occupy 8*N words (where N is an integer).
262 // Check this out by running: ./run.sh -z
263 // - look at those tagged ****, meaning they are not occupying full cache lines
264 // - look at those tagged <<<<, meaning they are larger than 32 words (something to watch)
265 // - Run "golint -min_confidence 0.81"