1[](https://goreportcard.com/report/github.com/buger/jsonparser) 
2# Alternative JSON parser for Go (10x times faster standard library)
3
4It does not require you to know the structure of the payload (eg. create structs), and allows accessing fields by providing the path to them. It is up to **10 times faster** than standard `encoding/json` package (depending on payload size and usage), **allocates no memory**. See benchmarks below.
5
6## Rationale
7Originally I made this for a project that relies on a lot of 3rd party APIs that can be unpredictable and complex.
8I love simplicity and prefer to avoid external dependecies. `encoding/json` requires you to know exactly your data structures, or if you prefer to use `map[string]interface{}` instead, it will be very slow and hard to manage.
9I investigated what's on the market and found that most libraries are just wrappers around `encoding/json`, there is few options with own parsers (`ffjson`, `easyjson`), but they still requires you to create data structures.
10
11
12Goal of this project is to push JSON parser to the performance limits and not sacrifice with compliance and developer user experience.
13
14## Example
15For the given JSON our goal is to extract the user's full name, number of github followers and avatar.
16
17```go
18import "github.com/buger/jsonparser"
19
20...
21
22data := []byte(`{
23 "person": {
24 "name": {
25 "first": "Leonid",
26 "last": "Bugaev",
27 "fullName": "Leonid Bugaev"
28 },
29 "github": {
30 "handle": "buger",
31 "followers": 109
32 },
33 "avatars": [
34 { "url": "https://avatars1.githubusercontent.com/u/14009?v=3&s=460", "type": "thumbnail" }
35 ]
36 },
37 "company": {
38 "name": "Acme"
39 }
40}`)
41
42// You can specify key path by providing arguments to Get function
43jsonparser.Get(data, "person", "name", "fullName")
44
45// There is `GetInt` and `GetBoolean` helpers if you exactly know key data type
46jsonparser.GetInt(data, "person", "github", "followers")
47
48// When you try to get object, it will return you []byte slice pointer to data containing it
49// In `company` it will be `{"name": "Acme"}`
50jsonparser.Get(data, "company")
51
52// If the key doesn't exist it will throw an error
53var size int64
54if value, err := jsonparser.GetInt(data, "company", "size"); err == nil {
55 size = value
56}
57
58// You can use `ArrayEach` helper to iterate items [item1, item2 .... itemN]
59jsonparser.ArrayEach(data, func(value []byte, dataType jsonparser.ValueType, offset int, err error) {
60 fmt.Println(jsonparser.Get(value, "url"))
61}, "person", "avatars")
62
63// Or use can access fields by index!
64jsonparser.GetString(data, "person", "avatars", "[0]", "url")
65
66// You can use `ObjectEach` helper to iterate objects { "key1":object1, "key2":object2, .... "keyN":objectN }
67jsonparser.ObjectEach(data, func(key []byte, value []byte, dataType jsonparser.ValueType, offset int) error {
68 fmt.Printf("Key: '%s'\n Value: '%s'\n Type: %s\n", string(key), string(value), dataType)
69 return nil
70}, "person", "name")
71
72// The most efficient way to extract multiple keys is `EachKey`
73
74paths := [][]string{
75 []string{"person", "name", "fullName"},
76 []string{"person", "avatars", "[0]", "url"},
77 []string{"company", "url"},
78}
79jsonparser.EachKey(data, func(idx int, value []byte, vt jsonparser.ValueType, err error){
80 switch idx {
81 case 0: // []string{"person", "name", "fullName"}
82 ...
83 case 1: // []string{"person", "avatars", "[0]", "url"}
84 ...
85 case 2: // []string{"company", "url"},
86 ...
87 }
88}, paths...)
89
90// For more information see docs below
91```
92
93## Need to speedup your app?
94
95I'm available for consulting and can help you push your app performance to the limits. Ping me at: leonsbox@gmail.com.
96
97## Reference
98
99Library API is really simple. You just need the `Get` method to perform any operation. The rest is just helpers around it.
100
101You also can view API at [godoc.org](https://godoc.org/github.com/buger/jsonparser)
102
103
104### **`Get`**
105```go
106func Get(data []byte, keys ...string) (value []byte, dataType jsonparser.ValueType, offset int, err error)
107```
108Receives data structure, and key path to extract value from.
109
110Returns:
111* `value` - Pointer to original data structure containing key value, or just empty slice if nothing found or error
112* `dataType` - Can be: `NotExist`, `String`, `Number`, `Object`, `Array`, `Boolean` or `Null`
113* `offset` - Offset from provided data structure where key value ends. Used mostly internally, for example for `ArrayEach` helper.
114* `err` - If the key is not found or any other parsing issue, it should return error. If key not found it also sets `dataType` to `NotExist`
115
116Accepts multiple keys to specify path to JSON value (in case of quering nested structures).
117If no keys are provided it will try to extract the closest JSON value (simple ones or object/array), useful for reading streams or arrays, see `ArrayEach` implementation.
118
119Note that keys can be an array indexes: `jsonparser.GetInt("person", "avatars", "[0]", "url")`, pretty cool, yeah?
120
121### **`GetString`**
122```go
123func GetString(data []byte, keys ...string) (val string, err error)
124```
125Returns strings properly handing escaped and unicode characters. Note that this will cause additional memory allocations.
126
127### **`GetUnsafeString`**
128If you need string in your app, and ready to sacrifice with support of escaped symbols in favor of speed. It returns string mapped to existing byte slice memory, without any allocations:
129```go
130s, _, := jsonparser.GetUnsafeString(data, "person", "name", "title")
131switch s {
132 case 'CEO':
133 ...
134 case 'Engineer'
135 ...
136 ...
137}
138```
139Note that `unsafe` here means that your string will exist until GC will free underlying byte slice, for most of cases it means that you can use this string only in current context, and should not pass it anywhere externally: through channels or any other way.
140
141
142### **`GetBoolean`**, **`GetInt`** and **`GetFloat`**
143```go
144func GetBoolean(data []byte, keys ...string) (val bool, err error)
145
146func GetFloat(data []byte, keys ...string) (val float64, err error)
147
148func GetInt(data []byte, keys ...string) (val int64, err error)
149```
150If you know the key type, you can use the helpers above.
151If key data type do not match, it will return error.
152
153### **`ArrayEach`**
154```go
155func ArrayEach(data []byte, cb func(value []byte, dataType jsonparser.ValueType, offset int, err error), keys ...string)
156```
157Needed for iterating arrays, accepts a callback function with the same return arguments as `Get`.
158
159### **`ObjectEach`**
160```go
161func ObjectEach(data []byte, callback func(key []byte, value []byte, dataType ValueType, offset int) error, keys ...string) (err error)
162```
163Needed for iterating object, accepts a callback function. Example:
164```go
165var handler func([]byte, []byte, jsonparser.ValueType, int) error
166handler = func(key []byte, value []byte, dataType jsonparser.ValueType, offset int) error {
167 //do stuff here
168}
169jsonparser.ObjectEach(myJson, handler)
170```
171
172
173### **`EachKey`**
174```go
175func EachKey(data []byte, cb func(idx int, value []byte, dataType jsonparser.ValueType, err error), paths ...[]string)
176```
177When you need to read multiple keys, and you do not afraid of low-level API `EachKey` is your friend. It read payload only single time, and calls callback function once path is found. For example when you call multiple times `Get`, it has to process payload multiple times, each time you call it. Depending on payload `EachKey` can be multiple times faster than `Get`. Path can use nested keys as well!
178
179```go
180paths := [][]string{
181 []string{"uuid"},
182 []string{"tz"},
183 []string{"ua"},
184 []string{"st"},
185}
186var data SmallPayload
187
188jsonparser.EachKey(smallFixture, func(idx int, value []byte, vt jsonparser.ValueType, err error){
189 switch idx {
190 case 0:
191 data.Uuid, _ = value
192 case 1:
193 v, _ := jsonparser.ParseInt(value)
194 data.Tz = int(v)
195 case 2:
196 data.Ua, _ = value
197 case 3:
198 v, _ := jsonparser.ParseInt(value)
199 data.St = int(v)
200 }
201}, paths...)
202```
203
204### **`Set`**
205```go
206func Set(data []byte, setValue []byte, keys ...string) (value []byte, err error)
207```
208Receives existing data structure, key path to set, and value to set at that key. *This functionality is experimental.*
209
210Returns:
211* `value` - Pointer to original data structure with updated or added key value.
212* `err` - If any parsing issue, it should return error.
213
214Accepts multiple keys to specify path to JSON value (in case of updating or creating nested structures).
215
216Note that keys can be an array indexes: `jsonparser.Set(data, []byte("http://github.com"), "person", "avatars", "[0]", "url")`
217
218### **`Delete`**
219```go
220func Delete(data []byte, keys ...string) value []byte
221```
222Receives existing data structure, and key path to delete. *This functionality is experimental.*
223
224Returns:
225* `value` - Pointer to original data structure with key path deleted if it can be found. If there is no key path, then the whole data structure is deleted.
226
227Accepts multiple keys to specify path to JSON value (in case of updating or creating nested structures).
228
229Note that keys can be an array indexes: `jsonparser.Delete(data, "person", "avatars", "[0]", "url")`
230
231
232## What makes it so fast?
233* It does not rely on `encoding/json`, `reflection` or `interface{}`, the only real package dependency is `bytes`.
234* Operates with JSON payload on byte level, providing you pointers to the original data structure: no memory allocation.
235* No automatic type conversions, by default everything is a []byte, but it provides you value type, so you can convert by yourself (there is few helpers included).
236* Does not parse full record, only keys you specified
237
238
239## Benchmarks
240
241There are 3 benchmark types, trying to simulate real-life usage for small, medium and large JSON payloads.
242For each metric, the lower value is better. Time/op is in nanoseconds. Values better than standard encoding/json marked as bold text.
243Benchmarks run on standard Linode 1024 box.
244
245Compared libraries:
246* https://golang.org/pkg/encoding/json
247* https://github.com/Jeffail/gabs
248* https://github.com/a8m/djson
249* https://github.com/bitly/go-simplejson
250* https://github.com/antonholmquist/jason
251* https://github.com/mreiferson/go-ujson
252* https://github.com/ugorji/go/codec
253* https://github.com/pquerna/ffjson
254* https://github.com/mailru/easyjson
255* https://github.com/buger/jsonparser
256
257#### TLDR
258If you want to skip next sections we have 2 winner: `jsonparser` and `easyjson`.
259`jsonparser` is up to 10 times faster than standard `encoding/json` package (depending on payload size and usage), and almost infinitely (literally) better in memory consumption because it operates with data on byte level, and provide direct slice pointers.
260`easyjson` wins in CPU in medium tests and frankly i'm impressed with this package: it is remarkable results considering that it is almost drop-in replacement for `encoding/json` (require some code generation).
261
262It's hard to fully compare `jsonparser` and `easyjson` (or `ffson`), they a true parsers and fully process record, unlike `jsonparser` which parse only keys you specified.
263
264If you searching for replacement of `encoding/json` while keeping structs, `easyjson` is an amazing choice. If you want to process dynamic JSON, have memory constrains, or more control over your data you should try `jsonparser`.
265
266`jsonparser` performance heavily depends on usage, and it works best when you do not need to process full record, only some keys. The more calls you need to make, the slower it will be, in contrast `easyjson` (or `ffjson`, `encoding/json`) parser record only 1 time, and then you can make as many calls as you want.
267
268With great power comes great responsibility! :)
269
270
271#### Small payload
272
273Each test processes 190 bytes of http log as a JSON record.
274It should read multiple fields.
275https://github.com/buger/jsonparser/blob/master/benchmark/benchmark_small_payload_test.go
276
277Library | time/op | bytes/op | allocs/op
278 ------ | ------- | -------- | -------
279encoding/json struct | 7879 | 880 | 18
280encoding/json interface{} | 8946 | 1521 | 38
281Jeffail/gabs | 10053 | 1649 | 46
282bitly/go-simplejson | 10128 | 2241 | 36
283antonholmquist/jason | 27152 | 7237 | 101
284github.com/ugorji/go/codec | 8806 | 2176 | 31
285mreiferson/go-ujson | **7008** | **1409** | 37
286a8m/djson | 3862 | 1249 | 30
287pquerna/ffjson | **3769** | **624** | **15**
288mailru/easyjson | **2002** | **192** | **9**
289buger/jsonparser | **1367** | **0** | **0**
290buger/jsonparser (EachKey API) | **809** | **0** | **0**
291
292Winners are ffjson, easyjson and jsonparser, where jsonparser is up to 9.8x faster than encoding/json and 4.6x faster than ffjson, and slightly faster than easyjson.
293If you look at memory allocation, jsonparser has no rivals, as it makes no data copy and operates with raw []byte structures and pointers to it.
294
295#### Medium payload
296
297Each test processes a 2.4kb JSON record (based on Clearbit API).
298It should read multiple nested fields and 1 array.
299
300https://github.com/buger/jsonparser/blob/master/benchmark/benchmark_medium_payload_test.go
301
302| Library | time/op | bytes/op | allocs/op |
303| ------- | ------- | -------- | --------- |
304| encoding/json struct | 57749 | 1336 | 29 |
305| encoding/json interface{} | 79297 | 10627 | 215 |
306| Jeffail/gabs | 83807 | 11202 | 235 |
307| bitly/go-simplejson | 88187 | 17187 | 220 |
308| antonholmquist/jason | 94099 | 19013 | 247 |
309| github.com/ugorji/go/codec | 114719 | 6712 | 152 |
310| mreiferson/go-ujson | **56972** | 11547 | 270 |
311| a8m/djson | 28525 | 10196 | 198 |
312| pquerna/ffjson | **20298** | **856** | **20** |
313| mailru/easyjson | **10512** | **336** | **12** |
314| buger/jsonparser | **15955** | **0** | **0** |
315| buger/jsonparser (EachKey API) | **8916** | **0** | **0** |
316
317The difference between ffjson and jsonparser in CPU usage is smaller, while the memory consumption difference is growing. On the other hand `easyjson` shows remarkable performance for medium payload.
318
319`gabs`, `go-simplejson` and `jason` are based on encoding/json and map[string]interface{} and actually only helpers for unstructured JSON, their performance correlate with `encoding/json interface{}`, and they will skip next round.
320`go-ujson` while have its own parser, shows same performance as `encoding/json`, also skips next round. Same situation with `ugorji/go/codec`, but it showed unexpectedly bad performance for complex payloads.
321
322
323#### Large payload
324
325Each test processes a 24kb JSON record (based on Discourse API)
326It should read 2 arrays, and for each item in array get a few fields.
327Basically it means processing a full JSON file.
328
329https://github.com/buger/jsonparser/blob/master/benchmark/benchmark_large_payload_test.go
330
331| Library | time/op | bytes/op | allocs/op |
332| --- | --- | --- | --- |
333| encoding/json struct | 748336 | 8272 | 307 |
334| encoding/json interface{} | 1224271 | 215425 | 3395 |
335| a8m/djson | 510082 | 213682 | 2845 |
336| pquerna/ffjson | **312271** | **7792** | **298** |
337| mailru/easyjson | **154186** | **6992** | **288** |
338| buger/jsonparser | **85308** | **0** | **0** |
339
340`jsonparser` now is a winner, but do not forget that it is way more lightweight parser than `ffson` or `easyjson`, and they have to parser all the data, while `jsonparser` parse only what you need. All `ffjson`, `easysjon` and `jsonparser` have their own parsing code, and does not depend on `encoding/json` or `interface{}`, thats one of the reasons why they are so fast. `easyjson` also use a bit of `unsafe` package to reduce memory consuption (in theory it can lead to some unexpected GC issue, but i did not tested enough)
341
342Also last benchmark did not included `EachKey` test, because in this particular case we need to read lot of Array values, and using `ArrayEach` is more efficient.
343
344## Questions and support
345
346All bug-reports and suggestions should go though Github Issues.
347
348## Contributing
349
3501. Fork it
3512. Create your feature branch (git checkout -b my-new-feature)
3523. Commit your changes (git commit -am 'Added some feature')
3534. Push to the branch (git push origin my-new-feature)
3545. Create new Pull Request
355
356## Development
357
358All my development happens using Docker, and repo include some Make tasks to simplify development.
359
360* `make build` - builds docker image, usually can be called only once
361* `make test` - run tests
362* `make fmt` - run go fmt
363* `make bench` - run benchmarks (if you need to run only single benchmark modify `BENCHMARK` variable in make file)
364* `make profile` - runs benchmark and generate 3 files- `cpu.out`, `mem.mprof` and `benchmark.test` binary, which can be used for `go tool pprof`
365* `make bash` - enter container (i use it for running `go tool pprof` above)