Note
TL;DR: In decode oriented use-case with big payloads JSON decoders often use disproportionate amounts of memory. I gave up on JSON and switched to Msgpack.
You should draw your own conclusions by running the test code yourself.
⸻
Based on various feedback [*] I've did the benchmarks again, using ru_maxrss instead of Valgrind and with few more implementations.
Updated results: *
- my data (240Mb)
- citylots.json (189MB)
- canada.json (2.2Mb)
I have a peculiar use-case where I need to move around big chunks of data (some client connects to some HTTP API and gets some data). For whatever reason [1], JSON was chosen as the transport format. And one day that big chunk became very big - around few hundred megabytes. And it turned out that processes doing the JSON decoding were using lots of RAM - 4.4GB for a mere 240MB JSON payload? Insane. [2]
I was using the builtin json library, and the first thing I thought - "there must be a better JSON parser". So I've started measuring ...
Now measuring memory usage is a tricky thing, you can look at ps or look around in /proc/<pid> but you'd get very coarse snapshots and would be very hard to find out the real peak usage. Luckily enough Valgrind can instrument any program to track allocations (as opposed to recompiling everything to use a custom memory allocator) and it has a really nice tool called massif.
So I've started building a little benchmark using Valgrind. My input looks like this:
{
"foo": [{
"bar": [
'A"\\ :,;\n1' * 20000000,
],
"b": [
1, 0.333, True,
],
"c": None,
}]
}
That generates a 240MB JSON with a structure pretty close to my app's problematic data.
Running valgrind --tool=massif --pages-as-heap=yes --heap=yes --threshold=0 --peak-inaccuracy=0 --max-snapshots=1000 ... for each parser gets me something like this on Python 2.7 (scroll down for results on Python 3.5):
Peak memory usage (Python 2.7):
cjson: 485.4 Mb
rapidjson: 670.5 Mb
yajl: 1,199.2 Mb
ujson: 1,862.0 Mb
jsonlib2: 2,882.7 Mb
jsonlib: 2,884.2 Mb
simplejson: 2,953.6 Mb
json: 4,397.9 Mb
Would you look at that. Now you can argue that my sample data is crazy but sadly, but that's just how my data looks sometimes. Few of the strings blow up to horrid proportions once in a while.
json has a severe weakness here, it needs a dozen more times memory than the input. WAT.
cjson is right there in my face, begging me to use it. There are some rumours that it has VeryBadBugs™ [6] but I think the lack of a bug tracker is what makes that project ultimately unappealing.
rapidjson seems to be a new player [3], however the Python 2 binding seems to have some gaps in essential parts. Still, it's interesting to at least get an idea of how it performs. The Python 3-only binding looks more mature, but sadly this app only run on Python 2 right now.
yajl and ujson appear to be mature enough but they simply still use lots of memory. There must be a better way ...
It looks like whatever I choose it's bad. There's a very good proverb [†] that applies here:
Best solution to problem is not having the problem in the first place.
Remember that time a customer asked for a thing but in fact he only needed something simpler and less costly. Talking through requirements and refining them solves lots of problems right there. This is that kind of situation. I wish I had realized I don't really need JSON at all sooner ...
I have to do more changes to switch the format of the HTTP API but that can't be worse than maintaining/fixing the cjson or rapidjson bindings myself.
If we try msgpack (and some old friends [‡], just for kicks) we get this:
Peak memory usage (Python 2):
pickle: 368.9 Mb
marshal: 368.9 Mb
msgpack: 373.2 Mb
cjson: 485.4 Mb
rapidjson: 670.4 Mb
yajl: 1,199.2 Mb
ujson: 1,862.0 Mb
jsonlib2: 2,882.7 Mb
jsonlib: 2,884.2 Mb
simplejson: 2,953.6 Mb
json: 4,397.9 Mb
If you look at the test code you'll notice that I use msgpack with very specific options. Because the initial version of Msgpack wasn't very smart about strings (it had a single string type [5]) some specific options are needed:
msgpack.dumps(obj, use_bin_type=True) - use a different type for byte-strings. By default Msgpack will lump all kinds of strings into same type and you can't tell what the original type was.
On Python 2:
- str goes into the bin type
- unicode goes into the string type
On Python 3:
- bytes goes into the bin type
- str goes into the string type
msgpack.loads(payload, encoding='utf8') - decode the strings (so you get unicode back).
What about speed? *
Using pytest-benchmark we get this [5]:
Speed (Python 2.7):
-----------------------------------------------
Name (time in ms) Min
-----------------------------------------------
test_speed[marshal] 59.2630 (1.0)
test_speed[pickle] 59.4530 (1.00)
test_speed[msgpack] 59.7100 (1.01)
test_speed[rapidjson] 443.0561 (7.48)
test_speed[cjson] 676.6071 (11.42)
test_speed[ujson] 681.8101 (11.50)
test_speed[yajl] 1,590.4601 (26.84)
test_speed[jsonlib] 1,873.3799 (31.61)
test_speed[jsonlib2] 2,006.7949 (33.86)
test_speed[simplejson] 3,592.2401 (60.62)
test_speed[json] 5,193.2762 (87.63)
-----------------------------------------------
Only the minimum time is shown. This is intentional - run the test code on your own hardware if you care about anything else.
Python 3 *
This app where I had the issue runs only on Python 2 for a very good (and also sad) reason. But no reason to dig myself further into a hole - gotta see how this performs on the latest and greatest. It will get ported one day ...
Peak memory usage (Python 3.5):
marshal: 372.1 Mb
pickle: 372.9 Mb
msgpack: 376.6 Mb
rapidjson: 668.6 Mb
yajl: 687.3 Mb
ujson: 1,578.9 Mb
json: 3,422.3 Mb
simplejson: 6,681.4 Mb
Speed (Python 3.5)
-----------------------------------------------
Name (time in ms) Min
-----------------------------------------------
test_speed[msgpack] 69.0613 (1.0)
test_speed[pickle] 69.9465 (1.01)
test_speed[marshal] 74.9914 (1.09)
test_speed[rapidjson] 337.5243 (4.89)
test_speed[ujson] 902.8647 (13.07)
test_speed[yajl] 1,195.4298 (17.31)
test_speed[json] 4,404.9523 (63.78)
test_speed[simplejson] 6,524.9919 (94.48)
-----------------------------------------------
No cjson or jsonlib on Python 3. I don't even know what's the story behind jsonlib2. Looks like Msgpack is a safe bet here.
Different kind of data *
Now this is highly skewed towards some might call a completely atypical data shape. So I advise you take the test code and run the benchmarks with your own data.
But if you're lazy here are some results with different kinds of data, just to get an idea of how much the input can change memory use and speed.
Lots off small objects *
The 189MB citylots.json gets us wildly different results.
It appears simplejson works way better on small objects, and json is quite improved on Python 3:
Peak memory usage (Python 2.7):
simplejson: 1,171.7 Mb
cjson: 1,304.2 Mb
msgpack: 1,357.2 Mb
marshal: 1,385.2 Mb
yajl: 1,457.1 Mb
json: 1,468.0 Mb
rapidjson: 1,561.6 Mb
pickle: 1,854.1 Mb
jsonlib2: 2,134.9 Mb
jsonlib: 2,137.0 Mb
ujson: 2,149.9 Mb
Peak memory usage (Python 3.5):
marshal: 951.0 Mb
json: 1,059.8 Mb
simplejson: 1,063.6 Mb
pickle: 1,098.4 Mb
msgpack: 1,115.9 Mb
yajl: 1,226.6 Mb
rapidjson: 1,404.9 Mb
ujson: 2,077.6 Mb
Speed:
Speed (Python 2.7):
-----------------------------------------------
Name (time in ms) Min
-----------------------------------------------
test_speed[marshal] 3.9999 (1.0)
test_speed[ujson] 4.2569 (1.06)
test_speed[simplejson] 5.1105 (1.28)
test_speed[cjson] 5.2355 (1.31)
test_speed[msgpack] 5.9742 (1.49)
test_speed[yajl] 6.1059 (1.53)
test_speed[json] 6.3822 (1.60)
test_speed[jsonlib2] 6.7880 (1.70)
test_speed[jsonlib] 6.9587 (1.74)
test_speed[rapidjson] 7.4734 (1.87)
test_speed[pickle] 18.8649 (4.72)
-----------------------------------------------
Speed (Python 3.5):
-----------------------------------------------
Name (time in ms) Min
-----------------------------------------------
test_speed[marshal] 1.1784 (1.0)
test_speed[ujson] 3.6378 (3.09)
test_speed[msgpack] 3.7226 (3.16)
test_speed[pickle] 3.7739 (3.20)
test_speed[rapidjson] 4.1379 (3.51)
test_speed[json] 5.1150 (4.34)
test_speed[simplejson] 5.1530 (4.37)
test_speed[yajl] 5.9426 (5.04)
-----------------------------------------------
Smaller data *
The tiny 2.2MB canada.json, again, gives us very different results. Memory use becomes irrelevant:
Peak memory usage (Python 2.7):
marshal: 35.2 Mb
cjson: 38.9 Mb
yajl: 39.0 Mb
json: 39.3 Mb
msgpack: 39.5 Mb
simplejson: 40.5 Mb
pickle: 42.1 Mb
jsonlib2: 47.4 Mb
rapidjson: 48.5 Mb
jsonlib: 48.8 Mb
ujson: 50.9 Mb
Peak memory usage (Python 3.5):
marshal: 38.3 Mb
pickle: 40.4 Mb
yajl: 42.1 Mb
json: 42.2 Mb
msgpack: 42.7 Mb
simplejson: 45.3 Mb
rapidjson: 52.3 Mb
ujson: 55.5 Mb
And speed is again different:
Speed (Python 2.7):
-----------------------------------------------
Name (time in ms) Min
-----------------------------------------------
test_speed[msgpack] 12.3210 (1.0)
test_speed[marshal] 15.1060 (1.23)
test_speed[ujson] 19.8410 (1.61)
test_speed[json] 48.0320 (3.90)
test_speed[cjson] 48.6560 (3.95)
test_speed[simplejson] 52.0709 (4.23)
test_speed[yajl] 62.1090 (5.04)
test_speed[jsonlib2] 81.6209 (6.62)
test_speed[jsonlib] 83.2670 (6.76)
test_speed[rapidjson] 102.3500 (8.31)
test_speed[pickle] 258.6429 (20.99)
-----------------------------------------------
Speed (Python 3.5):
-----------------------------------------------
Name (time in ms) Min
-----------------------------------------------
test_speed[marshal] 10.0271 (1.0)
test_speed[msgpack] 10.2731 (1.02)
test_speed[pickle] 17.2853 (1.72)
test_speed[ujson] 17.7634 (1.77)
test_speed[rapidjson] 25.6136 (2.55)
test_speed[json] 54.8634 (5.47)
test_speed[yajl] 58.3519 (5.82)
test_speed[simplejson] 65.0913 (6.49)
-----------------------------------------------
::...
免责声明:
当前网页内容, 由
大妈 ZoomQuiet
使用工具:
ScrapBook :: Firefox Extension
人工从互联网中收集并分享;
内容版权归原作者所有;
本人对内容的有效性/合法性不承担任何强制性责任.
若有不妥, 欢迎评注提醒:
或是邮件反馈可也:
askdama[AT]googlegroups.com
订阅 substack 体验古早写作:
点击注册~> 获得 100$ 体验券:
关注公众号, 持续获得相关各种嗯哼:
自怼圈/年度番新
关于 ~ DebugUself with DAMA ;-)
粤ICP备18025058号-1
公安备案号:
44049002000656
...::