• 设为首页
  • 点击收藏
  • 手机版
    手机扫一扫访问
    迪恩网络手机版
  • 关注官方公众号
    微信扫一扫关注
    公众号

fangq/jsonlab: JSONLab: compact, portable, robust JSON/binary-JSON encoder/decod ...

原作者: [db:作者] 来自: 网络 收藏 邀请

开源软件名称(OpenSource Name):

fangq/jsonlab

开源软件地址(OpenSource Url):

https://github.com/fangq/jsonlab

开源编程语言(OpenSource Language):

MATLAB 99.3%

开源软件介绍(OpenSource Introduction):

https://neurojson.org/wiki/upload/neurojson_banner_long.png

JSONLab: compact, portable, robust JSON/binary-JSON encoder/decoder for MATLAB/Octave

https://travis-ci.com/fangq/jsonlab.svg?branch=master

Table of Contents

What's New

We are excited to announce that the JSONLab project, as the official reference library for both the JData and BJData specifications, is funded by the US National Institute of Health (NIH) as part of the NeuroJSON project (https://neurojson.org). The goal of the NeuroJSON project is to develop human-readable, scalable and future-proof neuroimaging data standards and data sharing services. All data produced from the NeuroJSON project will be using JSON/Binary JData formats as the underlying serialization standards and use the lightweight JData specification as language-independent data annotation standard, all of which have been evolved from the over a decade development of JSONLab.

JSONLab v2.9.8 - code named "Micronus - beta" - is the beta-release of the next milestone - JSONLab v3.0 - containing a number of key feature enhancement and bug fixes. The major new features include

  1. exporting JSON Memory-Map for rapid disk-map like JSON/binary JSON reading and writing, implementing JSON-Mmap Spec v1 Draft 1
  2. supporting JSONPath query to MATLAB data and JSON/binary JSON file and streams,
  3. (breaking) upgrading the supported BJData spec to V1 Draft 2 where the default numerical data byte order changed from Big-Endian to Little-Endian,
  4. adding initial support to JData _DataLink_ decoding to link multiple JSON/binary JSON files

There have been many major updates added to this release since the previous release v2.0 in June 2020. A list of the major changes are summarized below (with key features marked by *), including the support to BJData Draft-2 specification, new interface functions savejd/loadjd, and options to use MATLAB/Octave built-in jsonencode/jsondecode functions. The octave-jsonlab package has also been included in the official distributions of Debian Bullseye and Ubuntu 21.04 or newer.

  • 2022-04-19*[2278bb1] stop escaping / to / in JSON string, see https://mondotondo.com/2010/12/29/the-solidus-issue/
  • 2022-04-01*[fb711bb] add loadjd and savejd as the unified JSON/binary JSON file interface
  • 2022-03-30 [4433a21] improve datalink uri handling to consider : inside uri
  • 2022-03-30 [6368409] make datalink URL query more robust
  • 2022-03-29 [dd9e9c6] when file suffix is missing, assume JSON feed
  • 2022-03-29*[07c58f3] initial support for _DataLink_ of online/local file with JSONPath ref
  • 2022-03-29 [897b7ba] fix test for older octave
  • 2022-03-20 [bf03eff] force msgpack to use big-endian
  • 2022-03-13 [46bbfa9] support empty name key, which is valid in JSON, fix #79
  • 2022-03-12 [9ab040a] increase default float number digits from 10 to 16, fix #78
  • 2022-03-11 [485ea29] update error message on the valid root-level markers
  • 2022-02-23 [aa3913e] disable TFN marker in optimized header due to security risk and low benefit
  • 2022-02-23 [f2c3223] support SCH{[ markers in optimized container type
  • 2022-02-14 [540f95c] add optional preceding whitespace, explain format
  • 2022-02-13 [3dfa904] debugged and tested mmap, add mmapinclude and mmapexclude options
  • 2022-02-10*[6150ae1] handle uncompressed raw data (only base64 encoded) in jdatadecode
  • 2022-02-10 [88a59eb] give a warning when jdatadecode fails, but still return the raw data
  • 2022-02-03*[05edb7a] fast reading and writing json data record using mmap and jsonpath
  • 2022-02-02*[b0f0ebd] return disk-map or memory-map table in loadjson
  • 2022-02-01 [0888218] correct typos and add additional descriptions in README
  • 2022-02-01*[03133c7] fix row-major ('formatversion',1.8) ND array storage order, update demo outputs
  • 2022-02-01 [5998c70] revert variable name encoding to support unicode strings
  • 2022-01-31 [16454e7] test flexible whitespaces in 1D/2D arrays, test mixed array from string
  • 2022-01-31*[5c1ef15] accelerate fastarrayparser by 200%! jsonlab_speedtest cuts from 11s to 5.8s
  • 2022-01-30 [9b25e20] fix octave 3.8 error on travis, it does not support single
  • 2022-01-30 [5898f6e] add octave 5.2 to travis
  • 2022-01-30*[2e3344c] [bjdata:breaking] Upgrade savebj/loadbj to BJData v1-draft 2, use little-endian by default
  • 2022-01-30*[2e3344c] [bjdata:breaking] Fix optimized ND array element order (previously used column-major)
  • 2022-01-30*[2e3344c] optimize loadjson and loadbj speed
  • 2022-01-30*[2e3344c] add 'BuiltinJSON' option for savejson/loadjson to call jsonencode/jsondecode
  • 2022-01-30*[2e3344c] more robust tests on ND array when parsing JSON numerical array construct
  • 2021-06-23 [632531f] fix inconsistency between singlet integer and float values, close #70
  • 2021-06-23 [f7d8226] prevent function calls when parsing array strings using eval, fix #75
  • 2021-06-23 [b1ae5fa] fix #73 as a regression to #22
  • 2021-11-22*[ ] octave-jsonlab is officially in Debian Testing/Bullseye
  • 2020-09-29 [d0cb3b8] Fix for loading objects.
  • 2020-07-26 [d0fb684] Add travis badge
  • 2020-07-25 [708c36c] drop octave 3.2
  • 2020-07-25 [436d84e] debug octave 3.2
  • 2020-07-25 [0ce96ec] remove windows and osx targets from travis-ci
  • 2020-07-25 [0d8baa4] fix ruby does not support error on windows
  • 2020-07-25*[faa7921] enable travis-ci for jsonlab
  • 2020-07-08 [321ab1a] add Debian and Ubuntu installation commands
  • 2020-07-08 [e686828] update author info
  • 2020-07-08*[ce40fdf] supports ND cell array, fix #66
  • 2020-07-07 [6a8ce93] fix string encoding over 399 characters, close #65
  • 2020-06-14 [5a58faf] fix DESCRIPTION date bug
  • 2020-06-14 [9d7e94c] match octave description file and upstream version number
  • 2020-06-14 [a5b6170] fix warning about lz4encode file name

Please note that the savejson/loadjson in both JSONLab v2.0-v3.0 are compliant with JData Spec Draft 3; the savebj/loadbj`` in JSONLab v3.0 is compatible to BJData spec Draft 2, which contains breaking feature changes compared to those in JSONLab v2.0.

The BJData spec was derived from UBJSON spec Draft 12, with the following breaking differences:

  • BJData adds 4 new numeric data types: uint16 [u], uint32 [m], uint64 [M] and float16 [h] (supported in JSONLab v2.0 or newer)
  • BJData supports an optimized ND array container (supported in JSONLab since 2013)
  • BJData does not convert NaN/Inf/-Inf to null (supported in JSONLab since 2013)
  • BJData Draft 2 changes the default byte order to Little-Endian instead of Big-Endian (JSONLab 3.0 or later)
  • BJData only permits non-zero-fixed-length data types as the optimized array type, i.e. only UiuImlMLhdDC are allowed

To avoid using the new features, one should attach 'UBJSON',1 and 'Endian','B' in the savebj command as

savebj('',data,'FileName','myfile.bjd','UBJSON',1, 'Endian','B');

To read BJData data files generated by JSONLab v2.0, you should call

data=loadbj('my_old_data_file.bjd','Endian','B')

You are strongly encouraged to convert all pre-v2.9 JSONLab generated BJD or .jamm files using the new format.

Introduction

JSONLab is an open-source JSON/UBJSON/MessagePack encoder and decoder written completely in the native MATLAB language. It can be used to convert most MATLAB data structures (array, struct, cell, struct array, cell array, and objects) into JSON/UBJSON/MessagePack formatted strings and files, or to parse a JSON/UBJSON/MessagePack file into a MATLAB data structure. JSONLab supports both MATLAB and GNU Octave (a free MATLAB clone).

Compared to other MATLAB/Octave JSON parsers, JSONLab is uniquely lightweight, ultra-portable, producing dependable outputs across a wide-range of MATLAB (tested on R2008) and Octave (tested on v3.8) versions. It also uniquely supports BinaryJData/UBJSON/MessagePack data files as binary-JSON-like formats, designed for efficiency and flexibility with loss-less binary storage. As a parser written completely with the native MATLAB language, it is surprisingly fast when reading small-to-moderate sized JSON files (1-2 MB) with simple hierarchical structures, and is heavily optimized for reading JSON files containing large N-D arrays (known as the "fast array parser" in loadjson).

JSON (JavaScript Object Notation) is a highly portable, human-readable and "fat-free" text format to represent complex and hierarchical data, widely used for data-exchange in applications. UBJSON (Universal Binary JSON) is a binary JSON format, designed to specifically address the limitations of JSON, permitting the storage of binary data with strongly typed data records, resulting in smaller file sizes and fast encoding and decoding. MessagePack is another binary JSON-like data format widely used in data exchange in web/native applications. It is slightly more compact than UBJSON, but is not directly readable compared to UBJSON.

We envision that both JSON and its binary counterparts will play important roles for storage, exchange and interoperation of large-scale scientific data among the wide-variety of tools. As container-formats, they offer both the flexibility and generality similar to other more sophisticated formats such as HDF5, but are significantly simpler with a much greater software ecosystem.

Towards this goal, we have developed the JData Specification (http://github.com/NeuroJSON/jdata) to standardize serializations of complex scientific data structures, such as N-D arrays, sparse/complex-valued arrays, trees, maps, tables and graphs using JSON/binary JSON constructs. The text and binary formatted JData files are syntactically compatible with JSON/UBJSON formats, and can be readily parsed using existing JSON and UBJSON parsers. JSONLab is not just a parser and writer of JSON/UBJSON data files, but one that systematically converts complex scientific data structures into human-readable and universally supported JSON forms using the standardized JData data annotations.

Installation

The installation of JSONLab is no different from installing any other MATLAB toolbox. You only need to download/unzip the JSONLab package to a folder, and add the folder's path to MATLAB/Octave's path list by using the following command:

addpath('/path/to/jsonlab');

If you want to add this path permanently, you can type pathtool, browse to the JSONLab root folder and add to the list, then click "Save". Then, run rehash in MATLAB, and type which savejson, if you see an output, that means JSONLab is installed for MATLAB/Octave.

If you use MATLAB in a shared environment such as a Linux server, the best way to add path is to type

mkdir ~/matlab/
nano ~/matlab/startup.m

and type addpath('/path/to/jsonlab') in this file, save and quit the editor. MATLAB will execute this file every time it starts. For Octave, the file you need to edit is ~/.octaverc, where ~ is your home directory.

To use the data compression features, please download the ZMat toolbox from https://github.com/fangq/zmat/releases/latest and follow the instruction to install ZMat first. The ZMat toolbox is required when compression is used on MATLAB running in the -nojvm mode or GNU Octave, or 'lzma/lzip/lz4/lz4hc' compression methods are specified. ZMat can also compress large arrays that MATLAB's Java-based compression API does not support.

Install JSONLab on Fedora 24 or later

JSONLab has been available as an official Fedora package since 2015. You may install it directly using the below command

sudo dnf install octave-jsonlab

To enable data compression/decompression, you need to install octave-zmat using

sudo dnf install octave-zmat

Then open Octave, and type pkg load jsonlab to enable jsonlab toolbox.

Install JSONLab on Debian

JSONLab is currently available on Debian Bullseye. To install, you may run

sudo apt-get install octave-jsonlab

One can alternatively install matlab-jsonlab if MATLAB is available.

Install JSONLab on Ubuntu

JSONLab is currently available on Ubuntu 21.04 or newer as package octave-jsonlab. To install, you may run

sudo apt-get install octave-jsonlab

For older Ubuntu releases, one can add the below PPA

https://launchpad.net/~fangq/+archive/ubuntu/ppa

To install, please run

sudo add-apt-repository ppa:fangq/ppa
sudo apt-get update

to add this PPA, and then use

sudo apt-get install octave-jsonlab

to install the toolbox. octave-zmat will be automatically installed.

Install JSONLab on Arch Linux

JSONLab is also available on Arch Linux. You may install it using the below command

sudo pikaur -S jsonlab

Using JSONLab

JSONLab provides a pair of functions, loadjson -- a JSON parser, and savejson -- a MATLAB-to-JSON encoder, to read/write the text-based JSON; it also provides three equivalent pairs -- loadbj/savebj for binary JData, loadubjson/saveubjson for UBJSON and loadmsgpack/savemsgpack for MessagePack. The load* functions for the 3 supported data formats share almost the same input parameter format, similarly for the 3 save* functions (savejson/saveubjson/savemsgpack). These encoders and decoders are capable of processing/sharing almost all data structures supported by MATLAB, thanks to jdataencode/jdatadecode - a pair of in-memory data converters translating complex MATLAB data structures to their easy-to-serialized forms according to the JData specifications. The detailed help information can be found in the Contents.m file.

In JSONLab 2.9.8 and later versions, a unified file loading and saving interface is provided for JSON, binary JSON and HDF5, including loadjd and savejd for reading and writing below files types:

  • JSON based files: .json`, ``.jdt (text JData file), .jmsh (text JMesh file), .jnii (text JNIfTI file), .jnirs (text JSNIRF file)
  • BJData based files: .bjd`, ``.jdb` (binary JData file), ``.bmsh (binary JMesh file), .bnii (binary JNIfTI file), .bnirs (binary JSNIRF file), .jamm (MATLAB session file)
  • UBJSON based files: .ubj
  • MessagePack based files: .msgpack
  • HDF5 based files: .h5, .hdf5, .snirf (SNIRF fNIRS data files) - require EasyH5 toolbox

In the below section, we provide a few examples on how to us each of the core functions for encoding/decoding JSON/UBJSON/MessagePack data.

savejson.m

jsonmesh=struct('MeshNode',[0 0 0;1 0 0;0 1 0;1 1 0;0 0 1;1 0 1;0 1 1;1 1 1],...
         'MeshElem',[1 2 4 8;1 3 4 8;1 2 6 8;1 5 6 8;1 5 7 8;1 3 7 8],...
         'MeshSurf',[1 2 4;1 2 6;1 3 4;1 3 7;1 5 6;1 5 7;...
                    2 8 4;2 8 6;3 8 4;3 8 7;5 8 6;5 8 7],...
         'MeshCreator','FangQ','MeshTitle','T6 Cube',...
         'SpecialData',[nan, inf, -inf]);
savejson(jsonmesh)
savejson('jmesh',jsonmesh)
savejson('',jsonmesh,'Compact',1)
savejson('jmesh',jsonmesh,'outputfile.json')
savejson('',jsonmesh,'ArrayIndent',0,'FloatFormat','\t%.5g','FileName','outputfile2.json')
savejson('cpxrand',eye(5)+1i*magic(5))
savejson('ziparray',eye(10),'Compression','zlib','CompressArraySize',1)
savejson('',jsonmesh,'ArrayToStruct',1)
savejson('',eye(10),'UseArrayShape',1)

loadjson.m

loadjson('{}')
dat=loadjson('{"obj":{"string":"value","array":[1,2,3]}}')
dat=loadjson(['examples' filesep 'example1.json'])
dat=loadjson(['examples' filesep 'example1.json'],'SimplifyCell',0)

savebj.m (saveubjson.m as an alias)

a={single(rand(2)), struct('va',1,'vb','string'), 1+2i};
savebj(a)
savebj('rootname',a,'testdata.ubj')
savebj('zeros',zeros(100),'Compression','gzip')

loadbj.m (loadubjson.m as an alias)

obj=struct('string','value','array',single([1 2 3]),'empty',[],'magic',uint8(magic(5)));
ubjdata=savebj('obj',obj);
dat=loadbj(ubjdata)
class(dat.obj.array)
isequaln(obj,dat.obj)
dat=loadbj(savebj('',eye(10),'Compression','zlib','CompressArraySize',1))

jdataencode.m

jd=jdataencode(struct('a',rand(5)+1i*rand(5),'b',[],'c',sparse(5,5)))
savejson('',jd)

jdatadecode.m

rawdata=struct('a',rand(5)+1i*rand(5),'b',[],'c',sparse(5,5));
jd=jdataencode(rawdata)
newjd=jdatadecode(jd)
isequaln(newjd,rawdata)

examples

Under the examples folder, you can find several scripts to demonstrate the basic utilities of JSONLab. Running the demo_jsonlab_basic.m script, you will see the conversions from MATLAB data structure to JSON text and backward. In jsonlab_selftest.m, we load complex JSON files downloaded from the Internet and validate the loadjson/savejson functions for regression testing purposes. Similarly, a demo_ubjson_basic.m script is provided to test the saveubjson and loadubjson functions for various matlab data structures, and demo_msgpack_basic.m is for testing savemsgpack and loadmsgpack.

Please run these examples and understand how JSONLab works before you use it to process your data.

unit testing

Under the test folder, you can find a script to test individual data types and inputs using various encoders and decoders. This unit testing script also serves as a specification validator to the JSONLab functions and ensure that the outputs are compliant to the underlying specifications.

Using jsave/jload to share workspace

Starting from JSONLab v2.0, we provide a pair of functions, jsave/jload to store and retrieve variables from the current workspace, similar to the save/load functions in MATLAB and Octave. The files that jsave/jload reads/writes is by default a binary JData file with a suffix .jamm. The file size is comparable (can be smaller if use lzma compression) to .mat files. This feature is currently experimental.

The main benefits of using .jamm file to share matlab variables include

  • a .jamm file can be 50% smaller than a .mat file when using jsave(..., "compression","lzma"); the only drawback is longer saving time.
  • a .jamm file can be readily read/opened among many programming environments, including Python, JavaScript, Go, Java etc, where .mat file support is not generally available. Parsers of .jamm files are largely compatible with UBJSON's parsers available at http://ubjson.org/?page_id=48
  • a .jamm file is quasi-human-readable, one can see the internal data fields even in a command line, for example using strings -n 2 file.jamm | astyle, making the binary data easy to be understood, shared and reused.
  • jsave/jload can also use MessagePack and JSON formats as the underlying data storage format, addressing needs from a diverse set of applications. MessagePack parsers are readily available at https://msgpack.org/

jsave.m

jsave    % save the current workspace to jamdata.jamm
jsave mydata.jamm
jsave('mydata.jamm','vars',{'var1','var2'})
jsave('mydata.jamm','compression','lzma')
jsave('mydata.json','compression','gzip')

jload.m

jload    % load variables from jamdata.jamm to the current workspace
jload mydata.jamm   % load variables from mydata.jamm
vars=jload('mydata.jamm','vars',{'var1','var2'}) % return vars.var1, vars.var2
jload('mydata.jamm','simplifycell',0)
jload('mydata.json')

Sharing JSONLab created data files in Python

Despite the use of portable data annotation defined by the JData Specification, the output JSON files created by JSONLab are 100% JSON compatible (with the exception that long strings may be broken into multiple lines for better readability). Therefore, JSONLab-created JSON files (.json, .jnii, .jnirs etc) can be readily read and written by nearly all existing JSON parsers, including the built-in json module parser in Python.

However, we strongly recommend one to use a lightweight jdata module, developed by the same author, to perform the extra JData encoding and decoding and convert JSON data directly to convenient Python/Numpy data structures. The jdata module can also directly read/write UBJSON/Binary JData outputs from JSONLab (.bjd, .ubj, .bnii, .bnirs, .jamm etc). Using binary JData files are expected to produce much smaller file sizes and faster parsing, while maintaining excellent portability and generality.

In short, to conveniently read/write data files created by JSONLab into Python, whether they are JSON based or binary JData/UBJSON based, one just need to download the below two light-weight python modules:

To install these modules on Python 2.x, please first check if your system has pip and numpy, if not, please install it by running (using Ubuntu/Debian as example)

sudo apt-get install python-pip python3-pip python-numpy python3-numpy

After the installation is done, one can then install the jdata and bjdata modules by

pip install jdata --user
pip install bjdata --user

To install these modules for Python 3.x, please replace pip by pip3. If one prefers to install these modules globally for all users, simply execute the above commands using

sudo pip install jdata
sudo pip install bjdata

The above modules require built-in Python modules json and NumPy (numpy).

Once the necessary modules are installed, one can type python (or python3), and run

import jdata as jd
import numpy as np

                      

鲜花

握手

雷人

路过

鸡蛋
该文章已有0人参与评论

请发表评论

全部评论

专题导读
热门推荐
阅读排行榜

扫描微信二维码

查看手机版网站

随时了解更新最新资讯

139-2527-9053

在线客服(服务时间 9:00~18:00)

在线QQ客服
地址:深圳市南山区西丽大学城创智工业园
电邮:jeky_zhao#qq.com
移动电话:139-2527-9053

Powered by 互联科技 X3.4© 2001-2213 极客世界.|Sitemap