]> NTV tabular format (NTV-TAB) Loco-labs
476 chemin du gaf de Famian BOLLENE 84 500 FR philippe@loco-labs.io https://github.com/loco-philippe/NTV/blob/main/README.md
General Internet Engineering Task Force JSON semantic data interchange format tabular ABNF This document describes a set of simple rules for unambiguously and concisely encoding semantic tabular and multidimensional data (NTV-TAB format). These rules are based on the NTV structure and its JSON representation (JSON-NTV format).  
Introduction
Presentation The main operational standard used to exchange textual tabular data is CSV format . Unfortunately CSV format is obsolete (last revision in 2005) and current CSV tools do not comply with the standard. It is therefore important to define an alternative format that meets the expectations of tabular and multidimensional data exchanges. The NTV-TAB format proposed here is a response to this need.
Key design features The format's focus is on simplicity, lightness and web usage. The key features of this format are the following:
  • JSON as the base format
    • JSON is simple and readable as simple text
    • JSON supports rich structure including nesting and basic types
    • JSON is web-native and very widely used and supported
    • JSON format has binary representation (i.e. CBOR format)
  • optimized representations
    • from the simplest to the most optimized, are available
    • avoid data duplication,
    • reduce the size of data,
    • allows strict and unambiguous reversibility (lossless round-trip)
  • high semantic level of data (JSON-NTV as a grammar)
    • Take into account most common data formats used in Internet standards
    • wide variety of data typing
    • meta-data (header or schema) can be integrate
    • common format between tabular and multidimensional data
  • simple, compact, extensible and self-describing
Conventions Used in This Document The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here. This document also uses the following terms:
JsonText, JsonValue, JsonObject, JsonMember, JsonElement, JsonArray, JsonNumber, JsonString, JsonFalse, JsonNull, JsonTrue :
These terms are defined in .
NTV, NTVlist, NVlist, Vlist, TVlist, NTVsingle, NVsingle, TVsingle, Vsingle, NTVname, NTVtype, NTVvalue, JsonNTVtype, JsonNTVname, JsonPrimitive, JsonUnnamed, JsonNamed:
These terms are defined in .
Row, Column, Table, Cell:
These terms are defined in .
Dataset
A Dataset is equivalent to a Table
Field
A Field is equivalent to a Column
Tabular data
Principles Tabular data is data that is structured into rows, each of which contains information about some things. Each row contains the same number of cells (although some of these cells may be empty), which provide values of properties of the thing described by the row. In tabular data, cells within the same column provide values for the same property of the things described by each row. This is what differentiates tabular data from other line-oriented formats. Two main uses are identified for tabular data:
  • a flow-oriented use for which each row is independent of the others. The dataset is then seen as a list of rows whose number can be variable. This is for example the case of a list of measurements from a sensor.
  • a structure-oriented use for which the rows are not independent and contribute to describing the same object. This is for example the case of a grade table for a class which integrates the students, courses, periods, etc.
This document deals with this second use.
Tabular structure In structure-oriented use, columns and rows are not equivalent, the columns (or Fields) represent the 'semantics' of the data and the rows represent a specific combination of Field's values according to the structure defined by the tabular data (Dataset). The nature of the rows is often implicit. Two basic patterns are present in Datasets:
  • Tree pattern: A tree is represented in tabular form by a list of paths between each leaf and the node. The columns then represent the levels of the tree.
  • Matrix pattern: A matrix (or multidimensional data) is represented in tabular form by a column of the values of the matrix and additional columns represent the coordinates of each of the values.
 and present an example of such patterns Tree pattern
Rootlevel 1level 2
ABD
ABE
ACF
ACG
Matrix pattern
Valuerowcol
1AC
2AD
3BC
4BD
Taking these structures into account leads to significant duplication of data. In the general case, Datasets mix these different structures. If we now observe the relationships between Fields , we can identify four main uses:
  • association: this consists of coupling each value of a Field to a single value of another Field ("coupled" relationship between two fields),
  • classification: This involves grouping the data by category in order - for example - to be able to make a statistical use of it, ("derived" relationship between two fields),
  • crossing: This consists of representing all the combinations between the two Fields, such as in matrix representations ("crossed" relationship between two fields),
  • characterization: It corresponds to the documentation of defined properties (no specific relationship).
Example: Price list of different foods based on packaging for the year 2022. Price list
IdProductFoodPackagingWeightPricePeriodAvailability
11applefruitbag1 kg12nd half 2022Yes
12applefruitcardboard10 kg92nd half 2022Yes
13orangefruitbag1 kg22nd half 2022end of 2022
14orangefruitcardboard10 kg182nd half 2022end of 2022
15peppervegetablebag1 kg1.52nd half 2022end of 2022
16peppervegetablecardboard10 kg132nd half 2022end of 2022
17bananafruitbag1 kg0.52nd half 2022Yes
18bananafruitcardboard10 kg42nd half 2022Yes
We find here:
  • association: between "Packaging" and "Weight",
  • classification: between "Product" and "Food",
  • crossing: between "Product" and "Weight",
  • characterization: between "Product" and "Availability"
Field structure A Field is an ordered set of Cells. To represent this structure, several representations are possible depending on the nature of the data:
  • the simplest format is to represent a Field by the list of Cells with the same order for all Fields. This format is interesting when the data is little duplicated,
  • when the data is repetitive, a second option is to represent on the one hand the list of different data and on the other hand their position in the list (i.e. categorical data),
  • another special case also concerns repetitive data for which one value is highly predominant (sparse data). In this case, it is sufficient to provide only the position in the list of data except for the one that is predominant,
  • a last option consists in representing the Field according to its dependence with another Field (coupled, derived or crossed relationship). This leads to an optimized data volume.
Representation Three representations are available for a tabular object : row-oriented (list of Rows), cells-oriented (list of Cells), field-oriented (list of Fields). The field-oriented representation is retained because it takes into account the semantics carried by the Fields as well as the inter-Field analysis presented above. A Dataset is then seen as a set of Fields representing the properties of the entire Dataset. The order of Fields or Rows is not relevant.
NTV-TAB format
NTV structure A Dataset is represented by the following NTV entities:
  • NTVcell represents a Cell. NTVcell is a NTVsingle.
  • NTVfield represents a Field. NTVfield is a NTV entity depending on the format chosen to represent the Field (simple format, default format, optimized format). A NTVfield contains a NTVlist of part of the NTVcells (Codec) and optionnaly coding data.
  • NTVdataset represents the Dataset. NTVdataset is a NVlist where the NTVname is the name of the Dataset and the NTVvalue is the list of NTVfields.
The JSON format of a NTVdataset is his JSON-NTV format.
simple NTVfield formats This category is the usual representation of a Field with different values (Full format) or with several identical values (Unique format). Full format : The Full format is the format that does not use any coding. Codec and NTVfield are identical. The NTVfield is therefore a NTVlist where the NTVname is the name of the Field, the NTVtype is the default type of the NTVcells and the NTVvalue is the list of NTVcells.
  • Example JsonNTVvalue ( "price" Field) :
    • [ 1, 9, 2, 18, 1.5, 13, 0.5, 4 ]
Unique format : The Unique format is used when all NTVcells are identical. The Codec is the NTVcell.Codec and NTVfield are identical (coding is implict). The NTVfield is therefore the NTVcell.
  • Example JsonNTVvalue ( "period" Field) :
    • "2nd half 2022"
Note :
  • The Unique format also makes it possible to represent tabular metadata
default NTVfield formats This category completes the simple formats with the other most common representations of a Field :
  • Categorical Field (Complete format)
  • Periodic Field (Primary format)
  • Sparse Field (Sparse format)
In those formats, Codec is explicit and is the TVlist of different Field NTVcells (Codec). The NTVfield is a NVlist where the NTVname is the name of the Field. Complete format : The "complete format" is equivalent to the format used to store categorical variables. The NTVfield is a NVlist composed with two NTV entities :
  • Codec: TVlist of different Field NTVcells (Codec),
  • Coding: Vlist of indexes of NTVcells in Component (Keys)
The list of NTVcells is reconstituted by replacing the integers in the coding Vlist with the NTVcell at the coding index in the Codec (e.g. pandas categories and codes).
  • Example JsonNTVvalue ( "product" Field) :
    • [ [ "orange" , "pepper" , "apple" , "banana" ], [ 2, 2, 0, 0, 1, 1, 3, 3 ] ]
Sparse format : A specific format (one dimensional sparse LIL format) is used for sparse data. It is defined by:
  • 'fill_value': it should be most common value
  • 'sp_value': it is a list storing only values distinct from the 'fill_value'
  • 'sp_index': list of index of 'sp-value' in the sparse data list
The NTVfield is a NVlist composed with three NTV entities :
  • Codec: TVlist of different Field NTVcells (Codec),
  • Ref: Vlist of indexes of Codec value in 'sp_value' ,
  • Coding: Vlist of 'sp_index'
The list of NTVcells is reconstituted by replacing in a list of 'fill_value', the values with index in the Coding Vlist by the corresponding value defined by the Ref index in the Codec TVlist.
  • Example JsonNTVvalue ( "food" Field) :
    • [ [ "vegetable" , "fruit"], [0, 0], [ 4, 5 ] ] 'fruit' is the 'fill_value' - 0 is the index of "vegetable"
Primary format : This format is equivalent to the Complete format where the Keys Vlist is calculated with the "repetition coefficient". The NTVfield is a NVlist composed with two NTV entities :
  • Codec: TVlist of different Field NTVcells (Codec),
  • Coding: Vlist with a single integer (Repetition coefficient)
The Keys Vlist is generated with the formula:
  • keys[ikey] = ( ikey % ( coef * period ) ) // coef
  • where:
    • keys: is the Keys Vlist
    • ikey: is the index of a key value
    • coef: is the Repetition coefficient
    • period: is the length of Codec
  • Example: coef = 2, period = 3, Keys length = 12
    • keys = [0, 0, 1, 1, 2, 2, 0, 0, 1, 1, 2, 2]
The Repetition coefficient is the number of adjacent identical values in the Keys list.
  • Example "packaging" :
    • Codec: [ "bag" , "cardboard" ]
    • Coefficient: 1
    • (implicit Keys : [ 0, 1, 0, 1, 0, 1, 0, 1 ] )
  • Example "product" :
    • Codec: [ "apple" , "orange" , "pepper" , "banana" ]
    • Coefficient: 2
    • (implicit Keys : [ 0, 0, 1, 1, 2, 2, 3, 3 ] )
Optimized NTVfield formats This category of formats reduces the size of Complete format with optimized Keys. The length of Keys is reduced with using of derived (Relative format) or coupled (Implicit format) relationships between two Fields. In those formats, Codec is explicit and is the TVlist of different Field NTVcells (Codec). The NTVfield is a NVlist where the NTVname is the name of the Field. Implicit format : This representation is associated with "coupled" Fields. These Fields have a one-to-one correspondence. The NTVfield is a NVlist composed with two NTV entities :
  • Codec: TVlist of different field NTVcells (Codec),
  • Ref: Vsingle entity index or name of the coupled Field.
This format is equivalent to the Complete format where Keys is the Keys of the Field (with Complete format) defined by Ref.
  • Example JsonNTVvalue ( "weight" Field is associated with "packaging" Field ) :
    • [ [ "1 kg" , "10 kg" ], "packaging"]
    • ( implicit Keys : [ 0, 1, 0, 1, 0, 1, 0, 1 ] )
Relative format : This representation is associated with "derived" Fields. These Fields have a one-to-many correspondence. The values of a "derived" Field are inferred from the values of the parent Field. The Field is a NVlist composed with three NTV entities :
  • Codec: TVlist of different field NTVcells (Codec),
  • Ref: Vsingle entity index or name of the parent Field,
  • Coding : Vlist of relative indexes of NTVcells in Codec (Relative Keys).
This format is equivalent to the Complete format where the Keys Vlist is obtained by replacing the values of the Keys Vlist of the parent Field with the corresponding values in the Relative Keys (the length of the Relative Keys is the length of the Codec of the parent Field).
  • Example JsonNTVvalue ( "food" Field - "product" Field is the parent Field of "food" Field) :
    • [ [ "fruit" , "vegetable" ], "product", [ 0, 1, 0, 0 ] ]
    • (the Vlist Keys is obtained by replacing the values 0, 1, 2, 3 of the Vlist Keys of the "product" Field by the values 0, 1, 0, 0 of the Relative Keys i.e.: [ 0, 0, 0, 0, 1 , 1, 0, 0] )
Synthesis The NTVfield structure corresponding to the format defined above are in : NTVfield formats
StructureCodecRefCoding
formatNTVTVlistVsingleVlist
RelativeNTVlistlen = 3xindexor nameRelative Keyslen < len(Field)
CompleteNTVlistlen = 2x Keyslen = len(Field)
Sparse NTVlistlen = 3xlist of indexsp_valuesp_index1<len<len(Field)
ImplicitNTVlistlen = 2xindexor name
Primary NTVlistlen = 2x coeflen = 1
UniqueNTVsingle
FullNTVlist len = len(Field)
Three levels are available to convert tabular data in JSON structure .
  • Level 0: "simple" is the usual representation of tabular data.Fields are converted with the Simple or Unique format.
  • Level 1: "default" avoids duplication of information by adding simple encoding. Fields are converted according to their own structure (simple, unique, categorical, sparse, periodic).
  • Level 2: "optimize" avoids duplication of information and minimizes encoding. It is the usual representation of multidimensional data. This level requires an analysis of the relationships between Fields ("partition")
NTVfield levels
LevelStructure
modeType Fieldformat
0simpleUniqueUnique
SimpleFull
1defaultUniqueUnique
SimpleFull
SparseSparse
CategoricalComplete
PeriodicPrimary
2optimizeUniqueUnique
Root coupledFull
Root derivedComplete
PrimaryPrimary
DerivedRelative
CoupledImplicit
Examples
Field examples The example in has the following JSON representation : NTVfield examples
FormatJsonNTV Representations
Full{ "price::float": [ 1, 9, 2, 18, 1.5, 13, 0.5, 4 ] } { "price": [ 1, 9, 2, 18, 1.5, 13, 0.5, 4 ] } [ 1, 9, 2, 18, 1.5, 13, 0.5, 4 ]
Complete{"product":[["orange","pepper","apple","banana"],[2,2,0,0,1,1,3,3]]} {"product": [ ["orange","pepper","apple","banana"],[2, 2, 0, 0, 1, 1, 3, 3] ]} [ ["orange","pepper","apple","banana"],[2, 2, 0, 0, 1, 1, 3, 3] ]
Unique{ "period": "2nd half 2022" }"2nd half 2022"
Implicit{"weight":[{"::string":["1 kg","10 kg"]},"packaging"]}[["1 kg","10 kg"],3]
Relative{"food": [ {"::string": [ "fruit" , "vegetable" ]},"product", [ 0, 1, 0, 0 ]] } [ [ "fruit" , "vegetable" ], 1, [ 0, 1, 0, 0 ] ]
Sparse{"food":[{"::string":["vegetable","vegetable","fruit"]},[4,5,-1]]} [["vegetable","vegetable","fruit"],[4,5, 1]]
Primary{"packaging":[{"::string":["cardboard","bag"]},[1]]}[["cardboard","bag"],[1]] {"product":[["apple","orange","peppers","banana"],[2]]}[["apple","orange","peppers","banana"],[2]]
Dataset examples The examples in below illustrate the optimize level: optimize level examples
DataOptimize level
typeFull formatJsonNTV
matrix[['a','a','b','b','c','c'],[10,20,10,20,10,20],[1,2,3,4,5,6]] [[['a','b','c'],[2]],[[10,20],[1]],[1,2,3,4,5,6]]
single[[1,2,3,4,5,6],['a','a','a','a','a','a']][[1,2,3,4,5,6],'a']
complete[[1,2,3,3,5,5]][[[1,2,3,5],[0,1,2,2,3,3]]]
coupled[[1,2,3,3,5,5], ['a','b','c','c','e','e']] [[[1,2,3,5],[0,1,2,2,3,3]], [['a','b','c','e'],0]]
derived[[1,2,3,4,5,6],['a','a','b','b','c','c'],[10,10,10,10,20,20]] [[1,2,3,4,5,6], [['a','b','c'],[0,0,1,1,2,2]],[[10,20],1,[0,0,1]]]
matrix+coupled[[6,6,7,7,8,8,9,9],[10,20,10,20,10,20,10,20], [1,1,2,2,3,3,4,4],[1,2,3,4,5,6,7,8]] [[[6,7,8,9],[2]],[[10,20],[1]],[[1,2,3,4],0], [1,2,3,4,5,6,7,8]]
matrix+coupled+derived[[6,6,7,7,8,8,9,9],[10,20,10,20,10,20,10,20], [1,1,2,2,3,3,4,4],[11,11,22,22,22,22,22,22],[1,2,3,4,5,6,7,8]] [[[6,7,8,9],[2]],[[10,20],[1]],[[1,2,3,4],0], [[11,22],0,[0,1,1,1]], [1 2,3,4,5,6,7,8]]
The examples in below illustre NTVdataset with a length equal to 0, 1 or 2: NTVdataset with length 0, 1 or 2
[ ] or { }Empty NTVdataset
[25] or [[25]]NTVdataset with 1 NTVfield and length 1
[2, 1] or [[2], [1]] or [2, [1]]NTVdataset with 2 NTVfield and length 1
[[2, 1]]NTVdataset with 1 NTVfield and length 2
[[2, 1], [4, 3]]NTVdataset with 2 NTVfield and length 2
Properties
JSON representation NTV-TAB format defines the representation of a Dataset into the NTV format. This conversion is reversible (lossless). Furthermore, the NTV format defines the conversion into JSON format. This conversion is also reversible. The exchange format (JsonText) of a Dataset is therefore obtained by a representation in NTV-TAB format then a conversion to JSON format and finally a conversion to text format (or binary format with CBOR conversion). The data is reconstituted identically by reverse conversions.
Dataset size As explain in cells are often duplicated in a Field. The principle of NTV-TAB format is to replace duplicated data with encoding based on integers. This optimization considerably reduces the size of a representation of a Dataset. details the methodology to optimize this size.
Nested NTV-TAB structure NTVcells in a NTVdataset are any NTVsingle. We can therefore include in a NTVdataset the data associated with the types defined in the NTV format. The 'tab' and the 'field' NTVtypes are associated to NTVdataset and NTVfield. A NTVcell can also include a NTVdataset or a NTVfield. is an example of nested Dataset. The 'nested' JsonNTV is the representation of a Dataset with length equal 2 and composed with two Fields 'field1' and 'field2'.
  • 'field1' is a Field with two Cells 'dataset1' ans 'dataset2' which are Dataset. 'field1' is represented with a Full format NTVfield.
  • 'field2' is a Field with two Cells 'field2_1' ans 'field2_2' which are Field. 'field2' is represented with a Full format NTVfield.
Nested Dataset
Parsing a JSON-value A NTV parser generates a NTV entity from a JSON-value. The decoding NTV entity is directly converted into the NTVdataset and a list of NTVfields. For each NTVfield the format is deduced following the structure defined in the table xxx. For each format, a decoder converts the NTVvalue of the NTVfield into the chosen object. Note :
  • Several NTVvalue are ambiguous to deduce the Field format :
    • [ list-data, integer, list-integer ] : Full or Relative format ?
    • [ list-data, string, list-integer ] : Full or Relative format ?
    • [ list-data, integer ] : Full or Implicit format ?
    • [ list-data, string ] : Full or Implicit format ?
    • [ list-data, list-integer ] : Full or Complete/Primary format ?
    • [ list-data, list-integer, list-integer ] : Full or Sparse format ?
  • The full format is not retained for those NTVvalue.
  • To avoid this ambiguity, precautions can be taken for Dataset with length = 2 or 3 and with a Full format:
    • a name can be added to the list-data (e.g. { "data": list-data}),
    • the order of data can be changed (e.g. [ integer, list-data ])
    • a type can be added (e.g. { "::json": [ list-data, list-integer ] } )
    • an additional field can be added (e.g. [ list-data, integer, list-integer, {"format": "full"} ] ).
 
IANA Considerations Any JsonValue is a JsonNTVValue and conversely, any JsonNTVvalue is a JsonValue. Thus, any JSON data may or may not be treated as JsonNTV data, so there is no need to create a specific MIME media type for JsonNTV. All properties of the MIME media type "application/json" are applicable.
Security Considerations The format used for NTV data exchanges is the JSON format. So, all the security considerations of apply. The NTV structure provides no cryptographic integrity protection of any kind.
  References Normative References Informative References Table Schema"FrictionLess" JSON semantic format (JSON-NTV) Tabular dataset analysis Recommendation : Model for Tabular Data and Metadata on the Web "W3C"  
Dataset sizing This appendix presents an analysis of NTVdataset size optimization with the defined formats.
Methodology The principle of defined formats is to replace duplicated data with encoding based on integers. We define the size of a Dataset representation (SZ) as the sum of the encoding size and the size of unencoded unduplicated values. The coding is modeled as being the product of the values remaining to be represented (nv - nc) with an average coding size (sc):
  • SZ = nc * sv + (nv - nc) * sc
  • where :
    • nv : number of values
    • sv : mean value size
    • nc : number of different values
    • sc : mean coding size
  • example :
    • Full format : {"product": ["orange","apple","apple","apple","orange","orange"]}
    • Complete format : {"product": [ ["orange","apple"], [1, 0, 0, 0, 1, 1] ]}
    • SZ = 9 + 8 + 7 + 6 * 1 = 30 (including double quotes),
    • nv = 9 (including the Field name),
    • sv = (9 + 8 * 3 + 7 * 3) / 7 = 7.71
    • nc = 3 (including the Field name)
    • sc = (30 - 3 * 7.71) / (9 - 3) = 1.15 (sc = (SZ - nc * sv) / (nv - nc))
  • In this example the JSON overhead (coma, space, curly bracket, square bracket) is not included.
SZ is maximal when there is no coding (sc = sv) and minimal when the coding is perfect (sc = 0):
  • SZmax = nv * sv
  • SZmin = nc * sv
We then define the following indicators:
  • unicity level UL = nc / nv
    • UL = SZmin / SZmax
    • 1 - UL = (SZmax - SZmin) / SZmax
    • UL characterizes the nature of the data independently of the coding and represents the maximum achievable gain (1-UL).
    • maximum UL = 1 (unduplicated data)
    • minimum UL = 0 (full duplicated data = empty data)
  • object lightness OL = sc / sv
    • OL = (SZ - SZmin) / (SZmax - SZmin)
    • 1 - OL = (SZmax - SZ) / (SZmax - SZmin)
    • OL characterizes coding efficiency
    • maximum OL = 1 (no coding)
    • minimum OL = 0 (perfect coding)
The optimization of the size of the representation is then evaluated by comparing the size obtained without coding and that obtained with coding:
  • G = (SZmax - SZ) / SZmax = (1 - UL) * (1 - OL)
  • R = 1 - G = SZ / SZmax = UL + OL - UL * OL
  • The maximum G gain is 1 - UL, the minimum G gain is 0.
  • If the data is empty, UL = OL = 0 and the gain is equal to 1.
The indicators are deduced from the following four measurable values:
  • number of cells in the dataset (nv)
  • number of different cells in the dataset (nc)
  • size of the dataset with the format to study (SZ)
  • size of the dataset with Full format (SZmax)
We then deduce sv = SZmax / nv as well as sc = (SZ - nc * sv) / (nv - nc) In the example above, the indicators are:
  • UL = 3 / 9 = 0.33
  • OL = 1.15 / 7.71 = 0.15
  • G = 0.67 * 0.85 = 0.57
  • SZmax = 9 * 7.71 = 69.4
  • SZmin = 3 * 7.71 = 23.1
  • The Complete format is close to the minimum size (SZ = 30) and its size is less than half the size of the Full format (43 %).
Formats The formats used to represent an NTVfield are in general form:
  • list of part of NTVcells
  • list of integers used to encode other NTVcells
The size of this format can then be written (without taking into account the overhead linked to the format):
  • SZ = nc * sv + k * nv * si
  • where :
    • nv : number of values
    • sv : mean value size
    • nc : number of different Field values
    • si : integer size
    • k: specific coefficient of the coding used
Comparison with the structure defined in the previous chapter allows us to deduce the parameters:
  • UL = nc/nv
  • sc = si * k * nv/(nv-nc)
  • OL = si/sv * k * nv/(nv-nc)
  • G = 1- nc/nv - si/sv * k
  • R = nc/nv + si/sv * k
The gain G is therefore equal to the maximum gain 1-UL reduced by the weight of the coding corresponding to the parameter k weighted by the average size of the values compared to an integer. below specifies the values of k for the different formats: coding coefficient
Formatk coefficientcomments
Full0R = 1
Unique0R = 1/nv (nc = 1)
Complete1R = nc/nv + si/sv
Primary1 / nvR = nc/nv + si/sv/nv
Coupled1 / nvR = nc/nv + si/sv/nv
Sparse2 * ns / nvR = nc/nv + 2*si/sv*ns/nv
Derivednd / nvR = nc/nv + si/sv*nd/nv
    • ns: number of values distinct from the 'fill_value'
    • nd: number of different values in the parent Field
Table schema compatibility This appendix presents the compatibility between Tableschema and the NTV-TAB format.
Table schema Table Schema is a simple language- and implementation-agnostic way to declare a schema for tabular data. A Table Schema is represented by a descriptor. The descriptor MUST be a JSON object with defined properties (JsonMember). Table Schema define following descriptors and properties:
  • Fields property
    • Fields property MUST be an array where each entry in the array is a field descriptor (as defined below).
  • Field descriptor
    • A field descriptor MUST be a JSON object that describes a single field.
  • Field properties
    • The field descriptor object MAY contain any number of other properties. Some specific properties are defined below. Of these, only the name property is REQUIRED.
    • Defined Properties:
      • name
      • title
      • description
      • example
      • type / format
      • constraints
    • The constraints property on Table Schema Fields can be used by consumers to list constraints for validating field values.
  • Table properties
    • In additional to field descriptors, there are the following "table level" properties:
      • missingValues
      • primaryKey
      • foreignKeys
Compatibility Three levels of compatibility are addressed :
  • Concepts
    • The concepts are equivalent between Table Schema and NTV-TAB :
      • Table is equivalent to Dataset
      • Field in Table Schema is equivalent to the NTVfield
      • Name in Table Schema is equivalent to the NTVname of the NTVfield
      • Type / Format in Table Schema is equivalent to the NTVtype of the NTVfield
  • Type / Format
    • NTVtype combines the concepts of type and format. The correspondence table in the NTV specification gives the link between an NTVtype and the corresponding type/format.
  • Constraints
    • Constraints are applicable to each value in a Table Field. Validating constraints for all values in a Table Field is equivalent to validating a constraint for all values in the Codec list.
These compatibility levels are reached, which makes it possible to validate an NTVdataset with a schema defined according to the Table Schema format. The following principles should then be considered to validate an NTVdataset:
  • The NTVfields names must be identical to the Schema names,
  • If the NTVtype of Codec data is not 'json', it must match the Type/Format defined in the Schema,
  • If the constrainst are valid with the Codec data, they are valid with the Field.
Example is an example of Dataset with Full format ('tab_data1' without NTVtypes) and with other formats ('tab_data2').
Dataset example
The schema in is valid with 'tab_data1' and 'tab_data2' formats
Schema example
Acknowledgements TBD
Contributors TBD