🔎

    Data Types

    Data in DataCogs is strongly typed. This helps ensure data is passed between compatible modules correctly and that the correct operations are performed on it. Data types are specified in a way similar to the RFC 6838 standard for Media Types / MIME types.

    Structure🔗

    [root type]/[semantic type]+[structural type];[parameter1]=[value1];[parameter2]=[value2]...
    • Root Type: This is the most general type of the data. Typically this is either application or text.
    • Semantic Type: This defines the meaning of the data. For example, vnd.datacogs.model is used to represent a model.
    • Structural Type: This defines the structure of the data. For example, json is used to represent JSON data.
    • Parameters: These are additional parameters that can be used to further specify the data type.

    It is recommended when defining outputs to include both the semantic type as well as structural type, as this will help downstream modules understand the meaning of the data and maximize compatibility.

    If only one "subtype" is provided (e.g. application/json), it is assumed to be the structural type. If two are provided (e.g. application/vnd.datacogs.model+json), the first is the semantic type and the second is the structural type. Providing both is preferable because some nodes and applets can unerstand the structural type and be useful for inspecting raw data.

    • application/vnd.datacogs.model+json: Nodes semantically working with models can understand the data type. Nodes that only understand json can also be used to view the data.
    • application/json: Nodes that only understand json can be used to view the data.
    • application/vnd.datacogs.model: Nodes semantically working with models can understand the data type, but nodes that only understand json will not be able to view the data.

    Common Data Types🔗

    application/json🔗

    This data type is used to represent JSON data. JSON is a lightweight data-interchange format that is easy for humans to read and write and easy for machines to parse and generate.

    text/csv🔗

    This data type is used to represent CSV data. CSV (Comma-Separated Values) is a simple file format used to store tabular data, such as a spreadsheet or database.

    vnd.datacogs.model+json🔗

    This data type is used to represent a DataCogs model in JSON format. A DataCogs model is a graph-based data structure that can be used to represent complex data and relationships between data.

    vnd.datacogs.graph+json🔗

    This data type is used to represent a DataCogs graph in JSON format. A DataCogs graph is a collection of nodes and edges that represent data and relationships between data.

    API🔗

    Internally, DataCogs has encapsulates data in either a serde_json::Value or a Vec<u8>. Apache Arrow streams of RecordBatches, for example, are converted to Vec<u8>.

    Serializing / Deserializing Objects🔗

    For whole objects, e.g. application/vnd.datacogs.model+json, the serde_json::Value is used directly.

    Strings and Numbers🔗

    For strings and numbers, specific variants of serde_json::Value are used directly.

    For example, text/plain is stored in a serde_json::Value::String. A node or applet receiving a text/plain data type can access the string directly.

    Apache Arrow Record Batch🔗

    For Apache Arrow Record Batches, the data is serialized to a Vec<u8>.