Canonical Dataset API v1

Canonical Dataset API v1

GPL-3.0-or-later

Each session stores dataset metadata in:

metadata/datasets.ini

A dataset record contains at least an ID, contract/kind, format, path, source module and integrity metadata where available.

Canonical variables

CSV columns are represented by canonical variables. For generic CSV import the default is:

original CSV header -> canonical variable with the same name

The user may change the canonical name and add:

• unit;
• semantic role;
• source column;
• source instrument channel;
• instrument ID;
• instrument channel/terminal.

The RAW input is never modified. acq.generic_csv writes a canonical CSV under datasets/ using the enabled variable map and canonical names. This prevents an alias from being only cosmetic: the analysis runtime sees the canonical column name that is actually present in the dataset.

Example:

CSV C1 header: Voltage
canonical:      Va_V
unit:           V
role:           independent
instrument:     scope_lab
channel:        CH1

Separation from modules

Dataset metadata does not encode knowledge of Curve Family Fitting, IR analysis or any other consumer. Analysis modules bind their required roles to canonical variables independently.

Structured Dataset metadata

Dataset Metadata API v1 associates one persistent namespaced key/value sidecar with a canonical Dataset ID. It is independent from Variable Schema and Dataset Notes, preserves fields unknown to CORE, and is deleted with its Dataset. See DATASET_METADATA_API_v1.md.

Session deletion

CORE exposes one dataset-scoped lifecycle operation:

set result [::GX::datasetDelete $session $datasetId]

The operation removes the Dataset ID and all of its registry metadata from the current session. It deletes the authoritative storage object only when its path is a regular file strictly below the session datasets/ directory and no other registered dataset uses that path. It also removes the dataset note when the note is stored below notes/datasets/.

RAW/source files are never deletion targets, including source paths inside the session raw/ directory or absolute paths outside the session. Analysis outputs, selections, graph configuration and other provider resources are not generically cascade-deleted. CORE Graph Series that directly reference the deleted Dataset ID are pruned immediately; provider-owned sidecars remain the provider’s responsibility and may reconcile their stale references through their public lifecycle.

The returned dictionary contains dataset_id, registry_removed, canonical_storage_deleted, note_deleted, metadata_deleted and cleanup_errors.

v0.8.8: publishing module-produced datasets

Analysis modules must not edit metadata/datasets.ini directly. Dataset Publication API v1 provides the framework-owned route for registering reusable files produced by modules, including contract, format, variables, units, semantic roles, provenance and authoritative SHA-256. See DATASET_PUBLICATION_API_v1.md.

v0.9.5 representation metadata

Canonical datasets may contain storage_kind=text|binary and optional media_type=<MIME>. These fields describe how the canonical dataset is stored; they do not replace or redefine the scientific dataset contract. See BINARY_DATASET_FOUNDATION_v1.md.

v0.9.7.3 variable schema extension

Dataset Variable Schema API v2 adds stable variable IDs, editable post-import schema and typed-value metadata while retaining this registry and existing Dataset IDs. See DATASET_VARIABLE_SCHEMA_API_v2.md.

Dataset deletion and Graph Series references (0.9.7.42)

Deleting a registered dataset also prunes CORE Graph Series records that directly reference that dataset. This prevents stale dataset IDs from making plot editors/runtimes unusable. Providers remain responsible for reconciling any provider-owned sidecar presentation state; plot.dataset 3.3.1 performs that reconciliation lazily and removes dead artifacts when no valid series remain. Surviving Graph Series are never rebound to another dataset automatically.

Canonical Dataset API v2

Canonical Dataset API v2

SPDX-License-Identifier: GPL-3.0-or-later

Canonical Dataset API v2 is the CORE-owned, backend-neutral read contract for GX-TXT 0.9.7.23. It extends Canonical Dataset API v1 without replacing the existing registry, Dataset ID, path resolution, or scientific contract.

Identity and ownership

A logical dataset is identified by its stable Dataset ID. Variables retain the stable pair dataset_id + variable_id. storage_provider selects a CORE implementation and is not a second scientific identity. Physical HDF5 object paths are private provider locators; they are not returned in Variable Schema v3 or Analysis Source runtime metadata.

The datasets.ini registry format remains readable by v1 consumers. Version 2 adds access_api_version=2, storage_provider, cheap counts and additive Variable Schema v3 fields.

Public Tcl operations

set handle [::GX::datasetAccessOpen $session $datasetId]
set metadata [::GX::datasetAccessMetadata $handle]
set capabilities [::GX::datasetAccessCapabilities $handle]

set all [::GX::datasetAccessReadVariables $handle {v0001 v0002}]
set range [::GX::datasetAccessReadRange $handle {v0001 v0002} $start $count]
set binary [::GX::datasetAccessReadRangeBinary $handle {v0002} $start $count]
set transport [::GX::datasetAccessReadRangeTransport $handle {v0001 v0002} $start $count]
set slab [::GX::datasetAccessReadHyperslab $handle v0003 {10 20} {4 8}]
set binarySlab [::GX::datasetAccessReadHyperslabBinary $handle v0003 {10 20} {4 8}]
set chunks [::GX::datasetAccessIterateChunks $handle {v0002} 65536 callback]

::GX::datasetAccessClose $handle

count=-1 means the remaining extent. Reads return a dictionary keyed by stable variable ID. Each result contains dtype, shape, values, start or hyperslab coordinates, count, and whether CORE synthesized the values. Multidimensional variables are never silently flattened by the generic range operation; consumers must request an explicit hyperslab.

datasetAccessWith provides a try/finally convenience wrapper. Closing an already closed handle is harmless. Operations on an unknown handle fail.

Metadata

datasetAccessMetadata returns:

• API version, Dataset ID, scientific contract, format and storage provider;
• row/sample count when known without a payload scan;
• the complete capability dictionary;
• Dataset Variable Schema v3 records;
• logical group IDs.

Metadata is cached per open handle. HDF5 metadata reads object headers, attributes, dtype, shape and chunk information without reading numeric array elements.

Capabilities

Every provider reports all capability keys with a boolean value:

read_full
read_selected_variables
row_range_read
random_access
chunk_read
hyperslab_read
multidimensional
groups
implicit_axis
typed_binary_read

The CSV provider supports full, selected, range and chunk reads and implicit axes. Its range operation scans the text prefix, so random_access=0.

The HDF5 provider supports every listed read capability. A CORE-owned Python worker opens one HDF5 file per access handle and remains alive until close. Repeated chunks therefore reuse one process and one file handle. Consumers do not depend on Python, h5py, HDF5 paths, or the worker protocol.

Uniform implicit axes

A variable with coordinate_role=coordinate, axis_kind=uniform, origin, step and count may have no stored payload. A range request synthesizes exactly the requested values:

value[i] = origin + (start + i) * step

No full coordinate column is allocated.

Elapsed-time numeric contract

For a CSV variable declared with value_kind=elapsed_time and value_format=hms, typed reads parse the supported HMS source representation and expose numeric elapsed seconds. A compatible source unit declaration of s, min, or h remains schema metadata and does not rescale that parsed number. Consumers must not reconstruct or reinterpret the original text. See Elapsed Time Contract v1.

Compatibility

• Existing CSV consumers may keep using Canonical Dataset API v1, Analysis Source paths, Table Stream v2, and Dataset Variable Schema v2.
• New consumers use the same v2 access calls for CSV and HDF5.
• This phase does not materialize HDF5 as CSV for old consumers.
• Publication, writing, indexed CSV, mmap, NetCDF, Zarr, plot loading, FFT, audio, extraction, Spectrogram and CFD remain outside this API version.

Typed binary numeric reads (GX-TXT 0.9.7.37)

Providers advertising typed_binary_read=1 may return compatible numeric ranges and hyperslabs as contiguous binary blocks. The public block dictionary contains:

__gxtxt_binary_block=1
dtype
endian=little|big
shape
byte_count
base64

datasetAccessReadRangeBinary requires a binary-capable numeric variable. datasetAccessReadRangeTransport selects a binary block when the variable and provider support it and otherwise returns the existing values item. This mixed transport operation is intended for generic language bridges. Dataset and variable IDs, schema metadata, range semantics and provider ownership are unchanged.

Supported transport dtypes are float32, float64, int8, int16, int32, int64, uint8, uint16, uint32, uint64 and bool. Text/string and unsupported numeric dtypes use the value fallback. Uniform implicit coordinates are synthesized only for the requested range and encoded with their declared float32 or float64 dtype.

See Typed Binary Read Transport v1 for the provider protocol, validation rules and Octave reconstruction.

Octave Script Bridge

Bridge API ID: CANONICAL_DATASET_API_v2. The provider registers:

read_range retains the Canonical Dataset API bounds and stable variable ID rules. Use a bounded count for large Datasets. The bridge does not expose a Dataset file path or an open storage handle to user scripts.

Dataset Variable Schema API v3

Dataset Variable Schema API v3

SPDX-License-Identifier: GPL-3.0-or-later

Dataset Variable Schema API v3 adds typed and multidimensional metadata to the stable dataset_id + variable_id model from API v2. The existing datasetVariableSchema operation keeps its v2 result fields. New consumers use datasetVariableSchemaV3 and datasetVariableV3ById.

Additive fields

Uniform axes require numeric origin and step plus a nonnegative integer count. Shape and dimension lists must contain exactly rank entries. Relationships reference stable variable IDs in the same logical dataset.

Public operations

set variables [::GX::datasetVariableSchemaV3 $datasetId]
set pressure [::GX::datasetVariableV3ById $datasetId v_pressure]
::GX::datasetVariableSchemaV3Update $datasetId [dict create \
    v_pressure [dict create dtype float32 rank 2 shape {100;5000} \
        dimension_ids {time;cell} coordinate_variable_ids {v_time}]]

API v3 metadata can describe tables, regular coordinates, matrices, fields, mesh coordinates, trajectories and vector components. It does not assign CFD meaning; domain plugins use the Semantic Registry for scientific roles.

CORE storage adapters may keep a private provider locator for each stable variable ID. datasetStorageMapVariable is a CORE integration operation. The locator is omitted from public schema and runtime records so an HDF5 path never becomes scientific identity.

Compatibility

Schema v2 consumers continue to receive the original v2 dictionary. Existing CSV datasets acquire safe v3 defaults only when the v3 API is queried. Use=no variables retain their stable variable IDs and may carry v3 metadata.

Acquisition Handler Plugin API v1 may now return an optional variable_schema declaration with its canonical Dataset descriptor. CORE validates and registers those stable IDs and v3 fields in the same Acquisition transaction as the RAW source, canonical payload and structured metadata. Without a declaration, the prior generated variable-ID behavior remains.

Octave Script Bridge

Bridge API ID: DATASET_VARIABLE_SCHEMA_API_v3. list accepts {dataset_id} and returns a cell of variable descriptor structs. get accepts {dataset_id, variable_id} and returns one descriptor struct. Both operations use the current Session and preserve the normal stable Dataset variable identities. For example:

v = gxtxt_api_call('DATASET_VARIABLE_SCHEMA_API_v3', 'get', ...
                   {'sample', 'v0001'});

Dataset Metadata API v1

Dataset Metadata API v1

SPDX-License-Identifier: GPL-3.0-or-later

Dataset Metadata API v1 stores one generic structured metadata resource for each canonical Dataset ID. It is separate from scientific data, Variable Schema, provenance fields in the Dataset registry, and free-form Dataset Notes.

CORE persists the resource and owns its lifecycle. Importers and other callers define the meaning of fields. CORE does not interpret audio, image, experimental or other domain metadata.

Storage model

The Dataset registry stores structured_metadata_file, a session-relative link to one UTF-8 INI sidecar under:

metadata/datasets/<dataset-stem>.ini

The sidecar records the Dataset ID and an arbitrary set of generic string keys. Dot-separated namespaced keys are recommended for domain-specific metadata, while simple keys are also valid for portable/generator metadata. Unknown keys are retained. Dataset IDs that need filesystem normalization receive a deterministic CRC suffix so distinct IDs cannot share a sidecar accidentally.

Public Tcl API

set metadata [::GX::datasetMetadataRead $session $datasetId]

::GX::datasetMetadataWrite $session $datasetId $metadataDictionary

set merged [::GX::datasetMetadataUpdate \
    $session $datasetId $updatesDictionary]

set remaining [::GX::datasetMetadataUnset \
    $session $datasetId {audio.encoder tags.legacy}]

datasetMetadataRead returns an empty dictionary when no resource has been written yet. datasetMetadataWrite replaces the complete resource explicitly. datasetMetadataUpdate reads and merges the existing resource, so fields unknown to the caller are preserved. datasetMetadataUnset removes only the named keys.

Keys may be simple generic identifiers such as description or function, or dot-separated namespaced identifiers such as audio.title, image.exposure, experimental.temperature or source.original_filename. Key segments may use letters, digits, underscores and hyphens. Values are opaque UTF-8 strings and may contain line breaks in the persistent API.

All write operations verify that the Dataset ID exists. The sidecar path is restricted to the session metadata/datasets directory.

Lifecycle

• A newly imported or published Dataset may receive metadata through the API.
• Metadata survives registry save and session reopen.
• Updating a Dataset does not discard fields owned by other importers.
• datasetDelete removes the sidecar in the same staged lifecycle transaction used for canonical storage and Dataset Notes.
• Deleting one Dataset never removes another Dataset resource, even if a malformed registry attempts to share a path.

Variable Schema and Dataset Notes remain independent APIs and storage resources.

Importer examples

A future audio importer can attach container tags after it creates or publishes the canonical Dataset:

set audioMetadata [dict create \
    audio.title "Calibration sweep" \
    audio.artist "Lab A" \
    audio.album "2026 tests" \
    audio.codec "pcm_s24le" \
    audio.container "wav" \
    source.original_filename "sweep-01.wav"]
::GX::datasetMetadataUpdate $session audio.recording01 $audioMetadata

A future experimental importer can add acquisition conditions without CORE knowing their scientific meaning:

set runMetadata [dict create \
    experimental.temperature_C "350" \
    experimental.pressure_kPa "240" \
    experimental.sample "catalyst-B" \
    experimental.instrument "reactor-02" \
    experimental.reactor_conditions "continuous flow"]
::GX::datasetMetadataUpdate $session reactor.run17 $runMetadata

The two importers use the same API and storage model. No domain-specific fields are built into CORE.

GUI

The Variables workspace exposes Metadata… for the selected imported Dataset. The generic editor lists all fields and supports add/update, remove, Import…, Export… and Save. Changing the selected Dataset closes a stale editor; reopening Metadata… loads the resource for the new selection.

Portable Import / Export text format (GX-TXT 0.9.7.74)

Import… and Export… are convenience operations on the editor’s current in-memory metadata dictionary. They do not introduce a second metadata store and they do not bypass Dataset Metadata API v1.

The portable file is UTF-8 plain text with one field per line:

reference.function=y=sin(x)+0.1*x
reference.sampling=sparse nonuniform
reference.noise=none
description=Interpolation validation dataset

Rules:

• the first = character separates the key from the value, so later = characters remain part of the value;
• blank lines are ignored;
• a line whose first non-whitespace character is # or ; is a comment;
• # and ; inside a normal value are ordinary value characters;
• surrounding whitespace around the key and value is trimmed on import; whitespace inside the value is preserved;
• the whole file is parsed and validated before the editor is changed;
• malformed lines report their line number and do not produce a partial import;
• duplicate keys are deterministic: the last occurrence in the imported file wins;
• importing merges into the current editor state: new keys are added, matching keys are replaced, and unrelated editor fields remain untouched;
• imported changes remain unsaved until the user selects Save;
• closing without Save keeps the existing editor semantics and discards uncommitted changes;
• Export… serializes the current editor state, including unsaved edits, without implicitly saving the Dataset;
• exported keys are sorted deterministically;
• the normal default filename is <dataset>.metadata.txt and ordinary .txt files are also accepted.

Because this portable representation is deliberately one physical field per line, values containing CR/LF line breaks cannot be represented losslessly. Export… therefore reports an explicit error for such a value instead of silently corrupting it. The persistent Dataset Metadata API itself continues to support multiline UTF-8 values.

A typical review flow is:

Open Dataset Metadata
→ Import...
→ review/edit merged fields
→ Save

Export can be used independently:

edit metadata
→ Export...

The exported file can later be imported back into the same or another Dataset Metadata editor, subject to the one-line value rule above.

Acquisition Handler handoff (GX-TXT 0.9.7.38)

Acquisition Handler Plugin API v1 Dataset result descriptors may include an optional structured_metadata dictionary. CORE registers the Dataset and then passes that dictionary to datasetMetadataWrite. Handlers therefore do not write sidecars or wait for registration. Missing structured_metadata preserves the original handler lifecycle. Malformed keys and persistence failures abort the import without retaining a partial Dataset registration.

standard_octave Analysis Source runtime bridge (GX-TXT 0.9.7.57)

Dataset Metadata API v1 remains the sole persistent authority. When a Standard Module runtime is generated, CORE can expose the current Dataset Metadata for each selected Analysis Source through the additive Analysis Source Dataset Metadata Runtime Extension v1.

The runtime bridge serializes indexed exact key / value pairs. Arbitrary Dataset Metadata keys are not normalized into Octave struct fields, so legal keys that differ only by punctuation cannot be silently merged. Missing metadata is represented as an empty/not-found resource and does not fail analysis.

Analysis modules consume this data only through the public Octave CORE helpers and must not open metadata/datasets/*.ini directly. Runtime is derived state: a later call to datasetMetadataWrite, datasetMetadataUpdate or datasetMetadataUnset becomes visible after normal runtime regeneration.

Octave Script Bridge

Bridge API ID: DATASET_METADATA_API_v1.

The current Session supplies the storage context. Keys and values follow the normal Dataset Metadata API validation rules.