Dataset

@dimah-form/dataset is the first-party export plugin. It projects stored responses into a versioned spec (dimah.dataset/v1): one DatasetRecord per response, a historical codebook, and optional CSV / Frictionless-style file maps.

Use it when the consumer app needs Excel / warehouse files that stay honest after the live form changes. It is not a dashboard — live counts are @dimah-form/insights.

Canonical interchange is JSON Lines + codebook, not a one-shot CSV download. HTTP stays paged (limit max 100). A zip of the whole form lives in your app — the example download route shows how.

Derived data only. The plugin does not add tables or a meta namespace. Records are built from the response definition snapshot, never the live questionnaire.

Install

pnpm add @dimah-form/dataset

import { datasetPlugin } from "@dimah-form/dataset";
import { dimahForm } from "@dimah-form/server";

export const form = dimahForm({
  database,
  forms,
  plugins: [
    datasetPlugin({
      onProject: async ({ record }) => {
        await warehouse.upsert(record.id, record);
      },
    }),
  ],
});

"use client";

import { datasetClientPlugin } from "@dimah-form/dataset/client";
import { createFormClient } from "@dimah-form/react";
import type { Form } from "@/lib/form";

const plugins = [datasetClientPlugin()] as const;
export const formClient = createFormClient<Form, typeof plugins>({
  plugins,
});

Import the server factory from @dimah-form/dataset and the browser companion from @dimah-form/dataset/client. The package depends on @dimah-form/scoring. getDatasetPage attaches scoreResponse when the snapshot has meta.scoring. A scoring validation error on one historical row omits scores; any other error propagates. Scores are never written into answers.

Guard operations are getDatasetPage, getDatasetCodebook, and getLiveCodebook. Treat them like listResponses (admin).

Two codebooks

formatted on each field is English formatAnswer (protocol). Select labels in the codebook and in responses.labels.csv keep the author's copy (any language). Pass booleanLabels on encode if Yes/No must not be English.

Spec

Every JSON object carries spec: "dimah.dataset/v1".

Record — one response, no raw definition. Identity: id, formId, status, submittedAt, createdAt, updatedAt, snapshotKey. snapshotKey is SHA-256 of the instrument under RFC 8785. HTTP includes respondentId (nullable); package encode omits it unless you pass includeRespondentId: true. fields is dense: every snapshot field, unanswered as value: null. attachment is set only when type is file (id / url / name / contentType / size). Inline bytes are not part of the file answer. Optional scores matches scoreResponse.

Codebook — built from snapshots in the dataset. Newest lastSeenAt wins for the canonical field (label, type, required, description, showWhen, constraints, options, scoring). Each older snapshot whose view differs gets one history object with that full view and its snapshotKey — not one row per response. Join record.snapshotKey to canonical inSnapshots or to history.

Scoring on the codebook comes from readScoringFormMeta / readScoringFieldMeta in @dimah-form/scoring/document. A bad variable, band, or formula row is skipped. A field may carry scoring.variable and scoring.reverse. Options carry points or add, never both; an invalid mix is omitted. scores.variables[] includes missing (zero / omit / incomplete) when the form sets it. Omitted means the scoring default, which is incomplete. scores.formulas[] is { id, op: "sum", vars } plus optional label. Variables, bands (from / to), and formulas get history when their definition differs. Do not merge the canonical label into the live form.

snapshotKey hashes RFC 8785 canonical JSON of title, description, fields, and meta. Form id / slug / status / timestamps are omitted so a touch-save does not fragment the codebook.

HTTP

Reuse core list limits (default 50, max 100). The plugin always passes limit + 1 into the store. Default status=submitted. Same filters as listResponses: respondentId, inclusive submittedFrom / submittedTo, exclusive updatedAfter. Each page includes total for the filter (not the page size).

• GET /dataset/responses (getDatasetPage) — formId required. Returns { spec, records, codebook, limit, offset, nextOffset, total }.
• GET /dataset/codebook/history (getDatasetCodebook) — historical codebook for the same filter. Walks pages on the server (default cap 10_000). Query maxRows can lower that cap, not raise it. truncated is true when more rows remain.
• GET /dataset/codebook (getLiveCodebook) — codebook of the live form.

Filename, zip, and timeout stay out of the plugin.

const page = await form.api.getDatasetPage({
  query: { formId: "gad7", limit: 100 },
});
const history = await form.api.getDatasetCodebook({
  query: { formId: "gad7" },
});

Full file in the app

Walk pages with createDatasetReader. CSV columns need a complete codebook first (getDatasetCodebook or reader.readCodebook()). If truncated is true, do not encode a full CSV — raise datasetPlugin({ maxRows }) or persist with onProject. readCodebook() returns { codebook, total, truncated }. readAll() buffers every page and has no truncated flag — fine for small N. JSONL does not need the historical codebook.

import { createCsvEncoder, createDatasetReader } from "@dimah-form/dataset";

const history = await form.api.getDatasetCodebook({
  query: { formId },
});
const encoder = createCsvEncoder(history.codebook);
const reader = createDatasetReader({
  page: ({ limit, offset }) =>
    form.api.getDatasetPage({
      query: { formId, limit, offset },
    }),
  codebook: async () => history,
  signal,
});

const chunks = [encoder.header()];
for await (const record of reader.records()) {
  chunks.push(encoder.row(record));
}
const csv = chunks.join("");

readAll() still exists for small packs:

import {
  createDatasetReader,
  toCsv,
  toDataPackage,
  toJsonl,
} from "@dimah-form/dataset";

const { records, codebook } = await createDatasetReader({
  page: ({ limit, offset }) =>
    form.api.getDatasetPage({
      query: { formId, limit, offset },
    }),
  signal,
}).readAll();

const jsonl = toJsonl(records);
const csv = toCsv(records, codebook, {
  fields: ["score_item"],
  omit: ["respondentId"],
});
const files = toDataPackage(records, codebook);

toDataPackage returns a filename → contents map (datapackage.json, codebook.json, responses.jsonl, responses.csv, responses.labels.csv). It is not a zip. datapackage.json uses the Frictionless tabular data package profile. The table schema on responses.csv includes missingValues: [""], primaryKey: ["id"], dialect (quoteChar, doubleQuote, \r\n), and codebook constraints (enum on select, minimum / maximum, required). JSONL stays nested records. responses.labels.csv is display text (UTF-8 BOM only there) and has the same dialect, not a schema. CSV is RFC 4180; multiSelect values join with ;. Object answers stringify as JSON in the codes CSV. Column order: identity (including updatedAt), then field columns (historical codebooks sort ids), then score.<id>.raw / .band / .complete / .missing. .missing is the unanswered item count.

A field id equal to an identity column, or starting with score., is written as field.<id>. If a field is also literally named field.<id>, encoding throws. The fields allowlist is still field ids. omit drops output column names.

Pass a live codebook if you want live form order.

Warehouse (onProject)

Same idea as scoring onScore. Persist the projected record in your table. The library will not.

datasetPlugin({
  onProject: async ({ record }) => {
    await warehouse.upsert(record.id, record);
  },
});

onProject runs from afterSubmit (after persist). Upsert by record.id: reopenResponse then submit runs the hook again. Use the response snapshot. Do not flatten against the live questionnaire. Throwing fails the submit HTTP response after the row is already stored.

Out of this package

SPSS / Parquet / FHIR encoders, attachment bytes, meta.dataset, census charts (@dimah-form/insights), LLM calls, raising LIST_MAX_LIMIT, and zip inside dimahForm().