Dataset

@dimah-form/dataset projects stored responses into dimah.dataset/v1: snapshot-correct records plus a codebook. It has no metadata namespace and adds no tables.

npm i @dimah-form/dataset

import { datasetPlugin } from "@dimah-form/dataset";

export const form = dimahForm({
  database,
  forms,
  plugins: [
    datasetPlugin({
      onProject: async ({ record }) => warehouse.upsert(record.id, record),
    }),
  ],
});

The client companion is datasetClientPlugin() from @dimah-form/dataset/client. Server plugins are not inferred by createFormClient. Authorize getDatasetPage, getDatasetCodebook, and getLiveCodebook as you would any other admin operation. The package depends on scoring: a valid scoring snapshot adds derived scores to its record, without changing stored answers.

import { datasetClientPlugin } from "@dimah-form/dataset/client";
import { createFormClient } from "@dimah-form/react";
import type { Form } from "@/lib/form";

const plugins = [datasetClientPlugin()] as const;
export const formClient = createFormClient<Form, typeof plugins>({ plugins });

import type { DatasetPluginOptions } from "@dimah-form/dataset";

Fetch pages and codebooks

const page = await form.api.getDatasetPage({
  query: { formId: "gad7", limit: 100 },
});

const history = await form.api.getDatasetCodebook({
  query: { formId: "gad7" },
});

Dataset response queries default to status: "submitted". Pass another status explicitly when exporting drafts or abandoned rows. The normal response filters—respondent, submitted range, and updated range—apply to pages and historical codebooks.

The history endpoint walks response pages and is capped at 10,000 rows by default. truncated means the codebook is incomplete. Narrow the response filter or raise the plugin's maxRows before creating a full export; query maxRows can lower that cap, never raise it.

Build an interchange file

HTTP is deliberately paged. Zip files, timeouts, and object storage belong in your application.

import { createDatasetReader, toCsv, toJsonl } from "@dimah-form/dataset";

const reader = createDatasetReader({
  page: ({ limit, offset }) =>
    form.api.getDatasetPage({ query: { formId, limit, offset } }),
  codebook: () => form.api.getDatasetCodebook({ query: { formId } }),
});

const { records, codebook } = await reader.readAll();
const jsonl = toJsonl(records);
const csv = toCsv(records, codebook);

Records carry a snapshotKey: an RFC 8785 SHA-256 fingerprint of the instrument that produced them. Join a record to the matching historical codebook entry rather than flattening it against the live form.

toDataPackage() produces a Frictionless data package; it returns files, not a zip archive. Encoders omit respondentId by default; include it only for an authorized export with { includeRespondentId: true }.

import type {
  CreateDatasetReaderOptions,
  DatasetRecord,
} from "@dimah-form/dataset";

DatasetRecord

CreateDatasetReaderOptions

EncodeOptions

onProject runs after a successful submit has been persisted. Reopening and submitting again runs it again, so external projections should upsert by record.id. Do not mutate response.answers.

Next