# Dataset (https://form.dimah.dev/docs/plugins/dataset)



`@dimah-form/dataset` projects stored responses into `dimah.dataset/v1`:
snapshot-correct records plus a codebook. It has no metadata namespace and adds
no tables.

```bash
npm i @dimah-form/dataset
```

```ts title="lib/form.ts"
import { datasetPlugin } from "@dimah-form/dataset";

export const form = dimahForm({
  database,
  forms,
  plugins: [
    datasetPlugin({
      onProject: async ({ record }) => warehouse.upsert(record.id, record),
    }),
  ],
});
```

The client companion is `datasetClientPlugin()` from
`@dimah-form/dataset/client`. Server plugins are not inferred by
`createFormClient`. Authorize `getDatasetPage`, `getDatasetCodebook`, and
`getLiveCodebook` as you would any other admin operation. The package depends
on scoring: a valid scoring snapshot adds derived scores to its record, without
changing stored answers.

```ts title="lib/form-client.ts"
import { datasetClientPlugin } from "@dimah-form/dataset/client";
import { createFormClient } from "@dimah-form/react";
import type { Form } from "@/lib/form";

const plugins = [datasetClientPlugin()] as const;
export const formClient = createFormClient<Form, typeof plugins>({ plugins });
```

```ts
import type { DatasetPluginOptions } from "@dimah-form/dataset";
```

<AutoTypeTable path="packages/dataset/src/plugin.ts" name="DatasetPluginOptions" />

## Fetch pages and codebooks [#fetch-pages-and-codebooks]

```ts
const page = await form.api.getDatasetPage({
  query: { formId: "gad7", limit: 100 },
});

const history = await form.api.getDatasetCodebook({
  query: { formId: "gad7" },
});
```

Dataset response queries default to `status: "submitted"`. Pass another status
explicitly when exporting drafts or abandoned rows. The normal response
filters—respondent, submitted range, and updated range—apply to pages and
historical codebooks.

| Call                 | What it describes                                        |
| -------------------- | -------------------------------------------------------- |
| `getDatasetPage`     | One page of records and its page codebook                |
| `getDatasetCodebook` | Historical union of snapshots in a filtered response set |
| `getLiveCodebook`    | The current live questionnaire only                      |

The history endpoint walks response pages and is capped at 10,000 rows by
default. `truncated` means the codebook is incomplete. Narrow the response
filter or raise the plugin's `maxRows` before creating a full export; query
`maxRows` can lower that cap, never raise it.

## Build an interchange file [#build-an-interchange-file]

HTTP is deliberately paged. Zip files, timeouts, and object storage belong in
your application.

```ts
import { createDatasetReader, toCsv, toJsonl } from "@dimah-form/dataset";

const reader = createDatasetReader({
  page: ({ limit, offset }) =>
    form.api.getDatasetPage({ query: { formId, limit, offset } }),
  codebook: () => form.api.getDatasetCodebook({ query: { formId } }),
});

const { records, codebook } = await reader.readAll();
const jsonl = toJsonl(records);
const csv = toCsv(records, codebook);
```

Records carry a `snapshotKey`: an RFC 8785 SHA-256 fingerprint of the
instrument that produced them. Join a record to the matching historical
codebook entry rather than flattening it against the live form.

`toDataPackage()` produces a Frictionless data package; it returns files, not a
zip archive. Encoders omit `respondentId` by default; include it only for an
authorized export with `{ includeRespondentId: true }`.

```ts
import type {
  CreateDatasetReaderOptions,
  DatasetRecord,
} from "@dimah-form/dataset";
```

### DatasetRecord [#datasetrecord]

<AutoTypeTable path="packages/dataset/src/spec.ts" name="DatasetRecord" />

### CreateDatasetReaderOptions [#createdatasetreaderoptions]

<AutoTypeTable path="packages/dataset/src/reader.ts" name="CreateDatasetReaderOptions" />

### EncodeOptions [#encodeoptions]

<AutoTypeTable path="packages/dataset/src/encode.ts" name="EncodeOptions" />

`onProject` runs after a successful submit has been persisted. Reopening and
submitting again runs it again, so external projections should upsert by
`record.id`. Do not mutate `response.answers`.

## Next [#next]

<Cards>
  <Card title="Insights" href="/docs/plugins/insights" description="Read aggregates for an administrative view." />

  <Card title="Responses" href="/docs/responses" description="Understand why every export is snapshot-based." />
</Cards>