Data: loading, writing and streaming
Everything a page needs to talk to a server comes from @duolith/client:
| To | Use |
|---|---|
| Load a value and keep it fresh | [state.load](#stateload-state-that-loads) |
| Write, with retries, optimistic updates and an offline queue | [action](#action-writes) |
| Follow a live source | [stream](#stream-live-state) |
| Load a long list a page at a time | [pagedList](#pagedlist-load-more) |
| Call a JSON API with typed errors | [api](#api-the-typed-client) |
| Use different settings per build | [environment](#environment-settings-per-build) |
| Check form input | [field, form and rules](#validation) |
| Search and sort a loaded list | [listView](#listview-filter-and-sort) |
| Know whether the device is online | [network](#network) |
Every value these give you is state: bind feed.loading or like.pending in markup and the screen updates when it changes. You never call .value, and nothing else on the screen re-renders.
The runnable example is examples/data, with its mock server in examples/data/server/:
node examples/data/server/server.ts # the mock API on port 4480
duolith dev examples/data --port 4400
state.load: state that loads
import { api, state } from "@duolith/client";
const server = api("https://api.example.com");
const feed = state.load("posts", () => server.get<Post[]>("/posts"));
<text>{feed.loading ? "Loading…" : `${feed.data?.length} posts`}</text>
<list each={feed.data ?? []} as="post"><text>{post.caption}</text></list>
<button on:press={() => feed.refresh()}>Refresh</button>
state.load(key, loader, options) starts loading as soon as it is declared and gives:
| Property | Means |
|---|---|
data | The loaded value, or initial until there is one |
loading | The first value is loading: nothing to show yet |
waiting | when says not to load yet |
showingPrevious | keepPrevious is showing the last key's value while the new key loads |
refreshing | Any load is running, including a refresh with a value already shown |
error, errorMessage | Why the last load failed, as the error and as a sentence |
refresh() | Load again now, even if the value is fresh |
dispose() | Stop following the key: no more reloads, and retries nobody else waits for stop. The last value stays readable. |
state | All of the above as one union: { status: "loading" }, { status: "ready", data, refreshing } or { status: "failed", error, message } |
One state, every case. <match> over feed.state.status must show each status, or the page does not build (E0500, decision 0098); inside a case, feed.state is narrowed, so ready's case can read feed.state.data. A value already shown stays ready while it refreshes and while a refresh fails.
<match value={feed.state.status}>
<case is="loading"><Skeleton lines={3} /></case>
<case is="failed"><ErrorState title="Could not load the posts" /></case>
<case is="ready"><list each={feed.data ?? []} as="post"><text>{post.caption}</text></list></case>
</match>
Loaders. The loader receives the key and { signal }, an AbortSignal that aborts once nothing watches the key any more (the state moved to another key or was disposed). Pass it on to cancel the request itself: (key, { signal }) => server.get("/posts", { signal }). Retries stop either way. Loaded state a page starts is disposed when its screen is dropped (when you go back past it, or its layout goes away); a screen you push past keeps its loaded state, so back shows what it had loaded (navigation).
Keys. The first argument names the value in the cache. Everything loading the same key shares one request and one value. A key is a string or a list: ["posts", 7] lives under "posts", so invalidating "posts" reaches it. Parts may be strings, numbers, bigints, booleans, null, undefined, dates, lists and plain objects; each is compared with its type, so 1 and "1" are different keys and field order in an object does not matter. Anything else (a Map, a Set, a function, a class instance) throws a TypeError naming the part. When the key is a function of other state, the loaded state follows it:
let selected = state(1);
const post = state.load(() => ["posts", selected], (key) => server.get<Post>(`/posts/${key[1]}`));
Options.
| Option | Default | Means |
|---|---|---|
staleTime | 0 | Milliseconds a loaded value counts as fresh. A fresh value is not loaded again when something else asks for it, on focus or on reconnect. |
retry | 3 | Retries for a failed load, after 500 ms, 1 s, 2 s… (see Retries) |
initial | none | The value shown before the first load finishes |
refreshOnFocus | true | Reload a stale value when the window regains focus or the tab becomes visible |
refreshOnReconnect | true | Reload a stale value when the network comes back |
refreshEvery | none | Reload every this many milliseconds while the state is followed and the device is online; a turn that falls while offline is skipped |
forgetAfter | five minutes | Milliseconds a value nobody watches is kept before it is forgotten; Infinity keeps it. A screen you go back past stops watching what it loaded. |
saveOnDevice | false | Keep the last value on the device (localStorage on the web), so the next start shows it at once while a fresh one loads. true keeps it a day; { keepFor } sets the milliseconds. The value must survive JSON.stringify. |
select | none | Picks or reshapes the loaded value: data is what it returns. It runs again only when the loaded value changes. The cache keeps the whole value, so another state.load of the key can select something else. |
keepPrevious | false | While a new key loads, keep showing the last key's value; showingPrevious is true meanwhile, and state stays ready |
when | none | A condition: nothing loads while it is false, and the state is waiting (its state is loading). It is followed like a key function, so when: () => signedIn loads once signedIn turns true. |
const titles = state.load("posts", load, { select: (posts) => posts.map((post) => post.title) }); // data: string[]
const page = state.load(() => ["posts", pageNumber], loadPage, { keepPrevious: true });
const profile = state.load("profile", loadProfile, { when: () => signedIn, saveOnDevice: true });
const clock = state.load("server-time", loadTime, { refreshEvery: 10_000 });
Unchanged parts stay the same. When a reload brings a value equal to the one shown, nothing bound to it updates. When part of it changed, every unchanged list item and object keeps its identity, so a keyed list redraws only the rows that changed. Plain objects and lists are compared part by part; anything else (a date, a Map, a class instance) counts as changed unless it is the same object.
Load ahead. prefetch(key, loader, options) loads a key before the screen that shows it, unless its value is fresh; it takes staleTime, retry, forgetAfter and saveOnDevice. Nothing watches it, so it is forgotten after forgetAfter unless a state.load asks for the key first, which then shows it at once:
import { navigate, prefetch } from "@duolith/client";
const open = (id: number) => prefetch(["posts", id], loadPost).then(() => navigate(`/post/${id}`));
Inspecting the cache. inspectData() lists every key the cache holds now, with its value, whether it is fresh or loading, how many loaded states watch it, waiting optimistic changes, when it was loaded and when it will be forgotten. watchData(listener) calls listener after anything in the cache changes, until the function it returns is called. In a browser's console the same two are __duolith_data.inspect() and __duolith_data.watch(...).
action: writes
const like = action(
(post: Post, { idempotencyKey }) => server.post<Post>(`/posts/${post.id}/like`, undefined, { idempotencyKey }),
{ invalidates: ["posts"] },
);
<button on:press={() => like.run(post)}>{like.pending ? "Liking…" : "Like"}</button>
<text>{like.errorMessage}</text>
run(input) resolves to the output, or to undefined if the write failed or was queued; it never throws, so a handler can call it without a try. The action gives pending, error, errorMessage, data (the last output) and queued.
Idempotency keys. The write function receives { idempotencyKey, attempt }. The key is new for each run and the same for every retry of it, and for a queued write's resend. Pass it to api as { idempotencyKey } (sent as the Idempotency-Key header) so the server applies the write once.
Refresh after a write. invalidates names keys whose values are stale once the write succeeds; all loaded state showing them loads again, and run resolves after they have. It can be a function of the input and output: invalidates: (post) => [["posts", post.id]].
Optimistic updates. Change cached state at once, and put it back if the write fails:
const like = action(sendLike, {
optimistic: {
key: "posts",
update: (posts: Post[], liked) => posts.map((each) => (each.id === liked.id ? { ...each, likes: each.likes + 1 } : each)),
},
invalidates: ["posts"],
});
optimistic takes one update or a list. Each run's change is kept apart from the value the server last sent and shown on top of it, so changes stack: a reload while a write is waiting shows the new value with the change still applied, and when a write fails only that run's change is taken out. Two likes in flight where the second succeeds and the first fails end on the server's count, not on the count from before either.
Queued writes. With queue: "notes", a write made while offline is stored (in localStorage on the web) and sent when the network is back, oldest first. Its optimistic update shows straight away. Writes that lose their connection mid-way are queued too, and tried again after 2 s, 4 s… if the device still says it is online. A queued write survives a reload: the next time an action with the same queue name is declared, it sends what is waiting. If the server refuses a queued write (a 4xx), the write is dropped, error says why, and its optimistic change is taken out (or, after a reload, the keys it touched are loaded again). A write that keeps failing to reach the server while the device says it is online (a wrong address, a request the browser blocks) is given up after maxSendAttempts tries, or once older than maxQueueAge; error is then a QueuedWriteGivenUp, and the writes behind it go ahead. A queued write sent later does not clear error; the next successful run does.
Actions that name the same queue share it: one list of waiting writes, each sent once. They should send the same kind of write, since whichever of them is declared can send the others' writes after a reload. Tabs share a queue too: each follows the others' changes through storage events, and only one tab sends at a time, using the browser's Web Locks. In a browser without Web Locks two tabs can send the same write; its idempotency key lets the server apply it once.
const addNote = action((text: string, { idempotencyKey }) => server.post("/notes", { text }, { idempotencyKey }), {
queue: "notes",
invalidates: ["notes"],
});
<list each={addNote.queued} as="text"><text>Waiting to send: {text}</text></list>
Inputs of a queued action must survive JSON.stringify.
| Option | Default | Means |
|---|---|---|
retry | 2 | Retries for a failed write (safe, because retries reuse the idempotency key) |
invalidates | none | Keys to load again after success |
optimistic | none | Updates applied at once and rolled back on failure |
queue | none | Queued-write mode, under this name |
maxSendAttempts | 10 | Queued mode: tries that cannot reach the server before a write is given up (about eight minutes of backoff) |
maxQueueAge | none | Queued mode: milliseconds after which a write that still cannot reach the server is given up |
stream: live state
const updates = stream<Update>("https://api.example.com/events", { keep: 20 });
<text>{updates.connected ? "Live" : "Connecting…"}</text>
<list each={updates.messages} as="update"><text>{update.text}</text></list>
An address starting with ws: or wss: uses a WebSocket; anything else uses server-sent events. A stream gives latest, messages (newest first), connected and reconnects, plus send(message) on a WebSocket (objects are sent as JSON) and close(). Messages are parsed as JSON, or kept as text when they are not JSON; pass parse to do it yourself. A dropped connection is opened again after 1 s, then 2 s, 4 s… up to 30 s, and back to 1 s once it connects.
| Option | Default | Means |
|---|---|---|
transport | from the address | "sse" or "websocket" |
keep | 50 | How many recent messages messages holds |
event | "message" | The server-sent event name to listen for |
parse | JSON or text | Turns a message's text into a value |
reconnectDelay, maxReconnectDelay | 1000, 30000 | The reconnect backoff, in milliseconds |
pagedList: load more
const products = pagedList((page) => server.get<Page<Product, number>>("/products", { query: { page: page ?? 1 } }));
<list each={products.items} as="product"><text>{product.name}</text></list>
<list each={products.hasMore ? [true] : []} as="more">
<button on:press={() => products.loadMore()}>{products.loading ? "Loading…" : "Load more"}</button>
</list>
The loader receives the cursor the previous page returned as next (undefined for the first page, or first if you pass one) and returns { items, next }. next can be a page number, an offset or the server's cursor; the list ends when a page has no next. It gives items, loading, hasMore, error, errorMessage, loadMore() (calling it again while a page loads does nothing) and refresh() (start again from the first page).
api: the typed client
const server = api("https://api.example.com", { headers: () => ({ authorization: `Bearer ${token}` }) });
const user = await server.get<User>("/users/1");
const results = await server.get<User[]>("/users", { query: { search: "ada", page: 2 } });
await server.post<User>("/users", { name: "Ada" }, { idempotencyKey });
await server.put("/users/1", user);
await server.patch("/users/1", { name: "Ada L." });
await server.delete("/users/1");
Bodies are sent and read as JSON; an empty answer resolves to undefined. Every failure is an ApiError:
kind | When | status, body |
|---|---|---|
"offline" | The device is offline; nothing was sent | unset |
"network" | The server could not be reached | unset |
"status" | The server answered with a status other than 2xx | set |
"parse" | The answer claimed to be JSON but was not | set |
Its message is the server's { message } or { error } when there is one, so errorMessage on loaded state or an action shows the server's own words. error.isConnectionProblem is true for offline and network.
A client typed from the server's OpenAPI description
Put the server's description in the project's api/ folder as <name>.openapi.yaml (or .json). Every build, and duolith dev, writes api/<name>.ts from it before the type check (or run duolith api generate api/<name>.openapi.yaml); duolith check checks against the client as it would be written and writes nothing: a type for each schema, and an interface of every path and its operations. Import it as a type and call the server with typedApi:
import { state, typedApi } from "@duolith/client";
import type { PostsApi } from "../api/posts.ts";
const server = typedApi<PostsApi>("https://api.example.com");
const feed = state.load("posts", () => server.get("/products", { query: { page: 2 } }));
const like = (id: number) => server.post("/posts/{id}/like", { params: { id } });
A path the description does not have for that method, a parameter of the wrong type, a missing required header, query member or body fails the build (E0340), and the answer is typed (feed.data is ProductPage | undefined). Calls go through api(), so options, errors and retries are the same; params fills in the path, query the query string, headers the declared headers (sent as text, with any others you add), body the JSON body. head and options are callable too. A path parameter that is missing, empty, . or .. is refused when the call is made, because it would change which path is called.
Everything in the description is typed, or the build says why not (decision 0109). A description Duolith cannot read is E0516. A schema it cannot type is E0517: a $ref to another file, an inline $ref that leads back to itself (move it under #/components/schemas/, where a schema may refer to itself), or a schema that says nothing about its shape ({}, not:, an array without items). An operation the client cannot call as described is E0518: a cookie parameter, a {name} in the path that no parameter declares, a form or multipart body, an answer that is neither JSON nor text, no 2xx answer (default alone is usually the error), and trace. The data example's pages/typed.duo calls its mock server this way.
Allowing the server in the Content-Security-Policy
Every web build allows requests only to the app's own origin (decision 0150). A server on another origin, the one given to api(), stream() or fetch, is added under connect-src in duolith.yaml; a wss:// source covers a web socket:
security:
content-security-policy:
connect-src: [https://api.example.com, wss://live.example.com]
An address written as text, such as api("https://api.example.com"), is checked when you build: one the policy does not allow is E0551, with the line to add. An address read from environment() is only known when the app runs, so list each origin it may hold; otherwise the browser blocks the request and the load fails with an ApiError of kind "network". examples/data allows its mock server this way. More in security.md.
Retries
Loaded state, actions and paged lists retry with a delay that doubles each time. retry takes a number of retries, false, or { times, delay, maxDelay, when }. By default they retry a dropped connection, 408, 429 and 5xx answers, and never retry other refusals (a 404 will not become a 200) or being offline (the network coming back reloads instead).
environment: settings per build
const settings = environment({ apiUrl: "http://localhost:4480", pageSize: 20, beta: false });
const server = api(settings.apiUrl);
Each setting is named with its default. When the setting is another origin, allow every origin it may hold under connect-src (see api). The build provides values as globalThis.__duolith_environment, an object set before the page's script runs; a provided value is converted to the type of its default ("50" becomes 50, "true" becomes true) and ignored if it cannot be.
Environment files. duolith build and duolith dev read duolith.env, then duolith.<mode>.env (the mode is production for duolith build and duolith check, and development for duolith dev, or --mode staging for any of them, so duolith check --mode development checks what duolith dev would), then the process environment; a later value wins. Each line is NAME=value or a # comment (E0513 otherwise). Only values named DUOLITH_PUBLIC_* reach the app, with the prefix removed and the rest in camel case: DUOLITH_PUBLIC_API_URL is apiUrl. Anything else stays on the build machine; everything that does reach the app can be read by anyone who has it. On the web the values are set by an inline script; a Content-Security-Policy must allow it (A Content-Security-Policy).
# duolith.env
DUOLITH_PUBLIC_API_URL=http://localhost:4480
# duolith.production.env
DUOLITH_PUBLIC_API_URL=https://api.example.com
environment() with no argument returns those values, typed from what they hold (true and false are booleans, numbers are numbers): environment().apiUrl is a string. A name no file defines fails the build (E0514), and so does one defined without the prefix (E0515). On native the bundle sets the same values in its first module, and values the shell sets first win (decision 0105).
Validation
const password = field("", required("Choose a password."), minLength(8));
const signUp = form({
name: field("", required(), maxLength(40)),
email: field("", required(), email()),
password,
confirmation: field("", custom((value) => value === password.value, "The passwords do not match.")),
});
function send() {
signUp.submit((values) => register.run(values));
}
<input value={signUp.fields.email.value} on:input={(event) => (signUp.fields.email.value = event.value)} />
<text class="text-danger">{signUp.fields.email.error}</text>
A field's error is the first failing rule's message, shown once the person has typed in the field or tried to submit; problem is the same message whether shown or not, and isValid says whether it passes. form(...) groups fields: isValid, values(), reset() and submit(send), which shows every error and calls send with the values only when the form is valid.
Rules: required(message?), email(message?), minLength(n, message?), maxLength(n, message?), matches(pattern, message) and custom(test, message). A rule is just a function from the text to a message or undefined, so you can write your own; because rules may read other state, a confirmation can compare against another field and stays up to date.
listView: filter and sort
const shown = listView(() => products.items, {
search: (product, query) => containsText(product.name, query),
where: (product) => product.inStock,
sorts: { name: byText((product: Product) => product.name), price: byNumber((product: Product) => product.price) },
});
<input value={shown.query} on:input={(event) => (shown.query = event.value)} />
<button on:press={() => (shown.sortBy = "price")}>Price</button>
<list each={shown.items} as="product">…</list>
<text>{shown.count} shown</text>
A list view is worked out from its source each time it is read, so it follows both the source (loaded state, a paged list, any state) and its own query, sortBy and descending, which you assign like variables. sortBy is "" for the source's order. Nothing is loaded again: it filters what has already loaded. filterItems and sortItems do the same work as plain functions.
network
network.online is state: true while the device is online. network.pretendOffline(true) makes the whole data layer behave as if it were offline, which is how the example's "Go offline" button works and a handy way to try queued writes; network.pretendOffline(false) brings it back, which sends queued writes and reloads stale loaded state.
Showing parts of a screen only sometimes
Wrap a part of the screen in <show when={...}> to show it only while the condition holds, and follow it with <otherwise> for what to show instead:
<show when={feed.error}>
<text class="text-danger">{feed.errorMessage}</text>
</show>
<otherwise>
<list each={feed.data ?? []} as="post" key={post.id}>
<text>{post.caption}</text>
</list>
</otherwise>
examples/control-flow shows <show>, <otherwise> and keyed lists.
How it is built
The cache behind state.load and actions is the data core (packages/client/src/data/core.ts), an interface with one implementation, MemoryCache, written for Duolith rather than wrapped from TanStack Query (decision 0019). Everything the data layer needs from the browser (fetch, timers, storage, focus and online events, EventSource, WebSocket) goes through one host object, which tests replace with a fake network and clock. The API shape, and why state that loads is state.load, is recorded in decision 0020.