Parity baselineCopy Anchor Link

English | 中文

This page records the parity baseline: upstream DeepSeek Harness's own test suites, run on Delorun's reference host at the upstream tag dsh-v0.2.2-rc.1. It holds two passes, a Node 24 check, the classification of each failure that remains, and the environment and host state those results depend on. It also records the green base on v5: the procedure, its prerequisites, its one exclusion and the fixes that made each failing file pass. Terms follow the Delorun glossary.

ContentsCopy Anchor Link

• Reference host and tree
• Passes
• Environment as upstream CI sets it
• Node 24 check
• Classification of the remaining failures
• Host state
• Green base on v5

Reference host and treeCopy Anchor Link

• Tree. The tree is upstream's tag dsh-v0.2.2-rc.1, checked out unmodified. Neither pass edited or committed it; afterwards only git-ignored build output differed.
• Host. Ubuntu 24.04.1 LTS on WSL2, kernel 6.6.87.2-microsoft-standard-WSL2, 22 CPUs, 15 GiB memory and 4 GiB swap.
• Tools. pnpm 11.28.5, the version package.json pins. Playwright 1.61.1. Docker server 29.9.2, reachable.
• Node. Passes 1 and 2 ran Node v26.12.1. The Node 24 check ran Node v24.22.0.
• Load. Other jobs shared the host during both passes. The load average reached about 20 in pass 1 and about 25 in pass 2.
• Prerequisites. pnpm install --frozen-lockfile and pnpm run build:native-system passed. pnpm exec playwright install chromium webkit, run in apps/web, passed and reported the missing WebKit libraries described in Host state. No system packages were installed. The snapshot suite needs built lib/ output, so pass 1 ran it once before any build and it failed.

PassesCopy Anchor Link

Pass 1 used the plain environment. Pass 2 first rebuilt with the CI variables, then ran each suite in turn with the variables in Environment as upstream CI sets it, confined to Linux.

Pass 1: plain environmentCopy Anchor Link

No CI variables, the default build profile, and the unmodified WSL environment.

Solo reruns passed 34 of the 44 failing unit files and 1 of the 12 failing web files. The reruns indicate that these failures came from load on the shared host.

Pass 2: as upstream CI sets itCopy Anchor Link

Pass 2 first ran DSH_TELEMETRY_DISABLED=1 DSH_BUILD_CLIENT_PROFILE=official pnpm run build: exit 0, 49 s, 10 of 10 gates. The official profile recorded 366 client artifacts with 4 public values; the default profile recorded 2.

The snapshot suite runs 9 files in pass 2, against 4 in pass 1, because DSH_EXAMPLE_MODE=lib adds the five apps/web/tests/*.snapshot.ts files. Each failing file of pass 2 was then run alone three times with the same environment; the results are in Classification of the remaining failures.

Changes between the passesCopy Anchor Link

• The unit lane fell from 44 failing files to 28, and from 147 failing tests to 37.
• The web lane fell from 12 failing files to 10, and from 15 failing tests to 11. context-menus.e2e.ts and smoke-real.e2e.ts passed in pass 2; their pass 1 failures did not reproduce.
• The expected lane has the same three failing files, with 6 failing tests instead of 5.
• spawn-runner.spec.ts failed alone in pass 1 and passed in the pass 2 full run.

Scenario countsCopy Anchor Link

The counts are the scenario directories under snapshots/ that contain a snapshot.yml.

Environment as upstream CI sets itCopy Anchor Link

Each variable below takes the value that the upstream workflow files give it for the command this baseline runs. No workflow file sets DSH_SNAPSHOT or sets DSH_EXAMPLE_MODE for the snapshot, expected or web commands; the gate definitions in scripts/run-gates.ts set those.

• DSH_TELEMETRY_DISABLED=1: the top-level env of .github/workflows/ci.yml, sandbox.yml and e2e.yml. Applied to every command in this baseline.
• DSH_BUILD_CLIENT_PROFILE=official: the "Run compatibility smokes" step of the node-compat job in ci.yml, and build-exe-for-python-sdk.yml. The build gates set it through CLIENT_BUILD_PROFILE_SELECTOR in scripts/client-build-environment.ts, which ciBuildGate and ciArtifactBuildGate in scripts/run-gates.ts use. Applied to the pass 2 build and to the build inside test:web.
• DSH_COVERAGE_TEST_TIMEOUT_MS=90000: the job env of node-24-coverage in ci.yml, and the "Unit tests (darwin parity)" step of sandbox.yml, which runs pnpm run check:ci:unit. Applied to pnpm run test.
• DSH_EXAMPLE_MODE=lib: e2e.yml sets it for pnpm run test:e2e, which this baseline does not run. The snapshot and expected gate definitions in scripts/run-gates.ts set it for pnpm run test:snapshot and pnpm run test:expected.
• DSH_SNAPSHOT=replay: the web snapshot gate definition in scripts/run-gates.ts sets it for pnpm run test:web.

Upstream's self-hosted-only variables, such as DSH_COVERAGE_MAX_WORKERS, DSH_GATE_CONCURRENCY and DSH_WEB_SNAPSHOT_WORKERS, resolve to empty strings on hosted runners. This baseline leaves them unset.

WSL confinementCopy Anchor Link

Each command ran under the Linux confinement that Delorun's check tooling applies on WSL, confineToLinux in delorun/apps/tools/src/checks/machine.ts. Upstream CI does not apply it. The shell's PATH lost the 44 entries under the Windows drive mount and kept 25, and these variables were unset: WSL_INTEROP, WSL_DISTRO_NAME, WSLENV, WSL2_GUI_APPS_ENABLED, DISPLAY, WAYLAND_DISPLAY, PULSE_SERVER and NoDefaultCurrentDirectoryInExePath. The variables were set in the shell for each command rather than in each test fork; the removed set is the same, so the suites see the same environment.

Node 24 checkCopy Anchor Link

The .nvmrc pin is 24, and ci.yml sets PRIMARY_NODE_VERSION to '24'. The check put Node v24.22.0 first on PATH, with the CI variables and the confinement kept.

• pnpm install --frozen-lockfile passed: exit 0, 3 s. The native addon was rebuilt under Node 24 and loads with module version 137.
• DSH_BUILD_CLIENT_PROFILE=official pnpm run build passed: exit 0, 95 s, 10 of 10 gates.
• Upstream's node-compat matrix also runs Node 24.9 and 26.

Each file below ran alone three times under Node 24 with the pass 2 environment. The table is a targeted comparison, not a full Node 24 pass, and it does not replace one.

The last five rows are five files drawn at random from class (c) of the classification below.

Node 24 and Node 26 agree on composer-focus.e2e.ts, on the serving case of public-url.expected.e2e.ts, and on the Chromium cases of context-meter.e2e.ts. Three files fail only on Node 26.12.1 in this check: subagent-diagnostic, subagent-inheritance and generation.spec.ts. The plain-HTTP warning case of public-url.expected.e2e.ts is flaky on Node 24.

Classification of the remaining failuresCopy Anchor Link

Pass 2's failing tests fall into four classes. A file's class follows from its solo runs and from the error its output shows. Each failing file belongs to one class, except public-url.expected.e2e.ts, whose IPv6 tests are in class (a) and whose other two tests are in class (d).

The expected column sums to 4 for 3 files because public-url.expected.e2e.ts is counted in classes (a) and (d). The web lane reports 11 failing tests in its total, but the named tests in classes (a) and (d) number 12 (9 WebKit tests, plus 3 in composer-focus.e2e.ts and context-meter.e2e.ts). This baseline does not resolve the difference.

(a) Host stateCopy Anchor Link

WebKit browser launch fails because the host lacks WebKit's GTK and WPE system libraries. Eight web files fail at browserType.launch with the error that the host is missing dependencies to run browsers, 9 tests in all. Each fails 3 of 3 when run alone:

• apps/web/tests/declared-reasoning.e2e.ts
• apps/web/tests/deepseek-messages-settings.e2e.ts
• apps/web/tests/markdown-wide-table.e2e.ts
• apps/web/tests/menu-material.e2e.ts
• apps/web/tests/models-settings.e2e.ts, two tests
• apps/web/tests/session-inspector.e2e.ts
• apps/web/tests/session-replay-reload.e2e.ts
• apps/web/tests/turn-tail-actions.e2e.ts

IPv6 loopback is disabled on the host, so ::1 listeners fail with EADDRNOTAVAIL. Three tests fail this way, each 3 of 3 when run alone:

• packages/test-support/llm-mock-server/tests/server.spec.ts: "formats an IPv6 listener as a valid base URL" (unit).
• apps/cli/tests/profiles/web/tests/public-url.expected.e2e.ts: "prints the IPv6 loopback root for ::1 and serves it", and "prints the IPv6 loopback root for the second IPv6 spelling and serves it" (expected).

(b) Upstream test bug on WSLCopy Anchor Link

These tests do not inject the platform or the kernel release. The WSL2 kernel's release string contains microsoft, so the production WSL branch runs. Each file fails 3 of 3 when run alone.

• packages/host/open-in-app/tests/resolver.spec.ts: "does not offer the Linux file manager without a desktop session". The file never injects osRelease, and isWsl() in packages/util/native-command/src/path-opener.ts falls back to os.release().
• packages/util/native-command/tests/path-opener.spec.ts: "rejects unsupported platforms, cancellation, and launcher errors", and "uses the native runner for a file-manager handoff when none is injected".
• packages/util/native-command/tests/file-applications.spec.ts: "uses the production command adapter and current platform when no override is supplied".
• packages/util/native-command/tests/file-applications-linux-native.spec.ts: "queries and launches an installed Linux desktop entry through GIO". The file injects no platform, so the WSL branch runs and calls powershell.exe, which the confinement removed.

(c) Flaky under the full runCopy Anchor Link

Twenty-two unit files fail in the full pnpm run test run and pass when run alone. Their first full-run error was a 30 s to 180 s budget timeout or a CPython probe timeout. Each passed 3 of 3 solo runs; the criterion is at least 2 of 3.

• packages/client/ui-settings-account/tests/apply.client.spec.ts
• packages/experimental/client-ui-forge/tests/controller.host.spec.ts
• packages/experimental/worktree/tests/worktree.spec.ts
• packages/experimental/ptc-runtime-python/tests/runtime.spec.ts
• packages/experimental/speech-to-text-sensevoice/tests/worker.spec.ts
• packages/schedule/schedule/tests/change-notice.spec.ts
• packages/subagent/subagent/tests/continuation.spec.ts
• packages/subagent/subagent-acp/tests/subagent-acp.spec.ts
• scripts/gen-third-party-notices.spec.ts
• scripts/persistence-epoch-header.spec.ts
• scripts/persistence-schema.spec.ts
• eleven files matching packages/experimental/liveshare/tests/viewer-*.client.spec.tsx

(d) OtherCopy Anchor Link

The causes below were not diagnosed further. Each file fails 3 of 3 when run alone.

• apps/web/tests/composer-focus.e2e.ts: "keeps mobile submission keyboard navigation in the composer while retaining desktop focus" (1 test). The teardown reports llm-replay: fixture not fully consumed: one recorded script was never bound to a live session.
• apps/web/tests/context-meter.e2e.ts: "keeps details visible while resizing the window" for Chat statistics: true and false (2 tests). The teardown error is the same as in composer-focus.e2e.ts. The file launches WebKit, but the WebKit library gap does not explain these two tests.
• apps/cli/tests/profiles/headless/tests/subagent-diagnostic.expected.e2e.ts: "discovers a catalogued child without reading its descriptor through the assembled headless app" (1 test). The expected output lacks the catalogued child's line.
• apps/cli/tests/profiles/headless/tests/subagent-inheritance.expected.e2e.ts: "confines a delegated child through the assembled headless app" (1 test). The deep comparison differs in delegationDepth.
• apps/cli/tests/profiles/web/tests/public-url.expected.e2e.ts: "serves the bound address and a declared authority, refusing any other name" (HTTP request timed out), and "warns about the plain-HTTP bind even when an HTTPS root is advertised" (expected 'SIGKILL' to be undefined), 2 tests.
• packages/session/session-persistence-jsonl/tests/generation.spec.ts: "accepts a valid append on a zstd target created by another migration" (1 test). The append fails because the current session generation already exists and its target bytes do not begin with the migrated generation. No WSL-specific cause was found.

Host stateCopy Anchor Link

• WebKit libraries. The reference host lacks WebKit's GTK and WPE system libraries, so WebKit launches fail in the class (a) files. Chromium resolves fully.
• IPv6 loopback. IPv6 loopback is disabled on the host, so ::1 listeners fail with EADDRNOTAVAIL.
• Owner decision. The owner deferred the host fixes for both items on 2026-11-06. The class (a) failures are host-state results, and this page records them as the host stands. Issue #58 tracks both fixes.
• Load. Other jobs shared the host during both passes, as described in Reference host and tree. The solo reruns of pass 1 and the class (c) files show load-dependent failures.

Green base on v5Copy Anchor Link

The green base is v5 with upstream's unit suite, upstream's recorded-session replay and Delorun's full gate passing, and no known failure carried forward: a test that cannot pass is fixed, or excluded by name with its reason recorded here. The run below was taken at v5 commit 2eb9c53b03 on Node v24.22.0; the fixes were made on top of 0b651c0d32.

ProcedureCopy Anchor Link

On Node 24 (nvm use 24), after pnpm install --frozen-lockfile, with no other test suite running on the host:

DSH_BUILD_CLIENT_PROFILE=official pnpm check:full
env -u NoDefaultCurrentDirectoryInExePath DSH_TELEMETRY_DISABLED=1 DSH_BUILD_CLIENT_PROFILE=official DSH_COVERAGE_TEST_TIMEOUT_MS=90000 \
  pnpm run test -t '^(?!.*(?:mock LLM server wire behaviors formats an IPv6 listener as a valid base URL))'
DSH_EXAMPLE_MODE=lib node_modules/.bin/tsx delorun/apps/tools/src/run-checks.ts --only tests:snapshots --no-cache

• pnpm run test runs upstream's own script, pnpm run build:native-system && vitest run, and pnpm appends the -t argument to it. pnpm run test -- -t … forwards the -- as well, vitest then reads the pattern as file filters, and the exclusion does not apply.
• Vitest matches -t against a test's full name joined with spaces, not against the >-separated form its listing prints.
• A file that fails in the full run is rerun alone. A failure that disappears alone is contention and is recorded under Contention; a failure that stays is fixed or excluded.

PrerequisitesCopy Anchor Link

Upstream's CI runs the suite through pnpm run test, which gives each spec the first, third and fourth settings below; the reference host's first green-base run called vitest directly and lacked them. The second comes from WSL: the Windows side passes NoDefaultCurrentDirectoryInExePath into Linux shells, and upstream's CI never sets it.

P4 is a lead decision for the reference host, made on 2026-11-07: the setting reproduces the value pnpm run test gives every upstream spec in upstream's CI, so it changes nothing the spec checks. The alternative, excluding release pack CLI > does not broaden an empty parallel family to the whole workspace by name, would carry a known failure.

The procedure above provides all four: pnpm run test gives P1, P3 and P4, and env -u gives P2. Delorun's check runner has no lane for upstream's whole unit suite; its lanes get the same settings. checkEnvironment in delorun/apps/tools/src/checks/machine.ts removes P2 with the other Windows-side variables for every check on WSL, and the runner reads P1, P3 and P4 from pnpm for any run that plans a check that runs pnpm or is marked pnpmScript, as tests:upstream-changed is (delorun/apps/tools/README.md).

ResultsCopy Anchor Link

Each failing unit file was rerun alone. 26 files passed alone: their 38 failing tests are contention. 15 files kept failing, with 58 failing tests: 50 that failed in the full run and the rerun, and 8 that failed only in the rerun. The table gives each file's cause and fix.

Each upstream spec edit and the helper have a delorun-feature entry in the divergence ledger.

After the fixes, each of the 15 files ran alone once on Node v24.22.0 with the prerequisites (env -u NoDefaultCurrentDirectoryInExePath pnpm exec vitest run <file>, with the variables of the procedure), and each passed: 800 tests in all, none failed. server.spec.ts with the exclusion passed 45 tests, with the excluded one skipped.

ExclusionCopy Anchor Link

One test is excluded by name: mock LLM server wire behaviors > formats an IPv6 listener as a valid base URL in packages/test-support/llm-mock-server/tests/server.spec.ts. It fails on every run alone with Error: listen EADDRNOTAVAIL: address not available ::1, because IPv6 loopback is disabled on the reference host (Host state). The exclusion ends when the host fix lands; issue #58 tracks it and the WebKit libraries.

ContentionCopy Anchor Link

The host is shared with other agents' suites, so these results depend on the load at the time of the run.

• The 26 files that passed alone were rerun while at least one other vitest process was running.
• packages/subagent/subagent/tests/continuation.spec.ts and packages/experimental/client-ui-forge/tests/local-diff-safety.host.spec.ts each failed in the full run and in a rerun that started while other vitest processes were running. Run alone at 2eb9c53b03, each passed three times out of three (164 and 17 tests); run alone at 0b651c0d32, each passed once. The host's load average was between 7 and 15 during these runs. Neither commit fails, so there was nothing to bisect; no commit between the tag and v5 touches either package, and both files are in class (c) at the tag.
• packages/experimental/client-ui-forge/tests/controller.host.spec.ts had two failures in the full run, both of which passed alone, and one 90000 ms timeout in a rerun that started beside another vitest process. Run alone at 0b651c0d32, with four other vitest processes on the host, it passed 118 tests of 118. It is in class (c) at the tag.

Where to go nextCopy Anchor Link

The parity checklist records the status of each difference from stock DeepSeek Harness. The harness feedback log records problems met while running these suites. The reference index lists the other developer pages.