Skip to content

Backport #5136: [world-postgres] Don't move an ended run back to running on a late run_started - #5138

Merged
VaguelySerious merged 1 commit into
stablefrom
backport/pr-5136-to-stable
Oct 31, 2026
Merged

VaguelySerious merged 1 commit into
stablefrom
backport/pr-5136-to-stable

Conversation

@github-actions

Copy link
Copy Markdown
Contributor

Automated backport of #5136 to stable (backport job run).

AI recommendation: This fixes a correctness race in @workflow/world-postgres: a late run_started could move an ended run back to running, which left a corrupt row that later failed runs.get with a ZodError. The same unguarded run_started UPDATE exists on stable in packages/world-postgres/src/storage.ts, so the bug is present there too. The change is a narrow guard plus a regression test, with a patch changeset (some surrounding names such as TERMINAL_WORKFLOW_RUN_STATUSES differ on stable and may need minor adaptation).

Merge conflicts were resolved by AI (opencode with anthropic/claude-opus-5.5). Please review the conflict resolution carefully before merging.

…n_started (#5136)

A `run_started` that arrives just as the run ends used to set the run
back to running. The row was left with its output and completion time
still set, and reading the run then failed. Now the late `run_started`
leaves the ended run alone and is refused, the same answer it would have
gotten had the run already ended when it was checked.

- The UPDATE that marks the run running now skips runs in a terminal
status. If it matches nothing because the run has ended, the event fails
with `RunExpiredError` and no `run_started` is written.

This showed up as `ZodError` failures from `runs.get` in the
world-postgres e2e lanes, in the `start({ hook })` "start or resume"
test: https://github.com/vercel/workflow/actions/runs/40935700151

Added a test that reproduces the race every time. A second connection
ends the run and holds its row lock while `run_started` waits on that
lock. Against a local Postgres 17 the test fails without the fix (the
run comes back `running`) and passes with it. The world-postgres storage
and spec tests pass locally.

---------

Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
@changeset-bot

changeset-bot Bot commented Oct 29, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: e8eeceb

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@workflow/world-postgres Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
example-nextjs-workflow-turbopack Ready Ready Preview, v0 Oct 29, 2026 7:47pm UTC
example-nextjs-workflow-webpack Ready Ready Preview, v0 Oct 29, 2026 7:47pm UTC
example-workflow Ready Ready Preview, v0 Oct 29, 2026 7:47pm UTC
workbench-astro-workflow Ready Ready Preview, v0 Oct 29, 2026 7:47pm UTC
workbench-express-workflow Ready Ready Preview, v0 Oct 29, 2026 7:47pm UTC
workbench-fastify-workflow Ready Ready Preview, v0 Oct 29, 2026 7:47pm UTC
workbench-hono-workflow Ready Ready Preview, v0 Oct 29, 2026 7:47pm UTC
workbench-nestjs-workflow Ready Ready Preview, v0 Oct 29, 2026 7:47pm UTC
workbench-nitro-workflow Ready Ready Preview, v0 Oct 29, 2026 7:47pm UTC
workbench-nuxt-workflow Ready Ready Preview, v0 Oct 29, 2026 7:47pm UTC
workbench-python-workflow Error Error v0 Oct 29, 2026 7:47pm UTC
workbench-sveltekit-workflow Ready Ready Preview, v0 Oct 29, 2026 7:47pm UTC
workbench-tanstack-start-workflow Ready Ready Preview, v0 Oct 29, 2026 7:47pm UTC
workbench-vite-workflow Ready Ready Preview, v0 Oct 29, 2026 7:47pm UTC
workflow-docs Ready Ready Preview, v0 Oct 29, 2026 7:47pm UTC
workflow-swc-playground Ready Ready Preview, v0 Oct 29, 2026 7:47pm UTC
workflow-tarballs Ready Ready Preview, v0 Oct 29, 2026 7:47pm UTC
workflow-web Ready Ready Preview, v0 Oct 29, 2026 7:47pm UTC

@github-actions

github-actions Bot commented Oct 31, 2026 •

Copy link
Copy Markdown
Contributor Author

🧪 E2E Test Results

❌ Some tests failed

Summary

Passed Failed Skipped Total
✅ ▲ Vercel Production 1099 0 78 1177
✅ 💻 Local Development 1198 0 86 1284
✅ 📦 Local Production 1198 0 86 1284
✅ 🐘 Local Postgres 1198 0 86 1284
✅ 🪟 Windows 107 0 0 107
❌ 🌍 Community Worlds 83 105 9 197
✅ 📋 Other 606 0 36 642
Total 5489 105 381 5975

❌ Failed Tests

🌍 Community Worlds (105 failed)

redis (20 failed):

  • hookWorkflow is not resumable via public webhook endpoint | wrun_01M67MCZ3JG5KT0SH2X9JYYVQM
  • parallelStepsThenWebhookWorkflow - no hook_conflict from same-tick replay race | wrun_01M67MD9P53Y39DADKK0D4FJCJ
  • sleepingWorkflow | wrun_01M67MECR5RS811W0S31EX3G7Q
  • outputStreamWorkflow negative startIndex (reads from end)
  • outputStreamWorkflow - getTailIndex and getStreamChunks getTailIndex returns correct index after stream completes
  • outputStreamWorkflow - getTailIndex and getStreamChunks getTailIndex returns -1 before any chunks are written
  • outputStreamWorkflow - getTailIndex and getStreamChunks getStreamChunks returns same content as reading the stream
  • error handling serialization failures step-argument serialization failure is catchable in workflow code
  • error handling serialization failures uncaught step-argument serialization failure fails the run as USER_ERROR without redelivery retries
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_01M67MPBC4THT8GTYRMMJZP9WK
  • hookGetConflictWorkflow - awaiting hook.getConflict() registers hook without payload | wrun_01M67MPP3G1VRHM4842YWHWG1R
  • hookGetConflictThenStepParallelWorkflow - hook.getConflict() continuation step runs alongside other steps | wrun_01M67MPXHWAT4BVE9QYQGRWPMS
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_01M67MQAX12EXWQDZ242D8DFM0
  • hookClaimOnlyMutexWorkflow - hook works as a pure run mutex without payload data | wrun_01M67MR6SVXES16CK2WFRPXF97
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_01M67MRBX1PF6BFQVYAHDZ0RM6
  • hookSignalOwnerWorkflow - duplicate forwards its payload to the owner via resumeHook | wrun_01M67MRJXJSB24MFPQX6FW8C0W
  • hookSupersedeOwnerWorkflow - duplicate cancels the owner and claims the released token | wrun_01M67MRR4QAX1H5XNYNW33FHDB
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_01M67MS0ESG50H54977R82RE6K
  • pages router sleepingWorkflow via pages router
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_01M67MY3GZVQRNTEAVKJ36DNH3

turso (85 failed):

  • addTenWorkflow | wrun_01M67MBJEH7HZ5YXNYMXTAFD9J
  • addTenWorkflow | wrun_01M67MBJEH7HZ5YXNYMXTAFD9J
  • deploymentId: 'latest' is a no-op in non-Vercel worlds
  • wellKnownAgentWorkflow (.well-known/agent) | wrun_01M67MCF3XC825A789V29XB6CC
  • should work with react rendering in step
  • promiseAllWorkflow | wrun_01M67MBTZ1TQKMBEXYEAMV6DYZ
  • promiseRaceWorkflow | wrun_01M67MC25BQ4RTTV6RA8NJ75XF
  • promiseAnyWorkflow | wrun_01M67MC4JY5JYAZJ1823BCMY2M
  • importedStepOnlyWorkflow | wrun_01M67MCWEVXZV5V0MMGJFMHY2D
  • readableStreamWorkflow | wrun_01M67MC74JXDMS1616P8V1RRCK
  • hookWorkflow | wrun_01M67MCPWYHD5Z458CX2MWB852
  • hookWorkflow is not resumable via public webhook endpoint | wrun_01M67MCZ3JG5KT0SH2X9JYYVQM
  • webhookWorkflow | wrun_01M67MD3YRS2J1BMNP3TKV98CV
  • parallelStepsThenWebhookWorkflow - no hook_conflict from same-tick replay race | wrun_01M67MD9P53Y39DADKK0D4FJCJ
  • sleepingWorkflow | wrun_01M67MECR5RS811W0S31EX3G7Q
  • parallelSleepWorkflow | wrun_01M67MEVAKAH5T5RM9C99S2XX4
  • sleepWinsRaceWorkflow | wrun_01M67MF02868EJQBJQ0EEW3P8A
  • stepWinsRaceWorkflow | wrun_01M67MF3W2CAWH9KT85TTFC3JW
  • nullByteWorkflow | wrun_01M67MF7FH46RMR6Y9M2X1DEC6
  • workflowAndStepMetadataWorkflow | wrun_01M67MF9X0PR0V8TFKN0TRD4X5
  • outputStreamWorkflow no startIndex (reads all chunks)
  • outputStreamWorkflow positive startIndex (skips first chunk)
  • outputStreamWorkflow negative startIndex (reads from end)
  • outputStreamWorkflow - getTailIndex and getStreamChunks getTailIndex returns correct index after stream completes
  • outputStreamWorkflow - getTailIndex and getStreamChunks getTailIndex returns -1 before any chunks are written
  • outputStreamWorkflow - getTailIndex and getStreamChunks getStreamChunks returns same content as reading the stream
  • outputStreamInsideStepWorkflow - getWritable() called inside step functions | wrun_01M67MHQYQWRYKXDS0HRBHM723
  • writableForwardedFromWorkflowWorkflow | wrun_01M67MJ5EXHJ46KMRS59PG9X4H
  • writableForwardedFromStepWorkflow | wrun_01M67MJ90C6HG35C982BW5JGMQ
  • fetchWorkflow | wrun_01M67MJCVDS0AHT30AG552Q56P
  • promiseRaceStressTestWorkflow | wrun_01M67MJGESFGSSD2DK4M214TS5
  • error handling error propagation workflow errors nested function calls preserve message and stack trace
  • error handling error propagation workflow errors cross-file imports preserve message and stack trace
  • error handling error propagation step errors basic step error preserves message and stack trace
  • error handling error propagation step errors cross-file step error preserves message and function names in stack
  • error handling retry behavior regular Error retries until success
  • error handling retry behavior FatalError fails immediately without retries
  • error handling retry behavior RetryableError respects custom retryAfter delay
  • error handling retry behavior maxRetries=0 disables retries
  • error handling catchability FatalError can be caught and detected with FatalError.is()
  • error handling serialization failures step-argument serialization failure is catchable in workflow code
  • error handling serialization failures uncaught step-argument serialization failure fails the run as USER_ERROR without redelivery retries
  • error handling not registered WorkflowNotRegisteredError fails the run when workflow does not exist
  • error handling not registered StepNotRegisteredError fails the step but workflow can catch it
  • error handling not registered StepNotRegisteredError fails the run when not caught in workflow
  • hookCleanupTestWorkflow - hook token reuse after workflow completion | wrun_01M67MP1587ZYM45DK8FEZRS34
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously | wrun_01M67MPBC4THT8GTYRMMJZP9WK
  • hookGetConflictWorkflow - awaiting hook.getConflict() registers hook without payload | wrun_01M67MPP3G1VRHM4842YWHWG1R
  • 'hookGetConflictWithPriorStepWorkflow' - hook.getConflict() does not block step execution | wrun_01M67MPRJS78MWYEB05DCGTDW4
  • 'hookGetConflictWithParallelStepWorkfl…' - hook.getConflict() does not block step execution | wrun_01M67MPV1YF39K12P16PRT64WK
  • hookGetConflictThenStepParallelWorkflow - hook.getConflict() continuation step runs alongside other steps | wrun_01M67MPXHWAT4BVE9QYQGRWPMS
  • hookGetConflictWorkflow - hook.getConflict() resolves with the conflicting run when token is already registered | wrun_01M67MQAX12EXWQDZ242D8DFM0
  • hookClaimOnlyMutexWorkflow - hook works as a pure run mutex without payload data | wrun_01M67MR6SVXES16CK2WFRPXF97
  • hookAdoptOwnerResultWorkflow - duplicate adopts the owner result via conflict.returnValue | wrun_01M67MRBX1PF6BFQVYAHDZ0RM6
  • hookSignalOwnerWorkflow - duplicate forwards its payload to the owner via resumeHook | wrun_01M67MRJXJSB24MFPQX6FW8C0W
  • hookSupersedeOwnerWorkflow - duplicate cancels the owner and claims the released token | wrun_01M67MRR4QAX1H5XNYNW33FHDB
  • resume-or-start route pattern - resumeHook retried after start() reaches the new run | wrun_01M67MS0ESG50H54977R82RE6K
  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running | wrun_01M67MS7SG9ANA8KG523PD6MF4
  • stepFunctionPassingWorkflow - step function references can be passed as arguments (without closure vars) | wrun_01M67MSNSXC6EW6988B3KCGHJB
  • stepFunctionWithClosureWorkflow - step function with closure variables passed as argument | wrun_01M67MSXAWWW42Y54ZG7R66WB5
  • closureVariableWorkflow - nested step functions with closure variables | wrun_01M67MT2C0ZDJ3AWTP6D542VQ1
  • spawnWorkflowFromStepWorkflow - spawning a child workflow using start() inside a step | wrun_01M67MT50WABQM1XH0G0N89A6S
  • health check (queue-based) - workflow and step endpoints respond to health check messages
  • health check (CLI) - workflow health command reports healthy endpoints
  • pathsAliasWorkflow - TypeScript path aliases resolve correctly | wrun_01M67MTJQ5ZB0J9RVA75QT84T7
  • Calculator.calculate - static workflow method using static step methods from another class | wrun_01M67MTQMSMC41PTZE71MR3QCA
  • AllInOneService.processNumber - static workflow method using sibling static step methods | wrun_01M67MTXQ3KB57Q4R91ASJB50D
  • ChainableService.processWithThis - static step methods using this to reference the class | wrun_01M67MV40EQKS4P9TJJ812KQ82
  • thisSerializationWorkflow - step function invoked with .call() and .apply() | wrun_01M67MVA4FQF6B6C451C8MA8GW
  • customSerializationWorkflow - custom class serialization with WORKFLOW_SERIALIZE/WORKFLOW_DESERIALIZE | wrun_01M67MVG7GTVDVPVSBVN6ZT6QH
  • instanceMethodStepWorkflow - instance methods with "use step" directive | wrun_01M67MVPC7RHHK591814F9J7ST
  • crossContextSerdeWorkflow - classes defined in step code are deserializable in workflow context | wrun_01M67MW1EF4S8799CMF6P5JTF7
  • stepFunctionAsStartArgWorkflow - step function reference passed as start() argument | wrun_01M67MW8S6DFDV54AGHXE8BP8E
  • cancelRun - cancelling a running workflow | wrun_01M67MWERN4QE6T69T8JEQ78NE
  • cancelRun via CLI - cancelling a running workflow | wrun_01M67MWK43C5JEH18WZ774EZK7
  • pages router addTenWorkflow via pages router
  • pages router promiseAllWorkflow via pages router
  • pages router sleepingWorkflow via pages router
  • hookWithSleepWorkflow - hook payloads delivered correctly with concurrent sleep | wrun_01M67MWS8P9P0XN7M7W404GHRC
  • hookWithSleepFinalStepWorkflow - step only on final payload | wrun_01M67MX5K0EX15B7FA2M6M2X2T
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration | wrun_01M67MXDGXPS6XV29B2FCNN0JZ
  • sleepWithSequentialStepsWorkflow - sequential steps work with concurrent sleep (control) | wrun_01M67MXRC38PGMAQSE1TCFPX44
  • importMetaUrlWorkflow - import.meta.url is available in step bundles | wrun_01M67MXYRBD3JW37JAM23WXCYS
  • metadataFromHelperWorkflow - getWorkflowMetadata/getStepMetadata work from module-level helper (#1577) | wrun_01M67MY1466FHGDBN9SA7697QA
  • resilient start: addTenWorkflow completes when run_created returns 500 | wrun_01M67MY3GZVQRNTEAVKJ36DNH3

Details by Category

✅ ▲ Vercel Production
App Passed Failed Skipped
✅ astro 99 0 8
✅ example 99 0 8
✅ express 99 0 8
✅ fastify 99 0 8
✅ hono 99 0 8
✅ nextjs-turbopack 104 0 3
✅ nextjs-webpack 104 0 3
✅ nitro 99 0 8
✅ nuxt 99 0 8
✅ sveltekit 99 0 8
✅ vite 99 0 8
✅ 💻 Local Development
App Passed Failed Skipped
✅ astro-stable 101 0 6
✅ express-stable 101 0 6
✅ fastify-stable 101 0 6
✅ hono-stable 101 0 6
✅ nextjs-turbopack-canary 88 0 19
✅ nextjs-turbopack-stable 107 0 0
✅ nextjs-webpack-canary 88 0 19
✅ nextjs-webpack-stable 107 0 0
✅ nitro-stable 101 0 6
✅ nuxt-stable 101 0 6
✅ sveltekit-stable 101 0 6
✅ vite-stable 101 0 6
✅ 📦 Local Production
App Passed Failed Skipped
✅ astro-stable 101 0 6
✅ express-stable 101 0 6
✅ fastify-stable 101 0 6
✅ hono-stable 101 0 6
✅ nextjs-turbopack-canary 88 0 19
✅ nextjs-turbopack-stable 107 0 0
✅ nextjs-webpack-canary 88 0 19
✅ nextjs-webpack-stable 107 0 0
✅ nitro-stable 101 0 6
✅ nuxt-stable 101 0 6
✅ sveltekit-stable 101 0 6
✅ vite-stable 101 0 6
✅ 🐘 Local Postgres
App Passed Failed Skipped
✅ astro-stable 101 0 6
✅ express-stable 101 0 6
✅ fastify-stable 101 0 6
✅ hono-stable 101 0 6
✅ nextjs-turbopack-canary 88 0 19
✅ nextjs-turbopack-stable 107 0 0
✅ nextjs-webpack-canary 88 0 19
✅ nextjs-webpack-stable 107 0 0
✅ nitro-stable 101 0 6
✅ nuxt-stable 101 0 6
✅ sveltekit-stable 101 0 6
✅ vite-stable 101 0 6
✅ 🪟 Windows
App Passed Failed Skipped
✅ nextjs-turbopack 107 0 0
❌ 🌍 Community Worlds
App Passed Failed Skipped
✅ mongodb-dev 4 0 3
✅ redis-dev 4 0 3
❌ redis 68 20 0
✅ turso-dev 4 0 3
❌ turso 3 85 0
✅ 📋 Other
App Passed Failed Skipped
✅ e2e-local-dev-nest-stable 101 0 6
✅ e2e-local-dev-tanstack-start-stable 101 0 6
✅ e2e-local-postgres-nest-stable 101 0 6
✅ e2e-local-postgres-tanstack-start-stable 101 0 6
✅ e2e-local-prod-nest-stable 101 0 6
✅ e2e-local-prod-tanstack-start-stable 101 0 6

📋 View full workflow run

@VaguelySerious
VaguelySerious enabled auto-merge (squash) October 31, 2026 00:03
@VaguelySerious
VaguelySerious merged commit 5c9b65d into stable Oct 31, 2026
94 of 98 checks passed
@VaguelySerious
VaguelySerious deleted the backport/pr-5136-to-stable branch October 31, 2026 00:16

This branch was successfully deployed

1 active deployment
Preview – workflow-docs — e8eeceb4 Deployed Oct 29, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant