Skip to content

fix: recover batch processors after dropped timers - #14208

Open
diaosj wants to merge 1 commit into
apache:masterfrom
diaosj:fix/batch-timer-recovery
Open

diaosj wants to merge 1 commit into
apache:masterfrom
diaosj:fix/batch-timer-recovery

Conversation

@diaosj

@diaosj diaosj commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Description

When OpenResty drops an expired batch timer at lua_max_running_timers, its callback never runs and its entries remain pending forever. Enough dropped batches exhaust the manager's backlog limit, so the logger keeps discarding new entries even after timer pressure has ended. A dropped buffer timer also leaves timeout-based flushing disabled.

Track scheduled timers until their callbacks begin. One worker-wide ngx.timer.every() sweep counts overdue, unstarted batches as dropped and restores lost buffer timers. The deadline includes any retry delay plus a one-second grace period. Running callbacks are excluded, and late callbacks cannot send or account for a recovered batch again. Buffer timer tokens keep an old callback from invalidating a replacement.

ngx.timer.every() schedules its next tick before OpenResty's running-timer check, so the recovery timer itself can miss a tick under pressure and still resume. Recovery reports aggregate dropped entries per processor.

This addresses the dropped-callback case explicitly left out of #14192. It does not include that PR's callback-exception or timer-creation-failure changes.

Which issue(s) this PR fixes:

Fixes #14204

Validation

  • make lint: passed, including 410 Lua files and test-file style checks.
  • Both changed documents passed markdownlint-cli@0.25.0, matching the version pinned in CI.
  • Six regression cases in t/utils/batch-processor-timer.t: their Lua bodies and output/log expectations passed in an isolated OpenResty 1.29.2.4 runner with real timers. Unrelated APISIX schema, metadata and metrics services used fixtures; this was not the full integration suite. Cases 1, 2, 5 and 6 fail against the unchanged implementation; the normal slow-send and delayed-partial-retry controls pass there.
  • The final patch merged cleanly in temporary files with fix: stop log entries from staying pending forever, going to another route's processor, or failing the log phase #14192 at 661609084bd88b61c03762f4719e42f06e256af9; the same six isolated regressions passed with both fixes applied.
  • git diff --check: passed.
  • Full Test::Nginx execution is pending CI. The local prove -I. t/utils/batch-processor-timer.t command could not start because Test::Nginx::Socket::Lua::Stream is not installed.

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

Signed-off-by: Shoujun (Adrian) Diao <shoujun.diao@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: a logger stops logging until reload after its batch timers are dropped at lua_max_running_timers

1 participant