Skip to content

fix: stop log entries from staying pending forever, going to another route's processor, or failing the log phase - #14192

Open
moonming wants to merge 12 commits into
apache:masterfrom
moonming:fix/batch-processor-callback-error
Open

moonming wants to merge 12 commits into
apache:masterfrom
moonming:fix/batch-processor-callback-error

Conversation

@moonming

@moonming moonming commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Description

Since #13826 every logger that goes through batch-processor-manager has a default backlog limit (max_pending_entries, 8192). The manager computes the backlog as total_pushed_entries minus the processed counts it can see, so an entry that never reaches one of those counts stays pending for the rest of the worker's life. Once enough of them have accumulated, the manager discards every new entry of that logger, and neither fixing the log server nor changing the route brings it back. A reload or restart clears the count; raising max_pending_entries only adds room above the entries already stuck as pending. Before 3.18 this only happened when max_pending_entries was configured. The PR also lets plugin.conf_version() encode a JSON null. Every caller shares that function: besides the loggers, limit-conn, limit-count, limit-req, ai-proxy-multi, graphql-proxy-cache, graphql-limit-count, ai-rate-limiting and workflow's limit-count and limit-conn actions no longer raise for a conf that holds one.

This PR closes five paths that left an entry pending forever (two that remain are listed under Notes for reviewers), one that stopped the stale cleanup for good, one that sent entries through another conf's processor, and two that made the log phase of every request fail:

Path Before After
The callback raises an error (e.g. the nil concatenation fixed for splunk-hec-logging in #14012) the timer is aborted with lua entry thread aborted, the batch skips the retry policy and is never counted handled like a false return: retried according to max_retry_count and counted when dropped; logged as failed to process entries: runtime error: ..., with the stack trace when the error is a string
The callback returns false with an error that ngx.log cannot print (a table without __tostring) logging the failure raises, which ends the timer before the batch is counted the error is logged through tostring
ngx.timer.at fails for a batch (too many pending timers, also in the fallback used while the worker exits) the batch is dropped without being counted counted when dropped, and the log line says how many entries were dropped
The stale cleanup (every 30 minutes) runs while a processor has batches running in a timer or waiting for a retry, e.g. any batch_max_size = 1 logger under steady traffic the processor is removed, and what its batches count afterwards goes to an object the manager no longer sums a processor is removed only once everything pushed to it has been counted
log_util.get_log_entry returns nil (for a request that a match condition leaves out, or, with the default log format, include_req_body and no log_format_extra, one whose body could not be read) and the plugin passes it on, as tcp, udp, kafka, rocketmq, clickhouse and, without global_tag, tencent-cloud-cls do counted as pushed and never counted as processed; with batch_max_size = 1, tcp, udp, kafka, rocketmq and clickhouse also sent it as a null record, and tencent-cloud-cls sent an empty request; elasticsearch-logger encoded it as a null document skipped by the manager, and by elasticsearch-logger
The stale-cleanup timer cannot be created stale_timer_running is set anyway, and the cleanup never runs again the next entry tries again
elasticsearch-logger stored the Elasticsearch major version in conf._version that is the field where plugin.conf_version() caches the version the manager keys processors by, so every elasticsearch-logger conf on the same major version shared one processor, bound to the conf that created it: entries of one route were sent to that conf's endpoints with its credentials and headers, and a changed conf kept using that processor until the stale cleanup removed it (since 3.14.0, when #12474 keyed processors by plugin.conf_version(); #12364 had put the major version in that field in 3.13.0) the version is kept outside the conf, and probed when a batch is sent
log_util kept the compiled include_req_body_expr and include_resp_body_expr on the conf, which plugin.conf_version() encodes with sorted keys with include_req_body or include_resp_body on, an exact IPv6 address in an ipmatch of the expression compiles to a matcher that mixes number and string keys, so the encoding raised in the log phase of every request of that conf (since 3.14.0, when #12474 made the manager call conf_version()): nothing was logged by that logger or by the log plugins that run after it (those of lower priority, and every route plugin when the logger is in a global rule), and the passive health check and the balancer's after_balance were skipped. elasticsearch-logger confs were spared only once a version probe had succeeded and set conf._version, which this PR no longer does the compiled expressions are kept outside the conf
a conf holds a JSON null, for example as the operand of ["http_x", "==", null] in an expression or a match plugin.conf_version() could not encode it (dkjson rejects the userdata that cjson decodes null to), with the same result as in the row above for any logger that goes through batch-processor-manager; limit-conn, limit-count, limit-req, ai-proxy-multi and graphql-proxy-cache call conf_version() as well, and so do graphql-limit-count, ai-rate-limiting and workflow's limit-count and limit-conn actions, which run limit-count's or limit-conn's code on their own conf conf_version() encodes it as null; the version of every other conf stays the same

The same field had a second effect: once conf_version() had run for a conf (for example when an earlier plugin rejected the conf's first request before elasticsearch-logger's access phase; the log phase still runs), the version was never probed for that conf, so Elasticsearch 5 and 6 never got the _type they need. The version is now probed when a batch is sent, and all action lines of a batch are built then, from the version of the cluster the batch goes to. The log phase only encodes the document; an entry that cannot be encoded is logged and skipped.

Reproduced with the unmodified apache/apisix:3.19.0-debian image, with worker_processes: 1. Its batch-processor, batch-processor-manager, log-util and elasticsearch-logger are the same as on master and in 3.20.0; splunk-hec-logging and tcp-logger have changed on master since (#14012, #14004):

  • splunk-hec-logging with its default batch settings, pointed at an endpoint that answers 403 {"message":"Forbidden"} (before fix(splunk-hec-logging): safely handle non-200 responses without text field #14012): after 9 batches the log shows total_pushed_entries: 8192 total_processed_entries: 0 and every new entry is discarded, also after the endpoint is changed to one that answers 200. With this PR: 10 failures go through the normal path, nothing is discarded, and the batch reaches the new endpoint.
  • tcp-logger with match on arg_log == yes and max_pending_entries: 3: three requests that do not match fill the backlog (total_pushed_entries: 3 total_processed_entries: 0), and the next matching request is discarded. With this PR the non-matching requests take no slot and the matching one is sent.
  • Two routes with elasticsearch-logger, A pointing at one Elasticsearch 8 endpoint with user a, B at another with user b: after one request to A, every bulk request of B goes to A's endpoint with A's credentials. With this PR each route's entries go to its own endpoint with its own credentials.

Tests

  • t/utils/batch-processor.t
    • TEST 14: a callback that raises is retried and then dropped, processed_entries becomes 1, the error is logged as runtime error: ... with its stack trace, and no lua entry thread aborted appears.
    • TEST 15 and 16: a batch dropped because ngx.timer.at fails is counted, both for a fresh batch and for a retry whose fallback timer fails while the worker exits.
    • TEST 17: a callback that returns false, {...} is counted, with and without first_fail.
    • TEST 18: a callback that raises a table is retried and then dropped, and its error is logged through tostring.
  • t/utils/batch-processor-manager.t
    • TEST 8: with max_pending_entries = 2, two entries whose callback raised do not block the next two.
    • TEST 9: a processor whose batches are still in flight survives the stale sweep, so the entries pushed after the batches finish are accepted.
    • TEST 10: once its batches have finished, the processor is still removed at the next sweep. This one guards against keeping processors forever and passes on master too.
    • TEST 11: nil entries take no pending slot, both when the processor already exists and when a caller goes straight to add_entry_to_new_processor().
    • TEST 12: when the stale-cleanup timer cannot be created, the next entry arms it.
    • TEST 13: plugin.conf_version() gives a conf without null the version it had before (the crc of the default encoding, with sorted keys), encodes null as null, does not give other userdata (cjson.empty_array) the version of a null, and still works after a conf it cannot encode. Broadening the encoder to treat every userdata as null makes this block fail on the new check; the current implementation passes.
  • t/plugin/tcp-logger.t TEST 19: through a real route, the first request builds the processor, the three requests that match leaves out after it do not fill a backlog of 2, and the next matching request is sent.
  • t/plugin/elasticsearch-logger2.t
    • TEST 11 and 12: two routes on the CI's two Elasticsearch 8 services (9200 without auth, 9201 with auth). The block first checks that both report the same major version, which the shared processor needed; then, in two rounds so that the second runs after both confs were probed, the first route's bulk requests go to 9200 and the second route's to 9201, never 9200, each conf's version is probed once, and no version probe or bulk request fails.
    • TEST 13 and 14: when the cluster rejects the version probe (wrong credentials), three requests with batch_max_size = 1 log the failed probe once, not once per request and batch.
    • TEST 15 and 16 (Elasticsearch 6 on 9501, behind a location that delays the version probe by two seconds): three concurrent requests do not wait for the probe, and their bulk requests carry _type.
    • TEST 17 and 18 (Elasticsearch 6 on 9501): a request that key-auth rejects before elasticsearch-logger's access phase is logged with _type.
    • TEST 19 and 20: a route whose include_req_body_expr and include_resp_body_expr have an exact IPv6 address in ipmatch is logged with both bodies, and its log phase does not raise.
    • TEST 21 and 22: a request that match leaves out sends no bulk request.
    • TEST 23 and 24 (Elasticsearch 6 on 9501): in a batch of three entries, every action line carries _type.
    • TEST 25 and 26: a route whose conf holds a JSON null is logged, its log phase does not raise, and the null reaches conf_version().
  • t/plugin/http-logger-json.t TEST 9: http-logger with an exact IPv6 address in both body expressions logs both bodies, and its log phase does not raise.
  • t/plugin/http-logger.t TEST 30 and 31: http-logger with a JSON null in its conf logs the request, its log phase does not raise, and the null reaches conf_version().

Every new test block fails on master, apart from the setup blocks (elasticsearch-logger2.t TEST 11, 13, 15, 17, 19, 21, 23 and 25, http-logger.t TEST 30) and those that pin what master already does: manager TEST 10 (an idle processor is still removed), and elasticsearch-logger2.t TEST 14 (a failed probe is not repeated), TEST 24 (on master the first request's access phase set the version before any action line was built) and TEST 20 (on master an elasticsearch-logger conf whose probe had succeeded never reached conf_version()'s encoding; http-logger-json.t TEST 9 shows the failure on master for http-logger). elasticsearch-logger2.t TEST 26 fails on master only because, for the same reason, its check that the null reaches conf_version() cannot match there; http-logger.t TEST 31 shows the raise on master. Each new guard was also removed on its own to check that a test fails without it, apart from three that need an injected failure to reach, which these tests do not use: the skip of an entry the JSON encoder rejects, a probe answer that is not an Elasticsearch info object, and keeping the version that one batch's probe got when a concurrent batch's probe fails afterwards. Locally (the CI's elasticsearch:8.12.0 image on 9200 and 9201, and elasticsearch:6.7.0 on 9501), the two t/utils files (118 tests) and t/plugin/elasticsearch-logger2.t pass. These logger test files give the same results with and without the change apart from the new tests, when each file runs against a fresh etcd: t/plugin/elasticsearch-logger.t, the seven http-logger*.t, sls-logger.t, loggly.t, syslog.t, tcp-logger.t, udp-logger.t, splunk-hec-logging.t, splunk-hec-logging2.t, the three google-cloud-logging*.t, datadog.t, file-logger.t, lago.t and tencent-cloud-cls.t. The failures that remain locally need other CI services (vector, syslog, Elasticsearch 7 and 9 and so on) and occur on master as well. The kafka, rocketmq, clickhouse, loki and skywalking logger tests need their services and are left to CI, as is luacheck; lj-releng and reindex pass.

Notes for reviewers

  • A raising callback is now retried like a failing one, from the start of the batch. A callback that had already sent part of the batch before raising sends that part again on retry, which is what already happens when it returns false without first_fail. Retries stay bounded by max_retry_count (default 0). The docs sentence (en and zh) says so.
  • elasticsearch-logger probes the version from the batch timer, so no request waits for it, and still does not repeat a probe that failed, as before (the crc that conf_version() stored in conf._version stopped every later probe). Only Elasticsearch 5 and 6 use the version, while retrying would cost each conf a request and an error line per interval for as long as the probe fails, which is for good when the credentials lack the monitor cluster privilege that GET / needs; so this PR does not retry.
  • With the default log format and include_req_body on, a request whose request body could not be read is still not logged, as before (log_util.get_full_log returns nil for it; with a custom log_format it is logged without the body). The loggers in the table's nil-entry row no longer give it a pending slot or send it as a null record or document. With log_format_extra set, in the conf or the plugin metadata, or on an AI route whose ai-proxy or ai-proxy-multi logging turns on summaries or payloads (ctx.llm_summary, ctx.llm_request, ctx.llm_response_text), log_util.get_log_entry itself still raises on that nil before the plugin sees it, as before. Logging such a request without the body would be a change to log_util and is not part of this PR.
  • match is declared only in file-logger's schema; the other loggers accept it as an undeclared property, which log_util honours. These loggers still index a nil entry in their log phase and raise, which this PR does not change: http-logger, loki-logger and, with global_tag, tencent-cloud-cls, both for a request that match leaves out and for one whose request body could not be read; google-cloud-logging and splunk-hec-logging for a request that match leaves out. For such a request, sls-logger, loggly and syslog send a record whose message is null, and skywalking-logger one whose body is null, also unchanged.
  • Not covered here: when the worker's running timers reach lua_max_running_timers, lua-nginx-module drops an expired timer without calling it (lua failed to run timer with function defined at ...: N lua_max_running_timers are not enough). A batch whose timer is dropped this way is never counted either, and with this PR its processor is kept by the stale sweep rather than removed; a dropped buffer timer leaves is_timer_running set with no timer behind it, and when the processor also lost a batch, the sweep no longer clears that by removing it. The processor cannot see these drops, so they need a different fix, tracked in bug: a logger stops logging until reload after its batch timers are dropped at lua_max_running_timers #14204. Also unchanged: when no buffer timer can be created at all (the worker already holds lua_max_pending_timers pending timers), the entries wait for the conf's next entry to arm one, so they stay pending if that conf gets no further entry.
  • The branch merges master to pick up fix(tcp-logger): return error when json encoding fails in batch processor #14004 and fix(splunk-hec-logging): safely handle non-200 responses without text field #14012.
  • Some intermediate commit messages describe code that later commits replaced (pcall instead of xpcall, conf._es_version, a create_buffer_timer change, marking a conf before the version probe). Please squash with the PR title and this description rather than the default list of commit messages.

Changes users may notice

  • elasticsearch-logger: each route's logs go to the endpoints, credentials and headers configured on that route. Since 3.14.0, routes whose clusters reported the same Elasticsearch major version used the processor of the route that logged first, so their logs were sent to that route's cluster.
  • elasticsearch-logger asks the cluster for its version when it sends a conf's first batch, instead of in the access phase of the conf's first request, so no request waits for it, and failed to get Elasticsearch version is now logged from the batch timer.
  • elasticsearch-logger confs no longer share one batch processor because their clusters report the same major version. Each conf batches and sends its own bulk requests again, as before 3.14.0, so many routes with different elasticsearch-logger confs send more, smaller bulk requests.
  • tcp-logger, udp-logger, kafka-logger, rocketmq-logger and clickhouse-logger with batch_max_size: 1 no longer send a null record, tencent-cloud-cls with batch_max_size: 1 and no global_tag no longer sends an empty request, and elasticsearch-logger no longer sends a null document, for a request that match leaves out or whose request body could not be read.
  • A logger that goes through batch-processor-manager and has an exact IPv6 address in an ipmatch of include_req_body_expr or include_resp_body_expr, or a JSON null anywhere in its conf, no longer fails in the log phase of every request. A JSON null no longer makes plugin.conf_version() raise for its other callers either (limit-conn, limit-count, limit-req, ai-proxy-multi, graphql-proxy-cache, and graphql-limit-count, ai-rate-limiting and workflow's limit-count and limit-conn actions).
  • A logger callback that raises is logged as Batch Processor[...] failed to process entries: runtime error: <error>, followed by its stack trace when the error is a string, and retried according to max_retry_count, instead of ending with lua entry thread aborted: runtime error: .... The two failed to create ... process timer lines now end with , dropped entries: N.

Which issue(s) this PR fixes:

Related to #14011, #14012, #14003, #14004

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change (one sentence in the batch processor docs, en and zh)
  • I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

execute_func called the logger callback directly, so a callback that
raised aborted the timer: the batch skipped the retry policy and its
entries were never added to processed_entries. Since the backlog limit
became the default (max_pending_entries = 8192) the backlog is
total_pushed_entries minus the processed entries, so those entries stayed
pending for the worker's lifetime, and after 8192 of them the worker
discarded every new entry of that logger until it was reloaded, even once
the log server was fixed.

Call the callback through pcall and handle a raised error like a false
return, so the batch is retried and counted when it is dropped. Count a
batch as processed when its timer cannot be created as well, which was
the other path that dropped a batch without counting it.
The backlog limit counts an entry as pending until it reaches a
processed count the manager can see. Three more paths kept entries out
of that count for the worker's lifetime:

- the stale cleanup removed a processor while batches of it were still
  running or waiting for a retry, so the count they added afterwards
  went to a processor the manager no longer sums; a processor is now
  removed only once everything pushed to it has been counted
- a nil log entry (log_util returns nil for a request that `match`
  excludes or whose request body could not be read) was counted as
  pushed but stored nowhere; the manager now ignores it
- an err that ngx.log cannot print (a table without __tostring) raised
  while the failure was being logged, ending the timer before the batch
  was counted; err is now passed through tostring

The callback now runs under xpcall with debug.traceback, so a raised
error keeps its stack trace in the log. The new tests wait by polling
instead of fixed sleeps, and the timer-failure tests restore the shared
module afterwards.
@moonming moonming changed the title fix(batch-processor): treat a raising callback as a failed batch fix(batch-processor): stop log entries from staying pending forever Nov 13, 2026
…._version

conf._version is where plugin.conf_version() caches the version of a
conf, and the batch processor manager keys a conf's processor by it.
elasticsearch-logger stored the Elasticsearch major version there, so
every elasticsearch-logger conf on the same major version got the same
key and shared the first conf's processor: entries of one route were
sent to the first route's endpoints with its credentials and headers,
and later conf changes did not take effect. When conf_version() ran
first for a conf, the version probe never ran for it either, so
Elasticsearch 5 and 6 did not get the _type they need.

Keep the major version in conf._es_version instead.
… fails

- create_buffer_timer() left is_timer_running set when flush_buffer
  could not extend the timer, so nothing armed one again and the buffer
  was never flushed by time; it now clears the flag, and also checks the
  fallback timer used while the worker exits
- check_stale() marked the cleanup timer as running even when it could
  not be created, so the stale cleanup never ran again; the next entry
  now tries again
- the lines for a batch dropped because no process timer could be
  created say how many entries were dropped, and a raised callback error
  is logged with a "runtime error: " prefix as it was before

Tests: TEST 16 now injects a failure lua-nginx-module can produce, TEST 17
covers the first_fail branch, manager TEST 11 pins both nil guards, and
tcp-logger TEST 19 checks the nil guard through a real route.
@moonming moonming changed the title fix(batch-processor): stop log entries from staying pending forever fix(batch-processor): stop log entries from staying pending forever or going to another route's processor Nov 13, 2026
…e conf

17bbe01 moved the major version from conf._version to conf._es_version.
That stopped confs on the same major version from sharing one processor,
but conf._version had also been what stopped the probe after a failure:
the log phase stores the conf's crc there, and _es_version was only set
when a probe succeeded. So while GET / failed (the cluster is down, or
the credentials lack the `monitor` cluster privilege), every request
probed again in its access phase, waiting up to `timeout`, and every
batch probed again before its bulk request.

Keep the version in a weak table keyed by the conf, mark the conf before
the probe yields, and do not retry a failed probe, as before. Nothing of
it is stored in the conf any more, so plugin.conf_version() never sees it.

Also:
- restore master's create_buffer_timer and drop batch-processor.t
  TEST 18: lua-nginx-module frees an expiring timer's pending slot before
  it runs the handler, and flush_buffer checks exiting() first, so
  flush_buffer extending its own timer cannot fail, and the case that
  change handled does not occur
- elasticsearch-logger2.t: TEST 12 also fails when a request is not
  answered with 200 or a probe or bulk request fails; TEST 13 and 14 pin
  that a failed probe is not repeated for every request and batch
- tcp-logger.t TEST 19: a matching request comes first, so that the
  requests `match` leaves out reach add_entry() with a processor in
  place; the plugin metadata is removed afterwards
- batch-processor-manager.t TEST 11: other_conf is cloned before conf
  caches its processor key, so that it gets a processor of its own
- docs: add_entry() also returns true for a nil entry
444602d decided `_type` per conf table, in the log phase, and marked a
conf as probed before the probe returned. Both can put an action line
without `_type` into a batch that goes to Elasticsearch 5 or 6, which
rejects the whole bulk request for it:
- requests that arrived while the first probe was in flight skipped it
  and were logged without `_type`
- conf tables with the same content share one processor, but only the
  table that had been probed got `_type`
- a request that an earlier plugin rejected before the access phase was
  logged before any probe, as on master

Probe the version only when a batch is sent, for the conf of the
processor that sends it, and build every action line of the batch then.
The log phase now only encodes the document; an entry that cannot be
encoded is logged and skipped instead of failing every batch it would be
part of. No request waits for the probe any more. Batches that run
before the first probe returns probe as well, and a failed probe is
still not repeated.

Also:
- elasticsearch-logger2.t TEST 12 checks that both clusters run the same
  major version and that the first route's bulk request goes to 9200
- TEST 14 lets each batch end before the next request, since batches
  now do the probing
- TEST 15 to 18 (Elasticsearch 6 on 9501): concurrent requests during a
  slow first probe, and a request that key-auth rejects, are logged with
  `_type`
log_util stored the compiled include_req_body_expr and
include_resp_body_expr on the conf. plugin.conf_version() encodes the
conf with sorted keys, and an exact IPv6 address in an `ipmatch`
compiles to a matcher that mixes number and string keys, so for such a
conf the encoding raised in the log phase of every request: the logger
sent nothing, the log plugins of lower priority did not run, and the
passive health check and the balancer's after_balance were skipped.
elasticsearch-logger confs did not reach that encoding while the version
probe kept conf._version set; since 7d5b35a they do. Keep the compiled
expressions in weak tables keyed by the conf instead.

elasticsearch-logger:
- skip a nil entry (a request that `match` leaves out, or whose body
  could not be read) instead of sending a `null` document, as the
  manager now does for the other loggers
- check the type of the version probe's answer before indexing it: a
  probe that raised failed its batch and left the version unset, so the
  next batch probed and failed again

Tests (elasticsearch-logger2.t):
- TEST 12 runs two rounds, the second after both confs were probed
- TEST 14 waits a second between requests
- TEST 16 delays the probe by a second, checks that the requests do not
  wait for it, and checks the action lines for `_type`; TEST 18 checks
  them too
- TEST 19/20: a conf with an exact IPv6 address in both body
  expressions is logged and its log phase does not raise
- TEST 21/22: a request that `match` leaves out sends no bulk request
A JSON null in a conf, such as the operand of ["http_x", "==", null]
in an expression or a `match`, is cjson.null, a userdata that dkjson
does not encode, so plugin.conf_version() raised for such a conf. For
every logger that goes through batch-processor-manager that happened in
the log phase of every request, with the same result as the IPv6 case
fixed in the previous commit. elasticsearch-logger confs were spared on
master once a version probe had succeeded and set conf._version; since
17bbe01 they are not.

Encode it as null through dkjson's exception handler. sortkey has to be
passed along with it, as it is the default only when no state is given;
the encoding of every conf without a null is unchanged.

Also (these complete the previous commit, whose message says ES confs
reach the encoding since 7d5b35a: it is since 17bbe01):
- batch-processor.t TEST 18: a callback that raises a table is retried
  and dropped, which pins the tostring() on the raised error
- http-logger-json.t TEST 9: http-logger with an exact IPv6 address in
  both body expressions, which fails on master
- elasticsearch-logger2.t: TEST 12 counts the version probes (one per
  conf), TEST 16 delays the probe by two seconds, TEST 16 and 18 reject
  an untyped action line anywhere in the body, TEST 20 checks both
  bodies, TEST 23/24 send a batch of three to Elasticsearch 6, and
  TEST 25/26 log a route whose conf holds a JSON null
@moonming moonming changed the title fix(batch-processor): stop log entries from staying pending forever or going to another route's processor fix: stop log entries from staying pending forever, going to another route's processor, or failing the log phase Nov 13, 2026
- batch-processor-manager.t TEST 13: conf_version() gives a conf without
  null the crc of the default encoding, as before, and encodes a conf
  with null deterministically
- http-logger.t TEST 30/31: http-logger with a JSON null in its conf,
  which fails in the log phase on master
- elasticsearch-logger2.t TEST 26 checks that the null reaches
  conf_version()
- plugin.lua: say why conf_version() builds a new dkjson state on every
  call
…ersion()

- batch-processor-manager.t TEST 13 compares conf_version() with the crc
  of a literal sorted encoding, checks that null is encoded as null, and
  that a conf dkjson cannot encode leaves nothing behind for the next
  one; it fails without sortkey and with a state reused across calls
- http-logger.t TEST 31 checks that the null reaches conf_version()
@moonming
moonming marked this pull request as ready for review November 14, 2026 23:28
@membphis

Copy link
Copy Markdown
Member

Could you add the batch-processor scope and use a shorter title that covers the fixes? Suggested title:

fix(batch-processor): prevent stuck, misrouted, and failed log entries

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants