Repository navigation
Conversation
execute_func called the logger callback directly, so a callback that raised aborted the timer: the batch skipped the retry policy and its entries were never added to processed_entries. Since the backlog limit became the default (max_pending_entries = 8192) the backlog is total_pushed_entries minus the processed entries, so those entries stayed pending for the worker's lifetime, and after 8192 of them the worker discarded every new entry of that logger until it was reloaded, even once the log server was fixed. Call the callback through pcall and handle a raised error like a false return, so the batch is retried and counted when it is dropped. Count a batch as processed when its timer cannot be created as well, which was the other path that dropped a batch without counting it.
4 of 5 tasks
The backlog limit counts an entry as pending until it reaches a processed count the manager can see. Three more paths kept entries out of that count for the worker's lifetime: - the stale cleanup removed a processor while batches of it were still running or waiting for a retry, so the count they added afterwards went to a processor the manager no longer sums; a processor is now removed only once everything pushed to it has been counted - a nil log entry (log_util returns nil for a request that `match` excludes or whose request body could not be read) was counted as pushed but stored nowhere; the manager now ignores it - an err that ngx.log cannot print (a table without __tostring) raised while the failure was being logged, ending the timer before the batch was counted; err is now passed through tostring The callback now runs under xpcall with debug.traceback, so a raised error keeps its stack trace in the log. The new tests wait by polling instead of fixed sleeps, and the timer-failure tests restore the shared module afterwards.
…._version conf._version is where plugin.conf_version() caches the version of a conf, and the batch processor manager keys a conf's processor by it. elasticsearch-logger stored the Elasticsearch major version there, so every elasticsearch-logger conf on the same major version got the same key and shared the first conf's processor: entries of one route were sent to the first route's endpoints with its credentials and headers, and later conf changes did not take effect. When conf_version() ran first for a conf, the version probe never ran for it either, so Elasticsearch 5 and 6 did not get the _type they need. Keep the major version in conf._es_version instead.
… fails - create_buffer_timer() left is_timer_running set when flush_buffer could not extend the timer, so nothing armed one again and the buffer was never flushed by time; it now clears the flag, and also checks the fallback timer used while the worker exits - check_stale() marked the cleanup timer as running even when it could not be created, so the stale cleanup never ran again; the next entry now tries again - the lines for a batch dropped because no process timer could be created say how many entries were dropped, and a raised callback error is logged with a "runtime error: " prefix as it was before Tests: TEST 16 now injects a failure lua-nginx-module can produce, TEST 17 covers the first_fail branch, manager TEST 11 pins both nil guards, and tcp-logger TEST 19 checks the nil guard through a real route.
…e conf 17bbe01 moved the major version from conf._version to conf._es_version. That stopped confs on the same major version from sharing one processor, but conf._version had also been what stopped the probe after a failure: the log phase stores the conf's crc there, and _es_version was only set when a probe succeeded. So while GET / failed (the cluster is down, or the credentials lack the `monitor` cluster privilege), every request probed again in its access phase, waiting up to `timeout`, and every batch probed again before its bulk request. Keep the version in a weak table keyed by the conf, mark the conf before the probe yields, and do not retry a failed probe, as before. Nothing of it is stored in the conf any more, so plugin.conf_version() never sees it. Also: - restore master's create_buffer_timer and drop batch-processor.t TEST 18: lua-nginx-module frees an expiring timer's pending slot before it runs the handler, and flush_buffer checks exiting() first, so flush_buffer extending its own timer cannot fail, and the case that change handled does not occur - elasticsearch-logger2.t: TEST 12 also fails when a request is not answered with 200 or a probe or bulk request fails; TEST 13 and 14 pin that a failed probe is not repeated for every request and batch - tcp-logger.t TEST 19: a matching request comes first, so that the requests `match` leaves out reach add_entry() with a processor in place; the plugin metadata is removed afterwards - batch-processor-manager.t TEST 11: other_conf is cloned before conf caches its processor key, so that it gets a processor of its own - docs: add_entry() also returns true for a nil entry
444602d decided `_type` per conf table, in the log phase, and marked a conf as probed before the probe returned. Both can put an action line without `_type` into a batch that goes to Elasticsearch 5 or 6, which rejects the whole bulk request for it: - requests that arrived while the first probe was in flight skipped it and were logged without `_type` - conf tables with the same content share one processor, but only the table that had been probed got `_type` - a request that an earlier plugin rejected before the access phase was logged before any probe, as on master Probe the version only when a batch is sent, for the conf of the processor that sends it, and build every action line of the batch then. The log phase now only encodes the document; an entry that cannot be encoded is logged and skipped instead of failing every batch it would be part of. No request waits for the probe any more. Batches that run before the first probe returns probe as well, and a failed probe is still not repeated. Also: - elasticsearch-logger2.t TEST 12 checks that both clusters run the same major version and that the first route's bulk request goes to 9200 - TEST 14 lets each batch end before the next request, since batches now do the probing - TEST 15 to 18 (Elasticsearch 6 on 9501): concurrent requests during a slow first probe, and a request that key-auth rejects, are logged with `_type`
log_util stored the compiled include_req_body_expr and include_resp_body_expr on the conf. plugin.conf_version() encodes the conf with sorted keys, and an exact IPv6 address in an `ipmatch` compiles to a matcher that mixes number and string keys, so for such a conf the encoding raised in the log phase of every request: the logger sent nothing, the log plugins of lower priority did not run, and the passive health check and the balancer's after_balance were skipped. elasticsearch-logger confs did not reach that encoding while the version probe kept conf._version set; since 7d5b35a they do. Keep the compiled expressions in weak tables keyed by the conf instead. elasticsearch-logger: - skip a nil entry (a request that `match` leaves out, or whose body could not be read) instead of sending a `null` document, as the manager now does for the other loggers - check the type of the version probe's answer before indexing it: a probe that raised failed its batch and left the version unset, so the next batch probed and failed again Tests (elasticsearch-logger2.t): - TEST 12 runs two rounds, the second after both confs were probed - TEST 14 waits a second between requests - TEST 16 delays the probe by a second, checks that the requests do not wait for it, and checks the action lines for `_type`; TEST 18 checks them too - TEST 19/20: a conf with an exact IPv6 address in both body expressions is logged and its log phase does not raise - TEST 21/22: a request that `match` leaves out sends no bulk request
A JSON null in a conf, such as the operand of ["http_x", "==", null] in an expression or a `match`, is cjson.null, a userdata that dkjson does not encode, so plugin.conf_version() raised for such a conf. For every logger that goes through batch-processor-manager that happened in the log phase of every request, with the same result as the IPv6 case fixed in the previous commit. elasticsearch-logger confs were spared on master once a version probe had succeeded and set conf._version; since 17bbe01 they are not. Encode it as null through dkjson's exception handler. sortkey has to be passed along with it, as it is the default only when no state is given; the encoding of every conf without a null is unchanged. Also (these complete the previous commit, whose message says ES confs reach the encoding since 7d5b35a: it is since 17bbe01): - batch-processor.t TEST 18: a callback that raises a table is retried and dropped, which pins the tostring() on the raised error - http-logger-json.t TEST 9: http-logger with an exact IPv6 address in both body expressions, which fails on master - elasticsearch-logger2.t: TEST 12 counts the version probes (one per conf), TEST 16 delays the probe by two seconds, TEST 16 and 18 reject an untyped action line anywhere in the body, TEST 20 checks both bodies, TEST 23/24 send a batch of three to Elasticsearch 6, and TEST 25/26 log a route whose conf holds a JSON null
- batch-processor-manager.t TEST 13: conf_version() gives a conf without null the crc of the default encoding, as before, and encodes a conf with null deterministically - http-logger.t TEST 30/31: http-logger with a JSON null in its conf, which fails in the log phase on master - elasticsearch-logger2.t TEST 26 checks that the null reaches conf_version() - plugin.lua: say why conf_version() builds a new dkjson state on every call
…ersion() - batch-processor-manager.t TEST 13 compares conf_version() with the crc of a literal sorted encoding, checks that null is encoded as null, and that a conf dkjson cannot encode leaves nothing behind for the next one; it fails without sortkey and with a state reused across calls - http-logger.t TEST 31 checks that the null reaches conf_version()
moonming
marked this pull request as ready for review
November 14, 2026 23:28
membphis
approved these changes
Nov 15, 2026
Member
|
Could you add the |
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Since #13826 every logger that goes through batch-processor-manager has a default backlog limit (
max_pending_entries, 8192). The manager computes the backlog astotal_pushed_entriesminus the processed counts it can see, so an entry that never reaches one of those counts stays pending for the rest of the worker's life. Once enough of them have accumulated, the manager discards every new entry of that logger, and neither fixing the log server nor changing the route brings it back. A reload or restart clears the count; raisingmax_pending_entriesonly adds room above the entries already stuck as pending. Before 3.18 this only happened whenmax_pending_entrieswas configured. The PR also letsplugin.conf_version()encode a JSONnull. Every caller shares that function: besides the loggers, limit-conn, limit-count, limit-req, ai-proxy-multi, graphql-proxy-cache, graphql-limit-count, ai-rate-limiting and workflow's limit-count and limit-conn actions no longer raise for a conf that holds one.This PR closes five paths that left an entry pending forever (two that remain are listed under Notes for reviewers), one that stopped the stale cleanup for good, one that sent entries through another conf's processor, and two that made the log phase of every request fail:
lua entry thread aborted, the batch skips the retry policy and is never countedfalsereturn: retried according tomax_retry_countand counted when dropped; logged asfailed to process entries: runtime error: ..., with the stack trace when the error is a stringfalsewith an error thatngx.logcannot print (a table without__tostring)tostringngx.timer.atfails for a batch (too many pending timers, also in the fallback used while the worker exits)batch_max_size = 1logger under steady trafficlog_util.get_log_entryreturns nil (for a request that amatchcondition leaves out, or, with the default log format,include_req_bodyand nolog_format_extra, one whose body could not be read) and the plugin passes it on, as tcp, udp, kafka, rocketmq, clickhouse and, withoutglobal_tag, tencent-cloud-cls dobatch_max_size = 1, tcp, udp, kafka, rocketmq and clickhouse also sent it as anullrecord, and tencent-cloud-cls sent an empty request; elasticsearch-logger encoded it as anulldocumentstale_timer_runningis set anyway, and the cleanup never runs againconf._versionplugin.conf_version()caches the version the manager keys processors by, so every elasticsearch-logger conf on the same major version shared one processor, bound to the conf that created it: entries of one route were sent to that conf's endpoints with its credentials and headers, and a changed conf kept using that processor until the stale cleanup removed it (since 3.14.0, when #12474 keyed processors byplugin.conf_version(); #12364 had put the major version in that field in 3.13.0)include_req_body_exprandinclude_resp_body_expron the conf, whichplugin.conf_version()encodes with sorted keysinclude_req_bodyorinclude_resp_bodyon, an exact IPv6 address in anipmatchof the expression compiles to a matcher that mixes number and string keys, so the encoding raised in the log phase of every request of that conf (since 3.14.0, when #12474 made the manager callconf_version()): nothing was logged by that logger or by the log plugins that run after it (those of lower priority, and every route plugin when the logger is in a global rule), and the passive health check and the balancer'safter_balancewere skipped. elasticsearch-logger confs were spared only once a version probe had succeeded and setconf._version, which this PR no longer doesnull, for example as the operand of["http_x", "==", null]in an expression or amatchplugin.conf_version()could not encode it (dkjson rejects the userdata that cjson decodesnullto), with the same result as in the row above for any logger that goes through batch-processor-manager; limit-conn, limit-count, limit-req, ai-proxy-multi and graphql-proxy-cache callconf_version()as well, and so do graphql-limit-count, ai-rate-limiting and workflow's limit-count and limit-conn actions, which run limit-count's or limit-conn's code on their own confconf_version()encodes it asnull; the version of every other conf stays the sameThe same field had a second effect: once
conf_version()had run for a conf (for example when an earlier plugin rejected the conf's first request before elasticsearch-logger's access phase; the log phase still runs), the version was never probed for that conf, so Elasticsearch 5 and 6 never got the_typethey need. The version is now probed when a batch is sent, and all action lines of a batch are built then, from the version of the cluster the batch goes to. The log phase only encodes the document; an entry that cannot be encoded is logged and skipped.Reproduced with the unmodified
apache/apisix:3.19.0-debianimage, withworker_processes: 1. Its batch-processor, batch-processor-manager, log-util and elasticsearch-logger are the same as on master and in 3.20.0; splunk-hec-logging and tcp-logger have changed on master since (#14012, #14004):splunk-hec-loggingwith its default batch settings, pointed at an endpoint that answers403 {"message":"Forbidden"}(before fix(splunk-hec-logging): safely handle non-200 responses without text field #14012): after 9 batches the log showstotal_pushed_entries: 8192 total_processed_entries: 0and every new entry is discarded, also after the endpoint is changed to one that answers 200. With this PR: 10 failures go through the normal path, nothing is discarded, and the batch reaches the new endpoint.tcp-loggerwithmatchonarg_log == yesandmax_pending_entries: 3: three requests that do not match fill the backlog (total_pushed_entries: 3 total_processed_entries: 0), and the next matching request is discarded. With this PR the non-matching requests take no slot and the matching one is sent.elasticsearch-logger, A pointing at one Elasticsearch 8 endpoint with usera, B at another with userb: after one request to A, every bulk request of B goes to A's endpoint with A's credentials. With this PR each route's entries go to its own endpoint with its own credentials.Tests
t/utils/batch-processor.tprocessed_entriesbecomes 1, the error is logged asruntime error: ...with its stack trace, and nolua entry thread abortedappears.ngx.timer.atfails is counted, both for a fresh batch and for a retry whose fallback timer fails while the worker exits.false, {...}is counted, with and withoutfirst_fail.tostring.t/utils/batch-processor-manager.tmax_pending_entries = 2, two entries whose callback raised do not block the next two.add_entry_to_new_processor().plugin.conf_version()gives a conf withoutnullthe version it had before (the crc of the default encoding, with sorted keys), encodesnullasnull, does not give other userdata (cjson.empty_array) the version of anull, and still works after a conf it cannot encode. Broadening the encoder to treat every userdata asnullmakes this block fail on the new check; the current implementation passes.t/plugin/tcp-logger.tTEST 19: through a real route, the first request builds the processor, the three requests thatmatchleaves out after it do not fill a backlog of 2, and the next matching request is sent.t/plugin/elasticsearch-logger2.tbatch_max_size = 1log the failed probe once, not once per request and batch._type._type.include_req_body_exprandinclude_resp_body_exprhave an exact IPv6 address inipmatchis logged with both bodies, and its log phase does not raise.matchleaves out sends no bulk request._type.nullis logged, its log phase does not raise, and thenullreachesconf_version().t/plugin/http-logger-json.tTEST 9: http-logger with an exact IPv6 address in both body expressions logs both bodies, and its log phase does not raise.t/plugin/http-logger.tTEST 30 and 31: http-logger with a JSONnullin its conf logs the request, its log phase does not raise, and thenullreachesconf_version().Every new test block fails on master, apart from the setup blocks (elasticsearch-logger2.t TEST 11, 13, 15, 17, 19, 21, 23 and 25, http-logger.t TEST 30) and those that pin what master already does: manager TEST 10 (an idle processor is still removed), and elasticsearch-logger2.t TEST 14 (a failed probe is not repeated), TEST 24 (on master the first request's access phase set the version before any action line was built) and TEST 20 (on master an elasticsearch-logger conf whose probe had succeeded never reached
conf_version()'s encoding; http-logger-json.t TEST 9 shows the failure on master for http-logger). elasticsearch-logger2.t TEST 26 fails on master only because, for the same reason, its check that thenullreachesconf_version()cannot match there; http-logger.t TEST 31 shows the raise on master. Each new guard was also removed on its own to check that a test fails without it, apart from three that need an injected failure to reach, which these tests do not use: the skip of an entry the JSON encoder rejects, a probe answer that is not an Elasticsearch info object, and keeping the version that one batch's probe got when a concurrent batch's probe fails afterwards. Locally (the CI'selasticsearch:8.12.0image on 9200 and 9201, andelasticsearch:6.7.0on 9501), the twot/utilsfiles (118 tests) andt/plugin/elasticsearch-logger2.tpass. These logger test files give the same results with and without the change apart from the new tests, when each file runs against a fresh etcd:t/plugin/elasticsearch-logger.t, the sevenhttp-logger*.t,sls-logger.t,loggly.t,syslog.t,tcp-logger.t,udp-logger.t,splunk-hec-logging.t,splunk-hec-logging2.t, the threegoogle-cloud-logging*.t,datadog.t,file-logger.t,lago.tandtencent-cloud-cls.t. The failures that remain locally need other CI services (vector, syslog, Elasticsearch 7 and 9 and so on) and occur on master as well. The kafka, rocketmq, clickhouse, loki and skywalking logger tests need their services and are left to CI, as isluacheck;lj-relengandreindexpass.Notes for reviewers
falsewithoutfirst_fail. Retries stay bounded bymax_retry_count(default 0). The docs sentence (en and zh) says so.conf_version()stored inconf._versionstopped every later probe). Only Elasticsearch 5 and 6 use the version, while retrying would cost each conf a request and an error line per interval for as long as the probe fails, which is for good when the credentials lack themonitorcluster privilege thatGET /needs; so this PR does not retry.include_req_bodyon, a request whose request body could not be read is still not logged, as before (log_util.get_full_logreturns nil for it; with a customlog_formatit is logged without the body). The loggers in the table's nil-entry row no longer give it a pending slot or send it as anullrecord or document. Withlog_format_extraset, in the conf or the plugin metadata, or on an AI route whose ai-proxy or ai-proxy-multiloggingturns onsummariesorpayloads(ctx.llm_summary,ctx.llm_request,ctx.llm_response_text),log_util.get_log_entryitself still raises on that nil before the plugin sees it, as before. Logging such a request without the body would be a change tolog_utiland is not part of this PR.matchis declared only in file-logger's schema; the other loggers accept it as an undeclared property, which log_util honours. These loggers still index a nil entry in their log phase and raise, which this PR does not change: http-logger, loki-logger and, withglobal_tag, tencent-cloud-cls, both for a request thatmatchleaves out and for one whose request body could not be read; google-cloud-logging and splunk-hec-logging for a request thatmatchleaves out. For such a request, sls-logger, loggly and syslog send a record whose message isnull, and skywalking-logger one whose body isnull, also unchanged.lua_max_running_timers, lua-nginx-module drops an expired timer without calling it (lua failed to run timer with function defined at ...: N lua_max_running_timers are not enough). A batch whose timer is dropped this way is never counted either, and with this PR its processor is kept by the stale sweep rather than removed; a dropped buffer timer leavesis_timer_runningset with no timer behind it, and when the processor also lost a batch, the sweep no longer clears that by removing it. The processor cannot see these drops, so they need a different fix, tracked in bug: a logger stops logging until reload after its batch timers are dropped at lua_max_running_timers #14204. Also unchanged: when no buffer timer can be created at all (the worker already holdslua_max_pending_timerspending timers), the entries wait for the conf's next entry to arm one, so they stay pending if that conf gets no further entry.conf._es_version, acreate_buffer_timerchange, marking a conf before the version probe). Please squash with the PR title and this description rather than the default list of commit messages.Changes users may notice
failed to get Elasticsearch versionis now logged from the batch timer.batch_max_size: 1no longer send anullrecord, tencent-cloud-cls withbatch_max_size: 1and noglobal_tagno longer sends an empty request, and elasticsearch-logger no longer sends anulldocument, for a request thatmatchleaves out or whose request body could not be read.ipmatchofinclude_req_body_exprorinclude_resp_body_expr, or a JSONnullanywhere in its conf, no longer fails in the log phase of every request. A JSONnullno longer makesplugin.conf_version()raise for its other callers either (limit-conn, limit-count, limit-req, ai-proxy-multi, graphql-proxy-cache, and graphql-limit-count, ai-rate-limiting and workflow's limit-count and limit-conn actions).Batch Processor[...] failed to process entries: runtime error: <error>, followed by its stack trace when the error is a string, and retried according tomax_retry_count, instead of ending withlua entry thread aborted: runtime error: .... The twofailed to create ... process timerlines now end with, dropped entries: N.Which issue(s) this PR fixes:
Related to #14011, #14012, #14003, #14004
Checklist