Repository navigation
Conversation
Refresh the Istio view after the publisher index is updated, so the first registered instance is not missed when event subscribers run out of order. Handle deregistration and client disconnect through the same notification.
Limit the new index notification to explicit instance registration and deregistration. Verify that client release still removes the index without triggering the additional notification.
|
Thanks for your this PR. 🙏 感谢您提交的PR。 🙏 |
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
|
Could we address this by publishing the existing I traced the publishing paths and the current consumers (Naming push, fuzzy watch, and AI agent projection). I couldn't find a consumer that needs the publisher index to remain in its previous state when handling The registration path used to update the index before publishing. The order changed alongside the fuzzy-watch classification logic, but computing the classification before the update does not require publishing the event before the update. With asynchronous dispatch, publishing first cannot reliably make the old index visible to a consumer anyway. A possible adjustment would be:
Reusing this event would also let health changes, instance metadata changes, and client release trigger Istio refreshes. Those paths deserve explicit regression coverage and corresponding clarification in the integration specs. I tried this alternative in an isolated checkout based on Based on that, I'd suggest fixing the ordering at the existing event's source and reusing it here. Is there another consumer or compatibility constraint that would require preserving the current publish-before-update order? |
|
@KomachiSion Thanks for the detailed investigation and suggestion. I’ve updated the PR to publish ServiceChangedEvent after the publisher index is updated and to have the Istio watcher subscribe to it. I haven’t found a consumer that requires the previous publish-before-update order in the paths I checked. I’ve also added regression tests and verified the first EDS response with a standalone Nacos server. One behavior I observed is that, with Istio enabled, client release can refresh the Naming cache sooner than on the baseline. I’ve documented this and the existing SDK test behavior in the PR description. The new CI runs are waiting for maintainer approval. Could you please approve them and take another look when you have time? Thanks! |
| if (changedType.get() != null) { | ||
| NotifyCenter.publishEvent( | ||
| new ServiceEvent.ServiceChangedEvent(service, changedType.get(), true)); |
There was a problem hiding this comment.
[P2] Preserve removal/addition event ordering
I need to correct the removal part of my earlier suggestion: moving this publication outside computeIfPresent opens an event-ordering window that I missed.
After the last publisher is removed, the following interleaving is possible:
- Thread A finishes
computeIfPresent, removing the map entry, and pauses before publishingDELETE_SERVICE. - Thread B registers a new publisher for the same service and publishes
ADD_SERVICE. - Thread A resumes and publishes the earlier
DELETE_SERVICE.
The publisher index now contains the new client, but fuzzy watch processes ADD_SERVICE -> DELETE_SERVICE and removes the live service from its matched set.
The normal client-event queue is serial, but NamingEventPublisher.publish() falls back to synchronous handling in the publishing thread when its queue is full, so concurrent index updates are reachable. I reproduced this with the actual publisher's queue-full fallback, a latch immediately after the map computation completes, and the actual fuzzy-watch context handler. At da18b3b1, the new client remained indexed while the fuzzy-watch matched set became empty. With the same interleaving on the base implementation, the event order was DELETE_SERVICE -> ADD_SERVICE and the service remained matched. The current concurrency test separates the registration and deregistration phases, so it does not cover this case.
Could we retain the registration-order fix and the Istio subscription change, while keeping the previous removal publication placement for this PR? The previous removal path already called ids.remove(clientId) before notifying consumers, and moving this notification past the map-entry removal is not required to fix the first-registration issue. If publication after the computation is needed, we should also preserve the ordering of index mutations and their notifications for the same service across both add/remove paths, with an interleaved deregistration/re-registration regression test.
There was a problem hiding this comment.
Thanks for catching this. I’ve restored the original removal event publication position inside computeIfPresent, while keeping the registration-order fix and the Istio subscription change.
I also added a regression test using a full Naming event queue and an interleaved removal/re-registration. It reproduces the incorrect event order before the fix and verifies that the service remains matched by fuzzy watch after the fix. All 51 focused tests passed locally.
The new CI runs are waiting for maintainer approval. Could you please take another look and approve the runs when you have time? Thanks!
修改目的 / Purpose
修复 #16023:Istio watcher 与 Naming 索引管理器异步处理实例注册时,watcher 可能在首个实例进入发布者索引之前刷新,从而遗漏该实例,直到后续变化再次触发刷新。
Fixes #16023. The Istio watcher could refresh before the first registered instance was added to the Naming publisher index, leaving it absent from the Istio view until a later change triggered another refresh.
修改内容 / Changes
复用现有
ServiceChangedEvent:注册时先更新发布者索引,再发布事件;注销时先移除客户端,并保持在索引计算内发布事件的原有位置,避免在索引项移除后延迟发布删除事件造成的乱序。Istio watcher 订阅该事件并保留回调串行化、推送防抖以及服务和实例元数据事件订阅。
补充注册、注销、客户端释放、健康状态、元数据变化、并发注册/注销,以及事件队列满时注销与重新注册交错的回归覆盖,并更新中英文集成规范。
Reuse
ServiceChangedEvent: update the publisher index before publishing a registration event. For deregistration, remove the client and keep publication inside the index computation, avoiding the ordering window caused by publishing the deletion after removing the map entry.Subscribe the Istio watcher to that event while retaining serialized callbacks, push debounce, and the service and instance metadata event subscriptions.
Add regression coverage for registration, deregistration, client release, health and metadata changes, concurrent registration/deregistration, and interleaved removal/re-registration under the queue-full fallback; update the English and Chinese integration specs.
验证 / Verification
Naming 索引与存储、模糊订阅和推送、AI Agent 投影及 Istio 共 51 项定向回归测试通过;Naming 与 Istio 的 Spotless 和 Checkstyle 检查通过。
交错回归测试使用实际填满的 Naming 事件队列:修复前复现
ADD_SERVICE → DELETE_SERVICE,修复后顺序为DELETE_SERVICE → ADD_SERVICE,且新客户端仍在索引和模糊订阅结果中。使用当前代码构建独立 Nacos 服务并启用 Istio:注册一个健康实例后,首次 EDS 响应及随后 7 次响应均包含该端点。事件处理仍是异步的,与注册竞争的查询不保证立即看到该实例。
通过独立服务对比基线与新方案的 Java SDK 客户端断开和注销不存在实例场景,分别检查启用与关闭 Istio 的行为。
51 focused Naming index/storage, fuzzy-watch/push, AI Agent projection, and Istio regression tests passed; Spotless and Checkstyle passed for Naming and Istio.
With an actually full Naming event queue, the interleaving regression reproduced
ADD_SERVICE → DELETE_SERVICEbefore the fix. After the fix it observedDELETE_SERVICE → ADD_SERVICE, with the new client still indexed and the service still matched by fuzzy watch.Built a standalone server with Istio enabled. After one healthy registration, the first EDS response and seven subsequent responses each contained its endpoint. Event handling remains asynchronous; a query racing registration is not guaranteed to see the instance immediately.
Compared Java SDK client disconnect and missing-instance deregistration against the baseline using standalone servers with Istio enabled and disabled.
调查中观察到的既有行为 / Existing behavior observed
启用 Istio 时,
NamingServiceJavaSdkITCase#testDuplicateRegisterAndMissingDeregisterAreIdempotent在基线和新方案中均在注销另一个实例后的断言处失败。关闭 Istio 时,两者的该用例均通过,但探针观察到后续缓存刷新后已注册实例仍可能消失。该 PR 不修改现有的注销实例身份语义。新方案会在客户端释放时触发 Istio 刷新;本地探针观察到启用 Istio 时,断开客户端的实例从查询结果消失的时间比基线更早。这是观察到的缓存刷新时机变化,不能据此保证所有时序下均相同。
With Istio enabled,
NamingServiceJavaSdkITCase#testDuplicateRegisterAndMissingDeregisterAreIdempotentfails at the assertion after deregistering a different instance on both the baseline and this change. With Istio disabled, the test passes on both, but the probe observed that the registered instance can disappear after a later cache refresh. This PR does not change the existing deregistration identity semantics.The new approach triggers an Istio refresh on client release. In the local probe with Istio enabled, the disconnected client's instance disappeared from query results earlier than on the baseline. This is an observed change in cache-refresh timing, not a guarantee for every execution schedule.