Skip to content

[ISSUE #16023] Fix missing Istio endpoints on first registration - #16027

Open
iyuan7 wants to merge 4 commits into
alibaba:developfrom
iyuan7:fix/issue-16023-istio-index-refresh
Open

iyuan7 wants to merge 4 commits into
alibaba:developfrom
iyuan7:fix/issue-16023-istio-index-refresh

Conversation

@iyuan7

@iyuan7 iyuan7 commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

修改目的 / Purpose

修复 #16023:Istio watcher 与 Naming 索引管理器异步处理实例注册时,watcher 可能在首个实例进入发布者索引之前刷新,从而遗漏该实例,直到后续变化再次触发刷新。

Fixes #16023. The Istio watcher could refresh before the first registered instance was added to the Naming publisher index, leaving it absent from the Istio view until a later change triggered another refresh.

修改内容 / Changes

  • 复用现有 ServiceChangedEvent:注册时先更新发布者索引,再发布事件;注销时先移除客户端,并保持在索引计算内发布事件的原有位置,避免在索引项移除后延迟发布删除事件造成的乱序。

  • Istio watcher 订阅该事件并保留回调串行化、推送防抖以及服务和实例元数据事件订阅。

  • 补充注册、注销、客户端释放、健康状态、元数据变化、并发注册/注销,以及事件队列满时注销与重新注册交错的回归覆盖,并更新中英文集成规范。

  • Reuse ServiceChangedEvent: update the publisher index before publishing a registration event. For deregistration, remove the client and keep publication inside the index computation, avoiding the ordering window caused by publishing the deletion after removing the map entry.

  • Subscribe the Istio watcher to that event while retaining serialized callbacks, push debounce, and the service and instance metadata event subscriptions.

  • Add regression coverage for registration, deregistration, client release, health and metadata changes, concurrent registration/deregistration, and interleaved removal/re-registration under the queue-full fallback; update the English and Chinese integration specs.

验证 / Verification

  • Naming 索引与存储、模糊订阅和推送、AI Agent 投影及 Istio 共 51 项定向回归测试通过;Naming 与 Istio 的 Spotless 和 Checkstyle 检查通过。

  • 交错回归测试使用实际填满的 Naming 事件队列:修复前复现 ADD_SERVICE → DELETE_SERVICE,修复后顺序为 DELETE_SERVICE → ADD_SERVICE,且新客户端仍在索引和模糊订阅结果中。

  • 使用当前代码构建独立 Nacos 服务并启用 Istio:注册一个健康实例后,首次 EDS 响应及随后 7 次响应均包含该端点。事件处理仍是异步的,与注册竞争的查询不保证立即看到该实例。

  • 通过独立服务对比基线与新方案的 Java SDK 客户端断开和注销不存在实例场景,分别检查启用与关闭 Istio 的行为。

  • 51 focused Naming index/storage, fuzzy-watch/push, AI Agent projection, and Istio regression tests passed; Spotless and Checkstyle passed for Naming and Istio.

  • With an actually full Naming event queue, the interleaving regression reproduced ADD_SERVICE → DELETE_SERVICE before the fix. After the fix it observed DELETE_SERVICE → ADD_SERVICE, with the new client still indexed and the service still matched by fuzzy watch.

  • Built a standalone server with Istio enabled. After one healthy registration, the first EDS response and seven subsequent responses each contained its endpoint. Event handling remains asynchronous; a query racing registration is not guaranteed to see the instance immediately.

  • Compared Java SDK client disconnect and missing-instance deregistration against the baseline using standalone servers with Istio enabled and disabled.

调查中观察到的既有行为 / Existing behavior observed

启用 Istio 时,NamingServiceJavaSdkITCase#testDuplicateRegisterAndMissingDeregisterAreIdempotent 在基线和新方案中均在注销另一个实例后的断言处失败。关闭 Istio 时,两者的该用例均通过,但探针观察到后续缓存刷新后已注册实例仍可能消失。该 PR 不修改现有的注销实例身份语义。

新方案会在客户端释放时触发 Istio 刷新;本地探针观察到启用 Istio 时,断开客户端的实例从查询结果消失的时间比基线更早。这是观察到的缓存刷新时机变化,不能据此保证所有时序下均相同。

With Istio enabled, NamingServiceJavaSdkITCase#testDuplicateRegisterAndMissingDeregisterAreIdempotent fails at the assertion after deregistering a different instance on both the baseline and this change. With Istio disabled, the test passes on both, but the probe observed that the registered instance can disappear after a later cache refresh. This PR does not change the existing deregistration identity semantics.

The new approach triggers an Istio refresh on client release. In the local probe with Istio enabled, the disconnected client's instance disappeared from query results earlier than on the baseline. This is an observed change in cache-refresh timing, not a guarantee for every execution schedule.

Refresh the Istio view after the publisher index is updated, so the first registered instance is not missed when event subscribers run out of order. Handle deregistration and client disconnect through the same notification.
Limit the new index notification to explicit instance registration and deregistration. Verify that client release still removes the index without triggering the additional notification.
@github-actions

Copy link
Copy Markdown

Thanks for your this PR. 🙏
Please check again for your PR changes whether contains any usage/api/configuration change such as Add new API , Add new configuration, Change default value of configuration.
If so, please add or update documents(markdown type) in docs/next/ for repository nacos-group/nacos-group.github.io


感谢您提交的PR。 🙏
请再次查看您的PR内容,确认是否包含任何使用方式/API/配置参数的变更,如:新增API、新增配置参数、修改默认配置等操作。
如果是,请确保在提交之前,在仓库nacos-group/nacos-group.github.io中的docs/next/目录下添加或更新文档(markdown格式)。

@codecov-commenter

Copy link
Copy Markdown

KomachiSion
KomachiSion previously approved these changes Nov 13, 2026
@KomachiSion

Copy link
Copy Markdown
Collaborator

Could we address this by publishing the existing ServiceChangedEvent after the publisher index update and having the Istio watcher subscribe to it? This seems to fit the existing event semantics and would avoid introducing ClientServiceIndexChangedEvent for the same service-view change.

I traced the publishing paths and the current consumers (Naming push, fuzzy watch, and AI agent projection). I couldn't find a consumer that needs the publisher index to remain in its previous state when handling ServiceChangedEvent. In particular, fuzzy watch uses the event's changedType to distinguish ADD_SERVICE from INSTANCE_CHANGED; it does not infer this by reading the old index.

The registration path used to update the index before publishing. The order changed alongside the fuzzy-watch classification logic, but computing the classification before the update does not require publishing the event before the update. With asynchronous dispatch, publishing first cannot reliably make the old index visible to a consumer anyway.

A possible adjustment would be:

  • For registration, determine changedType from the previous index state, update the index, then publish ServiceChangedEvent.
  • For deregistration, publish after computeIfPresent completes, so removal of the last publisher and the corresponding map entry has finished.
  • Have the Istio watcher consume ServiceChangedEvent, retaining the callback serialization and push debounce. Keep the necessary InfoChangeEvent subscriptions, especially ServiceInfoChangeEvent, since service-level metadata changes are not all covered by ServiceChangedEvent.

Reusing this event would also let health changes, instance metadata changes, and client release trigger Istio refreshes. Those paths deserve explicit regression coverage and corresponding clarification in the integration specs.

I tried this alternative in an isolated checkout based on 57847bf: 127 focused tests passed, including 8 additional cases covering immediate/asynchronous event handling, fuzzy-watch classification, client release, health/metadata updates, and burst registration/deregistration. Spotless and Checkstyle also passed. This validation used in-process event dispatch and resource generation; it did not cover a full standalone xDS/MCP stream.

Based on that, I'd suggest fixing the ordering at the existing event's source and reusing it here. Is there another consumer or compatibility constraint that would require preserving the current publish-before-update order?

@iyuan7

iyuan7 commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

@KomachiSion Thanks for the detailed investigation and suggestion. I’ve updated the PR to publish ServiceChangedEvent after the publisher index is updated and to have the Istio watcher subscribe to it.

I haven’t found a consumer that requires the previous publish-before-update order in the paths I checked. I’ve also added regression tests and verified the first EDS response with a standalone Nacos server.

One behavior I observed is that, with Istio enabled, client release can refresh the Naming cache sooner than on the baseline. I’ve documented this and the existing SDK test behavior in the PR description.

The new CI runs are waiting for maintainer approval. Could you please approve them and take another look when you have time? Thanks!

Comment on lines +156 to +158
if (changedType.get() != null) {
NotifyCenter.publishEvent(
new ServiceEvent.ServiceChangedEvent(service, changedType.get(), true));

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Preserve removal/addition event ordering

I need to correct the removal part of my earlier suggestion: moving this publication outside computeIfPresent opens an event-ordering window that I missed.

After the last publisher is removed, the following interleaving is possible:

  1. Thread A finishes computeIfPresent, removing the map entry, and pauses before publishing DELETE_SERVICE.
  2. Thread B registers a new publisher for the same service and publishes ADD_SERVICE.
  3. Thread A resumes and publishes the earlier DELETE_SERVICE.

The publisher index now contains the new client, but fuzzy watch processes ADD_SERVICE -> DELETE_SERVICE and removes the live service from its matched set.

The normal client-event queue is serial, but NamingEventPublisher.publish() falls back to synchronous handling in the publishing thread when its queue is full, so concurrent index updates are reachable. I reproduced this with the actual publisher's queue-full fallback, a latch immediately after the map computation completes, and the actual fuzzy-watch context handler. At da18b3b1, the new client remained indexed while the fuzzy-watch matched set became empty. With the same interleaving on the base implementation, the event order was DELETE_SERVICE -> ADD_SERVICE and the service remained matched. The current concurrency test separates the registration and deregistration phases, so it does not cover this case.

Could we retain the registration-order fix and the Istio subscription change, while keeping the previous removal publication placement for this PR? The previous removal path already called ids.remove(clientId) before notifying consumers, and moving this notification past the map-entry removal is not required to fix the first-registration issue. If publication after the computation is needed, we should also preserve the ordering of index mutations and their notifications for the same service across both add/remove paths, with an interleaved deregistration/re-registration regression test.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this. I’ve restored the original removal event publication position inside computeIfPresent, while keeping the registration-order fix and the Istio subscription change.

I also added a regression test using a full Naming event queue and an interleaved removal/re-registration. It reproduces the incorrect event order before the fix and verifies that the service remains matched by fuzzy watch after the fix. All 51 focused tests passed locally.

The new CI runs are waiting for maintainer approval. Could you please take another look and approve the runs when you have time? Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Istio adaptor: the first instance of a service can be missing from the Istio view

3 participants