Skip to content

Make sync include resolution ordered and match the Go SDK - #460

Merged
maximelb merged 6 commits into
masterfrom
fix-include-order
Nov 14, 2026
Merged

maximelb merged 6 commits into
masterfrom
fix-include-order

Conversation

@maximelb

Copy link
Copy Markdown
Contributor

Problem

sync push resolved include: through a set() and merged each file with dict.update. With two included files defining the same record (hive key, output name, ...) the winner depended on set ordering and changed between runs. The Go SDK (same file format) is deterministic.

Change (limacharlie/sdk/configs.py)

  • Includes are processed in listed order; a glob expands to its matches in sorted order; a file reached twice (listed twice, or by several patterns, compared by absolute path) is processed only at its first occurrence. Non-glob include with no match still errors.
  • Merge now follows go-limacharlie loadEffectiveConfig (thisConfig = thisConfig.Merge(incConf) per include, recursively) and OrgConfig.Merge:
    • maps (outputs, org-value, installation_keys, integrity, artifact, rules, fps): by record name, include wins, record replaced whole
    • hives: per hive name, per record key, include wins (mergeHives)
    • exfil: performance / events / watches merged by key (mergeExfil)
    • resources: per-name union without duplicates (mergeResources); extensions: union without duplicates (mergeExtensions)
  • Docs: doc/sdk/configs.md (new Includes section), sync group help, CHANGELOG.

Intentional differences from Go

  • extensions: Go builds the union from a map, so its order is random; here it is first-seen order.
  • Duplicates: Go has no globs and would re-apply a file listed twice ([a, b, a] → a wins); here the repeat is dropped (b wins).
  • yara: Go's mergeYara returns the include's yara outright (the merged copy it builds is discarded), so an include without yara erases the parent's. Here rules and sources merge by name.
  • Previously rules/fps in included files were silently ignored and resources lists were replaced; both now merge as in Go.

Tests

New TestConfigsIncludeOrder: later-listed wins for hive record and output (including under 8 PYTHONHASHSEED values, 12 includes), included-overrides-includer (incl. nested), glob order, duplicates, error/no-match cases, and a Go-parity fixture across hives/org-value/installation_keys/exfil/resources/extensions/rules/fps/yara. They fail on the old code. Full pytest tests/unit/ tests/microbenchmarks/: 5583 passed.

🤖 Generated with Claude Code

Includes were collected in a set, so when two included files defined the same
record the winner changed from run to run. Process them in listed order (glob
matches sorted, duplicates dropped at first occurrence) and merge like the Go
SDK: later include beats earlier, included beats includer; resources and
extensions are unions; rules/fps from includes are no longer dropped.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@maximelb

Copy link
Copy Markdown
Contributor Author

Self-review (adversarial)

  • Precedence vs Go: re-read loadEffectiveConfig and OrgConfig.Merge. thisConfig.Merge(incConf) makes the include win at every nesting level, including an include's own sub-include beating the file that includes it. The nested test pins this. Each category merges like its Go merge* function.
  • Hives/exfil: records are replaced whole, not deep-merged, as in mergeHives / mergeExfil.
  • Dedup: keyed on abspath after the per-file chdir, so a.yaml and ./a.yaml collapse. Symlinks are not resolved, so a symlink to an already-included file is processed again. This is harmless because the merge is idempotent for that file, though its position can change.
  • Self-include: include: ["*.yaml"] in the same directory still recurses on itself, as before. I left it out of scope.
  • Return value: all_included is now ordered (include, then its sub-includes), where it was previously all matches and then the sub-includes. The only callers are tests.
  • New errors: a list-vs-map mismatch between files raises ConfigException where it used to raise AttributeError/TypeError.
  • Behaviour change worth knowing: included resources lists used to replace the earlier list and now union (as in Go). Included rules/fps used to be dropped silently and now merge. The PR description lists these and the other intentional differences from Go (extensions ordering, duplicate files, yara).
  • Tests: the new tests fail on the old code (11 failures with the old configs.py). The hash-seed test runs a subprocess per seed with 12 includes, so a set-based implementation is caught.
  • No issues open.

@maximelb

Copy link
Copy Markdown
Contributor Author

/lc-review

lcbill
lcbill previously approved these changes Nov 14, 2026

@limacharlie-event-notifications limacharlie-event-notifications Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Code Review Summary

No high or medium priority issues. I found 2 low priority items, posted inline.

General Feedback

  • test-time-budget: no check run was found for this commit. The master branch does not require the check (branch protection and rulesets list no required checks), so it does not block this review.
  • Pre-existing, not introduced here: YAML anchors and aliases can make two records share one dict or list object. A later include that updates one of them in place (cur[name].update(...) or cur[name].extend(...)) then changes both. If you want to fully isolate files, deep-copy sub_config values before you assign them.

Blast Radius

  • Configs.push_from_file and limacharlie sync push with include: now give a different effective config whenever included files overlap. Shared keys now follow listed order. resources lists are now unioned instead of replaced. rules and fps from included files are now sent to the push extension, where before they were dropped. A user whose layout relied on the old behaviour (for example, an include that replaced a resources list to unsubscribe) will see different push results. With --force, that can change what gets removed. The CHANGELOG covers this.
  • Files that mix a list and a map for the same category now raise ConfigException, where before they raised AttributeError or TypeError.
  • No other module uses CONFIG_ROOTS or _load_effective_config.

This review was generated automatically by Claude Code.

Comment thread limacharlie/sdk/configs.py Resolved
Comment thread tests/unit/test_sdk_configs.py Resolved
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@maximelb

Copy link
Copy Markdown
Contributor Author

Addressed both inline items: includes are now returned as absolute paths (dedupe and the returned list share one key; test added for same-named files in different directories). The hash-seed test is cut to 3 seeds and sets PYTHONPATH to the repo root. Also deep-copying included values so YAML aliases can't share objects across records. /lc-review

@maximelb

Copy link
Copy Markdown
Contributor Author

/lc-review

@limacharlie-event-notifications limacharlie-event-notifications Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Code Review Summary

No high or medium priority issues. I found 2 low priority items about the new YAML-alias fix and posted them inline.

General Feedback

  • Both earlier inline items are fixed. Includes are now returned as absolute paths, and dedupe and the returned list use the same key. The hash-seed test now runs 3 seeds with PYTHONPATH set to the repo root.
  • test-time-budget: no check run was found for this commit. master does not require the check (branch protection and rulesets list no required checks), so it does not block this review.

Blast Radius

  • Configs.push_from_file and limacharlie sync push with include: give a different effective config when included files overlap. Shared keys now follow listed order, resources lists are unioned instead of replaced, and rules/fps from includes are now sent instead of dropped. With --force, this can change what gets removed. The CHANGELOG covers it.
  • Files that mix a list and a map for the same category now raise ConfigException instead of AttributeError/TypeError.
  • _load_effective_config now returns absolute paths. Only tests read this value.

This review was generated automatically by Claude Code.

Comment thread limacharlie/sdk/configs.py Resolved
Comment thread tests/unit/test_sdk_configs.py Resolved

@limacharlie-event-notifications limacharlie-event-notifications Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Code Review Summary

Found 2 medium priority issues, both posted inline. They concern the YAML alias fix that was added in response to the last review.

General Feedback

  • test-time-budget: no check run exists for this commit. master lists no required checks in branch protection or in rulesets, so this does not block the review.
  • Both earlier inline threads are fixed. Includes are now returned as absolute paths, and the hash-seed test uses 3 seeds and sets PYTHONPATH.
  • Low: subprocess.run in test_result_independent_of_hash_seed has no timeout=, so a hang in the child would use up the whole pytest timeout. Add timeout=60.

Blast Radius

  • Configs.push_from_file / limacharlie sync push with include: now give a different effective config when included files overlap. Shared keys follow listed order. resources lists are unioned. Included rules/fps are now pushed. With --force, these changes can alter what gets removed.
  • A list-vs-map mismatch between files now raises ConfigException.
  • _load_effective_config now returns absolute, ordered paths. Only tests call it.

This review was generated automatically by Claude Code.

Comment thread limacharlie/sdk/configs.py Resolved
Comment thread tests/unit/test_sdk_configs.py Resolved
lcbill
lcbill previously approved these changes Nov 14, 2026
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@maximelb

Copy link
Copy Markdown
Contributor Author

Both fixed: deepcopy keeps aliases shared, so each loaded file (root and includes) is now rebuilt without shared objects (_unalias). The test aliases at the hive-name level and fails against the deepcopy version.

@maximelb

Copy link
Copy Markdown
Contributor Author

/lc-review

@limacharlie-event-notifications limacharlie-event-notifications Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Code Review Summary

No high or medium priority issues. I found 1 low priority item and posted it inline.

General Feedback

  • Both open threads are fixed. _unalias rebuilds every loaded file (the root and each include) without shared objects. test_yaml_aliases_do_not_leak_between_hives aliases at the hive-name level in an included file, so it fails if _unalias is removed.
  • Low (from the last review, still open): subprocess.run in test_result_independent_of_hash_seed (line 385) has no timeout=. If the child hangs, the test uses up the whole pytest timeout. Add timeout=60.
  • Low: _unalias recurses without a cycle guard. A self-referencing anchor (&a [*a]) now raises RecursionError instead of a ConfigException. This only affects the user's own file, so it is not a security issue.
  • test-time-budget: no check run exists for this commit. master lists no required checks in branch protection or rulesets, so this does not block the review.

Blast Radius

  • Configs.push_from_file and limacharlie sync push with include: give a different effective config when included files overlap. Shared keys now follow listed order, resources lists are unioned instead of replaced, and rules/fps from includes are now pushed instead of dropped. With --force, this can change what gets removed on the server. The CHANGELOG covers it.
  • A list-vs-map mismatch between files now raises ConfigException.
  • _load_effective_config now returns ordered absolute paths. Only tests read this value.

This review was generated automatically by Claude Code.

Comment thread tests/unit/test_sdk_configs.py Resolved
lcbill
lcbill previously approved these changes Nov 14, 2026
…ests

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@maximelb

Copy link
Copy Markdown
Contributor Author

The deepcopy threads were already superseded by _unalias (previous push). This push also adds a ConfigException for self-referencing anchors, a 60s subprocess timeout, and test coverage for root-file and resources aliases.

@maximelb

Copy link
Copy Markdown
Contributor Author

/lc-review

@limacharlie-event-notifications limacharlie-event-notifications Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Code Review Summary

No high or medium priority issues.

General Feedback

  • The last open thread is fixed. test_yaml_aliases_do_not_leak_between_hives now checks the root-file extensions/resources aliases. subprocess.run now has timeout=60, and a self-referencing anchor raises ConfigException, with a test.
  • Low: _unalias (configs.py line 53) builds a separate copy for every alias. A file with nested aliases (anchors that alias other anchors several levels deep) now grows exponentially in memory and CPU at load time. Before, it stayed as shared objects, and yaml.dump wrote them back as anchors. This only affects the user's own config file, so it is not a security issue. If configs can come from less trusted sources (for example, a CI job that pushes configs from pull requests), consider capping the total node count in _unalias and raising ConfigException above the cap.
  • Low, pre-existing: include cycles (a includes b, b includes a, or *.yaml matching its own file) still recurse until RecursionError. The seen set is per file, so it does not catch them. The author has said this is out of scope.
  • test-time-budget: no check run exists for this commit. master lists no required checks in branch protection or rulesets, so this does not block the review.

Blast Radius

  • Configs.push_from_file and limacharlie sync push with include: now build a different effective config when included files overlap:
    • Shared keys follow listed order.
    • resources lists are unioned instead of replaced.
    • Included rules/fps are now pushed instead of dropped.
    • Aliased YAML is expanded before push.
  • With --force, these changes can alter what gets removed. The CHANGELOG covers them.
  • A list-vs-map mismatch between files now raises ConfigException.
  • _load_effective_config now returns ordered absolute paths. Only tests read this value.

This review was generated automatically by Claude Code.

lcbill
lcbill previously approved these changes Nov 14, 2026
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@maximelb

Copy link
Copy Markdown
Contributor Author

Added a node cap (1M) in _unalias that raises ConfigException for alias-expansion bombs, with a test. The include-cycle case stays out of scope (pre-existing).

@maximelb

Copy link
Copy Markdown
Contributor Author

/lc-review

@limacharlie-event-notifications limacharlie-event-notifications Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Code Review Summary

No high or medium priority issues. I found 1 low priority item and posted it inline.

General Feedback

  • The node cap in _unalias works for the case the test covers. It counts only dicts and lists, not their items, so a file with wide leaf lists can still expand a lot before the cap fires (see inline). It only affects the user's own config file.
  • Low, pre-existing: include cycles (a includes b and b includes a, or *.yaml matching its own file) still recurse until RecursionError. The author has said this is out of scope.
  • test-time-budget: no check run exists for this commit. master lists no required checks in branch protection or rulesets, so this does not block the review.

Blast Radius

  • Configs.push_from_file and limacharlie sync push with include: now build a different effective config when included files overlap. Shared keys follow listed order, resources lists are unioned instead of replaced, included rules/fps are now pushed instead of dropped, and aliased YAML is expanded before the push. With --force, these changes can alter what gets removed. The CHANGELOG covers them.
  • Config files that expand past 1M containers, or that hold self-referencing anchors, now fail with ConfigException. A list-vs-map mismatch between files also raises ConfigException now.
  • _load_effective_config now returns ordered absolute paths. Only tests read this value.

This review was generated automatically by Claude Code.

Comment thread limacharlie/sdk/configs.py Resolved
lcbill
lcbill previously approved these changes Nov 14, 2026
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@maximelb

Copy link
Copy Markdown
Contributor Author

The node cap now counts each container's items too, so wide leaf lists are bounded.

@maximelb

Copy link
Copy Markdown
Contributor Author

/lc-review

@limacharlie-event-notifications limacharlie-event-notifications Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI Code Review Summary

No high or medium priority issues.

General Feedback

  • The last open item is fixed. _unalias now charges each container for its items (_budget[0] -= 1 + len(value)), so wide leaf lists count toward the 1M cap. test_alias_expansion_bomb_is_a_config_error lowers the cap to 10,000 and checks that the error is raised early.
  • Low, pre-existing: include cycles (a includes b and b includes a, or *.yaml matching its own file) still recurse until RecursionError. The author has said this is out of scope.
  • test-time-budget: no check run exists for this commit. master lists no required checks in branch protection or in rulesets, so this does not block the review.

Blast Radius

  • Configs.push_from_file and limacharlie sync push with include: now build a different effective config when included files overlap:
    • Shared keys follow listed order.
    • resources lists are unioned instead of replaced.
    • Included rules/fps are now pushed instead of dropped.
    • Aliased YAML is expanded before the push.
  • With --force, these changes can alter what gets removed. The CHANGELOG covers them.
  • Config files that expand past 1M nodes, or that hold self-referencing anchors, now fail with ConfigException. A list-vs-map mismatch between files also raises ConfigException.
  • _load_effective_config now returns ordered absolute paths. Only tests read this value.

This review was generated automatically by Claude Code.

@limacharlie-refractionpoint

limacharlie-refractionpoint Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

LimaCharlie Cloud Security — code scan

No new code findings were introduced by this pull request.

This check reports and never fails: no gating.fail_on is set on the code_scanning policy.

Scanned refractionPOINT/python-limacharlie 162c743…76aa12d — only findings new in the head commit are listed; anything already on the base branch is the repository's own finding set, on the Cloud Security Code page.

This comment is updated in place on every push to this pull request.

@maximelb
maximelb merged commit 840ace9 into master Nov 14, 2026
8 checks passed
@maximelb
maximelb deleted the fix-include-order branch November 14, 2026 03:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants