Repository navigation
Make sync include resolution ordered and match the Go SDK - #460
Conversation
Includes were collected in a set, so when two included files defined the same record the winner changed from run to run. Process them in listed order (glob matches sorted, duplicates dropped at first occurrence) and merge like the Go SDK: later include beats earlier, included beats includer; resources and extensions are unions; rules/fps from includes are no longer dropped. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
Self-review (adversarial)
|
|
/lc-review |
There was a problem hiding this comment.
AI Code Review Summary
No high or medium priority issues. I found 2 low priority items, posted inline.
General Feedback
test-time-budget: no check run was found for this commit. Themasterbranch does not require the check (branch protection and rulesets list no required checks), so it does not block this review.- Pre-existing, not introduced here: YAML anchors and aliases can make two records share one dict or list object. A later include that updates one of them in place (
cur[name].update(...)orcur[name].extend(...)) then changes both. If you want to fully isolate files, deep-copysub_configvalues before you assign them.
Blast Radius
Configs.push_from_fileandlimacharlie sync pushwithinclude:now give a different effective config whenever included files overlap. Shared keys now follow listed order.resourceslists are now unioned instead of replaced.rulesandfpsfrom included files are now sent to the push extension, where before they were dropped. A user whose layout relied on the old behaviour (for example, an include that replaced aresourceslist to unsubscribe) will see different push results. With--force, that can change what gets removed. The CHANGELOG covers this.- Files that mix a list and a map for the same category now raise
ConfigException, where before they raisedAttributeErrororTypeError. - No other module uses
CONFIG_ROOTSor_load_effective_config.
This review was generated automatically by Claude Code.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
Addressed both inline items: includes are now returned as absolute paths (dedupe and the returned list share one key; test added for same-named files in different directories). The hash-seed test is cut to 3 seeds and sets PYTHONPATH to the repo root. Also deep-copying included values so YAML aliases can't share objects across records. /lc-review |
|
/lc-review |
There was a problem hiding this comment.
AI Code Review Summary
No high or medium priority issues. I found 2 low priority items about the new YAML-alias fix and posted them inline.
General Feedback
- Both earlier inline items are fixed. Includes are now returned as absolute paths, and dedupe and the returned list use the same key. The hash-seed test now runs 3 seeds with
PYTHONPATHset to the repo root. test-time-budget: no check run was found for this commit.masterdoes not require the check (branch protection and rulesets list no required checks), so it does not block this review.
Blast Radius
Configs.push_from_fileandlimacharlie sync pushwithinclude:give a different effective config when included files overlap. Shared keys now follow listed order,resourceslists are unioned instead of replaced, andrules/fpsfrom includes are now sent instead of dropped. With--force, this can change what gets removed. The CHANGELOG covers it.- Files that mix a list and a map for the same category now raise
ConfigExceptioninstead ofAttributeError/TypeError. _load_effective_confignow returns absolute paths. Only tests read this value.
This review was generated automatically by Claude Code.
There was a problem hiding this comment.
AI Code Review Summary
Found 2 medium priority issues, both posted inline. They concern the YAML alias fix that was added in response to the last review.
General Feedback
test-time-budget: no check run exists for this commit.masterlists no required checks in branch protection or in rulesets, so this does not block the review.- Both earlier inline threads are fixed. Includes are now returned as absolute paths, and the hash-seed test uses 3 seeds and sets
PYTHONPATH. - Low:
subprocess.runintest_result_independent_of_hash_seedhas notimeout=, so a hang in the child would use up the whole pytest timeout. Addtimeout=60.
Blast Radius
Configs.push_from_file/limacharlie sync pushwithinclude:now give a different effective config when included files overlap. Shared keys follow listed order.resourceslists are unioned. Includedrules/fpsare now pushed. With--force, these changes can alter what gets removed.- A list-vs-map mismatch between files now raises
ConfigException. _load_effective_confignow returns absolute, ordered paths. Only tests call it.
This review was generated automatically by Claude Code.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
Both fixed: deepcopy keeps aliases shared, so each loaded file (root and includes) is now rebuilt without shared objects ( |
|
/lc-review |
There was a problem hiding this comment.
AI Code Review Summary
No high or medium priority issues. I found 1 low priority item and posted it inline.
General Feedback
- Both open threads are fixed.
_unaliasrebuilds every loaded file (the root and each include) without shared objects.test_yaml_aliases_do_not_leak_between_hivesaliases at the hive-name level in an included file, so it fails if_unaliasis removed. - Low (from the last review, still open):
subprocess.runintest_result_independent_of_hash_seed(line 385) has notimeout=. If the child hangs, the test uses up the whole pytest timeout. Addtimeout=60. - Low:
_unaliasrecurses without a cycle guard. A self-referencing anchor (&a [*a]) now raisesRecursionErrorinstead of aConfigException. This only affects the user's own file, so it is not a security issue. test-time-budget: no check run exists for this commit.masterlists no required checks in branch protection or rulesets, so this does not block the review.
Blast Radius
Configs.push_from_fileandlimacharlie sync pushwithinclude:give a different effective config when included files overlap. Shared keys now follow listed order,resourceslists are unioned instead of replaced, andrules/fpsfrom includes are now pushed instead of dropped. With--force, this can change what gets removed on the server. The CHANGELOG covers it.- A list-vs-map mismatch between files now raises
ConfigException. _load_effective_confignow returns ordered absolute paths. Only tests read this value.
This review was generated automatically by Claude Code.
…ests Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
The deepcopy threads were already superseded by |
|
/lc-review |
There was a problem hiding this comment.
AI Code Review Summary
No high or medium priority issues.
General Feedback
- The last open thread is fixed.
test_yaml_aliases_do_not_leak_between_hivesnow checks the root-fileextensions/resourcesaliases.subprocess.runnow hastimeout=60, and a self-referencing anchor raisesConfigException, with a test. - Low:
_unalias(configs.pyline 53) builds a separate copy for every alias. A file with nested aliases (anchors that alias other anchors several levels deep) now grows exponentially in memory and CPU at load time. Before, it stayed as shared objects, andyaml.dumpwrote them back as anchors. This only affects the user's own config file, so it is not a security issue. If configs can come from less trusted sources (for example, a CI job that pushes configs from pull requests), consider capping the total node count in_unaliasand raisingConfigExceptionabove the cap. - Low, pre-existing: include cycles (
aincludesb,bincludesa, or*.yamlmatching its own file) still recurse untilRecursionError. Theseenset is per file, so it does not catch them. The author has said this is out of scope. test-time-budget: no check run exists for this commit.masterlists no required checks in branch protection or rulesets, so this does not block the review.
Blast Radius
Configs.push_from_fileandlimacharlie sync pushwithinclude:now build a different effective config when included files overlap:- Shared keys follow listed order.
resourceslists are unioned instead of replaced.- Included
rules/fpsare now pushed instead of dropped. - Aliased YAML is expanded before push.
- With
--force, these changes can alter what gets removed. The CHANGELOG covers them. - A list-vs-map mismatch between files now raises
ConfigException. _load_effective_confignow returns ordered absolute paths. Only tests read this value.
This review was generated automatically by Claude Code.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
Added a node cap (1M) in |
|
/lc-review |
There was a problem hiding this comment.
AI Code Review Summary
No high or medium priority issues. I found 1 low priority item and posted it inline.
General Feedback
- The node cap in
_unaliasworks for the case the test covers. It counts only dicts and lists, not their items, so a file with wide leaf lists can still expand a lot before the cap fires (see inline). It only affects the user's own config file. - Low, pre-existing: include cycles (
aincludesbandbincludesa, or*.yamlmatching its own file) still recurse untilRecursionError. The author has said this is out of scope. test-time-budget: no check run exists for this commit.masterlists no required checks in branch protection or rulesets, so this does not block the review.
Blast Radius
Configs.push_from_fileandlimacharlie sync pushwithinclude:now build a different effective config when included files overlap. Shared keys follow listed order,resourceslists are unioned instead of replaced, includedrules/fpsare now pushed instead of dropped, and aliased YAML is expanded before the push. With--force, these changes can alter what gets removed. The CHANGELOG covers them.- Config files that expand past 1M containers, or that hold self-referencing anchors, now fail with
ConfigException. A list-vs-map mismatch between files also raisesConfigExceptionnow. _load_effective_confignow returns ordered absolute paths. Only tests read this value.
This review was generated automatically by Claude Code.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
The node cap now counts each container's items too, so wide leaf lists are bounded. |
|
/lc-review |
There was a problem hiding this comment.
AI Code Review Summary
No high or medium priority issues.
General Feedback
- The last open item is fixed.
_unaliasnow charges each container for its items (_budget[0] -= 1 + len(value)), so wide leaf lists count toward the 1M cap.test_alias_expansion_bomb_is_a_config_errorlowers the cap to 10,000 and checks that the error is raised early. - Low, pre-existing: include cycles (
aincludesbandbincludesa, or*.yamlmatching its own file) still recurse untilRecursionError. The author has said this is out of scope. test-time-budget: no check run exists for this commit.masterlists no required checks in branch protection or in rulesets, so this does not block the review.
Blast Radius
Configs.push_from_fileandlimacharlie sync pushwithinclude:now build a different effective config when included files overlap:- Shared keys follow listed order.
resourceslists are unioned instead of replaced.- Included
rules/fpsare now pushed instead of dropped. - Aliased YAML is expanded before the push.
- With
--force, these changes can alter what gets removed. The CHANGELOG covers them. - Config files that expand past 1M nodes, or that hold self-referencing anchors, now fail with
ConfigException. A list-vs-map mismatch between files also raisesConfigException. _load_effective_confignow returns ordered absolute paths. Only tests read this value.
This review was generated automatically by Claude Code.
LimaCharlie Cloud Security — code scanNo new code findings were introduced by this pull request. This check reports and never fails: no Scanned This comment is updated in place on every push to this pull request. |
Problem
sync pushresolvedinclude:through aset()and merged each file withdict.update. With two included files defining the same record (hive key, output name, ...) the winner depended on set ordering and changed between runs. The Go SDK (same file format) is deterministic.Change (
limacharlie/sdk/configs.py)loadEffectiveConfig(thisConfig = thisConfig.Merge(incConf)per include, recursively) andOrgConfig.Merge:mergeHives)mergeExfil)mergeResources); extensions: union without duplicates (mergeExtensions)doc/sdk/configs.md(new Includes section),syncgroup help, CHANGELOG.Intentional differences from Go
extensions: Go builds the union from a map, so its order is random; here it is first-seen order.[a, b, a]→ a wins); here the repeat is dropped (b wins).yara: Go'smergeYarareturns the include's yara outright (the merged copy it builds is discarded), so an include withoutyaraerases the parent's. Here rules and sources merge by name.rules/fpsin included files were silently ignored andresourceslists were replaced; both now merge as in Go.Tests
New
TestConfigsIncludeOrder: later-listed wins for hive record and output (including under 8PYTHONHASHSEEDvalues, 12 includes), included-overrides-includer (incl. nested), glob order, duplicates, error/no-match cases, and a Go-parity fixture across hives/org-value/installation_keys/exfil/resources/extensions/rules/fps/yara. They fail on the old code. Fullpytest tests/unit/ tests/microbenchmarks/: 5583 passed.🤖 Generated with Claude Code