Skip to content

Fix DISTINCT hashing for set columns - #295

Closed
ryanduguid wants to merge 3 commits into
beancount:masterfrom
ryanduguid:fix/distinct-set-hashing
Closed

ryanduguid wants to merge 3 commits into
beancount:masterfrom
ryanduguid:fix/distinct-set-hashing

Conversation

@ryanduguid

@ryanduguid ryanduguid commented Oct 1, 2026 •

Copy link
Copy Markdown

Summary

Fixes #293. SELECT DISTINCT tags and SELECT DISTINCT links currently generate invalid Python when the set is the only column. With other columns, the hash also depends on set iteration order, so equal rows can survive deduplication.

Represent each set column as one frozenset hash component, preserve NULL, and always hash a tuple. Unique continues returning the first original row for each distinct value.

Related #242 changes Beancount column declarations to enable GROUP BY. This repair applies at the shared DISTINCT hashing boundary, including transaction columns and other sources that declare sets.

Evidence

  • Before the repair, all five new regression methods fail: the focused eight-test run has three failures and nine errors, including SyntaxError: cannot use starred expression here in all six tags/links query cases.
  • After the repair, all 298 tests pass on Linux with Python 3.8.20 / Beancount 2.3.6, Python 3.11.16 / Beancount 3.2.3 and Python 3.14.7 / Beancount 3.2.3.
  • Tests cover sole and multiple set columns, empty sets, NULL, mutable/frozen sets, forced opposite iteration orders, and tags/links queries over postings, transactions and entries. A separate Unique probe with five cases confirms retention of the first row and its cell identities.
  • Ruff 0.16 added PLR0917, which produced five baseline failures in the established renderer and command signatures. Five function-specific noqa comments restore the full repository check. AST comparisons confirm unchanged runtime code, and synthetic functions still trigger the rule in both affected files. All fork test, checks and documentation workflows pass at this PR head.
  • The existing parser and renderer coverage gates pass at 100%; hashable statement and branch coverage is 100%. Source distribution and wheel build, followed by wheel installation, pass on Python 3.11.16. After the CI repair, a fresh Linux Python 3.11 run again passes all 298 tests, both 100% coverage gates and both builds.

Aikido reports one inherited output-file path warning at shell.py:352, present in the upstream base. The five new lint comments do not change runtime code.

Merge danger

Door: reversible. Blast radius: DISTINCT keys.

A sole registered reducer now hashes its result inside a tuple. This changes its internal numeric hash but preserves the equality contract used for deduplication; existing reducer and dictionary controls pass.

Unverified

The remaining Python/Beancount combinations and documentation build passed in fork CI; upstream CI remains unverified. The Windows baseline has three temporary-file cleanup errors and one CSV newline failure; the focused regression tests pass on Windows, but a fully passing Windows suite is unverified.

@dnicolodi

Copy link
Copy Markdown
Collaborator

This is the same thing I wasted time reviewing in #294. The review comments in #294 didn't result in any interaction with someone understanding what they are doing, actually no interaction at all.

@ryanduguid what makes you think that it is valuable to contributed unverified and unfiltered LLM output to an open source project expecting the maintainers to review it? If I wanted LLM output I can very well use an LLM myself. I do not need a proxy to do that for me.

@ryanduguid the next interaction from your user account that looks like unfiltered LLM output will result in you being banned from the project.

@dnicolodi dnicolodi closed this Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

DISTINCT does not work with "tags"

2 participants