Repository navigation
[VOTE] Add details to zone map index #7548
Closed
westonpace
started this conversation in
Vote
Replies: 4 comments 1 reply
|
+1 (binding) |
0 replies
|
+1 (binding) I like this change. One thing worth considering is changing the layout of the protos. It feels really weird to keep updating |
1 reply
|
+1 |
0 replies
|
This vote has passed, with three +1 votes. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Currently we don't store anything in the zone map index details. I'd like to store two pieces of information, the
rows_per_zone(existing parameter) and a new parameter,use_seeds.The
rows_per_zoneparameter controls how many rows we want to fit into each zone. It's currently stored in the index file's metadata and we use that when we do updates. However, in #7427 I am developing a feature that indexes some columns at write time (via "index seeds") and so I want to be able to know the rows per zone without opening up the index itself.The second parameter,
use_seedscontrols whether or not we will generate seeds at write time. These are described in more detail in #7427 but the short version is that, at write time, we calculate a zone map for the file and put it into the metadata. Then, at update time, we only need to read the seeds, and not the data itself. This is very useful for large columns (e.g. binary, vectors, etc.) While min/max is not typically useful on this data type we still often use a zone map to know which columns are null.More details are in the PR: #7427
This only format change is adding the new parameters to the details section. This is consistent with other index types (e.g. vector, fts) where we populate the details with the index parameters. These changes are backwards and forwards compatible. Older readers can safely ignore them. Newer readers can fallback to default behavior (don't use seeds) if the fields are not present.
All reactions