Repository navigation
Add Enhanced Ingestion Mode to genddl Tool - #92
ron-daniel1 wants to merge 15 commits into
Conversation
e574abb to
9fd2ef6
Compare
|
Fix provided in PR: |
yabinma
left a comment
There was a problem hiding this comment.
Thank you @ron-daniel1 for adding/updating the PR. LGTM.
ethanyzhang
left a comment
There was a problem hiding this comment.
A few correctness/robustness notes on the enhanced ingestion mode, posted inline. (Nice work — the legacy path looks cleanly preserved.)
| SourceFileFormat string `json:"source_file_format"` // "CSV" or "TEXTFILE" | ||
| SourceCatalog string `json:"source_catalog"` // Source catalog name (optional) | ||
| TargetCatalog string `json:"target_catalog"` // Target catalog name (optional) | ||
| Engine string `json:"engine"` // "presto" or "spark" |
There was a problem hiding this comment.
engine is accepted as "presto"/"spark" and defaulted to presto at L397, but nothing ever branches on it — grep for Engine, spark, USING, TBLPROPERTIES finds no references in any template or code path. A config with "engine": "spark" silently emits the same Presto CREATE SCHEMA ... WITH (...), USE catalog.schema, and Presto-style table DDL, contradicting the PR description's claim of Spark support (USING iceberg + TBLPROPERTIES). Suggest either rejecting non-presto engines in loadSchemas for now, or implementing the Spark template path in this PR.
There was a problem hiding this comment.
Thanks for pointing this out. We have removed the partial Spark branches and DDL templates from this PR and strictly restricted engine to "presto" in validateEnhancedIngestion() (defaulting when omitted and returning a clear validation error on unsupported engines like "spark"). Full Spark DDL generation will be introduced in a dedicated follow-up PR.
| {{- end }}) | ||
| WITH ( | ||
| external_location = '{{ $.S3SourceLocation }}/{{ .Name }}/', | ||
| {{- if eq $.SourceFileFormat "CSV" }} |
There was a problem hiding this comment.
source_file_format is unvalidated and case-sensitive. The source template treats anything != "CSV" as TEXTFILE, while insert generation (insert_table.sql.tmpl:24) treats anything != "TEXTFILE" as the CSV branch but only emits column expressions when the value is exactly "CSV". A lowercase "csv" therefore creates the source table as TEXTFILE and generates an INSERT with an empty SELECT column list (SELECT\nFROM ...) — invalid SQL, no error. Please validate source_file_format ∈ {CSV, TEXTFILE} in loadSchemas for enhanced mode.
There was a problem hiding this comment.
Added input validation in loadSchemas() via validateEnhancedIngestion() to validate all required fields (source_schema, target_schema, s3_source_location, s3_target_location) and ensure source_file_format is strictly "CSV" or "TEXTFILE". Comprehensive unit tests have been added in main_test.go.
| SET SESSION {{ $key }}='{{ $value }}'; | ||
| {{ end }} | ||
|
|
||
| {{- if .TargetSchema }} |
There was a problem hiding this comment.
This template selects enhanced vs legacy by checking .TargetSchema presence rather than .Mode. Combined with the lack of required-field validation in loadSchemas, an enhanced_ingestion config that omits target_schema runs the legacy insert branch (referencing .SchemaName, .UncompressedName, etc.) while main.go still routes through the enhanced source/target generators — producing mismatched, silently-wrong output. Suggest branching on .Mode and validating that source_schema/target_schema/s3_*_location are set up front.
There was a problem hiding this comment.
Updated insert_table.sql.tmpl to branch explicitly on eq .Mode "enhanced_ingestion" instead of checking .TargetSchema, ensuring a clean separation between the legacy and enhanced ingestion workflows.
| @@ -161,15 +183,22 @@ func generateSchemaFromDef(schema *Schema, defDir string, configDir string, outp | |||
| func generateCreateTable(schema *Schema, currDir string, outputDirs []string, step int) { | |||
| genSubSteps := !schema.Iceberg && schema.Partitioned | |||
There was a problem hiding this comment.
genSubSteps is computed before the enhanced branch and the trailing if genSubSteps { generateAwsS3Mv...; generateAnalyze...; generateAwsS3Cp... } still fires in enhanced mode. An enhanced config with iceberg:false, partitioned:true would emit stray legacy 6b-s3-mv/6c-call-analyze/6d-s3-cp artifacts alongside the source/target files. Untested combo (all enhanced examples use iceberg:true), but the path is reachable and unintended — worth guarding with !schema.isEnhancedIngestionMode().
There was a problem hiding this comment.
Guarded genSubSteps with !schema.isEnhancedIngestionMode() in generateCreateTable(). In enhanced ingestion mode, only 1a-create-source-... and 1b-create-target-... DDL files are emitted, preventing legacy S3 move/copy and analyze sub-steps from being generated.Guarded genSubSteps with !schema.isEnhancedIngestionMode() in generateCreateTable(). In enhanced ingestion mode, only 1a-create-source-... and 1b-create-target-... DDL files are emitted, preventing legacy S3 move/copy and analyze sub-steps from being generated.
Adds enhanced ingestion mode to cmd/genddl for generating TPC-DS data
ingestion SQL files.
Key Features:
Files Changed:
New Templates:
Modified: main.go
Schema struct additions:
New functions:
Modified functions:
(iceberg=true → Iceberg only) instead of all 4 variants
based on mode
Modified: insert_table.sql.tmpl
Testing:
✅ All 10 tests pass
✅ Backward compatible - legacy mode unchanged
✅ Generated examples match golden files
Usage:
Enhanced:
go run main.go genddl config_enhanced_ingestion.jsonLegacy:
go run main.go genddl config.json(unchanged)