Repository navigation
fix(parameter): serialize Task instances using task_id in _DictParamEncoder - #3479
reginaldalfret wants to merge 2 commits into
Conversation
…ncoder Signed-off-by: reginaldalfret <reginaldalfret@gmail.com>
…ne-length Signed-off-by: reginaldalfret <reginaldalfret@gmail.com>
Manishram-ai
left a comment
There was a problem hiding this comment.
I tried this out and ran into a few problems with the round trip.
-
to_str_params()turns the tasks into task_id strings, butfrom_str_params()gives back a tuple of strings, not tasks. So a task that's rebuilt from its params gets strings fromrequires(). With a dynamic dependency,yield MainTask(subtasks=[SubTask("C"), ...])fails inluigi.buildwithrequires() must return Task objects but SubTask_C_87e101f821 is a <class 'str'>. -
Two different values now get the same task_id.
MainTask(subtasks=[t])andMainTask(subtasks=[t.task_id])end up as the same task. -
hasattr(obj, "task_id")catches any object with that attribute, not only tasks. If this stays,isinstance(obj, luigi.Task)would be safer.
Parameter values are supposed to come back the same after serialize() and parse(), so I think this needs a maintainer to weigh in on #3247 first: whether tasks should be allowed in List/DictParameter at all, or whether the task should take plain parameters and build the subtasks in requires().
Summary
Fixes #3247.
When passing \luigi.Task\ instances inside a \ListParameter\ or \DictParameter, serialization failed during parameter string conversion with:
\TypeError: Object of type is not JSON serializable\
Solution
In \luigi.parameter._DictParamEncoder.default, inspect if the object has a \ ask_id\ attribute (as all \Task\ instances do). If present, serialize it to \str(obj.task_id).
Tests
Signed-off-by: reginaldalfret reginaldalfret@gmail.com