Skip to content

fix(parameter): serialize Task instances using task_id in _DictParamEncoder - #3479

Open
reginaldalfret wants to merge 2 commits into
spotify:masterfrom
reginaldalfret:fix/issue-3247-task-in-list-parameter
Open

reginaldalfret wants to merge 2 commits into
spotify:masterfrom
reginaldalfret:fix/issue-3247-task-in-list-parameter

Conversation

@reginaldalfret

Copy link
Copy Markdown

Summary

Fixes #3247.

When passing \luigi.Task\ instances inside a \ListParameter\ or \DictParameter, serialization failed during parameter string conversion with:
\TypeError: Object of type is not JSON serializable\

Solution

In \luigi.parameter._DictParamEncoder.default, inspect if the object has a \ ask_id\ attribute (as all \Task\ instances do). If present, serialize it to \str(obj.task_id).

Tests

  • Added unit tests in \ est/parameter_test.py\ covering \ListParameter\ and \DictParameter\ populated with \Task\ instances.
  • Verified test suite passes: \pytest test/parameter_test.py.

Signed-off-by: reginaldalfret reginaldalfret@gmail.com

…ncoder

Signed-off-by: reginaldalfret <reginaldalfret@gmail.com>
…ne-length

Signed-off-by: reginaldalfret <reginaldalfret@gmail.com>

@Manishram-ai Manishram-ai left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried this out and ran into a few problems with the round trip.

  1. to_str_params() turns the tasks into task_id strings, but from_str_params() gives back a tuple of strings, not tasks. So a task that's rebuilt from its params gets strings from requires(). With a dynamic dependency, yield MainTask(subtasks=[SubTask("C"), ...]) fails in luigi.build with requires() must return Task objects but SubTask_C_87e101f821 is a <class 'str'>.

  2. Two different values now get the same task_id. MainTask(subtasks=[t]) and MainTask(subtasks=[t.task_id]) end up as the same task.

  3. hasattr(obj, "task_id") catches any object with that attribute, not only tasks. If this stays, isinstance(obj, luigi.Task) would be safer.

Parameter values are supposed to come back the same after serialize() and parse(), so I think this needs a maintainer to weigh in on #3247 first: whether tasks should be allowed in List/DictParameter at all, or whether the task should take plain parameters and build the subtasks in requires().

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

"Object of type ... is not JSON serializable" when ListParameters of Task instances

2 participants