Summary
The Kubeflow Pipelines API server's GET /apis/v1alpha1/runs/{run_id}/nodes/{node_id}/log handler retrieves logs without authorizing the caller against the selected run. A caller who can reach the backend and knows a victim run ID and a suitable pod or node identifier can read live or archived logs using the API server's permissions.
For live logs, the vulnerable handler also accepts the supplied node ID as a pod name in the run's namespace without verifying that the pod belongs to the run. Depending on the API server's pod-log permissions, a known run ID can therefore permit reading the main container logs of an unrelated pod in that namespace. Logs may contain sensitive workload data or credentials.
The affected handler does not authenticate the request itself. An external authentication gate or network policy can restrict reachability; public ingress exposure and unauthenticated enumeration of run IDs are not assumed.
Remediation
Upgrade the API server to Kubeflow Pipelines 2.18.0. Versions through 2.17.3 are affected; the first affected release has not been established.
PR #13684 authorizes the caller with the readLog verb before retrieving live or archived logs. For live logs, it also verifies that the pod's pipeline/runid label matches the requested run. Custom Kubernetes roles must grant readLog on runs in the pipelines.kubeflow.org API group; the bundled viewer and editor roles include it. Shared read access alone does not grant log access.
Until upgrading, restrict access to the backend log endpoint. Focused regression tests cover authorization denial, HTTP identity propagation, pod/run mismatch, archive retrieval, and cancellation; no new deployed-cluster end-to-end test was performed for this assessment.
Credit
bugbunny.ai discovered and reported the issue. Nayyar (@naruto-lgtm) authored the fix in #13684.
@patchmyday, @thesecguy45, @VitoCrl, @Grg0rry, @arpitjain099, and @BrianWillows made later submissions concerning the run-log authorization gap. MUTEDCYBERSECURITY S.L. separately reported a synthetic two-tenant reproduction, including the unrelated-pod log path.
References
bugbunny-research Reporter
Summary
The Kubeflow Pipelines API server's
GET /apis/v1alpha1/runs/{run_id}/nodes/{node_id}/loghandler retrieves logs without authorizing the caller against the selected run. A caller who can reach the backend and knows a victim run ID and a suitable pod or node identifier can read live or archived logs using the API server's permissions.For live logs, the vulnerable handler also accepts the supplied node ID as a pod name in the run's namespace without verifying that the pod belongs to the run. Depending on the API server's pod-log permissions, a known run ID can therefore permit reading the
maincontainer logs of an unrelated pod in that namespace. Logs may contain sensitive workload data or credentials.The affected handler does not authenticate the request itself. An external authentication gate or network policy can restrict reachability; public ingress exposure and unauthenticated enumeration of run IDs are not assumed.
Remediation
Upgrade the API server to Kubeflow Pipelines 2.18.0. Versions through 2.17.3 are affected; the first affected release has not been established.
PR #13684 authorizes the caller with the
readLogverb before retrieving live or archived logs. For live logs, it also verifies that the pod'spipeline/runidlabel matches the requested run. Custom Kubernetes roles must grantreadLogonrunsin thepipelines.kubeflow.orgAPI group; the bundled viewer and editor roles include it. Shared read access alone does not grant log access.Until upgrading, restrict access to the backend log endpoint. Focused regression tests cover authorization denial, HTTP identity propagation, pod/run mismatch, archive retrieval, and cancellation; no new deployed-cluster end-to-end test was performed for this assessment.
Credit
bugbunny.ai discovered and reported the issue. Nayyar (@naruto-lgtm) authored the fix in #13684.
@patchmyday, @thesecguy45, @VitoCrl, @Grg0rry, @arpitjain099, and @BrianWillows made later submissions concerning the run-log authorization gap. MUTEDCYBERSECURITY S.L. separately reported a synthetic two-tenant reproduction, including the unrelated-pod log path.
References