Skip to content

DPL Analysis: retry remote reads and fail over to other replicas in the native AOD reader - #16051

Merged
ktf merged 1 commit into
AliceO2Group:devfrom
ktf:pr16051
Nov 5, 2026
Merged

ktf merged 1 commit into
AliceO2Group:devfrom
ktf:pr16051

Conversation

@ktf

@ktf ktf commented Nov 5, 2026 •

Copy link
Copy Markdown
Member

Once a grid file was open, the native reader gave up on the first failed read,
and it sent a whole read plan at once, with plain reads of up to 64 MiB. On
Hyperloop train 767444 at ORNL, which reads most files remotely, native
subjobs failed on I/O errors from Poznan EOS: in 8 failure logs examined, 6
were "Operation expired" (a request taking longer than XRD_REQUESTTIMEOUT,
150 s in the ALICE environment) and 2 "bad address". ROOT does not retry
these either; it only moves to another replica when opening fails.

  • FailoverByteSource reads the replicas of a file: a read failing with an
    IoError, at any point of the file, is retried on the same replica, reopened,
    up to Options::readRetries (2) times per replica with a doubling delay from
    Options::retryDelay (2 s), and then continues on the next replica of the
    same size. Format errors are not retried. openNativeAOD() now goes through
    ByteSource::openReplicas() instead of trying the replicas only at open.
  • XRootDByteSource asks the data server for its kXR_readv limits, as
    TNetXNGFile does, splits large ranges into plain reads of at most 8 MiB, and
    sends a plan in rounds of at most Options::remoteInFlightBytes (32 MiB), so
    that no request has to wait for a whole plan on a slow link.

Tested: the AODNative unit tests, with new ones for FailoverByteSource (retry,
failover, size mismatch, open failures, concurrent readers), and the
AnalysisSupport and DataInputDirector tests. o2-test-framework-aod-native-vs-root
on alien:///alice/sim/2025/LHC25b4a6/2/559362/AOD/003/AO2D.root (2 DFs) from a
Mac: the CERN EOS replica does not exist on the server, both readers moved to
the next replica, and the native result is byte-identical with ROOT's. Not yet
measured on the Grid.

@ktf
ktf requested a review from a team as a code owner November 5, 2026 13:56
…he native AOD reader

Once a grid file was open, the native reader gave up on the first failed read,
and it sent a whole read plan at once, with plain reads of up to 64 MiB. On
Hyperloop train 767444 at ORNL, which reads most files remotely, native
subjobs failed on I/O errors from Poznan EOS: in 8 failure logs examined, 6
were "Operation expired" (a request taking longer than XRD_REQUESTTIMEOUT,
150 s in the ALICE environment) and 2 "bad address". ROOT does not retry
these either; it only moves to another replica when opening fails.

- FailoverByteSource reads the replicas of a file: a read failing with an
  IoError, at any point of the file, is retried on the same replica, reopened,
  up to Options::readRetries (2) times per replica with a doubling delay from
  Options::retryDelay (2 s), and then continues on the next replica of the
  same size. Format errors are not retried. openNativeAOD() now goes through
  ByteSource::openReplicas() instead of trying the replicas only at open.
- XRootDByteSource asks the data server for its kXR_readv limits, as
  TNetXNGFile does, splits large ranges into plain reads of at most 8 MiB, and
  sends a plan in rounds of at most Options::remoteInFlightBytes (32 MiB), so
  that no request has to wait for a whole plan on a slow link.

Tested: the AODNative unit tests, with new ones for FailoverByteSource (retry,
failover, size mismatch, open failures, concurrent readers), and the
AnalysisSupport and DataInputDirector tests. o2-test-framework-aod-native-vs-root
on alien:///alice/sim/2025/LHC25b4a6/2/559362/AOD/003/AO2D.root (2 DFs) from a
Mac: the CERN EOS replica does not exist on the server, both readers moved to
the next replica, and the native result is byte-identical with ROOT's. Not yet
measured on the Grid.
@ktf

ktf commented Nov 5, 2026

Copy link
Copy Markdown
Member Author

Tested on hyperloop. No regressions.

@ktf
ktf merged commit f136373 into AliceO2Group:dev Nov 5, 2026
13 of 16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant