Repository navigation
DPL Analysis: retry remote reads and fail over to other replicas in the native AOD reader - #16051
Merged
Merged
Conversation
…he native AOD reader Once a grid file was open, the native reader gave up on the first failed read, and it sent a whole read plan at once, with plain reads of up to 64 MiB. On Hyperloop train 767444 at ORNL, which reads most files remotely, native subjobs failed on I/O errors from Poznan EOS: in 8 failure logs examined, 6 were "Operation expired" (a request taking longer than XRD_REQUESTTIMEOUT, 150 s in the ALICE environment) and 2 "bad address". ROOT does not retry these either; it only moves to another replica when opening fails. - FailoverByteSource reads the replicas of a file: a read failing with an IoError, at any point of the file, is retried on the same replica, reopened, up to Options::readRetries (2) times per replica with a doubling delay from Options::retryDelay (2 s), and then continues on the next replica of the same size. Format errors are not retried. openNativeAOD() now goes through ByteSource::openReplicas() instead of trying the replicas only at open. - XRootDByteSource asks the data server for its kXR_readv limits, as TNetXNGFile does, splits large ranges into plain reads of at most 8 MiB, and sends a plan in rounds of at most Options::remoteInFlightBytes (32 MiB), so that no request has to wait for a whole plan on a slow link. Tested: the AODNative unit tests, with new ones for FailoverByteSource (retry, failover, size mismatch, open failures, concurrent readers), and the AnalysisSupport and DataInputDirector tests. o2-test-framework-aod-native-vs-root on alien:///alice/sim/2025/LHC25b4a6/2/559362/AOD/003/AO2D.root (2 DFs) from a Mac: the CERN EOS replica does not exist on the server, both readers moved to the next replica, and the native result is byte-identical with ROOT's. Not yet measured on the Grid.
Member
Author
|
Tested on hyperloop. No regressions. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Once a grid file was open, the native reader gave up on the first failed read,
and it sent a whole read plan at once, with plain reads of up to 64 MiB. On
Hyperloop train 767444 at ORNL, which reads most files remotely, native
subjobs failed on I/O errors from Poznan EOS: in 8 failure logs examined, 6
were "Operation expired" (a request taking longer than XRD_REQUESTTIMEOUT,
150 s in the ALICE environment) and 2 "bad address". ROOT does not retry
these either; it only moves to another replica when opening fails.
IoError, at any point of the file, is retried on the same replica, reopened,
up to Options::readRetries (2) times per replica with a doubling delay from
Options::retryDelay (2 s), and then continues on the next replica of the
same size. Format errors are not retried. openNativeAOD() now goes through
ByteSource::openReplicas() instead of trying the replicas only at open.
TNetXNGFile does, splits large ranges into plain reads of at most 8 MiB, and
sends a plan in rounds of at most Options::remoteInFlightBytes (32 MiB), so
that no request has to wait for a whole plan on a slow link.
Tested: the AODNative unit tests, with new ones for FailoverByteSource (retry,
failover, size mismatch, open failures, concurrent readers), and the
AnalysisSupport and DataInputDirector tests. o2-test-framework-aod-native-vs-root
on alien:///alice/sim/2025/LHC25b4a6/2/559362/AOD/003/AO2D.root (2 DFs) from a
Mac: the CERN EOS replica does not exist on the server, both readers moved to
the next replica, and the native result is byte-identical with ROOT's. Not yet
measured on the Grid.