Abstract

In statistically hungry science domains, data deluges can be both a blessing and a curse. They allow the narrowing of statistical errors from known measurements, and open the door to new scientific opportunities as research programs mature. They are also a testament to the efficiency of experimental operations. However, growing data samples may need to be processed with little or no opportunity for huge increases in computing capacity. A standard strategy has thus been to share resources across multiple experiments at a given facility. Another has been to use middleware that “glues” resources across the world so they are able to locally run the experimental software stack (either natively or virtually). We describe a framework STAR has successfully used to reconstruct a ~400 TB dataset consisting of over 100,000 jobs submitted to a remote site in Korea from STAR's Tier 0 facility at the Brookhaven National Laboratory. The framework automates the full workflow, taking raw data files from tape and writing Physics-ready output back to tape without operator or remote site intervention. Through hardening we have demonstrated 97(±2)% efficiency, over a period of 7 months of operation. The high efficiency is attributed to finite state checking with retries to encourage resilience in the system over capricious and fallible infrastructure.

Details

Title
Automated Finite State Workflow for Distributed Data Production
Author
Hajdu, L 1 ; Didenko, L 1 ; Lauret, J 1 ; Amol, J 2 ; Betts, W 1 ; Jang, H J 3 ; Noh, S Y 2 

 Software and Computing Group, RHIC/ STAR experiment, Brookhaven National Lab, PO Box 5000, Upton, NY 11973-5000, USA 
 Korea Institute of Science and Technology Information, 245 Daehangno, Yuseong, Daejeon 305-806, KOREA; Korea University of Science and Technology, Yuseong, Dajeon 305-350, KOREA 
 Korea Institute of Science and Technology Information, 245 Daehangno, Yuseong, Daejeon 305-806, KOREA 
Publication year
2016
Publication date
Oct 2016
Publisher
IOP Publishing
ISSN
17426588
e-ISSN
17426596
Source type
Scholarly Journal
Language of publication
English
ProQuest document ID
2575309735
Copyright
© 2016. This work is published under http://creativecommons.org/licenses/by/3.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.