Abstract

The HTCondor high throughput computing system is heavily used in the high energy physics (HEP) community as the batch system for several Worldwide LHC Computing Grid (WLCG) resources. Moreover, it is the backbone of GlidelnWMS, the pilot system used by the computing organization of the Compact Muon Solenoid (CMS) experiment. To prepare for LHC Run 2, we probed the scalability limits of new versions and configurations of HTCondor with a goal of reaching 200,000 simultaneous running jobs in a single internationally distributed dynamic pool.

In this paper, we first describe how we created an opportunistic distributed testbed capable of exercising runs with 200,000 simultaneous jobs without impacting production. This testbed methodology is appropriate not only for scale testing HTCondor, but potentially for many other services. In addition to the test conditions and the testbed topology, we include the suggested configuration options used to obtain the scaling results, and describe some of the changes to HTCondor inspired by our testing that enabled sustained operations at scales well beyond previous limits.

Details

Title
How much higher can HTCondor fly?
Author
Fajardo, E M 1 ; Dost, J M 1 ; Holzman, B 2 ; Tannenbaum, T 3 ; Letts, J 1 ; Tiradani, A 2 ; Bockelman, B 4 ; Frey, J 3 ; Mason, D 2 

 University of California San Diego, La Jolla, CA, USA 
 Fermi National Accelerator Laboratory, USA 
 University of Wisconsin, Madison, WI, USA 
 University of Nebraska, Lincoln, NE, USA 
Publication year
2015
Publication date
Dec 2015
Publisher
IOP Publishing
ISSN
17426588
e-ISSN
17426596
Source type
Scholarly Journal
Language of publication
English
ProQuest document ID
2576454190
Copyright
© 2015. This work is published under http://creativecommons.org/licenses/by/3.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.