Condor
From GridInfo
Contents |
Executive summary
The Condor software system allows a series of computing tasks to be sent to a group of computers. This allows many computing tasks to be carried out simultaneously using many processors, instead of one by one on a single processor. As Condor pools can consist of hundreds or thousands of computers, the number of calculations done in a particular time can increase greatly. This is referred to as high-throughput computing.
The Condor software allows what is called cycle-stealing or scavenging. The software is installed on normal desktop computers and when the computer is not in use by its owner it is available to Condor to run calculations. This does not impinge on use by the normal user, who regains full control of their machine as soon as they start to use it again. Incomplete calculations are sent by Condor to another computer in the pool.
The two main advantages of using Condor are more efficient use of machines as idle computers are put to work, and greater availabilty of resources as larger numbers of computers are made available to individual researchers.
Introduction
The Condor Project has performed research in distributed high-throughput computing since 1988, and maintains the Condor High Throughput Computing resource and job management software originally designed to harness idle CPU cycles on heterogeneous networks of computers to create high-throughput computing resources. Since its conception, Condor has gained a level of maturity and sophistication that make it a widely popular system, used by hundreds of government, academic, and commercial organizations worldwide.
The Condor system, which can best be described as a distributed job scheduler and resource management system, provides means for users to submit jobs in the form of executable programs, and manages the execution of these jobs on suitably selected resources in a pool, based on job requirements and community, resource owner and workload distribution policies. In this manner Condor differs from traditional batch scheduling systems for compute intensive jobs, as it does not require the underlying resources to be dedicated: Condor will select machines on which to run jobs which are not currently in use according to an arbitrary policy dictated by the owner of the resource. This ensures that underused resources in a pool are properly harnessed and may vacate and migrate jobs when the machine is required, effectively protecting resource owner policies.
Whilst the range of services provided by Condor is wide and varied, we can simplify Condor’s rich functionality into the following three categories:
- Job scheduling: A condor scheduler is responsible for managing job execution requests - consisting of input files, execution binaries, environment requirements, and other parameters - from multiple users and maintaining a persistent queue of jobs. Users may assign different priorities to their jobs, and specify workflow dependencies between jobs.
- Resource management services: A central manager is responsible for collecting resource characteristic and usage information from machines in a Condor pool. It is based on this collected information, and on user priorities, that job requests can be matched to suitable resources for execution.
- Job execution management: Condor manages remote execution of jobs, providing file transfer mechanisms enabling required files to be staged on the remote machine, checkpointing mechanisms enabling job state to be saved and subsequently resumed, and migration of jobs from one machine to another. Furthermore, Condor enables system calls performed by the application running remotely to be performed on the client machine, providing a degree of transparency with regard to the execution of the job.
The service-based decomposition of the Condor architecture (covered below) has enabled many of these services to be adapted for different uses and purposes, represented by different Universes, or Condor run-time environments. Apart from the standard universe - providing the entire set of Condor functionality, such as checkpointing and migration, to programs re-linked with a special Condor library – and the Vanilla Universe – providing less features but suited to a wider range of programs, Condor supports specialized universes for Java, PVM (parallel applications), and MPI applications, as well as interaction with Grid resources managed by an array of grid middleware technology including the Globus Toolkit 2.x, Globus Toolkit 3.x, Unicore, and others. Many of these features will be explored in more detail in later sections.
ClassAds
Condor Class Advertisements (ClassAds) are central to Condor’s matchmaking capabilities. A ClassAd is a set of uniquely named expressions, using a schema-free semi-structured model. ClassAds enable a mapping between attributes and expressions to be specified and evaluated with respect to another ClassAd. A ClassAd in Condor expresses a job’s characteristics, as well as requirements (e.g. memory, OS, etc.) and preferences upon matching a ClassAd representing a computing resource. Symmetrically, ClassAds representing computing resources specify attributes about the resource, and any requirements or preferences upon a job it is willing to service. The ClassAd framework provides a very effective means for job ClassAds to be matched to suitable resource ClassAds in a distributed policy environment.
Architecture
Condor’s key activities - job-resource allocation, job startup and execution, and metadata collection and display – are kept separate, allowing compartmentalization of Condor into clearly defined components. This daemon-based decomposition of the Condor architecture is distributed amongst submission site, central manager and execution site. Whilst there can only be one central manager, there can be any number of submit and execute machines in a given Condor pool, and the same machine may hold multiple roles:
- Central Manager: A single central manager exists for every Condor pool. It is responsible for collecting information about the pool and its machines, and the matchmaking of job requirements to suitable machines based on community policies in the form of job pre-emption rules and user priorities. Matchmaking is a symmetric process; both job and machine requirements and ranks are considered when these are paired up. Two daemons reside here, the negotiator and the collector. The collector is, as its name indicates, responsible for collecting ClassAds from all other daemons in the system (such as schedulers, or resource-side components), enabling us to determine the state of all machines in a Condor pool. It is based on this information that the negotiator will, during a regular negotiation cycle, match schedulers with waiting resource requests to available resources in priority order. The negotiator also performs accounting for the pool and maintains and adapts user priorities according to current resource usage. A previously established match may be revoked if a higher priority user/job requires a particular resource.
- Execute Machine: The execute machine runs jobs on behalf of clients. It advertises its capabilities and usage information - as well as requirements and preferences upon a match - with the central manager. The daemon specific to the execute machine is called the Startd, and represents a resource on the condor pool. When a successful match has been established by the negotiator, and the submit machine specified by the match has established contact with the Startd, it will spawn a starter daemon responsible for managing the local execution of the job. The starter spawns the actual job on the execute machine, setting up the execution environment and monitoring the job’s progress once it starts. The starter communicates with the shadow daemon running on the submission machine (described below), returns status information, cleans up the execute machine upon job completion and exits. Resource owners can choose to cleanly vacate running jobs in order to regain control over the machine.
- Submit Machine: This machine allows users to submit jobs to a local virtual ‘queue’, and this function is controlled by the schedd daemon. In order to service queued jobs, it will advertise to the collector the number of idle jobs it has in the queue so it can be contacted by the negotiator of the central manager during the negotiation cycle. It is then that the scheduler will submit requirements for its jobs in priority order so that it can be allocated suitable resources. Once a match has been established, the schedd will spawn a shadow daemon responsible for managing the remote execution of the job. The shadow will interact with the starter daemon on the remote machine and monitor its execution. It will manage the checkpointing of the job, perform system calls on the local machine on behalf of the remote application, and reschedule the job in case of failure. The scheduler may attempt to re-use a matched resource for another job or hold it as a spare, until the match is relinquished.
Common to all types of machines, is the master daemon. This daemon will run on every machine in a Condor pool. It is responsible for ensuring that the daemons for the corresponding type of machine are running and restarts them in case of failure. It is also through this daemon that administrative requests such as start/stop or reconfigure daemons can be serviced remotely.
It is important to mention the clear distinction between matchmaking and claiming. The role of the central manager, or more specifically the negotiator is a suggestive role. Whilst the negotiator may recommend a match between client and server, it is left for the resource itself to approve the match, based on owner configurations. Claiming a resource is hence kept distinct from matchmaking. When a client and resource have been informed of a successful match, the client will then proceed to claim the resource, which after mutual authentication based on information provided by the negotiator, can be authorized or rejected. This distinction enables Condor to cope with the fact that the matchmaking process may have been performed on potentially outdated information, and protect resource owner policies expressed independently from the community policies specified by a pool administrator. Furthermore, the claiming protocol may vary according to the nature of the resource and security protocols.
We should also consider the role of these components when dealing with parallel applications and the master worker paradigm. Essentially this paradigm allows a central master process to coordinate the execution of individual tasks by worker processes on remote nodes, and Condor can act as the resource management platform on which such applications can be deployed. Whenever the master requires additional nodes with specific characteristics on which to run tasks, it will request these from Condor’s resource management framework, which will spawn a new worker on a node to whom the master can then submit tasks. The Condor shadow in this case acts as the resource management service for this type of application, through which masters can request resources. In turn the shadow will proceed to submit these requests to the local scheduler. This master worker paradigm is one that should be noted, as it will drive many of the considerations as to how Condor resource management services can be exploited in the Grid.
Condor and the Grid
The capability of managing jobs in an inter-domain setting, across independently managed resources, has been explored and introduced in Condor via the following mechanisms:
- (direct) Flocking: Flocking allows jobs idling in the queue of a given Condor pool to be run on spare resources on a different Condor pool. If a job cannot be serviced by resources of the local pool, the scheduler can be configured to submit the job execution request to the central manager of another.
- Condor-G: Condor-G enables the scheduler daemon to submit jobs to resources managed by Globus and other middleware packages. The Globus Resource and Allocation Management (GRAM) architecture essentially defines a 2-tier scheduling architecture: meta-schedulers can submit their jobs to a wide range of individual local schedulers through a standard and secure interface. Condor-G enables the scheduling component of Condor to be used to manage the submission of jobs to resources on the Grid presenting the GRAM interface, by providing job management services on the client side.
- Condor glide-in: Condor glide-in is a mechanism by which temporary resources managed by Globus can be added to a Condor pool. Condor glide-in effectively exploits the daemon-based decomposition of services in Condor by enabling Condor execution daemons to be submitted as jobs to local schedulers through the Globus GRAM interface. In effect, this mechanism enables users to build a personal Condor pool on resources independently allocated by different underlying scheduling systems.
These Condor execution daemons will report back to a Condor collector when run on allocated resources, and Condor-G can then be used to submit jobs directly to these daemons. This strategy allows a distinction to be made between the allocation process of local schedulers and the actual job execution, and enable higher level schedulers to bind a job to a resource only when its availability is determined. Further more, the entire set of functionality of Condor can now be exploited through this strategy across multiple independently managed resources. In effect, Condor glide-in results in an end-to-end solution: in order to present semantic guarantees to the end user, such as at-most-once, at-least-once, and once-and-only once, a distributed job execution model must provide end-to-end management.
Condor Pools in the UK
The UCL Condor Pool has been running since 2003 and currently operates at a maximum of 940 nodes, with plans for expansion in the future. Each node is a WIN2K Intel Pentium3, with 1GHz CPU speed and between 256Mb and 512Mb RAM. 750Mb of hard disk space is available on each of these nodes. These machines are part of the UCL Information systems Cluster WTS available to students throughout UCL. Between October 2003 and July 2005, UCL Condor has managed, processed and returned near to 2,000,000 hours of computational chemistry, physics and geological programs for users all over the UK academic community, a total roughly equivalent to 220 years of CPU time.
Cambridge, RI, etc.. also operate large pools. (to be completed)
