Using GridSAM at NIEeS

From GridInfo

Jump to: navigation, search

GridSAM is an open-source job submission and monitoring web service. The aim of GridSAM is to provide a Web Service for submitting and monitoring jobs managed by a variety of Distributed Resource Managers (DRM). The modular design allows third-parties to provide submission and file-transfer plug-ins to GridSAM. Moreover the job management API used by the GridSAM web service can be embedded into a grid application that requires job submission and monitoring capabilities. The Job Submission Description Language (JSDL) has been developed in the Global Grid Forum (GGF).

NIEeS has deployed GridSAM which connects to the NIEeS condor pool. The topography can be seen in the following Figure. Description

At this moment, both the GridSAM server and client are installed at agorilla.niees.group.cam.ac.uk and talk to the Condor pool central manager, which is cete.niees.group.cam.ac.uk. One can login to agorilla and use the Open Middleware Infrastructure Institute (OMII) or GridSAM client tools to submit jobs to the NIEeS condor pool. However, you are welcome to install your own GridSAM client on your own machine if it is insde the NIEeS network. The NIEeS firewall blocks OMII services at the moment. Currently, GridSAM is using UK-eScience Certificate Authority (CA) and will support the NIEeS CA as well. One can apply for a NIEeS certificate and submit jobs using the GridSAM client tools.

When you have obtained a NIEeS account and user certificate, you can ssh into the GridSAM client, which is agorilla (128.232.232.41) at this moment. You can create your GridSAM job submission description language files (JSDL). The following are some examples:

In this demo, all JSDL files will be stored and executed at this location: /home/gtniees/gridsam-jobs/examples.

[edit]

Example 1: submission with no staging

In this case, the user is authorised to submit jobs to the GridSAM instance. The following script assumes the backend system is UNIX-based with the POSIX /bin/echo command available. The JSDL file is called "hello-NIEeS.jsdl". 

 <JobDefinition xmlns="http://schemas.ggf.org/jsdl/2005/06/jsdl">
     <JobDescription>
         <Application>
             <POSIXApplication xmlns="http://schemas.ggf.org/jsdl/2005/06/jsdl-posix">
                 <Executable>/bin/echo</Executable>
                 <Argument>Hello NIEeS!!</Argument>
             </POSIXApplication>
         </Application>
     </JobDescription>
 </JobDefinition>

Then, you can run the following command: 

 [gtniees@agorilla examples]$ gridsam-submit -s "http://agorilla.niees.group.cam.ac.uk:18080/gridsam/services/gridsam?wsdl" hello-NIEeS.jsdl

Then, you will able to get a unique job id, in this case, this is urn:gridsam:006868a90a74befa010a753725350037

The job status can be monitored using the command "gridsam-status" (some lines have been broken to improve legibility): 

[gtniees@agorilla examples]$ gridsam-status -s "http://agorilla.niees.group.cam.ac.uk:18080/gridsam/services/gridsam?wsdl" \
    urn:gridsam:006868a90a74befa010a753725350037
  Job Progress: pending -> staging-in -> staged-in -> active -> executed -> staging-out -> staged-out -> done
  --- pending - 2006-04-07 17:37:29.0 ---
  job is being scheduled
  --- staging-in - 2006-04-07 17:37:29.0 ---
  staging files...
  --- staged-in - 2006-04-07 17:37:29.0 ---
  no file needs to be staged in
  --- active - 2006-04-07 17:37:29.0 ---
  job is being launched through condor
  --- executed - 2006-04-07 17:37:45.0 ---
  04/07 17:38:48 Job terminated. (1) Normal termination (return value 0) Usr 0 00:00:00, Sys 0 00:00:00 
      - Run Remote Usage Usr 0 00:00:00, Sys 0 00:00:00 
      - Run Local Usage Usr 0 00:00:00, Sys 0 00:00:00 
      - Total Remote Usage Usr 0 00:00:00, Sys 0 00:00:00 
      - Total Local Usage 0 
      - Run Bytes Sent By Job 13888 
      - Run Bytes Received By Job 0 
      - Total Bytes Sent By Job 13888 
      - Total Bytes Received By Job
  --- staging-out - 2006-04-07 17:37:45.0 ---
  staging files out...
  --- staged-out - 2006-04-07 17:37:45.0 ---
  no file needs to be staged out
  --- done - 2006-04-07 17:37:45.0 ---
  Job completed
  --------------
  Job Properties
  --------------
  urn:condor:classad=universe=vanilla
  when_to_transfer_output=ON_EXIT
  should_transfer_files=IF_NEEDED
  notification=Never
  log=/tmp/condor.log

  executable=/bin/echo
      arguments=Hello NIEeS!!
  queue
  urn:condor:clusterid=287
  urn:gridsam:exitcode=0

The job workflow can be seen below:

Description

[edit]

Example 2: submission with input and output staging

This example is going to stage input and output files, but no executable files. It will download either via http, ftp or sftp. The JSDL file used in this example is "cat-staging.nodir.jsdl" 

<JobDefinition xmlns="http://schemas.ggf.org/jsdl/2005/06/jsdl">
     <JobDescription>
        <JobIdentification>
            <JobName>cat job</JobName>
            <Description>cat job description</Description>
            <JobAnnotation>no annotation</JobAnnotation>
            <JobProject>gridsam project</JobProject>
        </JobIdentification>
        <Application>
            <POSIXApplication xmlns="http://schemas.ggf.org/jsdl/2005/06/jsdl-posix">
                <Executable>/bin/cat</Executable>
                <Argument>file1.txt file2.txt</Argument>
                <Output>stdout.txt</Output>
                <Error>stderr.txt</Error>
            </POSIXApplication>
        </Application>
        <DataStaging>
            <FileName>file1.txt</FileName>
            <CreationFlag >overwrite</CreationFlag>
            <DeleteOnTermination>false</DeleteOnTermination>
            <Source>
                <URI>http://128.232.232.46/gridsamfiles/hello.txt</URI>
            </Source>
        </DataStaging>
        <DataStaging>
            <FileName>file2.txt</FileName>
            <CreationFlag>overwrite</CreationFlag>
            <DeleteOnTermination>false</DeleteOnTermination>
            <Source>
                <URI>ftp://anonymous:anonymous@128.232.232.41:19245/subdir/input-file.txt</URI>
            </Source>
        </DataStaging>
        <DataStaging>
            <FileName>stdout.txt</FileName>
            <CreationFlag>overwrite</CreationFlag>
            <DeleteOnTermination>false</DeleteOnTermination>
            <Target>
                <URI>ftp://anonymous:anonymous@128.232.232.41:19245/output/stdout.txt</URI>
            </Target>
        </DataStaging>
    </JobDescription>
</JobDefinition>

This example will stage in two input files, one from the NIEeS website and the other one from a local ftp server. Finally, it will stage out the output result and put it in the default ftp directory. 

 [gtniees@agorilla examples]$ gridsam-submit -s "http://agorilla.niees.group.cam.ac.uk:18080/gridsam/services/gridsam?wsdl" cat-staging.nodir.jsdl

Again you can use gridsam-status to monitor the job status: 

 [gtniees@agorilla examples]$ gridsam-status -s "http://agorilla.niees.group.cam.ac.uk:18080/gridsam/services/gridsam?wsdl" \
     urn:gridsam:006868a90a74befa010a755aadc00048

After the job is done, one can go to the default FTP root directory and check the output data. 

 [gtniees@agorilla output]$ less stdout.txt
   Hello! chaps This is NIEeS GridSAM file staging test!
[edit]

Example 3: submission with executable, input and output staging

In this example, you are authorised to submit a job to the GridSAM instance. You are also authorised to stage an executable to the system. The backend system can execute foreign executables in the form of a sh shell script. The client host can provide a FTP service (at port 19245) to the remote GridSAM instance. The JSDL file used in this example is "remotecat-staging.jsdl". 

<JobDefinition xmlns="http://schemas.ggf.org/jsdl/2005/06/jsdl">
    <JobDescription>
        <JobIdentification>
            <JobName>cat job</JobName>
            <Description>cat job description</Description>
            <JobAnnotation>no annotation</JobAnnotation>
            <JobProject>gridsam project</JobProject>
        </JobIdentification>
        <Application>
            <POSIXApplication xmlns="http://schemas.ggf.org/jsdl/2005/06/jsdl-posix">
                <Executable>bin/concat</Executable>
                <Argument>file2.txt</Argument>
                <Output>stdout.txt</Output>
                <Error>stderr.txt</Error>
                <Environment name="FIRST_INPUT">file1.txt</Environment>
            </POSIXApplication>
        </Application>
        <DataStaging>
            <FileName>bin/concat</FileName>
            <CreationFlag>overwrite</CreationFlag>
            <Source>
                <URI>http://panda.niees.group.cam.ac.uk/gridsamfiles/concat.sh</URI>
            </Source>
        </DataStaging>
        <DataStaging>
            <FileName>file1.txt</FileName>
            <CreationFlag>overwrite</CreationFlag>
            <Source>
                <URI>http://panda.niees.group.cam.ac.uk/gridsamfiles/hello.txt</URI>
            </Source>
        </DataStaging>
        <DataStaging>
            <FileName>file2.txt</FileName>
            <CreationFlag>overwrite</CreationFlag>
            <Source>
                <URI>ftp://anonymous:anonymous@128.232.232.41:19245/subdir/input-file.txt</URI>
            </Source>
        </DataStaging>
        <DataStaging>
            <FileName>stdout.txt</FileName>
            <CreationFlag>overwrite</CreationFlag>
            <DeleteOnTermination>true</DeleteOnTermination>
            <Target>
<URI>ftp://anonymous:anonymous@128.232.232.41:19245/output/remotecatout.txt</URI>
            </Target>
        </DataStaging>
    </JobDescription>
</JobDefinition>

Again, submit jobs using gridsam-submit (some lines have been broken to improve legibility): 

[gtniees@agorilla examples]$ gridsam-submit -s "http://agorilla.niees.group.cam.ac.uk:18080/gridsam/services/gridsam?wsdl" \
    remotecat-staging.jsdl urn:gridsam:006868a90a74befa010a83070d670059

[gtniees@agorilla examples]$ gridsam-status -s "http://agorilla.niees.group.cam.ac.uk:18080/gridsam/services/gridsam?wsdl" \
    urn:gridsam:006868a90a74befa010a83070d670059
Job Progress: pending -> staging-in -> staged-in -> active -> executed -> staging-out -> staged-out -> done

--- pending - 2006-04-10 09:59:38.0 ---
job is being scheduled
--- staging-in - 2006-04-10 09:59:38.0 ---
staging files...
--- staged-in - 2006-04-10 09:59:38.0 ---
3 files staged in
--- active - 2006-04-10 09:59:38.0 ---
job is being launched through condor
--- executed - 2006-04-10 10:02:45.0 ---
04/10 10:04:14 Job terminated. (1) Normal termination (return value 3) Usr 0 00:00:00, Sys 0 00:00:00 
  - Run Remote Usage Usr 0 00:00:00, Sys 0 00:00:00 
  - Run Local Usage Usr 0 00:00:00, Sys 0 00:00:00 
  - Total Remote Usage Usr 0 00:00:00, Sys 0 00:00:00 
  - Total Local Usage 54 
  - Run Bytes Sent By Job 146 
  - Run Bytes Received By Job 54 
  - Total Bytes Sent By Job 146 
  - Total Bytes Received By Job
--- staging-out - 2006-04-10 10:02:45.0 ---
staging files out...
--- staged-out - 2006-04-10 10:02:45.0 ---
1 files staged out
--- done - 2006-04-10 10:02:45.0 ---
Job completed

--------------
Job Properties
--------------
urn:gridsam:Description=cat job description
urn:gridsam:JobProject=gridsam project
urn:gridsam:JobAnnotation=no annotation
urn:gridsam:JobName=cat job
urn:condor:classad=universe=vanilla
when_to_transfer_output=ON_EXIT
should_transfer_files=IF_NEEDED
notification=Never
log=/tmp/condor.log


executable=bin/concat
    arguments=file2.txt
output=stdout.txt

error=stderr.txt
    environment=FIRST_INPUT=file1.txt;
transfer_input_files=bin/concat,file1.txt,file2.txt
transfer_output_files=stdout.txt
queue
urn:condor:clusterid=289
urn:gridsam:exitcode=3

Check the output file: 

[gtniees@agorilla output]$ less remotecatout.txt
  Hello!
  chaps This is NIEeS GridSAM file staging test!

  It is working fine!!

There are more different job submission scenarios. You can write your own JSDL file and execute your own jobs. For more information about how to write JSDL, you can look at the Global Grid Forum (GGF) website and obtain JSDL document from this link: http://forge.gridforum.org/projects/jsdl-wg

For more information about how to use NIEeS GridSAM for further testing, please contact NIEeS technical staff (technical@niees.ac.uk)

Retrieved from "http://gridinfo.niees.ac.uk/index.php/Using_GridSAM_at_NIEeS"
Personal tools