Virtual Organisation

From GridInfo

Introduction

The concept of the Virtual Organisation arose around 10–15 years ago, but many of the practices of virtual organisations can be traced back at least four decades. For example, Sor (1999) has described how many of the features of virtual organisations can be discerned within the organisation of the housing construction industry in Western Australia in the early 1960’s. Much has been written about the more specific concept of the ‘Virtual Business’, and many definitions of VO’s are particularly pertinent to the industrial sector. The strong motivation for the formation of VO’s in industry is the need to reduce costs – this was the driving factor that saw the drive towards collaborations in Western Australia. The economic idea is that costs can be saved if partners with complementary expertise work together towards some common objective. In particular, the infrastructure costs of a group of small units are likely to be much lower in total than the corresponding cost to a large organisation.

As a buzz phrase, there appears to be no agreed definition of the term ‘Virtual Organisation’, but there are a number of key characteristics that can be said to be implicit in the idea. Our working understanding of the concept of the VO is that it is a collection of people working together within an organisational structure that is distinct from their formal allegiances (probably not relevant in an academic context, but some members of a VO may be freelance without any institutional allegiance). A VO will have a particular mission, and may be time limited. Its members will inevitably be geographically dispersed, and will have responsibilities on behalf of their employer institute as well as on behalf of the VO. Membership is also likely to be dynamic, with members joining and leaving when their roles begin and end, rather than remaining members for the while duration of the project around which the VO is established. Some aspects of the VO are similar to those of a more traditional working organisation, but there are other aspects, such as a flatter hierarchy and a voluntary commitment, that are more peculiar to the VO. It is being recognised that the dependence on IT tools is one of the main characteristics of a VO, but it has also been argued that since VO’s existed in practice before the IT revolution, the reliance on IT is not a defining characteristic.

It is useful to distil out of the general idea those concepts that are pertinent to an academic project, taking account of the fact that we do not see the same constraints as would be central to the concept of a Virtual Business. In particular, there is not the equivalent of a fixed objective – research objectives have to be sufficiently flexible to be able to develop during the lifetime of a project in response to new discoveries by the project team and competing research groups, and to adapt to the common situation where proposed research meets unpredictable problems. Moreover, there is no corresponding cost-reduction motivation, and usually it is expected that member groups will stay together for the whole lifetime of a particular project (rather than joining for short periods). The key point we can distil is that of the joining together of dispersed research groups to work together on a topic of common interest with a commonly-agreed management structure (e.g. one nominated leader and a steering committee) that is selected by acclamation rather than imposition. One might ask how this differs from a standard collaboration of distributed groups? We argue here that there are two features of a VO that are qualitatively different from features of a looser collaboration. The first is that there is a sharing of resources that is more akin to the manner in which resources are shared within a formal organisation. For example, there may be semi-formal policies on access to some of the shared resources, and a commitment on the part of the donor to ensure that access is properly maintained. The second particular feature of a VO is an interdependence between member groups that is built into the VO from the outset. It is possible for members of a collaboration to gain benefit from other members but to not be dependent on each other. One of the points of this paper is to show how IT can be used to ensure that the interdependence between VO partners is exploited to the fullest extent possible.

Case study: Mapping the eMinerals project onto the concept of the Virtual Organisation

Consortium grants are increasingly common in the modern funding era, and eScience projects are very much in this mould. Often such consortia are formed by groups with similar skills and related interests. In such a case, there may be no in-built interdependence on the consortia members. Thus there will be a tendency towards working within the traditional model of collaboration that is built upon regular but not frequent face-to-face meetings where progress is reviewed (a good consortium will gain a lot from these meetings), irregular email contacts where help/advice is sought (and the telephone used when this help is urgent), and the reading of manuscripts sent between partners. This is often as much as the partners expect out of the collaboration.

The UK eScience testbed projects have the interdependence between partners built into them from the outset. The eMinerals project, along with many others, consists of scientists, code developers and computer scientists (and some people who straddle two or three of these specialisations). Collaboration is essential if the project is to achieve its goal of constructing an integrated grid structure that meets the needs of the scientists. The scientists need to inform the grid team of their needs, and the grid team need to develop something that the scientists will genuinely find useful. The scientists will also need a lot of help adapting their usual work practices to the new grid-based way of working. The code developers will need to select their priorities based on the needs of the users through working closely with the scientists (as distinct from the more usual case where groups who use a particular code formulate periodic wish lists). The code developers can also use the grid structure to their benefit, and will need to interact with the grid team to ensure they get as much from the grid structure as the science team does. Thus we have sought to build collaboration between all project partners right from the outset. We need to work within the constraint that our teams are based in geographically distinct locations, and yet we want interactions to be much more frequent than would be possible if restricted to face-to-face meetings. It is our aim to eventually be able to enlarge the team, including bringing in international collaborators, and the concept of the VO will need to be robust enough to accommodate enlargement.

It should be noted that the science community from which the eMinerals project is drawn is not used to working within large close collaborations. It is much more characterised by individuals working with their own resources and codes. Consortia may be formed in order to gain access to high-capacity computing facilities, and partnerships may be formed between groups working on common problems. But these are a long way from the concept of the VO outlined in the previous section. We recognise that other areas of science, particularly the particle physics and astronomy communities, have a much stronger track record of the need to work within interdependent collaborations. It is thus an interesting experience to see how the eMinerals project will adapt to the new possibilities afforded by eScience to develop as a functioning VO.

Behind the development of the eMinerals VO is the deployment of the eMinerals minigrid, as described in Calleja et al (2004a), Tyer et al (2004) and Blanshard et al (2004). This is an integration of both compute and data resources, wrapped within Globus security (based on GT 2.4, but also implemented within GT 3.2). The minigrid consists of project-owned and contributed compute resources; the former are a group of three 16-node Linux clusters running PBS, and the latter are condor pools (including a very large pool of 930 teaching PC’s at UCL running Microsoft Windows, and a smaller 24 machine mixed-platform Condor pool in Cambridge) and parallel computing facilities (including a 24-node IBM pSeries machine at Reading). We will shortly add a second 40-node linux cluster and extend the Cambridge Condor pool, and anticipate adding access to a Cambridge-wide Condor grid as described by Calleja et al (2004b).

The data resources are based around the twin pillars of the Storage Resource Broker (SRB) from the San Diego Supercomputing Centre and the Daresbury data portal. The use of the SRB overcomes one of the limitations of Globus, namely difficulties in copying files through the Globus gatekeeper. The SRB allows data to be handled through a server, and allows location-independent access to the data for the users. The data portal handles the metadata associated with the various studies carried out within the project.

Access to the eMinerals minigrid is via eScience X.509 digital certificates. We have also established an eMinerals certificate authority to grant access to members of the VO who are unable to obtain eScience certificates (usually these members are international collaborators). The eMinerals compute resources for production tasks can only be accessed through remote Globus run commands, apart from one cluster for which we allow access via gsissh in order to allow code compilation and testing. We are developing command-line scripts and a compute portal to make access for the users as straightforward as possible.

Developing the eMinerals minigrid has required close collaboration between members of the grid team, and only through equally close collaboration has it been possible to help the scientists make use of the resources of the minigrid. This collaboration has made extensive use of the tools discussed in the following two sections. The development of the eMinerals minigrid illustrates both the resource sharing and team interdependence that, as we have argued, differentiate the VO from a loose collaboration.

We make one final remark in this section. The success of business VO’s depends on the use of standards, so that all partners work to the same system and interoperability is built into the VO from the outset. In the case of the eMinerals project, one standard is the use of the Chemical Markup Language (CML) to describe the simulation data. CML is an application of XML that is designed to handle the science that drives our project, and it enables data to be imported and exported into and out of many of the codes used in the eMinerals project. Similarly, for interoperability at the grid level, the eMinerals project makes use of the benefits of grid standards such as Globus.