Overview

Data discovery and delivery are inherent components of many aspects of science. They can be considered part of a processing chain that starts with raw data from a variety of sources, and ends with the graphical production of information that is directly used in scientific research. The NERC DataGrid provides tools and information products that can be used to facilitate this process.

Can I use the NDG now?

The NDG is still under active development, but you can certainly use some NDG  components now. The most useful components now are:

1. The NDG Production Discovery gateway (provides access to thousands of datasets).
   
2. The NDG development discovery gateway (this is a technology preview, and may not be stable).
   
3. The Climate Science Modelling Language (CSML).
4. The NDG Vocabulary Server.

See the NDG trac site for the most up to date information about the NDG.

What is the NDG?

See the NDG flyer.

The NDG consists, or will consist of, the following components:

1. The Climate Science Modelling Language information Model. (CSML is both a UML description and an XML schema which is an “Application Schema” of the “Geographic Markup Language”). CSML provides format independent descriptions of the parameters and organization of datasets according to the “Sampling Features” and “Observations and Measurements” frameworks of the Open Geospatial Consortium (OGC).

2. The CSML Toolbox: provides code which creates and manipulates documents which conform to the CSML schema.

3. The CSML Data Services: A set of web services which can expose CSML documents and the data which they point to via the Open Geospatial Consortium protocols: WFS, WMS and WCS, and which provide limited processing options on the datasets exposed by those services (via the WPS). The CSML Data Services are optionally secured by the NDG Security Infrastructure described below.

4. CSML documents and underlying data are held at the NDG partners: British Oceanographic Data Centre, the British Atmospheric Data Centre, the Plymouth Marine Laboratory, and the National Oceanographic Centre, Southampton.

5. The NDG Data Graphical User Interface: provides the web interface which allows a user to exercise the CSML web services to manipulate data.

6. The NDG Data Graphical Interface and the CSML Data Services are deployed at each of the NDG partners.

7. The Metadata Objects for Linking Environmental Sciences, MOLES, schema (an XML schema). MOLES provides a common and standardized way of describing key aspects of datasets: the Activities which generate data, the Observation Stations at which the data are collected (or produced in the case of simulations), the Data Production Tools, and the Data Entities themselves.

8. The MOLES XQuery definitions and associated software: Because MOLES documents are intended to provide a common way of accessing definitions, software is needed to import and export information stored in other common formats. The MOLES XQuery system supports import and export (in three languages: XQuery, Java and Python) from the NASA Global Change Master Directory Directory Interchange Format (DIF), Dublin Core, the Marine Data Information Partnership, and ISO19139 metadata formats.

9. MOLES documents have been produced for datasets held at all the NDG partners (this like all the other population tasks within the NDG has, and continues to be, a very time consuming task).

10. The NDG Vocabulary Service: MOLES, CSML, and other schema describing data need to make use of “controlled vocabularies” (constrained and standardized ways of describing components within the documents – it is these standardized vocabularies which will support the “Semantic Web”, ie software which can go beyond simply reporting document contents, but make “inferences” based on their contents). The NDG Vocabulary service itself consists of a web service deployed at the British Oceanographic Data Centre, the work required to maintain the vocabularies exposed by the lists, and some prototype inference engines which map terms from one list to another.

11. The MOLES browser: software which exposes MOLES documents to users, and allows them to navigate between the various components of the data descriptions and make leaps from one dataset to another based on common components (activities, data production tools, observation stations, parameters etc).

12. Deployment of the MOLES browser at the NDG partners, coupled to the NDG security infrastructure, and underlying MOLES databases.

13. The NDG Discovery Infrastructure: databases of documents designed to support data location are produced at all the NDG partners, and stored at each site in databases which support the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH). Clients (not limited to NDG clients) which harvest these documents can expose the records in their own websites and provide links to NDG data. These discovery records are produced using the MOLES XQuery software and include links to both browse services and

14. The NDG Discovery Gateway: provides a user interface to discovery records harvested from the NDG Discovery Infrastructure, and supports both SOAP and REST web service interfaces to the underlying discovery database. (These interfaces can and are used by other projects, e.g. the Marine Data information Partnership allow their discovery records to be harvested by the NDG, and have developed their own interface to those records based on the SOAP interface).

15. The NDG Security Infrastructure, Server Toolkit: provides the services and software libraries needed to

    1. deliver an authentication infrastructure (based on PKI technology and X509 proxy certificates held in multiple MyProxy databases – the “Session Managers”)

    2. connect a preexisting user database to the NDG authentication infrastructure,

    3. control access to resources (the “gatekeepers”)

    4. control authorization information and map between authorization paradigms at different NDG partner (the “attribute authorities”)

16. The NDG Security Infrastructure, Client Toolkit: a software library to allow programmers to modify existing, and develop new, Java and Python applications which are secured by NDG technologies. The initial NDG applications using this client toolkit are the MOLES browser and the NDG Data Graphical User interfaces described above.

17. The NDG security infrastructure is deployed at each of the NDG partners (including bespoke software at each site to integrate the local access control to the NDG infrastructure via the server toolkit)