Materials Data Centre - EP2DC Blog

EP2DC - Future plans....

Fri, 08 Jan 2010 05:56:22 -0800

With the project officially ended, team members are keen to ensure EP2DC finds broad utilisation. The team at Southampton has no misconceptions about the fact encouraging uptake will prove a greater challenge than the development effort of the past 6 months.

It is clear though that the scientific community recognizes the merits and benefits of effective research data management, and we are optimistic that our efforts to encourage uptake will be well received. A core team of Kenji Takeda, Tim Austin, and Mark Scott will thus continue to maintain and develop the facility in anticipation of opportunities to serve the research data management needs of the scientific community.

EP2DC Final Progress Post

Tue, 22 Dec 2009 02:16:00 -0800

Title of Primary Project Output: EP2DC

Screenshots or diagram of prototype: Here are some screenshots that show the EP2DC data deposit and retrieval screens

Users of EP2DC access the data service via an EPrints repository. We have appended the standard EPrints document workflow to include a data deposit step. It also means that our plugin can just be dropped into any customised EPrints deployment. The aim is to use a workflow that is familiar to users, but simply add extra functionality in a seamless way.

This screenshot above shows the EPrints data deposit page that we have added. Here users can upload an XML data document. This is then sent, via Web Services, to a data centre. In this prototype we are using the Materials Data Centre (www.materialsdatacentre.com) as our data repository. The federated architecture and REST services mean that we can support many-to-many relationships between EPrints and data repositories.

The EP2DC stage for uploading experimental data includes an option (collapsed in the previous figure) that allows metadata associated with the test data to be entered. The fields marked with a red star are mandatory. As shown in the adjacent figure, one of these madatory fields defines the access control. This field affects the data retrieval process, as follows:

Open—allows data retrieved through the EP2DC EPrints repository to be downloaded by anyone.
Restricted to registered users—allows data retrieved from the EP2DC EPrints repository to be downloaded by registered users.
On demand—data is supplied on request direct from the owner.

The last stage in the EPrints default workflow is to deposit the unit of work (meaning all of the documentation, figures, etc. together with the accompanying data). The data will be deposited in the remote data centre, which is responsible for validating the data against the corresponding XML Schema Definition. If the data sets are validated and deposited successfully, a page similar to that shown in the above figure is displayed. Note this shows all debug information, that will normally be hidden for the user but is included here to illustrate the service calls.

This screenshot above shows the Materials Data Centre (MDC) back-end that we are using. It is a Microsoft SharePoint site that includes document libraries to hold the XML schemas that we validate against (mdcschemas folder), and the data files uploaded through the EPrints interface (mdcdata folder). Users will be able to upload data into MDC directly through a separate web interface. EPrints talks to MDC via a REST interface directly so that the data centre can be made transparent to the end user if they just want to use an EPrints front-end. This federated approach means that many-to-many architecture is supported. i.e. multiple EPrints repositories can talk to multiple data centres.

This screenshot above shows the data retrieval screen from EPrints. When a user selects an article they can immediately see what datasets have been uploaded that relate to that document. This link to the underlying data allows researchers to much more readily access data that promotes data re-use, and should also encourage more citations, subject to the access control defined by the autorh at upload time.

Description of Prototype: The objective is to develop a prototype module to enable the EPrints repository to support the submission of XML-formatted experimental data together with the manuscript to which they correspond. The basic motivations for the work are to promote the conservation of experimental data and to link data to publications. As Semantic technologies become established, the ever-increasing body of experimental data will provide new opportunities for knowledge discovery within and across disciplines.

End User of Prototype: EPrints is used worldwide as a repository for scholarly publications. This prototype allows users to, probably for the first time, include datasets in a remote dtata repository that relate to the paper content and link to it. This is a new publishing model for data, as it provides seamless upload and retrieval of data using a federated architecture. This helps to elevate data as a first-class citizen in the open access world, accelerating research and also increasing citation rates - an important incentive for academics given current/future research audit processes. By supporting XML schemas, we hope to improve data quality and consistency and further develop semantic linking capabilities. The federated SOA architecture means that the system can support many-to-many repository/data centre relationships in an internet-scalable way.

A typical upload user scenario is:

Scientist writes paperScientist uploads paper to EPrints repository.
Scientist includes dataset (from 1) in EPrints upload, using EP2DC service developed here
Scientist finalises deposit and is shown similar datasets and papers related to her uploaded paper/data. This disproportionate feedback is an incentive for scientist to continue to use the service.

A typical user search scenario is:

Scientist searches EPrints repository for paper on topic of interest
User finds paper
User sees dataset is available and downloads it
User uses downloaded dataset in their own research, citing paper found in EPrints

Link to working prototype: http://ep2dc-s1.soton.ac.uk - to try the demo you can browse papers and datasets by year or type. To try the upload functionality you need to create an account (simply click 'Create Account' at top of the screen) and follow the instructions.

Link to end user documentation: http://ep2dc.codeplex.com/documentation

Link to code repository or API: http://ep2dc.codeplex.com/

Link to technical documentation: http://ep2dc.codeplex.com/documentation

Date prototype was launched: 11 December 2009

Project Team Names, Emails and Organisations:

          o Product Owner: Philippa Reed
          o Project Concept and Design: Tim Austin
          o Project Advisors: Kenji Takeda , Leslie Carr
          o Developers: Mark Scott , Tim Austin , Dr Steven Johnston
          o All team members are from the University of Southampton

Project Website: http://www.materialsdatacentre.com/ep2dc.html

Table of Content for Project Posts: listed on the right here as categories

Software release on Codeplex

Fri, 11 Dec 2009 00:00:00 -0800

We are delighted to announce that EP2DC is now available for download from Codeplex @ http://ep2dc.codeplex.com/

SharePoint upgrade

Wed, 25 Nov 2009 14:50:23 -0800

We're using Microsoft Sharepoint for our MDC back-end. This already has great out-of-the-box functionality. The team went to the latest conference in October and were delighted to see that SharePoint 2010 has a built in taxonomy engine. Ontologies here we come.

Great Tech

Wed, 25 Nov 2009 14:48:57 -0800

As with any new project, we love to use the latest and greatest tech to get the job done.

EPrints. The core of our document repository work. Written in PERL it's a great platform for development.

REST and SOAP/XML. Interoperable distributed systems architecture, SOA all the way.

C# and Windows Communciation Foundation (WCF). The MDC back-end is written using .NET. WCF means we are able to simultaneously publish REST and SOAP/XMl interfaces from the same code base.

SWORD. We didn't use this, but are planning to include it for the MDC middleware in the very near future.

Inside our toolbag

Wed, 25 Nov 2009 14:42:49 -0800

We've been using a wide variety of tools for teamworking and software development. These are:

www.huddle.net - a great Web 2.0 collaborative teamworking site. Very usual given that the team is spread across 4 buildings at Southampton and Tim Austin in Amsterdam. Huddle was started by a Southampton graduate too!!!

EPrints. Great repository software - need we say more!

Microsoft SharePoint. Here we've used it for the Materials Data Centre back-end. Provides lots of out-of-the-box functionality, and the new version is even better!

Visual Studio 2008 and 2010 Beta. The MDC software has been built on .NET, coded in C#. So Visual Studio is the natural IDE of choice. We love it for its productivity benefits.

SVN. The University of Southampton centrally hosts our SVN server, meaning we don't have to worry about it. We're using the AnkhSVN plug-in for Visual Studio, so our version control is truly seamless.

Skype. Great for video-conferencing, particularly between the UK and Holland. Means we don't have to cancel/postpone meetings when people are out of the office. Works a treat :)

XML validator - www.validome.org - used to check out our XML schema validation. MDC uses the built-in XML validator in SharePoint, but we used this tool to double-check the validity of our MatDB schema.

SWOT analysis

Wed, 25 Nov 2009 14:27:11 -0800

Now we're getting to the end of the EP2DC project it's a good time to reflect. So here's a brief SWOT analysis....

STRENGTHS
Users, users, users... That's what we're all about! We've been completely focussed on what is useful for our research users, materials scientists in this case. We've consulted with them online, in interviews and meetings. This has been particularly easy as the principal investigator, Philippa Reed, is a materials professor and not an IT person. So she's the one we have to convince with our software!

Teamwork. We've had an awesome team on the project. Tim Austin came up with the idea and has been making sure that we've stayed focussed and been doing plenty of EPrints development. Steven Johnston and Mark Scott have been putting together the back-end data centre and web services, using their years of experience. Seb and Tim M-B in the EPrints Services group have delivered in spades, we expect nothing less from these gurus. Philippa Reed and Kenji Takeda have been managing the project and keeping things moving forwards at a real pace. Great people working together, that's what this project has been all about.

Cashing in on JISC investments. We've been lucky enough to be able to build on other JISC proejcts, notably EPrints and the Materials Data Centre (MDC). We've also benefited from the extensive experience from the Microsoft Institute for HPC.

Interoperability. We love mixing it up! We're using EPrints on Linux and MySQL, talking to SharePoint 2007 on top of SQL Server. It all talks seamlessly over REST and Web Services, SOA really does work.

WEAKNESSES
Short Timescales. With only a few months to complete this project it has been challenging. Once everyone was up to speed and working together, several members of the team were off at JISC and other conferences. All good stuff, and certainly worthwhile, particularly the Microsoft SharePoint and Tech Ed conferences. While this saved development time, it proved tricky to balance everyone's workloads.

Distributed teamwork. The project was carefully split to minimise dependencies, so that work could be carried out by each team member independently. That said, Tim Austin was in Amsterdam, and the rest of the team scattered across the Southampton campus.Projects are always easier when everyone is in the same room!

Dependencies. With so many interacting components we had to be careful about dependencies. The SOA approach helped this, but we did rely on the Materials Data Centre to be up. This project started a bit late, so the MDC wasn't ready when we wanted it to be. However, with lots of hours (and late nights by Mark!) we managed to catch up.

OPPORTUNITIES
Data deluge. This project is really just the tip of the iceberg for managing the data deluge starting to swamp scientists and engineers. By providing linked data, particualrly directly to publications, we can help users manage this brave new world.

New models for research. This project hopes to help develop this new mode of research - linked data and publications. It is part of the Fourth Paradigm vision of the late, great Jim Gray. We really are entering a new era in scientific discovery.

Extensibility. The generic nature of the implementation means that it can be extended to other disciplines, and not just science and engineering.

Protecting research investments. By creating a usable framework for researchers to store and link their data we can help preserve the knowledge gained during years of research. This increases the longevity and usefulness of research, giving better value for money in the long-term, and reducing the danger of duplication by making existing research openly available.

THREATS
Driving uptake. As with any approach, getting users to take it onboard is a challenge. Making it usable is a key enabler, and this is why we are so focussed on what users want. Still, it is a challenge to drive uptake, but one which we are relishing.

Overhead for users. Adding the feature to include data with EPrints uploads is great, but adds additional steps to the process. This may be off=putting for users, as they are busy and just want to get on with their research. We've therefore focussed on making it as painless as possible, and minimise the overhead to users.

Federated security. To make the system usable we need to have seamless, single-sign-on. More difficult is the federated security we need to introduce so that the back-end services can talk to each other securely. This is a classic internet federated security problem. While we have not implemented a full, elegant solution, we now know how to do this :)

Policy barriers. Technology is one thing, but policies are another. We believe we have implemetned a very usable system. However, organisational policies must be defined before wider uptake, particularly regarding data access and ownership. As time goes on and the idea of sharing data becomes more accepted, we are optimistic that policies will modernise to allow this.

MatDB schema

Tue, 10 Nov 2009 14:49:26 -0800

A review of materials schemas has identified 3 potential candidates for the MDC back-end, MatML @ www.oasis-open.org/committees/materials, EC MatDB @ odin.jrc.ec.europa.eu, and NMC MatDB @ www.nims.go.jp/vamas_twa10. These schemas are at various stages of development, each with their own benefits and limitations. Most promising appears to be the EC MatDB schema, about which you can read more @ www.jstage.jst.go.jp/article/dsj/7/0/7_179/_article.

Shibboleth for EPrints- BIG win!

Tue, 10 Nov 2009 14:38:10 -0800

In order to allow users from different organisations to access our EP2DC system, we have implemented Shibboleth for EPrints. This is a new feature, so thousands of EPrints users worldwide will benefit from this development :):):)

Disproportionate feedback

Thu, 05 Nov 2009 00:00:00 -0800

After the great meeting with David Flanders, we're now implementing our disproportionate feedback to the user. When the user makes a deposit in the repository, instead of a simple 'OK' message, we return a cascade of extra information that they might be interested in. Simliar to the Amazon's "Customer who bought this item also bought" feature. We return other datasets that are related to the one deposited. Hopefully this will delight the user and make them come back for more ;)