[myexperiment-hackers] Re: myExperiment and Trident workflows

[Jiten Bhagat]

Hi Roger,

Apologies for the delayed response. Hope you're well too.

Thanks very much for the info. We've looked this over with the team here 
and the whole scenario looks fine, and for the most part very doable.

Comments:

- For stage 2 in the diagram: we currently have two mechanisms to upload 
workflow packages:
 - via the HTTP/HTML interface, for authenticated users only (as you 
mentioned).
 - via the RESTful API (HTTP/XML based; which currently requires HTTP 
Basic Auth for authentication of a user's account to allow the upload 
under their myExp user account. We're looking to make this SSL enabled 
soon). See: 
http://wiki.myexperiment.org/index.php/Developer:API#POST_workflow for 
documentation on how to do this. The "content_type" identifier for 
Trident OPC packages is "trident_opc" and for Trident XOML files alone 
it's "application/xaml+xml" (let us know if you would like any of these 
changed). For the latter I suspect we won't be able to parse any 
metadata out of the XOML file, but this can still be provided in the 
HTML form and/or the RESTful API.

- I should also mention that it's possible to have full myExperiment 
search and tag cloud functionality (and more) within the Trident system, 
through the RESTful API. See: 
http://wiki.myexperiment.org/index.php/Developer:Taverna_plugin for an 
example of a plugin I developed for Taverna 1 (which we are improving 
for Taverna 2, with more write abilities). For API reference: 
http://wiki.myexperiment.org/index.php/Developer:API (this is maintained 
by Don Cruickshank <address@hidden> who is able to provide support).

- The running of Trident workflows from within the myExp interface might 
be something that we cannot do just yet and maybe something we should 
plan for further down the line?

- In terms of parsing metadata from the OPC packages to display in 
myExperiment, the list you provided is very comprehensive and exactly 
what we need, thanks. Few comments on this:
 - The workflow description can have most HTML elements in it (allowing 
for rich text).
 - Currently, we use sequential version numbers in myExp that the 
system automatically generates. What we can do if a 'version' field is 
provided is to store that as a special "version label".
 - We require just one preview image, which we will resize to the 
various different sizes (for thumbnails etc).
 - If you could also provide an SVG version of the preview image, in 
the package, that would be great (but not compulsory).
 - We support the following license options (if you need to support 
different licenses, let us know):
   - "by-nd" - Creative Commons Attribution-NoDerivs 3.0 License
   - "by-sa" - Creative Commons Attribution-Share Alike 3.0 License
   - "by" - Creative Commons Attribution 3.0 License
 - For credits: could you make these either names of people, or full 
URLs to people's myExperiment profiles? This way we can match what we 
get to existing users on myExp. We're not sure about actually creating 
users based on new data. In a worst case scenario, we'll try and store 
these as names/URLs directly so they still show up under credits.
 - For attributions: these are about attributing existing 
workflows/files/etc that were used to help build the workflow (and it's 
code and so on). So these would be URLs to existing workflows/files/etc, 
much like for credits, above, but rather than being about people it's 
about things.

- In terms of downloading a workflow, this is either possible through a 
direct download URL 
(http://www.myexperiment.org/workflows/{id}/download. Note: only users 
who have download permissions will be able to get a file back from this 
URL) or via the RESTful API.

- Speaking of permissions, when uploading a workflow through the RESTful 
API, default credits, sharing and license* settings are applied (see 
attached image). The user would then need to go the workflow manage page 
to change these (http://www.myexperiment.org/workflows/{id}/edit).

* If credits and license are not provided.

In terms of taking this forward, one of the things required from the 
myExperiment side is the development of a "workflow processor" (in 
Ruby), that acts as the bridge between the Trident OPC package and the 
myExp system. The interface/skeleton of this processor is here: 
http://myexperiment.rubyforge.org/svn/trunk/lib/workflow_processors/interface.rb 
and an example is the Taverna 1 processor here: 
http://myexperiment.rubyforge.org/svn/trunk/lib/workflow_processors/taverna_scufl.rb

You will notice that this interface does not currently support all the 
metadata elements you have mentioned in your list. However this isn't a 
problem and we can easily add new methods to this interface to cater for 
the new metadata elements. (We'll then write the appropriate routines in 
the create workflow process to add these to our database).

At this stage, would it be possible for one of your devs to write an 
initial version of this workflow processor so we have something to build 
on? I am at hand to provide info/support (especially since the interface 
isn't fully documented yet).

Hope this is useful.

Kind regards,
Jits


Roger Barga wrote:
> Hi David, Jits:
>
>  
>
> I hope all is going well for you both.  Below is a first draft of our 
> view of publishing Trident workflows to myExperiment.  We would very 
> much appreciate your feedback on this and especially on metadata 
> provided with the OPC package.  Once we have this in hand we would 
> like to begin creating these packages to test out the end-to-end 
> implementation.
>
>  
>
> Roger
>
>  
>
> *From:* Dean Guo (DI)
> *Sent:* Thursday, April 09, 2009 5:26 PM
> *To:* Roger Barga
> *Subject:* myExperiment and Trident workflows
>
>  
>
> Roger,
>
>  
>
> Here is the overview of publishing Trident workflows to myExperiment 
> and downloading Trident workflows from it.    Please review it and 
> modify it as needed.  More details below the diagram.
>
>  
>
>  
>
> Currently Trident can export a workflow package (a zip file) with core 
> entities required to run Trident workflows.   We are working on 
> importing a package back to Trident to complete the round trip. 
>
>  
>
> We will add more data to a Trident workflow package specifically 
> targeted for myExperiement.    Trident UI design will support data 
> entries for it.   We need to get more information from myExperiment in 
> terms of the key meta data needed.    Here is a preliminary list of 
> data elements that we have identified for myExperiment:
>
>  
>
> ·         Workflow category
>
> ·         Workflow name
>
> ·         Workflow description
>
> ·         Keywords
>
> ·         Version
>
> ·         Type: Trident (XOML)  (Windows Workflow MIME type)
>
> ·         WF Preview image
>
> ·         WF Thumbnail image
>
> ·         License (Open source license name)
>
> ·         Credits (People or groups)
>
> o   Name
>
> o   email
>
> o   Website
>
> o   Location (city, state, and country)
>
> ·         Attributions
>
> o   WF XOML
>
> o   WF assemblies
>
> o   WF Input parameters
>
> o   WF output parameters
>
> ·         Others
>
> o   Documentation (PDF)
>
>  
>
> We need feedback from myExperiment on this list asap.     Thanks.
>
>  
>
> Dean