Merge & Upload support (version 2)

FYI, I would like to replace /REST/ with /data/ in the URIs. I'm not sure if this should be done in 1.5, or should wait for 2.0. The change should be backwards compatible (i.e. /REST/ would still be supported).

Anything which allows creating/archiving of data should allow XML Path for customization of any field in the xml.

Services

/services/import (supports upload of zip, tar.gz, etc) Should support merging to existing sessions.
  • ?dest=/data
  • ?dest=/data/projects/PROJECT
  • ?dest=/data/projects/PROJECT/subjects/SUBJECT
  • ?dest=/data/projects/PROJECT/subjects/SUBJECT/experiments/EXPERIMENT
  • ?dest=/data/experiments/CNDA_E023423
  • ?dest=/prearchive
  • ?dest=/prearchive/PROJECT
  • ?dest=/prearchive/PROJECT/timestamp/session

/services/ data /copy (Duplicate an experiment into a different project, including xml and data)
  • ?src=/data/experiments/CNDA_E023423&dest=/data/projects/A/subjects/B/experiments/C
  • ?src=/prearchive/PROJECT/timestamp/SESSION&dest=/data/projects/A/subjects/B/experiments/C
/services/ data /move (Move a resource (diff name or project)). Currently supported through other means
  • ?src=/data/experiments/CNDA_E023423&dest=/data/projects/A/subjects/B/experiments/C
  • ?src=/prearchive/PROJECT/timestamp/SESSION&dest=/data/projects/A/subjects/B/experiments/C
/services/data/refresh (This is currently called PullDataFromHeaders and would refresh a session or scan based on the DICOM header contents) Currently supported through other means.
  • ?resource=/data/projects/A/subjects/B/experiments/C
  • ?resource=/prearchive/PROJECT/timestamp/SESSION




All services should use URI chunks to define resources. Here is a preliminary list of supported URIs (they should also support GETs)

Supported Prearchive URIs
/prearchive/ (GET,PUT(refresh))
/prearchive/projects (GET,PUT(refresh))
/prearchive/projects/[PROJECT_ID} (GET,PUT(refresh))
/prearchive/projects/{PROJECT_ID}/timestamp/SESSION (GET,DELETE,PUT(refresh))

Supported Archive URIs
/data/projects/ID
/data/projects/ID/subjects/ID
/data/projects/ID/subjects/ID/experiments/ID
/data/projects/ID/subjects/ID/experiments/ID/scans/ID
/data/projects/ID/subjects/ID/experiments/ID/reconstructions/ID
/data/projects/ID/subjects/ID/experiments/ID/assessors/ID
/data/experiments/ID
/data/experiments/ID/scans/ID
/data/experiments/ID/reconstructions/ID
/data/experiments/ID/assessors/ID


Questions


/services/data/import

What should the configuration variables here be? First thought:
  • overwrite: whether to allow import if resource already exists
  • append: whether to append to existing resource, or completely overwrite the resource
  • backup: whether the pre-existing resource should be backed up.

/services/Rename

This should move an experiment from one label to another. It needs to physically move the files on the file system, modify the URIs in the XML, and rename the label.
  • Should check permissions, label collision
  • Should create a workflow entry while In Progress, and Complete (Should not proceed when a workflow is In Progress)
  • Should be transaction based (failure at any point will roll back).
  • PARAMETERS:
    • src=/data/projects/X/subjects/X/experiments/X or /data/experiments/X
    • dest=/data/projects/X/subjects/X/experiments/Y or /data/projects/X/experiments/Y
  • REQUEST BODY: empty
  • CODES:
    • 409: Label already in use
    • 404: Session not found
    • 403: Inadequate permissions for the session or its assessors
    • 423: Locked (meaning processing is currently active for this session)
  • RESPONSE BODY: URI=new URI

Task API (Workflow)

This isn't currently in development, but was modeled as an example of other APIs we could support)
/tasks
(PUT to create, can translate from data URI)
/tasks/ID

<task ID="autoGen" status="" ...>
   <user>tolsen</user>
   <input>
      <dataURI...>
   </input>
   <output>
      <dataURI...>
   </output>
</task>


Pipeline API

Supported URI

/pipelines
/pipelines/{ID}
/project/{ID}/pipelines
/project/{ID}/pipelines

Services

/pipelines/Launch (launches a pipeline and creates a task)
  • pipeline URI - optional (default=AutoRun)
  • data URI

Notification API

This isn't currently in development, but was modeled as an example of other APIs we could support)
Services
/notifications/Monitor
Inputs
  • Task URL or search criteria
  • Data URL or search criteria
  • Action
    • 1: Text
    • 2: Email
    • 3: Site List
  • user or user group


Merge & Upload support (version 1)



URLs to upload files to archive (using DICOM analysis tools)

/REST/projects/ID/subjects/ID/experiments/ID?action=import (ZIP)
/REST/projects/ID/subjects/ID/experiments/ID/scans/5?action=import (ZIP) (ELIMINATED: Should POST to session level)

URLs to run DICOM analysis tools on existing files

/REST/projects/ID/subjects/ID/experiments/ID?action=pullDataFromHeaders
/REST/projects/ID/subjects/ID/experiments/ID/scans/5?action=pullDataFromHeaders

URLs to copy files from archive

/REST/projects/ID/subjects/ID/experiments/ID?action=copyTo&project=B&subject_id=B1&label=B2
/REST/projects/ID/subjects/ID/experiments/ID?action=copyToPrearc&project=B

URLs to upload Files to the prearchive

/REST/projects/ID/prearchive?action=import (ZIP)
/REST/projects/ID/prearchive/12324214214/identifier/scans?action=import (ZIP) (ELIMINATED: Should POST to session level)
/REST/projects/ID/prearchive/12324214214/identifier?action=rebuild

URLs to merge prearchive and archive

/REST/projects/ID/subjects/ID/experiments/ID?action=mergeFrom&path=/prearchive/1321231231/identifier
/REST/projects/ID/prearchive/12324214214/identifier?action=mergeFrom&path=/prearchive/1321231231/identifier

URL to archive prearchive sessions

/REST/projects/ID/prearchive/12324214214/identifier?action=archive&project=B&subject_id=B1&label=B2





Merge & Upload support details

POST ZIP /REST/projects/ID/subjects/ID/experiments/ID?action=import
Description:
Supports uploading files to a new session. The added files will go through the file analysis tools (DICOM, ECAT, etc) and be restructured into scan directories. The files will then be transferred into the archive space and the standard auto run processing will be called.
Open questions:
Can this be asynchronous? (if not it must somehow wait for Transfer, but not the AutoRun)... Pretty sure it needs to be synchronous.
Should it allow adding files to existing sessions (Merge)?... I don't think so. If you want to merge, it should go through prearchive so we are only supporting a single merge point. Per Dan, it should. This means the scans can be POSTed at this level (eliminating the need for separate support)
Should this run the auto-run pipeline?... I don't think so.
Should multiple actions be supported (action=import&action=auto-run&action=somethingelse)... Maybe. not for now.
Implementation:
Mimic how the Upload Images works in Auto-archive mode.
  • Use PrearcImporter to extract/organize/review files into prearchive space.
  • Fix and store the generated session xml (fail if there is more then one)
  • Transfer files to archive space
  • Auto-Run pipeline

Consider De-indentification and De-compression




POST ZIP /REST/projects/ID/subjects/ID/experiments/ID/scans/5?action=import
Description:
Supports uploading files to a new scan. The added files will go through the file analysis tools (DICOM, ECAT, etc) and be restructured into a scan directory. The files will then be transferred into the archive space and the standard auto run processing will NOT be run.
Open questions:
Can this be asynchronous? (if not it must somehow wait for Transfer)
Should it allow adding files to existing scans (Merge)? ...Please, no.
Implementation:
Mimic how the Upload Images works in Auto-archive mode.
  • Use PrearcImporter to extract/organize/review files into prearchive space.
  • Fix and store the generated scan xml (fail if there is more than one)
  • Transfer files to archive space




POST /REST/projects/ID/subjects/ID/experiments/ID?action=pullDataFromHeaders
Description:
Supports running our file analysis tools (DICOM, ECAT, etc) on existing image sessions. It will re-generate the xml. It will not delete any data from the existing data (so existing recons, assessments, etc will be preserved).
Implementation:
In testing




POST /REST/projects/ID/subjects/ID/experiments/ID/scans/5?action=pullDataFromHeaders
Description:
Supports running our file analysis tools (DICOM, ECAT, etc) on an existing image scan. It will re-generate the scan xml. It will not delete any data from the existing data.
Implementation:
In testing




POST ZIP /REST/projects/ID/subjects/ID/experiments/ID?action=copyTo&project=B&subject_id=B1&label=B2
Description:
Will copy the session to another project. It will do this by copying the files and modifying the session xml.
Open questions:
Should it copy the reconstructions and assessments?
This should support copying only some scans.




POST ZIP /REST/projects/ID/subjects/ID/experiments/ID?action=copyToPrearc&project=B
Description:
Will copy the session to the prearchive space of another project. This could be used to merge the session with another archived session.




POST ZIP /REST/projects/ID/prearchive?action=import
Description:
Post files to the prearchive for a given project. Files will be reorganized by the file analysis tools into one timestamped folder with one or more sessions in it.




POST ZIP /REST/projects/ID/prearchive/12324214214/identifier/scans?action=import
Description:
Post files to the prearchive for a given session within a project. Files will be reorganized by the file analysis tools into one more scans in the session directory. The session xml will be regenerated




POST ZIP /REST/projects/ID/prearchive/12324214214?action=rebuild
Description:
Will use our file analysis tools to rebuild the session xml for the existing session files.




POST ZIP /REST/projects/ID/subjects/ID/experiments/ID?action=mergeFrom&path=/prearchive/1321231231/identifier
Description:
Will merge files from the prearchive space into an existing session. The files in the prearchive will be checked for compatability with the files in the archive space.
Implementation:
1. Rebuild prearchive session XML to verify that it is complete and
correct.
2. For each scan in prearchive session, check whether scan already
exists in archived session. If the session already exists, copy
any missing data from archived session into prearchive session.
Existing, identical data is acceptable. Existing, conflicting data
is an error that aborts the merge operation and requires manual
intervention.
3. If any data were retrieved from the archived session, rebuild the
prearchive session XML.
4. Check all prearchive scans for compatibility with archived scans.
Any type conflicts are errors that abort the merge operation.
5. POST all new or modified scan elements to the archived session
6. Invoke merge/transfer pipeline to move files from prearchive into
archive session directory.






POST ZIP /REST/projects/ID/prearchive/12324214214/identifier?action=mergeFrom&path=/projects/X/prearchive/1321231231/identifier
Description:
Will merge the files form two separate prearchive folders.
Implementation:
Check the Study Instance UIDs of the two sessions. If identical, we
can simply move all the data into a single, unified session directory
and rebuild the session XML. If the Study Instance UIDs are different,
the merge operation is similar to the procedure described above for
merging into the archive.





POST ZIP /REST/projects/ID/prearchive/12324214214/identifier?action=archive&project=B&subject_id=B1&label=B2
Description:
Will archive a prearchive session into the archive space. If the session already exists it will return an error code of 409 (Conflict)?