icon-usingXNAT.png
Working With XNAT


Advanced Topics

[Edit Nav]




Advanced Topics for XNAT Users: Batch Uploading

Introduction

Much of XNAT's structure is designed to facilitate ongoing research. In that model, data is gradually added to an XNAT installation as it is acquired during the project. Often, users of XNAT will want to host data that was previously collected. There are certain best practices which should be followed when uploading large amounts of legacy data.

Meta-data Creation

All data in XNAT must be assigned to a project. Furthermore, most data also needs to be assigned to a subject. In order to upload data to your XNAT server, you will need to create these meta-data entries.

Projects

Generally, you don't need to create to many projects. Often all of your data is going into one project. Perhaps, your data will be placed into a few projects (determined by DICOM headers). Either way, the number of projects to create is usually fairly minimal. Thus, it is usually advisable to create your projects one at a time through the web interface.

In the rare circumstance where you need a large number of projects created, you can create Projects via the REST api (PUT /REST/projects/PROJECT_ID). More...

Subjects

Commonly, you will need to pre-create your subject entries. This is particularly necessary if you need to specify any subject demographic data, or if you are uploading anything other then DICOM data.

You have several options for how you can create the subjects:

Manual Creation

If your list of subjects is fairly small, the easiest course of action is probably to create the subjects through the standard user interface. This will give you the most latitude in the population of subject demographics, identifiers, etc. More...

REST API

The REST API allows users to create subjects and specify any level of variables associated with that subject. In most cases, it will not be necessary to build XML documents to create subjects via REST. Most of the relevant fields can be populated via query string variables using the XML Path Shortcuts. More...

Spreadsheet Upload

A popular option for creating subjects in batch mode is to upload a spreadsheet (csv). Although XNAT's Upload Spreadsheet feature is still considered to be experimental, it can still be very useful in the simple case of uploading subjects. The spreadsheet feature allows you to build a template describing the spreadsheet you are going to upload, and then upload a csv file which corresponds to that spreadsheet. More...

Auto-create option

If you are uploading DICOM and have no demographics to populate for your subjects, then you can use the auto-create feature in XNAT. To use this feature, you must upload your data via the common DICOM services (DICOM Server), and enable auto-archiving. When sessions are received for auto-archiving, XNAT will attempt to match the data to existing subjects via the DICOM Patient Name header. If it cannot match an existing subject, it will create a new subject using the DICOM Patient Name header.

Demographics

Most demographic data in XNAT is stored as part of the the subject entry. If you would like to store demographic data, you will need to do it as part of the subject creation step (previous). To do this, you will need to use the Manual Creation, REST API, or Spreadsheet Upload. The key step will be to identify which existing XNAT subject fields correspond to the demographics you are trying to store (review the subject creation form and the xnat.xsd). If the fields don't already exist, you will need to create them using the Managing Custom Variables feature or by building a custom schema.

Subject Assessors

If you've created custom subject assessor data types to store your non-imaging data, you have a few options for uploading that data. The creation process should have generated edit forms which would allow you to manually create the entries. The REST API will allow you to build XML documents for your data type and uploading them (PUT /REST/projects/X/subjects/Y/experiments/Z). And, if your data type doesn't include any maxOccurs='unbounded' elements, you can use the Spreadsheet Upload tool to create them.

Image data uploading

A common batch upload need is to upload large amounts of legacy imaging data. XNAT has in-depth support for data of the DICOM format. If you are uploading DICOM data, you will have an easier time of it. That said, it is possible (and has been done frequently) to upload non-DICOM image data. It just requires a bit more work.

Scan Data

Scan data (RAW, DICOM, NIFTI, etc) is by far the most common data that is uploaded to an XNAT server. As such, it is generally the easiest.

DICOM

XNAT has a large amount of support built in for the DICOM format. Most importantly, the XNAT team has developed a DICOM Receiver (DICOM Server ) which can receive DICOM data and place it directly into XNAT using auto-archiving. Furthermore, the XNAT team has developed a client tool, DICOM Browser , which can be used to easily manage the anonymization and upload of DICOM data.

When uploading large amounts of DICOM data via DICOM server, it is advisable to pre-configure your project to use auto-archiving. You can do this on the Manage tab on your Project Report. Once you've enabled this feature, your image data will be placed directly in the permanent file archive and you will not need to transfer the sessions from the prearchive space.

NOTE: The DICOM browser utility has a command prompt version which allows you to upload files from a command line script (skipping the user interface).

Non-DICOM

If you are uploading non-DICOM files, you will need to build a simple script which uploads your data via REST. The script will need to create your imaging session (MR), your scans, and attach the files to those scans. More...

Processed Data

Uploading processed data requires a bit more investigation. In the end, you will need to use the REST api to upload your data. The first step is to determine where in the XNAT hierarchy your data belongs.

Processed data w/ non-image derived measures

If you have derived measures, which you would like to be searchable via the web interface, then you have two options:

Custom Variables
If your variables are straight-forward and will only occur once for a given session, then you could add them as variables of the session using the Managing Custom Variables feature. The drawback of this method is that other data of this type uploaded by other users may not be handled in the same way, and will be difficult to share down the road. Also, XNAT will not be able to perform an validation on your variables. However, if that is not a concern, this is the easiest method for handling your custom variables. When using this method, you can upload your physical files to the additional resources level of your session.

Custom Schema
If your measures are more complex (include unboundeds or could occur multiple times for a session), you want to enforce some validation, or you want to enable sharing with other data sets of this type, then you should build a custom schema to represent your data as image assessors. More... After registering your data types with XNAT, you will need to build XML (according to your schema) and upload it, and the files using the REST API.

Processed data w/o derived measures

Processed data w/o derived measures are a bit easier to upload. If your processed data relates to one and only one scan, then you can upload it as seperate resource with your scan (PUT /REST/experiments/X/scans/1/resources/SOMETHING/files...). If your processed data relates to more then one scan, then you should upload it as a session reconstruction (PUT /REST/experiments/X/scans/1/reconstructions/SOME_UNIQUE_ID/resources/SOMETHING/files...).

Anonymization

Anonymization is a very important step when uploading image data. The XNAT team encourages installations to prevent the inclusion of any PHI in your data. To facilitate this, we would encourage you to develop a plan for how PHI will be removed from your data.

DICOM

DICOM Browser is built with a specific eye towards anonymization. It gives users a user interface for modifying header values in large amounts of DICOM data easily. Also, it provides a scripting engine for the anonymization of data from the command line or within XNAT's DICOM Upload Applet.