It's not supposed to happen... but it can. Here's how to fix it.
Sometimes you go to download data from a subject, and it's not there. I don't mean the series are missing from the NiDB website, but the data is actually missing from the disk.
This can happen for a lot of reasons, usually because studies are moved from one subject to another before they are completely archived. Also for the following reasons
Subjects are merged, but data is not completely copied over on disk
Subject ID is incorrectly entered on the MR scanner console. This causes a new ID to be generated. If the study is later moved to the correct ID, some data might not be moved over on disk
A subject is deleted. But since data is never really deleted from NiDB, it's possible that a study was moved to that subject and not all data on disk is copied over
Suppose we have subject S1234ABC. This subject has one study, and ten series in that study. We'd expect to see the following on the website for subject S1234BC study 2.
But, we go to export the data through the search page or through a pipeline, and not all of the series have data! If we look on the disk, we see there are series missing.
That's not good. This could also appear as though all series directories do exist, but if we dig deeper, we find that the dicom directory for each series is missing or empty. So, where's the data? We have to do some detective work.
Let's look around the subject's directory on the disk.
That's interesting, there appears to be another directory. Our study is 2, but there's also a study 1, and it doesn't show up on the NiDB website. Maybe our data is in there? Let's look.
That looks like our data! We can verify by doing a diff between directories that exist in both studies.
If this is the data we are looking for, we can copy all of the data from study 1 to study 2.
After the copying is done, you should be able to go back to the study page, and click the View file list button at the bottom and see all of expected series.
Tutorial on how to import DICOM data into NiDB
There are two main methods to import DICOM data into NiDB
(a) Global Import - used by the DICOM receiver. All files go into the same directory to be archived completely unattended. Filenames must be unique. Preferable if you have a large batch of disorganized DICOM data
(b) Individual Import - import a single zip file or directory. The import can contain subdirectories. This will parse and display the contents of the import and wait until you select which series to archive before any data will be imported. Preferable if you have smaller batches of data, or data that must be parsed differently than the default global method
DICOM files are parsed into subject/study/series groups using 3 DICOM header tags (or set of tags). These are parsed in order.
Subject - PatientID (0010,0020) - this uniquely identifies the subject. PatientID will match to existing subjects in the database (regardless of project enrollment) by comparing the UID, and alternate UID fields in NiDB
Study - Modality (0008,0020) & StudyDate (0008,0020) & StudyTime (0008,0030) - this set of tags uniquely identifies the study. This will match to existing studies within NiDB. Those existing studies must also be associated with the Subject from the previous step.
Series - SeriesNumber (0020,0011) - this uniquely identifies the series. This will match to existing series in NiDB based on this series number, as well as the study and subject from the previous step.
Check your configuration (Admin-->Settings-->NiDB Config) for the incomingdir variable. It will most likely be /nidb/data/dicomincoming. This will be the directory NiDB will search every minute for new data, which will then be automatically parsed and archived.
From a Linux terminal on the NiDB server, run the following commands as the nidb user to find and copy all dicom files.
You can also move files, instead of copying, by replacing cp with mv. If your files have a different extension, such as .IMG, or no extension, you can change that in the command as well.
Go to Admin-->Modules-->import-->View Logs to view the log files generated by the import process. The global import process expects to process a stream of data, where there is no beginning and no end, so the log file will not delineate this particular import from any other data that were found and archived. This import method is also designed to take a stream of potentially random data, and only utilize readable DICOM files.
The global import method will only archive readable DICOM files. Any unreadable or non-DICOM files will be moved to the /nidb/data/problem directory.
Here's a sample section of import log file. Log files can be very detailed, but any errors will show up here. Problems
You may ask... where's my data? You can search, on the Search page, by ID, dates, protocol, and other information.
The global import method will group files by the method specified above. If one of those fields are blank for some or all of your data, that could cause the archiving process to create a subject/study/series hierarchy that does not match what you are expecting. Sometimes you will find that each series is placed in it's own study. Or each study is placed in a unique subject.
To troubleshoot these issues, try using the individual import method described below. This allows you to select different matching criteria and preview the data found before archiving it.
This tutorial is based on the Importing data section of the User's guide, but the content on the current page is more detailed. See link to the user's guide:
Go to Data-->Import Imaging. Click the New Import button.
Fill out the required information. Choose if you are uploading a file, or if the data is located in an NFS path. Select the modality and project. Then select the matching criteria, which will determine how the data will structured into a subject/study/series hierarchy. When everything is all set, click Upload.
Check the status of the import by going to Data-->Import Imaging and finding the import that was just created. The current import step will be displayed, and you can click View Import to view more details. Details of the import will be displayed.
If the import has finished parsing, it will ask for your attention. You'll need to review the subjects, studies, and series that were found and then select which series you want to archive.
How to upload data into a sample project
A project will often need imaging data of different modalities uploaded to an instance of NiDB. All of the data must be associated with the correct subject, and each modality must have it's own study.
Follow this order of operations when uploading data
Create the subject(s) - Subjects must exist in NiDB and be enrolled in a project before uploading any imaging data
Upload EEG and ET data before MRI data - MRI data is automatically sorted into subject/session during import which is different than how EEG and ET are imported. Uploading the EEG and ET first makes sure that all of the subjects and associated IDs exist before attempting to upload the MRI data
Upload small MRI imaging sessions (less than 1GB in size) using the NiDB website - This is useful to upload data for a single subject.
Upload large MRI imaging sessions (greater than 1GB in size, or dozens of subjects), or data that must be anonymized, using the NiDBUploader - This is useful if you need to upload thousands of MRI files. Sometimes a single session might generate 10,000 files, and maybe you have 20 subjects. Might be easier to use the NiDBUploader.
Make sure you have permissions to the instance and project into which you are uploading data.
Select the correct instance.
On the top menu click Subjects --> Create Subject
Fill out the information. First name, Last name, Sex, Date of Birth are required
Click Add and confirm on the next screen. The subject is now created
On the subject's page, select a project from the Enroll in Project dropdown (you might need to scroll down in the dropdown), and click Enroll.
On the subject's page, click the Edit Subject button on the left.
In the IDs section, enter the extra ID(s) in the specific project line. Separate more than one ID with commas, and put a * next to the primary ID. Such as *423, P523, 3543
Click Update and confirm at the next screen.
If demographic data are stored in a redcap (or other) database and NiDB is storing imaging data, make sure to put each ID in each database. In other words, put the redcap ID into the the NiDB ID field and store the S1234ABC ID in redcap.
On the subject's page, find the Create New Imaging Studies dropdown. Expand that and find the New empty study dropdown. Select the ET or EEG modality, and click Create. Your study will be created.
On the subject's page, click the Study number that was just created, and it will show the study.
Fill out the protocol name, the date/time of the series, and any notes, then click Create series.
Drag and drop your file(s) onto the Upload button. The status of the upload will be shown below the button.
Don't click Refresh or press ctrl+R to reload the page. Instead, click the Study n link at the top of the page.
If you need to rename or delete files, click the Manage N file(s) button on the study page.
Upload the data
On the top menu, click Data. Then click the Import imaging button.
Click the New Import button.
Choose the files you want to upload. These can indivudual files, or zip files containing the DICOM or par/rec files.
Data modality should be Automatically detect.
Select the destination project
Leave the other matching criteria as the defaults
Click Upload.
A new row will be created with your upload. MRI data can contain just about anything, so NiDB needs to read through all the files and see what's there.
Once NiDB has parsed the data you uploaded, you'll need to decide which data to actually import.
Click the yellow Choose Data to Import button
Details about the import will be displayed. On the bottom will be a list of subjects, studies, and series. You can deselect certain series if you don't want to import them, but likely you'll just want to import all of the series, so click the Archive button.
Click on the Back button on the page to go back to the import list.
Refresh this page and eventually your import should change to a status of Archived.
Download the NiDBUploader from github: https://github.com/gbook/nidbuploader/releases
Install it and run the program.
Create a connection
Fill in the server: https://yourserver.com, and enter your username/password. Click Add Connection.
Click on the connection and click Test Connection. It should say Welcome to NiDB after a little while.
Select the data
Select a Data Directory at the top of the program. This should be the parent directory of your data.
Change the Modality to MR. Uncheck the Calculate MD5 hash...
Click Search. This will slowly populate that list with DICOM/ParRec files that it finds.
Once it is done loading files, you can select multiple files and click Remove Selected if you need to.
Set anonymization options
Make sure Replace PatientName is checked.
Set the destination
Click the ... button for the Instance, which will populate the list of instances. Select your instance. Then select the Project.
Click the ... for the Site and Equipment to load the lists. Select the Site and Equipment.
Upload the data
Click Upload.
It will take a while. Like a long time. Be prepared for that. Depending on the number of files, it could take hours to upload.
If any files fail, it will be displayed along with a reason. If you fix the errors, then you can click Resend Failed Objects.
MR scans quality parameters
MR scan parameters defining the quality of a MR scan are displayed on this page. The information is available for each NiDB project. This MR scan quality information page can be accessed from project's main page by clicking on the "MR Scan QC" sub-menu as shown below.
The MR scan quality control page shows the following information for each MR scan stored in the corresponding project. These parameters includes:
Series Desc. : Description of the MR scan
Params good? : Green if parameters are within range. Yellow if criteria is not defined and Red if parameters are out of specified range
Files on disk? : Green if files are found on the specifies location on the disc. Red if the files are found on the specified location.
Num Files : Actual number of files
Avg Rating : Average ratings
Basic QC : Red if parameters are out of specification, yellow if limit is not defined and green for within range
Disp (X,Y,Z) : X, Y and Z displacement value
Mot(X,Y,Z) : Motion in X, Y and Z direction.
Rot(P,R,Y) : Rotation pitch, roll and yaw values.
SNR : Signal to noise ratio
FD : Frame-wise Displacement
DVARS : Root mean square of temporal change of the voxel-wise signal at each time point. (D is for temporal derivative of timecourses, VARS refers to RMS variance over voxels.)
The QC scan quality control table can be download as ".csv" file by clicking "Save As CSV" button at the end of the MR QC table.
