File-Based Data Processing Workflow🔗

Goal: Maintain an up-to-date mapping of identifiers for de-identification workflows.
Contacts: Reed McEwan, Dan Duhon
Frequency: Daily (<1 hour)
Inputs: N/A
Outputs: s3://midb-hbcd-main-pr-deidentification-list/release_identifiers.csv
Notes: Phantom data may not yet be included.

Goal: De-identify and upload raw BIDS sessions

Criteria for De-identification:

• Subject is listed in the release ID mapping
• No existing session files in the de-ID bucket
• Session files are available in the LORIS bucket and are ≥1 day old

De-identification Procedure Overview:

• De-identify and upload all supported session files to the de-ID bucket
• Update session metadata (sessions.<tsv|json>) in de-ID bucket
• Tag each file with its loris-versionid (corresponds to VersionId in original LORIS files) for traceability

Contacts: Sriharshitha Anuganti, Erik Lee
Frequency: Daily (PM CST; ensure completion within 24 hours)
Inputs: s3://midb-hbcd-main-pr/assembly_bids (raw BIDS data with DCCIDs)
Outputs: s3://midb-hbcd-main-deid/assembly_bids (with Release Candidate IDs)
Notes: EEG sourcedata files eventlogs.edat3 and eeg_flags.json are not yet supported.

Goal: Make CBRAIN aware of subjects available for processing.
Contacts: Monalisa Bilas, Erik Lee
Frequency: Daily (<1 hour)
Inputs: s3://midb-hbcd-main-deid/assembly_bids
Outputs: Internal CBRAIN records indicating existence of subject folder within BIDS directory
Notes: Each subject has a single CBRAIN BidsSubject File Collection linking all sessions, though sessions are processed independently.

Goal: Run de-identified BIDS data through BIDS App pipelines in CBRAIN (e.g., `Nibabies`, `BIBSNet`, `QSIPrep`) to generate derivatives.

Workflow Steps:

1. Detect available sessions sessions in the BIDS directory and CBRAIN
2. Check for existing outputs or prior processing attempts
3. For sessions without outputs or attempted processing, verify that required prerequisite files exist and pass QC (from scans.tsv)
4. Select files for processing based on modality-specific rules (e.g., best T1w image only, all fMRI images passing QC)
5. Confirm dependencies between pipelines (e.g., BIBSNet outputs are required by Nibabies)
6. Launch CBRAIN processing tasks using predefined settings and including only files selected for processing
7. CBRAIN uploads successful job outputs to session-specific folders on S3 and records the corresponding processing tasks and generated file collections internally.

Contacts: Erik Lee, Monalisa Bilas
Frequency: Daily (initial routine <1 hour; processing jobs may take ~1 day)
Inputs: s3://midb-hbcd-main-deid/assembly_bids (raw BIDS data)
Outputs: s3://midb-hbcd-main-deid/derivatives/ses-{V0X} (session-specific subject folders with Release Candidate IDs)
Notes: See the GitHub repository and Documentation for the code that manages processing. CBRAIN logs and file collections are stored internally for traceability.

Goal: Preserve CBRAIN processing logs for failed tasks before the jobs are deleted (a few weeks after completion). Logs from successful jobs are already archived in the .cbrain logs included with the S3 outputs. Note that CBRAIN only transfers outputs to S3 for successful jobs.
Contacts: Monalisa Bilas, Erik Lee
Frequency: Daily (<1 hour)
Inputs: CBRAIN task directories stored on MSI under /scratch.global
Outputs: s3://midb-hbcd-main-deid/cbrain_std_logs/ (Files named {CBRAIN_Task_ID}.<out|err>.out)
Notes: CBRAIN task IDs are unique, so duplicates pose no issue.

Goal: Detect and remove sessions where LORIS and de-ID data diverge.

Process:
1. Compare file counts between de-id and LORIS session folders
2. If files counts are the same, compare the loris-versionid of the de-id files to ensure they match
3. If session counts or loris-versionid mismatch, delete all associated derivatives, CBRAIN task records, and raw BIDS data. The next time the query scripts are run that look for new subjects to process, the processing will be re-initiated for these subjects.

Contacts: Erik Lee, Monalisa Bilas
Frequency: Daily (runtime varies by data volume)

Inputs:
• Raw BIDS data: s3://midb-hbcd-main-deid/assembly_bids and s3://midb-hbcd-main-pr/assembly_bids
• Derivatives: s3://midb-hbcd-main-deid/derivatives
• CBRAIN records of userfiles and tasks

Outputs: N/A
Notes: Cleanup enables the next de-ID workflow to rerun cleanly for that session.

Goal: Re-identify de-identified derivatives by replacing Release Candidate IDs with DCCIDs, enabling upload to LORIS. Ensures derivatives are accurately linked back to participant DCCIDs to facilitate internal QC with Workgroups and tabulation of derivatives.

Process:

• Download de-identified derivatives from s3://midb-hbcd-main-deid/derivatives.
• Replace all Release Candidate IDs with corresponding DCCIDs in both filenames and file contents (using file type–specific routines) for (1)text-based files (.csv, .html, .json, .txt, .toml, .tsv, .log) and (2) EEG .set files (.set, .mat)
• Replace anonymized site IDs with real site IDs.
• Upload re-identified files to s3://midb-hbcd-main-pr/reid_derivatives and set the metadata field cbrain-timestamp based on the original file’s LastModified value.

Re-identification is performed on derivatives for the following pipelines:

Contacts: Sriharshitha Anuganti, Erik Lee
Frequency: Runs daily
Inputs: s3://midb-hbcd-main-deid/derivatives (de-identified derivatives)
Outputs: s3://midb-hbcd-main-pr/reid_derivatives (re-identified derivatives)

Notes:
• Update re-ID routines whenever pipeline filenames or formats change.
• Previous documentation referenced VersionId metadata; this has been replaced with LastModified since the de-ID bucket is non-versioned.

Goal: Remove re-identified derivatives from LORIS when they become out of sync with corresponding de-identified derivatives.

Process:

• For each subject/session/pipeline, compare LastModified (de-ID) and cbrain-timestamp (re-ID) values between:
  • s3://midb-hbcd-main-deid/derivatives
  • s3://midb-hbcd-main-pr/reid_derivatives
• If the number of files or timestamps differ, delete the corresponding re-identified data from s3://midb-hbcd-main-pr.

Contacts: Sriharshitha Anuganti, Monalisa Bilas, Erik Lee
Frequency: Daily
Inputs: s3://midb-hbcd-main-pr/reid_derivatives and s3://midb-hbcd-main-deid/derivatives
Outputs: N/A
Notes: Ensures only synchronized derivatives remain in LORIS and prevents outdated or mismatched data from being used.