Error Checking Process
NACC Data Validation Pipeline
Submitting a form data CSV file to center's Flywheel ingest project will trigger the NACC data validation pipeline in Flywheel.
For the centers who are using REDCap direct entry, the records that are marked as "ready for Flywheel upload" will be transferred to the center's Flywheel ingest project each night by NACC, and the same validation pipeline will be triggered.
The stages of the validation pipeline are
-
CSV format check:
Check whether the submitted CSV has correct headers and datatypes. The CSV should include only NACC accepted variables for the module, entire file will be rejected if there are extra fields. -
NACCID lookup:
For each record in CSV file, look up the NACCID using the adcid and ptid for that record. Record will not be processed further if there's no matching NACCID. -
Generate JSON file:
Create a JSON file per CSV record and store in Flywheel ingest project as aquisition data. A Flywheel hierarchy (Subject/Session/Aquisition) will be created for the file if it doesn't already exist. -
Extract metadata:
Extract information from the JSON files and store as Flywheel metadata, these will be used for search, dataviews, reports, etc. -
NACC QC checks:
Run NACC data quality checks according to NACC published error checks.
Each stage of the validation pipeline will trigger a job in Flywheel.
Check the job completion status in the Jobs Log.
Error Report
Once all the jobs in the pipeline are complete, you will be able to view the error report in the Flywheel Issue Manager interface.
The Issue Manager UI will display all the errors and alerts that are currently unresolved in the project. It has a tab for each stage of the data validation pipeline. If there are no errors occurred in a certain stage, that tab will be empty.
The Issue Manager UI supports sort, search, and download (as CSV) functionalities for the error report.
