Skip to main content

Upload data programmatically

Programmatic uploads to the Data Platform use capabilities provided by the underlying Flywheel system. There are two options:

  • use the Flywheel CLI tool, or
  • use a Flywheel provided SDK, available for both Python and MatLab.

Flywheel CLI​

The first option is to use software that is already built and can be used to perform transfers from a computer using the command line. Flywheel has two CLI options, but only one currently allows file transfers from a local computer: the Classic Flywheel CLI tool. (If you are transferring files to cloud storage such as AWS S3, the Beta Flywheel CLI tool is the better option.)

We recommend using the CLI if your goal is to upload files from the command line. The tool can be downloaded from the Classic Flywheel CLI tool page and installed locally.

Once you have the CLI running, the command

fw ls

will list the Flywheel groups that you can access. Most people will only see one, which is for their center. (If it looks weird, please ask.)

To see which pipeline projects you can access, use the command fw ls followed by the center group name. For instance, the command

fw ls sample-center

shows the projects in the group labeled sample-center. Use the label for your center instead and you will see the projects you can access.

The pipelines that allow upload start with either ingest- or sandbox-. Use the sandbox- pipeline project while you are figuring things out, and when you are ready to submit actual data use the ingest- project. The pipelines are specific to data types:

  • If you are authorized to upload research form data you will see ingest-form and sandbox-form.
  • If you are authorized to upload enrollment form data you will see ingest-enrollment and sandbox-enrollment.

If you don't have any of these pipeline projects, you have not been authorized by the center administrator.

To upload a file to sandbox-form, you need the group label from running fw ls. We will use sample-center as the group. The command

fw upload dummy_data_uds.csv fw://sample-center/sandbox-form

will upload the file dummy_data_uds.csv in the current working directory.

Note: File name should have _[module].csv suffix. Find the list of file name suffixes for currently supported modules here.

Flywheel SDK​

The second option is limited to scenarios where a center wants to transfer data directly from their data systems without direct human involvement. Building this integration will require someone who is able to build the software that performs the transfer.

For more details on how to build integrations between systems, see the NACC Data Platform Demos.