Smart File Uploads are a simple way to get your data in to Crunch by auto-detecting the type of data file(s) you are loading and customizing the import workflow based on the type of data.

It supports SPSS files as well as Parquet files (coupled with corresponding schema/metadata files). See below for details about how each data type is handled and how it differs from the Legacy File Uploads.

You can find the Smart File Upload as part of the refreshed New Dataset flow:

Note: Smart File Uploads are currently being rolled out to customers in a phased manner - you may not see the Smart File Upload menu item. Contact Support if you are not seeing this new experience. Your view of the Import data workflow may differ based on your subscription.

SPSS Files

SPSS (otherwise known as .sav) files using the Smart File Upload are processed in the following way:

• All variable names are converted to lower case variables. 
  SPSS supports case insensitive variable names, and data processing routinely encounters problems because SPSS files inadvertently change variable names E.G. 'Q1' becomes 'q1'. The smart file upload normalizes all variables to the latter form.
• Automation scripts are auto-generated to reduce the data processing workload. The following scripts may be generated depending on the content of the SPSS file:
  •  Categorical Grids
    • Often, SPSS files contain multiple categorical variables that share the same leading or trailing label and are part of a Grid (or Matrix) question. These variables are detected and Automation scripts are generated to automatically create Arrays from this raw data.
  • Multiple Response Data
    • Generated Grids
      • Similar to Categorical Grids, groups of Multiple Response variables are turned in to array variables so they can be analysed together.
    • Dashboards
      • Smart File Upload detects when multiple response variables are part of a Grid/Matrix question and creates Dashboards for the transposed view of the grid to ease analysis.
    • Missing Data Recoding
      • Variables in MR Sets are automatically detected as Non-Dichotomous format (No nulls) and are re-coded into Dichotomous format.

All generated automation scripts are available in the Automation menu as a Generated Scripts tab. E.G.

The user can choose which of the generated scripts to apply, edit the scripts for correctness. Use the Copy button and paste in to the Scripts tab and edit accordingly before applying.

This is an optional step - the generated scripts can be ignored if they are not desired.

Parquet Files

Smart File Uploads accept long format parquet files. Every Parquet file must be accompanied by corresponding JSON schema file. Metadata JSON files are optional but helpful in making the data more readable. for details about the JSON files structure see Crunch Logical Schema.

For details about how to structure data in parquet files, refer to Long Format Data Model.

Long format parquet files bring many benefits and efficiencies to data pipelines - a consistent data file schema allows for simpler and faster processing.