APIs enable your survey platform to directly communicate with Crunch so that your data automatically syncs with Crunch, without the need for downloading and uploading datafiles (such as SPSS and .csv).
For some platforms, such as SurveyMonkey and Decipher, Crunch is already set up to ingest your data. But if you are collecting data in your own data collection platform, then you need to set up the connection.
Getting data into Crunch
There are two ways to get data into Crunch:
- You can set up your platform so it pushes data into Crunch. In programmer-speak, the data is the payload and Crunch is the target.
- The alternative is that Crunch pulls data from your data collection platform.
Generally, proprietary data collection platforms are not built to facilitate another system pulling information from it. Commercial platforms such as SurveyMonkey and Decipher, on the other hand, are designed with this in mind (and so Crunch works well in pulling data from them). For your internal data collection platform, it is more likely you will want to push data into Crunch.
Setting up the API process
As every data collection platform is different, a program needs to translate the information from one platform so it can be read by the other. This process is called data mapping. Data mapping happens at your end before the information is transmitted to Crunch. The program that is written by the programmer gathers up the required data and prepares it for transmission to Crunch (which is typically contained within a JSON file). The preparation of data includes putting the information into the right format (called schema) and correcting/adding appropriate metadata (which is information about the data, such as variable labels, and so on).
API best practices
Here are a few upfront tips and considerations relating to setting up a direct API that interacts with Crunch:
- Test via incremental trial-and-error — The setup process isn’t instant and it is perfectly normal to go through several iterations of the process, where variables are incrementally pushed into Crunch, to see how it works before proceeding further. This trial-and-error approach also gives the programmer an opportunity to better understand how our API works (as some APIs are harder than others to understand—but hopefully it won't prove to be too difficult). The programmer will be pleased to know we use REST as an API protocol (which is a standard for web-based data transmission and offers good flexibility).
- Avoid shortcuts by piggybacking on existing output methods — For example, a programmer uses existing methods of extracting data from your system, such as producing an SPSS or CSV output, and then transforms that datafile for API transmission. Even though it involves more setup time, it is far more robust to set up a direct data connection (i.e., with the data mapping) without an intermediary format.
- Check your variables (e.g., base sizes) early on in the setup process to detect any issues — Your programmer may not be as familiar with survey-related concepts such as the role of missing data. For example, with multiple-response questions it’s important to distinguish between a non-response and survey skip. A non-response is when a respondent is shown an option, but they don’t check the box. This is ideally recorded for that respondent (row) as a zero (0). On the other hand, a survey skip, should be encoded as a missing data (i.e., blank, or sometimes referred to as null).