This article describes how you can compare between the different elements (e.g., brands or statements) of a categorical array ("grid") variable using both the web app and Excel exports:
- In the web app, you choose a reference column to which the others are all compared.
- In Excel export, you can get the ABCD letters for all possible comparisons between columns.
In both cases, the statistical tests and assumptions are the same (between a reference column or between every pair of columns). For more information on statistical testing in Crunch, please see Crunch's introductory article.
How to highlight significant differences within a categorical array using the web app
Consider the following categorical array below, which is a series of statements the respondents must agree or disagree with on a scale from “Strongly disagree” to “Strongly agree”:
We may want to look at the differences between the statements and which differences are large enough to be considered statistically significant.
To get started, the variables must be in a categorical array (shown in the app image above). Then:
- Show the variable in the Tables and Graphs mode:
- To compare between statements, swap the rows/columns so that the statements are the columns and their categories are the rows:
- Hover the mouse under a particular column to Set Comparison.
The column turns grey and then the other cells are highlighted if the figures are deemed significant when compared to the grey column. Again, this shading is on a scale of significance (see the p-value scale in the bottom left):
In the above example, “I believe in eating only organic foods” is set as the basis of comparison. The 34.5% figure (the subtotal figure) compares to all the other figures in that row. We can see that “I am worried about the future” is significantly higher (green) with 73.8% and “I am conscious of my weight” is significantly lower (with 17.5%). Notice that the subtotal row (“All agree”) and the mean scores are being compared to the grey column along with all the individual categories on the scale.
You can set other columns to be the reference column and then the comparisons are made to that column, by toggling that column with Set Comparisons that appears under each column:
Unlike tab books, the web app doesn’t show all the letters at once. This is because the web app provides a clean, focused, and interactive environment, where you can flexibly change the columns to highlight comparisons to that specific column. If you want column letters you see in traditional tab books, you can achieve that with exports from the deck as discussed in the following section.
How to achieve column comparisons (ABCD letters) using Excel exports
- In Tables and Graphs mode, make sure the table is oriented with the variables along the top and the scale down the rows.
- Save your table to the deck.
- Go to Edit properties of the deck slide:
- In Export settings, go to Custom and then ensure that the column t-tests are switched on (checked):
- Export to Excel (click the button at the bottom of the Deck).
Frequently asked questions:
How is the significance computed?
- Crunch computes a margin of error for each element (e.g., brands as columns) in the categorical array.
- Table cells are shaded where the confidence intervals do not overlap.
How is the overlap in respondents treated?
- The ‘overlap’ in this context is whether a respondent provides a response on both variables (statements, brands). In some studies, the same respondents give answers to some or all of several variables. In theory, those people’s responses would tell us more about the differences between the variables than if the variables were asked of entirely different people. The most ‘conservative’ (least likely to detect a statistically significant difference) choice is to treat variables as if they were asked of independent groups, even if the groups are not (fully or partially) ‘independent’. Adjustments for ‘overlap’ attempt to balance statistical conservatism and the knowledge that some respondents may have responded to some of the same questions.
- The confidence intervals are calculated separately for each percentage. That is, they are treated as independent variables.