How to Replicate Survey Reporter Significance Tests

Survey Reporter is an analytics software that has been around for a long time. Many customers and their clients who want to transition to more modern software may want to replicate the significance testing for legacy analyses.

Method - Configure Stat Testing in Displayr

1. Select a table and go to Properties .
2. Go to Appearance > Appearance > Significance > Advanced.
3. Make the following changes in Statistical Assumptions settings:
   • Change Show significance to Compare columns.
   • Change both Multiple comparison correction settings in Exception Tests and Column Comparisons to None.
   • Set the Minimal sample size for testing in the Significance Levels tab to its level in Survey Reporter (which defaults to 30).
   • Change the Proportions setting in the Test Type tab to Survey Reporter Proportions.
   • Change the Means setting in the Test Type tab to Survey Reporter Means.
   • Change Weights and significance in the Significance Levels tab to Kish's approximation.
   • Uncheck Recycle column letters in the Column Comparisons tab.
   • Replace the No test symbol in the Column Comparisons tab with **.
   • In the Significance levels and appearance grid on the Significance Levels tab, copy the one uppercase Column letters fields (i.e., A,B,C,...,) and paste it over the top of each of the lowercase fields (i.e., a,b,c,...,). Note that this will remove Displayr's default of using uppercase and lowercase to denote different levels of significance. However, it will not precisely replicate Survey Reporter's use of column letters, as Displayr assigns a letter to NET fields but Survey Reporter does not. Two ways of resolving this are to:
     • Remove the NET categories from within Spans.
     • Manually change the contents of Column letters to exactly replace Survey Reporter if so required, but this will need to be redone for each question used in the columns of tables.
     • Where a Binary - Multi question is in the columns of a table or analyzing a Numeric - Grid, Binary - Multi Grid, or Nominal - Multi SUMMARY tables set Overlaps to Dependent.

Method - To Look into Differences in Results

1. Move the NET categories to be at the beginning top/left of the tables (and sections within a table where Spans are used).
2. Ensure that the base used in statistical comparisons is equivalent (e.g., on a Binary - Multi Grid, Survey Reporter's base is the total number of responses, whereas Displayr's is the total number of respondents.
3. Ensure that the rounding is being computed in a way consistent with Displayr (Displayr rounds up; by default, Survey Reporter rounds to even).
4. Check that the percentages and means being compared in the statistical tests are the same. Where they differ this likely relates to the base. Sometimes Filters can be used to get Displayr to replicate Survey Reporter's percentages (however, keep in mind that this is something that is only done to aid in comparison rather than being a desirable approach to compute percentages).
5. Ensure that any filters are equivalent (including Survey Reporter's Interview Filter settings).

Next

How to Apply Significance Testing in Displayr

How to Replicate Quantum Significance Tests