This article walks through steps on what to look into when you have an issue with a weight in your document. This could be an error or simply the weight is not weighting targets as you expect. Work through the steps below to uncover what is going on.
1. Gather the details about the issue
2. Find what is causing the issue
3. Figure out a workaround or a solution
Requirements
- Familiarity with How to Configure a Weight from Variable(s).
1. Gather the details about the issue
You will use these details and information to help in the troubleshooting steps below.
-
- If the weight was created in Displayr, do you see any error messages in the bottom right of the weighting tool? We have an article that discusses particular weight errors and how to troubleshoot them here: Common Errors in Weights.
- Apply the weight to a table of each adjustment variable used to build the weight to see whether the target values are what you expect. Note any issues.
- What documentation and help information is available about the tool(s) or issue? Consult How to Configure a Weight from Variable(s) and How To Apply Different Weighting Structures To Sub-Samples. These are two useful documents that walk through what the various fields can do and common techniques for creating complex weights. You can search the Displayr Help Center, Displayr wiki, and Displayr blog/website for any other relevant material.
- If comparing a weighted table from Displayr to one created outside of Displayr, you will need to gather the underlying information about the data used to create both tables. This includes the weighted and unweighted values for: Counts and the relevant Sample Sizes. Do note, Displayr rounds to 13 significant digits (not decimals) and “away from zero”.
- If comparing respondent-level weight variable values created in Displayr vs those created outside, know that these likely will not match due to various algorithms and tools used to create respondent-level weighting values. This is expected.
2. Find what is causing the issue
What aspects of the issue do you have control over (i.e. structure of the table, order of rules, user permissions, etc.) that you can tweak and test to flag as the root cause of the problem? Once you figure out what is causing the issue, then you can go to step 3. Figure out a workaround or a solution.
- Confirm your current inputs and process match the documentation (if there is any available).
- Confirm the weight variable is checked Usable as Weight in the object inspector.
- Confirm any tables in question have the correct weight variable selected in the Properties > FILTERS & WEIGHT > Weight dropdown.
- Confirm any settings in the weighting tool are used appropriately according to the documentation. Three fields to double check specifically are what it means to use: Recompute weights for, Making weights sum to, and Design Weights. For more information see: How to Configure a Weight from Variable(s)
- Confirm the structure of the adjustment variables are single Nominal variables and any recompute variables are Nominal or Binary - Multi variables.
- Check that the categories in the Value Attributes are what you expect to be using. For example, when using a variable to recompute the weighting, all respondents in categories - even respondents with missing data for that variable - will be recomputed. Remember you can merge together categories within the tool by dragging and dropping or using the steps here: How to Create a New Variable with Merged/Combined Categories.
- If using a combined weight variable (a weight created using R code or JS code to stitch together weight values from individual weight variables as-in here), ensure the correct groups are being used in the R conditional statements and those are paired with the appropriate sub-sample or individual weight variable. You can hover over the sub-sample variable name to see the categories used to confirm. Keep in mind if you want to leave any respondent unweighted in a combined weight variable, they will be given a weight of 1. To exclude respondents from the final weight you will give them a value of 0. Ensure the last line of code is the final variable name, this will return the full lists of weights for the variable.
- If applying a weight to a custom Calculation, remember that weights only can be applied to raw data values - not tables referenced in the code. Weights should be applied to input tables referenced by the code on the table directly. If using raw data in the Calculation, you can adjust it by multiplying it by the variable QPopulationWeight (which will be the respondent-level weight values of the weight selected in the Weight dropdown on the calculation), see How to Perform Mathematical Calculations Using R for more info.
- Test aspects you have control over to see what triggers the issue.
- If an issue with values shown on a table using rules. Remove all rules from the table to see if that fixes the issue. If it does, add back the rules one at a time to see which one triggers the problem.
- If using a combined weight variable (see explanation in 2.1.f above), apply the individual weights to the adjustment variable tables to confirm that they are working as expected. If they aren't, troubleshoot potential issues with the weighting tool restarting with 1 above for confirming inputs and processes. Otherwise, if the individual weights are fine, use How to Troubleshoot R Code in Displayr to figure out why the individual weights aren't getting put together correctly in the R code of the combined variable.
- If using a weight created outside Displayr, compare the unweighted Counts and Sample sizes of the tables that aren't matching to confirm the data used to create the tables is the same. Then, apply the weight and compare the weighted statistics to identify a particular category that is off. Note that you should look at the Weighted Count and Weighted Sample Size statistics. Then compare the individual weight values for respondents in that category to confirm the values of their weight in Displayr vs the outside program.
- For weights created inside Displayr, try changing the field Recompute weights for (if using), Making weights sum to, removing any maximums and minimums, and changing the algorithm back to the default Raking algorithm, if you have modified any of those fields.
- If step d doesn't fix anything, rebuild the weight from scratch, using just the first adjustment variable, and any supplementary settings (Recompute weights for and Making weights sum to, maximum and minimum). Create a table of the adjustment variable (crossed by the recompute variable if used) and apply the weight to confirm that the targets are as expected. If not, then start changing supplementary settings back to their default one at a time until you get what you expect.
- If things look fine after step e, continue adding adjustment variables and checking the adjustment variables tables to confirm their targets one at a time. Eventually adding an adjustment variable will trigger the issue.
- Once you find the adjustment variable that creates the issue, create crosstabs of it and the other adjustment variables to see if any cells have very small sample sizes. This can affect how the raking algorithm converges and may be seen in the rounding of aggregated numbers in summary tables.
- Use what you've learned and confirmed to identify what causes the trigger to create the issue.
- Are any of the adjustment variables modified or different in a way that may affect the weight?
- Do you need to reorder or remove any rules applied to the table? Do read the technical documentation of the relevant rules to ensure they are appropriate to apply or note any caveats regarding the weights.
- Are the settings appropriate given the limitations of your data? If dealing with small sample sizes for the intersection of some targets (found in f above) having min and max restrictions on weighting values may cause the algorithm to not converge.
3. Figure out a workaround or a solution
If the steps above don't lead to an obvious solution, use what you've learned to work toward a solution.
- If dealing with small sample sizes, can you merge categories or remove any adjustment variables to allow the weighting algorithm to converge? Can you weight a portion of your data differently to account for the small sample? (see How To Apply Different Weighting Structures To Sub-Samples)
- Is the setting or issue that caused the issue required or can you work without it?
4. Contact Support
If you've worked through these steps and haven't found a solution to the issue or need help in making the solution, please contact our support team at support@displayr.com or by clicking on Help > Contact support from within your specific document in Displayr. Please provide the details of what you learned in the steps above.