Create a Separate Data Preparation Document

Separate data preparation documents are used to remove the heavy data cleaning, merging, and other processing out of your actual reporting document. Once the data is prepared and exported, you can Update the document with your report with the new data set. This workflow alleviates some of the load on the reporting document causing it to run faster. There are many cases for using a data preparation document. It can be used to:

• Save your data set into a more compressed format (a leaner .sav or .QDat).
• Automatically merge live survey data from an integration/API data source (such as Qualtrics or Forsta) into a historical data file - this is especially handy for tracking studies. 
• Change the shape of your data, such as with stacking or making data wide.
• Merge together multiple data sources, using the Combine > By case and By variable widgets.
• Removing unneeded variables and cases.
• Create lots of constructed variables that do not need to be updated in real-time in the report, such as top 2 box scores, text categorizations, segmentation membership, etc. Variables created will only have their values, not formulas, included in the exported data file. So you should not include dynamic filters (that use a combo box or control) and Banners in your data preparation document, see Additional notes for more detail. 

The general setup for this workflow can be grouped into two higher-level steps:

1. Method - Creating the data preparation document
2. Method - Updating Your Report Document

Requirements

• A data set of raw observation data.
• If setting any data up to export automatically, you'll need Server time for private dashboards and API hours (view hours) on your account.
• Familiarity with User Group and Displayr Cloud Drive permissions. See How to Manage User Groups and Permissions in Displayr and How to Use the Displayr Cloud Drive for more details.

Method - Creating the data preparation document

The data preparation document is separate from the document containing the final report, as mentioned above. In it, you can do data cleaning, configuring, merging, and stacking for your final report. The purpose of the data preparation document is to export a more efficient data file to the Displayr Cloud Drive. 

1. Decide if you would like to start with the raw data and edit from there, or take the Data Set you've created (with constructed variables) and whittle down your report to what you want to include in the data preparation.
   • Starting from the raw data - create a New > Document from your Documents page, append something like Data Preparation to the report name, and import the raw data used for your report.
   • Starting from your current data set - Duplicate the document from your Documents page (via the ellipses button when hovering over the document's name), appending something like Data Preparation to the name.
2. In the data preparation document, edit the data to add in constructed variables that you'd like hardcoded in the report, hide/delete unneeded variables/cases, and delete any banner or variables that need to be recalculated in real-time in the report (if working from a duplicated document).

3. Save to your cloud drive for the most efficient workflow. Click Share > Export Data > To Displayr Cloud Drive as Q Data File (.QDat) (the leanest format type, as it was created specifically for Q/Displayr), or you can choose To Displayr Cloud Drive as SPSS File (.SAV) if you are using it in the merging or stacking widget later in the workflow. 

   Note: Ensure you save your file to an appropriate Displayr Cloud Drive location with appropriate permissions. If the relevant files are located within a Displayr Cloud Drive folder with configured access, ensure your document also has access to this folder by including the user groups that can access it in either the permissions of the document's folder or the document itself. See How to Manage User Groups and Permissions in Displayr and How to Use the Displayr Cloud Drive for more details.

4. On the next screen, select an option below, then click OK.
   • If you'd like to set up automatic exporting, select Now and whenever document is published.
   • To export manually whenever you want, select Just now.
5. Give the exported file a name and click OK. If saving to a Cloud Drive folder, additionally select the appropriate folder location (if applicable), and just click Save.
6. [OPTIONAL]: Set up the automatic updating of this data set by scheduling the document to be republished on a schedule (it doesn't matter that it's not been published to a dashboard yet). See How to Automatically Republish a Dashboard for how to set up the Scheduler to do this.
7. [OPTIONAL]: Perform any merging or stacking that you require using the exported file from your Displayr cloud drive. If desired, use the Automatic updating sections in their Properties to set up a schedule for exporting the merged/stacked data to the cloud (note this will happen whether or not the input data sets have been changed).

Method - Updating your report document

Now that you've used the data prep document to do the heavy data lifting, you will end up with a new data file that was exported to the Displayr Cloud drive. You will then need to update your final report document with this updated data set from the cloud.

1. In the document used for the final reporting, select your existing data set in Data Sources.
2. Press Update in Properties to update it with a new version.
3. Click Displayr Cloud Drive and select the appropriate folder (if applicable) and file from your cloud drive that you created in your data preparation document.
4. Review any warnings or errors that pop up in the document.
5. Repeat steps 1-4 for any other data sets you'd like to update with a file from your data preparation document.
6. OPTIONAL: Once you've linked the data set in your final report document to the appropriate file on the Displayr Cloud Drive, you can also set up automatic updates for this using a Schedule, if desired; see How to Set Up Schedules in Displayr. Keep in mind that if you are also automatically updating and exporting from your data preparation document, you will need to allow enough time for that to complete before scheduling your final report to update.

Additional notes

As mentioned, you shouldn't add dynamic filtering variables and Banners to your data preparation document. This is because all data is essentially hardcoded :

• Filter variables that depend on user selections in Combo Boxes or other Controls on the page, as they need to be recalculated when the control selection changes.
• Constructed Banners created by our drag-and-drop tool, as they use a dynamic interface and have extra nets, spans, and other structures integrated into them.

Additionally, changes to the Data Reduction of a variable set that you have changed on a table directly, such as combining categories, creating NETs, removing categories, reordering rows, etc., will not pass through to the saved file.

Next

Reduce the Number of Unnecessary Variables and Variable Sets

How to Automatically Update Dashboards in Displayr

Tracking Study Best Practices

How to Set Up Schedules in Displayr