Installation and configuration

How to configure the DHIS2-based WHO Data Quality Tool

How to download and install the app

[Note to the facilitator: this section can best be presented using live online projection of the play instance of DHIS2 (https://play.dhis2.org/2.33.4)..To begin practice with configuration, the table that appears on the “Numerators” page after clicking on More-Administration should have blank values for all numerators for “Data element/indicator” and “Dataset”. After Exercises 1 to 3 have been completed, this table should appear as shown in Annex 1. This guide deals with configuration of only the numerators for the Tool (the first 5 of the 8 tabs of the Administration function). Configuration of the Tool denominators and the Tool external comparisons is discussed in an annex of the companion guide entitled “How to Use the WHO Data Quality Tool.docx” ]

1. Use the App Management application to download and install the latest version of the DQ Tool

   a.From the Home page of DHIS2, someone with Super-User authority should type “app” on the search apps line then click on App Management

   

   b. Click on App Store then, under WHO Data Quality Tool, click on INSTALL V 1.2.3¹

   

   c. From the Home page of DHIS2, type “WHO” on the search apps line and the shortcut for The WHO Data Quality Tool should then appear. Click on the shortcut to launch the app.

   

   Note: At first, the app may not be visible/accessible except to persons who have higher level administrative privileges. Once the DQ Tool has been configured and is ready to be viewed and used more widely, the data manager can change the access requirements so that people with ordinary authorities can access/view the app.  To do this, from the Home page a Super-User should find and click on the “Users” app. Click on “User Role” to launch “User role management”. In the list of “Role”, find and click on the role to which you want to assign the authority (e.g. “User”). From the drop-down menu click on “Edit”. Scroll down to find the “Authorities” section. Scroll down in this section to find and place a check In the box for “WHO Data Quality Tool app”. Click on “Save”. The icon for the WHO Data Quality Tool should now be visible to the user when clicking on the search apps icon on the home page.

How to configure the numerators for the Tool

Anyone with authority to add indicators should be able to configure the app.

Demonstration of how to map each “reference numerator” to a DHIS2 data element or indicator

1. From the Home page of DHIS2, type “WHO” on the search apps line and the shortcut for The WHO Data Quality Tool should then appear. Click on the shortcut to launch the app.

Notice the message that appears the first time that you launch the Tool. Click OK and this message will disappear.

1. Before we begin to configure the tool, let us familiarize ourselves with some of the different pages of the Tool. Each time that you launch the Tool, you will see the main menu at the top. This is used to select between several different functionalities. Notice that the word “Dashboard” is highlighted. The Dashboard function is selected by default.

1. The Dashboard has 4 different pages. You use the tabs near the top to select a page. The Completeness page is shown by default. Notice that this page is blank. The Tool does not display any results until it is configured.

1. The Dashboard also has its own menu. You open and close this menu by clicking on the blue menu icon on the left of the page, near the top.

Later, we will learn how to use this menu to select a Group of data sets, a period and group of organization units (i.e. a geographic region) for review

1. To configure the Tool, click on More from the main menu of the Tool then select Administration

1. At first you will see a message saying that you do not have access to change the configuration. You will see this message, at least briefly, even if you have Super-User access.

1. Be patient and wait a minute or two for the message to disappear and for a table to appear:

1. Notice the tabs that appear in a horizontal row across the top of the table: Numerators; Numerator groups; Numerator relations; Numerator quality parameters; Denominators; Denominator; External data comparison. Part 1 of this guide discusses how to configure the first 4 of these tabs.

2. The 1^st column of this table provides the name of a “Group”. As the term is used here, a “Group” is a set of numerators. Groups are typically for a specific program like HIV/AIDs, Immunization or Maternal health.

3. The 2^nd column of this table lists “Reference numerators”. These are standard indicators which are recommended by WHO. Later we will learn how to customize this list by adding new numerators or ignoring some Reference numerators. Notice that this column lists numerators (e.g. DPT 1 doses) rather than indicators (e.g. DPT 1 coverage). The reason for this is that the Tool assesses numerators separately from denominators.

4. The 3^rd column is headed “Core”. There is a check mark in the cell for each Reference numerator which WHO recommends be included as part of a small (less than 10) multi-program set of indicators to feature on the dashboard of the DQ Tool and in an annual report on the core datasets. Later, we will learn how to modify the list of Core numerators.

5. The 4^th column is headed “Data element/indicator”. This is the column in which we will specify the DHIS2 data element or indicator which corresponds to the Reference numerator. Notice that this column is now blank. To configure the Tool you must identify which country-specific DHIS2 data element or indicator corresponds to each Reference numerator. This process of matching is called “mapping the Reference numerator to a Data element/indicator”.

6. The 5^th column is headed “Dataset”. This too is now blank. As part of the next step, when you map a Reference indicator to a DHIS2 data element or indicator you will also specify which DHIS2 data set the data for this data element or indicator comes from.

7. To understand the process of mapping, click on the Edit button for the Reference numerator “DPT 1”. By default, the first 3 fields for this WHO standard Reference numerator have already been filled in. The box for Core is not checked. However, if you decide to include this indicator in your small core set, you should check this box.

1. Under Data mapping, you leave “Data element” selected if you want to map the Reference numerator to a DHIS2 data element. Alternatively, you may want to map the Reference numerator to an Indicator configured to add together the value of multiple data elements. As an example, suppose that there was one data element for “DPT 1, < 12 months” and a separate data element for “DPT 1, ≥ 12 months”. Instead of choosing one data element or the other you could configure an indicator such as “Total DPT 1” = “DPT 1, < 12 months” + “DPT 1, ≥ 12 months”. If this indicator did not already exist, you would have to exit the Tool then use the Maintenance-Indicator app to configure it. The denominator of this new indicator would be the number 1. Hence, we call this type of indicator a “numerator only indicator”.

2. Notice the box beneath “Data element/Indicator”. It asks for the ‘Data element group” (or, if Indicator is selected, the “Indicator group”. There is no option to select from a list of “All data elements” or “All indicators”. Instead, a data element group or indicator group must first be selected. This means that if the data element you want is not part of a data element group then you must exit the Tool and use the Maintenance-Data element group app to assign the data element to a group. Likewise any indicator you want to select must first be assigned to an indicator group by using the Maintenance-Indicator group app.

3. For our example, let’s leave Data mapping set to “Data element” then click on the down arrow to the right of “Select data element group” to view the drop-down menu. Find and select “Immunization”. Note: the name of the relevant data element group or indicator group may be different on your own DHIS2.

4. Click on the down arrow to the right of “Select data element” to see the drop-down menu. Find and select “Penta 1 doses given”. Note: the name of the relevant data element or indicator may be different on your own DHIS2.

5. Next, we must fill in the “Select data set” field under “Data set for completeness”. Click on the down arrow to the right of “Select data set” to view the drop-down menu. The Tool has pre-selected the possible data sets. Confirm that the correct data set is listed (“Child health”) and select it. This is the data set that the Tool will use to assess for the completeness of reporting. Note: the name of the relevant data set will likely be different on your own DHIS2.

6. Next, we must fill in the field for “Select variable” under Variable for completeness. When you click on the down arrow to the right of “Select variable”, notice that the list consists of fully disaggregated detailed data elements with their corresponding categories. You cannot select the aggregated total data element that you just picked (under Data mapping). Instead you are forced to select a detailed data element. It is usually best to select as a variable the detailed data element for which data is most likely to be reported. In our example, this is “Penta 1 does given Fixed, < 1y”. This is the detailed data element which the Tool will use to assess “Completeness of indicator data” – assessment of the reporting for a specific data element as opposed to assessment of the reporting for an entire data set².

7. Only when you have filled in ALL of the Fields (the box for “Core” is optional) will the Tool permit you to click on Save.

1. Once you have successfully Saved the mapping for a Reference numerator, the Tool will again show the full table of numerators. Notice how the cells for “Data element/indicator” and “Dataset” are now filled in for the numerator you have just mapped. Also notice that the Clear button is brighter. If you wanted to, you could clear the mapping. Clearing a WHO standard Reference numerator will simply clear the mapping information – it will not remove the Reference numerator from the table. On the other hand, if the row was one that you had added (using the blue Add button at the bottom of the page), if you clicked on “Clear” the row would disappear from the table.

1. For more practice, let’s repeat the process for DPT3. Click on Edit. Notice that the box is checked for “Core”. WHO recommends DPT 3 as the Core indicator for immunization programmes. You could change this, but let’s leave it as it is. Next, fill in all of the other fields and Save your mapping.

1. Notice that, so far, DPT 1 and DPT 3 are the only numerators with “Data element/indicator” and “Dataset” specified.

If we go back to the dashboard of the Tool we will see that, as soon as you have mapped at least one reference numerator for a Group, the name of that Group will appear in the drop-down list for “Data”. In this way, the Tool can be set to assess all Core numerators or to assess all of the numerators that have been mapped for a single Group.

1. Next, let’s go back to the More-Administration function and the Numerators page. To add a new numerator, we can click on the blue Add button at the bottom right of the Numerators page

Let’s add “Number of physicians”. We must assign the new numerator to a Group – there is no default Group. For now, choose the “General Service Statistics” group. Complete the remainder of the fields. The existing data element “Doctor” is in the “Staffing” data element group. Once you have filled in all fields, you can Save the new numerator with its mapping.

1. If we don’t want to assign the new numerator to an existing Group then we must use the “Numerator groups” tab to add a new Group. Click on this tab now.

1. Then click on the Add group button at the bottom left of the “Numerator groups” page, type in the name for the new Group and Save it.

1. The name of this new Group will then appear in various tables and menus. For example, it now appears on the “Numerator groups” page. Notice that no numerator has yet been assigned to this Group.

1. We can assign to this Group a numerator that we have already mapped by clicking on the down arrow to the right of “Select item to add …” and selecting the numerator from the drop-down menu,

Notice that the “Add” button is now brighter. Click on the “Add” button.

The numerator you just selected will appear under the new Group.

1. The name of the new Group will now appear next to our new numerator on the Numerators page

1. If we want, we can edit this new numerator so that it is only in the Human Resources Group. Notice, however, that you must Save all edits and the Save button is greyed out (inactive). The Tool will not permit you to Save unless all fields have been completed.

1. Unfortunately, the Tools editor does not remember anything about the previous settings so you must re-enter everything. Then you can Save.

And the new numerator is now correctly mapped.

Now that you understand the mechanics of how to “map” reference numerators to DHIS2 data elements/indicators, you have reached the critical step of deciding which data elements/indicators to map to. This is discussed in the next section

Select the data elements/ indicators to map to

1. For several reasons, it is not always easy to choose the best data element or indicator for a given reference numerator: [Note to the facilitator: the following points are presented in the ppt named “Configuring the tool with the right data elements”.]

a. It may be hard to find the data element/indicator that you want

i. There may be a very long list of data elements, making it difficult to find the data element you want.

Figure 1: Screenshot from the Maintenance-Data element app showing that this particular instance of DHIS2 has 8,171 data elements.

Figure 2: Screenshot showing how a Filter can be used to identify data elements with "DPT" in the name. Note: other names such as "DTP" and "Penta" should also be tried. Also, try the same filters with [All indicators]

However, the best solution is to have a well-designed data element group which includes only the essential data elements. Remember that you will not be able to map a reference numerator to a data element unless the data element has already been assigned to data element group. If there are many data element groups, you may want to add a 2 digit prefix in front of the names of the most important ones so that they appear near the top of the drop down lists. DHIS2 sorts to the top of drop-down lists any data elements which begin with numbers. The same is true for drop-down lists of indicators.

Figure 3: With the Maintenance-Data element app you can configure Data element groups for the most essential data elements. If the name of the group begins with a digit, it will appear near the top of the list.

Figure 4: Assign to the group only those data elements which you will frequently use for analysis or select for the Data Quality Tool

ii. There may be no single data element to which a numerator can be mapped. For example, some DPT 1 doses may be counted with the data element “DPT 1, fixed, <12 months”, some with the data element “DPT1, fixed, ≥ 12 months”, some as “DPT 1, outreach, <12 months” and some as “DPT1, fixed, ≥ 12 months”. One solution would be to map to only one of these 4 data elements (e.g. “DPT 1, fixed, <12 months”). However, if you took this approach, then the Tool would not identify data quality problems with the other 3 data elements. A better solution would be to configure a new “numerator only indicator” equal to the sum of the 4 data elements. If the Tool identified a suspicious value for this indicator then it would be easy to use DHIS2 to identify the specific data element which was responsible for the suspicious value for the indicator. In many cases, such an indicator has already been defined. In some cases, however, it may be necessary to create a new indicator using the Maintenance-Indicator app.

Figure 5: An example of how to define the numerator of a "Numerator only indicator". The denominator for this indicator is 1.

iii. Be alert to the possibility that some of the data elements or indicators which you wish to select may not be properly defined.  For some of the more complex data elements and indicators (e.g. currently on ART, total TB notifications) it may be best to use the Maintenance-Indicator app to review the definition of the indicator³ and, if necessary, to change it. Any changes that are made using this app will change the data viewed by everyone who accesses the production instance. Therefore, indicator definitions should never be changed without first consulting and getting explicit permission from the national data manager. 

Figure 6: Currently on ART is a cumulative data element. For this reason, the data element should be configured with the unusual aggregation type "Last value (sum in org unit hierarchy)"

Figure 7: Some indicators have complex definitions. "Total TB notifications" is calculated by adding together the values for many different types of TB. The definition should be double checked to confirm that all required data elements have been included.

iv There may be no DHIS2 data set with data for certain important numerators. This problem is especially common for numerators related to HIV, TB, inpatient care, human resources and supply of commodities. One solution is to obtain an Excel version of the relevant data set then use the Excel-based version of the WHO DQR Tool⁴. An even better solution is to arrange for data from another electronic data system to be routinely imported into DHIS2.

b. It may be difficult to select between data elements/indicators with similar names

i. As noted, with the Pivot Table app, the list of data elements can be filtered by typing something on the search line. This may reveal that there are multiple data elements with very similar names. This makes it difficult to select the data element with the most complete and most reliable data.

Figure 8: The reference numerator "DPT 1" should be mapped to which data element?: 1. DPT+HiB+HEP1 Dose Administered; or 2. DPT/Hep+HiB1 doses Administered"?

Again, the best solution is to make sure that the relevant data element group does not include data elements with highly incomplete or unreliable data.

A good way to screen data elements is to use the Data Visualizer to configure a chart showing the trend over the last 5 years to assess which data element appears to have the most complete data and whether completeness appears to have varied.

Figure 9: A chart showing the multi-year trend of the two data elements shows that only one of them has reliable values. The program target for each year has been added to assess whether the data are reasonably complete

ii. If DHIS2 shows reliable Reporting rates for the respective data set, you can also use the Data Visualizer to review the trend in completeness. Such a chart will also tell you for which years there are data for the data element. You may find that the Reporting rate of one or more important data sets is low (less than 75%) or has varied from year-to-year.

Figure 10: A chart showing a low and variable Reporting rate for the Maternal health dataset.

Figure 11: A chart showing that the reporting rates for multiple data sets were substantially lower prior to last year. Note: When reporting rates were so much lower during previous years, the year-to-year consistency is likely to be poor.

If, for a particular numerator, there is no relevant data set with a Reporting rate that is high and consistent from year to year, then you may have no choice but to map to a data set with incomplete data. You should keep in mind, however, that the Tool is likely to find many problems with the quality of incomplete data due simply to variations in the completeness over time and between geographic sub-units. If the reporting rate for last year is below 50%, the data set/data element should not be designated as Core. Explore whether there is a non-DHIS2 data set with more complete data for the data element. If so, you may be able to use the Excel-based DQR Tool to review the quality of the data.

If the reporting rate was substantially lower during previous years, then the Annual Report is more likely to show poor year-to-year consistency even if there was no other problem with data quality. For assessment of year-to-year consistency, it is best to exclude data from years for which there is no data set with a DHIS2 reporting rate nationwide of at least 50%.

iii. DHIS2 may not reliably reflect the Reporting rates for data that are imported. The best solution is to also import the meta-data required for DHIS2 to track the Reporting rate. Special guidance has been drafted for this purpose (see “How to update the reporting completeness of imported DHIS2 data”).

Figure 12: Screenshot showing how DHIS2 and the DHIS2-based WHO Data Quality Tool may sometimes not show reliable findings on the reporting rate of imported data.

iv. For certain of the data elements that you want to select (or which are used to calculate “numerator only indicators” that you want to select), it may not be clear which data set they belong to. When there is such uncertainty, use the Maintenance-Data set to find the data set to which you believe it may belong. Click on “edit” to confirm that the data element has been assigned to the data set.

Figure 13: Screenshot of showing some of the data elements assigned to the “Maternal health” dataset, as viewed using the Maintenance – Data set app.

35.Exercise 1: Complete the following table. Configure any new indicators or data element groups or indicator groups which are necessary.

Configure the Numerators page of the WHO Data Quality Tool

Exercise 2:

36.Return to the Numerators page of the Tool (launch the Tool then click on More then click on Administration).

37.For each of the first 4 numerators in your table from Exercise 1, click on Edit, complete the configuration and Save.

38.For the 5^th numerator (Total RDT tests), use the blue Add button to add it. Complete the configuration.

Demonstration of how to configure the Numerator relations page of the Tool

[Note to the facilitator: this section can best be presented using live, online projection of the play instance of DHIS2 (https://play.dhis2.org/demo/dhis-web-commons/security/login.action )]

39.While still in More-Administration, click on the tab labeled “Numerator relations”. This is where you will specify pairs of data elements which should be compared to each other. By default, the Tool may recommend some pairs.

In addition, there is a blue Add button in the lower right corner of the page that can be used to add new “Numerator relations”.

40.We have already mapped DPT1 and DPT3, so the Tool is able to automatically configure the relation of DPT 1 to DPT3. Numerator A (DPT 1) has been mapped to “Penta 1 doses given”. Numerator B (DPT 3) has been mapped to “Penta 3 doses given”. By default, this relation is set to Type = Dropout rate.

41.To see how “Numerator relations” are configured, click on the Edit button to the right of “ANC 1 – IPTp 1 ratio”. A default relationship “Type” is suggested: A = B. And a default Threshold is suggested: 10%. The meaning of “Threshold” depends upon the relationship type. For relationship type of A = B, the threshold is defined as the maximum % that Numerator A can differ from Numerator B. For example, if ANC 1 and IPTp 1 differed by more than 10%, then this might suggest a possible data quality problem.

42.ANC 1 is often significantly greater than IPT 1⁵. So we do not want to assess whether A = B. Instead, we want to assess whether the ratio of A to B for each sub-national unit (region or district) is the same as the ratio of A to B at national level. For this comparison, change “Type” to “Equal across orgunits”. For relationship type of “Equal across orgunits”, the threshold is defined as the maximum % that Numerator A can differ from Numerator B.

For example, if ANC 1 and IPTp 1 differed by more than 10%, then this might suggest a possible data quality problem.

43.You cannot Save this relation until you have specified Numerator A and Numerator B. Numerator A is filled in by default because you have already mapped ANC 1.

44.If you click on the down arrow to right of Numerator B, however, you will not find the numerator that you want. This is because the drop down list includes only numerators that you have already mapped. So, if you want to review the data using this relation, you must first go back and map the numerator “IPTp 1”. Click Cancel to abandon for now the attempt to configure this particular numerator relation.

45.Next, let us click on the Add button at the bottom right of the “Numerator relations” page

Again, all field must be filled before you will be permitted to Save the relation. You will do this as part of the next exercise.

Configure the “Numerator relations” page of the Tool

46.Exercise 3: From the “Numerator relations” page, configure the following relations:

a. DPT 1 to DPT 3 dropout rate

b. ANC 1 – DPT 1 ratio; A = B; Threshold = 10%

c. If time permits, you should also configure one other relation of your choice.  Remember that you will not be able to Save a “Numerator relation” unless Numerator A and Numerator B have both already been mapped.

Review the numerator quality parameters

47.While still in More-Administration, Click on the tab labeled “Numerator Quality Parameters”. Don’t worry if, at first, you find this busy page to be a bit overwhelming. Fortunately, the Tool comes with default quality parameters. You do not need to adjust any of these parameters until you understand how they function and have thought about how you might want to re-set them.

1. As one example, let us consider the parameters set for Completeness at the bottom of the page. We can think of a threshold as being like a target. By default, a threshold/target is set for 90% completeness. Suppose that reporting completeness for a particular data set averages only 80% nationwide. In this case, if you used a threshold of 90% to assess each district, you might find that almost all districts failed to meet this threshold/target.

In that case, it would be appropriate to lower the completeness target to identify only those districts which had especially low completeness. You can do this by placing the cursor in the cell. Up and down arrows will then appear which permit you to change the threshold (or you could simply type a number in the cell). Click on the blue “Save changes” button in the lower right of the page and the Tool will then use the adjusted threshold. The screen will not change when you click the button but the Tool will have indeed saved the change.

49.As another example, consider the parameters which define moderate and extreme outliers for an epidemic prone or seasonal disease such as malaria. In most settings, we do not expect the number of reported cases of confirmed malaria to remain roughly constant from month to month. During outbreaks or months of increased transmission we may have values that are several standard deviations above the mean monthly value. Hence we may want to ignore moderate outliers and set the threshold for extreme outliers to 4 standard deviations rather than 3.

Annex 1: Eventual configuration of the Tool on the “play instance”

Note: To begin practice in configuration of the Tool, this table of numerators should at first have blank cells for “Data element/indicator” and “Dataet” for ALL reference numerators.

Notes

Usage and interpretation

How to use and interpret WHO’s Data Quality Tool

[Note to the facilitator: this section can best be presented using a live, online projection of the DQ TrainingLand instance of DHIS2 (https://who.-demos.dhis2.org/dq ). The document can be presented in its entirety, followed by the 3 exercises. The instructions for the 3 exercises should be printed together as a single document, a copy of which should be given to each participant. Guidance on configuration of the numerators for the Tool is provided in a separate document entitled “How to Configure the WHO Data Quality Tool.docx”. Presentation related to assessment of denominators and comparison with survey estimates is dealt with in a separate documents entitled “Quality Denominators.ppt”. ]

Contents

Contents

• Launch the Tool
• Use the Completeness dashboard
• Use the “Consistency-time” dashboard
• Use the Tool for a program-specific review of data quality
• Use the “Consistency-data” dashboard
• Use the “Outliers” dashboard
• Use the “Analysis-Consistency” function
• Use the “Analysis- Outliers and Missing Data” function
• Exercise 1: Use the dashboard and the analysis functions to identify suspicious values with your DHIS2
• Exercise 2: Discuss how to deal with very suspicious values
• Use the “Annual Report” function
• Exercise 3: Use the Tool to prepare an Annual Data Quality Report
• Annex 1: How to configure Denominators, Denominator relations and External data comparisons
• Notes
• “DPT-HepB-Hib 3 doses given < 1 year” is reported with the EPI dataset whereas “Penta 3 doses given is reported with the HMIS dataset.

Launch the Tool

1. Log into the DHIS2 instance https://who.-demos.dhis2.org/dq
   Username: demo Password: District1#

2. Type “WHO” on the search apps line in the upper right of the DHIS2 Home page. Click on WHO Data Quality Tool.

   

3. Quickly review the main menu of the DQ Tool: After you launch the DQ Tool, you will see the 5 tabs at the top of the page. These are used to select between several different functionalities. Notice that the word “Dashboard” is highlighted and the Dashboard function is selected by default.

   

4. Quickly review the Dashboard menu: The Dashboard has its own set of 4 tabs. The Completeness tab is selected by default.

Use the Completeness dashboard

1. Review the Completeness dashboard: Scroll down and review the Completeness page of the data quality dashboard. The page shows completeness for each of 5 “core datasets”.

Note: When you review the completeness with your DHIS2 instance, if you find that completeness of one or more dataset is 0%, consider the possibility that this is because the data was imported. DHIS2 cannot be used to monitor the completeness of imported data unless you also import the completeness meta-data. For the training instance of DHIS2, the immunization dataset is quite incomplete. It is best to review the quality of the ANC data which are more complete.

a. Review the graphs on the left side of the page: The graphs on the left show, for each dataset, the completeness and timeliness by month for each of the last 12 months.

b. Review the graphs on the right side of the page: The graphs on the right show, for each dataset, the completeness by Region for the last month of the period being analyzed. Note: with this instance of DHIS2, the country is divided into 4 regions and each region is divided into multiple districts. Completeness for the most recent month is actually a measure of timeliness. It is often more revealing to show completeness for the month before last. Click on the menu icon.

c. Change the “Organisation unit” that is being analyzed: By default, the DQ Tool analyzes all data nationwide, disaggregated by Region. The DQ Tool can also be used at district level with data disaggregated by individual health facility. In the Organisaton unit section of the menu, click on Other and select “District D” of Region A. The graphs on the left side of the completeness page now show the 12 month trend in the reporting completeness of District D. By default, the graphs on the right side of the completeness page show results disaggregated for one level below – facility. Question: Which individual facilities have not yet reported ANC data for last month? For 3 months ago? For 6 months ago? After you have experimented with changing the organization unit and the month, reset it national with disaggregation by District and period = April of this year¹.

Use the “Consistency-time” dashboard

1. Review the Consistency-time dashboard: Click on the tab for “Consistency-time”. Scroll down and review the page. Move the cursor around on each chart to view the pop-up details.

a. Review the graphs on the left side of the page: The graphs on the left show, for each core numerator, the trend in the numerator value over the period 1 to 12 months previously (dark blue in the above chart), the period 13 to 24 months previously (light blue in the above chart) and the period 25 to 36 months previously (orange in the above chart). Note the sharp rise in the value of ANC 1 in May 2018 and again in January 2019. When there is such a dramatic rise at national level, this is certainly suspicious.

b. Review the graphs on the right side of the page: Each graph on the right is an example of "scatter plot". When the Tool is set to disaggregated by District, each dot represents the value for a single district. On this chart, the position of the dot on the vertical axis represents the value of the numerator for the month selected (remember that we re-set the menu to show results for Period = April 2020). The position of the dot on the horizontal axis is the average value in the same district over the 11 previous month. Dots that appear above the lower grey line or above the upper grey line are suspicious (e.g. in the example below, the number of ANC 1^st visits in District F was 1,425 in April 2020 as compared to an average of 4,596 / month from May 2019 to March 2020.

Hence, the "Completeness-time dashboard" may be useful to identify some suspicious values. However, other components of the Tool do a better job of identifying suspicious values. So this page of the Tool is not used as frequently as others².

Use the Tool for a program-specific review of data quality

1. Review the list of “Core” data element/indicators: On the menu of the Tool, Data is now set to “Core”. As a result, the Tool presents findings for the data elements/indicators which were included in the “Core” grouping³. Scroll down the “Completeness-time” page to review the list. Notice that only two immunization indicators⁴ are featured.

2. Change Data to “Immunization”– On the menu (click on the icon if the menu is not open) change Data from “Core” to “Immunization”. Notice what happens to the list of data elements/indicators featured. There are now 10 immunization indicators featured on the “Completeness-time” page. In this way, the Tool can be used for a program-specific review of data quality – to review the full range of program-specific indicators. Before you continue to the next step, use the menu to change Data back to “Core”.

Use the “Consistency-data” dashboard

1. Review the Consistency-data dashboard: Click on the tab for “Consistency-data”. This page shows whether there is a plausible relationship between numerators that are related. These are the “numerator relations” that are configured using More-Administration-Numerator relations. Scroll down and review the page. Move the cursor around on each chart to view the pop-up details.

   a. Review any scatterplots: The graph comparing ANC 1 to ANC 4 is a scatterplot with each dot representing the total values, over the last 12 months, for one district. Districts with values that fall outside of the grey threshold lines are represented with a diamond shape. As one example, for District F (represented by the enlarged diamond), over the last 12 months, ANC 4 was only 2,641 while ANC 1 was 51,976. The discrepancy is suspicious and should be investigated.

   

   b. Review any graphs assessing dropout rates: The graph comparing DPT1 to DPT3 assesses for any “negative dropout rate”. The graph shows that there are 5 districts which, over the 12 months up to and including April 2020, had a negative DPT 1 to DPT 3 dropout rate. In other words, these 5 districts reported a higher value of DPT 3 than DPT 1. A negative DPT dropout rate for a district for an entire year is usually a sign of poor data quality. This should be investigated.

   

Use the “Outliers” dashboard

1. Use the “Outliers” dashboard to identify suspicious district-level values: Click on the tab for “Outliers”. If necessary, use the menu to set Disaggregation to District. To free up space on the page, click on the menu icon to make the menu disappear. The page shows a “Result” table with a large number of rows. Each row has one or more values highlighted in red. How are the values highlighted in red different from the other values in the same row? The Tool finds the values highlighted in red to be suspicious because they are either significantly higher or significantly lower than the other values in the row. The rows are sorted in order of “weight”. The “weight” can be thought of as the difference between the suspicious value and the average of other values in the row. There are so many rows that the table will not fit on a single page. You can view the subsequent pages by clicking on the controls at the bottom of the page.

   

   a. Investigate outliers with a weight that exceeds 5% of a district’s annual total. With so many outliers, how can you decide which ones are most important? One approach is to focus on the rows at the top which have the largest weight. As a rule of thumb, all values with a weight that is more than 5% of the district’s expected annual total should be carefully investigated because they can significantly distort the district statistic. Consider, as examples, the 1^st and 2^nd rows in the above table showing extreme outliers in the values of ANC 1. The expected annual number of pregnancies in districts F is 20,000 while the expected number of pregnancies in district E is 110,000. In both cases the extreme outliers must be investigated because the outlier weights greatly exceed 5% of the district’s expected annual pregnancies.

   b. Investigate outliers with an extreme Z-score. There is another way to identify the outliers which are most suspicious by using their “Z-score”⁵. To use this method, click on the Options button to open the Options window. To see the Z-score for each row, click on the box “Include Z-score”. Review the Standard Z-score values in the Result table. Notice that the values range from less than 2 to more than 3. Next, click on the following buttons: 1) Outliers; 2) Standard Score; and 3) Extreme. If the Result table has no rows at all or only 1 or 2 rows, click on “Modified Z-Score”⁶ to identify more outliers.

   

   c. Inspect each row– Questions: Review the Standard Z-scores. Notice how the Result table reduces to only a few rows. Notice that the Z-score for each row is greater than 3.0. Do you think these suspicious value might be due to an error? How would you investigate to determine whether a highlighted value is due to an error? Click on the menu icon at the end of the first row and select “Visualize” from the drop-down menu.

   

   d. “Drill down” to investigate the source of the suspicious value – Click on the Close button to close the graph, click again on the menu icon and select “Drill down” from the drop-down menu. If necessary, again use the Options feature to filter the table to Outliers, Standard Score and Extreme. Do you think these suspicious value might be due to an error? Very large extreme outliers such as the one in the first row are very likely to be due to errors. Extreme outliers which are much less than the average monthly value, such as the one seen in the second row, may be due to missing data if they are seen at district or higher level. A small but extreme outlier seen at facility level is also likely to be due to an error.

   

   If there is another level of the organizational hierarchy below this level, you can drill down further. If, however, this is the lowest level in the hierarchy then you get a message such as the following:

   

   e. Consider sending a message⁷ to the person responsible. Click again on the menu icon and select “Contact” from the drop-down menu. A window should appear showing the Contact person for notifying of the extreme outlier. Click on “Write message”. Use the drop-down menus to fill in the information below Recipient, write a message and click on “Send message”. A message will then be sent via the DHIS2 internal messaging service.

   

   

   As we have seen, the dashboard of the DQ Tool is easy to use. A limitation of the dashboard, however, is that it can only be used to review those data elements/indicators for which the dashboard has been configured (see “How to configure the WHO Data Quality Tool”). Next we will learn how the Analysis function of the DQ Tool can be used to review the quality of any data element or indicator for which DHIS2 has data.

Use the “Analysis-Consistency” function

1. Review the Analysis – “Consistency” function: In addition to the Dashboard, the Tool provides a convenient way to customize your review of data quality.

2. Click on the Analysis tab at the top of the page, then select Consistency.

A window appears which allows you to configure a customized assessment of consistency-over-time or consistency between related numerators. By default, consistency “Between Indicators” (between related numerators) is selected and Compare organization units to “Overall result” is selected. With this setting, A = B, A > B and Dropout cannot be selected⁸. Leave “Consistency analysis type” set to “Between Indicators” but click on “Expected result” and select “Dropout” (see figure).

• Click on Data to choose your numerators. Configure the window as shown in the figure. (Note that “Indicator” is selected for both numerators. This is because, for this instance of DHIS2, data for DPT 1 is disaggregated into two separate data elements: DPT1 from fixed sites and DPT1 from outreach. To simultaneously review the data for both of these data elements, an indicator has been configured which is the sum or the two data elements. Likewise, an indicator has been configured for the sum of DPT3 from fixed sites and DPT3 from outreach.

• Click on Period to customize the period as shown in the figure – set to last year.

• Click on Organisation unit and set disaggregation to District.

Click on the blue Analyze button in the lower right of the page and the results will appear.Note the following:

1. The chart on the left side of the page is similar to what we saw on the Consistency-data dashboard. There are 6 districts which had a negative DPT 1 to DPT 3 dropout rate in 2017. For details, you can move the cursor over the red bars.

2. The table on the right side of the page has a row for the entire country followed by one row for each district. The districts are sorted according to their dropout rates. The 6 districts with negative dropout rate are at the top of this list. The rows continue on two more pages but it is less important to view these other pages because the dropout rates are all positive.

   

3. If you click on any row, the color of the row changes to pale yellow and a one row table and another chart appear at the bottom of the page. The chart shows you the results by month. In this example, DPT 3 was greater than DPT1 during 7 of the 12 months of last year. Thus the negative dropout rate is not just the result of a single outlier – at least for 2019 for District G there was a recurrent problem.

   

4. You can click on the menu icon at the end of this single row to “drill down” and see the results dis-aggregated to a level below. The chart shows that the negative dropout rate is not just the result of data reported by a single health facility – 11 out of 18 health facilities in district G reported more a higher DPT 3 than DPT 1 in 2018. Conclusion: District G appears to have a problem with systematic over-reporting of DPT 3. The problem affected reporting by multiple health facilities and for multiple months. This may be due to a tendency of health facilities in district G misclassifying first doses of DPT as third doses of DPT.

1. Use the “Analysis – Consistency” function to identify all facility values which warrant investigation

2. Set Analysis Type as shown in the figure: Consistency analysis type = “Over Time”; Compare organization units to = “Expected result”; Expected trend = “Constant”; Criteria = 0%.

• Click on Data and select a single data element or indicator. For example, select “Penta 3 doses given (HMIS)” in the data element group “HMIS”

• Click on Period to customize the period as shown in the figure. With these settings, the value for February 2020 will be compared to the values for the preceding 12 months.

• Click on Orgunit, select a specific district and set disaggregation to Facility.

With these settings, the Penta 3 values for February 2020 for all health facilities in District E will be compared to the average values reported by these same facilities during the preceding 12 months.

• Click on “Analyze and wait for the chart to appear.

How do we interpret this chart?

• Each dot represents the data reported by one health facility in the selected district (District E);
• The chart compares the value reported in February 2020 to the average value reported by the same health facility during the preceding 12 months;
• The line represents equality. Almost none of the dots falls exactly on the line. The distance of each point from the line is given by the weight. The vast majority of dots lie so close to the line that their weight is small and we can ignore them.
• Each month, the district reports a total of 6,536 Penta 3^rd doses on the HMIS form. We can use this number to determine a threshold for investigation of outliers. We could, for example, decide that we must investigate any outlier which diverges from the line by more than 5% of the district total = 325.
• Only the top two dots have a weight greater than 325. Can you spot these two dots in the chart? :
  • Facility 567 reported 961 when it has previously reported an average of 146/month. Thus, this suspicious value is given a weight of 803;
  • Facility 222 has failed to report any data at all when it has previously reported an average of 349/month. Thus, this missing value is given a weight of 349;
• These two values must be investigated

As a first step of our investigation we can click on each row. The selected row turns yellow and a chart appears at the bottom of the DHIS2 screen.

The chart shows us how different the February 2020 value for Facility 567 is from the values reported previously. This is probably due to erroneous data. How could we use DHIS2 to investigate further? What more could you do to investigate this value? ` Here is what we find when we click on the row for Facility 222.

Notice not only that the value is missing for February 2020, but there is a suspiciously high value for May of 2019. This illustrates how this analysis can not only identify extreme outliers for the month we have selected for review (February 2020) but it can sometimes identify outliers from previous months (if the outlier is so large that it increases the 12 month average by a large amount).

In summary, with this type of analysis, a district can review the most recent values from all health facilities and identify a small subset of values which require intensive investigation. The analysis needs to be carried out one data element at a time.

Use the “Analysis- Outliers and Missing Data” function

1. Review the Outliers and missing data function: Click on the Analysis tab at the top of the page, then select “Outliers and missing data”. The Analysis-Outliers function works exactly the same way as the Outliers dashboard. The advantage that it has over the dashboard is that it is much more flexible. You can configure it to assess any data element or indicator⁹ – not just the data elements/indicators for which the dashboard of the DQ Tool has been configured. In fact, you can configure it to simultaneously assess any combination of data elements and indicators you want. You could even choose to simultaneously assess all data elements or indicators in a data set, data element group and/or indicator group.

   

You can also use it to assess data over any period.

… and for any grouping of health facilities.¹⁰

Exercise 1: Use the dashboard and the analysis functions to identify suspicious values with your DHIS2

1. Review the “Consistency-time” dashboard – Click on the tab for the “Consistency-time” dashboard. Review the left and right graphs for the first data element/indicator (ANC 1).

   a.Set Data to [Core], Period to April of this year, and Disaggregation to District.

   b. Review the graph on the left.

   Has the data element/indicator had a consistent value from year-to-year and from month-to-month. If not, how do you explain any dramatic inconsistency?

   c. Quickly go to the Completeness dashboard to review the trend in Completeness of the ANC dataset over the last 12 months (the graph on the left side of the page).

   To what extent is the inconsistency in the value of the data element/indicator due to a change in reporting completeness?

   d. Return to the Consistency-

   Time dashboard and review the graph on the right for ANC 1. This chart is a scatterplot with each dot representing one district. The position of the dot on the vertical axis shows the value of ANC 1 for one district for April of this year. The position of the dot on the horizontal axis shows the average value of ANC 1 for that district during the 11 months preceding April (i.e. May 2019 to March 2020). Move the cursor over the dots which lie above or below the two grey lines to view the suspicious values.

   Question: List the districts which have values for last month which are most inconsistent with their average value for the preceding 11 months. Do you have any theory as to why the data from these districts was inconsistent over time? Could it be due to incomplete reporting for last month? Could it be due to real changes in delivery of services last month? Note that there are multiple reasons for inconsistency-in-time. Therefore inconsistency-in-time does not necessarily indicate a problem with data quality.

2. Review “Consistency-data” – Click on the tab for the “Consistency-data” dashboard. Each chart on this dashboard assesses the relationship between a pair of related data elements/indicators.

   a. Review the first chart comparing ANC 1 to ANC 4:

   This chart is also a scatterplot with each dot representing the values for ANC 1 and ANC 4 for one district, over the 12 months up to and including April of this year. Districts with values that fall outside of the grey threshold lines are represented with a diamond shape. Move the cursor over these dots to review the suspicious values of the related data elements/indicators.

   Questions: Do you see why the Tool has flagged these related values as suspicious? List the districts which have suspicious values. Do you have any theory as to why these particular districts have suspicious values? Why would incomplete reporting NOT explain the suspicious values?

   b. Review the graph assessing DPT dropout rates:

   Questions: Which districts have a negative DPT 1 to DPT 3 dropout rate? Which district has a dropout rate which is the most negative? Why do you think this particular district has the most negative DPT dropout rate? Could the negative dropout rate be due to a single, large erroneous value for a single health facility for a single month? Or is the dropout rate due to multiple values from multiple health facilities and multiple months? With the “Consistency-data” dashboard we cannot really answer this question.

   c. Use Analysis-Consistency to assess DPT dropout rates: Click on the Analysis tab then select “Consistency” from the drop-down menu. Under Analysis Type, click on each of the following: “Between Indicators”, “Expected result” and Dropout”.

   

   Open the Data window then configure it as shown in the figure to the right. Note: With your DHIS2, DPT 1 < 1 year (or Penta 1 < 1 year) and DPT 3 < 1 year (or Penta 3 < 1 year) may be data elements rather than indicators.

   

   Open the Period window and set it to year = 2019.

   Open the Organisation unit window and set it to National (or the equivalent ) with disaggregation to District. Click the Analyze button.

   i. Review the chart and the table (note: the top row of the table will not be shaded yellow until you complete the next step). Move the cursor over the vertical red bars in the chart to view the details. Compare the names of the districts with vertical red bars to the names of the districts in the table with Dropout rate shaded in red.

   Questions: Which district has the most negative dropout rate? Find it in the chart and find it in the table. For that district, during 2019, what was the value of DPT 1 and what was the value of DPT 3 (see table)?

   

   ii. Click on the row of the table for the district with the most negative dropout rate. A new table and chart should appear at the bottom of the page.

   Question: Is the negative dropout rate due to a value for a single month or is it due to values for multiple months?

   

   iii. Click on the menu icon at the right end of the small table and select “Drill down” (due to a bug in the Tool, the words may not all show on the screen). A new chart and table should appear showing the dropout rates for 2019 for each org sub-unit (for example, each health facility if they are immediately beneath district in the organizational hierarchy).

   Questions: For the district which you have selected, is the negative dropout rate due to a value from a single facility or is it due to values from multiple facilities? Does it appear that the district you have selected has a systematic problem with over-reporting of DPT 3 – for multiple months and multiple sub-districts? How do you explain this?

   

3. Use the “Outliers” dashboard to Identify important outliers-- Click on the tab for the “Outliers” dashboard. Review the menu window to confirm that Disaggregation is set to District. Click on the menu icon to hide the menu window and free up space on the screen. Click on the Options button then check the box for “Include Z-score” and click on “Outliers”, “Standard Score” and “Extreme”. If the “Result” table has no rows or only 1 row, click on “Modified Z-score”.

   

   Click on the menu icon at the end of the first row of the table and select “Visualize”. Review the graph. Question: Do you think this extreme outlier is due to an error in the data? After you have finished viewing the graph, click “Close”.

   Click again on the menu icon at the end of the first row of the table and select “Drill down”. A new “Result” table should appear with 1 row for each facility in the selected district which has an extreme outlier during the 12-month period. Consider the outlier in the first row of this new table. Repeat the process of visualizing the outlier. If you want, try to drill down further. Question: Do you think this extreme outlier is due to an error in the data?

4. Use Analysis-Outliers to identify extreme outliers of additional data elements—

   Click on the tab for Analysis, then select “Outliers and Missing Data”. Select a data element or indicator of your choice. Then set Period to 2019 and set Orgunit to national with Disaggregation by district. Click on Analyze then proceed to use the Tool to identify extreme outliers using the same steps as with the Outliers dashboard.

   

Exercise 2: Discuss how to deal with very suspicious values

1. List the measures that you would take if you identified a value that was very suspicious and so large that it probably has a significant effect on a nationwide statistic.¹¹

2. Who has authority to edit the data of your DHIS2?

3. How do you plan to use the Tool to identify extreme outliers – at district level as well as at national level -- each month or quarter as well as each year.

Use the “Annual Report” function

1. Generate the Annual Report:

   Go to the menu at the top of the DQ Tool and click on the Annual Report. Use the drop down menu to select “[Core]” under Data – Group. Under Period, select the last calendar year, then specify the number of years for which you have at least one data set with completeness greater than 50%. Ideally this would be 2 or 3 years¹². Under Organization unit set Disaggregation to the equivalent of district. It is not recommended that you leave Disaggregation set to the equivalent of Region (e.g. Province), because the Annual Report will then fail to identify many important data quality problems. Click on “Generate” to view the report.

   

2. Quickly review the SUMMARY of the Annual Report:

   The report begins with a SUMMARY section followed by a more detailed report which includes much more information. Quickly scroll through the SUMMARY and note the sub-sections devoted to Domain 1 (Completeness), Domain 2 (Internal consistency), Domain 3 (External comparisons) and Domain 4 (Consistency of population data).

3. Review the detailed report on 1a: Completeness of facility reporting:

   The detailed report begins about ⅓^rd of the way through the Annual Report.

   

   Note that table 1a shows an “Overall score” for each of the core datasets. This is the nationwide completeness of reporting for the dataset, during 2018. The column labeled “Percent” shows the percent of districts which had completeness below the threshold. The table identifies the specific districts with apparent problems with data quality.

   Question: What is the threshold for ANC Reporting rate? How many and what percentage of districts had a completeness below this threshold?

4. Type something in the box for Interpretation:

   Note the box where interpretations and notes on follow-up actions should be typed.

   THE ANNUAL REPORT IS NOT FINALIZED UNTIL THE BOXES FOR INTERPRETATIONS AND FOLLOW-UP ACTIONS HAVE BEEN FILLED IN FOR KEY SECTIONS OF THE REPORT.

5. Review the graph of Reporting completeness over time: Scroll down to the graph showing “Reporting completeness over time”. For robust analysis, a dataset should be at least 75% complete. If the completeness of a dataset changes by more than 5%, then the data should be adjusted before analyzing for a year-to-year trend.

   Questions: 1) For which datasets can robust analysis of 2019 data be performed? 2) To assess for trend from 2018 to 2019, should the ANC data be adjusted for a change in completeness?

   

6. Review table 2a: Extreme outliers:

   The table shows the Threshold for an extreme outlier - A monthly value that is more than 3 SD (standard deviations) from the mean monthly value for the period being review. The table cites an “Overall score (%)”. This is the percentage of reports expected during the year (12 times the number of facilities that are expected to report) for which the value of the indicator was an extreme outlier. The “Number” is the number of districts and the “Percent” is the % of districts which, during 2019, reported an extreme outlier for the indicator.

   

7. Review 2d: Consistency of indicator values over time:

   Scroll down to section 2d. This section compares, for the 5 core indicators, the total value reported in 2019 to the average annual total value for preceding years. For https://who.dhis2.net/dq , we have data that are at least 50% complete only for 2018 and 2019. So we compare the 2019 value to the 2018 value. When values of an indicator are consistent from year to year, this is reassuring. If values jump up and down erratically from year to year it can be a sign of real changes in program performance (e.g. as a result of a stock out). However, inconsistency from year to year may be a sign of a data quality problem.

   Review the scatterplot showing consistency over time of ANC 1 visits:

   Each dot represents the data from one district. The position of the dot on the vertical axis is the number of 1^st ANC visits reported by the district in 2019. Notice from the small graph in the lower left corner that there has been a nationwide increase in reporting of ANC 1^st visits (mostly due to an increase in completeness of reporting). Hence, we don’t expect the 2019 value to be equal to the 2018 value. The “Overall score” of 125% indicates that the 2019 nationwide total value of ANC 1 was 125% of the value for the preceding year. The difference between the 2018 value and the 2019 value is also reflected in the slope of the black line which represents the national relationship (e.g. when the black line crosses 60,000 on the horizontal axis, it is at roughly 67,000 on the vertical axis). The grey lines show ± 15% from the national ratio. The dots that fall outside of the grey lines have a different shape because they have suspicious values. For example, District C reported 19,988 ANC 1^st visits in 2018 but the value increased to 32,522 in 2019. This increase is significantly more than the nationwide increase. This suspicious value should be investigated. Notice that District C is included in the box listing districts with “divergent scores”.

   

   Interpret your findings-

   Ask yourself whether there have been any sub-national changes in program operations (e.g. stock outs, new interventions) or reporting completeness or changes to administrative boundaries (i.e. splitting of districts) which affected the districts with divergent scores.

   Consider changing the threshold for assessment of consistency over time

   For some indicators, you may wish to change the threshold to identify a larger or smaller number of units with inconsistencies over time. For example, as an optional exercise, perhaps you want to focus only on the districts with the most extreme inconsistency over time. Click on “More” in the menu at the top of the DQ Tool, then “Administration” then “Numerator quality parameters” then scroll down to “Indicator parameters” and use the control arrows to raise the consistency threshold to ± 33%.

   Click on the “Save changes” button. Return to the Annual Report (set up as previously – with Period set to compare to the same number of years and with district dis-aggregations), again scroll down to section 2d (about 60% of the way through the Annual Report). Note the new Quality threshold in the table on the left. Note how the grey lines of the chart are now spread further apart and more of the dots now fall between the grey lines.

   Question: How many districts now have divergent scores?

   

   Review the scatterplot showing consistency over time of DPT 3

   Scroll down and find the table and chart for this indicator. Note how the dots are more scattered than for ANC 1. Ordinarily, we would not expect for DPT 3 values to be so unpredictable. However, as shown by the graph in the lower right corner, reporting of EPI data has risen dramatically. With data that are less 70% complete we should not expect consistency from year to year even in indicators such at immunizations and ANC visits which normally should be quite stable.

   

   In summary, there are many reasons that data may not be consistent from year to year. However, when we find consistency-over-time, it is reassuring that the data are of better quality.

8. Review 2e: Consistency between related indicators

   Scroll down to section 2e. The graphs in this section are the same as those which appear on the “Consistency-data” dashboard. The only difference is that the period is the last calendar year (i.e. 2019) rather than the last 12 months or whichever other 12-month period you selected from the dashboard menu. The Annual Report provides you with an opportunity to explain your findings and summarize your follow-up actions. For example, what will you do to supervise and support the districts which have negative DPT dropout rates?

9. Use the Tool to make external comparisons (Section 3a)

   “Domain 3” of a data quality desk review involves comparison of estimates based upon routine data with estimates obtained from other sources. In particular, routine data are used to estimate coverage with maternal and infant health services: antenatal care, institutional delivery, postnatal care and immunization. These “routine estimates” of coverage can be compared with estimates obtained from surveys of representative samples of households: Demographic and Health Surveys (DHS), Multiple Indicator Cluster Survey (MICS), immunization coverage surveys and other.

   Such survey estimates are sometimes considered the “gold standard” for assessing the reliability of routine data. However, it is important to keep in mind that discrepancies between routine estimates and survey estimates may be due to several factors other than problems with the reliability and completeness of routine data:

   a. Other reasons that the estimate based on routine data may not be reliable: 1. The denominator (i.e. the target) may not be over-estimated or under-estimated; 2. Persons may live in one district and obtain health services in another district; 3. Person may obtain health services from private providers who do not report. For example, antenatal care and delivery services may be provided by private midwives.

   b. Reasons that survey estimates may not be reliable: 1. Sampling error – too small a sample for precise estimation. Even with the largest of samples, it is seldom practical to estimate at district level. 2. Non-random sampling – this has been a problem with surveys conducted with WHO’s old EPI survey methodology 3. Non-sampling error –e.g. recall bias. This is a special problem where a home-based record is not available and immunization status is assessed based on recall of the care provider.

   Review Section 3a of the Annual Report–

   Note: Before the Tool can generate findings for section 3a, several steps must be completed.

   These are discussed in Annexure 1:

   1. New DHIS2 data elements must be configured for the survey estimates and the survey estimate data must be imported. This data is typically disaggregated by region.
   2. The Denominators and the External comparisons pages of the Tool must be configured.

   Scroll down to Section 3a to see how some coverage estimates based upon routine data compare to survey estimates. In the example, notice that the findings are now disaggregated by region rather than by district. Notice that the “Routine” estimates of coverage are each reasonably close to the “Survey” estimates. The match will seldom be this good. However, even in this example the match is not perfect. The routine estimates are consistently less than the survey estimates. The “Overall score” is 92.2%, meaning that the routine estimate at national level is 92.2% of the survey estimate at national level. Can you think of any reasons that the routine estimates would be lower? One reason is that, as we have seen, the completeness of reporting of ANC data is only about 92%. We shall see when we examine Domain 4 whether over-estimation of the denominator may be another reason.

   

   In our example, the match is poor for the DPT 3 coverage. The reason, over course, is the low completeness of the EPI data set.

   

   As shown by the figure below, the match is considerably better if we use as the numerator the data on Penta 3^rd doses that is reported on the HMIS form rather than the data on DPT 3 that is reported on the EPI form. Again, we note some tendency for the routine estimate to be slightly lower than the survey estimate.

   

10. Use the Tool to assess denominators (Section 4a)–

    Note: before the Tool can generate findings for section 4a, a couple of steps must be completed. Both of these steps are discussed in Annex 1:

    1. A new indicator or data element must be configured for the UN estimate of live births¹³.

    2. The denominator relations page of the Tool must be properly configured.

    “Domain 4” of a data quality desk review assesses the denominators used by DHIS2. Here we discuss findings related to section 4a which compares the official estimate of the live births to the U.N. estimate of live births.

    

    Here, with our example, we have identified one more reason why the estimates of coverage based on routine data (and based on the official estimates of pregnancies, live births or population < 1 year) are somewhat lower than the survey estimates – if we believe the U.N. estimate, the official estimate of live births may be too high. If we used the U.N. estimate as the denominator, then the routine coverage estimate would be higher.

Exercise 3: Use the Tool to prepare an Annual Data Quality Report

1. Generate an Annual Data Quality Review Report for 2019

   a. Click on Annual Report. Set Data to Core. Set Period to 2019. Set Preceding years for reference to as many preceding years as you have a core data set that is at least 50% complete. Set Disaggregation to the equivalent of district. Click Generate.

   b. Review the following sections of the detailed report. For each of these sections, in the box for Interpretations, type your interpretation and summarize your follow-up actions:

   1. 1a (completeness of reporting)
   2. Review and interpret the graph shown at the bottom of 1d (consistency of data set completeness over time
   3. 2a (extreme outliers)

   4. 2d (consistency of indicator values over time). Comment on at least the following two indicators:

      i. ANC 1^st visits

      ii. DPT 3^rd doses

   5. 2e (consistency between related indicators). Comment on at least the DPT 1 to DPT 3 dropout rate

Annex 1: How to configure Denominators, Denominator relations and External data comparisons

Before the Tool can generate findings for section 3a (external data comparisons) and 4a (consistency of denominator estimates) of the Annual Report the Tool must be configured properly. Configuration of Tool numerators is discussed in a separate document. This Annex explains how to configure Denominators, Denominator relations and External data comparisons.

1. New DHIS2 data elements must be configured for the survey estimates and the survey estimate data must be imported. This data is typically disaggregated by region. These data elements can be assigned to a new data element group with an appropriate name such as “External data”

2. A new data element or indicator must be configured for the UN estimate of live births.¹⁴ One approach would be to define a new indicator which calculated the live births by multiplying the official estimate of the total population times the U.N. estimate of the crude birth rate (available from https://data.worldbank.org/indicator ) divided by 1,000. This assumes that the official estimate of the total population is reliable. This new indicator can be assigned to an appropriate indicator group (either an existing group or a new one such as “Population estimates”. The other approach would be to create a new data element then for the UN estimate of live births then use the Data Entry screen to enter the value for each year. Only a single nationwide value is needed for each year – no disaggregation is required. This new data element can be assigned to the same new data element group that you created for the survey estimates.

3. Configure the denominators used by the Tool– Click on More, then Administration then Denominators.

   

   There are no denominators set up by default. The ones you see in the example above had to be added one-at-a-time by clicking on the Add button. A window will appear for configuring the denominator (see figure to the right). Notice that a special data element group has been configured for “Population estimates”. This includes the data element “Expected pregnant women”. Data for this data element have been imported as part of the standard configuration of DHIS2 denominators. Notice that “Lowest available level” has been set to “Region”. This is because reliable DHS survey estimates are not available below the level of region. Even if you configure other outputs of the Tool to disaggregate findings by district, the external comparison findings will only be disaggregated by region.

   

   Make sure that you add to this the following denominators:

   a) Expected pregnancies,

   b) Live births;

   c) Population < 1; and

   d) the new indicator or data element that you created for the U.N. estimate of live births.

   4.Configure the Tool to make external comparisons – Click on “More” then “Administration” then “External data comparison”.

   

   There are no default settings. The ones shown above had to be added one at a time by clicking on the Add button. A window will appear for configuring the comparison (see figure to the right). The DHS coverage estimates and the UN population estimates have been imported as part of the newly configured “External data” data set. These same data elements have also been assigned to a newly configured “External data” data element group. You see in the exmaple that this data element group has been selected, then “ANC 1 coverage (DHS 2014)” can be selected from the drop-down menu.

   

   The “Routine data numerator” must be selected from the list of numerators that have already been configured for the Tool. The “Routine data denominator” must be selected from the list of denominators already configured for the tool. Notice that the Survey level has been set to “Region”.

   

   5.Configure Denominator relations– When you click on More-Administration –Denominator relations, you should find that the table has one default row labeled “Total population – census to UN projection.

   

   To configure this relation, click on Edit. Edit the Name to read “Live births – official to UN estimate”. Leave Type set to “UN population projection”. Click on the down arrows to the left of Denominator A and Denominator B to select from the list. The list is limited to the denominators which you configured in step 3 above. For Denominator A, choose the official estimate of live births (or the best available equivalent). For Denominator B, choose the UN estimate of live births. Save your configuration.

   

   

Notes

1. Conduct further desk review.  For example, use DHIS2 to compare the suspicious value with the value for related data elements (e.g. OPV 3, DPT 1) for the same facility and same month;
2. Use DHIS2 to send a message to the person responsible for the data’
3. If you are at district level, you can inspect the paper form;
4. Telephone the person-in-charge at the health facility and ask them to check the paper form;
5. If you have authority, edit the value.