Data validation¶

Validation¶

To generate a data validation summary you can interact with the validation resource. The dataSet resource is optimized for data entry clients for validating a data set / form, and can be accessed like this:

GET /api/33/validation/dataSet/QX4ZTUbOt3a.json?pe=201501&ou=DiszpKrYNg8

In addition to validate rules based on data set, there are two additional methods for performing validation: Custom validation and Scheduled validation.

The first path variable is an identifier referring to the data set to validate. XML and JSON resource representations are supported. The response contains violations of validation rules. This will be extended with more validation types in the coming versions.

To retrieve validation rules which are relevant for a specific data set, meaning validation rules with formulas where all data elements are part of the specific data set, you can make a GET request to to validationRules resource like this:

GET /api/validationRules?dataSet=<dataset-id>

The validation rules have a left side and a right side, which is compared for validity according to an operator. The valid operator values are found in the table below.

The left side and right side expressions are mathematical expressions which can contain references to data elements and category option combinations on the following format:

${<dataelement-id>.<catoptcombo-id>}

The left side and right side expressions have a missing value strategy. This refers to how the system should treat data values which are missing for data elements / category option combination references in the formula in terms of whether the validation rule should be checked for validity or skipped. The valid missing value strategies are found in the table below.

Validation results¶

Validation results are persisted results of violations found during a validation analysis. If you choose "persist results" when starting or scheduling a validation analysis, any violations found will be stored in the database. When a result is stored in the database it will be used for 3 things:

1. Generating analytics based on the stored results.

2. Persisted results that have not generated a notification, will do so, once.

3. Keeping track of whether or not the result has generated a notification.

4. Skipping rules that have been already checked when running validation analysis.

This means if you don't persist your results, you will be unable to generate analytics for validation results, if checked, results will generate notifications every time it's found and running validation analysis might be slower.

Query validation results¶

The validation results persisted can be viewed at the following endpoint:

GET /api/33/validationResults

You can also inspect an individual result using the validation result id in this endpoint:

GET /api/33/validationResults/<id>

Validation results can also be filtered by following properties:

• Organisation Unit: ou=<UID>
• Validation Rule: vr=<UID>
• Period: pe=<ISO-expression>

Each of the above filter properties can occur multiple times, for example:

GET /api/36/validationResults?ou=jNb63DIHuwU&ou=RzgSFJ9E46G

Multiple values for the same filter are combined with OR, results have to match one of the given values.

If more then one filter properties is used these are combined with AND, results have to match one of the values for each of the properties.

For the period filter matching results have to overlap with any of the specified periods.

In addition the validation results can also be filtered on their creation date:

GET /api/36/validationResults?createdDate=<date>

This filter can be combined with any of the other filters.

Trigger validation result notifications¶

Validation results are sent out to the appropriate users once every day, but can also be manually triggered to run on demand using the following API endpoint:

POST /api/33/validation/sendNotifications

Only unsent results are sent using this endpoint.

Delete validation results¶

Validation results can be manually deleted by ID,

DELETE /api/36/validationResults/<id>

or using filters

DELETE /api/36/validationResults?<filters>

Supported filter parameters include:

• ou=<UID> to match all validation results of an organisation unit; multiple units combine OR when the parameter is provided more than once
• vr=<UID> to match all validation results of a validation rule; multiple rules combine OR when the parameter is provided more than once
• pe=<ISO-expression> to match all validation results related to a period that overlaps with the specified period
• created=<ISO-expression> to match all validation results that were created within the provided period
• notificationSent=<boolean> to match either only validation results for which a notification was or wasn't sent

If filters are combined all conditions have to be true (AND logic).

Some examples:

To delete all validation results related the organisation unit with UID NqwvaQC1ni4 for Q1 of 2020 use:

DELETE /api/36/validationResults?ou=NqwvaQC1ni4&pe=2020Q1

To delete all validation results that were created in week 1 of 2019 and for which notification has been sent use:

DELETE /api/36/validationResults?created=2019W1&notificationSent=true

Any delete operation will require the authority Perform maintenance tasks.

Outlier detection¶

The outlier detection endpoint allows for detecting outliers in aggregate data values.

GET /api/36/outlierDetection

This endpoint supports two algorithms for detecting outliers:

• Z-score: The z-score is defined as the absolute deviation between the score and mean divided by the standard deviation. A threshold parameter referring to the number of standard deviations from the mean must be specified with the z-score algorithm to define the upper and lower boundaries for what is considered an outlier value.
• Modified Z-score: Same as z-score except it uses the median instead of the mean as measure of central tendency. Parameters are same as for Z-score.
• Min-max: Min-max data element values refers to custom boundaries which can be inserted in DHIS 2 based on data element, org unit and category option combination.

The outlier values will be ordered according to significance, by default by the absolute deviation from the mean, with the most significant value first. This is helpful to quickly identify the outlier values which have the biggest impact on data quality and data analytics.

Request query parameters¶

The following query parameters are supported.

[*] You must specify either data sets with the ds parameter, which will include all data elements in the data sets, or specify data elements with the de parameter.

At least one data set or data element, start date and end date, and at least one organisation unit must be defined.

The startDate and endDate parameters are mandatory and refer to the time interval for which you want to detect outliers. The dataStartDate and dataEndDate parameters are optional and refer to the time interval for the data to use when calculating the mean and std dev, which are used to eventually calculate the z-score.

Usage and examples¶

Get outlier values using the default z-score algorithm:

GET /api/36/outlierDetection?ds=BfMAe6Itzgt&ds=QX4ZTUbOt3a
  &ou=O6uvpzGd5pu&ou=fdc6uOvgoji&startDate=2020-01-01&endDate=2020-12-31

Get outlier values using a specific algorithm and a specific threshold:

GET /api/36/outlierDetection?ds=BfMAe6Itzgt&ds=QX4ZTUbOt3a
  &ou=O6uvpzGd5pu&startDate=2020-01-01&endDate=2020-12-31
  &algorithm=Z_SCORE&threshold=2.5

Get outlier values ordered by z-score:

GET /api/36/outlierDetection?ds=BfMAe6Itzgt
  &ou=O6uvpzGd5pu&startDate=2020-01-01&endDate=2020-12-31
  &orderBy=Z_SCORE

Get the top 10 outlier values:

GET /api/36/outlierDetection?ds=BfMAe6Itzgt
  &ou=O6uvpzGd5pu&startDate=2020-01-01&endDate=2020-12-31
  &maxResults=10

Get outlier values with a defined interval for data to use when calculating the mean and std dev:

GET /api/36/outlierDetection?ds=BfMAe6Itzgt
  &ou=O6uvpzGd5pu&startDate=2020-01-01&endDate=2020-12-31
  &dataStartDate=2018-01-01&dataEndDate=2020-12-31

Get outlier values using the min-max algorithm:

GET /api/36/outlierDetection?ds=BfMAe6Itzgt&ds=QX4ZTUbOt3a
  &ou=O6uvpzGd5pu&ou=fdc6uOvgoji&startDate=2020-01-01&endDate=2020-12-31
  &algorithm=MIN_MAX

Response format¶

The following response formats are supported.

The response contains the following fields:

The mean, stdDev and zScore fields are only present when algorithm is Z_SCORE.

The response will look similar to this. The metadata section contains metadata for the request and response. The outlierValues section contains the outlier values.

{
  "metadata": {
    "algorithm": "Z_SCORE",
    "threshold": 2.5,
    "orderBy": "MEAN_ABS_DEV",
    "maxResults": 10,
    "count": 10
  },
  "outlierValues": [
    {
      "de": "rbkr8PL0rwM",
      "deName": "Iron Folate given at ANC 3rd",
      "pe": "202011",
      "ou": "Pae8DR7VmcL",
      "ouName": "MCH (Kakua) Static",
      "coc": "pq2XI5kz2BY",
      "cocName": "Fixed",
      "aoc": "HllvX50cXC0",
      "aocName": "default",
      "value": 9000.0,
      "mean": 1524.5555,
      "stdDev": 2654.4661,
      "absDev": 7475.4444,
      "zScore": 2.8161,
      "lowerBound": -5111.6097,
      "upperBound": 8160.7208,
      "followUp": false
    },
    {
      "de": "rbkr8PL0rwM",
      "deName": "Iron Folate given at ANC 3rd",
      "pe": "202010",
      "ou": "vELbGdEphPd",
      "ouName": "Jimmi CHC",
      "coc": "pq2XI5kz2BY",
      "cocName": "Fixed",
      "aoc": "HllvX50cXC0",
      "aocName": "default",
      "value": 8764.0,
      "mean": 1448.0833,
      "stdDev": 2502.3031,
      "absDev": 7315.9166,
      "zScore": 2.9236,
      "lowerBound": -4807.6745,
      "upperBound": 7703.8412,
      "followUp": false
    }
  ]
}

Constraints and validation¶

The following constraints apply during query validation. Each validation error has a corresponding error code.

Data analysis¶

Several resources for performing data analysis and finding data quality and validation issues are provided.

Note: This endpoint is deprecated and will be removed in 2.38. Use the outlierAnalysis endpoint instead.

Validation rule analysis¶

To run validation rules and retrieve violations:

GET /api/dataAnalysis/validationRules

The following query parameters are supported:

Sample output:

[{
    "validationRuleId": "kgh54Xb9LSE",
    "validationRuleDescription": "Malaria outbreak",
    "organisationUnitId": "DiszpKrYNg8",
    "organisationUnitDisplayName": "Ngelehun CHC",
    "organisationUnitPath": "/ImspTQPwCqd/O6uvpzGd5pu/YuQRtpLP10I/DiszpKrYNg8",
    "organisationUnitAncestorNames": "Sierra Leone / Bo / Badjia / ",
    "periodId": "201901",
    "periodDisplayName": "January 2019",
    "attributeOptionComboId": "HllvX50cXC0",
    "attributeOptionComboDisplayName": "default",
    "importance": "MEDIUM",
    "leftSideValue": 10.0,
    "operator": ">",
    "rightSideValue": 14.0
}, {
    "validationRuleId": "ZoG4yXZi3c3",
    "validationRuleDescription": "ANC 2 cannot be higher than ANC 1",
    "organisationUnitId": "DiszpKrYNg8",
    "organisationUnitDisplayName": "Ngelehun CHC",
    "organisationUnitPath": "/ImspTQPwCqd/O6uvpzGd5pu/YuQRtpLP10I/DiszpKrYNg8",
    "organisationUnitAncestorNames": "Sierra Leone / Bo / Badjia / ",
    "periodId": "201901",
    "periodDisplayName": "January 2019",
    "attributeOptionComboId": "HllvX50cXC0",
    "attributeOptionComboDisplayName": "default",
    "importance": "MEDIUM",
    "leftSideValue": 22.0,
    "operator": "<=",
    "rightSideValue": 19.0
}]

Standard deviation based outlier analysis¶

To identify data outliers based on standard deviations of the average value:

GET /api/dataAnalysis/stdDevOutlier

The following query parameters are supported:

Min/max value based outlier analysis¶

To identify data outliers based on min/max values:

GET /api/dataAnalysis/minMaxOutlier

The supported query parameters are equal to the std dev based outlier analysis resource described above.

Follow-up data analysis¶

To identify data marked for follow-up:

GET /api/dataAnalysis/followup

At least one data set or data element, start date and end date or period, and at least one organisation unit must be defined.

The following query parameters are supported.

[*] You must specify either data sets with the ds parameter, which will include all data elements in the data sets, or specify data elements with the de parameter. Equally, either startDate and endDate or period must be specified.

The startDate and endDate parameters refer to the time interval for which you want to detect outliers. If a period pe is provided instead the interval start and end is that of the period.

If no option combos coc are provided all data elements of numeric value type are considered.

Data integrity¶

The data integrity capabilities of the data administration module are available through the web API. This section describes how to run the data integrity process and retrieve the results. The specific details regarding each check are described in the user manual.

Listing the available data integrity checks¶

A description of the available checks is returned by a request to:

GET /api/dataIntegrity

[
    {
        "name": "data_elements_without_groups",
        "displayName": "Data elements lacking groups",
        "section": "Data Elements",
        "severity": "WARNING",
        "description": "Lists all data elements that have no data element groups",
        "issuesIdType": "dataElements",
        "isSlow": false
    }
]

The name member of the returned check elements is the identifier used for the checks parameter to declare the set of checks to run.

Note

Each check will indicate whether it may require significant time and resources to complete with the isSlow field. Users should be cautious about running these checks on production systems as they could lead to decreased performance. These checks can be run individually, but will not be run unless specifically requested.

Checks are grouped semantically by the section member and categorised in one of four severity levels:

The available checks can be filtered using the checks parameter.

GET /api/dataIntegrity?checks=<pattern1>,<pattern2>

One or more exact names or patterns using * as a wildcard can be provided.

Additional results can be filtered using a section parameter.

GET /api/dataIntegrity?section=Categories

The section filter will return all exact matches which have the specified section.

Running data integrity summaries¶

Since version 2.38, data integrity checks have two levels of specificity: - a summary level that provides an overview of the number of issues - a details level that provides a list of issues pointing to individual data integrity violations.

To trigger a summary analysis for a set of checks run:

POST /api/dataIntegrity/summary?checks=<name1>,<name2>

This triggers a job that runs the check(s) asynchronously. Individual check results will be returned to the application cache as soon as the check has completed.

Alternatively the list of checks can also be given as BODY of the POST request. This can be useful if the list becomes to long to be used in the URL.

To fetch the data integrity summary of the triggered check(s) use:

GET /api/dataIntegrity/summary?checks=<name1>,<name2>

When the checks parameter is omitted, all checks are fetched from the server cache.

The response is a "map" of check results, one for each check that has completed already. This information is cached for one hour or until the check is rerun.

To wait for the summary to be available in the cache a timeout in milliseconds can be added:

GET /api/dataIntegrity/summary?checks=<name1>,<name2>&timeout=500

An example of a summary response could look like:

{
  "<name1>": {
    "name": "<name1>",
    "displayName": "<displayName1>",
    "startTime": "2023-01-11T06:12:56.436",
    "finishedTime": "2023-01-11T06:12:57.021",
    "section": "...",
    "severity": "WARNING",
    "description": "...",
    "count": 12,
    "percentage": 2.3
  },
  "<name2>": {
    "name": "<name2>",
    "displayName": "<displayName2>",
    "startTime": "2023-01-11T06:12:57.345",
    "finishedTime": "2023-01-11T06:12:58.007",
    "section": "...",
    "severity": "WARNING",
    "description": "...",
    "count": 4,
    "percentage": 5.1
  }
}

Each summary response will contain the name, section, severity, description and optionally an introduction and recommendation.
Each summary contains the number of issues found in the count field. When possible, an optional percentage field will provide the percentage of objects with data integrity issues when compared to all objects of the same type. The startTime field indicates when the check was initiated. Using the finishedTime the duration which was required to execute the check can be calculated.

Should a check analysis fail due to programming error or unforeseen data inconsistencies both the summary and the details will have an error field describing the error that occurred. The count of any checks which failed will be set to -1. No percentage will be returned in such cases.

{
  "<name1>": {
    "name": "<name1>",
    "displayName": "<displayName1>",
    "finishedTime": "2022-02-15 14:55",
    "section": "...",
    "severity": "WARNING",
    "description": "...",
    "error": "what has happened",
    "issues": []
  }
}

Note

Each metadata check is run asynchronously on the server. Results will be returned as soon as each check completes. The safest way to ensure that you have retrieved the latest set of results which has been requested is to compare the timestamp of when the request was made with the finishedTime in the response.

To get a list of the names of checks that are currently being performed by the server use:

GET /api/dataIntegrity/summary/running

To get a list of the names of checks for which results are available already use:

GET /api/dataIntegrity/summary/completed

Running data integrity details¶

To run a selection of details checks first trigger them using a POST request:

POST /api/dataIntegrity/details?checks=<name1>,<name2>

Similar to the summary the list of checks can also be given as the POST body.

Then fetch the results from the cache using:

GET /api/dataIntegrity/details?checks=<name1>,<name2>&timeout=500

When the checks parameter is not provided, all checks which have not been marked as isSlow will be scheduled to be run on the server.

Omitting the timeout will not wait for results to be found in the cache, but instead not have a result for the requested check.

The /details response returns a map similar to the summary, but does not contain a count or percentage. Instead, a list of issues is returned.

{
  "<name1>": {
    "name": "<name1>",
    "displayName": "<displayName1>",
    "startTime": "2023-01-11T06:12:56.436",
    "finishedTime": "2023-01-11T06:12:57.021",
    "section": "...",
    "severity": "WARNING",
    "description": "...",
    "issuesIdType": "<object-type-plural>",
    "isSlow": false,
    "issues": [{
      "id": "<id-or-other-identifier>",
      "name": "<name-of-the-id-obj>",
      "comment": "optional plain text description or hint of the issue",
      "refs": ["<id1>", "<id2>"]
    }]
  },
  "<name2>": {
    "name": "<name2>",
    "displayName": "<displayName2>",
    "startTime": "2023-01-11T06:12:57.345",
    "finishedTime": "2023-01-11T06:12:58.007",
    "section": "...",
    "severity": "WARNING",
    "description": "...",
    "issuesIdType": "<object-type-plural>",
    "isSlow": false,
    "issues": []
  }
}

The comment and refs fields are optional for each issue. A comment may provide more context or insight into why this particular issue is regarded to be a data integrity problem. The refs list may also give the identifiers of other objects that contributed to the violation. The finishedTime field shows when the particular check finished processing on the server. The cache will store the result of each completed check for one hour.

Tip

A set of checks can also be specified using wild-cards. To include all checks with element in the name use checks=*element*. Like full names such patterns can be used in a comma-separated list and be mixed with full names as well. Duplicates will be eliminated. Also a check can be given by its code. A code consists of the first letters of each word in the name as upper case letter. For example, orgunits_invalid_geometry has the code OIG.

Similar to the summary a set of names of the currently performed and the already completed details checks can be obtained using:

GET /api/dataIntegrity/details/running
GET /api/dataIntegrity/details/completed

Complete data set registrations¶

This section is about complete data set registrations for data sets. A registration marks as a data set as completely captured.

Completing data sets¶

This section explains how to register data sets as complete. This is achieved by interacting with the completeDataSetRegistrations resource:

GET /api/33/completeDataSetRegistrations

The endpoint supports the POST method for registering data set completions. The endpoint is functionally very similar to the dataValueSets endpoint, with support for bulk import of complete registrations.

Importing both XML and JSON formatted payloads are supported. The basic format of this payload, given as XML in this example, is like so:

<completeDataSetRegistrations xmlns="http://dhis2.org/schema/dxf/2.0">
  <completeDataSetRegistration period="200810" dataSet="eZDhcZi6FLP"
    organisationUnit="qhqAxPSTUXp" attributeOptionCombo="bRowv6yZOF2" storedBy="imported"/>
  <completeDataSetRegistration period="200811" dataSet="eZDhcZi6FLP"
    organisationUnit="qhqAxPSTUXp" attributeOptionCombo="bRowv6yZOF2" storedBy="imported"/>
</completeDataSetRegistrations>

The storedBy attribute is optional (as it is a nullable property on the complete registration object). You can also optionally set the date property (time of registration) as an attribute. It the time is not set, the current time will be used.

The import process supports the following query parameters:

The idScheme, dataSetIdScheme, orgUnitIdScheme, attributeOptionComboIdScheme, dryRun and strategy (note the dissimilar naming to parameter importStrategy) can also be set as part of the payload. In case of XML these are attributes, in case of JSON these are members in the completeDataSetRegistrations node.

For example:

<completeDataSetRegistrations xmlns="http://dhis2.org/schema/dxf/2.0"
      orgUnitIdScheme="CODE">
    <completeDataSetRegistration period="200810" dataSet="eZDhcZi6FLP"
    organisationUnit="OU_559" attributeOptionCombo="bRowv6yZOF2" storedBy="imported"/>
</completeDataSetRegistrations>

Should both URL parameter and payload set a scheme the payload takes precedence.

Reading complete data set registrations¶

This section explains how to retrieve data set completeness registrations. We will be using the completeDataSetRegistrations resource. The query parameters to use are these:

The period, startDate, endDate, created and createdDuration parameters provide multiple ways to set the time dimension for the request, thus only one can be used. For example, it doesn't make sense to both set the start/end date and to set the periods.

An example request looks like this:

GET /api/33/completeDataSetRegistrations?dataSet=pBOMPrpg1QX&dataSet=pBOMPrpg1QX
  &startDate=2014-01-01&endDate=2014-01-31&orgUnit=YuQRtpLP10I
  &orgUnit=vWbkYPRmKyS&children=true

You can get the response in xml and json format. You can indicate which response format you prefer through the Accept HTTP header like in the example above. For xml you use application/xml; for json you use application/json.

Un-completing data sets¶

This section explains how you can un-register the completeness of a data set. To un-complete a data set you will interact with the completeDataSetRegistrations resource:

GET /api/33/completeDataSetRegistrations

This resource supports DELETE for un-registration. The following query parameters are supported: