Ověření dat¶

Ověření¶

To generate a data validation summary you can interact with the validation resource. The dataSet resource is optimized for data entry clients for validating a data set / form, and can be accessed like this:

GET /api/33/validation/dataSet/QX4ZTUbOt3a.json?pe=201501&ou=DiszpKrYNg8

In addition to validate rules based on data set, there are two additional methods for performing validation: Custom validation and Scheduled validation.

The first path variable is an identifier referring to the data set to validate. XML and JSON resource representations are supported. The response contains violations of validation rules. This will be extended with more validation types in the coming versions.

To retrieve validation rules which are relevant for a specific data set, meaning validation rules with formulas where all data elements are part of the specific data set, you can make a GET request to to validationRules resource like this:

GET /api/validationRules?dataSet=<dataset-id>

The validation rules have a left side and a right side, which is compared for validity according to an operator. The valid operator values are found in the table below.

Tabulka: Operátoři

The left side and right side expressions are mathematical expressions which can contain references to data elements and category option combinations on the following format:

${<dataelement-id>.<catoptcombo-id>}

The left side and right side expressions have a missing value strategy. This refers to how the system should treat data values which are missing for data elements / category option combination references in the formula in terms of whether the validation rule should be checked for validity or skipped. The valid missing value strategies are found in the table below.

Tabulka: Chybějící hodnotové strategie

Validation results¶

Validation results are persisted results of violations found during a validation analysis. If you choose "persist results" when starting or scheduling a validation analysis, any violations found will be stored in the database. When a result is stored in the database it will be used for 3 things:

1. Generování analytiky na základě uložených výsledků.

2. Persisted results that have not generated a notification, will do so, jednou.

3. Keeping track of whether or not the result has generated a notification.

4. Přeskakování pravidel, která již byla zkontrolována při spuštění validační analýza.

This means if you don't persist your results, you will be unable to generate analytics for validation results, if checked, results will generate notifications every time it's found and running validation analysis might be slower.

Query validation results¶

The validation results persisted can be viewed at the following endpoint:

GET /api/33/validationResults

You can also inspect an individual result using the validation result id in this endpoint:

GET /api/33/validationResults/<id>

Výsledky ověření lze filtrovat také podle následujících vlastností:

• Organizační jednotka: ou=<UID>
• Pravidlo ověření: vr=<UID>
• Období: pe=<ISO-expression>

Každá z výše uvedených vlastností filtru se může vyskytnout několikrát, například:

GET /api/36/validationResults?ou=jNb63DIHuwU&ou=RzgSFJ9E46G

Více hodnot pro stejný filtr je kombinováno s OR, výsledky musí odpovídat jedné z daných hodnot.

Pokud je použito více než jedné vlastnosti filtru, jsou kombinovány s AND, výsledky musí odpovídat jedné z hodnot pro každou z vlastností.

U filtru období se musí výsledky shody překrývat s některým ze zadaných období.

Kromě toho lze výsledky ověření filtrovat také k datu jejich vytvoření:

GET /api/36/validationResults?createdDate=<date>

Tento filtr lze kombinovat s jakýmkoli z ostatních filtrů.

Trigger validation result notifications¶

Validation results are sent out to the appropriate users once every day, but can also be manually triggered to run on demand using the following API endpoint:

POST /api/33/validation/sendNotifications

Pomocí tohoto koncového bodu se odesílají pouze neodeslané výsledky.

Delete validation results¶

Výsledky ověření lze ručně odstranit pomocí ID,

DELETE /api/36/validationResults/<id>

nebo pomocí filtrů

DELETE /api/36/validationResults?<filters>

Mezi podporované parametry filtru patří:

• ou=<UID>, aby odpovídal všem výsledkům ověření organizační jednotky; více jednotek kombinuje OR, pokud je parametr poskytnut více než jednou
• vr=<UID>, aby odpovídal všem výsledkům ověření pravidla ověření; kombinuje více pravidel NEBO pokud je parametr poskytnut více než jednou
• pe=<ISO-expression>, aby se shodovaly všechny výsledky ověření týkající se období, které se překrývá se zadaným obdobím
• created=<ISO-expression>, aby odpovídalo všem výsledkům ověření, které byly vytvořeny v zadaném období
• notificationSent=<boolean>, aby odpovídal pouze výsledkům ověření, pro které bylo nebo nebylo odesláno oznámení

If filters are combined, all conditions have to be true (AND logic).

Nějaké příklady:

Chcete-li odstranit všechny výsledky ověření související s organizační jednotkou s UID NqwvaQC1ni4 pro 1. čtvrtletí roku 2020, použijte:

DELETE /api/36/validationResults?ou=NqwvaQC1ni4&pe=2020Q1

Chcete-li odstranit všechny výsledky ověření, které byly vytvořeny v 1. týdnu 2019 a pro které bylo odesláno oznámení, použijte:

DELETE /api/36/validationResults?created=2019W1&notificationSent=true

Jakákoli operace odstranění bude vyžadovat oprávnění Provádět úlohy údržby.

Detekce odlehlých hodnot¶

Koncový bod detekce odlehlých hodnot umožňuje detekci odlehlých hodnot v agregovaných hodnotách dat.

GET /api/36/outlierDetection

Tento koncový bod podporuje dva algoritmy pro zjišťování odlehlých hodnot:

• Z-skóre: Z-skóre je definováno jako absolutní odchylka mezi skóre a průměrem děleno standardní odchylkou. Parametr prahové hodnoty vztahující se k počtu standardních odchylek od střední hodnoty musí být zadán pomocí algoritmu z-skóre, aby bylo možné definovat horní a dolní hranici pro to, co je považováno za odlehlou hodnotu.
• Upravené Z-skóre: Stejné jako z-skóre s tím rozdílem, že jako měřítko centrální tendence používá medián místo průměru. Parametry jsou stejné jako u Z-skóre.
• Min-max: Min-max data element values refers to custom boundaries which can be inserted in DHIS2 based on data element, org unit and category option combination.
• Invalid numeric values: This algorithm detects data values which cannot be cast to numeric values. This usually occurs when data values have been entered with a very large number of digits.

The outlier values will be ordered according to significance, by default by the absolute deviation from the mean, with the most significant value first. This is helpful to quickly identify the outlier values which have the biggest impact on data quality and data analytics.

Parametry požadavku databázového dotazu¶

Jsou podporovány následující parametry dotazu.

[*] Musíte zadat buď datové sady s parametrem ds, který bude zahrnovat všechny datové prvky v datových sadách, nebo specifikovat datové prvky s parametrem de.

Musí být definován alespoň jeden datový soubor nebo datový prvek, datum zahájení a datum ukončení a alespoň jedna organizační jednotka.

Parametry startDate a endDate jsou povinné a odkazují na časový interval, ve kterém chcete zjistit odlehlé hodnoty. Parametry dataStartDate a dataEndDate jsou volitelné a odkazují na časový interval, který mají data použít při výpočtu střední hodnoty a std dev, které se použijí k případnému výpočtu z-skóre.

Použití a příklady¶

Získejte hodnoty odlehlých hodnot pomocí výchozího algoritmu z-skóre:

GET /api/36/outlierDetection?ds=BfMAe6Itzgt&ds=QX4ZTUbOt3a
  &ou=O6uvpzGd5pu&ou=fdc6uOvgoji&startDate=2020-01-01&endDate=2020-12-31

Získejte hodnoty odlehlých hodnot pomocí konkrétního algoritmu a konkrétní prahové hodnoty:

GET /api/36/outlierDetection?ds=BfMAe6Itzgt&ds=QX4ZTUbOt3a
  &ou=O6uvpzGd5pu&startDate=2020-01-01&endDate=2020-12-31
  &algorithm=Z_SCORE&threshold=2.5

Získejte hodnoty odlehlých hodnot seřazené podle z-skóre:

GET /api/36/outlierDetection?ds=BfMAe6Itzgt
  &ou=O6uvpzGd5pu&startDate=2020-01-01&endDate=2020-12-31
  &orderBy=Z_SCORE

Získejte 10 nejlepších hodnot odlehlých hodnot:

GET /api/36/outlierDetection?ds=BfMAe6Itzgt
  &ou=O6uvpzGd5pu&startDate=2020-01-01&endDate=2020-12-31
  &maxResults=10

Získejte odlehlé hodnoty s definovaným intervalem pro data, která se mají použít při výpočtu střední hodnoty a standardní hodnoty:

GET /api/36/outlierDetection?ds=BfMAe6Itzgt
  &ou=O6uvpzGd5pu&startDate=2020-01-01&endDate=2020-12-31
  &dataStartDate=2018-01-01&dataEndDate=2020-12-31

Získejte hodnoty odlehlých hodnot pomocí algoritmu min-max:

GET /api/36/outlierDetection?ds=BfMAe6Itzgt&ds=QX4ZTUbOt3a
  &ou=O6uvpzGd5pu&ou=fdc6uOvgoji&startDate=2020-01-01&endDate=2020-12-31
  &algorithm=MIN_MAX

Formát odpovědi¶

Podporovány jsou následující formáty odpovědí.

Odpověď obsahuje následující pole:

Pole mean, stdDev a zScore jsou přítomny pouze když algorithm je Z_SCORE.

Odpověď bude vypadat podobně. Sekce metadata obsahuje metadata pro požadavek a odpověď. Sekce outlierValues obsahuje hodnoty odlehlých hodnot.

{
  "metadata": {
    "algorithm": "Z_SCORE",
    "threshold": 2.5,
    "orderBy": "MEAN_ABS_DEV",
    "maxResults": 10,
    "count": 10
  },
  "outlierValues": [
    {
      "de": "rbkr8PL0rwM",
      "deName": "Iron Folate given at ANC 3rd",
      "pe": "202011",
      "ou": "Pae8DR7VmcL",
      "ouName": "MCH (Kakua) Static",
      "coc": "pq2XI5kz2BY",
      "cocName": "Fixed",
      "aoc": "HllvX50cXC0",
      "aocName": "default",
      "value": 9000.0,
      "mean": 1524.5555,
      "stdDev": 2654.4661,
      "absDev": 7475.4444,
      "zScore": 2.8161,
      "lowerBound": -5111.6097,
      "upperBound": 8160.7208,
      "followUp": false
    },
    {
      "de": "rbkr8PL0rwM",
      "deName": "Iron Folate given at ANC 3rd",
      "pe": "202010",
      "ou": "vELbGdEphPd",
      "ouName": "Jimmi CHC",
      "coc": "pq2XI5kz2BY",
      "cocName": "Fixed",
      "aoc": "HllvX50cXC0",
      "aocName": "default",
      "value": 8764.0,
      "mean": 1448.0833,
      "stdDev": 2502.3031,
      "absDev": 7315.9166,
      "zScore": 2.9236,
      "lowerBound": -4807.6745,
      "upperBound": 7703.8412,
      "followUp": false
    }
  ]
}

Omezení a validace¶

Během ověřování dotazu platí následující omezení. Každá chyba ověření má odpovídající kód chyby.

Analýza dat¶

Several resources for performing data analysis and finding data quality and validation issues are provided.

Poznámka: Tento koncový bod je zastaralý a bude odstraněn ve verzi 2.38. Místo toho použijte koncový bod outlierAnalysis.

Analýza ověřovacích pravidel¶

Spuštění ověřovacích pravidel a načtení porušení pravidel:

GET /api/dataAnalysis/validationRules

Jsou podporovány následující parametry dotazu:

Tabulka: Parametry dotazu analýzy ověřovacích pravidel

| Parametr dotazu | Popis | Možnost | Požadované | Výchozí | |---|---|---|---| | vrg | Skupina ověřovacích pravidel | ID | false | If omitted, all validation rule groups will be used | | ou | Organizační jednotka | ID |true | | | startDate | Start date for the time span | Date (yyyy-MM-dd) | false | Dnes | | endDate | End date for the time span | Date (yyyy-MM-dd) | false | Dnes | | trvalé | Zda přetrvávat porušení v systému | false | true | false | false | | oznámení | Zda se mají zasílat oznámení o porušení | false | true | false | false | | maxResults | Max limit for the output | Integer, greater than zero. Maximum as specified by system setting keyDataQualityMaxLimit| false | 500 |

Sample POST body request:

{
    "startDate":"2024-01-01",
    "endDate":"2025-04-10",
    "ou":"ImspTQPwCqd",
    "notification":false,
    "persist":false,
    "vrg":"UP1lctvalPn",
    "maxResults": 500
}

Sample output:
```json
[{
    "validationRuleId": "kgh54Xb9LSE",
    "validationRuleDescription": "Malaria outbreak",
    "organisationUnitId": "DiszpKrYNg8",
    "organisationUnitDisplayName": "Ngelehun CHC",
    "organisationUnitPath": "/ImspTQPwCqd/O6uvpzGd5pu/YuQRtpLP10I/DiszpKrYNg8",
    "organisationUnitAncestorNames": "Sierra Leone / Bo / Badjia / ",
    "periodId": "201901",
    "periodDisplayName": "January 2019",
    "attributeOptionComboId": "HllvX50cXC0",
    "attributeOptionComboDisplayName": "default",
    "importance": "MEDIUM",
    "leftSideValue": 10.0,
    "operator": ">",
    "rightSideValue": 14.0
}, {
    "validationRuleId": "ZoG4yXZi3c3",
    "validationRuleDescription": "ANC 2 cannot be higher than ANC 1",
    "organisationUnitId": "DiszpKrYNg8",
    "organisationUnitDisplayName": "Ngelehun CHC",
    "organisationUnitPath": "/ImspTQPwCqd/O6uvpzGd5pu/YuQRtpLP10I/DiszpKrYNg8",
    "organisationUnitAncestorNames": "Sierra Leone / Bo / Badjia / ",
    "periodId": "201901",
    "periodDisplayName": "January 2019",
    "attributeOptionComboId": "HllvX50cXC0",
    "attributeOptionComboDisplayName": "default",
    "importance": "MEDIUM",
    "leftSideValue": 22.0,
    "operator": "<=",
    "rightSideValue": 19.0
}]

Analýza odlehlých hodnot založená na směrodatné odchylce¶

To identify data outliers based on standard deviations of the average value:

GET /api/dataAnalysis/stdDevOutlier

Jsou podporovány následující parametry dotazu:

Tabulka: Parametry dotazu analýzy odlehlých hodnot standardní odchylky

Analýza odlehlých hodnot založená na minimální / maximální hodnotě¶

Identifikace odlehlých dat na základě min / max hodnot:

GET /api/dataAnalysis/minMaxOutlier

The supported query parameters are equal to the std dev based outlier analysis resource described above.

Následná analýza dat¶

Identifikace údajů označených pro následnou kontrolu:

GET /api/dataAnalysis/followup

Musí být definována alespoň jedna datová sada nebo datový prvek, počáteční a koncové datum nebo období a alespoň jedna organizační jednotka.

Jsou podporovány následující parametry dotazu.

[*] You must specify either data sets with the ds parameter, which will include all data elements in the data sets, or specify data elements with the de parameter. Equally, either startDate and endDate or period must be specified.

The startDate and endDate parameters refer to the time interval for which you want to detect outliers. If a period pe is provided instead the interval start and end is that of the period.

Pokud nejsou zadány žádné kombinace voleb coc, berou se v úvahu všechny datové prvky typu číselné hodnoty.

Integrita dat¶

The data integrity capabilities of the data administration module are available through the web API. This section describes how to run the data integrity process and retrieve the results. The specific details regarding each check are described in the user manual.

Výpis dostupných kontrol integrity dat¶

A description of the available checks is returned by a request to:

GET /api/dataIntegrity

[
    {
        "name": "data_elements_without_groups",
        "displayName": "Data elements lacking groups",
        "section": "Data Elements",
        "severity": "WARNING",
        "description": "Lists all data elements that have no data element groups",
        "issuesIdType": "dataElements",
        "isSlow": false
    }
]

The name member of the returned check elements is the identifier used for the checks parameter to declare the set of checks to run.

Note

Each check will indicate whether it may require significant time and resources to complete with the isSlow field. Users should be cautious about running these checks on production systems as they could lead to decreased performance. These checks can be run individually, but will not be run unless specifically requested.

Checks are grouped semantically by the section member and categorised in one of four severity levels:

Dostupné kontroly lze filtrovat pomocí parametru checks.

GET /api/dataIntegrity?checks=<pattern1>,<pattern2>

One or more exact names or patterns using * as a wildcard can be provided.

Additional results can be filtered using a section parameter.

GET /api/dataIntegrity?section=Categories

The section filter will return all exact matches which have the specified section.

Furthermore, to filter (select) only checks marked as isSlow use slow=true,

GET /api/dataIntegrity?slow=true

or to filter (select) only checks that are not performed via database query (programmed checks) use programmatic=true:

GET /api/dataIntegrity?programmatic=true

The slow, programmatic and section filters can be combined in which case all conditions must be met.

Running data integrity summaries¶

Since version 2.38, data integrity checks have two levels of specificity:

• a summary level that provides an overview of the number of issues
• a details level that provides a list of issues pointing to individual data integrity violations.

Chcete-li spustit souhrnnou analýzu pro sadu kontrol:

POST /api/dataIntegrity/summary?checks=<name1>,<name2>

This triggers a job that runs the check(s) asynchronously. Individual check results will be returned to the application cache as soon as the check has completed.

Alternatively the list of checks can also be given as BODY of the POST request. This can be useful if the list becomes to long to be used in the URL.

Chcete-li načíst souhrn dat o integritě spuštěných kontrol(y), použijte:

GET /api/dataIntegrity/summary?checks=<name1>,<name2>

When the checks parameter is omitted, all checks are fetched from the server cache.

The response is a "map" of check results, one for each check that has completed already. This information is cached for one hour or until the check is rerun.

To wait for the summary to be available in the cache a timeout in milliseconds can be added:

GET /api/dataIntegrity/summary?checks=<name1>,<name2>&timeout=500

An example of a summary response could look like:

{
  "<name1>": {
    "name": "<name1>",
    "displayName": "<displayName1>",
    "startTime": "2023-01-11T06:12:56.436",
    "finishedTime": "2023-01-11T06:12:57.021",
    "section": "...",
    "severity": "WARNING",
    "description": "...",
    "count": 12,
    "percentage": 2.3
  },
  "<name2>": {
    "name": "<name2>",
    "displayName": "<displayName2>",
    "startTime": "2023-01-11T06:12:57.345",
    "finishedTime": "2023-01-11T06:12:58.007",
    "section": "...",
    "severity": "WARNING",
    "description": "...",
    "count": 4,
    "percentage": 5.1
  }
}

Each summary response will contain the name, section, severity, description and optionally an introduction and recommendation.
Each summary contains the number of issues found in the count field. When possible, an optional percentage field will provide the percentage of objects with data integrity issues when compared to all objects of the same type. The startTime field indicates when the check was initiated. Using the finishedTime the duration which was required to execute the check can be calculated.

Should a check analysis fail due to programming error or unforeseen data inconsistencies both the summary and the details will have an error field describing the error that occurred. The count of any checks which failed will be set to -1. No percentage will be returned in such cases.

{
  "<name1>": {
    "name": "<name1>",
    "displayName": "<displayName1>",
    "finishedTime": "2022-02-15 14:55",
    "section": "...",
    "severity": "WARNING",
    "description": "...",
    "error": "what has happened",
    "issues": []
  }
}

Note

Each metadata check is run asynchronously on the server. Results will be returned as soon as each check completes. The safest way to ensure that you have retrieved the latest set of results which has been requested is to compare the timestamp of when the request was made with the finishedTime in the response.

To get a list of the names of checks that are currently being performed by the server use:

GET /api/dataIntegrity/summary/running

To get a list of the names of checks for which results are available already use:

GET /api/dataIntegrity/summary/completed

Retrieving completed checks as Prometheus metrics¶

Metadata integrity checks which are present in the cache, can be retrieved in the Prometheus metrics format by making a request to : GET /api/dataIntegrity/metrics

The response should return a plain text format in the Prometheus plain text exposition format. Several metrics are available for each data integrity check. - Count: A count of the number of issues identified by the metadata check. - Percentage: When available, provides a percentage of the objects which have been identified by the check relative to a baseline. For instance, if check "Organisation units with trailing spaces" has a percent of 2.13, the percentage is calculated by dividing the number of organisation units with trailing spaces by the total number of organisation units. Note that this percentage may not be available for all metadata integrity checks. - Duration: Number of milliseconds that the check took to execute the last time it was run.

An example of the output of this endpoint is provided below:

# HELP dhis_data_integrity_check_count Data integrity check counts { #help-dhis_data_integrity_check_count-data-integrity-check-counts } 
# TYPE dhis_data_integrity_check_count gauge { #type-dhis_data_integrity_check_count-gauge } 
dhis_data_integrity_check_count{check="orgunits_invalid_geometry"} 1
dhis_data_integrity_check_count{check="user_groups_scarce"} 0
dhis_data_integrity_check_count{check="indicator_no_analysis"} 13
# HELP dhis_data_integrity_check_percentage Data integrity check percentages { #help-dhis_data_integrity_check_percentage-data-integrity-check-percentages } 
# TYPE dhis_data_integrity_check_percentage gauge { #type-dhis_data_integrity_check_percentage-gauge } 
dhis_data_integrity_check_percentage{check="orgunits_invalid_geometry"} 0.13054830287206268
dhis_data_integrity_check_percentage{check="user_groups_scarce"} 0.0
dhis_data_integrity_check_percentage{check="indicator_no_analysis"} 16.0
# HELP dhis_data_integrity_check_duration Data integrity check durations { #help-dhis_data_integrity_check_duration-data-integrity-check-durations } 
# TYPE dhis_data_integrity_check_duration gauge { #type-dhis_data_integrity_check_duration-gauge } 
dhis_data_integrity_check_duration{check="orgunits_invalid_geometry"} 11
dhis_data_integrity_check_duration{check="user_groups_scarce"} 1
dhis_data_integrity_check_duration{check="indicator_no_analysis"} 0

Data integrity checks which are not currently in the cache will not be returned by this endpoint. A request would need to be made to the /summary endpoint to trigger the checks to be run or alternatively through a scheduled job.

Running data integrity details¶

To run a selection of details checks first trigger them using a POST request:

POST /api/dataIntegrity/details?checks=<name1>,<name2>

Similar to the summary the list of checks can also be given as the POST body.

Poté načtěte výsledky z mezipaměti pomocí:

GET /api/dataIntegrity/details?checks=<name1>,<name2>&timeout=500

When the checks parameter is not provided, all checks which have not been marked as isSlow will be scheduled to be run on the server.

Omitting the timeout will not wait for results to be found in the cache, but instead not have a result for the requested check.

The /details response returns a map similar to the summary, but does not contain a count or percentage. Instead, a list of issues is returned.

{
  "<name1>": {
    "name": "<name1>",
    "displayName": "<displayName1>",
    "startTime": "2023-01-11T06:12:56.436",
    "finishedTime": "2023-01-11T06:12:57.021",
    "section": "...",
    "severity": "WARNING",
    "description": "...",
    "issuesIdType": "<object-type-plural>",
    "isSlow": false,
    "issues": [{
      "id": "<id-or-other-identifier>",
      "name": "<name-of-the-id-obj>",
      "comment": "optional plain text description or hint of the issue",
      "refs": ["<id1>", "<id2>"]
    }]
  },
  "<name2>": {
    "name": "<name2>",
    "displayName": "<displayName2>",
    "startTime": "2023-01-11T06:12:57.345",
    "finishedTime": "2023-01-11T06:12:58.007",
    "section": "...",
    "severity": "WARNING",
    "description": "...",
    "issuesIdType": "<object-type-plural>",
    "isSlow": false,
    "issues": []
  }
}

The comment and refs fields are optional for each issue. A comment may provide more context or insight into why this particular issue is regarded to be a data integrity problem. The refs list may also give the identifiers of other objects that contributed to the violation. The finishedTime field shows when the particular check finished processing on the server. The cache will store the result of each completed check for one hour.

Tip

A set of checks can also be specified using wild-cards. To include all checks with element in the name use checks=*element*. Like full names such patterns can be used in a comma-separated list and be mixed with full names as well. Duplicates will be eliminated. Also a check can be given by its code. A code consists of the first letters of each word in the name as upper case letter. For example, orgunits_invalid_geometry has the code OIG.

Similar to the summary a set of names of the currently performed and the already completed details checks can be obtained using:

GET /api/dataIntegrity/details/running
GET /api/dataIntegrity/details/completed

Custom Data Integrity Checks¶

Users of DHIS2 can now create and supply their own Data Integrity Checks. This can be useful if users want to avail of this functionality and extend upon the supplied set of core data integrity checks.

Tip

Users are also encouraged to share their custom checks with others by opening a pull request in the dhis2-core repository containing their .yaml file(s). Please select platform-backend as reviewer to put the PR on our radar early on. The team will take care of checking and linking the check correctly, so it becomes part of the provided suite of checks with the next release.

An example of a custom check could be for determining if certain users are members of specific user groups. This type of check would be very specific to an implementation, and not generally applicable across all installs. These types of metadata checks can be used to extend the default checks which are included with DHIS2.

Custom checks can be implemented by satisfying the following requirements, each of which we will go into detail: - Supplying your own list of custom data integrity checks in a list file named custom-data-integrity-checks.yaml in your DHIS2_HOME directory - Having a directory named custom-data-integrity-checks in your DHIS2_HOME directory - Supplying your valid custom data integrity check yaml files

Custom Data Integrity Check List File¶

DHIS2 will only try to load data integrity files when they are needed. e.g. when making a call to view all data integrity checks:

GET /api/dataIntegrity

DHIS2 will look for a file named custom-data-integrity-checks.yaml in your DHIS2_HOME directory when loading data integrity files. If you are not using custom checks and the file is not present, a warning log like this will be present:

08:29:57.729  WARN o.h.d.d.DataIntegrityYamlReader: Failed to load data integrity check from YAML. Error message `{DHIS2_HOME}/custom-data-integrity-checks.yaml (No such file or directory)

If you are implementing custom data integrity checks then this file must be present. To see what the core data integrity checks file looks like as an example, check out this file.

The custom-data-integrity-checks.yaml file should list all of your custom data integrity checks. As an example, it could look something like this:

checks:
  - categories/my_custom_check.yaml
  - users/my_user_group_check.yaml
  - base_check.yaml

Check names in this file can be preceded with a directory name for logical grouping. From the 3 example checks listed above, the directory structure should look like this:

├── DHIS2_HOME
│   ├── dhis.conf
│   ├── custom-data-integrity-checks.yaml
│   ├── custom-data-integrity-checks
│   │   ├── categories
│   │   │   ├── my_custom_check.yaml
│   │   ├── users
│   │   │   ├── my_user_group_check.yaml
│   │   ├── base_check.yaml

Name and Code constraints¶

Each data integrity check name and code must be unique. If there are any clashes then the violating custom check will not be loaded.

Note

System data integrity checks are always loaded first. Any name or code clashes resulting from custom checks will not affect these core system checks.

An example data integrity check yaml file is located here for reference. Note the name property.

The data integrity code is calculated dynamically by using the first letter of each word in the name. Some examples:

If there is a name clash, a warning log like this will be present:

09:48:43.138  WARN o.h.d.d.DefaultDataIntegrityService: Data Integrity Check `my_custom_check` not added as a check with that name already exists

If there is a code clash, a warning log like this will be present:

09:48:43.138  WARN o.h.d.d.DefaultDataIntegrityService: Data Integrity Check `my_custom_check` not added as a check with the code `MCC` already exists

Data Integrity Check Schema¶

A data integrity check file must comply with this JSON schema. If a check does not comply with the schema then a warning like this will be present:

09:48:43.136  WARN o.h.d.d.DataIntegrityYamlReader: JsonSchema validation errors found for Data Integrity Check `categories/my_custom_check.yaml`. Errors: [$.name: is missing but it is required]

Any schema violations must be fixed before that check can be loaded and used.

If a data integrity check file contains invalid yaml then a warning log like this could be present:

10:30:37.858  WARN o.h.d.d.DataIntegrityYamlReader: JsonSchema validation errors found for Data Integrity Check `my_custom_check.yaml`. Errors: [$: string found, object expected]

To view and use the custom checks please refer to the main Data Integrity section

Note

It is recommended to follow any naming and format conventions seen in the provided examples above when implementing your own custom checks to help avoid any issues

Data Integrity File¶

Details of the data integrity check yaml file, taken from the JSON schema file

Example custom data integrity check¶

An example of a custom check could be for determining if users have an email. Emails are useful to be able to communicate with users and sent them notifications, as well as password recovery. So, in some instllations of DHIS2, it could be a policy that all users should have emails. An example of this type of custom check is shown below.

---
name: users_should_have_emails
description: Users should have emails.
section: Users
section_order: 6
summary_sql: >-
  WITH users_no_email as (
  SELECT uid,username from
  userinfo where email IS NULL)
  SELECT COUNT(*) as value,
  100*COUNT(*) / NULLIF( ( select COUNT(*) from userinfo), 0) as percent
  from users_no_email;
details_sql: >-
  WITH users_no_email as (
  SELECT uid,username from
  userinfo where email IS NULL)
  SELECT uid,username as from users_no_email;
severity: WARNING
introduction: >
  Users should have defined emails. This is important for password recovery and to be able
  to send notifications to users.
recommendation: >
  Make sure that all users have defined emails.
details_id_type: users

More examples of different types of metadata integrity checks can be found in the DHIS2 source code here.

Dokončená registrace datových sad¶

This section is about complete data set registrations for data sets. A registration marks as a data set as completely captured.

Dokončení datových sad¶

This section explains how to register data sets as complete. This is achieved by interacting with the completeDataSetRegistrations resource:

GET /api/33/completeDataSetRegistrations

The endpoint supports the POST method for registering data set completions. The endpoint is functionally very similar to the dataValueSets endpoint, with support for bulk import of complete registrations.

Importing both XML and JSON formatted payloads are supported. The basic format of this payload, given as XML in this example, is like so:

<completeDataSetRegistrations xmlns="http://dhis2.org/schema/dxf/2.0">
  <completeDataSetRegistration period="200810" dataSet="eZDhcZi6FLP"
    organisationUnit="qhqAxPSTUXp" attributeOptionCombo="bRowv6yZOF2" storedBy="imported"/>
  <completeDataSetRegistration period="200811" dataSet="eZDhcZi6FLP"
    organisationUnit="qhqAxPSTUXp" attributeOptionCombo="bRowv6yZOF2" storedBy="imported"/>
</completeDataSetRegistrations>

The storedBy attribute is optional (as it is a nullable property on the complete registration object). You can also optionally set the date property (time of registration) as an attribute. It the time is not set, the current time will be used.

Proces importu podporuje následující parametry dotazu:

Tabulka: Parametry dotazu na registraci kompletní datové sady

The idScheme, dataSetIdScheme, orgUnitIdScheme, attributeOptionComboIdScheme, dryRun and strategy (note the dissimilar naming to parameter importStrategy) can also be set as part of the payload. In case of XML these are attributes, in case of JSON these are members in the completeDataSetRegistrations node.

Například:

<completeDataSetRegistrations xmlns="http://dhis2.org/schema/dxf/2.0"
      orgUnitIdScheme="CODE">
    <completeDataSetRegistration period="200810" dataSet="eZDhcZi6FLP"
    organisationUnit="OU_559" attributeOptionCombo="bRowv6yZOF2" storedBy="imported"/>
</completeDataSetRegistrations>

Should both URL parameter and payload set a scheme the payload takes precedence.

Čtení registrací úplných datových sad¶

This section explains how to retrieve data set completeness registrations. We will be using the completeDataSetRegistrations resource. The query parameters to use are these:

Tabulka: Parametry dotazu sady hodnot dat

Parametry period, startDate, endDate, created a createdDuration umožňují nastavit časový rozměr požadavku více způsoby, a proto je třeba použít pouze jeden z nich. Například nemá smysl nastavovat jak datum začátku / konce, tak období.

Příklad požadavku vypadá takto:

GET /api/33/completeDataSetRegistrations?dataSet=pBOMPrpg1QX
  &startDate=2014-01-01&endDate=2014-01-31&orgUnit=YuQRtpLP10I
  &orgUnit=vWbkYPRmKyS&children=true

You can get the response in xml and json format. You can indicate which response format you prefer through the Accept HTTP header like in the example above. For xml you use application/xml; for json you use application/json.

Zrušení dokončení datových sad¶

Tato část vysvětluje, jak můžete zrušit registraci úplnosti datové sady. Chcete-li zrušit dokončení datové sady, budete pracovat s prostředkem completeDataSetRegistrations:

GET /api/33/completeDataSetRegistrations

This resource supports DELETE for un-registration. The following query parameters are supported:

Tabulka: Parametry dotazu na registraci kompletní datové sady