Annual PEPFAR MER Update

Notes:

• PEPFAR Monitoring, Evaluation, and Reporting (MER) reference indicators are released annually in a PDF document, providing narrative around the indicators to be reported by PEPFAR sites.
• The names of Codelists change each year -- the current codelists do not have a period in the name, whereas older lists do
• Codelists may be reused in more than one year

Export latest metadata from DATIM DHIS2

• This step should not be done until the new MER Guidance has been released in DATIM (usually happens the first week of January)
• Open settings.py and set DATIM_USERNAME and DATIM_PASSWORD to credentials that have metadata API access
• Run python export_datim_metadata.py and just sit back and relax while it retreives the exports
• Output files are saved to data/*

Prepare IHUB export

• The IHUB team is able to provide an export of IHUB's derived data elements and their rules as an Excel spreadsheet (eg ihub_mer_derived_data_rules_20210108.xlsx)
• After receiving the IHUB export, open the file in Excel and save as a CSV in the input data folder (eg data/ihub_mer_derived_data_rules_20210108.csv)
  • Note that the export file may have an empty row at the top -- this should be removed first so that the column headers are treated correctly in python

Update settings.py for the new period

• Update filenames to latest exports from the previous steps
• Update INPUT_PERIODS and OUTPUT_PERIODS (eg to include FY21)
• Update MSP_ORG_ID (eg PEPFAR, or PEPFAR-Test10 for a test import) and MSP_SOURCE_ID (eg MER)
  • An org will be created for MSP_ORG_ID if it does not already exist, otherwise it will not be updated
  • If a source corresponding to MSP_SOURCE_ID already exists, it will be dropped and recreated for the new import

Prepare codelists

• Open Google Spreadsheet:
• Update spreadsheet with latest codelists from ZenDesk: https://datim.zendesk.com/hc/en-us/articles/115002334246-DATIM-Data-Import-and-Exchange-Resources
• Add CSV of exported dataSets into spreadsheet- columns must be in this order: id, code, shortName, name, description, href, periodType
• Update VLOOKUP cells to re-populate code, name, shortName and description
• Export spreadsheet as a CSV and save into the data folder (eg data/codelists_RT_FY16_21_20210106.csv)
• Open settings.py and make the following changes:
  • Update FILENAME_DATIM_CODELISTS to match the filename of the CSV exported in the previous step (eg data/codelists_RT_FY16_21_20210106.csv)
  • Update FILENAME_DATIM_CODELISTS_WITH_EXPORT with the desired filename that will be generated in the next step (eg data/codelist_collections_with_exports_FY16_21_20210106.json)
• Run python save_codelists_to_file.py to retrieve codelist exports from DATIM DHIS2 and save along with the collection definitions

Prepare reference indicators from the newly released MER guidance

• For each MER reference indicator (e.g. AGYW_PREV), there is a row in an import spreadsheet, which has columns for OCL attributes, including common concept attributes (e.g. Name, Description, and Concept Class) and extra attributes that are specific to MER (e.g. Numerator, Denominator, Period, and Indicator Group).
• Text from the MER guidance PDF is copied and pasted into the reference indicator's row.
  • Each MER Guidance section, such as "Reporting frequency" and "How to use" has its own column in the spreadsheet and thus has its own attribute when loaded into OCL.
  • PDF formatting is also reflected in the spreadsheet using Markdown formatting, which is added manually during extraction.
  • Images from the MER guidance must be loaded using a different method, which is still to be determined.
• Export the spreadsheet and save to the data folder (eg data/mer_indicators_FY21_20200106.csv)
• Open settings.py and update FILENAME_MER_REFERENCE_INDICATORS to include the reference indicator spreadsheet saved in the previous step

Build the OCL bulk import script

• Example python build_ocl_import.py > logs/build_pepfar_test10_20210106.log

Validate the bulk import script

• tbd

Load into OCL

• tbd

Validate the content via OCL and MSP

• tbd