Consultant - Urban WASH Programme Final Evaluation

Background:

Goma and Bukavu have seen rapid urban growth over the past fifteen years as a consequence of conflict in rural areas driving people towards perceived safe havens in urban centres. However, in both cities the availability of clean water and sanitation facilities have not kept pace with the increase in demand as a consequence of a multitude of factors including lack of investment, poor management of existing infrastructure, restrictive legislative codes on management of the market, and poor hygiene practices. Cumulatively as a result; water borne diseases such as diarrhoea are rampant – leading to high incidences of child mortality. The cost of easily accessible potable water remains prohibitively high for most families – and women and girls continue to suffer from sexual harassment or assault when travelling long distances to collect it, in addition to missing out on education and economic opportunities.

Purpose / Project Description:

Responding to this, with the support of the UK Department for International Development, Mercy Corps is mid-way through the implementation of a five year, DFID funded urban Water, Sanitation and Hygiene (WASH) programme based in Goma and Bukavu under the overall envisaged outcome of improved hygiene behaviours and safe, equitable and sustainable access to potable water and sanitation for close to 1 million Congolese men, women, boys and girls. The overall envisaged impact is reduced diarrhoea rates in children under 5. The project name is “Integrated Maji Infrastructure and Governance Initiative for eastern Congo” – or IMAGINE – and the four main programme outputs are as follows:

• Increased availability of improved water, sanitation and hygiene infrastructure from source to tap; [Infrastructure]
• Viable and sustainable water, sanitation and hygiene service provision market; [Service Delivery]
• Effective governance and systems that are accountable to all citizen’s WASH needs; [Governance]
• Improved knowledge, awareness and perceptions of essential water, sanitation and hygiene behaviours. [BCC]

An interesting web-map that outlines the rationale for the project can be viewed through the following link: http://arcg.is/28NUKwU

Mercy Corps is looking for a consultant to undertake two final evaluations of the programme. The first final evaluation will focus on the BCC and Governance components and will take place between April and July 2019, whereas the second final evaluation will focus on the Infrastructure and Service Delivery components and will take place between April and July 2020. While this SOW covers the first final evaluation only, Mercy Corps is planning to hire the same consultant for the second final evaluation, contingent upon satisfactory performance.

Consultant Objectives:

Objective 1: A final evaluation is conducted which provides evidence-based recommendations and enables Mercy Corps to understand progress made against results of log-frame indicators from baseline and midline evaluations and biannual surveys.

Objective 2: A final evaluation is conducted which enables Mercy Corps to indicatively identify both the joint and individual impact of the BCC and Governance components in contributing to our overall goal of reducing the rate of diarrhoea for children under 5.

Objective 3: Research questions on IMAGINE’s BCC approach as well as on the impact of the program on household resource allocation are integrated into the survey in close coordination with researchers of the London School of Hygiene and Tropical Medicine.

Sampling Approach:

Quantitative Data Collection: In order to be able to fulfil the objectives, a relatively complex sampling model will need to be established – that will enable us to replicate the sampling approach applied to the baseline study and mid-term evaluation (systematic random sampling distributed proportionally across quartiers of intervention), while employing a random pre-selection of households to be interviewed based on satellite imagery. The consultant will be required to support with a critique and review of the proposed sampling approach in order to ensure that the model is sufficiently robust so as to enable us to achieve our stated objectives. Sampling should also ensure that a reasonable balance of male/female respondents is reached.

Overall as a minimum standard the quantitative data collected (for Objective 1) should be at no more than 3.5% margin of error against the totality of the population of each target location (Goma and Bukavu), with a 95% confidence level. For gender disaggregated data, MC would accept 5% margin of error against the totality of the total population across all target locations. For Objective 2 results, we can accept a margin of error of 7% for each control group.

In Goma, data for Objective 3 will be collected in the IMAGINE intervention area as well as in neighborhoods outside the scope of IMAGINE, so as to include a control group of the population not served by the program. This will provide a point of comparison between two different population groups as well as multiple time points (the two final evaluations), allowing LSHTM researchers to conduct difference-in-difference analysis. Control group data will be analyzed in a separate database before drawing comparisons between the results for the different geographical areas.

Qualitative data collection: Qualitative data collection may take place in complement to the quantitative data collection process – with the focus topics to be identified in conjunction with the M&E team and Programme Director prior to the launch of the consultancy. Focus Group discussions or Key Informant interviews will most likely be used to probe progress on the Governance and BCC components in more detail and/or obtain the perspectives of specific categories of beneficiaries (e.g. people living with disabilities). Focus group discussions and key-informant interviews would be required to take note of gender and age ethical and cultural sensitivities and requirements.

Specific Activities:

In chronological order, to be able to achieve Objectives 1 and 2, the consultant will be expected to carry out the following activities. The amount of time spent in country and/or working from home will be subject to discussion at interview.

• Document Review: The IMAGINE programme team will share key documents and required literature (Log-frame, Inception Report, Baseline/Midline/Biannual survey reports and Master Plans).
• Reach pre-agreement of the sampling frame with the Mercy Corps technical support unit, the IMAGINE programme M&E and Research Manager and Programme Director, and researchers of the London School of Hygiene and Tropical Medicine.
• Liaise with the Mercy Corps M&E and logistics team over logistical planning for the final evaluation.
• Review all research questions proposed by the London School of Hygiene and Tropical Medicine, and, where necessary, comment on their comparability to questions posed in the baseline and mid-term evaluation as well as on their ability to show the quality, direction, pace and results of change – and suggest changes if necessary.
• Finalise the draft survey questionnaire (which should be consistent with the baseline and mid-term evaluation) to ensure capture of relevant information and share with Mercy Corps M&E team and Gender Advisor.
• Conduct training of data collectors in conjunction with IMAGINE M&E team on data collection, validation, ethical aspects and quality assurance criteria and pilot through a preliminary field test.
• Conduct pilot field testing – in conjunction with IMAGINE M&E team.
• Produce brief inception report (5 pages) documenting field test and work plan to include clear description of methods to be applied for collecting, processing, cleaning, analysing and interpreting the data collected.

Subsequent to the conduct of the final evaluation in the field, the consultant will be expected to:

• Prepare and conduct focus group discussions and key informant interviews to analyse more in-depth the reasons behind unexpected trends.
• Conduct all the analysis pertinent to the final evaluation - in conjunction with the IMAGINE M&E team - disaggregating findings by age, gender and, where pertinent, disability – and establishing/refuting correlations between different variables and the diarrhoea rate of children under 5.[1]
• Present findings to Mercy Corps staff – with subsequent Q&A (up to 4 will be conducted – with different audiences targeted). These shall be conducted in French and/or English.
• Present findings to DFID staff – with subsequent Q&A. This shall be conducted in English and will take place in Kinshasa or Goma.
• An accessible data-set covering all quantitative data (analysed and raw) collected will be developed and handed over to Mercy Corps M&E team. The language of the data-set can be French or English. The data-set will consolidate findings from both localities. An accompanying file with all qualitative data collected (as applicable) will be developed and handed over to Mercy Corps M&E team. The file should be in French.
• Produce a comprehensive final evaluation report including (where relevant) tables, maps and graphs and circulate for review to Mercy Corps. The report will highlight considerations on gender, age and, where pertinent, disability aspects and will propose relevant recommendations to increase inclusion of vulnerable groups.
• Produce a standalone, expanded executive summary of the final evaluation report of up to 10 pages highlighting findings and key issues.
• Provide Mercy Corps staff with the opportunity to respond to the findings of the report (two weeks). Up to three rounds of comments by Mercy Corps will be provided with the consultant expected to provide a revised report within five working days after each review.
• Amend the report following feedback from Mercy Corps and send back to Mercy Corps for final validation.

Consultant Deliverables:

• Literature & documentation review.
• Refine Tools/Methodology + Training Materials for Enumerators.
• Review sampling of control group and integration of research questions.
• Meetings with Mercy Corps and stakeholders in DRC to review programme documentation/strategy.
• Oversee staff training for data collection team in Goma.
• First week data collection in Goma, teething problems resolved.
• Oversee staff training for data collection team in Bukavu.
• First week data collection in Bukavu, teething problems resolved.
• Inception report produced and signed off.
• Remote availability for questions during survey implementation period.
• Data review and analysis – including review of findings with programme team remotely via skype and email.
• Data-set of raw and analysed data produced and signed off.
• Conduct focus group discussions and key informant interviews in Goma and Bukavu.
• Produce PowerPoint of indicative findings produced and presented to DFID, MC and other stakeholders in DRC.
• First draft of Evaluation Report and Executive Summary Report produced.
• Evaluation Report finalized and signed off by Mercy Corps.

[1] For the analysis, Mercy Corps seeks to understand the extent to which different outputs and outcomes of the programme contribute towards the overall programme impact – both in isolation and cumulatively – and question/confirm some of the key assumptions made by the programme.

Timeframe / Schedule:

In consultation with the IMAGINE M&E team, the schedule for the deliverables will be determined with the consultant and aligned with the team’s needs and implementation timelines. The amount of time spent in country and/or working from home will be subject to discussion with the consultant. The consultancy must start in April 2018.

Review Process:

Mercy Corps requires a review period before approval of certain deliverables. Of greatest importance is the sampling frame and initial proposed methodology – which will be reviewed by Mercy Corps Design, Monitoring and Evaluation and Learning (DMEL) unit based in our headquarters in Portland, Oregon, US. The anticipated review period for this will be three days – and the statistical relevancy of the sampling approach proposed will be the primary issue for review. For all other deliverables requiring approval (see table on page one above), review will take place in country, with approval from the M&E Manager/Director, in consultation with the Programme Director in accordance with the time-table outlined through the link above.

Management Relationships:

The M&E consultant will report to the IMAGINE M&E Manager, with the majority of tasks to be undertaken in collaboration with implementing partners, programme staff and, particularly, M&E personnel and primary stakeholder groups to enable a learning process and inclusive decision making and to maximise transparency and accountability. The M&E consultant will have management responsibility for the staff directly engaged in the conduct of the baseline in the field – with the exception of disciplinary action or removal of staff from the field – which remains within the purview of the M&E Manager. In such a case where disciplinary action is necessary, the consultant will be free to raise concerns to the M&E Manager and/or Programme Director for subsequent action to be taken (as required).

Information and Services Provided by Mercy Corps:

For the duration of their consultancy, the consultant(s) will be provided with:

• Up to 15 Android based smartphones for data collection using ODK.
• Full and active collaboration and support of the IMAGINE M&E team in country.
• Translators and Interpreters from Swahili to French.
• French speaking enumerators (50% female if possible). Estimated 10 per locality for 20 days including training, subject to change based on the methodology proposed. Data collectors will be expected to undertake a training on gender and age-sensitive data collection prior to field work inception (3 days, with Gender and M&E component).
• Access to all project documentation at the beginning of the consultancy.
• Accommodation in hotel or Mercy Corps premises in Goma and Bukavu.
• Assistance of the M+E team in each locality (1 expatriate Manager and 4 national staff in total) for set up, training and recruitment of enumerators.
• All internal flights or boat transport (potentially in the case of Bukavu) to the locations required for the final evaluation and subsequent presentation of findings (Kinshasa likely).
• Access to Mercy Corps Office for working.
• Transport in all sites targeted for the final evaluation (max two vehicles per location).
• A per diem at the rate of $57 USD per day whilst in Bukavu and USD $53 per day whilst in Goma (including weekends).
• A per diem at the rate of $44 USD per day whilst in Kigali is permissible for transits exceeding 8 hours in this location.

The Mercy Corps M&E team will be available to work directly with the Consultant throughout the duration of the consultancy – including in the supervision process for data collection, the technical set up of the ODK forms, training of enumerators and data analysis. In addition, Mercy Corps has a database of potential enumerators (in both Goma and Bukavu) and will be able to hire them in advance of the arrival of the consultant. The consultant should take this into account when drafting his/her proposition.

Qualifications and experience required:

In addition to outlining their capacity to meet the specific requirements of this consultancy, through their application and throughout the selection process, the consultants must demonstrate that they have:

• Strong statistics background – acquired through either academic studies, practical experience or both. The consultant will need to be able to quality check the proposed sampling approach in addition to conducting quality data-analysis (using SPSS/STATA/R) that goes beyond descriptive presentations of data.
• Fluency in English and French are required.
• Strong experience conducting major research exercises in support of major development programmes – preferably in WASH – in challenging operational environments.
• A strong approach to assuring quality assurance of data collected.
• A strong ethical approach to data collection – while still being able to meet the objectives of the consultancy.
• All data collected and study outputs in soft and hardcopy will be the property of Mercy Corps

Instructions for Detailed Proposals:

Parties interested in bidding for this should submit the following to Mercy Corps by the closing date of 31/02/2019:

• CVs of the consultant(s) envisaged for deployment.
• Proposed total cost of consultancy including cost per day
• 2 references

Bids can be submitted in either English or French.

Selection criteria:

Candidates for the consultancy will be selected according to the following criteria:

Relevant technical qualifications (focus on statistics) and experience for the tasks to be performed 30

Relevant experience of similar assignments in challenging political/operational environments. 20

Experience with NGO agencies 10

Language capacity (francophone) 10

Knowledge of the region 10

Lowest Price 20

The Consultant will report to:

Moritz Schuberth – IMAGINE M&E and Research Manager

The Consultant will work closely with:

Patrick Cantin – IMAGINE Programme Director

Caitlin Snyder – Mercy Corps DRC M&E Director