Evaluation Subject, Purpose and Scope
UNHCR Sudan commissioned this evaluation for the purpose of accountability and learning, with an emphasis on generating lessons learnt and identifying examples of good practice to support improvements to the ongoing refugee response in White Nile State (WNS) which may also be applied to the wider refugee response in other parts of Sudan. The evaluation aimed to assess the extent to which protection, including assistance needs of refugees, have been met and to gauge the degree to which timely operational adjustments or revisions in strategic direction or coordination mechanisms have been made since the beginning of the crisis to meet the emerging needs of the refugee population. UNHCR anticipated that the conclusions and recommendations resulting from this independent evaluation will contribute to developing its strategic engagement in the ongoing response in Sudan, highlighting key lessons that can influence future ongoing activities and planning. This evaluation covered the UNHCR and partner response between December 2013 and April 2018
The evaluation was divided into three-phases: inception, data collection and analysis. The data collection phase included both a desk review and a three week visit to Sudan to collect data relevant to the five evaluation questions (EQ) listed in the terms of reference (TOR) for this evaluation. The team interviewed a total of 202 stakeholders, either face-to-face, or by phone, including government officials, bilateral donors, United Nations (UN) Agencies, national and local authorities, international and national Non-Governmental Organisations (NGOs). This number includes 15 Focus Group discussions (FGD) with representatives of refugee and host communities. The team conducted a desk review of policy and strategy documents, evaluations, reviews, studies and other documents. An interagency workshop was held prior to circulation of the draft report where staff from UNHCR and partners were given the opportunity to provide high level feedback on preliminary findings and provide perspectives on the relevance and achievability of emerging recommendations. The evaluation team found it challenging with some of UNHCR’s interventions to assess achievements due to variable quality of monitoring data and relied to a large extent on qualitative data collected during interviews and FGDs.
Summary of Findings
A summary of findings based on each of the five evaluation questions is given below.
1. Relevance of UNHCR and Partner Strategies
UNHCR RRRPs were appropriately focused on three priorities: 1) maintenance of emergency response capacities, 2) achievement of minimum emergency sectoral standards and 3) facilitation of durable solutions. At the same time the UNHCR RRRP did not provide sufficient strategic guidance for UNHCR operations in WNS for two main reasons. Firstly, it was only an annual strategy covering a single calendar year. Secondly, it was a regional strategy covering several countries and lacked sufficient context-specific detail to guide investments and operations.
Channelling of assistance and assessment – up until 2016 the Government of Sudan (GoS) required UNHCR to channel their assistance through the Sudanese Red Crescent Society (SRCS). UNHCR did this while simultaneously positioning themselves to work with a wider range of partners. UNHCR used the Refugee Coordination Model (RCM) to guide collaboration with UN strategic partners, notably UNICEF, World Food Programme (WFP) and the World Health Organisation (WHO), who faced fewer GoS restrictions to operate in WNS. Until mid-2016 UNHCR was thus largely dependent on assessment data from secondary sources, primarily from SRCS. UNHCR’s remote management and monitoring systems were insufficient to allay doubts amongst many donors about the appropriateness and effectiveness of the assistance provided. The operating environment in WNS significantly improved for UNHCR after mid-2017 when they were able to gain regular access to refugees in WNS.
Protection Strategy – UNHCR’s emphasis on registration and legitimization was an appropriate priority that helped to enable refugees to support themselves. UNHCR’s Protection Strategy gave priority to providing targeted support to persons with disabilities and other specific needs, though it lacked guidance on mainstreaming protection in different sectors. At the same time, the scale of unmet needs in shelter and sanitation raised the question of whether UNHCR should not have given more priority to investments in these sectors.
Persons of Concern (PoC) with specific needs – UNHCR’s focus on assessing relief needs without considering gaps in participation/inclusion and communication/transparency was inconsistent with policy guidance in UNHCR’s global Age, Gender and Diversity (AGD) policy. A key element missing in virtually all of UNHCR’s sectoral strategies was about promoting community participation and ownership.
2. Achievement of Results
UNHCR was challenged by the lack of financial and human resources from the beginning of the crisis, which influenced the efficiency and effectiveness of their response. The initial decision by GoS to designate South Sudanese as “brothers and sisters” rather than as refugees left UNHCR with an ambiguous role in the response. After refugee status was accorded to the South Sudanese by GoS in 2016, UNHCR was able to take on a more central role in the refugee response in accordance with its mandate and provide more systematic technical support and directly monitor quality.
Assistance – Apart from an initial lack of physical access, UNHCR faced multiple challenges, including availability of land, low funding levels, poor road infrastructure, low national capacities and GoS policies that restricted intervention options. Mortality rates have remained below emergency thresholds, indicating that UNHCR’s contributions together with other humanitarian agencies, have helped to ensure the South Sudanese refugee population received life-saving support they required, even though emergency thresholds were exceeded in many sectors. There were considerable variations in standards of assistance between camps and there continued to be significant unmet needs in the sanitation, nutrition and shelter and education sectors. Some of the gaps observed were partly attributed to GoS requirements that UNHCR invest in relatively costly permanent community infrastructure, which decreased coverage due to limited funding.
UNHCR’s approach has helped to reduced fuelwood consumption by some 50 percent in the camps, although wood cutting remained a priority environmental concern for host communities. NFI and shelter assistance used a targeting approach since 2017. Other assistance, including WFP-supplied food aid, was provided via blanket distributions.
Beneficiaries were found to be selling relief assistance to raise money to purchase food and non-food items, pay for medical expenses and meet other basic needs.
Protection – Protection has been a central focus of UNHCR’s RRRP. Additional protection staff based in Kosti since 2017 has enabled UNHCR to increasingly fulfill its protection mandate, though many challenges remain, notably a reluctance of local authorities to allow UNHCR staff to conduct confidential interviews with refugees and strong sensitivities around attempts to address sexual and gender-based violence (SGBV). Delays in refugee registration and verification processes and credibility gaps in data have contributed to inefficiencies and has led to the perception that considerable numbers of refugees registered spent most of their time outside camps. Since the change of UNHCR’s government counterpart from Humanitarian Aid Commission (HAC) to the Commission for Refugees (COR), their involvement has helped to improve the knowledge and understanding amongst stakeholders of UNHCR’s protection mandate and the role of the host government.
Building of National Capacities: Although UNHCR’s response has been predominantly led by national staff and agencies, UNHCR has not had a capacity building strategy and capacity needs assessments did not appear to be used during periodic partner reviews.
Monitoring and Information Management: Donor representatives confirmed there had been a significant improvement in the quality of information provided by UNHCR.
Information management systems tended to prioritise donor reporting, with lower priority given to continuous improvement and learning for field operations. With the exception of shelter and non-food items (NFI), which has been systematically collecting post-distribution monitoring (PDM) data since 2017, reporting by UNHCR and partners has been mainly at the activity/output level. There was no evidence of UNHCR-led or partner lessons-learned reviews that could inform annual planning exercises. UNHCR staff in Sudan had no previous experience of remote management systems, which became less important for camps once access had improved, but the lack of remote management systems continued to affect assistance and protection for PoCs staying in difficult-to-access areas.
Staffing and Funding: Numbers of UNHCR staff in Sudan were significantly lower in comparison with other countries in the region dealing with significant South Sudanese refugee influxes. By the end of 2017 UNHCR had considerably expanded their technical capacities, resulting in a decreased dependence on deployments from UNHCR’s Emergency Response Team (ERT) and standby partners. UNHCR has only been able to mobilise between 19% and 37% of their annual funding requirements in the RRRPs. UNHCR Sudan has been able to cover its own operations budget but has lacked resources to adequately cover unmet life-saving and longer-term needs.
Value for Money (VFM): Relatively high operating and staff costs put pressure on UNHCR to demonstrate VFM for South Sudanese refugees. While UNHCR’s specific technical expertise and overall contributions have helped in keeping mortality rates below emergency thresholds, areas where UNHCR could have improved VFM included, more systematic monitoring of major cost drivers, minimising delays in updating/verifying registration data, promoting community ownership of infrastructure, more systematic use of PDM data and feedback from community complaints systems, increased use of CBI options, more strategic partner selection and improved preparedness based on lessons learned.
3. UNHCR’s Refugee Response Coordination Responsibilities
Humanitarian Coordination Team (HCT) – UNHCR was praised for its teamwork by its peer HCT members for its coordination role in Sudan for CERF funding processes during 2014. The subsequent deterioration in team work with HCT peers observed was mainly attributed to a combination of UN strategic partners not delivering according to expectations along with UNHCR’s increased technical capacities. These capacities made them less dependent on capacities of strategic partners but one result has been less of a development perspective in UNHCR’s programming.
Refugee Consultation Forum (RCF)/Refugee Multi-Sector (RMS) – UNHCR’s coordination systems were perceived as relatively inefficient until the RMS, later renamed as RCF, was launched in 2016 and the improvements in coordination helped to improve UNHCR’s image both at a national level and in WNS. The RCF in Sudan was subsequently promoted by UNHCR’s Regional Bureau as a model of good practice. Development actors such as UNDP and UNHABIT have been invited to the RCF but have yet to participate even though they are funding refugee-related interventions in refugee hosting areas.
Participation by donors has also been irregular.
Harmonising approaches –UNHCR has faced major challenges in aligning its standards to meet basic protection and assistance needs using participatory approaches while at the same time satisfying GoS requirements to invest in relatively costly contractor-driven permanent community infrastructure with minimal participation from either refugees or host communities.
4. Consideration of Medium- and Long-term Objectives
UNHCR standards have been mainly designed for protracted refugee crises and were used to inform site planning for camps and related infrastructure in WNS. UNHCR’s programme supported medium- to long-term objectives in a number of ways, including formalising agreements with state-level line ministries, advocacy with donors to support livelihood interventions, piloting durable solutions to refugee settlement, registration/legitimisation of the refugee population, investments in permanent community infrastructure.
Areas for improvement included the lack of a multi-year strategy specifically for UNHCR’s programme, an accompanying strategy for building national capacities, more sustainable exit strategies that promote ownership of refugee and host communities and site planning that is area-based, rather than camp-focused. Interviews with different UN agencies and donors indicated general agreement with the out-of-camp solution. Donors nevertheless expressed reservations about UNHCR’s lack of a long-term strategic view.
5. Protection and Assistance for Persons of Concern
UNHCR has made progress in identifying vulnerable groups and individuals amongst the PoCs. Leaders in both refugee camps and host communities were used to regularly interacting with staff from UNHCR and partners, as were refugee women and youth groups.
Despite these regular discussions, findings from Focus Group Discussions (FGDs) indicated basic levels of Accountability to Affected Populations (AAP). Refugee and host community members displayed little knowledge of plans, technical specification or budgets of interventions. Refugees demonstrated little responsibility for monitoring implementation of interventions or maintaining community infrastructure and were constrained in giving honest feedback to agencies.
A qualitative assessment by the team based on UNHCR’s ten AGD obligatory core actions found that good progress had been made in meeting commitments relating to AGD-inclusive programming data and the registration and legitimisation of vulnerable groups in camps. The main areas for improvement related to participation and inclusion. As access by UNHCR and their partners to PoC in WNS has improved since 2016, UNHCR has been able to improve their understanding of refugee profiles in camps, including PoC with specific needs.
Since 2017, UNHCR has used SOPs that included eligibility criteria for persons with specific needs to prioritise distribution of NFIs and temporary shelter materials though UNHCR had only begun to look at specific needs in out of camp populations. There have nevertheless been significant problems with coverage and the quality of assistance provided by UNHCR’s NGO partner tasked since 2014 to provide community services.
This section on conclusions begins with an overall statement on UNHCR Sudan’s interventions in WNS from the end of 2013 until April 2018. This statement is followed by concise versions of conclusions linked to corresponding recommendations.
Overall Statement on UNHCR’s interventions in White Nile State
UNHCR in Sudan was challenged from the start of the South Sudanese refugee crisis both
by a lack of access to PoCs and by the low levels of financial and human resources. These
factors had a significant influence on the efficiency and effectiveness of the response. Sudan
was one of four major countries of asylum for South Sudanese refugees and UNHCR Sudan
arguably faced the greatest challenges compared to other UNHCR offices in the region in
scaling up, starting from the lack of a clear role due to a GoS decision to designate South
Sudanese in Sudan as “brothers and sisters”. This changed after refugee status was given in
2016 and, by the beginning of 2018, UNHCR was playing a lead role in providing protection
and coordinating the response as they scaled up their capacity in WNS. Although UNHCR
have continued to face important constraints, significant improvements in the operating
environment from the end of 2017 should pave the way for UNHCR to address critical gaps
and apply their coordination role to facilitate a move towards durable solutions for South
Sudanese refugees in Sudan