For organizations thinking about conducting a data ecosystem assessment, there is an array of initiatives, tools, and approaches from which to choose. Based on our experiences deploying our own tools and supporting a variety of stakeholders, IREX and Development Gateway: An IREX Venture (DG) would like to share a few insights and questions to help you get started.

Just like a natural ecosystem is the result of the interaction between certain organisms and the physical environment in which they live, data ecosystems are also the result of all the people (individuals), communities, and institutions interacting with each other in the way data is being produced, used, re-used, and/or shared. For international development stakeholders (everyone from national governments to international organizations, private sector, civil society as well as individuals and communities), harnessing the power of data for good has been a priority ever since the launch of the Sustainable Development Goals in 2015. From the onset of the Sustainable Development Goals agenda implementation, quality access and use of data was considered essential for decision-making, accountability, and for seeking solutions to complex and multifaceted development challenges. It was the start of the so-called “data revolution” for sustainable development. 

For initiatives aiming to harness the power of data for good, it is critical to start with a well defined problem and a thorough understanding of the data ecosystem around it. It is, therefore, very important for stakeholders driving such initiatives to have the tools and resources that will help them assess data ecosystems specific to a sector where the development challenge takes place or even specific to a nation for more effective and impactful policy making or strategic planning. Investing time to assess a data ecosystem will ensure the relevance, sustainability, and potential for scale of a solution to a development challenge. At the same time, data ecosystem assessments will help to close any significant data gaps and strengthen the ecosystem, enabling quality and timely data production, facilitating collaboration between stakeholders, increasing the use of data for decision-making, improving service delivery, building responsible and ethical data practices, and/or increasing government data transparency. 

Three Questions to Get You Started 

The number of tools, frameworks, and initiatives to help you understand, analyze, map, and integrate data ecosystems at various levels can seem daunting. Below, we have included a select list of tools you can use for your assessment. Each methodology has a unique set of outputs, focuses on different parts of a data ecosystem, and analyzes different types of ecosystems. Before selecting a tool, ask yourself the following questions:

Q1: What is the objective?

This is the most important question to ask at the beginning of the process. The success of an ecosystem assessment depends on having a clear understanding of the reason for conducting it. This is important for assessments led both internally and externally. We see two major motivations for leading these assessments:

●  To find possible solutions to a known problem, need, or interest – In this case, the assessment should be problem-driven and will help identify specific recommendations and investments that help find a solution to the known problem. Some notable examples are assessments to help identify possible data and digital investments, operationalizing strategic plans, or designing Monitoring and Evaluation (M&E) systems. A specific example illustrating this would be IREX and local actor D-Lab’s use of the Data Compass tool (as part of Data Zetu program) to help the Ministry of Health in Tanzania evaluate and sustain investments in public health care that included building capacities of rural health care facilities for more effective and systematic use of health data.

●   To evaluate where an ecosystem stands in relation to other ecosystems – Some data ecosystem assessment tools have been designed around evaluative frameworks used to conduct multi-dimensional research on the state of a data ecosystem — typically — across several countries. A reason for using these tools would be to determine where an ecosystem fits relative to other ecosystems or an ideal state. Examples of tools that help this type of assessment are scorecards, benchmarks, or indexes, like the recently released Global Data Barometer.

Q2: At what level do you want to focus?

Data ecosystems are multilayered, and there are multiple levels at which an assessment can be led:

●  Country Level – These data ecosystem assessments are tools that provide a general landscape of a country. A country-level tool could be useful during investment or development planning. For example, a donor could use a tool that assesses the statistical capacities of a country to inform the design of a program or project that is data heavy. 

●  Sector Level – These data ecosystem tools help actors to better understand opportunities to harness data within a sector. It considers all actors involved in a sector, including the private sector, government, civil society organizations, nongovernmental organizations, and donors. For example, DG led a data and digital assessment in Guadalajara, Mexico in the open procurement sector. The objective of this sectoral assessment was to identify key areas of investments in open procurement from the perspective of multiple actors, including government contractors, civil society organizations working in the corruption sector, and government employees making purchases.

●  Organizations or Institutions Level – These data ecosystem assessments allow actors to understand the data ecosystem of a single organization or institution. For example, a social impact organization may want to understand the extent to which they are utilizing their data (i.e., data maturity level) in order to identify areas for improvement to become a data-driven and data-informed organization.

●  A Combination of Assessments at Multiple Levels – Sometimes multiple data ecosystem assessments are needed. For example, a country-level assessment can help a donor identify areas of investment, and a second assessment can allow us to drill down on a possible solution. This was the case of the Results Data Initiative Program led by DG in Malawi. First, an assessment was led to understand the data ecosystem of the agriculture sector in the country. During this assessment, we identified the need for a National Agriculture Data System as a key priority across all actors involved. During a second separate assessment, however, DG focused on the Ministry of Agriculture to design the National Agriculture Management Information System (NAMIS).

Q3: Which aspects of the ecosystem do you want to understand?

A data ecosystem consists of many different elements. Having clarity on which aspects of the ecosystem you want to assess will ensure that the final product responds to your needs. Below are some examples of aspects you could focus on during the assessment. This is an indicative list that can be used as a starting point, but you can add more or select multiple foci.

●  Available data – A key part of a data ecosystem is understanding what data exists. Additional information can be associated with available data, including data gaps; who produced the data; how it is collected; its infrastructure; how it is shared; the format in which it is collected; frequency of collection; interoperability(i.e., the ability of data systems to exchange and make use of information); data quality analysis; or the legal framework.

●  Data openness – Knowing the level of open data of a government can help governments and international organizations recognize areas for investment. The level of data openness can refer to multiple variables, including the format in which data is published; the quality and frequency of data published; and coverage or legal framework.

●  Data capacities and skills – Governments, donors, and civil society organizations need to have the capacity to produce and use available data. Depending on the assessment tool you select, you can assess statistical capacities, skills, access to training, or digital and data literacy.

●  Use and reuse of data – Leveraging data to make decisions to advance the implementation of the Sustainable Development Goals is one of the key premises of the data revolution. Each assessment tool will evaluate the use of data differently. Some examples include having a positive impact, using data for decision-making, or incentives or disincentives for data use. IREX recently helped the Millennium Challenge Corporation (MCC) to better understand data ecosystems within Lesotho's healthcare sector with a focus on the use of data to understand key processes ranging from budgeting to maternal mortality rates; recommendations also helped inform investments in data ecosystems driven by MCC’s new compact plan for Lesotho.

IREX and DG has created a curated list of tools to be used for data ecosystem assessments.

DATA ECOSYSTEM ASSESSMENT TOOLS