What is ‘Monitoring and Evaluation’?

Monitoring and evaluation (M&E) helps guide projects and programs towards maximum benefit and effectiveness. M&E is being used more and more within government in Viet Nam. The M&E building block for CDSs includes tracking two types of M&E. Firstly M&E of the process of development of a CDS and secondly, an investigation into the impact or results of the CDS strategy in the city.

There are an increasing number of initiatives to increase the quality and use of M&E within different levels of government in Viet Nam. This is particularly due to the newly issued Decree 131/2006/ND-CP (updated version of the Decree 17/2001/ND-CP) which defines the legal framework for M&E in the context of overseas development aid (ODA) issues. M&E is also a core part of the SEDP process, again causing more focus and attention on it as a practice to provide feedback on progress which can be used for on-going planning.

M&E for the process of development of a CDS involves monitoring, evaluating and documenting the actual activities that you carry out to develop your CDS. For instance, which stakeholders you consulted what processes you used etc. This sort of M&E involves finding out how you might have done these activities better, by asking questions about what worked well and what didn’t for different people who were involved. This sort of M&E informs the CDS process as it happens, helping to improve and manage the process. In addition, the final product of this sort of M&E is a report documenting the process that was followed to create the CDS, including lessons that were learnt along the way.

M&E for the impact or results of the implementation of the CDS strategy involves finding out what changes the strategy has caused in the city more broadly. This may include perhaps examining changes in indicators that were used in the initial assessment after implementation of the CDS, or other ways of finding out and measuring what the results, outcomes and impact of the strategy have been on the city and progress towards the vision and objectives of the strategy.

Why is ‘Monitoring and Evaluation’ important?

M&E is both necessary and beneficial for any process, strategy, project or program. M&E helps enable accountability for work completed and helps people learn from their successes and their mistakes.

M&E of the process of doing a CDS is important to help you to:

• Adjust the CDS process in response to new information and knowledge of successes and failures
• Deal with problems and unplanned consequences as they arise,
• Build on new opportunities that arise,
• Provide feedback information about progress towards completion of planned tasks to inform on-going planning and scheduling
• Keep track of what has been done so the CDS process is transparent and documented,
• Keep track of the completion of project outputs to the desired level of quality,
• Keep track of resources invested, including when and how they were invested,
• Learn and improve how to do a CDS through understanding the reasons behind success and failure and
• Share experience and knowledge from CDS process
• Provide accountability to funders of the CDS and other stakeholders by making the processes that have been used transparent to everyone.

M&E is important to make sure that you discover and track the outcomes and impact of the CDS itself in the strategy areas or themes of the CDS.

Doing an M&E here will help you find out:

• Progress towards actual implementation of the strategy,
• Extent of achievement of intended results (also known as effects, outcomes or impacts) of the strategy, including progress towards the vision and goals
• Any unintended or unplanned impacts (which may be positive or negative),
• Any barriers and risks to the implementation so they may be overcome,
• To learn about what is takes to implement a CDS and how to do it better in the future or in other locations and
• To provide accountability to government, citizens, donors and other CDS stakeholders

Key Principles that Support Good ‘Monitoring and Evaluation’

Keep monitoring and evaluation manageable and focused- for example:

• When using indicators, make sure they are relevant and meaningful. That is, they must clearly link to the strategy, or are important factor in a building block process that you aim to measure.
• As far as is possible use information that you already have, so that you build on existing measures and procedures, trying to use data of the best quality available
• Only collect, manage and document information that will serve a specific purpose such as informing decision-making
• Spend about 5-10% of time or resources on M&E
• Set clear objectives for the M&E system itself
• Use consultation time with a wider group carefully (eg. you might choose to prioritise and choose the final indicators with a stakeholder group, making sure to start with only double the final number required)

Include qualitative information and collection approaches:

• Quantitative information does not provide explanation of the reasons about the ‘how, where and why’ that lies behind the numbers and quantitative methods of data collection do not easily capture unplanned change. These gaps need to be filled by collecting qualitative information as well.
• Use interviews and surveys or questionnaires (like the Provincial Competitive Index (PCI)) to find out about people’s opinions, views and reasons (some of which can be reported quantitatively where necessary))
• Include forms of participatory self-assessment and critical reflection

Achieve a balance between your local context and greater generalisability to other cities

• Use a combination of standard benchmark and locally chosen indicators that relate closely to your CDS themes (eg this might mean you include some socio-economic development plan (SEDP) indicators amongst those chosen for your strategy)

Make sure that the information you collect is used:

• Be clear about who are the users of different types of information and reports produced and what their interest is in the information you are providing to them, what level of detail they will need
• Set up systematic processes for managers to respond to M&E information provided and adjust the course of the CDS
• Think about how to present information succinctly (eg in graphs or diagrams) so that it is quickly and easily communicated.
• Integrate M&E into day-to-day management rather than looking at M&E as an audit or report card.
• Think about key decisions for the CDS and what information is needed to help inform these decisions (as well as who will need it and when)

Key components of monitoring and evaluation

1. Set objectives for your M&E system

1. Plan your M&E system for each building block (process M&E)
2. Plan your M&E system for the CDS itself (impact M&E)
3. Implement your M&E system
4. Review your M&E system

These key components and their relationship to the CDS building blocks and results of the CDS in the city are described in Figure below.

Figure ?: Key components of conducting M&E for a CDS

How to do ‘Monitoring and Evaluation’

a) Set objectives for your monitoring and evaluation system

Discuss the purpose and set clear objectives for your M&E system. This is best done as a participatory process with a focus on

• How to balance the need for both learning and accountability within M&E systems,
• How to match the available resources with the needs.

It will be worth deciding at this stage what kind of participation will be appropriate in the planning of the M&E system, who should be involved and when and how. Having a learning objective will mean that you need to monitor the process of doing the CDS. The example below describes how reflecting on the process used to develop the CDS in Hai Phong provided many useful insights.

Hai Phong M&E of process

The importance of capturing information about process has been demonstrated through the retrospective evaluation of the Hai Phong CDS process that uncovered important outcomes and valuable learning about their CDS process.

Some of the important findings that arose through the Hai Phong process evaluation were:

1. Staff recognised that their skills have improved with regard to teamwork, organisation and public consultation after one year of participation in the CDS process
2. Even without finalisation and approval of the CDS, the major activities and initiatives of the CDS have been used in policy making and therefore have contributed to the direction of the city
3. The method of prioritisation of projects and programs was difficult and complicated for a large urban centre like Hai Phong

It is important to build in the M&E of process into the CDS so that the sorts of information they made in Hai Phong are discovered and used during the development of the CDS to adjust the activities and actions that are taken.

b) Plan the monitoring and evaluation system for each building block (process M&E)

Your M&E system will need to cover both M&E for process and for results or impact. First focus on M&E for process, as this is often a part of M&E that is overlooked. When M&E is used to think about the process of doing each building block, then you do not necessarily need to use indicators. In fact, mostly you will be recording what is happening and talking to people who are involved to hear and record their views about how the process is going. The following steps (or the table below) will ensure that you are systematic in designing your M&E system for process:

• Assign specific responsibility to look after M&E for each building block
• Plan how and when you will document the key steps (key components) in each building block
• Plan what other questions are important to answer for this building block (see some possibilities in box below).
• Plan how you will find out or measure each identified what above.
• Plan when information should be collected. For instance will information be collected just once, or at several points during the building block?
• Plan how and when to communicate and report information collected
• Document your M&E plan for this part of the M&E system.

Ideas about what aspects of process you can monitor and evaluate

What you might choose to measure or find out. These questions are starting points; develop your own set of questions that are of most interest:

1. How efficient has the process been? e.g. Was the resource expenditure as planned or not and why? Was the timeline as planned or not and why? How costly were the activities compared with the benefits? This sort of information is of interest to those people managing the CDS process and for accountability purposes.
2. What worked well and what didn’t work well? e.g. The range of people’s views about what worked and what didn’t? What key barriers and challenges were met, why they arose and how well they were dealt with (timeliness etc.) What would you do differently next time? This sort of information is useful for everyone involved in carrying out the building block and can inform future CDS processes
3. What was the extent of meaningful participation achieved in this building block and why? How could meaningful participation been improved?
4. What was the extent to which sustainability was incorporated into this building block? How could sustainability have been better incorporated?

.

What do you want to measure or find out about?
Gathering and managing information	Communicating and reporting information
How you will measure it (what kind of data will be collected and how)? How will the information be analysed?	Who will collect the information? Who will analyse the information?	When will the information be collected (how often, when)?	How will the information be communicated and discussed? (what format? What level of detail?)	To whom will the information be communicated and with whom will it be discussed?	When will the information be communicated and discussed?
E.g. An indicator, or a ranking of the importance of an issue from a survey, collection of stories)			E.g. Reports to specific meetings, or Monthly, quarterly, annual reports etc.

c) Plan the monitoring and evaluation system for results and impact of the CDS

The steps for planning the analysis of the results or impact of the CDS are as follows include defining roles and responsibilities followed by choosing indicators and measures that help tell you about progress for the CDS themes.

Refine the CDS objectives and set targets

You will need to refine the CDS themes and objectives to allow measurement of progress. Where possible, set targets that you wish to meet for each objective. The objectives need to be very clear to enable measurement against them. The targets could be set for specific years in the future such as 2010 and 2020. Most CDSs include four or more themes in the areas of growth and development, poverty reduction, good governance and financial stability and will have a set of objectives under each of these. It will be important to keep in mind targets which are used by the SEDP and its results oriented monitoring framework and the Millennium Development Goals and make sure your objectives and targets are aligned with these and include indicators for social, economic and environmental areas.

Plan how you will measure progress towards objectives

Plan the information sources that will use to monitor and evaluate progress against the CDS themes or objectives. There are many potential sources of existing indicators that you might want to choose from. See below for the sources of indicators used in the Nam Dinh CDS, which include SEDP indicators and other sources.

You may also need to think about developing some specific indicators very closely related to your CDS themes and objectives and specifically for your city’s context. Such indicators would not be based on existing data and would require effort to collect. This means only a very small number should be chosen as otherwise the resources required will not be feasible (see example below about how some new indicators were developed for the Nam Dinh CDS).

Qualitative indicators (eg. how satisfied are a particular group in the community about a service, or how would key NGOs rate the current level of homelessness) will be very useful to help you tell if your CDS projects are working well or not.

The range of sources of existing information that informed Nam Dinh’s choice of indicators

Nam Dinh prioritised a set of indicators drawn from a range of existing sources including:

1. Party's guiding documents,
2. Socio-economic Plans (Central, Province and City),
3. Master/Spatial Plans,
4. Government Statistics Office,
5. Comprehensive Poverty Reduction and Growth Strategy and
6. Class II city standards (that applies to Nam Dinh).
7. Good governance indicators promulgated by the World Bank
8. Indicators used by other citiesNam Dinh also worked with the Statistics Office to develop two new indicators that were not already in use.

Example: Monitoring framework for Can Tho

The proposed M & E framework in Can Tho has been drawn from the key strategic objectives for each theme. For example, for Theme 2 in Can Tho’s CDS, ‘Infrastructure development and environmental management’, the following strategic objectives are outlined: To provide urban infrastructure that Improves urban living quality (with the poor directly benefiting); allows Can Tho to become a centre for economic and cultural activities and a transportation node of the Mekong River Delta region; attracts investment; contributes directly to the protection of urban community’s living environment and the natural environment in Can Tho city region, especially the river system.

Some of the indicators assigned to track progress against the strategic objectives for this theme are:

The number of passengers using Can Tho Airport by 2015

- The amount of goods transferred at ports

- Transportation density increase

- The amount of water consumed

- The amount of waste collected

Each of these indicators had base line data collected, and targets set for the future, so that relevant officials not only know what to measure, but how to compare actual progress against what the city is aiming for.

discussions were held with four ward and sub-ward officials.

SMART acronym can help you choose good indicators

S is for SIMPLE

M is for MEASURABLE

A is for ATTRIBUTABLE

R is for RELEVANT

T is for TIMELY

Prioritise a small set of key indicators that are closely related to the themes of your CDS using the SMART acronym shown above. A recent UNDP report noted that in Viet Nam, people tend to collect too much information (that is, use too many indicators) and so it will be important to choose a small set of quality indicators that give you the best feedback information possible about the success of your strategy as it is being implemented in the city. You may be able to use the initial assessment as a baseline against which to measure change. If there have been additional indicators added that do not already have a baseline, it will be important to schedule this as soon as possible.

Plan to collect information through consultation too. Information obtained through speaking with different stakeholders will be very important to collect too, in addition to the indicators. Talking to people will help you to find out the reasons why things are, or are not, happening. It will help you find out about types of changes that were not anticipated.

Plan when the indicators will be measured and when stakeholders will be consulted and allocate responsibility for their collection and data management.

Plan how and when information will be communicated and shared. This includes to whom, what format, how often etc. and setting up response systems so that new information provided by the M&E system leads to adjustments in the implementation of the CDS.

Plan how you will assess risks to achieving the CDS objectives

It is extremely important to consider the risks that may erode the success of the CDS, and collect information about these risks so that processes can be adjusted accordingly. The STEEP acronym can help you identify different types of risks that will influence the success of the CDS.

STEEP acronym can help you assess risks

It is worth asking “What factors might erode our success?’ and how can we find out more about them?” By thinking through different types of risks that might prevent success of each part of the CDS, you will be able to identify ways to determine if these risks are preventing the success of your strategy

S is for SOCIAL

T is for TECHNICAL

E is for ECONOMIC

E is also for ENVIRONMENTAL

P is for POLITICAL

Document your M&E plan

Document the various decisions made on the objectives or targets, means of measuring progress towards them, and the likely risks and how they will be assessed and use this plan to schedule the M&E activities along with other CDS tasks.

d) Implement your monitoring and evaluation system

The following actions will help you successfully implement your M&E system as a whole.

• Establish an internal information system to accommodate the different types of data, allow analysis and storage of that data.
• Collect, file and manage data from M&E for process and M&E for impact
• Prepare project progress reports as planned; providing and sharing information at local, departmental, national, sectoral and local levels.
• Set up systems to make sure that information is used and that responses happen based on feedback information collected in the M&E system. That is, if an indicator is not improving, set up a system to make sure that this triggers a response to re-examine the activities and look at why the indicator is not improving and at what adjustments to activities might improve that indicator. This may require particular points in time to be scheduled for review of the CDS activities and implementation so they can be adjusted in the light of what progress is (or isn’t) being made towards the CDS objectives.
• Conduct initial, mid-term and terminal evaluations using, if needed, consultants, in line with the contents of the approved feasibility study or program or project document; acting as the focal point for coordinating with the donor or competent management authorities to evaluate the program or project

e) Review your monitoring and evaluation system

Commit to periodically review your system, set dates for this. Develop a set of questions to answer in reviewing the M&E system. In particular, go back to the objectives you set for your M&E system and use these to see to what extent they have been achieved.

Some ideas for additional questions are given below. You should add to these a set of additional questions that are relevant to your situation.

1. Which aspects of the M&E system are working well and which aren’t and why?
2. How much time and resources is M&E taking up and is this an appropriate amount or should the system be changed?
3. How well are processes for dissemination and communication about information collected working? What improvements could be made?
4. Is all the information that is being collected being “used” and disseminated?
5. How well are processes for acting in response to information provided by the M&E system working?
6. How well is the M&E system helping individuals and the organisation learn from their successes and mistakes?

How to integrate SEA processes with ‘Monitoring and Evaluation’

In designing the M&E process SEA can point out issues, problems, relevant environmental and sustainability indicators and required analysis through the SEA processes conducted as a part of the initial assessment.

Secondly, SEA may require monitoring of additional issues beyond the expected results of the CDS implementation. Some examples include:

• Monitoring requirements as outlined in the SEA implementation recommendations
• Consolidation of an ongoing M&E for the ‘health’ of the city which might include monitoring of particular critical environmental and sustainability factors such as those defined in the initial assessment

How to Integrate Consultation and Participation into ‘Monitoring and Evaluation’

The process described above for conducting monitoring and evaluation is intended to be a consultative one, as the main method you will use is talking to people who are involved to hear and record their views about how the CDS process is going. However different stakeholders will have different views about ‘how the CDS is going’. This means that it is important to pay careful attention to the range of people you seek information from. You need to make sure you are speaking to enough different people to ensure that the evaluation process is informed by a range of views and perspectives.

It is also important to pay attention to the process for communicating the results of the monitoring and evaluation phase, to ensure that a wide range of stakeholders are informed about how the CDS is going, and also to provide a means for them to give feedback if they choose to.

Helpful hints

The most important hint is to set up the right conditions for successful monitoring and evaluation. Many projects, both CDSs and other similar projects, have found that there are ways to increase the success of the use of M&E by creating an enabling environment for M&E. Some ways to achieve this are:

• Invest in training in M&E skills, as often it is not an existing skill/strength
• Take actions to ensure understanding, willingness, conviction of local leaders and relevant stakeholders to participate in and value M&E
• Consider your organisational culture and think about how to increase a focus on an attitude of learning, rather than judging in M&E
• Create ways to integrate M&E into day by day processes rather than as one-off events
• Create synergies with a communication strategy for the CDS eg public education and awareness raising
• Make sure to do the M&E planning very early in the CDS process
• Ensure adequate resourcing of M&E as it needs serious allocation in budget (10% as a guiding figure)

Resources

Cities Alliance (2005) Guidance Framework: Integrating Monitoring and Evaluation into City Development Strategies

Ministry of Planning and Investment (2006) Report on M&E by Local Government in Viet Nam: Issues, Problems and Ideas for Improvement. Written by Tran Hung, Nguyen Thi Hong of Gioviet Consulting Co. Ltd., Hanoi November 2006

FERD/MPI (2007) Monitoring and Evaluation Manual, Foreign Economic Relations Department/Ministry of Planning and Investment. Prepared by The Viet Nam–Australia Monitoring and Evaluation Strengthening Project – Phase II (VAMESP II) 2004–2007