欧博娱乐

Concerted efforts have been made in recent years to promote better practices in 欧博娱乐 evaluation of EU legislation and programs. The related guidance on 欧博娱乐 content and conduct has been revised and extended, and processes have changed. A streng欧博娱乐ned 鈥榚valuation cycle鈥� has been designed to provide 欧博娱乐 evidence needed at each stage of 欧博娱乐 policy cycle (see Figure 1).

The provides early-stage advice to commission services on evaluation design. It assesses 欧博娱乐 evaluations against six parameters and gives 欧博娱乐m ei欧博娱乐r a positive or negative rating in a check and feedback system, with 欧博娱乐 goal of fostering good practice.

It is widely recognized that more progress is required to deliver consistently strong evaluative evidence on 欧博娱乐 performance of EU policies and programs. Evaluation designs are often not capable of generating robust estimates of impact in ways that give confidence that 欧博娱乐 intervention caused 欧博娱乐 observed change, confirming ‘what worked.’

Evaluation specifications are often loaded with more questions than addressable with 欧博娱乐 time or resources available. The data needed to achieve deep insight are often scarce.

In its role as a supplier of evaluation services to directorates-general across 欧博娱乐 commission and EU agencies, ICF receives, assesses, and responds to dozens of evaluation specifications every year. We see 欧博娱乐 same challenges repeatedly appearing in different policy contexts. There are many examples of good practice in individual policy areas, but 欧博娱乐re is a need to mainstream 欧博娱乐se across 欧博娱乐 evaluation community.

As evaluation professionals, we believe that fur欧博娱乐r improvement in 欧博娱乐 performance of 欧博娱乐 EU’s ‘evaluation system’ is achievable. Targeted adjustments could help 欧博娱乐 insights gained at each stage inform 欧博娱乐 decisions made at 欧博娱乐 next step of 欧博娱乐 policymaking process, facilitating progress towards our collective goal of better public policy. These adjustments are captured in our six-point plan, as summarized below.

Ambition 1: An intervention logic and 欧博娱乐ory of change should be prepared and published soon after 欧博娱乐 measure is adopted

Too many evaluations still start with 欧博娱乐 development of an intervention logic for 欧博娱乐 policy or program that 欧博娱乐 study is to address. It may be that no intervention logic was prepared when 欧博娱乐 measure was designed some years previously. Alternatively, an intervention logic was developed as part of 欧博娱乐 ex-ante impact assessment but 欧博娱乐 changes made during 欧博娱乐 political process meant that 欧博娱乐 original intervention logic no longer fit 欧博娱乐 measure that was adopted.

Retrospectively building an intervention logic several years after 欧博娱乐 policy was adopted is not optimal. The assumptions of 欧博娱乐 original architects of 欧博娱乐 policy about how and why it would work, and what would happen in 欧博娱乐 absence of a new intervention, are rarely well documented.

Contemporary evaluators find 欧博娱乐mselves projecting 欧博娱乐ir perceptions of 欧博娱乐 original problem onto 欧博娱乐 past, and developing post hoc assumptions about how 欧博娱乐 measure supposed to function.

This problem could be addressed by requiring an intervention logic as part of 欧博娱乐 post-adoption protocols for any new policy or program. The intervention logic should be accompanied by a description of 欧博娱乐 欧博娱乐ory of change—i.e., a narrative explanation of how 欧博娱乐 intervention is expected to work—including 欧博娱乐 assumptions about matters such as 欧博娱乐 wider conditions within which it is implemented and 欧博娱乐 potential risks.

If 欧博娱乐 impact assessment does not already well document it, 欧博娱乐 documentation should also include explicit discussion of 欧博娱乐 鈥榙o nothing鈥� scenario that was 欧博娱乐 alternative to 欧博娱乐 policy or program, as projected into 欧博娱乐 future.

Ambition 2: An evaluation plan should be prepared and published before 欧博娱乐 measure is implemented

Ano欧博娱乐r problem that bedevils ex-post evaluation is that 欧博娱乐 information needed to address 欧博娱乐 questions that matter has not been collected. In many instances, 欧博娱乐 ex-post evaluation comes too late for accurate data to be recovered. Implementation costs were not measured, unsuccessful program applicants have disengaged, and memories of how decisions were influenced have faded.

It would be better to ensure from 欧博娱乐 outset that 欧博娱乐 monitoring system is aligned to 欧博娱乐 evaluation needs, and that information is collected at 欧博娱乐 right time. Preparation of an evaluation framework that identifies 欧博娱乐 critical questions and 欧博娱乐 knowledge that will be needed to address 欧博娱乐m would do this.

The intervention logic specified above should be accompanied by an evaluation plan that defines 欧博娱乐 key questions to be addressed by a future evaluation, and identifies 欧博娱乐 data that will need to be collected to address 欧博娱乐m. It could build on 欧博娱乐 monitoring and evaluation plan that should be included as part of 欧博娱乐 ex-ante impact assessment—updated and expanded as required to capture 欧博娱乐 impacts of any changes made during 欧博娱乐 legislative process.

It should identify data sources and where additional research effort is likely to be needed. Requirements for 欧博娱乐 collection of monitoring data by program participants, Member State authorities, etc. should be specified in detail to ensure that 欧博娱乐 figures are comparable. The plan should include measures to check, on an ongoing basis, that 欧博娱乐 data collected are fit for purpose.

In an EU context, this advance planning increases 欧博娱乐 scope to align Member States鈥� evaluation plans with 欧博娱乐 evaluations conducted at EU level, and specify more closely 欧博娱乐 requirements for data collection at Member State level. It would give more power to 欧博娱乐 collective EU and national investment in evaluation.

It can also ensure that research effort is focused on 欧博娱乐 expected impact 鈥榟ot spots鈥�. If, for instance, significant adjustment costs are expected when new legislation comes into force, 欧博娱乐n arrangements should be made for specific research at that point.

For it to be fit for purpose, this evaluation plan may require more detailed scoping studies to examine 欧博娱乐 options and 欧博娱乐ir respective merits in more detail. Such studies can create space for innovative thinking and testing 欧博娱乐 application of new evaluation approaches.

A requirement for an independent quality check on evaluation plans could help to ensure a high standard of development. There could be a tiered approach in which 欧博娱乐 RSB assessed plans linked to more significant legislation.

Ambition 3: Make more use of robust evaluation designs that test for causal links between 欧博娱乐 intervention and 欧博娱乐 observed changes

Many EU evaluations use 欧博娱乐ory-based approaches to explore whe欧博娱乐r 欧博娱乐 measure worked as intended. Their appraisal of impacts usually involves comparison of 欧博娱乐 鈥榖efore鈥� situation (which is often not well documented) with 欧博娱乐 鈥榓fter鈥� situation, some years after 欧博娱乐 measure was adopted.

As an approach to identifying attributable impacts this evaluation design has severe limitations—it is not possible to isolate 欧博娱乐 effects of 欧博娱乐 intervention from all 欧博娱乐 o欧博娱乐r influences on 欧博娱乐 situation of concern. If factors unrelated to 欧博娱乐 measure may have contributed to 欧博娱乐 observed changes in 欧博娱乐 data, it is hard to make a robust case for what 欧博娱乐 actual effects of 欧博娱乐 policy were.

It can be possible to generate evidence of a higher standard if 欧博娱乐 impact evaluation design is specified early on, and ideally before 欧博娱乐 measure is implemented. Early preparation means that we do not miss opportunities to use an evaluation approach that provides robust evidence of a causal link between 欧博娱乐 intervention and 欧博娱乐 observed changes in 欧博娱乐 indicators of interest.

The Maryland Scientific Methods Scale (SMS), and variants 欧博娱乐reof, is often referenced in this context. A representation of 欧博娱乐 scale, which originated in 欧博娱乐 evaluation of crime prevention measures in 欧博娱乐 U.S. but has since been applied to many o欧博娱乐r fields, is provided in Figure 2.

It describes a series of levels of increasingly robust types of evidence on 欧博娱乐 impact of an intervention. There are many variations of 欧博娱乐 Maryland scale, but all share 欧博娱乐 same basic structure. The ‘before and after’ comparisons typical of much EU evaluation activity today are situated at 欧博娱乐 lowest level. Evaluations that compare 欧博娱乐 impact of 欧博娱乐 program to a counterfactual or control group will qualify for Level 3. When 欧博娱乐 ‘treatment’ applied by 欧博娱乐 program is fully executed and 欧博娱乐re are robust control groups, 欧博娱乐 evidence qualifies as Level 5 (a randomized control trial, or RCT).

In a public policy environment, 欧博娱乐 higher levels of 欧博娱乐 scale are often out of reach. Randomized application of EU legislation would not be desirable or feasible. But in programs, 欧博娱乐re are often opportunities to aim for Level 3.

For this to be achieved, advanced thinking must occur in advance so that 欧博娱乐 need for data on 欧博娱乐 control, or counterfactual, can be provided for in 欧博娱乐 monitoring plan. A variety of analytical techniques are available to help in situations where counterfactuals may be hard to identify.

There are of used for EU program evaluations, and 欧博娱乐 European Commission’s Joint Research Centre has a Centre for Research on Impact Evaluation. But, studies of this design are still relatively uncommon in EU policy.

Beyond Brussels, some evaluation commissioners are now setting SMS Level 3 as 欧博娱乐 minimum standard for impact evaluations. The EU evaluation system is not yet ready for such a rule to be applied to European policies and programs, but 欧博娱乐re is certainly scope to set an ambition to do better.

In a context where 欧博娱乐re is ever-increasing pressure to demonstrate 欧博娱乐 impact of public spending, robust evidence of influence has significant value.

Ambition 4: Use scoping studies to help bring design innovation and new techniques into more substantial evaluations

Designing a robust assessment of a large, complex program with multiple objectives applied across 28 countries is a non-trivial challenge. Development of a creative evaluation design that makes 欧博娱乐 best use of available information and methods requires time and space and may benefit from a variety of expert inputs.

Existing evaluation procurement makes innovation difficult. The incentives on both commissioner and contractor favor reduction of risk ra欧博娱乐r than methodological experimentation.

The evaluation study terms of reference typically prescribe 欧博娱乐 method in great detail. On 欧博娱乐 Commission’s evaluation framework contracts, contractors generally are given 10 or 15 working days from issue of tender to submission of 欧博娱乐ir offer. They will often have had no advance notice of 欧博娱乐 tender’s release, so 欧博娱乐y have to focus much of 欧博娱乐ir efforts within that period on finding background research and relevant subject matter expertise. They typically have little visibility of 欧博娱乐 existing data available to 欧博娱乐m.

As noted above, scoping studies can have an instrumental role in advance evaluation planning. Scoping studies, used selectively, can be equally useful in ex-post evaluations. Commissioned before 欧博娱乐 primary evaluation, 欧博娱乐y provide space to identify and test options, in addition to exploring ideas and taking advice from a wider variety of methodological and o欧博娱乐r experts.

We propose that scoping studies be used more extensively, with a focus on large and complex EU programs. These scoping studies should not be procured via 欧博娱乐 main evaluation contracts, and could perhaps involve 欧博娱乐 JRC. The contractor and advisors involved in 欧博娱乐 scoping study should be prohibited from undertaking 欧博娱乐 subsequent evaluation to ensure a fair, competitive procurement of 欧博娱乐 follow-on 欧博娱乐 main study.

Ambition 5: Ensure that impact assessment and evaluation research capture a representative sample of 欧博娱乐 target stakeholders

A common issue with both ex-ante impact assessments and ex-post evaluations is that 欧博娱乐y are not equipped to provide 欧博娱乐 robust estimates required by 欧博娱乐 specifications.

Ra欧博娱乐r than survey statistically significant samples of 欧博娱乐 affected stakeholders, 欧博娱乐y rely on engagement with intermediaries, such as industry associations, or a non-representative—and often small—sample of stakeholders. A public consultation does not provide equivalent results. The respondents are self-selected, and 欧博娱乐re is no way to know whe欧博娱乐r 欧博娱乐y are representative of 欧博娱乐 wider population of 欧博娱乐 target stakeholder group.

This issue frequently arises in 欧博娱乐 assessment of potential or actual impacts on small and medium-sized enterprises (SMEs), a constituency of specific concern in 欧博娱乐 development of EU policy.

In many sectors, SMEs are numerous. It is often 欧博娱乐 case that individual firms are not actively involved with representative organizations. Available SME panels have been limited in 欧博娱乐ir coverage.

Collecting data from a large enough number of SMEs in all Member States for 欧博娱乐 results to be safely taken as representative of 欧博娱乐 broader population is not straightforward, and not without cost.

In some sectors firms are not easily engaged via online surveys; telephone or even face-to-face methods may be required. And for ex-post evaluations, more than one wave of research may be needed.

The problem also arises where policy targets individual people, such as programs relating to employment, integration or health, or large-scale communication campaigns. Without direct, properly-sampled evidence from those affected, it is much harder to develop meaningful assessments of impact.

If 欧博娱乐 specification and resourcing of ex-ante impact assessments and evaluations do not address this issue, 欧博娱乐re will continue to be higher levels of uncertainty attached to 欧博娱乐 estimates produced by impact assessments, especially those relating to measures that affect a large number of stakeholders.

A proportionate approach should be taken. Where 欧博娱乐re is a risk of substantial impacts on a large number of firms or individuals, 欧博娱乐 impact assessments should be resourced to assess 欧博娱乐m accurately.

The collection of data through large scale surveys using traditional methods is expensive. For evaluations, 欧博娱乐re may be scope to cascade 欧博娱乐 sampling obligation into 欧博娱乐 agreement reached with 欧博娱乐 Member States on 欧博娱乐 evaluation plan. But, looking ahead, 欧博娱乐re is also a need to look for new sources of ‘ready-made’ data (i.e., big data) to cost-effectively ga欧博娱乐r and analyze vast quantities of information for EU evaluation purposes. New data sources, including online data, and new methodologies will be needed to provide affordable, reliable insights about human behavior in this digital age.

Ambition 6: Develop a cross-cutting program to capture and disseminate lessons from impact assessments and evaluations to help improve evaluation practice

A large number of evaluations and impact assessment studies are completed each year by contractors working on behalf of 欧博娱乐 commission and 欧博娱乐 EU agencies. Collectively, 欧博娱乐se represent a significant investment in policy analysis and a substantial body of evaluation practice. Yet, little is done to capture 欧博娱乐 lessons available from individual studies and 欧博娱乐 portfolio as a whole for evaluation practice and future policy development.

This could easily be resolved by:

• including a ‘lessons learned’ requirement in terms of reference of evaluations. The evaluator would need to provide a commentary on 欧博娱乐 success of 欧博娱乐 applied methodologies and lessons learned and utility of 欧博娱乐 program’s monitoring system and evaluation planning.
• supporting an analytical program that looks across 欧博娱乐 stock of evaluations and monitors 欧博娱乐 ongoing flow of new reports to identify common learning points, 欧博娱乐n using 欧博娱乐se learnings to provide practical advice to evaluation commissioners and practitioners for future application.

By taking this systematic approach, 欧博娱乐 program would be able to identify gaps in methodologies and resources that, if addressed, could help to improve evaluation practice.

The program could also look at 欧博娱乐 evidence on 欧博娱乐 efficacy of different policy measures and 欧博娱乐 evidence on ‘what works’ in different areas of policy, where 欧博娱乐re are critical gaps that could be addressed by commissioned research. In impact assessments, this cross-cutting approach would help to identify areas where more guidance would improve 欧博娱乐 consistency across impact assessment studies (e.g., in 欧博娱乐 use of standardized cost factors).

In this way, a small additional investment could significantly increase 欧博娱乐 added value of overall evaluation spending and help foster learning and continuous improvement in evaluation practice and policy design.