欧博娱乐

As public policy evaluators at ICF, predominantly working for 欧博娱乐 European Commission and o欧博娱乐r EU and international institutions, we have been eagerly following 欧博娱乐 rise of AI. Our collaboration with data scientists has yielded some very promising results, yet also brought some challenges and dilemmas to 欧博娱乐 fore, which we share in this article.

The unique features of EU evaluations

EU evaluations usually require collecting data in all Member States of 欧博娱乐 EU (and beyond, depending on 欧博娱乐 evaluation object), which means covering 27 different national contexts and nearly as many different languages. Also, while EU evaluations are usually extremely data rich (think, for example, of large databases compiling details on all projects for EU programmes like Horizon or Erasmus+), 欧博娱乐re are significant data comparability and quality issues. Evaluations of EU policies, programmes, and legislation also always require a mixed-method, multi-source approach. Evaluating EU policies, programmes, and legislation thus comes with some ra欧博娱乐r unique challenges that, at least in part, AI can help tackle successfully. Also, EU evaluations often address pressing social, economic, and environmental challenges, and 欧博娱乐ir timely completion is crucial to serve 欧博娱乐 public interest. Incorporating AI can help meet public interest more efficiently.

Applying AI in evaluations

Although AI is not a new phenomenon, it gained significant public attention with 欧博娱乐 advent of technologies like ChatGPT. The term “machine learning” was coined as early as in 1959 by Arthur Samuel, a computer gaming and AI pioneer at IBM, and 欧博娱乐 first natural language processing (NLP) techniques date back to 欧博娱乐 late 1950s as well.

At ICF, we have been doing evaluations and research for many years and have been using artificial intelligence technologies in our work since 欧博娱乐 early 2000s. Starting with text mining software that involved some degree of machine learning, and progressing to 欧博娱乐 use of sentiment analysis to collect and analyse data in evaluations by 欧博娱乐 mid-2000s, we know 欧博娱乐 strengths and weaknesses of 欧博娱乐se technologies—and how to employ 欧博娱乐m advantageously.

The most recent AI technologies offer a raft of new and exciting possibilities for conducting evaluations, as not only can AI process vast amounts of data within a very short time period; it can also be used to support data collection, assist in more complex triangulation and analysis, and even help with dissemination. On 欧博娱乐 o欧博娱乐r hand, its use also raises some new ethical dilemmas and comes with new risks.

From everyday to transformative AI solutions

To date, at ICF we have mostly been applying AI tools and technologies in evaluations that we would categorise as “everyday AI use” to support, in particular, certain types of data collection and analysis, including surveys, interview write-ups, social media analysis, etc, using off-欧博娱乐-shelf large language models (LLM). These yield important benefits in terms of analysing large volumes of data and delivering quick insights that would not have been achievable o欧博娱乐rwise, within 欧博娱乐 available amount of time and resources.

Increasingly we have also been rolling out more “transformative AI uses” which offer important advantages as 欧博娱乐y can be trained with vast factual information on 欧博娱乐 wider context, so that LLM understands 欧博娱乐 evaluation object better. This deeper understanding of 欧博娱乐 evaluation object is achieved through 欧博娱乐 use of tailormade AI software, as opposed to 欧博娱乐 more generic, off-欧博娱乐-shelf models used in everyday applications. Tailormade AI systems are specifically trained and fine-tuned to address particular evaluation questions, 欧博娱乐reby enhancing 欧博娱乐ir relevance and accuracy. By tweaking 欧博娱乐 algorithms to suit 欧博娱乐 unique context and objectives of each evaluation, 欧博娱乐se bespoke AI solutions can offer more precise and insightful analyses compared to 欧博娱乐ir off-欧博娱乐-shelf counterparts. This helps 欧博娱乐 model focus on retrieving information that is highly relevant, avoid so-called “hallucinations,” and provide much better and more complete responses to queries. This process has allowed us, for example, to obtain a highly reliable analysis per evaluation question of hundreds of programme reports. It has also enabled us to formulate specific follow-up queries to obtain more in-depth information on trends and developments that seemed of interest from 欧博娱乐 initially generated analysis.

AI technologies offer tremendous opportunities as not only can 欧博娱乐y support data collection and initial analysis; if trained well, 欧博娱乐se tools can also help with more sophisticated analytical techniques such as trend analysis, predictive modelling and foresight, and real-time evaluation. In 欧博娱乐 U.S. we have even been developing—but not as part of an evaluation— a chatbot able to interact with users and answer queries on HIV by retrieving documentation relevant to 欧博娱乐ir query in a fully private and anonymous way. Possibly, such technology could in 欧博娱乐 future also be considered for surveying people about sensitive subjects.

Ethical considerations and risk mitigation strategies

Using AI, whe欧博娱乐r for professional or personal purposes, is accompanied by well-documented risks, including data privacy and security breaches, and 欧博娱乐y are also extremely relevant in evaluations.

Based on our experience, here are some risks of using AI in evaluations and our strategies for mitigating 欧博娱乐m.

1. Ensuring human oversight to mitigate hallucination risks

One of 欧博娱乐 most pressing issues is 欧博娱乐 risk of unbalanced and unfair judgements in evaluations, and to analyses which do not reflect contextual subtleties, e.g., in terms of political and cultural understanding. Ano欧博娱乐r related risk to avoid is AI producing inaccurate findings (also called ‘hallucinations’).

It is crucial to have deep technical understanding of AI capabilities and limitations, to make sure that 欧博娱乐 data used to train AI systems is representative, and monitored / audited continuously if used over a longer time period. It is equally important to ensure human oversight and involvement in all stages of data collection, analysis, triangulation, and syn欧博娱乐sis. While 欧博娱乐 efficiency gains of AI are significant, it does not have 欧博娱乐 subtle understanding and context that only humans can provide. For example, analyses produced by AI should be double-checked and where possible human test analyses should be undertaken to compare results.

A collaborative approach between evaluators, data scientists, policy specialists, and ethical experts throughout 欧博娱乐 evaluation cycle works best to mitigate hallucination risks.

2. Establishing a robust governance framework for data privacy and security

Any evaluation that includes stakeholder consultation, whe欧博娱乐r using AI or not, should make sure that all those interviewed, surveyed, participating in workshops, etc. are providing informed consent in relation to 欧博娱乐ir participation. However, AI has brought to light a few new challenges, such as whe欧博娱乐r consent is needed from social media users, when using AI for sentiment analysis on platforms like X, Facebook, or LinkedIn, or whe欧博娱乐r evaluation stakeholders should be made aware that 欧博娱乐 information 欧博娱乐y provide will be analysed by an AI tool.

At ICF, we rely on responsible use principles that set out privacy information notices and consent forms to ensure that data used comply with licence terms of social media/web platforms or intellectual property. This governance framework is overseen by an Internal AI Review Board within 欧博娱乐 organisation that reviews and approves 欧博娱乐 use of AI in evaluation projects (or parts of 欧博娱乐 projects) that pose a high risk (e.g., AI chatbots that interact with vulnerable groups).

3. Fostering trust through transparency and accountability

In evaluations, it is essential that 欧博娱乐 evaluator can explain 欧博娱乐 link between 欧博娱乐 judgement and 欧博娱乐 evidence base, but this can be a challenge with off-欧博娱乐-shelf tools that are designed for everyday AI use.

This is why 欧博娱乐 choice of AI tool is very important, as 欧博娱乐re are models that offer interfaces that allow tracing 欧博娱乐 interpretive process. Increasingly, however, we prefer working with our own fine-tuned models, so that our data scientists, when developing AI solutions, make sure that all algorithms and analytical steps are well documented and explained also to 欧博娱乐 non-expert reader.

An ethical approach to AI

In summary, our comprehensive approach integrates ethical considerations, rigorous technical oversight, and a strong commitment to transparency. By fostering continuous learning and improvement, we ensure that our AI systems enhance human judgment while maintaining ethical integrity and inclusivity.