The activities of ReSPA are
funded by the European Union

PA Awards - Evaluation Guidelines

Important Aspects for Assessors and Jurors

Your engagement as an assessor or juror, and the decisions that you make in the process, will have an impact on the awards and its organisers, the individuals whose work you evaluate, and your colleagues who join you as assessors or jurors.

  • Before accepting the invitation to serve as assessor or juror, please familiarise yourself with the application rules and the assessment criteria established by the award organiser.
  • Your agreement to serve as an assessor or juror establishes a bond of trust between you and the award organiser. The consequences, should that trust be broken, are outlined in this document.
  • The assessment and judging process for the competition should be clearly understood and carefully observed.
  • As assessor or juror, you are contributing to the success and further development of the award and its improvement for future editions. You will be encouraged to share observations and recommendations about the methodology and process between the pool of colleague assessors and jurors and the organisers of the Award. By discussing the methodology and process and agreeing on the criteria and procedures, you are significantly contributing to the success, transparency and fairness of the award.
  • Assessors and jurors need to state a potential conflict of interest with any of the submitted applications and will not be able to evaluate, vote on or comment on these applications. These applications will be assigned to another assessor or juror.

A conflict of interest is possible if the application is submitted by an organisation from the same country as the assessor or juror, or an application where the assessor or juror has been directly involved in any project phase, or when there are close relationships (family or work related) with the people involved in the PA Award.

  • Assessors and jurors must treat all information about the products, applicants, process and results as strictly confidential.
  • The evaluation process and voting will be conducted via the Public Administration Awards in the Western Balkans platform.


Evaluation Process - Objectives

Process:

  • Independent
  • Fair
  • Consensus based
  • Transparent

Persons (Jurors, Assessors):

  • Multi-stakeholder
  • Gender balanced
  • Professional expertise
  • Balanced in terms of national and regional background

Content/Outcome:
Excellent, inspiring, practical solutions (tools, systems, approaches) from the region, demonstrating the diversity within countries, creating a benchmark and learning opportunity for the region.


Evaluation Process


Phase 1: Eligibility CheckAward secretariat will assess all submitted initiatives confirming that:

  • The initiative has been submitted by a public sector institution from Albania, Bosnia and Herzegovina, North Macedonia, Montenegro, Serbia or Kosovo*.
  • The application form is filled out completely and adequate information has been submitted to permit evaluation of the initiative.
    Clarification: “False applications” and applications that are not relevant to any of the categories will be disqualified.
  • Applicants whose submission is incomplete will be contacted by telephone or email by the award secretariat in order to provide further information.
  • Notification of ineligible applications will be given by the secretariat.


Phase 2: Screening and Interview Phase for Assessors

Phase 2a – Assessment Phase: initial screening, overview of all applications and adjustment of methodology

Introductory video call with all assessors and award secretariat:

  • Explaining the methodology, process and platform functionality to the assessors.
    Depending on the number and quality of submissions, the assessors may adapt the process in agreement with the award secretariat.
  • Forming the assessor teams, depending on the number of applications (each submission will be evaluated by two assessors).
  • Teams may change if an assessor states a conflict of interest for an assigned application.
  • Agreeing on a schedule for video calls, discussing the process, criteria, methodology, answering questions.
  • Introducing the awards platform and its technical functions.

Assessors start screening all submitted applications that meet the technical awards criteria and start to:

  • Note questions and comments for each submitted initiative.
  • Note comments about unclear or missing information in the platform.
  • Assessors submit their individual votes using the platform.
  • Assessor teams discuss their votes, rank applications and comments and agree on:
    • applications that need a certain clarification, more information, etc – the award secretariat will collect this additional information from the applicants.
    • the best applications that they have evaluated and that should be considered for the interview and data collection phase.
  • The number of applications considered in phase 2b may vary between the teams of assessors, depending on the quality of the randomly-assigned applications.

Phase 2b - Interview Phase: telephone calls with the most promising applicants

  • The award secretariat will schedule calls between applicants and assessors to find out further information, answer open questions and get a feeling for the team behind the initiative.
  • The award secretariat will provide a questionnaire for these calls and schedule the calls between assessors and applicants, supported by an online tool.
  • Assessors take minutes during the calls and upload them on the contest platform.
  • After the calls, assessors review their votes through the platform.

Phase 2c - Shortlisting Phase: award secretariat and assessors

  • The award secretariat reviews the final votes from the assessors in a combined list of rankings and recommends a list of shortlisted projects (reflecting the quality and diversity of projects).
  • Assessors will have the opportunity to speak for/against specific applications and agree on the suggested shortlist to be presented to the jury.
  • The award secretariat informs all shortlisted projects and those not selected and announces the finalists/shortlist on the awards webpage.
  • Assessors prepare the list of rankings of projects with brief comments and explanations and present the suggested shortlist to the secretariat, including comments on why an application was or was not selected for the shortlist.


Phase 3: Jury Meeting

In the case of a small number of applications (up to 20), the entire process will be conducted by the International Experts Jury as explained in the description of the Phase 2 (2a, 2b and 2c).

The awards secretariat will provide support with telephone calls, requesting information, etc.
Jury meeting with international experts (on-site), resulting in the decision on the winning initiatives of the first Public Administration Awards in the Western Balkans (depending on the coronavirus (COVID-19) measures, this meeting might take place online):

  • The award secretariat presents the methodology and process to the jury members.
  • The award secretariat presents the complete list of rankings to the jury members and presents the shortlisted solutions.
    The jury of experts can request the removal applications from the shortlist, or their inclusion, if there is agreement by the majority of the jury members and sufficient justification.
  • All agreed shortlisted applications will be presented by the assessors to the jury for evaluation, including information on:
    • why the initiative could be a winner
    • why is it better/more effective than others
    • what was the concrete challenge and solution
    • any numbers, impact achieved, further information
    • the local context.
  • The Jury is responsible for discussing all selected applications in terms of award criteria and carefully considering all of the parameters of the initiatives.
  • The jury can access each application through the platform and review all submitted material, including the comments of assessors, the information gathered and the insights provided through the telephone calls.
  • Following the presentations and discussions, jury members will vote on each project through the system.
  • The secretariat will present the voting results to the jury, who make the final decision on award winners and provide the reasons for their decision.


Evaluation Criteria for Assessors and Jury Members

Criteria Definitions Sub-Criteria

Adaptable

Evidence shows that the initiative would also improve the level of adaptability of a government in possible future situations of crisis.

The initiative must provide regulations, procedures, mechanisms and means to ensure its sustainability and resilience in the future.

  • The initiative should reflect the process of change adaptation caused by external factors (at societal / global scale).
  • Adjustment of management style during the COVID-19 pandemic

Forward looking

The initiative must introduce new ways of working or delivering services in the public sector and an innovative approach in addressing problems occurring in extraordinary circumstances.
The translation of new ideas into practice and harnessing new technology.

  • The initiative should present new ideas, approaches
  •  The initiative uses modern tools and contemporary solutions.
  • Inventive in strategy, process, and/or system.
  • New ways of mobilisation and use of resources (human, financial, technical, etc.) in a crisis context.

Effective & Impactful

  • Changes and transformations have been noticed since the implementation of the initiative in addressing the problems identified when designing the project as a response to the crisis concerned.
  • Relevance: The initiative should have made a positive impact on a group(s) of the population and/or address a significant issue of public concern within the context of a given country or region in a the context of a crisis.
  • The evidence should demonstrate that the initiative was effective, efficient (if applicable), had an economic impact for the Institution and was implemented using transparent methods in the context of a crisis.
  • Existence of statistics, ratios, etc. is encouraged.

 

Participatory & Inclusiveness

  • The initiative is based on an open approach including all concerned and affected parties in the context of a crisis, and captures diverse views, through:
  • First, the involvement of the beneficiaries and other stakeholders in the design, operation, management, monitoring and evaluation of the project in the context of a crisis;
  • Second, the ownership of the project by the said beneficiaries and other stakeholders in the context of a crisis.
  • Identification and mapping of the beneficiaries and other stakeholders
  • Citizen engagement.
  • Information, consultation, participation, involvement of the beneficiaries and other stakeholders.
  • Leaving no one behind in terms of taking into account disadvantaged, marginalised, and vulnerable populations.
  • Gender mainstreaming.
  • Building and promoting Partnership: The initiative should demonstrate that it has engaged relevant stakeholders and partners.

Transferable & Replicable

The possibility of replicating the project, partially or fully, in a different setting or context in the Western Balkan Countries during and after the crisis context.

  • Dissemination of the initiative as successful and best practice at the state level and in the region.
  • Potential of universal appeal, replication and transfer.
  • the initiative can be adopted to solve problems in similar organisations and in similar, extraordinary situations.
  • It can be adopted to solve problems in similar country set-ups and similar, extraordinary situations.
  • Documentation and building on (developing) the initiative.
  • The initiative can serve as a case study for ReSPA and its Network.


How to Nominate Guidelines EVALUATION GUIDELINES DOWNLOAD

Sign in with the username and password you received from the contest organisers
Login link: https://db.icnm.net/Signup/PAA/2020/

Please note: For technical reasons, assessors and jurors will be both called “juror/jury” in the database.

1

Sign In

2

Complete your profile before you start your evaluation

3

Evaluation overview page

4

Evaluation of an application

5

Evaluation

Overview for assessors/jurors – you now see that the evaluated project has been moved to “Evaluated projects”.

THANK YOU!

If you have any questions related to the rules, contest, etc, please contact:
pa-awards@respaweb.eu

If you have any difficulties with the awards database, please contact
office@worldsummitawards.org