To support this evaluation, Frontex initiated an activity in 2018 called ‘Performance Assessment Methodology for Document Inspection Systems (PAMDIS)’. The aim of this activity was to propose a methodological framework for operational testing and evaluation of DIS in border checks, and to test the methodology in real-life border control scenarios.
To overcome the limitations of the first methodology in terms of completeness and applicability, Frontex initiated another activity in 2021 called ‘PAM4DIS 2.0’ to develop a practical capability tool for operational testing and evaluation of DIS. This tool will support the operational performance assessment of DIS and through testing an evaluation of DIS under realistic conditions.
As major results of the said activity, the PAM4DIS toolkit introduces a practical approach for applying the methodology in a stepwise process, describes the creation and compilation of test document sets, provides for the preparation of protocols and questionnaires, details the process of conducting the test, and supports the interpretation of results through guidelines.
Measuring the performance of DIS is complex. It must therefore be broken down into measurable, elementary dimensions (i.e. performance indicators). Proper weights need to be developed for and assigned to each of the elementary dimensions, the combined score of which will equal the overall performance value or “score” for a DIS. In the context of this activity a complete list of all relevant performance indicators – authentication correctness, data capture quality, speed, usability, availability of service, and interoperability – has been developed with the proper weight assigned to each performance indicator.
The PAM4DIS 2.0 Guidance Document provides Frontex and Member States with the means to evaluate a DIS as part of an operational border check system. One main strength is the practical application to various border scenarios and its robustness against low testability of current DIS installations. Another positive aspect is the support by an additional toolset of files that implements the procedure, formulas, and weighting.