Forward
A proficiency test (PT) is a blind test to participants that is intended to prove or disprove participant’s measurement or testing competence. Any action that breaks the integrity of that process ultimately destroys the value the PT is intended to provide. It starts with selecting a reputable provider: one who adheres to ISO/ IEC 17043 and ISO/IEC 13528; one who has invested in the properly qualified staffing to manage the PT process; one who has built the necessary safeguards into their PT process to preserve integrity; one who will help ensure a laboratory receives constructive feedback throughout their PT participation providing a cornucopia of opportunities to verify and improve laboratory operations and avoid potential pitfalls.
Provider Services
Like most consumer services, proficiency test providers differ in their scope, quality, and faithfulness (integrity) in providing required services. For providers, the following are fundamental services to be performed congruent with industry accepted practices:
• Development and dissemination of test plans and instructions • Acquisition of participants’ test data
• Analysis of participants’ test data
• Analysis evaluation and results reporting
Industry accepted practices for providers are given in ISO/IEC 17043, Second edition 2023-05, Conformity assessment – General requirements for the competence of proficiency testing providers, used in conjunction with ISO/IEC 13528, Statistical methods for use in proficiency testing by interlaboratory comparison. ISO/ IEC 17043 is characteristically used in assessing providers for accreditation. Providers are accredited by accreditation bodies such as the American Association for Laboratory Accreditation (A2LA) which denotes accreditations as:
Accreditation is synonymous with both the quality and competence of an organization, based on international standard(s). Accreditation refers to the recognition given to an organization by an authoritative body such as A2LA. It is a process by which an authoritative body gives formal recognition that a Conformity Assessment Body (CAB) fulfills specified requirements and is competent to carry out specific tasks. Accreditation is the most appropriate way to ensure an organization’s competence in performing a specified task.
ISO/IEC 17043 denotes PTs as:
Proficiency testing (PT) is widely recognized as an essential tool for demonstrating the competence of conformity assessment bodies. PT can provide evidence of competence and it can be an indicator of an underlying or emerging problem. This document is intended to promote confidence in the operations of PT providers. It contains requirements for PT providers to enable them to demonstrate that they operate competently and can generate valid evaluations of participant performance.
PTs determine laboratory performance by means of comparing and evaluating calibrations or tests on the same or similar items or materials by two or more laboratories in accordance with predetermined conditions.
This paper will focus on ISO/IEC 17043 compliance as related to topics to consider when selecting a provider. Documented evidence is deemed essential for determining a provider’s successful compliance. Provider advertisements, promotions and customer correspondence should always be delivered congruently within the context of successful compliance... this cannot be overly emphasized.
Provider Considerations
Essentially the main reasons laboratories participate in PTs are:
• Accreditation Requirements – By far the biggest reason
• Evaluation and Improvement of current processes
• Gain confidence in the ability to provide valid results
• Simply because it is good laboratory practice
ISO/IEC 17043 lists servals reasons why laboratories engage in PT’s. First and foremost is to evaluate the performance of laboratories in regard to specific measurements in order to identify differences in measurements results. These differences may or may not be satisfactory given each laboratory’s uncertainties and the known value of the measured PT artifact. Other reasons given are Identification of measurement problems, validation of measurement uncertainty claims, assigning values to reference materials, education of participating laboratories based on comparison outcomes, and additional user confidence in their measurement and test results.
Regarding laboratory accreditation and PTs, the International Laboratory Accreditation Cooperative (ILAC), ILAC P-9 denotes several requirements associated with obtaining laboratory accreditation such as demonstrated competency i.e. satisfactory PT results, minimum PT activities required for accreditation, and PT testing policies, procedures, and plans as well as identifying alternatives to proficiency testing.
ISO/IEC 17025, General requirements for the competence of testing and calibration laboratories, specifies laboratories shall monitor performance by comparison with other laboratories where available and appropriate. This monitoring shall be planned, reviewed and include, but not be limited to, participation in PTs and/or participation in interlaboratory comparisons other than proficiency testing. ISO/IEC 17025 contains the following note:
NOTE: ISO/IEC 17043 contains additional information on proficiency tests and proficiency testing providers. Proficiency testing providers that meet the requirements of ISO/IEC 17043 are considered to be competent.
PT providers do not request participants to make a statement of “Pass – Fail” relative to a certain specification. Rather, providers are looking to see if a participant is making a measurement that is in acceptable agreement via statistical means (Z-Score¹, En², etc.), with the other participants making like measurements. Participation in PT services provides documented evidence of the participant’s ability to perform testing within acceptable criteria as defined in ISO/IEC 17043 and is an accepted means to substantiate the participant’s competence to perform a test. Providers essential services must ensure:
• Test plans and instructions are fit for purpose, provide sufficient guidance and are accurate.
• Acquisition of customer data is protected, standardized and error checked
○ Ensuring the confidentiality of client data is paramount. providers must implement robust data protection measures to safeguard sensitive information. Often, providers will use non-disclosure agreements (NDAs) to formalize their commitment to confidentiality.
• Analysis of customer data is appropriate, nonbiased and provides meaningful insight
○ Providers should conduct tests and analyze data without bias. This impartiality is crucial for producing accurate and reliable results that customers can trust.
• Analysis evaluations (methodologies) are industry accepted, performed competently, and validated
○ Providers must be accountable for their methods and results. This means being open to audits, peer reviews, and providing customers with the necessary information to understand and verify the processes used.
○ Providers should only employ qualified and competent staff. This ensures that all aspects of the analysis evaluation are conducted by individuals with the necessary expertise and knowledge.
• Evaluation reporting is confidential, impartial and easily understood from a layman’s perspective
○ Transparent reporting practices are essential. This includes providing detailed and understandable reports that clearly communicate the results and their implications.
When shopping around for a PT provider, what are some of the steps one should consider helping to ensure rendered services meet your laboratory’s needs and provide reported results that are accurate and truthful? Some helpful steps are:
• Determining if a provider is accredited, this is an assurance that the provider has demonstrated compliance to ISO/IEC 17043. Using an accredited provider assures an assessor that a laboratory accreditation required PTs have been administered, evaluated, and reported per industry accepted practices.
• Determining if a provider provides tests which are applicable and beneficial to one’s laboratory activities.
• Querying how long the provider has been providing services.
• Reviewing list of companies that have used their services. • Reviewing any customer testimonials.
• Querying the provider if it has ongoing training and development for their staff to maintain and enhance their skills and knowledge.
• Determining a provider’s non-profit status. Many reputable proficiency testing providers operate as nonprofit organizations. This structure helps ensure that their services are driven by the goal of maintaining high standards rather than generating profit, reducing potential conflicts of interest.
• Determining if a provider has a system in place to gather and act on feedback from customers to address any issues promptly.
These beneficial steps help to filter out potential PT provider candidates that do not meet your laboratory’s needs as well as bringing to light any issues needing additional fact finding. The above steps provide a top-level screening of provider candidates which can then be evaluated for additional salient criteria such as confidentiality and impartiality. Confidentiality safeguards regarding customer data and evaluation reporting is clearly defined in ISO/IEC 17043 section 4.2 with sub-sections 4.2.1 and 4.25 especially noted:
4.2.1 The PT provider shall be responsible, through legally enforceable agreements, for the management of all information obtained or created during the performance of PT activities. The PT provider shall inform the client in advance of the information it intends to place in the public domain. Except for information that the client makes publicly available, or when agreed between the PT provider and the client, all other information is considered proprietary information and shall be regarded as confidential.
NOTE: The terms “proprietary” and “confidential” do not preclude publication for academic and new insights of information purposes, provided that neither clients nor participants can be identified, including by inference.
4.2.5 The identity of participants in a PT scheme shall be confidential and known only to persons involved in the operation of the PT scheme, unless the participant or the customer waives confidentiality. It can be easily ascertained that unauthorized disclosure of a participant’s data may reveal insight as to a customer’s testing methodology or capability which may be proprietary, confidential, or otherwise inappropriate to disclose. Unauthorized disclosure of evaluation results has the potential to be negatively used by a laboratory’s competition to gain unfair advantage over a laboratory either publicly or via word of mouth. It is prudent to take the time to understand a provider’s confidentiality policy prior to utilizing their services. Provider impartiality directly impacts the integrity of rendered services. ISO/IEC 17043 addresses impartiality as follows:
4.1 Impartiality
4.1.1 PT activities shall be undertaken impartially.
4.1.2 The PT provider shall be structured and managed so as to safeguard impartiality.
4.1.3 The PT provider shall be responsible for the impartiality of its PT activities and shall not allow commercial, financial or other pressures to compromise its impartiality.
4.1.4 The PT provider shall monitor its activities and its relationships to identify threats to its impartiality.
This monitoring shall include the relationships of its personnel.
NOTE A relationship can be based on ownership, governance, management, personnel, shared resources, finances, contracts or marketing (including branding). Such relationships do not necessarily present a PT provider with a threat to impartiality
4.1.5 If a threat to impartiality is identified, its effect shall be eliminated or minimized so that the impartiality is not compromised. Without impartiality, the integrity of provider services may be compromised.
Lack of objectivity, e.g. compromised neutrality, due to financial and/ or reputational interests can influence a provider’s judgment and actions. The mere perception of a conflict of interest can damage the credibility of a PT as well as PT participants in terms of a perceived rigged, (i.e. predetermined), outcome.
A laboratory’s failure to successfully pass a PT can be viewed as an embarrassment for laboratory personnel. More importantly, it could lead to negative consequences such as potential product recalls, retesting costs, negatively impacting reputations, etc. and, in some cases, failure is to be avoided at all costs. These scenarios can result in pressure on the provider to manipulate test data to avoid negative consequences for the participant. Providers must never insinuate or promise to compromise PT data integrity to entice new customers or to satisfy and retain existing customers.
Lastly one other impartiality issue has to do with laboratory accreditation assessors advocating the use of a particular PT provider. Per industry accepted guidelines, assessors cannot suggest, recommend, provide incentives, or inflict retaliatory measures regarding the use or non-use of a particular PT provider. Compromised PT provider integrity typically manifests as:
• Doubts about the validity and reliability of the tests, undermining the fundamental purpose of a PT
• Reputational damage if conflicts of interest are determined
Summary
Because there is no guarantee that all PT providers conform to a high level of performance, consumers are advised to do their homework when selecting a PT provider. Take the time to ask tough questions. Take the time to look at the PT provider’s ISO/ IEC 17025 accreditation and their validation reports to ensure they are following ISO/IEC 17043 requirements and adhere to ISO/IEC 13528 standards for the statistical methods implemented in their calculations.
Consumers are encouraged to report any noncompliance of industry accepted practices as well as any impartiality issues such as insinuations or offers to influence a PT outcome via appropriate channels such as the BBB, the provider’s accreditation body, or other avenues. It is up to consumers to demand professionalism of the highest caliber and integrity beyond reproach.
Think of it this way: The assessors are the scouts looking to ensure all of the controls are in place for professional services to be provided. The consumers are the watchdogs that have the power to verify those professional services have been delivered consistently and to report violations so that all consumers are protected.
Endnotes
¹ Z-Score is a statistical measurement of a score’s relationship to the mean in a group of scores. |z| ≤ 2 = satisfactory performance, 2 |z| ≥ 3 = unsatisfactory performance
² En is a statistic that is derived by dividing the difference between test data and assigned data by the square root of the sum of the squares (RSS) of test data uncertainty and assigned data uncertainty. |En| ≤ 1 = satisfactory performance, |En| > 1 = unsatisfactory performance
Christopher L. Grachanen is an American Society for Quality (ASQ) Fellow, Member of the National Association for Proficiency Test (NAPT) Board of Directors and an A2LA Assessor in Training