Conducting Targeted In-Depth Verification ReviewsI. PURPOSE
What is the purpose of this directive?
This directive describes FSISí In-Depth Verification (IDV) reviews. It explains what an IDV review is, the procedures for performing targeted IDV reviews, how an IDV review team is formed, the teamís responsibilities, and the components and purpose of an IDV report. This directive also provides a specific set of procedures to be followed when reviews are being performed because an establishment has failed two consecutive Salmonella performance standard sets.II. [RESERVED]
9 CFR Part 416, 417, 500
FSIS Directives 5000.1 and 5400.5
A. What is an FSIS IDV review?
1. An IDV review is an assessment as to whether an establishment is carrying out activities that meet the requirements of the Pathogen Reduction/HACCP (PR/HACCP) final rule, published July 25, 1996. The IDV review is one of several tools that enable FSIS personnel to judge the level of compliance in individual establishments.
2. At present, in-plant inspection program personnel use the Basic Compliance Checklists for HACCP Systems, Sanitation SOPs, and generic E. coli testing to determine whether establishments meet minimum regulatory requirements. In-plant inspection program personnel also perform PBIS HACCP 01 and 02 procedures, as described in FSIS Directive 5400.5, on selected HACCP system features or the entire system to determine whether the system is functioning as planned and is having its intended effect. In-plant inspection program personnel take samples from carcasses or ground products and send them to FSIS laboratories for Salmonella analyses. The results indicate whether the establishment is meeting the pathogen reduction performance standards for these raw products, and whether the HACCP system is accomplishing its pathogen reduction purposes. In-plant inspection program personnel also take samples of ready-to-eat (RTE) products to verify that the establishment's HACCP system is effectively controlling the pathogens of concern in these products.
3. IDV reviews supplement these existing verification tools and address, in a more rigorous and integrated manner, the technical and scientific merit of an establishment's HACCP system. The IDV review is not designed, and will not be used, to assess any aspect of the performance of inspection program personnel.
4. IDV reviews will be conducted by multi-disciplinary teams or by individuals but will not be done by the establishment's in-plant inspection program personnel or their immediate supervisor.
VI. The Components of the IDV review
A. Is the IDV review an audit?
The IDV review is not an audit, although it is designed to incorporate some features of audits, including:
1. The IDV review is conducted over a specific period of time by a team or an individual not involved with inspecting the plant. It results in a carefully and accurately presented account of the non-conformances identified in the establishmentís system. Determinations of non-conformance are based on established regulatory, technical, and scientific standards with which team members are thoroughly familiar;
2. The IDV review will include entrance and exit conferences with the establishment. At or slightly before the entrance conference, the establishment will be provided with a copy of the IDV review checklists so that it can begin to assemble and organize its documents. The exit conference will include an oral report by the team to the establishment of the teamís observations and findings.
3. The IDV report prepared by the team consists principally of team membersí findings based on observations of nonconformance with regulatory, technical, and scientific standards. Finally, the IDV report may include other factual information necessary to understand the establishment Ďs situation.
B. What instruments are used in conducting IDV reviews?
The IDV review Protocol consists of two sets of information gathering instruments: Attachment 1 addresses the Sanitation Standard Operating Procedures (Sanitation SOPs) and HACCP systems; Attachment 2 addresses Pathogen Reduction activities and their outcomes.
1. Attachment 1 consists of 10 separate checklists, each focusing on one of the elements of HACCP or the Sanitation SOPs. Each checklist has a Documents review and a Systems review section. At the beginning of each checklist, there are both regulatory citations and references to the technical/scientific measures of adequacy that are the standards against which establishment performance is judged.
2. Attachment 2 consists of three forms designed to record and analyze microbial data that provide insight into pathogen reduction efforts and accomplishments in the establishment.
3. Attachment 1 checklists may be used as a complete set, or a subset may be used in cases where certain aspects of an establishmentís activities are to be targeted. For example, it may be decided by Field Operations (FO) and the team to focus the review on reassessment efforts and only use the reassessment checklist; or the team may decide to use several checklists but to apply them to only one product or one process category that is of particular concern. The Documents review portions of the checklists may be used without proceeding with the Systems review portions. Attachment 2 forms may be used in conjunction with Attachment 1 or independently. The team should not alter the checklists.
VII. Conducting IDVs
A. How will the Agency decide when to perform IDV reviews?
1. IDV reviews will be either targeted or random.
2. FSIS will perform targeted IDV reviews when problems are identified within an establishment, and these problems persist in spite of repeated efforts by the in-plant inspection program personnel to secure appropriate and effective company responses and actions. There are other circumstances that could suggest the need for a targeted IDV review:
a. before the Agency makes a decision on instituting proceedings for the withdrawal of inspection;b. after the Technical Assessment Group (TAG) has requested specific information from an establishment in order to determine regulatory compliance and the
TAG needs to know whether the information is accurate, and whether the HACCP outcome has been achieved;
c. if the company's products have been repeatedly implicated in recalls or illnesses, or when the Agency testing has found repeated RTE product positive results; or
d. because of a general policy decision, such as the decision to perform IDV reviews in all slaughter or processing establishments that have failed two consecutive Salmonella sample sets.
3. FSIS will also perform IDV reviews on a random basis, after the Agency has established a full complement of trained FSIS personnel. All establishments will be subject to such a review within a 5-year time frame.
(Note: This document focuses on the conduct of targeted reviews.)
B. What steps will the Agency use to initiate a targeted IDV review?
1. On a case-by-case basis, FO personnel will recommend targeted IDV reviews. District Managers (DM) who believe that an establishment in their jurisdiction should be the subject of a targeted review will describe the situation in the establishment and furnish this information to the Assistant Deputy Administrator (ADA) for District Inspection Operations. FSIS will consider the facts surrounding the DM recommendation as well as available resources in determining whether there will be an IDV review, and whether it will be a comprehensive or a limited IDV review.
2. At the beginning of each quarter, the designated FO ADA will confer with other FO leadership and then convene the Deputy Administrators and Associates who will establish priorities and a rough schedule for the IDV reviews that FO has decided are needed. Such prioritizing and scheduling will permit the programs whose participation is expected in the IDV reviews to identify and plan for their participation in an orderly manner. The Office of Public Health and Science (OPHS) and the Office of Policy, Program Development and Evaluation (OPPDE) will supply technical experts to participate in targeted IDV reviews.
VIII. The IDV team
A. What is the expected team composition for a targeted IDV review, and what skills and competencies should be included?
1. Teams will usually include FSIS personnel with the following expertise:
a. a person with experience and expertise in synthesizing scientific and technical judgments about HACCP with regulatory enforcement protocols as they are applied in inspected establishments (frequently, this person is the Inspection Coordinator (IC) in the DO);
b. a person with experience and expertise in documenting in-plant findings and explaining them and their significance in plain language (this person is expected to be from FO);
c. persons with scientific or technical credentials and experience relevant to the processes performed and under review in the subject establishment (these persons should be from OPHS or OPPDE);
d. a person with expertise in interpreting FSIS regulations, directives, notices, and guidance material, and recognizing issues that present novel policy questions requiring further development.
2. FSIS expects that IDV review team members will receive training that includes appropriate auditing and analytic techniques as well as report writing skills.
(Note: Sometimes the team may need to include several people to ensure that there is relevant scientific, technical, and policy experience. In other cases, one person may be able to fill multiple roles. In forming teams, FSIS will take care to ensure representation from all relevant Agency program areas. FSIS will also avoid an overly large team that may be difficult to manage and disruptive to establishment activities. FSIS will not use targeted IDV reviews for training purposes.)
B. How will the team operate before, during, and after the on-site visit?
The Deputy Director of the TSC will determine team membership by using lists of potential participants supplied by OPPDE, OPHS, and FO. In forming a team, FO may consult with any program area about the availability or special expertise of specific individuals. FO will designate a team leader (TL). In most cases the TL will be the DO representative. The Deputy Director of the TSC will notify the TL, and he or she will assume leadership of the effort at that time.
C. What are the TLís responsibilities?
1. If the TL is the DOís representative, he or she will assist in start-up of the review. The TL will facilitate accommodation arrangements, identify and secure an appropriate workspace for the team, notify the subject establishment of the IDV review, and provide the establishment with advance copies of checklists and procedures. (Note: If the DOís representative is not the TL, someone from the DO will need to perform these tasks. When making these arrangements, local conditions, including the establishmentís size and resources will need to be considered.)
2. The TL will lead and manage the team from the time of its identification as a team through the development and delivery of the IDV report.
3. The TL will engage the cooperation and collaboration of local in-plant inspection program personnel, without drawing them into the IDV review, by requesting their assistance in an initial walk-through of the establishment to familiarize the team with the operation and layout. (Note: The TL will promptly refer to local personnel any team observations that might involve immediate food safety problems. In-plant inspection program personnel are expected to take any necessary actions)
4. The TL will correlate with all members of the team about their understanding of the technical, scientific, and regulatory standards against which establishment activities are to be measured.
5. If the IDV does not involve all operations in the establishment, the TL will determine and define the scope of the review. During the course of the review, the TL may change the scope of the review. For example, if the team determines that a fundamental requirement, such as conducting the hazard analysis and identifying critical control points (CCPs) for all "food safety hazards reasonably likely to occur," has not been met, the TL should terminate the IDV review and inform the establishment, as well as the DM, of this determination and its basis.
6. The TL will make assignments for performing various aspects of the IDV review, taking into account the expertise of individual team members and the sensitivity of the review. Wherever possible, more than one person should be involved in the review and in the analysis of documents or activities.
7. The TL will hold team meetings as necessary during the course of the IDV review to ensure that work is proceeding on time and on target.
8. The TL will serve as, or designate a single person to serve as, the teamís contact with the establishment regarding follow-up questions, requests for further information, etc., during the course of the review. All individual team member requests should go through the TL.
9. The TL will schedule and lead the exit meeting with the establishment.
10. The TL will ensure that the draft team report is finished within one week of completing the on-site review, and that the report is submitted to team members and the DM, plus the DAs of OPPDE, FO, and OPHS. The TL is not expected to write the report alone. All team members should contribute, and members with skills in documentation should be major contributors.
D. What are the team membersí responsibilities?
1. The team will be fully available for this priority effort by ensuring that their routine/on-going assignments have been delegated to colleagues for the relevant time period.
2. The team will study, in advance, the checklists and the reference materials that define the benchmarks. This will prepare team members to perform the review efficiently and effectively.
3. The team will bring with them technological tools, such as portable computers, that will enable them to record their observations and perform data analyses.
4. The team will bring or have ready access to any information pertaining to the subject establishment that is important to their areas of expertise.
5. The team will be skilled in objective reporting of facts without making judgments. Team members are expected to provide factual descriptions of what they read in documents or of what they observe in operations. Team members should not formulate judgments in circumstances where only facts are necessary.
6. The team will assist in writing the report within specified time frames.
A. What are the key characteristics of a good IDV report?
1. Good IDV reports are clear and understandable to a general audience, including attorneys, consumer groups, and the interested public. Many people who will review and use the report will not have extensive technical or scientific experience and education. This means that the report must both communicate the technical and scientific findings and provide insightful analysis that supports their significance. The report should be written in a way that will be useful both to the Agency in deciding what follow-up action with respect to the establishment is necessary and appropriate and to an establishment that wants to use the report as a basis for improving its design and execution of HACCP systems and Sanitation SOPs.
2. Good IDV reports are concise and present findings supported with factual observations. Such reports do not speculate about what might have happened or what might have been the thinking of establishment management or personnel.
3. Good IDV reports present scientific and technical analyses that explain why observations rise to the level of findings. These scientific and technical discussions are framed within the principles of HACCP, as articulated in FSIS regulatory requirements.
4. Good IDV reports contain regulatory citations from the checklists but do not rely on extensive quotations from the regulations or Agency Directives and Notices. This scientific and technical emphasis is appropriate because the Agency has other verification tools to address regulatory compliance. Personal interpretations of regulatory requirements are not appropriate.
B. How will the Agency review and use IDV reports?
1. After the IDV review team has formulated its report, the report will be submitted to the DM. The DM will direct the IC to analyze the report and make recommendations about how to proceed with the establishment based on the reportís findings. In performing this analysis, the IC should draw on the following resources as necessary: the IDV review team itself; scientific and technical experts; regulatory and policy experts; and colleagues who might have had experience in performing such analyses.
2. There are various possible outcomes of IC analysis:
a. when the targeted IDV was performed to determine whether selected information indicates that the establishment meets regulatory requirements, IC analysis may confirm that the establishment does meet the requirements or does not meet the requirements;
b. when a targeted IDV was performed because an establishment has multiple, continuing findings of pathogens in RTE products, and such products are implicated in illnesses, IC analysis may develop relationships between in-plant occurrences and RTE product failures;
c. when the targeted IDV was performed for almost any purpose, IC analysis may define issues other than those on which the IDV was focused or, more rarely, may even uncover instances of basic non-compliance;
d. because the potential outcomes of IC analysis are so many and varied, the potential significance of the findings of the IDV will also vary. The Agency may decide, based on the findings, that any of a variety of approaches are in order, from no further action to beginning an action to withdraw inspection. However, establishments generally should be given the opportunity to let their HACCP systems work before FSIS acts.
e. FSIS generally will give the establishment an opportunity to reassess its HACCP plan within a specific timeframe in which this reassessment and any consequent actions are to be accomplished. If the reassessment and consequent actions result in a HACCP system that appears to be adequate, the changes made by the establishment may be described in an addendum to the IDV report that is included as part of the report that is made final, publicly available, and closed.
f. The final IDV reports will be distributed to the DO, TSC, ADADO, OPHS, and OPPDE.
A. What specific procedures are used in targeted IDVs performed because an establishment has failed two consecutive Salmonella performance standard verification sets?
1. The specific procedures to be used in these cases begin with clear notification to the establishment of the second set failure and its consequences, that is:
a. the occurrence of a second consecutive Salmonella set failures is rapidly communicated to FO and the establishment by OPHS. The DO receives the Pathogen Reduction Enforcement Program (PREP) Completed Set report;
b. the DO communicates the information from the completed set report to the IIC and prepares a NR that includes a citation to the regulatory requirement (9 CFR 310.25(b)(3)(ii); 381.94(b)(3)(ii)) that an establishment reassess its HACCP plan; and
c. the DO sends a form letter that reiterates this regulatory requirement and gives the establishment information about the Agencyís next action and its likely timing.
2. After the establishment has sufficient time to perform the required reassessment, the next Agency activity is the initiation of a targeted IDV review.
3. As in other cases, the DM will receive the team report on the IDV, which contains the teamís findings. The DM will have the IC analyze the findings and make recommendations about regulatory outcomes. In targeted IDVs following a second set failure, it is often difficult to determine whether the reassessment that the establishment performed in response to the regulatory requirement has resulted in a HACCP system that is likely to pass the third set. If the IC has any doubt about this outcome, the DM should request that the establishment perform a new reassessment based on the IDV in the 30 days following receipt of the IDV. Most targeted IDVs performed because of two consecutive Salmonella set failures have resulted in this type of request. The IC analysis may result in a high level of confidence that the establishmentís HACCP system is adequate and likely to be successful in controlling production, so that the next sample set will succeed. In this case, the IDV is transmitted to the establishment, and the third sample set is initiated.
4. The staffing and scope of targeted IDVs that are performed following two Salmonella set failures may also be special. The team performing these targeted IDVs may be as few as two people. However, one of them needs to have an understanding of, and experience in, food microbiology sufficient to provide insight about the impact of establishment activities on the prevalence of Salmonella. The scope of this IDV may be limited. It may focus on the reassessment checklist only, or it may include several other pertinent checklists but apply them only to some of the establishment's HACCP systems. In this way, available Agency resources can be used to enhance knowledge about which CCPs are successful in controlling this pathogen in various operations.
Philip S. Derfler
II. Hazard Analysis
Regulatory Standard: 9 CFR
III. CCP Determination
Regulatory Standard: 9 CFR
IV. Critical Limits
9 CFR 417.2 (b) (3)
V. Monitoring Procedures
Regulatory Standard: 9 CFR
417.2 (b) (4)
B. Systems Review
IV. Corrective Actions
9 CFR 417.2 (b) (5)
VII. Validation Procedures
Standard: 9 CFR 417.4
VIII. Verification Procedures
Regulatory Standard: 9 CFR 417.4(a)
A. Documents Review
IX. Reassessment Procedures
Standard: 9 CFR 417.4(a)(3)
*Note NACMCF does not use the term reassessment, but includes this activity as part of verification, the occasions for reassessment are identified under B.
A. Document Review
X. Record Keeping Systems
Regulatory Standard: 9 CFR
Reviewers may find that all these questions have been answered under
Send mail to
webmaster with questions
or comments about this web site.