I. PURPOSE

This directive integrates AssuranceNet enhancements implemented since March of 2008 and provides updated specific instructions to OFO managers at Agency headquarters (HQ) and at the district offices (DOs) on reviewing, monitoring, analyzing, and responding to AssuranceNet results. (NOTE: The instructions in this directive do not apply to non-supervisory in-plant inspection program personnel.) In addition, this directive provides:

A. Clarification that AssuranceNet reports and analyses are appropriate for viewing organizational performance at the district and circuit levels.

B. OFO manager requirements for documenting organizational performance findings and any required followup.

II. (RESERVED)

III. (RESERVED)

IV. REFERENCES

AssuranceNet User Guide

PBIS User Guide

V. ABBREVIATIONS 

The following appear in their shortened form in this directive:

• AA Assistant Administrator
• DA District Analyst
• DAA Deputy Assistant Administrator
• DAIG Data Analysis and Integration Group
• DM District Manager
• DO District Office
• EARO Executive Associate for Regulatory Operations
• FLS Frontline Supervisor
• HQ Headquarters
• IPPS In-Plant Performance System
• OFO Office of Field Operations
• PBIS Performance Based Inspection System
• STAR Supervisory Tool for Assessment Results

VI. POLICY

It is FSIS policy that each level of OFO management conducts systematic analyses in their areas of responsibility using AssuranceNet data and results to identify where organizational performance is not meeting established performance measures, or where performance indicates a vulnerability to failure in achieving Agency objectives and mandates.

VII. DEFINITIONS

A. Circuit Level. A defined geographic area for which FLSs oversee inspection activities performed by public health veterinarians, supervisory consumer safety inspectors, and food inspectors.

B. District Level. A defined geographic area covering one or more states that oversee inspection activities performed at the circuit level.

C. Establishment Level. A specific establishment in a circuit of a district.

D. National Level. The level at which inspection activities are carried out within the district and managed by the OFO AA, DAA, and EARO.

VIII. BACKGROUND

A. AssuranceNet is a Web-based system that OFO uses to monitor organizational performance at the circuit, district, and national levels for mission critical and key administrative functions and activities.

B. The purpose of reviewing AssuranceNet data and assessing actual organizational performance levels against the targets on the performance measures is to make sure OFO managers are actively focusing on the key food safety, food defense, and management activities represented by the performance measures.

C. The focus of AssuranceNet data should not be on the numbers themselves, but on the activities they represent. To the extent that OFO managers and supervisors have control over the source data used in the AssuranceNet calculations on these performance measures, they are expected to ensure the integrity of the data being used by the system. The performance measures are a means to an end, not the end itself. OFO managers are expected to focus on the meaning of the data and to react to it appropriately rather than focusing on controlling the data so that targets are met consistently.

IX. ESTABLISHMENT LEVEL DATA

A. AssuranceNet data are broken down to the establishment level for certain activities. However, that is not to imply that the performance measure targets apply at the establishment level. (NOTE: Individual establishments are monitored through supervisory oversight using other systems and reporting tools.)

B. Establishment level data has value and meaning in assessing circuit performance when a circuit is not meeting a performance measure or is barely meeting a performance target. The OFO manager reviews establishment level data to determine where there are outliers that would warrant further investigation. (NOTE: FLSs are not required to address why establishment level performance does not meet the circuit, district, or national performance measures as there is no expectation that performance at the establishment level meets the performance measure targets.)

X. REVIEWING AND MONITORING PERFORMANCE DATA AND ACTION LEVELS

A. Action levels serve as a guide for determining whether performance at the circuit, district, or national level warrants investigation or followup action. Action levels are listed on the AssuranceNet Action Level Matrix (See Attachment 1). The following applies based on the availability of objective, measureable data:

1. Objective Data. These measures take into account the normal variations that can be expected in performance. For these objectively determined performance measures, the action level is the same as the target level.
2. Subjective Data. For these measures, OFO must rely on more subjective assessments made by supervisors using the IPPS. Most target levels for these measures are set at 100 percent. While the supervisor is expected to follow up on individual performance in every case and ensure that there is appropriate management of individual employee performance, the district is expected to investigate further when a circuit or the district falls below 90 percent (action level) on these IPPS-driven measures. In addition, the district must monitor the performance of each circuit on these IPPS-driven measures over time (example: compare performance each quarter with previous quarters) to note any trends that may be developing, such as performance for the circuit trending downwards or showing little or no improvement.

B. When a circuit’s or district’s performance is below the action level, the DM or designee will look into the reasons the measure was not met and will document their findings and follow up. (See subparagraph XII.) (NOTE: Not every instance in which a performance measure falls below the action level should be construed as a failure to meet mission objectives. The point of establishing an action level and requiring further inquiry and followup is to ensure that attention is paid to critical areas of organizational performance and vulnerabilities and potential failures are addressed before they become actual failures.)

C. To determine the frequency of reviewing and monitoring performance data, OFO managers use the Action Level Matrix which lists the Analysis Window and the Frequency of Review. The Analysis Window indicates the amount of data that needs to be reviewed when performing the analysis. The Frequency of Review indicates how often the data will be reviewed. (EXAMPLE: For performance measure 1.2.1, the analysis window is 6 months and the frequency of review is quarterly. So, every quarter, 6 months of data should be reviewed and analyzed.)

XI. DATA REVIEW AND ANALYSIS

A. The district and FLSs have access to reports in AssuranceNet and they are encouraged to utilize them frequently. However, for the purpose of conducting monthly, quarterly or annual review and analysis of the AssuranceNet data, reports will be provided by either OFO HQ or the DAIG in the Office of Data Integration and Food Protection to the DOs and HQ management staff at the required frequencies as outlined in subparagraph XI. D.

B. Depending on the performance measure involved and the activity the measure represents, the data reviewed in a given month or quarter may actually reflect a larger window of time than 1 month or 1 quarter. (EXAMPLE: Recall management measures have to be viewed for the entire year as recalls do not occur in every district every month, if at all, during a given year.) For performance measures that draw data from IPPS assessments, the organizational performance data makes little sense if viewed for less than a 6-month moving window because IPPS visits are not performed at the same rate and do not cover the full complement of elements and measures evenly within a given month or even within a given quarter.

C. The reporting tools available for reviewing AssuranceNet results are as follows:

1. Circuit and District Level Performance Measure Reports. These reports are available to FLSs and district and HQ users.
2. Custom Reports and Dashboard Manager. These reports are available to DMs, deputy DMs, DAs, and HQ users.

D. There are other tools and options available for viewing data at a greater level of detail for which action may be warranted or to further explore results and determine vulnerabilities. The AssuranceNet Action Level Matrix provides a detailed description of the functions, measures, and action levels. The performance measures table (see Attachment 2) provides brief information about the measures, and whether data sources are IPPS-driven or non-IPPS driven. The following outlines the expectation at each level and the reports that should be utilized, at a minimum:

1. Headquarters Level.

a. The OFO AA or DAA conducts a review of district performance on all measures using a specially designed report reflecting the past 4 quarters of data on each measure, for each district. This review will be done at least twice a year. This report is also available to the EAROs to review at least twice per year to look for performance trends in their assigned districts.

b. At a minimum, the EAROs conduct a quarterly review of the performance measure data for the previous quarter for their assigned districts. They also review the following set of reports covering the applicable quarter for each assigned district. (NOTE: Reports can be found in the AssuranceNet Custom Reports Public Folders under AssuranceNet/Master Custom Reports.)

• (1) AssuranceNet custom reports showing district performance for all measures under Functions 1, 2, 6, 7, and 17 for the past quarter.
• (2) AssuranceNet performance measure reports showing district performance under Functions 3 and 5 for the past quarter.
• (3) AssuranceNet performance measure report showing district performance for Function 4 for the past year.
• (4) AssuranceNet custom report showing district performance on Function 8 measures, year to date beginning with the first day of the current performance rating cycle.
• (5) DA HACCP and Data Analysis DA pages for applicable districts, to review the findings and comments entered for the 3 months falling within the quarter under review.
• (6) Administrative (RMA) measures, Functions 9-14, using custom reports every quarter.

2. District Level. The DM is responsible for ensuring that district and circuit performance data is assessed, acted on, and appropriately documented. At a minimum, the DM ensures that a review of district and circuit performance with select performance measures is conducted monthly, with all other performance measures assessed at least quarterly. (NOTE: The custom reports listed below are available in the AssuranceNet Custom Reports Public Folders under AssuranceNet/Master Custom Reports in the subfolder with the district's name (example: AssuranceNet/Master Custom Report/Atlanta).)

• a. AssuranceNet custom report showing circuit and district performance for all measures under Functions 1, 2, 6, 7, and 17 for the past quarter. EXCEPTION: Function 1 and 2 performance measures will be grouped for review by either monthly or quarterly reports, depending on the specific measure.
• b. AssuranceNet performance measure report showing district performance under Functions 3 and 5 for the past quarter.
• c. AssuranceNet performance measure report showing district performance for Function 4 for the past year.
• d. Function 8 showing circuit and district performance on all Function 8 measures, year to date beginning with the first day of the current performance rating cycle to locate where circuits and the district may not be making expected progress at the current point in the rating cycle on these annual measures.
• e. Custom reports for administrative Functions 9 -14.

3. Circuit Level.

a. The FLS reviews AssuranceNet data for the circuit on a monthly basis covering the previous month, for the following performance measures:

• (1) 1.5.2
• (2) 1.5.3
• (3) 2.1.1
• (4) 2.2.1
• (5) 2.3.1
• (6) 2.4.1
• (7) 2.4.2
• (8) 17.2.1

b. The circuit level review should be conducted late in the month to ensure all data for the previous month has been captured in the system. The FLS will be provided with a custom or other report reflecting the actual performance against targets for the above measures. The FLS may also use other reports to further analyze the data on the custom reports and to determine if there is any outlier establishment level performance that would be of concern and that might have led to the performance level of the circuit being below the target or action level. Additional reports include:

• (1) Performance measure reports in AssuranceNet.
• PBIS reports entitled "AssuranceNet Summary" and "AssuranceNet Details.” (NOTE: The PBIS User's Guide provides instructions for generating these reports.)

c. For the remaining measures that apply to circuits, the FLS reviews the custom or other reports provided by the district on a quarterly basis, to determine if there are any performance measures falling below the action levels and to identify the followup action indicated as a result, if any. The remaining circuit measures include:

• (1) All Function 1 measures except 1.3.3.
• (2) All Function 2 measures except 2.3.3, 2.3.5, 2.3.8, and 2.4.3.
• (3) All Function 6, 7, and 17 measures.

d. The FLS should view establishment level data for Functions 1, 2, and 17 where available, to determine whether there are results that are out of the range of the performance levels at most other establishments in the circuit.

e. The FLS reviews and observes subordinate supervisors' IPPS and STAR assessments as required and makes comments in AssuranceNet on the IPPS report and STAR form.

f. The supervisory public health veterinarian reviews and observes IPPS assessments as required and makes comments in AssuranceNet on the IPPS reports.

g. The FLS uses the performance measure report for Function 8, year to date beginning with the first day of the current performance rating cycle to determine their progress on performance measures for conducting, reviewing, and observing IPPS assessments. The FLS also uses the IPPS standard report to determine progress.

XII. REPORTING AND FOLLOWING UP

A. Each level of management is required to provide documentation to establish that applicable data have been reviewed and analyzed, and that followup has been initiated as appropriate. OFO management also determines whether there has been a failure to meet or exceed the action levels listed on the AssuranceNet Action Level Matrix. (See Attachment 1.)

B. Each level of management uses the Action Level Matrix as a guide for determining whether performance is trending downward or failing, thus indicating that attention needs to be given to the functions and activities that the performance measures address. The following describes specific reporting repsponsibilities by review level:

1. Headquarters Level.

• a. The AA or DAA will discuss with the EAROs any concerns that emerge from review of district performance as reflected in the AssuranceNet data. The EARO documents any followup as outlined in subparagraph b.
• b. At a minimum, the EAROs should document the results of their quarterly review on the EARO/AA Oversight form in AssuranceNet, summarizing their specific conclusions, function by function, and documenting discussion of the findings and follow up with the respective districts.

2. District Level.

• a. The DM is responsible for ensuring that the data assessment is performed, accurate, and appropriately documented. The DM ensures that AssuranceNet reports and analyses provided by DAIG DAs and HQ or district personnel are reviewed and correlated with other relevant reports and analyses. The DM will brief the EARO for their district when they have concerns about the district’s failure to meet or exceed the action levels. The district is expected to keep a record of its review and analysis and to document the followup actions taken.
• b. At a minimum, the districts are to complete the data analysis DA form in AssuranceNet each month to reflect the outcome of this monthly analysis and followup action in detail. The AssuranceNet User Guide has specific instructions on completing this form. (NOTE: There will be additional information entered for the third month of each quarter to reflect the quarterly AssuranceNet data reviews. (EXAMPLE: For the first quarter of the calendar year, January and February reporting would reflect district review and analysis of the monthly reports and data, whereas the March reporting would reflect not only the monthly analysis but also analysis for the previous quarter.))

3. Circuit Level. The FLS documents any followup that they conduct in response to AssuranceNet results for a given month or quarter, as appropriate, either in response to their own review of the data or in response to the district's review of the data. Specifically, if the circuit falls below an action level for the relevant period, the FLS should be prepared to provide an explanation to the DO. The FLS is not expected to provide an establishment-by-establishment explanation as to why establishment level performance fell below the performance targets set for the circuit or district. FLSs provide explanations only when the circuit’s performance falls below action levels or is trending downward. FLSs provide these explanations to the DO in accordance with the district’s established protocol for this purpose.

XIII. ADDITIONAL INFORMATION

Refer questions concerning this directive to OFO’s Regulatory Operations staff. Refer all AssuranceNet application questions to AssuranceNet.HelpOFO@fsis.usda.gov.

See full PDF for attachments

• Sample AssuranceNet Action Level Matrix.
• Performance Measures Table Identifying Non-IPPS Driven and IPPS-Driven Data Sources.