Evaluation Report - eLearning Pilot Final Report

United States Department of Agriculture
Food Safety and Inspection Service
Washington, DC 20250-3700

Evaluation Report
eLearning Pilot
Final Report

June 2002

Introduction

This is an evaluation report on a pilot of eLearning (distance learning conducted over the Internet) in FSIS from September to June 2002. The evaluation sought to measure the success of the pilot with three criteria:

	participants take advantage of needed training;
	participants report a positive experience regarding the ease of use of courses taken via the Internet;
	participants report they benefit from learning.

The methods for evaluation included review of automated participation reports, telephone interviews with a sample participants and supervisors, an online evaluation of courses taken, a follow-up Email survey to participants, and a review of online and printed literature. An interim report summarized the experience during the first five months of the pilot.

Key Findings

Overall, the pilot met all three criteria for success.

	66% of the participants took needed courses.
	Most reported a positive experience with the pilot meeting their expectations, reporting they would use eLearning again and they would recommend it to others.

Participants and their supervisors reported benefits from the pilot by using skills learned on the job including writing, computer skills, better communication and management skills. Supervisors also reported increased satisfaction among participants.

Additional advantages from the pilot participants and the literature included:

	Ability to schedule classes conveniently,
	Consistent training for all,
	Convenient locations and less travel time,
	More time to study, resulting in better, more thorough responses.

Despite these advantages, one-third of the participants were unable to take the courses anticipated. Most of those unable to take a course reported that busy schedules and lack of time prevented their taking courses. This was a greater problem for those in Headquarters where there were both more participants who were unable to complete courses and those who reported more problems with time. Of the thirty unable to complete a course, fourteen did not wish to continue with the pilot.

Twenty-nine participants also reported computer-related problems such as getting working access codes, connecting to courses, marking course progress and computer lock-ups in the middle of a course. This was a greater problem for those in the field.

Key Recommendations

The following recommendations address problems with time management, technology, and the need to maintain sufficient student contact.

Expand pilot to a group of Agency employees who are not volunteers to assess success of those who may not be as experienced computer users.

Use monthly report to monitor progress with eLearning and to identify participants who are experiencing problems. Follow-up with these participants to problem solve issues of computer access or time that they may be facing. If these problems cannot be solved, provide an opportunity for others to replace them in the pilot.

To meet issues of time,

Communicate with supervisors, if time is an issue, to provide uninterrupted time periods within the workweek for course completion.

Grant credit for off-duty time spent on eLearning, and cover the cost of off-site phone connections.

Provide private workspace and encourage options away from the desk such as computer training center.

To meet issues of computer access,

Ensure that all participants have adequate computers and software.

Provide comprehensive instructions and orientation for new participants.

	Ensure good quality and timely help is available throughout the pilot.
	Publicize where to get help – providing names and numbers.

Problem solve with vendors possible reasons for lesson interruptions due to technology.

To ensure participant contact,

Maintain communication with participants and their supervisors through continuation of the eLearning newsletter.

Provide more complete information on courses available to assist in course planning.

Provide additional social dimensions to eLearning through meetings or discussion groups, either onsite or using chat rooms or other computer resources.

Participants and Their Learnings

The original pilot group included 100 employees who volunteered to participate. Most were from Field Offices (49) or Headquarters (35) with 13 from TSC and 3 from the College Station.

The main reasons for participation in the eLearning pilot were the following:

interest in trying eLearning,

need to take the course, and

limited resources for offsite learning.

The main reasons supervisors encouraged participation were:

	for staff to get needed training,
	to encourage professional development and enhanced knowledge,
	to use a reasonably inexpensive option for training.

As of May 2002, 66¹ of the participants were able to take at least one course. The average number of classes taken by each respondent was 4. (Range 1 – 19) They spent an average of 2 hours per course for a total of 8 hours.

The classes selected covered a wide range of topics the most popular of which were:

	communication,
	writing and grammar,
	computer software applications, and
	business management.

Most participants took the eLearning courses in the office. Some also reported taking courses at home and very few used the Microcomputer lab or other sites such as the local library.

Most (83%) of those who did not complete a course reported problems with time due to:

	staff shortages,
	additional office commitments,
	the need for uninterrupted time.

Six of those who did not complete a course also reported problems with computers and lack of interest in courses and eLearning format.

Most supervisors surveyed reported that the pilot experience met their expectations, was positive for their employees, and led to increased satisfaction.

Examples reported by supervisors surveyed of applications to the job from the courses included observing improved management, mentoring, and budget skills on the job, application of improved communication and writing skills, and increased technical and computer skills.

The supervisors surveyed support the expansion of the program and provided additional suggestions.

	Make the program available to office staff and student interns who may not get training opportunities.
	Emphasize increasing computer skills through coursework.
	Include formal credit for courses taken, perhaps toward a degree or other program, and provide certificates of completion.

¹An automated report provided information on courses taken. Originally this report did not include complete information. Comparing the results with the supplemental surveys and problem solving with vendors led to report adjustments that provide a reasonably accurate profile. Continued monitoring of this report is necessary because it will provide ongoing evaluation information on program participation.