Online Course Evaluations at
Princeton Universit
y
y
From Paper to Pilot: Online Course Evaluations
and How We Got There
AACRAO 2010
Session ID: 110
1
PresenterPresenter
• Jonathan R. LeBouef
Associate Registrar for Reporting and
Institutional Research
[email protected]
2
AgendaAgenda
• Background and History
• Planning
• System Overview
•
Final Analysis
•
Final
Analysis
3
Princeton UniversityPrinceton University
¾¾ Chartered in 1746Chartered in 1746
¾¾ 7,500 full7,500 full--time studentstime students
¾¾
850+ full850+ full
--
time facultytime faculty
¾¾
850+
full850+
full
time
facultytime
faculty
¾¾ Undergraduate and graduate programs Undergraduate and graduate programs
in 40 disciplinesin 40 disciplines
4
Paper Course EvaluationsPaper Course Evaluations
¾¾ Included Included Most Undergraduate Courses and Most Undergraduate Courses and
Graduate Courses in Natural Sciences and Graduate Courses in Natural Sciences and
EngineeringEngineering
¾¾ Two Forms: Narrative and Quantitative Two Forms: Narrative and Quantitative
¾¾ Process Process Occurred Occurred Twice a Year: Fall and SpringTwice a Year: Fall and Spring
¾¾ 1 Full1 Full--Time Staff Member vs. 1 Time Staff Member vs. 1 ScantronScantron MachineMachine
+=
5
EveryEvery Semester . . . Semester . . .
¾¾ 20,000 Scantron Forms20,000 Scantron Forms
¾¾ 20,000 Narrative Forms20,000 Narrative Forms
¾¾ 20,000 Golf Pencils20,000 Golf Pencils
¾¾ 5,000 Envelopes 5,000 Envelopes
¾¾ Each Scantron Form Touched 4 Times!Each Scantron Form Touched 4 Times!
¾¾ Each Narrative Form Touched 2 Times!Each Narrative Form Touched 2 Times!
6
Problems with PaperProblems with Paper . . . . . .
¾¾ Time Consuming Time Consuming
¾¾ No Instant GratificationNo Instant Gratification
¾¾ 5 Customizations to PeopleSoft5 Customizations to PeopleSoft
¾¾ Low Response Rate ~60%Low Response Rate ~60%
¾¾ Inefficient and WastefulInefficient and Wasteful
¾¾ Difficult to Maintain: Complex WorkflowDifficult to Maintain: Complex Workflow
7
8
9
Export Data
to
CollegeNet
Verify
Forms &
Term Set-
Evaluate
Open Access
to Canned
Reports
Import
Results back
to Princeton
Online Evaluation ProcessOnline Evaluation Process
Assign
Holds
Up
10
Once the Evaluations Left the Office . . .Once the Evaluations Left the Office . . .
11
PlanningPlanning
¾¾ Customize vs. 3Customize vs. 3
rdrd
Party (Architecture Review)Party (Architecture Review)
¾¾ Phased Implementation vs. Big BangPhased Implementation vs. Big Bang
¾¾ TimelineTimeline
¾¾ Map out SpecificationsMap out Specifications
¾¾ Identify Populations for PilotsIdentify Populations for Pilots
¾¾ CollegeNetCollegeNet MagicMagic
¾¾ Test! Test! Test!Test! Test! Test!
¾¾ GoGo--LiveLive
¾¾ PostPost--MortemMortem
12
Online RequirementsOnline Requirements
¾¾ Anonymity must be maintainedAnonymity must be maintained
¾¾ Results held until evaluation period over and grades Results held until evaluation period over and grades
submitted.submitted.
¾¾ Ability to ask courseAbility to ask course--based and instructorbased and instructor--based questionsbased questions
¾¾ Parse course components Parse course components ––Dynamic FormDynamic Form
¾¾
Support More than One Evaluation FormSupport More than One Evaluation Form
¾¾
Support
More
than
One
Evaluation
FormSupport
More
than
One
Evaluation
Form
¾¾ Provide students with a way to “optProvide students with a way to “opt--out”out”
¾¾ User friendly (faculty, students, & admin)User friendly (faculty, students, & admin)
¾¾ Distributed reportingDistributed reporting
¾¾ Send raw data back to PrincetonSend raw data back to Princeton
¾¾ Hook evaluations to the selfHook evaluations to the self--service grade viewservice grade view
13
Authentication
14
15
Term Set-Up
16
Adding Administrative Users
17
Validation
18
E-Mail Announcements
19
Custom Text
20
Form Viewer
21
Real Time Statistics
22
Statistics Continued . . .
23
Reporting Summaries
24
Reporting Summaries
25
Viewing Reports for Individuals
26
Quantitative Report View
27
Report View for Narratives
28
View Individual Responses
29
Fall Semester 08-09
By the Numbers
• Three Forms (FRS, WRI, Main)
• 21,478 Evaluations Across 1,030 Courses
• 18,062 Evaluations Completed (84%)
• 2,822 Evaluations Declined (13%)
• 594 Evaluations Not Completed (3%)
• Overall 97% Response Rate
30
Spring Semester 08-09
By the Numbers
• 3 Forms (FRS, WRI, and Main)
• 20,215 Evaluations Across 1041 Courses
• 15,215 Evaluations Completed (75%)
• 3,875 Evaluations Declined (19%)
• 1,125 Evaluations Not Completed (6%)
• Overall 94% Response Rate
31
Fall Semester 09-10
By the Numbers
• 3 Forms (FRS, WRI, and Main)
• 21,929 Evaluations Across 1034 Courses
• 16,864 Evaluations Completed (77%)
32
• 4,036 Evaluations Declined (19%)
• 1,002 Evaluations Not Completed (5%)
• Overall 96% Response Rate
What Students Think . . .What Students Think . . .
Thank you for using Princeton's new online course evaluation system. What was Thank you for using Princeton's new online course evaluation system. What was
your experience of evaluating your seminars online?:your experience of evaluating your seminars online?:
¾¾ “Mind“Mind--blowing”blowing”
¾¾ “Easier. Saves paper.“Easier. Saves paper.”
¾¾ “I enjoyed evaluating my seminar online because I was able to do it “I enjoyed evaluating my seminar online because I was able to do it
when I had the time to devote to thinking about my answers.”when I had the time to devote to thinking about my answers.”
¾ “The online evaluation system is intuitive.”The online evaluation system is intuitive.”
¾¾ “It is more convenient than filling it out in paper, partially because I “It is more convenient than filling it out in paper, partially because I
can choose my own time in which to do it.can choose my own time in which to do it. “
¾¾ “I like this format a lot more than the in“I like this format a lot more than the in--class format. Even though I class format. Even though I
know the professors don't see what we write when we do it in class, I know the professors don't see what we write when we do it in class, I
feel like I can be more candid with this structure.feel like I can be more candid with this structure. “
¾¾ “It was really easy to use! I much prefer it to the paper evaluations.“It was really easy to use! I much prefer it to the paper evaluations. “
33
Final Analysis
• Keep evaluation forms short
• Easier to administer than paper
• Course information system is more accurate
• More control over who and what get’s evaluated
• Make sure specifications are as detailed as possible
• After initial set-up, annual costs are substantially less than paper
format
• Students can get really annoyed if you send out too many reminder
mails!
-
mails!
• What to do with withdrawn students?
• Pop- Up blockers!
• Be prepared for data analysis
• All in all the our first year was successful and no major problems
reported
34
Questions???
35
PresenterPresenter
• Jonathan R. LeBouef
Associate Registrar for Reporting and
Institutional Research
[email protected]
36