Strategic Review of Online Education

Report of the Faculty Council on Teaching and Learning

Princeton University

September 2015

Table of Contents

1. Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

2. Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

3. Overview and Guiding Principles for Strategic Planning . . . . . . . . . . . . . . . . . . . . . . 4

3.1. Princeton’s Current Activities in Online Education . . . . . . . . . . . . . . . . . . 4

3.2. Evaluating Princeton’s Current Initiatives . . . . . . . . . . . . . . . . . . . . . . . . . . 5

3.3. Experiences at Other Institutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3.4. Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

4. Priorities for Future Experimentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4.1. Supplemental Online Materials for STEM Preparation . . . . . . . . . . . . . . . 9

4.2. Small Private Online Courses (SPOCs) . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

4.3. Recommendation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

5. Infrastructure and Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

5.1. Pedagogical and Technical Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

5.2. Faculty Incentives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

5.3. Strategic Partnerships . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

5.4. Research and Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

5.5. Recommendations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

6. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Appendices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

A. Council on Teaching and Learning Members AY 2014 – 15 . . . . . . . . . . . . . 20

B. Summary of Interviews with Peer Institutions . . . . . . . . . . . . . . . . . . . . . . . . . 21

C. Survey Questions and Responses from On-Campus Studies . . . . . . . . . . . . . . 27

D. Princeton MOOCs Since 2012 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

E. MOOC Production by Peers Since 2012 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

F. Flipped and Blended Courses at Princeton Since 2012 . . . . . . . . . . . . . . . . . . 32

G. Princeton Online Course Projects by Division . . . . . . . . . . . . . . . . . . . . . . . . . 33

1. Executive Summary

This decade has the potential to be a watershed moment in the history of higher education as

hundreds of colleges and universities explore the potential of online technology to enhance

teaching and learning. Many welcome this phenomenon as the beginning of a revolution that

will not only expand the availability of educational resources to students throughout the world,

but also stimulate improvements in pedagogy. Others are less sanguine about these

developments and warn that online instruction will, among other things, lead to a substantial

decline in the personalized, face-to-face instruction that is an essential and defining element of

liberal arts education. At this time, it is simply too early to know how and to what extent the

dissemination of online technology will transform higher education, but the impact could very

well be major.

After extensive consultation with faculty members, Princeton decided to join Stanford

University, the University of Michigan and the University of Pennsylvania in the April 2012

launch of Coursera, now the largest platform for the delivery of Massive Open Online Courses

(or MOOCs). Princeton participated in the Coursera launch with the primary goal of

understanding how MOOCs might be used to enhance the quality of education on campus while

also sharing educational materials with the world (http://bit.ly/1IHD901). Through the

development and delivery of free, non-credit course materials that have reached millions of

students across the world, Princeton has gained a great deal of insight into the nature, costs, and

relative benefits of online education. Additional insights have been gained through creative

experiments by Princeton faculty in the use of online technology to enhance the quality of

undergraduate education here on campus.

In September 2014, a little more than two years after Princeton offered its first MOOC on

Coursera, Provost David Lee charged the Faculty Council on Teaching and Learning to survey

the “rapidly changing landscape” of online instruction and prepare a report that recommends

strategic priorities for Princeton’s ongoing approach to this area. This report, the Council’s

response to the Provost’s charge, follows a year-long process of investigation that included

reading in the literature on online education, consultation with authorities in this area, interviews

with representatives of peer institutions and Princeton faculty members who have experimented

with online education, and anonymous surveys of Princeton students and faculty members.

The Council’s recommendations fall into three groups. First, we recommend that Princeton’s

strategic planning in this area be guided by the following principles:

· Princeton’s strategy with respect to online education should be informed by the pervasive

uncertainty about its effectiveness.

· Princeton should give priority to the development of online materials and tools that will

build upon our strengths as a residential learning community and that will enhance and

complement -- not supplant -- time-tested modes of teaching and faculty-student

interaction.

· Princeton should encourage the public dissemination of online course materials only to

the extent that those efforts are likely to yield clear benefits to Princeton students.

· Broad consultation with the faculty regarding Princeton’s strategy for online education is

critical, especially in light of faculty members’ strong and widely diverging views on this

subject.

Second, we recommend that Princeton give highest priority to the development of supplemental

online materials as well as online courses for summer study that will help students make their

way into the STEM disciplines and other fields where entry has been particularly challenging.

To the extent possible, the fruits of our efforts in this area should be disseminated for public

benefit.

Finally, we recommend that the infrastructure and resources available to support Princeton’s

ongoing experimentation with online education be augmented. In particular, Princeton should

offer special incentives (e.g., summer salary and teaching relief) for the development of online

materials of high priority. We should also increase our efforts to communicate with faculty

members regarding online education and the resources available to support innovation in this

area. Princeton should also explore opportunities for collaboration with other institutions and

offer additional resources for faculty and departments that want to undertake research on the

effectiveness of various types of online learning.

2. Process

In September 2014, Provost David Lee charged the Faculty Council on Teaching and Learning to

study the “rapidly changing landscape” of Massive Open Online Courses (MOOCs) and other

forms of online instruction, and to prepare a report that recommends strategic priorities and

specific measures for guiding Princeton’s ongoing approach to online instruction. According to

the Provost’s charge (http://bit.ly/1TpK0jL), “[t]he overarching questions in this area include

how Princeton should take advantage of new developments in online learning and technology in

the classroom to enhance the quality of education on our campus, whether and how Princeton

should uses MOOCs or other technology to expand the reach of its teaching, and whether and

how Princeton can contribute to research about the efficacy and value of online teaching.”

The Council met monthly during the 2014-15 academic year. (The members of the Council are

listed in Appendix A.) To begin, the Council received general orientation in the field of online

education. In addition, without attempting a comprehensive review of the burgeoning literature

in this area, members of the Council read a number of key documents and held conversations

with several authorities in the field.

These authorities included William Bowen and Kevin

Guthrie of ITHAKA, authors of several leading studies of online education; Anant Agarwal, the

founding CEO of edX; and Kimberly Cassidy, President of Bryn Mawr College. The Council

next divided itself into two groups:

· Group A focused on online education at a sampling of seven peer institutions that

responded positively to requests for in-depth information: Columbia University, Duke

University, University of Michigan, Massachusetts Institute of Technology, Yale

University, Stanford University and Brown University. They interviewed administrative

and faculty leaders at each of these institutions to gauge the nature and magnitude of their

activities in online education (and the policies and infrastructure in place to support them)

and to understand their challenges and successes as well as their priorities for the future.

(A sample of the questions and a summary of the interviews are attached as Appendix B.)

· Group B focused on online education at Princeton and administered a series of

anonymous surveys designed to ascertain the views and experiences of three campus

constituencies: (1) students who have taken courses involving substantial online

components; (2) faculty members who have taught such courses; and (3) all other faculty

members. The survey instruments and responses are attached as Appendix C. Group B

also conducted an analysis of existing student evaluations from courses that have

involved substantial online components. (The Council reviewed these evaluations only

after securing the permission of the professors whose courses were involved.)

With all this information in hand, the Council deliberated about the principles and priorities that

should guide Princeton’s initiatives in online education in the future.

For an overview of the main findings in the literature, see Siemens, George, Dragan Gasevic and Shane

Dawson, Preparing for the Digital University: A Review of the History and Current State of Distance,

Blended, and Online Learning, (Gates Foundation, February 2015); and Allen, I. Elaine, Jeff Seaman,

Grade Change: Tracking Online Education in the United States (Formerly Known as the Sloan Online

Survey), (Babson Survey Research Group, January 2014).

3. Overview and Guiding Principles for Strategic Planning

3.1 Princeton’s Current Activities in Online Education

Hundreds of colleges and universities are actively exploring the potential of online technology to

enable new teaching methods, to cut costs, and to deliver educational material to students around

the world. One of the most visible sites of such exploration is Coursera, an online learning

platform where millions of students can find hundreds of online courses (now known as

“Massive Open Online Courses” or MOOCs) produced by faculty from more than 110

institutions, including Princeton and several of its peers. An equally prominent platform is edX,

the MIT-Harvard collaboration that offers more than 550 MOOCS from more than 60

institutional partners. At the same time, faculty members at many of these institutions are also

using online technology to “flip” their classrooms on campus. This mode of instruction requires

students to view pre-recorded lectures online in advance of class and then dedicates regularly

scheduled class time to group discussion and other forms of “active learning.” The Council’s

research indicated that “online education” encompasses a wide variety of other tools and

methods that extend well beyond the MOOC and the “flipped” classroom.

One of four institutions to participate in the launch of Coursera, Princeton has experimented

actively with MOOCs since 2012.

Over the last three years, 16 members of the Princeton

faculty, representing 10 disciplines, have stepped forward to develop MOOCs. (A summary

table of Princeton’s MOOCs since 2012 and an overview of MOOC production among its peers

are attached as Appendices D and E.) Princeton’s MOOC offerings have varied widely in terms

of their format, assessment methods, length, and subject matter. The total enrollment in these

MOOCs exceeds 2.4 million. Importantly, as a matter of policy, Princeton decided not to offer

fee-based certificates to those who complete these offerings.

Several faculty members have also used online lectures to flip their classes on the Princeton

campus or have otherwise endeavored to integrate recorded lectures or online learning materials

into their Princeton courses (see Appendix F). To support these efforts, Princeton offers course

development stipends and the resources and professional expertise of the McGraw Center for

Teaching and Learning. (See Section 5.1 of this report for further information regarding

Princeton’s resources for supporting online education.)

The interest among faculty in developing courses with a significant online component has varied

widely by department and division. Over time, the distribution of Princeton courses on online

platforms has skewed heavily to the STEM fields. Over 62 percent are science and engineering

courses, roughly 31 percent are social science courses, and 7 percent are humanities courses (see

Since entering into its non-exclusive agreement with Coursera in 2012, Princeton has expanded its

platform options to include NovoEd (http://bit.ly/1ptGcRU) and Kadenze (http://bit.ly/1UANDFX

which is specifically designed to support the arts. Princeton is currently evaluating edX, the MIT-

Harvard collaboration. Some faculty members have also developed their own platforms for the

dissemination of educational content. For example, Professor Mung Chiang created 3ND (or “Three

Nights and Done”), a platform that offers short courses consisting of three one-hour videos.

Appendix G).

When Princeton joined the launch of Coursera in 2012, it did so with the primary objective of

enhancing the quality of education on Princeton’s own campus. Princeton also embraced this

opportunity to share educational materials with the world and, thereby, advance our mission “in

the nation’s service, and in the service of all nations.” In particular, interested Princeton faculty

sought to determine whether online modes of instruction might enhance their on-campus teaching

in the following ways:

· Flipping the lecture: In principle, students first work through recorded “lectures” at their

own pace, answering in-video questions as they go. Professors are then free to devote

more face-to-face class time to discussion, problem-solving and other interactive forms of

teaching. Students’ responses to in-video quizzes, for example, may reveal particular

difficulties that professors can then address in a targeted manner during class. It is

important to note that the “flipped” model does not require that the recorded materials be

made publicly available; a number of Princeton faculty members have used closed

platforms to post materials for Princeton students only.

· Importing a global perspective: Some faculty use MOOCs in order to import a global

perspective into the Princeton classroom, an especially helpful feature in comparative and

interpretative disciplines such as world history. Video-conferencing technologies, such as

Google Hangout, are sometimes employed to conduct “global precepts” that connect

Princeton students with Coursera students from around the world.

· Crowd sourcing: As thousands of students work through online course materials, they

often offer candid, instantaneous feedback to one another and to the instructor that leads to

improvements in the materials and pedagogical methods of the course.

· Assessments: Online courses in quantitative disciplines often use quizzes and problem

sets that can be graded automatically, greatly reducing the amount of time it takes for

students to receive feedback and freeing up faculty time to consult with students having

difficulty

. While instructors in the more qualitative disciplines have not designed quizzes

that are machine-gradable, some have developed peer-graded exercises that can also give

students meaningful feedback.

· Catalyst for innovation: An ancillary benefit is that the process of developing and

delivering a MOOC – designing and recording lectures, collaborating with the McGraw

Center for Teaching and Learning, designing assessments, mediating global discussion

forums – can lead faculty members and the graduate students teaching alongside them to

rethink their general approach to pedagogy, thus invigorating even their “conventional”

teaching.

3.2 Evaluating Princeton’s Current Initiatives

Have these experiments in various modes of online learning been successful? Several Princeton

faculty members have seen many of these putative benefits materialize in the course of their early

experimentation with MOOCs and the “flipped” classroom.

They also note that the benefits

associated with the public dissemination of online lecture materials are significant. Providing

free educational materials is consistent with Princeton’s mission and also improves the visibility

of a Princeton education in the eyes of prospective students across the world. Conversations

with officials at the Alumni Council indicate that substantial numbers of Princeton alumni are

also attracted to MOOCs offered by the Princeton faculty.

However, many faculty members who have experimented with online education at Princeton have

also encountered a number of significant challenges. Flipping the lecture is not easy. It requires a

great deal of time to record, edit and update lectures, create in-video quizzes and other features,

and then develop a plan for using face-to-face class time more effectively. Posting recorded

lectures online can also weaken class attendance and reduce the quality and frequency of

interaction between students and faculty, a defining component of residential education at

Princeton. The high volume of discussion on MOOC forums can be difficult to manage,

compromising faculty attention to Princeton’s students.

After holding discussions with several members of the faculty who were using online teaching

methods, the Council decided to address the evaluation of Princeton’s current efforts in a more

systematic fashion by surveying both faculty and students. Both the faculty and student responses

to our survey questions regarding the effectiveness of the “flipped” classroom model were

decidedly mixed. The key findings were:

· Faculty members hold strong and widely diverging opinions about which forms of online

education are best and, indeed, whether technology has anything at all to add to their

current pedagogical approaches. In some cases, these views are informed by hands-on

experiences with online learning; in others, they are based on other sources of

information.

· Students also have polarized (roughly 50/50) views about online learning based on their

specific experiences in Princeton courses. Most student comments are strongly in favor

or against courses with online learning components also include remarks about how that

student best absorbs new material and what type of teaching sustains their attention.

· Interestingly, both faculty and students agree that many of the perceived problems with

flipped classrooms relate to the quality of the classroom component of instruction, as

opposed to the quality of the recordings themselves. Students who were negative about

the experience complained that the classroom instruction was boring and unfocused, and

faculty who have flipped their classrooms indicated that it is a difficult, labor-intensive

way to teach that requires careful preparation and continued experimentation.

· Some faculty members regard the time and effort invested in thinking about how to flip a

See Adelman, Jeremy, "History a la MOOC," Perspectives on History: The Newsmagazine of the

American Historical Association (February 2014); and Connell, Christopher, “The Future of Education?

As the World Gets a Taste of Princeton, Princeton Gets Ideas to Improve At Home,” Princeton Alumni

Weekly (May 15, 2013).

classroom or how to create online content as beneficial to their teaching pedagogy.

· Students who are positive about their online learning experience (roughly 50%) usually

express surprise that they enjoyed the online videos or flipped classrooms, and write

about how these experiences enhanced their approach to learning.

· Faculty respondents generally did not support the view that Princeton should be making a

deliberate effort to increase its production of MOOCs for audiences beyond our student

body.

3.3 Experiences at Other Institutions

Our investigation of online learning at a number of peer institutions is summarized in Appendix

E. The level of experimentation with MOOCs and other forms of online education at Princeton

over the past four years appears to be near the middle of our peer group (Appendix D). One

should note that to the extent that schools like Harvard, Penn and Stanford have been more active

than Princeton in the development of MOOCs, it is because of their professional schools, the

sector of higher education that has historically been more open to the development of fee-based

models for online coursework.

In any case, as indicated in Appendix B, our peers’ approaches and experiences have been

similar to ours. Their faculty members are primarily interested in how online instruction,

especially the flipped model, can enhance the quality of education in their classroom. Further,

while faculty experimentation in this area has stimulated a great deal of interest in pedagogy

more generally, they are also having mixed results.

The wide variety of reactions and experiences among the faculty at Princeton and elsewhere is not

surprising given that “online education” is a rapidly changing area that encompasses many

different modes of instruction, none of which has emerged as most compelling. Indeed, the

leading studies of online instruction and its efficacy confirm that the relative benefits of online

instruction remain unclear.

Moreover, our research does not indicate that there is a compelling

“first-mover” advantage for institutions that use online education as a means of enhancing the

education of their students on campus. Princeton should remain nimble enough to adopt the

most effective models and methods as they become clear.

3.4 Recommendations

· In our conversations with individuals at other schools, it was striking that not a single one

was willing to venture a guess as to what online approaches would be used at his or her

institution in five years. No one knows how this field will evolve. Princeton’s strategy with

respect to online education should be informed by the pervasive uncertainty about its

See, e.g., Wu, D. Derek, Online Learning in Postsecondary Education: A Review of the

Empirical Literature (2013-2014), Ithaka S+R (2015) and Reich, Justin, “Rebooting MOOC Research:

Improve assessment, data sharing, and experimental design,” Science (January 2, 2015). Reich observes,

“We have terabytes of data about what students clicked and very little understanding of what changed in

their heads”(pp. 34-5).

effectiveness. In the words of President emeritus William Bowen, “walk deliberately, don’t

run, toward online education.”

· Princeton should give highest priority to the development of online materials and tools that

will build upon our strengths as a residential learning community and that will enhance and

supplement -- not supplant -- time-tested modes of teaching and faculty-student interaction.

As a corollary, reduction in instructional costs per se should not be the goal of introducing

online instruction. Indeed, our research suggests that the development of pedagogically first-

rate online teaching materials might actually increase costs.

· Princeton should encourage the public dissemination of online course materials only to the

extent that those efforts are likely to yield clear benefits to Princeton students.

· Broad consultation with the faculty regarding Princeton’s strategy for online education is

critical, especially in light of faculty members’ strong and widely diverging views on this

subject. To that end, the Council on Teaching and Learning should serve as a forum for

addressing faculty and student concerns, advising the University on priorities in this area, and

periodically reviewing the use of online materials in Princeton courses to ensure that they are

not diminishing the quality and frequency of student-faculty interaction.

Bowen, William G., “Walk Deliberately, Don’t Run, Toward Online Education,” The Chronicle of

Higher Education (March 25, 2013).

4. Priorities for the Future Experimentation

While the future of online learning is characterized by a high degree of uncertainty, the Council

identified two types of initiatives that appear to be especially promising and worthy of future

experimentation at Princeton: (1) the development of supplemental materials designed to

improve student retention in the STEM disciplines; and (2) the development of small private

online courses (SPOCs) that improve student options for summer study. These two types of

initiatives are especially appealing because they are naturally aligned with Princeton’s greater

efforts to support the complex educational needs of its increasingly diverse student body.

4.1. Supplemental Online Materials for STEM Preparation

The Council found that a number of our peer institutions are using supplemental online materials

to prepare students for introductory courses that often prove to be stumbling blocks in the STEM

disciplines (i.e., science, technology, engineering, and math). Here are three notable examples:

· Bryn Mawr’s TIDES initiative (Teaching to Increase Diversity and Equity in STEM)

offers supplementary online modules that students can use at any time to strengthen their

mathematical skills in physics, biology, chemistry and geosciences

(http://bit.ly/1Tl83Wa). Bryn Mawr also offers two beginning mathematics courses

(http://bit.ly/1f5uxYC) that use a “playlist” of modules that students use as a

supplemental resource.

· Penn’s SAIL initiative (Structured, Active, In-Class Learning) encourages STEM faculty

to increase active engagement in their classes by using online tools to deliver content

outside of class and spend class time on highly participatory and active learning activities

(http://bit.ly/1Tl8YpL). Preliminary analysis of learning outcomes indicates that students

in SAIL classes “are better able to explain complex concepts.”

· MIT offers an introductory chemistry course that uses online assessment tools in a

“mastery-based” approach to learning: the students must pass a minimum number of

these online assessments, but are allowed to repeat them as often as needed to pass within

a 14-day period. According to the professor teaching the course, the goal of this

approach is “to bring up the bottom of the class and raise the pass rate to 100%.”

(http://bit.ly/1UMf9yx).

For a recent statement of the University’s commitment to equitable access to our curriculum, academic

support and advising, see the Report of the Special Task Force on Diversity, Equity and Inclusion

(Princeton University, May 2015), especially pp. 9-10.

Quoted from conversation between McGraw Instructional Designer Mona Fixdal and Director of Penn’s

Center for Teaching and Learning Bruce Lenthall (August 5, 2015).

These types of self-guided, online resources provide beginning students with an opportunity to

master essential skills and concepts at their own pace, free of the logistical constraints (and

possibly the personal and cultural inhibitions) that can limit the degree to which they take

advantage of tutoring sessions, study halls, and other forms of supplemental instruction. Bryn

Mawr, Penn and a number of other peer institutions are optimistic that these initiatives will be

particularly useful in making STEM disciplines more accessible to underrepresented groups and

students from low socio-economic backgrounds. However, these initiatives are so new that there

is not a substantial body of published research to demonstrate their efficacy.

From the

standpoint of Princeton’s ability to move forward in this area, it is important to note that a

number of funding agencies and foundations, such as the American Association of Colleges and

Universities and Andrew W. Mellon Foundation, have demonstrated a high degree of readiness

to support the development of online materials that will improve STEM retention.

Princeton faculty members have also expressed and demonstrated considerable interest in

developing supplementary online teaching tools for introductory STEM courses. For example,

the faculty teaching the pre-med physics sequence and organic chemistry recently approached

the McGraw Center to discuss online modules on foundational concepts such as “chemical

bonding” and “trigonometry for physics.” McGraw has also worked with faculty members

teaching introductory statistics courses to develop self-paced modules that teach students how to

use the programming software “R.”

It is worth noting that the Council’s survey of Princeton

students indicated that they are generally comfortable using online lectures and course materials

as supplemental resources.

4.2 Small Private Online Courses (SPOCs)

The Council also found that a number of our peers have had some encouraging results in their

early development of what are known as “small private online courses” (or SPOCs). Typically

offered during the summer months, these courses have limited enrollments (approximately 20-25

students) and consist of live, faculty-mediated discussions as well as course-site activities that

For some initial studies that point to the promise of self-paced and “just-in-time” online interventions,

see: Bowen, William et. al., “Interactive Learning Online at Public Universities: Evidence from

Randomized Trials,” ITHAKA S+R (2012); Stevenson, Katherine and Louis Zweier, “Creating a Learning

Flow: A Hybrid Course Model for High-Failure-Rate Math Classes,” Educause Review (December 15,

2011); and Twigg, Carol, “Using Asynchronous Learning in Redesign: Reaching and Retaining the At-

Risk Student,” Journal of Asynchronous Learning Networks, Vol. 13: Issue 3 (2009).

Bryn Mawr’s TIDES initiative received $170,000 from the Helmsley Charitable Trust, and its “Math

Fundamentals” initiative to improve STEM completion received $1.65 million from the U.S. Department

of Education’s “First in the World” grant program. Penn’s SAIL initiative was seeded with a $500,000

grant from the Association of American Universities and the Helmsley Foundation.

The modules were introduced in the spring 2015 semester. Roughly half of the students enrolled in

statistics courses visited a module; of those, approximately 1/3 watched a lecture, and half of them

submitted an exercise. Student responses to a survey aimed at gauging the effectiveness of the modules

were too few to draw meaningful conclusions. McGraw intends to undertake more thorough investigation

of this question in the fall 2015 and subsequent terms.

students can complete on their own time (such as writing in online discussion forums). Here are

three notable programs that offer SPOCs for summer study:

· Yale’s “Summer Online” program offers over 20 courses for credit in two four-week

sessions (http://bit.ly/1Elpyd0). Courses are developed and taught by Yale faculty

using virtual classroom technology that enables students and teachers to come

together “face-to-face” in synchronous discussion sessions each week.

· The “Undergraduate Summer Session” at Brown offers enrolled undergraduates the

option to choose from a variety of online, for-credit courses developed by Brown

faculty (http://bit.ly/1L46YO3). In these courses, students study together in small

groups and participate in live discussions via web conferencing technology. Seven

courses were offered in the summer of 2015; students are allowed to enroll in up to

two courses for credit in a single summer.

· “Penn Summer Online” offers 30 fully-online courses that carry degree credit and can

fulfill degree requirements for Penn undergraduates (http://bit.ly/1ElpBWb).

Although delivered from a distance, Penn’s online summer courses are highly

interactive learning experiences that are designed and led by Penn faculty members.

In addition, Cornell allows enrolled students to receive degree credit for online summer courses

with the approval of their department. That four of our Ivy peers now offer summer SPOCs for

credit provides an opportunity for Princeton to learn about the most effective ways to take

advantage of this mode of online instruction.

SPOCs have the potential to serve an important need at Princeton. Every summer, hundreds of

our undergraduates take courses at other institutions for credit toward their Princeton degree.

(Princeton students are permitted to take three of the 31 courses required for the A.B degree and

four of the 36 courses required for the B.S.E. degree at other institutions, subject to decanal and

departmental approval.) In fact, over the past three years, nearly 22 percent of our students have

transferred in one or more degree credits for summer courses taken elsewhere, and over 40

percent of these were introductory courses in STEM disciplines.

Moreover, students in the

bottom 40 percent of the GPA distribution took more than 55 percent of these courses.

Princeton students take these summer courses for a variety of reasons. Sometimes the goal is to

make their course load more manageable during the fall and the spring semesters. In other cases,

they are required to take these courses in order to repair deficiencies in their progress to degree

because they have failed or dropped essential courses during the preceding academic year.

Unfortunately, the pool of available summer courses is especially limited for Princeton students

because most summer programs, including those offered by our peers, start before the end of our

academic year. Because there is not much choice, sometimes the quality of instruction in these

courses is questionable.

These courses do not include Princeton’s summer offerings such as the Global Seminars program and

its intensive language study programs.

SPOCs offered for summer study also might be adapted for use during the regular academic year

to serve the needs of Princeton students who wish to study abroad but are reluctant or unable to

do so because they are tied to certain courses on campus. For example, the SPOC model might

be used to enable students to take certain types of prerequisite courses in math or statistics as

well as certain pre-med courses while studying abroad at institutions that may not have adequate

offerings in these particular areas.

4.3 Recommendation

· While supporting experimentation with a variety of promising forms of online education,

Princeton should give highest priority to developing supplemental online materials and

SPOCs that will help students make their way into the STEM disciplines and other fields

where entry has been particularly challenging. This work should be Princeton’s signature

initiative in the field of online education. To the extent possible, we should disseminate

the fruits of our efforts for public benefit.

5. Infrastructure and Resources

5.1 Pedagogical and Technical Support

Princeton’s online course initiatives are administered through the McGraw Center for Teaching

and Learning, which reports to the Office of the Dean of the College.

Over the past four years,

at least 23 members of the Princeton faculty have worked closely with McGraw’s staff to design

and create online course materials for public dissemination and/or use in Princeton courses on

campus. The Online Learning team at McGraw includes instructional designers and video

producers who work closely with faculty members to create online teaching materials, develop

strategies for connecting online environments to Princeton classes, and implement related

interactive classroom experiences for students. The McGraw staff also consults with faculty

members on the development of funding proposals for online and blended projects,

communicates with online platform providers, and offers guidance on relevant University

policies.

In their anonymous survey responses, Princeton faculty consistently expressed a high degree of

satisfaction with the pedagogical and technical expertise that McGraw provides in support of

their online teaching projects. “Support from McGraw is astounding,” writes one, “I can’t say

enough positive things about the support there.” Among faculty who have not yet experimented

with online modes of teaching, many identified “professional production support,” and

“instructional seminars on how to do it” as factors that would make them more likely to develop

a flipped classroom.

While McGraw serves as the sole administrative home for Princeton’s experimentation with

online education, a number of our peers use separate offices to support externally focused

MOOCs, on the one hand, and internally focused teaching projects, on the other. Of course, this

distinction can be nebulous at times. For example, as discussed earlier (section 3.1), the process

of developing and delivering an externally focused MOOC can lead to a variety of benefits for

students on campus. As one Princeton professor observed after teaching his first MOOC, an

externally-focused summer offering, “I had begun worrying about how I could bring the New

Jersey campus experience to [the world]; I ended by thinking about how to bring the world back

to the classroom in Princeton.”

Nevertheless, in many cases, the delivery of externally focused

MOOCs, especially those offered on an “on demand” basis requiring no further involvement

from the instructor, can over time lose their power to drive innovations and insights that will

benefit the teaching mission on campus. The maintenance of such materials should not fall

within the purview of McGraw.

Following an external review in 2012, Princeton moved its Educational Technologies Center and its

Broadcast Center, which were then part of the Office of Information Technology, into the McGraw

Center so that our pedagogical and technical expertise would be more fully integrated under the direction

of a single administrative unit.

Duneier, Mitchell, “Teaching the World from Central New Jersey,” Chronicle of Higher Education

(September 3, 2012).

5.2 Faculty Incentives

As noted above, our research indicated that substantial time and effort are required to develop

first-rate online materials, and these costs almost certainly reduce the number of faculty who are

willing to experiment with this approach to teaching. According to the results of our survey,

faculty interest depends on several factors: (1) financial incentives (e.g., course development

stipends and revenue-sharing policies associated with the development of materials that are

marketed); (2) availability of time (e.g., reduced teaching loads for faculty members who

dedicate time to the development of online course materials); (3) compelling evidence of the

effectiveness of online learning environments; and (4) in the case of materials intended for

audiences outside the university, the possibility of enhanced visibility for themselves and their

departments that comes from teaching on a global stage.

Princeton currently offers stipends -- but not reduced teaching loads – to faculty members who

make compelling proposals for MOOCs and other types of online projects. In many cases,

faculty members use their stipends to engage graduate students to assist with the development

and delivery of the online materials, a labor-intensive process that often requires a dedicated

team with expertise in the subject matter of the course.

In the case of MOOCs, many institutions, including a number of Princeton’s peers, defray their

production costs by selling branded certificates to students who complete a given course.

This

monetization strategy typically generates revenue for both the institution and the faculty member

who offers the course. Princeton decided not to offer fee-based, branded certificates for several

reasons: (1) the difficulty of verifying the identity of online students; (2) uncertainty regarding

the educational value and effectiveness of an online course; (3) potential dilution of the Princeton

“brand”; and (4) the administrative burdens and “customer service” issues that come with the

scale of MOOCs. In 2013, an ad hoc faculty committee on online courses, chaired by Professor

Gideon Rosen, recommended that the University continue this policy, but conjectured that free,

unbranded certificates might improve student engagement and completion rates (see the 2013

Report of the Ad Hoc Faculty Committee on Online Courses. (http://bit.ly/1Kdi7Gv).

However, Princeton faculty members are free to monetize their online course materials by other

means, provided they follow the University’s long-standing rules against “teaching elsewhere”

See figures Q08 and Q09 on page 3 of Appendix C. The response rate to the faculty survey was low

(see figure Q01), so these results should be interpreted with caution.

The cost of developing and delivering an online course depends on several factors, including the length

of the course, the faculty members’ stipend (if any), the number of graduate student assistants needed to

support the course, video production charges, copyright clearance fees, and so on. At Princeton, the

combined cost for the video production and faculty stipend tends to fall between $25,000 and $35,000,

not including overhead (e.g., administrative support and the time of the professional staff in the McGraw

Center).

Following the recommendation of the Rosen committee, Princeton recently authorized the use of

unbranded Statement of Accomplishment (SoA) in 2 courses. Consistent with research at other

institutions, we found that approximately 50 percent of MOOC students who complete the first

assignment in an online course continue and complete all course requirements and receive a statement.

and agree to standard terms regarding reimbursement for their use of substantial University

resources.

As noted earlier, Princeton faculty are decidedly heterogeneous with respect to their willingness

to experiment with online learning. Nearly all of the Princeton faculty members who have

experimented thus far have demonstrated their commitment and enthusiasm by putting their

projects through multiple iterations. For example, Jeremy Adelman in the Department of History

offered his “World History since 1300” course online on three consecutive occasions, each time

experimenting with increasingly interactive modes of teaching, both online and in the classroom.

Likewise, faculty in the School of Engineering and Applied Science such as Claire Gmachl,

Howard Stone, and Mung Chiang continue to experiment as they seek the most effective balance

between traditional lectures and active learning in their “flipped” classrooms. Obviously, these

individuals do not comprise a random sample of the faculty. Among the faculty who choose not

to be involved, our survey indicated that in addition to the incentive issues mentioned above,

another possible roadblock is lack of awareness of the resources that are currently available to

support online projects.

5.3 Strategic Partnerships

Our interviews with individuals at peer institutions revealed a high degree of interest in forming

partnerships for purposes of advancing large-scale projects in online education (Appendix B,

finding #7). In principle, such partnerships can help institutions cover curricular gaps, conserve

resources (including faculty time), more efficiently assess student learning, and reach a broader

audience.

The Council noted two types of partnerships that seem particularly worthy of

consideration:

· Large quantitative courses: These courses attract high enrollments of students with

varying levels of preparation, can be difficult to staff, and often have high attrition rates.

The Yale Computer Science Department, facing rising enrollments that far outpace the

growth of their faculty, recently opted to join forces with Harvard in the teaching of

introductory computer science. Beginning in the fall of 2015, Yale students will watch

live-streamed lectures from Cambridge, and students on both campuses will complete

tests and assignments online, and interact via web conferencing technology. To Harvard

faculty, “a shared course allows for interactions not possible within a single physical

classroom . . . cultivating a healthy diversity of viewpoints.”

We imagine that this

While intellectual property issues are important, the Provost’s charge to the Council explicitly excludes

them from its consideration.

See Figure Q05 on page 3 of Appendix C.

Straumsheim, Carl, “Our Powers Combined,” Inside Higher Ed (March 19, 2014); Griffiths, Rebecca,

“Best Practices in Collaborative Multi-Campus Online Learning,” ITHAKA S+R, Plenary Session at the

2014 Annual Meeting of the North Carolina Conference of Graduate Schools (November 7, 2014).

Vilensky, Mike, “Coming Soon to Yale: A Class Taught by Harvard,” WSJ (December 4, 2014).

approach might be particularly attractive to faculty in certain departments with large

enrollments in introductory courses.

· Less commonly taught languages. Several of our peers are pooling resources to create

high quality online materials to enable instruction in a number of languages that they

could not otherwise afford to teach on their campuses. Princeton might consider joining

an existing consortium or creating a similar partnership to provide a greater variety of

language courses for undergraduate and graduate students.

As is true with other issues relating to online learning, the efficacy of strategic partnerships, and

the conditions under which they are most likely to be successful, are uncertain. External funding

agencies such as the Andrew W. Mellon and Teagle Foundations are actively soliciting proposals

for collaborative online projects and the formation of consortia of liberal arts colleges. We

expect that the Mellon and Teagle efforts will provide a good deal of information about the best

approaches to designing and implementing partnerships with other institutions and to supporting

faculty who seek to establish them.

5.4 Research and Assessment

Online teaching and learning is an important topic of scholarly inquiry in a variety of fields, and

many of the centers for teaching and learning at peer institutions (e.g., Harvard, Yale, Penn and

Michigan) have staff positions that are dedicated to supporting faculty members with the design

and implementation of assessment tools for online learning. A number of Princeton faculty and

instructors engaged in online experiments have contributed to the literature on learning science;

others are actively involved in ongoing studies of online interventions in their courses.

Indeed,

the report of the Rosen committee recommended that “every proposal for a new on-line course

should include a plan for assessing the course after the fact” (pp. 8). The McGraw Center does

not, however, have the capacity to support such research.

5.5 Recommendations

· Given its mission and the expertise of its staff, the McGraw Center should support only those

This recommendation is seconded in the draft report of Princeton’s Regional Studies Task Force,

chaired by Professor Mark Beissinger. For an example of this kind of effort, see Schmitz, Emily, “Duke,

UVA partner to teach Creole Tibetan,” The Chronicle of Higher Education (March 21, 2013).

Brinton, Christopher G., Mung Chiang, Shaili Jain, Henry Lam, Zhenming Liu, Felix Ming Fai Wong,

“Learning About Social Learning in MOOCs: From Statistical Analysis to Generative Model,” IEEE

Transactions on Learning Technologies, Vol. 7, No. 4 (October-December 2014: pp. 346-359); Brinton,

Christopher G., Swapna Buccapatnam, Mung Chiang, H.V. Poor. "Mining MOOC Clickstreams: On the

Relationship Between Learner Video-Watching Behavior and Performance,” Cornell University Library

(March 2015); Duneier, Mitchell, “Teaching to the World From Central New Jersey,” Chronicle of

Higher Education (September 3, 2012).

online projects that have clear potential to enhance teaching and learning at Princeton and

should not support the continuing delivery of MOOCs once their pedagogical value to our

students has diminished. Such MOOCs should be housed and maintained elsewhere, perhaps

in the Office of Communications and/or Alumni Relations.

· To encourage future experimentation in areas of priority, the university should offer special

incentives beyond what we currently provide. We recommend the following:

o Augment the 250

Fund for Innovation in Undergraduate Education, which is already

oversubscribed.

o Allocate a modest number of FTEs to the McGraw Center that it can use to negotiate

teaching reductions in order to support projects of the highest priority.

o Offer additional AI support for courses with significant online components, which

will have the additional benefit of providing our graduate students with opportunities

to develop skills and know-how that will enhance their prospects on the job market.

· There is considerable room for improvement in our communications with faculty regarding

the benefits and limitations of online education, the resources available to support online

projects, and the processes for developing and reviewing proposals. The McGraw Center

should lead our communications in this area. Additional resources would be needed for

website enhancements and other efforts to improve the visibility of this initiative and to

support further experimentation.

· Faculty should be encouraged to explore collaborations in online learning with other

institutions. Proposals for collaboration should be developed in consultation with the Office

of the Dean of the College.

· We should give high priority to making resources available for faculty who want to pursue

research on the effectiveness of various types of online learning. Staff with appropriate

training in statistics and project evaluation will be needed at the McGraw Center to support

this work. Also, one important criterion for evaluating current and prospective online course

platforms should be the quality of their assessment tools.

6. Conclusion

During the past few years, members of Princeton’s faculty, guided and supported by the McGraw

Center, have done a great deal of experimentation with various approaches to online education,

and in the process have learned a great deal about their limitations and advantages. While it is

too soon to know how and to what extent online technologies may transform the landscape of

higher education, it is clear that continued faculty experimentation has the potential to enhance

the quality of teaching on our campus. A top priority should be the provision of incentives for

faculty members to create online materials that will make our curriculum, especially in STEM

disciplines, more accessible to our increasingly diverse student body. The McGraw Center has

the professional expertise to support these and other initiatives, but it will need additional

resources to establish a more robust infrastructure for assessing the efficacy of various

approaches to online learning, communicating the available options to faculty, and facilitating

the development of partnerships with other institutions. Above all, as Princeton moves forward

in this area, we must remember that utilizing new technology should be neither an end in itself

nor a means simply for cutting costs. Rather, online learning is a tool for building upon our

strengths as a residential learning community by enhancing and complementing -- not

supplanting -- time-tested modes of teaching and faculty-student interaction.

Appendices

Appendix A: Council on Teaching and Learning Members AY 2014-15

Wendy Belcher, Assistant Professor of Comparative Literature and African American Studies.

Robert K. Root University Preceptor, Department of Comparative Literature

Cole Crittenden, Associate Dean for Academic Affairs, Office of the Dean of the Graduate

School

Edward Felten

, Robert E. Kahn Professor of Computer Science and Public Affairs,

Department of Computer Science

Carol Greenhouse, Arthur W. Marks '19 Professor of Anthropology. Chair, Department of

Anthropology

Lisa Herschbach, Director of the McGraw Center for Teaching and Learning, Associate Dean

of the College

Jeff Himpele, Director of Teaching Initiatives and Programs, McGraw Center for Teaching and

Learning

Fred Hughson, Professor of Molecular Biology, Department of Molecular Biology

Adam Maloof, Associate Professor of Geosciences, Department of Geosciences

Simone Marchesi, Associate Professor of French and Italian, Department of French and Italian

Clayton Marsh, Deputy Dean of the College, Office of the Dean of the College

Rodney Priestley, Assistant Professor of Chemical and Biological Engineering, Department of

Chemical and Biological Engineering

Harvey Rosen, John L. Weinberg Professor of Economics and Business Policy, Department of

Economics, Chair of the Council

Professor Felten served on the Council during the academic year 2014-15, but took a leave of absence

in June 2015 to serve as Deputy U.S. Chief Technology Officer at the White House Office of Science and

Technology Policy. He was not available to review and comment on this report.

Appendix B: Summary of Interviews with Peer Institutions

1. What are your goals in investigating and using online learning? Would you describe your activities as oriented

more externally or internally? If you are externally oriented, does any of that external online activity currently

connect with internal activities, for example admission recruitment?

Columbia

Internal: improve teaching on campus; lifelong learning for alumni; fundraise. External:

showcase faculty and programs; attract applicants; share knowledge.

Duke

Internal: enhance student learning; promote innovation in teaching. [Advantage: internal

circulation of digital material between MOOCs, flipped courses, alumni courses, external

professional schools.] External: showcase academic excellence; share knowledge with the world.

[Highlight: Coursera Translation Partners; Learning Hubs].

Michigan

Mostly internal. Enhance residential education. Promote educational research on campus.

MIT

MIT is a recognized leader in digital learning with a significant platform of activities that appear

to work together in an integrated manner. MIT’s digital learning aim to “impact lives and society

in ways not previously thought possible.” Focus on two areas: 1) enhance residential education

via online and blended environments, and 2) meet the demand for online education worldwide via

MOOCs.

Yale

Internal: complement offerings (Yale Summer Online Courses; Freshman Scholars program);

impact teaching practices on campus. External: MOOCs via Coursera; other courses through

Open Yale. Recruitment not a factor.

Stanford

Internal: experiment with new pedagogy; revive attention to pedagogy. Online education also a

subject for study.

Brown

Both. Operationally a committee was formed: split the responsibility between the Sheridan Center

and the School of Professional Studies. School of Professional Studies interested in outward

facing curriculum. Create courses with faculty and distribute them outside. Goal is to enhance

curriculum; ramp up online and blended teaching in the professional study. Internally:

Undergraduate courses for credit during the summer: pre-college audience + piloted two courses

for credit (small, boutique courses: writing and literature courses). Looking ahead at developing

classes that help develop competency (intro chem. or calculus).

2. Are you developing your own technologies or significantly modifying existing technologies for internal use?

Columbia

Develop Mediathread [platform for exploration, organization, and analysis of multimedia content]

and Forest [simple educational modules].

Duke

Coursera, massively. But also 2U platform for flipping courses. Develop Duke Coursera

Specialization [sequence of courses, capstone project, certification].

Michigan

Develop Open.Michigan into a teaching platform to compete with proprietary platforms.

MIT

MIT and Harvard co-founded EdX in 2012. MIT utilizes the EdX program to operate MITx

(residential-setting content delivery, creation of material for world-wide distribution; self-study).

Also: MIT OpenCourseWare, an online platform on which MIT makes course lecture material

free to the general public worldwide.

Yale

Coursera, though OpenEdX considered intriguing.

Stanford

OpenEdX, being adopted, but not passively (contributing code). Coursera and Udacity born here.

Brown

Technology not the driver: adopt and adapt rather than develop. But open to experimentation:

faculty-driven; collaborative learning a goal.

3. What percentage of your current courses have an online component? What percentage of students are taking

courses that have a significant online component?

Columbia

No number available. Depends on schools.

Duke

Twenty-five courses in 2012-14. Twelve courses have added a significant online component in

just last year. Courses originate from all quarters: humanities, sciences, professional school.

Junior and senior faculty alike; Professor of the practice takes leading role.

Michigan

No statistics available. Professional schools all use online learning. Medical School leading.

MIT

2,200 courses are available on MIT OpenCourseWare – some courses have been translated to

other languages.

Yale

Few. Between 5 and 10% of students in a given semester.

Stanford

Best estimate is 50%.

Brown

Low. Generational issue at play: more senior faculty reluctant; less senior faculty under time

constraints.

4. Can you provide examples of successful courses? What factors contributed most to their success?

Columbia

For MOOCs: High production, open discussion, careful sectioning. For flipped courses: enable

constant participation (regular quiz on pre-class lecture assignments, in-class lecture on specifics

and with live polling on mobile devices); foster collaborative and problem-based learning; elicit

feedback.

Duke

For MOOCs and hybrid Duke-credit courses: writing-program course as testing ground for peer-

to-peer evaluation; intro chemistry course developed to test student learning and peer-to-peer

interactions (both Gates funding to run assessment). Highlight: Coursera’s Learning Hubs

(organizations in locations worldwide provide internet access and in-person instruction to support

MOOC students).

Michigan

No examples. [Development/implementation seems to be happening at the level of

School/College.]

MIT

Introduction to Solid State Chemistry has been considered a successful course at MIT. However,

one must carefully define the definition of success. For this online course, students were given

unlimited chances to pass exams.

Yale

Flipped courses: success depends on the real pedagogical thinking that goes into the course. Good

model: offer lecture in video, quiz on lecture, assess weaknesses, address them in class. Once

freed from conveying content, the teacher may focus on inspire learning, nurture critical thinking,

help develop the ability to manipulate complex information.

Stanford

Machine Learning; Introduction to Mathematical Thinking; How to Learn Math; Quantum

Mechanics for Scientists and Engineers; Principles of Economics. No comments on qualities.

Brown

Blended course most successful. Undergraduate courses for credit during the summer: pre-college

audience + piloted two courses for credit (small courses, boutique: writing and literature courses).

Looking ahead at developing courses that help develop competency (intro chem. or calculus) and

summer bridge course.

5. Have you developed a way to assess the pedagogical advantages of incorporating online components in your

courses? Do you find course evaluations or learning assessments more useful in assessing the success of these

courses?

Columbia

Nothing systematic yet. Evaluation done via ad-hoc studies of individual courses or modules.

Duke

Ten studies currently being conducted on comparing traditional and online content delivery.

Faculty successful in securing Gates funding for such studies. Developing scholarship on digital

pedagogy a crucial goal.

Michigan

Ongoing PhD-level research projects to develop assessment strategies. For Open.Michigan,

download volume considered significant.

MIT

The Office of Digital Learning is actively supporting research to address this question.

Yale

No system beyond course evaluation. Case-by-case assessments tried: comparative study of same

course in flipped and unflipped format (Math 115). Flipped won. Research agenda: are the

learning outcomes higher enough to justify higher production costs?

Stanford

Intensely, through the Graduate School of Education. Standard student evaluations likely to

include questions about online components. For MOOCs, assessment via post-course surveys and

ad-hoc studies.

Brown

Under development: Data in STEM most useful: hard thinking is done here. Grant from the AAU

to transform STEM education. For other disciplines goal is larger: integrating evaluation into the

starting projects for online pedagogy (not only gpa, entry scores, test scores, retention, teaching

evaluations; but a larger holistic approach). Develop a system of advising, mentoring, enhancing

commitment to courses.

6. Are online resources used in the framework of academic support services, for example online tutoring for your

students or developing and assigning cross-course preparatory modules in specific subjects?

Columbia

Yes. Modules, especially.

Duke

Yes. Developing material that better prepares student to be successful in a course is a priority.

Modules to present intro-level and accelerate progress in a subject.

Michigan

The focus is on other, more institutional aspects: developing, producing, assessing, and assisting

with use of digital education.

MIT

N/A

Yale

Not really. Yale Summer Online are ‘regular’ (synchronous, faculty-taught, credit-bearing)

courses which are simply delivered online. Serve students to advance through initial stages of a

subject or to boost quantitative literacy of incoming students. (Freshman Scholars Program)

Model: on-line video combined with personal contact/coaching.

Stanford

Yes. Active experimentations with pre-freshman teaching. Introducing students to a Stanford-

level learning environment before they get to campus.

Brown

Not at the moment.

7. Do your faculty adopt best practices and avail themselves of the best materials available to them online or do

you/they prefer to develop that material internally? In other words, do you see yourself as a producer, consumer, or

both when it comes to online materials?

Columbia

Both, but with a preference to develop internally.

Duke

Enthusiastically a producer, but not in isolation. Worldwide as well as US partnerships.

Michigan

Definitely producers.

MIT

MIT is a producer of online course material.

Yale

Both. Potentially exporting Yale Coursera courses as digital textbooks for other schools.

Importing lectures streaming from Harvard and having on-the-ground teaching and support staff

on campus.

Stanford

Mostly a content producer, but envisioning to become increasingly a content consumer.

Portability issues should be reduced by open platforms (OpenEdX), and sharing between peer

institutions should increase.

Brown

Producers, though collaborations are an essential part of the long-term strategy.

8. What do you think will be the impact of online learning on the nature of in-class learning in, say, twenty years?

Columbia

Twenty years too far out. In a five-year span, slow buildup to radical changes. Direction: custom-

content and instruction; new ways to assess performance; apply knowledge to new problems.

Duke

Five years max. On campus: more blended learning; more modules to provide further,

supplementary, remedial material; more repurposing of teaching material, more grab-and-go

within peer institutions. Outside: sharing with broader audience, work with global partners, de-

localize teaching (courses jointly developed and jointly taught in various locations worldwide);

create consortia around common courses (in different perspectives, with no central certification).

Michigan

No real good answer. For professional schools online possibly dominant platform, to allow

students more field time.

MIT

N/A.

Yale

Blending pedagogy. Opening the classroom to different students and perspectives. Dialogue with

other institutions. Internationalize.

Stanford

Blurring between in-class and online pedagogy. More integration of technology in the class and

more collaboration outside. Potentially new, more efficient, ways of learning will develop.

Quality crucial in success and survival of courses online. Expansion of online teaching in

continuing education and with alumni. Online courses potentially a meeting (and screening) space

for prospective students.

Brown

Shorter-term forecasting necessary. Trend is blended learning at the college level; online presence

increased in professional school pedagogy. Effort produced new climate at the departmental

level. Deconstructing and modularizing the experience of teaching. Also, a conversation ignited

on how can new pedagogies help students learn new ways to learn. Engage students in the

iterative process of thinking in a discipline.

9. What resources are you providing to faculty to experiment with online learning? What form do these

resources take? Does your university provide incentives to faculty who do such experimentation, such as summer

salary, release from some teaching responsibilities, and so on?

Columbia

Centralized support to faculty: educational technologists, programmers, media producers,

designers. Small grants for experimentation. Local support: some schools provide release time.

Duke

For MOOCs, honorarium, full production team, shared revenue. For digital pedagogy:

consultants, equipment at no charge, IT Office, Digital media services. Department and schools

determine release time and teaching assistants. Dedicated course IT support are provided for first

two instantiations of course.

Michigan

Financial support and infrastructure for research at both central and school or college level.

MIT

The Office of Educational Innovation and Technology (http://oeit.mit.edu) aims to assist faculty in

developing online courses from experimentation to implementation. GS and postdoc fellowships

available for online course development.

Yale

Salaries for Summer Online Course are same as in residential format. For Coursera courses,

education committee evaluates proposals, provost decides which courses to support. Faculty

receive 16,000$ salary and logistical, financial, and production support. No release time.

Stanford

Centralized system: design team, production team (digital media), studios. Graduate School of

Education pursues research on digital pedagogy. 12 seed grants per semester (up to $20,000) to

develop MOOCs.

Brown

Issue: Junior faculty mostly advanced in their incorporation of digital components in the course.

Mid-career and upper level senior are less involved. Time is crucial factor. Institutional

recognition for development and implementation of online courses not yet there. Strategies:

Office of the Provost is investing energy in creating a stimulus package for faculty. Grants for

STEM disciplines have worked very well: engineering, physics, applied math, chemistry. Provost

committed to resourcing the initiatives. Incentives at the central level.

10. Do you have institutional policies guiding online learning? For examples, are there policies on how much of a

course may be online? Any other policies?

Columbia

IP policies and recommendations in place since 2000. Upcoming updating.

Duke

Collaboration with Coursera on policies. Key principle: no fully on-line courses exist at Duke.

Only modules or summer courses are non-residential.

Michigan

No policies in place at the moment.

MIT

The development of policies has been recommended.

Yale

No formal policies. No credit-bearing undergraduate courses are taught solely online.

Stanford

Intellectual property rules and ordinary policies (FERPA, accessibility, compliance, privacy, etc.).

Credit-bearing full online courses exist.

Brown

No formal policies. Guidelines in the Strategic Planning report. Keyword is engagement with

new pedagogies in a holistic fashion.

11. Do you see virtues in partnering with other institutions in this area?

Columbia

Yes, of course.

Duke

Yes, absolutely.

Michigan

Yes, of course. [Partnered with other institutions into forming Unizin (www.unizin.org)]

MIT

Yes. EdX was co-founded with Harvard.

Yale

Collaboration is essential. No need to develop in-house what is available elsewhere. Regular

conversations ongoing between Yale, Harvard, Stanford, Rice, Michigan, U. Washington, U.

Illinois, Penn, and Duke (“G9”).

Stanford

Yes, enthusiastically. Developing tools and sharing them is the key to the endeavor.

Brown

Yes, absolutely. Dialogue with different institutions (from large to small universities, from

professional schools to community colleges) essential to devise high-impact classes that engage

students deeply. Creating those connections is crucial for learning, whatever the field.

Appendix C: Survey Questions and Responses from On-Campus Studies

Students who have taken courses with substantial online components

Appendix C: Survey Questions and Responses from On-Campus Studies

Faculty members who have taught courses with substantial online components

Appendix C: Survey Questions and Responses from On-Campus Studies

All other faculty members

Appendix D: Princeton MOOCs Since 2012

Sessions are offerings of the course that are open for a specific time period. Course materials are

released on a schedule and enrolled students progress through the course as a cohort.

Course Launched Sessions Enrolled

A History of the World since 1300 Sept. 2012 2 180,914

Algorithms, Part 1 Aug. 2012 8 676,472

Algorithms, Part 2 Sept. 2012 5 236,801

Analysis of Algorithms Sept. 2012 5 157,328

Analytic Combinatorics Feb. 2013 5 69,894

Buddhism and Modern Psychology Feb. 2014 2 68,250

Computer Architecture Sept. 2012 3 246,047

Fog Networks and the Internet of Things Mar. 2015 1 21,009

Imagining Other Earths Feb. 2014 3 52,215

Introduction to Sociology June 2012 1 44,721

Networks Illustrated; Principles without Calculus July 2013 3 111,049

Networks: Friends, Money, and Bytes Sept. 2012 5 183,637

Paradoxes of War June 2014 2 67,667

Practical Ethics Mar. 2014 1 38,854

Statistics One Sept. 2012 2 264,447

Effective Altruism June 2015 1 5,886

Software Defined Networking May 2015 1 41,713

Bitcoin and Cryptocurrency Technologies Sept. 2015 1 NA

Global History Lab, Part 1 Sept. 2014 1 6,992

Global History Lab, Part 2 Oct. 2014 1 2,122

Writing Case Studies: Science of Delivery May 2015 1 193

Making Government Work in Hard Places Jan. 2015 1 2,362

Art of Structural Engineering Spring 2016 - -

Reinventing the Piano Spring 2016 - -

2,478,573

Total MOOC Enrollments

Appendix E: MOOC Production by Peers Since 2012

Appendix F: Flipped and Blended Courses at Princeton since 2012

Appendix G: Princeton Online Course Projects by Division