Impacts of college-level courses via Asynchronous Learning Networks:
Focus on Students

Starr Roxanne Hiltz, NJIT
Sloan Conference on Asynchronous Learning Networks, Philadelphia, October 1995
DRAFT, October 3
Copyright, 1995


Abstract

New Jersey Institute of Technology has been delivering college courses via an ALN system called the Virtual Classroom[TM] for a decade, using various media mixes. Currently, two complete undergraduate degree programs are available, via a mix of video plus Virtual Classroom, the B.A. in Information Systems and the B.S. in Computer Science. This paper presents preliminary findings about impacts on students, and touches on some issues and potential impacts on faculty, individual universities, and the structure of higher education. Overall ratings of courses by students who complete Asynchronous Learning Network (ALN) based courses are equal to or superior to those for traditional courses. Dropout or Incomplete outcomes may be somewhat more prevalent, while grade distributions for those who complete tend to be similar to or better than those for traditional courses. For both students and faculty, more startup time devoted to solving the "logistics" of ALN delivery seems to be required at the beginning of courses, perhaps akin to encountering traffic jams and being unable to find a parking space, on the way to traditional courses. ALN delivery is not just a "different" way of doing the same thing, however; it is likely to change the nature and structure of higher education.

1. Introduction

The sophistication and flexibility of software structures for supporting distance education vary widely, from simple electronic mail systems to conferencing systems that have been specially enhanced to support classroom-like experiences, particularly group discussions and joint projects. This paper focuses on what we have learned about teaching techniques that are effective online as a result of a decade of experiences at New Jersey Institute of Technology (NJIT) using versions of a computer conferencing system enhanced with features to support collaborative learning, called the Virtual ClassroomTM. However, most of the themes emphasized are echoed in reports on similar projects (Mason & Kaye, 1989; Harasim, 1990; Harasim et. al. 1995).

Generically, the Virtual Classroom is a teaching and learning environment located within a computer-mediated communication system. Rather than being built of steel and concrete, it consists of a set of group communication and work "spaces" and facilities that are constructed in software. Thus it is a "virtual" facility for interaction among the members of a class, rather than a physical space. Specifically, the Virtual ClassroomTM is NJIT's trademarked name for versions of its Electronic Information Exchange System (EIES2) with special software structures designed to support collaborative learning, including structures to force active participation, to allocate unique assignment topics, and an electronic gradebook (Hiltz, 1986, 1994; Turoff and Hiltz, in press). Participation is generally asynchronous; that is, the Virtual Classroom participants may dial in at any time around the clock, and from any location in the world accessible by the Internet or a reliable telephone system. The fact that the educational process is asynchronous means each student may engage in more reflective thinking before having to answer or discuss issues, and it also means that student can participate at their own convenience, and thus better fit the demands of a college degree program into busy lives.

It should also be noted that a Virtual Classroom type of environment can be used successfully in many different media mixes:

One cannot segregate the "on-campus" and "distance" markets for ALN courses. If students who take most of their courses on campus are permitted to, they will choose to take a significant portion of their courses via a "distance" mode such as Virtual Classroom. This is because they experience scheduling conflicts with other courses, their jobs, or their family obligations, which mean that they either must take a "distance" course, or take longer to complete their degrees. In the fall of 1995, for instance, about half of the students enrolled in the Virtual Classroom + Video undergraduate courses are taking the majority of their degree courses on campus, and half are "distance" students who are taking the majority of their courses in distance modes. There is also a synergy; if there are two sections, one face to face + VC and the other video + VC, they can be combined in the same online conferences and activities, which seems to motivate students in both sections.

1.1 Collaborative Learning

It does not "work" to simply make an ALN available and tell students that they can use it to ask questions about the readings or discuss aspects of the course at any time. If it is not a "required" and graded, integral part of the course, the majority of the students will never use it at all; and those who start to use it, will generally decide that "nothing is going on there" and will stop using it. The way in which ALN's can best enrich distance delivery, is to serve as the means to create the feeling of a true "class" or group of people who are learning together, and to structure and support a carefully planned series of collaborative learning activities which constitute the assignments for the course.

Collaborative learning is defined as a learning process that emphasizes group or cooperative efforts among faculty and students. It stresses active participation and interaction on the part of both students and instructors. Knowledge is viewed as a social construct, and therefore the educational process is facilitated by social interaction in an environment that facilitates peer interaction, evaluation and cooperation (Bouton & Garth, 1983; Bruffee, 1984; Johnson, 1981; Johnson & Johnson, 1975). A variety of social-psychological mechanisms make collaborative learning effective, including "self-explanation," "internalization," and "appropriation" (Dillenberg & Schneider, 1995).

The Virtual Classroom is an environment that was designed to facilitate collaborative learning -- among students, between students and instructors, among teachers, and between a class and wider academic and nonacademic communities. It also supports independent learning and generative, active learning techniques that are self-paced by each participant. For distance education students, the increased ability to be in constant communication with other learners is obvious. But even for campus-based courses, the technology provides a means for a rich, collaborative learning environment that exceeds the traditional classroom in its ability to "connect" students and course materials on a round-the-clock basis.

It is important to establish collaborative learning through substantive contributions by students to the class discussion from the very beginning of a course. However, with distance students, there are often problems in obtaining the books, videotapes, or other materials by the first week of the course. (Students are told to order the materials ahead of time, but they often do not get around to it until the day before the course is supposed to start!) Therefore, the first week's assignment, in particular, should draw upon the students' own experiences and general knowledge, rather than requiring the reading and synthesis of specific assigned materials.

An example of a collaborative learning strategy applied in the VC that is included in most courses is the "seminar" type of interchange in which the students become the teachers. Individuals or small groups of students are responsible for making a selection of a topic; reading material not assigned to the rest of the class; preparing a written summary for the class of the most important ideas in the material; and leading a discussion on the topic or material for which they are responsible.

Seminar-style presentations and discussions are thus an example of a collaborative learning activity that is often difficult in the Traditional Classroom (TC), but which tends to work very well in the Virtual Classroom environment, even with fairly large classes of undergraduates. Another example is to assign students to identify key concepts or skills in each module of the course, make up a question suitable for an exam to test mastery of this material, and answer each other's questions. Exams then actually include selections from the student-generated questions. Students are thus made partners in deciding what it is that is important to know related to course topics, and summarizing this key knowledge. Other examples of collaborative learning strategy in the VC include debates, group projects, case study discussions, simulation and role-playing exercises, "ask an expert," sharing of solutions to homework problems, and collaborative composition of essays, stories, or research plans. (See Harasim et. al. 1995, for descriptions and examples of these collaborative learning strategies).

2. EXPERIENCES WITH ALN'S AT NJIT

Our first extensive experimentation with the Virtual Classroom began in 1986 and involved the comparison of a large number of courses over a period of two years in many different disciplines which used this medium of communication, including undergraduate courses in Sociology, Communication, English Composition, Management, Computer Science, and Statistics. For some of the courses, there was a "matched" section of the same course offered by the same instructor in a traditional classroom and using the Virtual Classroom (as the sole means of delivery, or in combination with a reduced number of face-to-face meetings). For other courses, there was no "match," and the comparison was subjectively made by the students and instructors to previous, traditional courses.

For this study and the projects which have followed, we used a "multi-method" approach to evaluation. This includes pre and post-course questionnaires completed by students, direct observation of online activities, interviews with selected students, comparison of test or course grades or other "objective" measures of performance, and regular reports by faculty, which follow a common outline. The results presented here will rely primarily on the post-course questionnaires. These were administered to all the Video+ VC sections, plus sections of the same course taught by the same teacher(s) in a different mode; either face to face plus VC, completely face to face, or video only.

2.1. Virtual Classroom + Video: The Degree Program

This project was designed to develop, offer, and assess the effectiveness of an undergraduate major in Information Systems delivered via Virtual Classroom plus video, to attain five objectives:

Some of the courses incorporating video for "lectures" use standard public television courses, such as "Discovering Psychology," produced by PBS. Most video segments are filmed by NJIT in its "candid classroom" and then distributed to remote students on videotape, or via broadcast on a cable channel or satellite. In all video variations, the Virtual Classroom is used for all assignments and additional discussions.

As of the fall of 1995, we are starting the third of the three years of the supported project. We have completed production of all of the major courses for the BA in Information Systems, and will complete production of the courses needed for the B.S. in Computer Science by the fall of 1996. Enrollments have shown a fairly steady upwards trend, aided by a substantial investment in recruiting efforts (See Figure 1).

2.2. Access: Overcoming Logistical Problems

The majority of NJIT's students are "first generation" college students, who must work while attending school and who commute rather than live on campus. They are faced with overcrowded classes with sections that fill up early and then become closed (due to budget cuts that laid off staff and raised class sizes). They are also faced with degree programs that because of their technological nature, have extensive sequences of courses that build on one another and must be taken in order. If a student is closed out of a course one semester or must withdraw, he or she might lose a whole semester waiting for another opportunity to take this course which is a pre-requisite for subsequent courses.

In addition, many of the courses are offered more slowly than would be possible for the better prepared or more motivated students to proceed. The "mass lectures" that have emerged as a result of budget cuts tend to slow down to the pace of the "lowest common denominator" and frustrate the better students. Both video and VC allow "fast forward" and "replay" self-pacing.

By using a combination of video (tape and/or broadcast) plus Virtual Classroom to deliver courses, students may more easily fit them into busy lives as employees and family members as well as students. Since the vast majority of NJIT students work 20 hours a week or more, and many work 40 hours a week or more (in order to pay their own educational expenses and/or to help support their families) it is very difficult for them to fit courses they need to graduate into their schedules. By offering distance education sections, scheduling difficulties can be alleviated. Students can graduate in fewer calendar years because they can more easily fit in all the courses they want.

In addition to the asynchronous nature of the VC helping to overcome course enrollment and completion difficulties, we are trying several innovations that may better facilitate self-pacing and improved rates of progress towards the degree, while maintaining or enhancing quality. These innovations include double courses offered during the same semester, independent study opportunities during the summer, and "late start" courses.

Many of the courses in the CIS majors form a series of prerequisites and must be taken in order. For example, the calculus perquisites must be taken before the first introductory course(113), which must be taken before 114; which serves as a pre-requisite to more advanced courses. Students can lose many semesters trying to get through this rigid sequence of courses, one at a time, meanwhile being closed out of some of the courses in the sequence some semesters.

We have redesigned a version of the introductory courses in this series of pre-requisites as an integrated "double course" in the Spring semester, for those students who were making up pre-requisites in the fall and were unable to start their major. It covers 113 at twice the normal pace, devoting twice the normal number of hours to this task. By the middle of the semester, some of these "fast track" students may decide that they need a full semester after all; they drop into a slower paced group online. Those who can learn at the accelerated pace complete the first course in the first half of the semester, and then go on to take the follow on course during the second half of the semester. The participating faculty members coordinate their efforts to provide a "seamless transition" between the two courses. We have found that many students are able to learn at the increased pace because of the intensive interaction and the support available online. Accelerated (7 week) versions of many of the ALN courses are also offered during the summer, for students who need to "make up" a missing course. Every time students successfully complete an accelerated course, they can conceivably cut an entire semester off of the number of semesters needed to finish their degree program.

3. Hypotheses and Findings on Student Impacts: Interim Results

The most basic hypotheses in our initial study about impacts on students are listed below; all were supported (see Hiltz, 1994, for complete data and analysis). They are also guiding some of the data analysis for the current Video + VC project; some interim results will be presented below.

H1: Mastery of course material in the Virtual Classroom will be equal or superior to that in the traditional classroom.

H2: VC students will report higher subjective satisfaction with the VC than the TC on a number of dimensions, including improved overall quality, whereby the student assesses the experience as being "better" than the TC in some way, involving learning more on the whole.

H3: Those students who experience "group learning" in the virtual classroom are most likely to judge the outcomes of online courses to be superior to the outcomes of traditional courses.

3.1. Problems reported by Students

On the post-course questionnaire, students in all modes were asked a series of questions about the course, the instructor, and their experiences. The relative distribution of responses allows us to compare different delivery modes indirectly. The face to face and video only mode were combined into "no VC" for these analyses, because of the low number of video only responses. Most of the items showed no significant differences among modes; the differences in course and instructor ratings are evidently more affected by factors other than mode.

More students using the VC experienced logistical problems than should be true. For example, about 13% of the video + VC students indicated that access to a PC was a serious problem for them (using response 1 or 2 on a 7 point scale, where 1 was "serious problem" and 7 was "not a problem"; see Figure 2). These students should not have signed up for a course which required PC access. All registration materials, for instance, note that access to a "PC and modem are required." It seems that students are oblivious to information they do not want to see, or do not bother to look at the printed course descriptions before self-registering into any open section.

The most severe access problem is that 40% to 50% of students using VC at NJIT experience serious problems with busy signals when they try to dial in (see Figure 3). The university does not seem to be able to provide enough modem access; there were 48 dial-in ports to campus computing systems last year. This year, another 48 ports were added, but within a week, they were saturated, too. One solution would be for students to subscribe to an Internet service for $15. a month or so, and come in through the Internet, rather than direct dial; however, they are very resistant to spending this additional money.

In terms of disadvantages of learning networks as compared to traditional courses, there are two items on the post-course questionnaire that are worthy of note. Whereas the majority of students meeting in face to face courses develop new friendships in class, only a third of the VC + Video students report doing so (see Figure 4). It is good to see that a substantial proportion can make new friends via a written mode of communication, but it is obvious that close personal relationships are more difficult. In addition, responding to a different format of question in which VC students were asked to compare their experiences to those they had in traditional courses, VC students report that they are more likely to stop "attending class" when they are busy with other things (see Figure 5). Since the class does not meet at any particular time, it is easy to "postpone" it, and this procrastination all too easily turns into falling seriously behind.

3.2 Objective Assessments: Relative Frequency Distributions for Different Modes

In looking at how interesting the course content was, the differences among delivery modes are not significant (p=.21), but those in the experimental Video + VC mode were most likely to strongly agree that the course content was interesting (Figure 6). Students in a mode using VC were slightly more likely to strongly agree that they "participated actively in the class discussion" (p=.06; Figure 7). This is a welcome result, since one of the objectives of the Virtual Classroom is to encourage/force active participation. On the overall course rating, the VC + video students are slightly more likely to rate it as either excellent or very good (p= .10; Figure 8). A few items yielded statistically significant differences, but nothing very striking. For example, for the question on whether the course was a "waste of time," the Video + VC mode students were more likely to strongly disagree (Figure ; p= .001; Figure >).

A final section of the post-course questionnaire was administered only to those sections using the Virtual Classroom. The instructions to students were to compare their experiences in the course they had just completed, to their experiences in traditional, face to face college courses. The results of these items are presented in graphical form to show the distributions visually. The vast majority (71%) feel that ALN's provide better access to their professor (Figure 10). Despite the reported problems of some in reaching the computer facilities, over 2/3 (69%; see Figure 11) feel that the Virtual Classroom is, overall, "more convenient." Only 15% did not "feel more involved in taking an active part" in the course (Figure 12).

The results indicate that "collaborative learning" did take place and did tend to have its intended motivational and learning consequences. For example, 55% felt more motivated to work hard on their assignments because other students would be reading them (Figure 13), and only 9% disagreed that reading the assignments of other students was useful (Figure 14).

Because of the convenient daily access to the class interaction, and the need and motivation to keep up with what their instructor and classmates are contributing, students (as well as faculty) tend to work harder in ALN based courses. For our interim results, the item on total amount of work is phrased as "I didn't have to work as hard for the online class," and the vast majority (67%) disagree with this, while only 13% agree (Figure 15).

One overall measure of student satisfaction is whether students would choose to take another course using an ALN. Only 20% of the students agreed with a statement that they would not do so; 58% indicated that they would (Figure 16). This is a measure which undoubtedly would improve if logistical problems of reaching the network were lessened. In terms of the total amount that they learned, the modal response is that students are not sure whether they learned more or not than they would have in a traditional course; but 40% felt that they had learned more, where as 21% felt that they had not (Figure 17).

The final overall measures of subjective perceptions of the quality of the educational experience are phrased in terms of one to seven Likert-type scale items i which the polar points are 1="definitely yes" and 7= "defininitely not." The modal response on whether the Virtual Classroom increased the efficiency of educational delivery is that students are not sure; but 47% feel that it does, as compared to 23% who feel that it does not (Figure 18). In terms of whether the VC "increased the quality of education," the students are more sure; 58% say yes, while 20% say no (Figure 19).

3.3 Outcomes: Grades and Completion rates

So far, the results support the hypothesis of equal or better performance. Some preliminary results for all courses showed no significant differences, thought here is a suggestion that the withdrawal rate may be higher than for traditional FtF sections. However, one really needs to look within each course in order to reach more definitive conclusions on this matter, since otherwise mode of delivery is confounded by differences in the level of difficulty and grading tendencies of different courses and instructors. Figure 20 shows the current results for the first introductory course in CIS, the various sections of which are taught by the husband and wife team of Fadi and Maura Deek, who follow exactly the same syllabus, grading policies, etc. Once again, as for the combined results, there are no statistically significant differences thus far. However, there is a strong suggestion that Video + VC is much more effective than video alone (so much so that we do not intend to offer any more video only sections, which may be educationally sound but will prevent us getting a large enough "N" to reach statistical significance on these conclusions!) There is also a suggestion that a larger proportion of students do well (earning an A or B) in Video + VC sections than in traditional sections (50% vs. 31%).

4. Impacts on Faculty

The following are a few preliminary impressions on impacts on faculty, gathered from completed course reports and discussions in our online "faculty lounge" where the issue of reporting and discussing perceived impacts has been raised.

  1. Preparing the video portions is a LOT of work. One spends perhaps 10 hours to prepare the materials for two hours of taping, plus review or rehearsal ahead of time.
  2. Considering the VC portion and the delivery, once the videotape is "in the can"-- The first time you prepare the materials for weekly modules and moderate the conferences, just this part is a LOT more work than just delivering lectures FtF.
  3. Once the course has had its basic VC delivery materials developed and you have done it once and gained some experience:
If the class is say 25 or less, the actual amount of work to conduct it is about the same as for a traditional class.

The amount of work is directly proportional to the number of students, since there is no limit to the amount of time each student can ask questions, etc.

For more than about 30 students, class conferences have to be divided to be manageable, thus increasing the faculty workload by an order of magnitude.

There are also qualitative differences in the way a faculty member functions. For example, at the beginning of the semester, the instructor feels a bit like a "cybernetic cowboy," trying to herd all the students into the class corral. Willy Rossak, in the second week of a VC + Video course, says:

Well, at the moment I feel I am more busy to try to round up my students instead of discussing the course topic. It seems the instructor becomes system administrator for the conference, head-hunter, and only then can be a lecturer.

Grading becomes a logistical problem Julian Scher says:

One unexpected impact was for the mentor to organize himself/herself for the grading process of homeworks/projects, much different than in the FtF mode. In the traditional class, I walk out of a class with a "batch" of Projects/homeworks to grade, with an obvious batch grading. In video+vc, I gave students the option of submitting projects through e-mail, regular mail or in person. The completed homework arrived at all different times and days. And I would feel guilty grading on the "first arrived-first graded" basis due to the desire to maintain consistency.

Looking to the Future

When most students have multi-media work stations, the awkward logistics of tape distribution or student recording of broadcasts will probably be supplanted with CD-ROM based or World Wide Web-based digitized video modules. However, for the short term, the VC + video mix seems to work well for a wide variety of courses.

The future of this technology is tied to overcoming some of the difficulties related to the current situation of budget cutting and increased course loads for faculty in higher education. The first difficulty is the initial burden placed upon instructors to completely rethink the nature of their courses and adapt their teaching to a facilitative role. It is also necessary to provide some training for faculty on how to utilize collaborative learning approaches. There is also an initial workload in terms of creating materials in electronic form that is quite large the first time one teaches utilizing this medium. Faculty may be far slower to change and adopt this technology than students; therefore, one has to consider the incentives to do so in the particular educational institution.

Institutions may initially think that ALN based courses are a "cheap" way to deliver education. If done right, with full time faculty conducting courses, it is actually more expensive. There is much to be done in discovering which tasks can be offloaded from faculty to lower priced teaching assistants, without substantially decreasing the quality of the product delivered.

In long run, ALN's will increase the competition for students and probably lead to a decrease in the total number of colleges and universities in the U.S.. Large institutions and consortia of smaller institutions will develop programs that students anywhere can enroll in, and they will choose this rather than enrolling in a "second rate" local college. ALN's will also contribute to the change in the role of colleges and universities, from being primarily residential institutions to take care of 18-21 year olds while they finish growing up, to being centers for a variety of degree programs designed to support students of all ages.

Acknowledgments

Development and research on the B.A.I.S. degree via a combination of video and VC is supported by the Alfred P. Sloan Foundation. Continuing research on appropriate software structures for collaborative learning via computer is partially supported by a grant from the National Science Foundation (NSF-IRI-9015236).

References

Bouton, C., & Garth, R. Y. (1983). Learning in Groups. (New Directions in Teaching and Learning, No. 14). San Francisco: Jossey-Bass.

Bruffee, K. A. (1984). Background and history to collaborative learning in American colleges. College English, 46 (7), 635-652.

Dillenberg, P. & Schneider, D (1995). Collabortive Learning in the Internet. Proceedings, Fourth Int. Conference on Computer Assisted Instruction, Taiwan, S10-6 to S10-13.

Harasim, L. Ed. (1990). Online education: Perspectives on a new medium. New York: Prager/Greenwood.

Harasim, L., Hiltz, S. R., Teles, L., & Turoff, M. (1995). Learning Networks: A Field Guide to Teaching and Learning Online. Cambridge MA: MIT Press.

Hiltz, S. R. (1986). The virtual classroom: Using computer-mediated communication for university teaching. Journal of Communication, 36 (2), 95-104.

Hiltz, S. R. (1994). The Virtual Classroom: Learning Without Limits via Computer Networks. Norwood NJ: Ablex Publishing Corp., Human-computer Interaction Series.

Johnson, David W. (January, 1981). Student-student interaction: The neglected variable in education. Educational Research (pp. 5-10).

Johnson, D. W., & Johnson, R. T. (1975). Learning together and alone: Cooperation, competition, and individualization. Englewood Cliffs, NJ: Prentice Hall.

Mason, R., & Kaye, A. (1989). Mindweave: Communication, computers and distance education. Oxford: Pergamon Press.

Turoff, Murray and Hiltz, S.R. Designing and evaluating a Virtual Classroom. J. Information Technology for Teacher Education, 4,2, 1995 (in press).