2003 Self Study: Educational Technology

2003 Self Study

Self Study: Educational Effectiveness

Self Study: Educational Technology

Self Study: Student Research

Self Study: Integrative Essay

Self Study: Educational Technology

Preface

This report is designed to be read online or with a web browser handy. If it is printed and read and none of the links are followed, a great deal will be lost. Most readers will want to follow at least a few of the links related to the topics of most interest to them.

A reader may wish to consult Criterion 3.7 of our Capacity Report where there is an overview of information and educational technology at UC Davis and where additional references are available.

Introduction

Educational technology has long been a subject of substantial interest to the UC Davis faculty. Consequently we welcomed the opportunity to use the WASC reaccreditation process to review recent developments in educational technology on our campus. In this study, we discuss examples of both institutional and individual projects. In the cases where there has been some form of evaluation, we document that as well.

The goals are to study our programs that prepare students for the use of information technology and to review the development of educational technology and its impacts on pedagogy. In addition to giving an overview of educational technology developments, we will give particular attention to the projects that include an evaluation of their educational effectiveness.

The first part of this discussion of educational technology will be organized according to a chronology from the perspective of an undergraduate student. The second part takes an institutional perspective.

There have been a large number of innovative and ambitious projects at UC Davis. Some of them have been carefully evaluated. We find that in certain circumstances, an appropriate use of technology is a sound pedagogical option, and when it can be used to enhance the efficient use of classroom space and thereby avoid the expense of a new building, it can be cost effective. However, on balance, one cannot expect dramatic impacts on student learning or costs.

We find that our evaluation of educational effectiveness is still uneven, but that it is beginning, through the Teaching Resources Center and Mediaworks, to become more institutionalized. In the summary, we offer some suggestions for further progress.

Educational Technology Organized by the Chronological Experiences of a Typical Student

In this section, we describe the aspects of educational technology that a typical undergraduate student is likely to encounter during his or her career at UC Davis. It is in chronological order, so there are initial experiences, technologies that are likely to be encountered in lower division classes, a sample of upper division possibilities, and integrative experiences. The results of surveys on student satisfaction with their IT preparation and experiences at UC Davis are included.

Initial student experiences

While many potential UC Davis undergraduate students are very familiar with some aspects of computer use, their first experience with educational technology in the context of the University of California is likely to be at the beginning of the application process for undergraduate admission. The University provides an array of websites to give potential students information and help them decide if the University of California is right for them. The application process itself can be carried out completely online.

Included in that material is the UC Davis expectation for computing hardware and software for incoming students. This is intended to be a baseline computer configuration and student competence that instructors can count on and build upon if it is appropriate for their classes. In this way, the expectations of students and instructors are matched, and a foundation is laid for the use of educational technology in classes.

A new UC Davis student is strongly encouraged to come to summer orientation. At that time, s/he will receive a UC Davis computer account and register online for fall classes. Davis was the first UC campus to codify such an expectation. It was done as a collaboration among faculty, campus Information and Educational Technology (IET) professionals, and student affairs staff. Because of the wording of this expectation, students eligible for financial aid are better able to receive support for the purchase of a computer.

Upon arriving on campus in the fall, a student in the residence halls will find that the room is wired for a high-speed network connection and that there are staff from IET available to help with getting connected. For additional help with configuring a computer for UC Davis, the bookstore sells a CD called Bovine Online for $4.50. It contains setup, connectivity, anti-virus, and other utility software.

There are a number of other ways that students can get computing help:

Having achieved some familiarity with the UC Davis computing environment, a student can use the Student Information System (SISWEB) to enroll in classes, adjust class schedules, view and print class schedules, print unofficial academic records, make address changes, view accounts, view financial aid status, etc.

As a part of orientation to the campus digital infrastructure, it is important for students to become familiar with the library and especially its electronic tools and resources. Information available on the library link shows that it provides a very wide range of instruction, orientations, tours, and tutorials that new students can use to become familiar with the library in general and with the electronic resources in particular. As more collections move to a digital format, the library is participating in the Collection Management Initiative to evaluate the impact of digital vs. print collections.

In first year classes, students will discover that email and class websites are commonly used for some of the communication between instructors and students. The campus has developed the infrastructure to automatically generate an Electronic Mailing List for each class. From the point of view of an instructor, these features are very easy to setup and maintain. It is also very simple for the students, who are added to the list automatically.

An ambitious tool that is under continuous development and that is also suited to enhancing communication for classes is the web portal MyUCDavis. It is a centerpiece of the New Business Architecture (NBA). MyUCDavis has many features that are designed to facilitate student learning through parallel use by students and faculty. At the portal, students will find their records, access to class webpages, online quizzes, grades, email, financial information, a planner, search engines, library services, the Internship and Career Center, and the Degree Navigator.

As students make choices for majors or approach graduation, student advising by faculty and staff can be augmented with Degree Navigator. It is an online advising tool to help students develop a study plan with their adviser and evaluate their progress toward a degree.

Another feature of MyUCDavis is the QuickSurvey, which allows Student Affairs Research and Information (SARI) to do timely, informal surveys of student opinion.

The MyUCDavis project has benefited from extensive, careful planning. See for example: a report to AC4, a report to a joint meeting of the Academic Computing Coordinating Council and the Administrative Computing Coordinating Council, and an IT Times special issue (check the contents link in left sidebar and in particular the article on student involvement in the design).

Technology in the classroom

Lower division

Upon entering a classroom, a student will see that it has a Smart Panel. The student's instructor can connect a laptop computer to it and gain access to a network connection and, in most classrooms and lecture halls, to an installed digital projector.

Some areas of instruction have built up a more specialized and advanced technology environment. For example, an entering student is likely to take an English composition course. Many of those are taught withcomputer aided instruction (CAI). The CAI classrooms are used for some of the lower division writing classes and for a majority of the upper division ones. The class structure and the classroom are set up so that students must use the technology and so that close collaboration among students is facilitated.

For language instruction, a student might be in the Language Learning Center. The IT Times has an interesting related article about Professor Robert Blake's work. He was an "early adopter," and one of his innovative classes is described below.

Many students enroll in the Chemistry 2ABC sequence, which is one of the largest enrollment classes on campus. It is a required course for many majors and is a prerequisite for other required courses. The Chemistry 2ABC courses have associated labs. The Department of Chemistry is developing pre- & post-laboratory activities that students access on-line for Chemistry 2C with the expectation of extension to the entire series. The project was undertaken to improve the students' educational experiences in the laboratory by aiding the students in preparing for the laboratory, by familiarizing the students with the equipment and chemical reactions involved in the experiments, by focusing the students' attention on the principles underlying the chemistry, and by cementing the students' laboratory experiences with a post-laboratory synopsis. The Dean's Office of the College of Letters and Science, Mathematical and Physical Sciences Division and Mediaworks have been instrumental in providing resources to bring the project to fruition.

The pre-laboratory activities are two-fold in nature, a presentation of the exercise to be undertaken in the laboratory of the week and a short quiz to verify the student's comprehension of conceptual material, laboratory procedures, and essential safety practices. Use of a multi-media mode of instruction permits the student to receive this material visually and audibly and is a mechanism for student testing with immediate feedback. The post-laboratory activities guide the student through detailed calculations using the student's personal data, collected during the laboratory class, to arrive at a verified result. This process provides a valuable opportunity to reinforce instruction in the concepts pervading the calculations.

The first full implementation was the 2C labs in spring 2002. Guest access to the pre-labs is available here.

Although only prelab quizzes and postlab exercises are required, the students are using the online prelabs to prepare for the lab as well as the quiz. There will be more description of the broader context of this project below.

Entering freshmen have general education requirements to fulfill. A study funded by a Mellon grant focuses upon introductory general education classes. The purpose of the project is to implement and evaluate the use of information technology in large undergraduate courses. The project includes extensive effectiveness studies, which will be described below.

Upper division

A student's discipline-related computing experiences are highly dependent on his or her choice of major. For the most part, the major-specific computing needs for upper division students are handled at the department level. When financial support for department-based student computing is needed, deans make large, crucial contributions. But generally it is the department that identifies the needs and decides how they should be met. Thus there are a huge number of varied examples. To give some feel for the different approaches, we will briefly discuss three: Environmental Design, Physics, and social sciences.

The Design program has been teaching the use of computer technologies since 1989, when they purchased twelve Mac II computers for their first departmental computer lab. Initially all computer instruction was integrated within individual Visual Communication courses. As the curriculum moved further from traditional graphics methods to more emphasis on computer technologies and as the other emphasis areas in the department began to integrate the use of technologies into their instruction, the need for comprehensive and consistent computer instruction became apparent. After experimenting with various methods of integrating computer instruction into the curriculum, the department developed a new required course for all design students, DES 16: Introduction to Computers in Design. Currently, the Design Program teaches eight courses entirely in computer labs.

There are two main approaches to the teaching of technologies in a classroom setting. In one the focus of instruction is on computer and software proficiency. This is the model used by most community colleges. It can also be useful in one-unit workshops, overseen by graduate students, to bring the students within the "comfort zone" for specific software. The Design Program chose a different model. They teach design concepts on the computer while the students are mastering the computer technologies. For all assignments, the students write concept statements and do drawings before working on the computer. Their belief is that if the students know what they want to create, there is an incentive to figure out a "work-around" rather than settling for the standard techniques determined by menu options. In comparing slides of student projects over the last six years, they have observed a remarkable increase in the sophistication of student work.

Not only has the Design Program devoted FTE to the instruction of computer technologies, but it devotes a good deal of departmental funding for computer technologies. They support an 18-station Macintosh G4 lab with high-end color printing. To serve 300 majors, they keep the lab open 75 hours per week. While most of the computers and software are paid for through Instructional Use of Computers (IUC) funding, the department supports two full-time staff members, and has purchased a $14,000 color plotter with department funds.

Physics majors are required to take both a lower division programming course and an upper division computer lab associated with the junior level mechanics course. This provides a basic skill foundation that can be built upon in other courses. In addition there is a new upper division course: Computational Methods of Mathematical Physics. The department emphasizes programming. This gives students greater depth of understanding and more flexibility in the application of computing to physics.

The department has several faculty members whose research is heavily computational. It recognizes that along with theory and experiment, computational physics has become a third approach to discovery in physics. For students who would like to pursue it more seriously, the department offers an applied physics emphasis in the field.

The Social Sciences Division of the College of Letters and Science has taken an approach that enhances access to computing for students in several departments. To meet the common needs of students in the quantitative areas of the social sciences, Dean Sheffrin started a dedicated computer lab for upper-division social science classes that use specialized software. The lab was designed to offer upper-division students more hands-on experience in computing. This need was revealed in extensive discussions with faculty who are were engaged in quantitative instruction on the campus, particularly with the use of specialized software. To date, the lab has been used by Sociology, Political Science, and Economics. Sociology is perhaps the heaviest user of the lab to date.

In another project, the Institute for Governmental Affairs and the Social Sciences Division collaborated on an Instructional Use of Computing (IUC) project to provide support for faculty use of data in instruction. Activities included investigation of appropriate software and data sources, a new website with information for faculty and students, and an upcoming workshop on the use of spreadsheets in the undergraduate classroom. An initial prototype of some of these activities was a program called WEBSTAT, which allowed students in large classes to do data analysis on the web. It was used by several instructors. This makes it possible for large introductory classes to do data analysis. Continuing development of this project is planned.

The Social Science Data Service (SSDS), a part of IGA, provides computing access for honors undergraduates for their senior projects. Normally SSDS is reserved for graduate students and faculty.

The Division of Social Sciences invested in developing a 24 computer facility in the Department of Psychology. It is used heavily in Psychology upper-division statistics courses. Undergraduates also use this lab in their senior projects.

Psychology also has totally revamped their lower-division methods course, Psychology 41, to make it an interactive web-based course. They did this with the assistance of Mediaworks, and it will prevent a bottleneck for those majors. The course is being offered in its new format this quarter for the first time.

Economics has developed an undergraduate data course. This was developed as an intermediate step between introductory statistics and advanced econometrics.

Finally, Dean Sheffrin encouraged faculty to continue innovating in this area, and departments continue to examine this. As an example, Political Science is now re-thinking its approach to undergraduate opportunities for quantitative work. They are also looking into sources of external funding to increase the scope of their activities.

Integrative work

The idea of encouraging students to produce electronic portfolios is gaining support. UC Davis faculty attending the 2001 Chancellor's Fall Conference had an opportunity to increase their understanding of how other institutions have instituted portfolio projects. As a result of consultation between IET professionals and faculty, a portfolio component for MyUCDavis is being developed. This could be used by advanced students as a capstone experience to demonstrate their cumulative learning in their major and in relation to the Educational Objectives. It could also serve as the basis for a resume.

There are many examples of integrative work by advanced students which display both impressive research accomplishments and excellent use of technology. Integrative work of this type is an indicator of educational effectiveness. Examples are available from the classes of Professor Geoffrey Wandesforde-Smith such as Environmental Science, Policy Making in Natural Resource Agencies, and International Law.
See Awards for Excellence in Research and Writing "Best of the Web" Sites by Students.

Professor Geoffrey Wandesforde-Smith was one of the presenters at the 2001 Summer Institute on Technology in Teaching (SITT).
Teaching with Various Technologies: Moving Beyond "Content Delivery" Across Courses and Subject Matters

Another example of electronic presentation of research by students comes from Professor Jeffrey Mount's capstone course in geology.

Students in the McNair Program produce full portfolios including research reports, CVs, and other material.

A collection of posters (Powerpoint format) that they have developed to present their research results at the Undergraduate Research Conference and at other conferences can be accessed here.

These student presentations are evidence for the effectiveness of our methods in that they demonstrate cumulative student learning, involvement in research, and appropriate use of communication technology.

General observations by instructors

Many instructors have found, through evaluation by student response, that a lecture format with a mix of the traditional and the digital is the most effective. A description of one experience can be read here. An excellent and very experienced instructor also reached the conclusion that a mixed format is optimal.

Evaluation surveys

We have conducted surveys to gauge the effectiveness of our efforts to prepare students for the use of information technology both during their time at UC Davis and after graduation. A 1996 survey of enrolled students showed that the vast majority of students were using computers on a regular basis and that most were satisfied with the internet access and related resources at UCD. Only about a quarter of students felt that their instructors expected them to know more about computing than they did.

The survey of 1999 graduates addressed the question: are UC Davis graduates equipped with the computer skills needed to excel in a technological era? In other words, has UC Davis successfully met the educational needs and goals of undergraduate students in this fundamental area? Half of the June 1999 alumni respondents are satisfied or very satisfied with the use of information technology in instruction, 37% are neutral and 13% are dissatisfied or very dissatisfied. This cohort had been very heavy users of basic computing during their years at UCD, with over half analysing data and using applications specialized to their field of study. 80% felt that UCD had prepared them adequately or better in the area of information technology. 1999 and 1996 graduates rate their preparation slightly better than did 1993 and 1990 graduates.

The 2001 survey of undergraduates contained questions about the use of information technology. The full results allow either general or very specific reports to be generated. There is also a report with more description and commentary. A part of the summary reads:

"There were five items that addressed the use of computers and IT applications to provide or support the delivery of instruction. The survey found students to be well satisfied with both the amount of computer use in teaching and the amount of time they spent using computers. Students were also open-minded about exclusively on-line courses but the opinions of those who had taken on-line courses were split, with about half rating a traditional class as better and half stating that online course were equivalent or better. "On average, students agreed that classes using computers for more than simple tasks did significantly improve instruction. Engineering students and seniors led and HArCS students were least in agreement. The second item in this section asked about the amount of IT use in instruction. In response to this question, the overwhelming majority (76%) said that it was about right. Notably, only 3% said that instructors were using computers too much. A much larger group (21%) wanted to see more. Unlike those items that asked about specific IT applications, this general item found little to no variation by major, class level, sex, or origin (transfer or not). The third item in the series asked students whether they would enroll in a course that was on-line and had no regular class meetings. The responses produced a very flat distribution. Some 28% were negative, 39% were positive and 34% said maybe or did not know what they would do. Only one-in-ten students were definitely opposed, and there were 16% who were definitely inclined to enroll in an exclusively on-line course. The last item in this section asked students with experience in UC Davis on-line courses whether they were better than, equivalent to, or worse than traditional courses. Only 16% claimed experience and their responses were tallied. Of these 16% with on-line course experience, 46% said that the traditional class was better, 33% thought that the two types were equivalent, and 21% reported that the on-line course was better."

Section summary

We have described numerous examples of our programs to prepare students for the use of information technology at multiple levels of sophistication. Some impressive examples of student work are included. In addition, general surveys indicate the success of our efforts. A missing element in the picture is the evaluation of the department-based upper division computing projects. Barriers to serious evaluation are the facts that most faculty probably think that they are self-evidently beneficial and that these efforts are either new or very rapidly changing and present a difficult target for evaluation. It will be challenging to evaluate these diverse and decentralized programs in a way that is meaningful and cost effective.

Support for faculty

The campus offers many opportunities for faculty to prepare for the use of educational technology. One of the most innovative, long-running, and successful is the Summer Institute on Technology in Teaching(SITT), which is run by the Teaching Resources Center (TRC). In July 2002, the 9th annual session was presented. The Summer Institute has been very successful in providing faculty members an opportunity to efficiently learn how to use new technology in their classes. The demand from faculty for the week-long institute was so great that the TRC doubled to 70 the number of people that it can accommodate. There are now several hundred faculty who have participated. This summer, programmers from the MyUCDavis team attended sessions so that the IET efforts on the MyUCDavis portal will stay coordinated with the needs of faculty. Over the years, a real camaraderie has developed among the faculty who participate in SITT. The organizers use past faculty participants to train new classes of faculty. Faculty evaluations of the Institute show a great appreciation for the emphasis on teaching evident throughout the week-long workshop.

A recent addition to MyUCDavis that is a response to faculty needs is the quiz builder, which is explained on the MyUCDavis tutorials page. In fact, the flip side to the student view of MyUCDavis shows the many tools it has to help faculty with class email, websites, and recording and submitting grades.

Another major resource for faculty is Mediaworks. Some background on the formation of this unit can be found in the Summary associated with Criterion 3.7 of our Institutional Portfolio. Mediaworks provides a range of educational technology support for instructors. It includes things as varied as the partners program and full-blown multimedia development. In this way, a faculty member can move to any technical level that makes sense for his or her instructional purposes.

Educational technology projects at Mediaworks are primarily for regularly scheduled undergraduate, graduate, and professional courses, with an emphasis on large undergraduate courses that are experiencing enrollment pressure. The Educational Technology group has expertise in multimedia and computer programming. Members of this group serve as consultants and producers for instructors who want to use media to present their course content in compelling and engaging ways.

Since UC Davis aspires to be a community of learners whose members learn from one another in a reciprocal fashion, it is fitting that the ET Partners program harnesses the time and expertise of students to help faculty members learn about and use educational technology effectively. Students bring both a familiarity with technology and an expectation that their education will be enriched by the appropriate use of modern communication tools.

The ET Partners program, part of IET at Mediaworks, hires, trains, and manages the student partners, and selects the faculty partners. The primary aim of this program is to use the concept of the community of learners so faculty members will use educational technology effectively and easily. Mediaworks is evaluating the effectiveness of the ET Partners program with questionnaires on computer attitudes and skills as well as focus groups on the impact of the program on both student and faculty partners.

In an effort to make sure that faculty are aware of Mediaworks services, an openhouse was held recently.

As an example of the careful institutional planning discussed in Standard 4 of our capacity report, these projects are coordinated closely with the Teaching Resources Center, a responsibility of the Vice-Provost-Undergraduate Studies, and the Classroom Technology Services, which is part of Information and Educational Technology. Classroom Technology Services works closely with the Registrar and the Instructional Space Advisory Group. The Vice Provost for Undergraduate Studies, the Vice Provost for Information and Educational Technology, and the directors of Mediaworks and the TRC meet regularly.

Yet another important resource for faculty is the Arbor, which provides support to faculty in the use of instructional technology to improve the quality of learning at UC Davis. Examples can be seen in theirshowcase.

The TRC also offers a great deal of teaching and technology support:

A comprehensive reference is the faculty technology guide.

Thus there is a strong institutional commitment to supporting faculty in their use of educational technology.

Institutional perspective

For an institutional perspective, it is instructive to outline the evolution of educational technology at UC Davis. In this part, we include references to evaluations of the educational effectiveness of the technology.

The impacts of educational technology on pedagogy are determined largely by faculty adoption of the new methods. Faculty attitudes to new technology cover the entire spectrum. Many faculty members were quick to embrace the possibilities of the emerging technology and experimented with its use in their classes. These were primarily dedicated and enthusiastic innovators. They operated mostly individually, i.e. not as part of an institutional initiative. However, the campus has encouraged and supported this adoption with funding opportunities through the TRC and Instructional Use of Computers grants. It would be difficult to identify most of these projects. Among the most ambitious early projects were those of Professors Matthews, Falk, Blake, and Walters.

Professor Harry Matthews taught a molecular biology course for first year students in the medical school. Over a period of several years in the late 1990s, he moved an increasing amount of the lecture content into electronic form. This included animations to illustrate key concepts. With much of the course content available via the web or CD, there was more time for interaction between the professor and small groups of students. This successful experiment is described in several articles:

The one line summary of these effectiveness results is that student performance improved, and their opinions of the course were positive.

Professor Richard Falk taught an introductory biology class for nonscience majors. The discussion sections became email listserver-based virtual discussion sections with topics drawn from current events. They included a writing component. The lecture content was made available on the web. It was restructured to include links to related sites, animations, and self-test questions. This was a complete replacement for the lectures. However, the lectures were still given, and a minority of the students chose to attend them. Evaluation based on student surveys, student work, and grades is also described in links 2, 3, and 4 above. Most of the students liked the web-based material. The virtual discussion sections were successful in producing much more student involvement and interest in biology and its relationship to issues of interest to them. The quality and depth of the discussion were also enhanced. In this electronic introductory biology course, student performance was comparable to that in the traditionally delivered course, and again student acceptance was very good.

Professor Richard Walters undertook an extremely ambitious project to develop the Remote Collaboration Tool (RCT ne RTA). This is a large software suite, which brings many of the benefits of face-to-face interaction to interactions at a distance through multimedia synchronous and asynchronous electronic communication. This tool was used successfully by Professor Robert Blake in introductory Spanish language courses. It was also used by Professor Walters in an online independent study version of an introductory computing class. Students in the online version were satisfied with the course and performed as well as those in the parallel conventional version. For more information, see the Remote Collaboration Tool website and the list of papers there, e.g.

The students who used this tool for synchronous pairwise discussions showed improvement in their language proficiency comparable to the control group.

In parallel with these innovative projects, there were two institutional efforts. SITT already has been described. The Learning Environment Architecture Development (LEAD) project was an assessment to help identify the tools, architecture, and infrastructure for the learning environment at UC Davis. The final report was published in early 2000. Among the projects that followed from this is the web portal MyUCDavis described above.

The second phase of the development of educational technology at UC Davis includes a more institutionalized approach. The central elements are Mediaworks and the Mellon Project. The courses of Professors Falk and Matthews mentioned above became the pilots for the ten general education courses for full study in the Mellon Project. The aim of the project is to implement and evaluate the use of information technology in large undergraduate general education courses. The evaluation includes both educational effectiveness and cost. The first course to receive a full evaluation is Food Science 2. The papers are

The results from these studies were mixed. After accounting for lecture hall space, the online version of the course is less expensive per student than the traditional version. The students who relied mostly on the web material rather than going to lecture did slightly worse on the exams. Since there are many uncontrolled variables in this study, it is difficult to know whether a random assignment of students to groups rather than the self-selection would produce the same results.

In a very recent and more comprehensive study, six of the Mellon classes have now been analyzed. A draft report and its tables and figures are available. This analysis for six courses reveals that in terms of student performance and student preference, a traditional course is favored over an online course. However, the differences are very small. For a more complete and nuanced view, the reader is strongly encouraged to examine this careful report at the two links above.

A course that is not part of the Mellon Project was also recently evaluated. It is a virtual lab section for a large introductory psychology course on research methods. In this case, the students responded favorably to the format, and their exam scores were no different from those in the traditional lab sections. See A virtual lab in research methods.

Another case is the online laboratories that were developed for the large enrollment course Application of Microcomputers in Agriculture. In this case, the evaluation showed the same performance for the online group and the traditional group.

The third phase in the deployment of educational technology at UC Davis involves projects in a strategic choice of classes. One example is the introductory chemistry sequence, Chemistry 2ABC, which was mentioned and for which links were given above. The availability of rooms for the laboratory sections is a major limiting factor on the number of students who can be accommodated in the courses. The construction of more lab space is a very slow and expensive process. However, a significant fraction of lab time is spent on pre- and post-lab discussions that do not actually require the physical facilities of the laboratories. So that more students can be accommodated in the present physical facilities, the pre- and post-lab discussions are going online. This means that the lab rooms will be used more efficiently. Another motivation is to provide a more uniform pre-lab student preparation and a more uniform lab experience for the students that is less dependent upon individual lab TA styles. In particular, the quality and uniformity of safety instruction is enhanced greatly.

This project has been a collaboration of the Department of Chemistry, Mediaworks, and the Division of Mathematical and Physical Sciences (MPS) Dean's office. In the Chemistry Department, William Fink and Dianne Meador provide the chemistry content, the lab setups, and the descriptions. Mediaworks does the programming and comes over to record the video and audio. Moby (Coldfusion and Flash) are used to present the web pages.

The spring 2002 offering has been evaluated. A Powerpoint presentation is available. A new edition of the student evaluation form has just been released to this quarter's students. Also the TAs report that the students are coming to lab better prepared to do the experiments. To date, all indications are that the project is working well.

A number of other projects indicate the increasing commitment of the campus to educational technology and its evaluation. Most of the general assignment classrooms have been updated to the Smart Panels for digital display and internet connections. Mediaworks, the new technical production, consulting, and support unit for educational technology was established. The Teaching Resources Center, which provides support for many aspects of teaching and learning at UC Davis, has increased its commitment to educational technology. Three years ago, they hired a director for technological programs. They plan to add further strength in instructional design and project evaluation. They have also increased their commitment to technology by including more software and hardware for consideration in IUC grants.

Among our standard review procedures are program reviews and student evaluations of teaching. For both of these, we are considering ways in which the evaluation of the educational effectiveness of technology could play a bigger role. Course evaluations are administered at the department level with the approval of individual faculty for their courses. Thus it may be sometime before such questions are widely used. One possible approach is to encourage SITT participants to include specific questions about the effectiveness their new endeavors. The guidelines for program review are being revised. It may be possible to include in them further encouragement for evaluating the effectiveness of educational technology.

In 1998 the Academic Computing Coordinating Council (AC4) was formed. Its mission statement reads: "The Academic Computing Coordinating Council reviews proposals and plans that seek to promote the use of information technology in instruction, research, and public service at UC Davis and makes appropriate recommendations to the Provost and Executive Vice Chancellor. The Council may also recommend its own proposals and plans. The Council's broad representation assists the campus administration in ensuring that campus resources are deployed to their most strategic advantage. The Council recommends academic computing-related policy to the Information Technologies Policy Board. Policy interpretation and financial decision impasses are also referred to the Policy Board." AC4 has subcommittees for education and course management systems. The institutional context for AC4 and some of the developments that led to its formation are described in Criterion 3.7 of our capacity report.

A very exciting new program is Technocultural Studies. It grew out of the ArtsVision Initiative. Students in the major will develop interdisciplinary skills that include the use of software technology for digital media arts. While a program that does not yet have any students cannot very well have a large evaluation component, it should be noted that the plans for student learning of technology are based on the past successful use of those techniques. We anticipate that this hybridization of the "two cultures" will bear abundant fruit and will have a major impact on the understanding and use of educational technology at UC Davis.

Summary

This description of the development and use of educational technology at UC Davis reveals a careful approach. Most of the development for classroom use has been at the initiative of individual faculty members or small groups. From that perspective, the campus has taken a bottom up approach. However, in its direct support for faculty efforts by TRC and Mediaworks and in its indirect support via infrastructure, the campus has an organized, strategic, and timely approach.

The evaluation of educational effectiveness has been variable. Individual faculty members tend to be guided by student evaluations and gut feelings. Serious studies of educational effectiveness are not usually done by individual faculty members. However, if evidence emerges that using such evaluations results in better student outcomes, then UC Davis faculty members will be likely to embrace the proven assessment tools. On the other hand, the larger projects, e.g. the Mellon Grant and the Chemistry online pre- and post-labs, are already being seriously evaluated for their effectiveness.

In those cases where a careful evaluation has been done, the results indicate that student learning, student satisfaction, and costs are not dramatically affected by the use of technology. In cases where new technology is mixed with traditional techniques, student acceptance is good. In cases where the use of technology can avoid the construction of a new building, it can be justified on a cost basis.

A more uniform approach to the evaluation of educational technology might include the following elements:

Faculty who use significant educational technology should be encouraged to include specific questions about its effectiveness on student class evaluations.
Departments should be encouraged to discuss the use and effectiveness of educational effectiveness in program reviews.
Mediaworks and the TRC should coordinate in increasing their commitment to instructional design and project evaluation. They should develop standard evaluation tools that could be used with the projects they fund. In addition, they should create readily accessible and publicized descriptions of case histories and best practices from UC Davis and other institutions.