STUDENT OPINION OF TEACHING

1. Please take the completion of this instrument seriously. The data collected from this process will be returned to the faculty member after the completion of the semester for the purposes of reflection and personal development.
   
   
2. On the Narrative Response Form, copy the computer number, the name of the instructor, the course and section number, and the semester and year from the board/overhead and place this information in the boxes where indicated on the forms. On this form, you are asked to make comments concerning various aspects of your experience in this course. Please take some time to answer these items as they provide me with qualitative information regarding the course and my instruction.
   
   
3. Place the computer number on the scan form in the box indicated for it. (If applicable, fill in the team-taught number in the box to the right of the computer number). Also, fill in the student information box on the reverse side of the form. Read each item carefully and fill in the circles under the appropriate column indicating your response to each item. Please note that a "1" means "Strongly Disagree" or "Weak" and a "6" means "Strongly Agree" or "Strong."
   
   
4. When you have finished, please sit quietly until everyone has finished filling out the instrument. I have asked (student's name) to collect the instruments, make sure narratives are separated from scannable forms, face all forms in the same direction, put them into their separate, original envelopes, seal the envelopes, and deliver them to the department head. You will have 15 minutes to complete the instrument. Any questions? You may begin."

Special Conditions for Non-Traditional Courses

1. Student Teaching, Practicum Courses, Thesis Courses, Independent Studies, and Internships
   
   
   1. As noted in the SOT Policies, no SOT procedures will be conducted for Student Teaching, Practicum Courses, Thesis Courses, Independent Studies, and Internships. The SOT instrument is not designed to accommodate the distinctive differences in content, presentation, and activities contained in these non-traditional types of courses.
      
      
   2. A master list of classes will be sent to each Department Head each semester for review of classes that fit the qualifications for exemption.
      
      
   3. Departments where these courses are offered will be responsible for developing procedures appropriate for evaluating the faculty members delivering these courses.
      
      
   4. Any exceptions to this policy must be approved by the respective Department Head and Dean.
   
   
2. Half-term and Team-taught Courses
   
   
   1. It is the responsibility of Department Heads to update team-taught courses to include all faculty members teaching those courses.
      
      
   2. For half-term courses, the Office of Institutional Research & Assessment will make deliver the SOT instruments to Department Heads at least one week prior to the final exam (or before the last two class meetings prior to the final exam during summer sessions).
      
      
   3. For team-taught courses, the Office of Institutional Research & Assessment will make sure that an envelope containing instruments for each instructor which has been identified as being involved in the teaching of the course is included in the packets delivered to Department Heads.
   
   
3. Special Teaching Conditions
   
   
   1. Courses such as those delivered via the Internet or compressed video may require special arrangements for arrangements for administration and/or special forms.
      
      
   2. When a special teaching situation arises, the Department Heand and the instructors should contact Institutional Research and Assessment with questions about special teaching conditions.

Purpose of Analyzing the Narrative Comments

Evaluation by students, peers, and administrators is a critical element in effective educational programs. Self-assessment, often cited as the weak link in achieving a quality evaluation, is also important in producing a holistic evaluation. The results of the quantitative analysis of student comments furnishes insight. Making sense of the often conflicting written responses, however, can be both frustrating and confusing. Should such comments be destroyed, ignored, or should we order and use the wealth of this additional information?

Validity Issues Related to Student Feedback

Validity of student opinions, that is, the correctness or credibility of a description, conclusion, explanation, or interpretation offered by students, is often a major concern. In self-evaluation, it is not always necessary to attain some ultimate truth in order for the evaluation to be useful and believable. External generalizability of narrative comments is often not a crucial issue. In contrast, the faculty member's personal interpretation within the context of this particular class at this particular time is what makes the evaluation meaningful.

How Qualitative Analysis Complements Quantitative Analysis

What is the purpose of evaluating narrative comments? Results from scannable forms can be used alone to assess teaching environment, instructor-student interaction, and other important elements of instruction. For example, scannable data may be used to determine the number of students who believe that the instructor's speech is not clear and understandable, while qualitative data collection could elaborate on why students do not consider the instructor's speech clear and understandable. The comments might explain that the noise from the air conditioning unit makes it difficult to hear the lectures. In the same way, a quick review of comments might also explain any outliers that could have influenced the scannable summaries.

The methods are not simply different ways of doing the same thing. Instead, they have different strengths and logics, together providing a holistic approach. The strengths of narrative data derive primarily from the focus on specific situations or people, and the emphasis on words rather than numbers. This type of research often supplies the answers to the "Why?" questions left by quantitative designs.

Uses for Improvements in Teaching and Learning

If the instructor plans to benefit from the narrative comments, structure must be imposed so the comments will make sense and offer insight. A variety of approaches may be adopted in order to analyze qualitative data. This variety stems from the range of talents, the diversity of settings, and the goals of the faculty members. What links all the approaches is a central concern with transforming and interpreting seemingly unconnected data in a scholarly way.

One way to analyze the data is to use the narrative response form that SLU students use. The comments are already categorized as those pertaining to (1) the instructor, (2) the activities and tests, (3) the books and materials, and (4) the classroom. These comments are already two dimensional--strengths and suggestions for improvement.

The instructor summarizes students' written statements by grouping similar comments and then counting the number of times such comments are made. Then, the comments are listed in each category on the narrative form in order of decreasing frequency, making sure to specify the number of times each statement was made. For example, one summary might begin as follows:

Instructor Strengths
(25) does concise or good job presenting material / explains clearly
(15) sense of humor
(12) friendly / polite / accessible
(10) knows the material

Areas for Improvement
(16) not enough review in class / quiz not same as class material/ not enough time
(9) unclear explanations / needs to explain in more than one way
(6) unorganized / needs to stay on topic
(5) monotone voice / needs to speak louder

Once the information is in an organized format, the faculty member can more easily make inferences. For example, according to the above information, students perceive this professor as an effective one. Most of the students seem to understand the material presented; however, if additional examples were used, more students might succeed. Perhaps transparencies or handouts with the lecture outlined would help both professor and students stay on task. Finally, many students indicate that they need additional review, more time for taking tests, and test questions taken directly from lecture or from the text A few changes in the instructor's delivery might make a big difference in how students perceive and rate this instructor. This analysis, including both scannable and narrative data, could offer additional ideas for professional development and could easily be translated into teaching goals.

Another example of an analysis of narrative comments is given in the following table and commentary.

Focusing the Comments

If the instructor is interested in specific dynamics of a particular class, additional information could be supplied to the students at the time of the evaluation. For example, if a new text has been adopted or a new teaching method has been incorporated, the instructor might ask students to include feedback on this specific aspect of the course in addition to their other comments. This focus helps students structure written responses and gives the instructor additional desired feedback which might have been obscured by a more random presentation.

Southeastern Louisiana University Student Opinion of Teaching -- Narrative Response Form
Computer Number: 0649	name of Instructor: S. Smith	Course: Nuclear Physics 101	Section Number: 04	Semester/Year: Fall 1997
	STRENGTHS		SUGGESTIONS FOR IMPROVEMENT
The Instructor	Presentation is organized (6 comments) Friendly, caring, concerned (4 comments) Knowledge of subject is good (4 comments)		Needs to speak louder, monotone (1 comment) Needs less talking (4 comments)
The Activities and Tests	Challenging material (3 comments) Lab demonstrations are helpful, good (4 comments) Peer learning was helpful, good (2 comments)		Tests-too difficult, unfair (11 comments) Need more feedback (2 comments)
The Books and Materials	Overheads were helpful (2 comments) OK (1 comment)		Book was not good, outdated, not clear (5 comments)
The Classroom (i.e., temperature, acoustics, etc.)	OK (1 comment)		Couldn't hear (2 comments) Room was too cold (1 comment)

20 students / 53 comments

Analysis of Comments:

53 comments from 20 students

Summary:

In general many of the comments were positive and seemed to correspond with the scores on the scannable forms.

The highest item averages I received were for planning and management. These averages were higher than the dept. or college averages. The lowest averages were for items 17, 19, 20 and 21.

Based on the narrative comments, students seem to be having difficulty with the exams. This does not seem to be related to their perceptions of me in general, as comments indicated positive perceptions of my relationships with students.

Suggestions for Further Course Development:

1. Review exams in terms of content and difficulty to make sure they are appropriate.
   
   
2. Investigate sending exam study guides (with answers) to students via e-mail. This would help also with the need to get more feedback. Also investigate getting students to use these study guides collaboratively via e-mail since students seem to like peer interaction.
   
   
3. Review course and look for more ways to use visuals -- overheads and demonstrations. Students seem to like them and find them helpful. Also would reduce need for "teacher talk."
   
   
4. Investigate getting a new book.

Narrative data:

1. provide a "larger picture" of faculty performance.
   
   
2. provide a qualitative layer for Department Heads/Deans to help interpret the quantitative data.
   
   
3. provide spontaneous responses from students who choose to volunteer information and ask for non-structured responses (unlike the quantitative portion).
   
   
4. provide an opportunity for students to speak frankly about the instructor and the course.
   
   
5. provide a potential source of information for the faculty member and the Department Head for the development of performance plans.
   
   
6. provide sections for input that may be out of the control of the faculty member (i.e., books, materials, classroom). Although useful information for the faculty member, these areas were included for the benefit of administrators who might have more control over these conditions than the individual instructor.
   
   
7. provide a holistic representation of instruction. While narratives possibly may be a source for alerting supervisors to serious problems, Departments Heads/Deans should avoid placing undue emphasis on the negative comments of one or two students.
   
   
8. provide the faculty member with information for the purpose of modifying or enhancing the course and/or the instructor's instruction.

1. Statement of Philosophy

   The annual review of faculty performance, including the results of the SOT Narrative Response Forms, should be seen as a supportive and positive experience for both the faculty member and the Department Head, with its principal goal to enhance the learning experience of all students.
   
   

2. Role of the Narrative Response Forms

   It is a belief of the SOT Review Committee that information from the narrative response forms must be kept in perspective when it is used in annual faculty evaluations. The information obtained from the narrative responses should not be weighted more heavily than any other information available to the Department Head. Moreover, the narrative responses should be seen as only one part of an overall picture of teaching performance over time. Following is a list of materials that might also be used in the annual evaluations of faculty teaching:
   
   

   • SOT Questionnaire Summaries
   • Self-assessment of Teaching
   • Peer Evaluations
   • Department Head Observations and Evaluation
   • Course Syllabi
   • Tests
   • Grade Distribution
   • Work Load
   • etc.
   
   
3. Narrative Evaluation

   Prior to the annual evaluation conference between the Department Head and the faculty member, the Department Head should review the SOT Questionnaire Summary, the Narrative Responses, and any "Special Factors in SOT Administration" forms. Parts of the narrative responses may contain information that is not solely focused on teacher performance. For this reason, the narrative response form may be best examined by quadrants (please see page 15). Quadrant I, composed of the top two boxes, deals with Strengths and Suggestions for Improvement for the Instructor. Likewise, Quadrant II deals with Strengths and Suggestions for Improvement of Activities and Tests for the class. Quadrants III and IV are composed primarily of information that may or may not be beyond the instructors' control: books and materials, classroom environment, etc.

The Department Head may want to scan SOT Narrative Response forms for the entire department in order to develop a holistic perspective of comments and statements made by students about instructors, learning environments, and areas that many students suggest need to be improved.

1. Courses with Multiple Sections

   If a number of different faculty teach sections of the same course, then the narrative responses from all sections of the course should be compared to identify course commonalities, strengths, problems, and special needs. These data, as well as information identified in section II above, could be used as a source of data for Major Field Assessment and other program evaluation.
   
   

2. Quadrants III and IV

   Information in Quadrants III and IV will be primarily used for administrative purposes such as program evaluation, major field assessment, and establishing budget priorities. Nevertheless, this information can also be used to qualify the comments which may be influenced by learning conditions (environment), textbook adoption policies, technology resources, lab requirements, crowded conditions, broken furniture, handicap access, accommodations for disabilities, etc.

After scanning the narrative responses, the Department Head should prepare for the end-of-year conference. Progress toward attainment of goals may require effort on both the Department Head's part in the form of materials, scheduling, and other resources, and on the faculty member's part in the form of identifiable and measurable improvements. Actions agreeable to both parties may be documented. The students' narrative comments should, in some form, be used to evaluate the faculty member's teaching performance.

The conference itself should focus on the faculty member's narrative summary and the goals and conclusions it includes. The Department Head may also wish to offer his/her own observations regarding the narrative comments. Where there is agreement on the conclusions, a note to this effect may be made on the faculty member's narrative summary. Where there is not agreement, further discussion should take place, with the Department Head offering suggestions to the faculty member as deemed appropriate.

Southeastern Louisiana University Student Opinion of Teaching -- Narrative Response Form
Computer Number:	name of Instructor:	Course:	Section Number:	Semester/Year:
	STRENGTHS		SUGGESTIONS FOR IMPROVEMENT
The Instructor	Ia		Ib
The Activities and Tests	IIa		IIb
The Books and Materials	IIIa		IIIb
The Classroom (i.e., temperature, acoustics, etc.)	IVa		IVb

CLASS________________________________________________COMP.#________
                    (prefix)          (Course #)          (Section #)

THIS SHEET MAY BE USED BY FACULTY MEMBERS TO NOTE ANY UNUSUAL FACTORS THAT MIGHT AFFECT THE RESULT OF THE STUDENT OPINION OF TEACHING FOR THIS COURSE (e.g., an unanticipated attendance problem, an emergency or unexpected event, experimentation with new teaching techniques, problems or special circumstances regarding student behavior, classroom environment and equipment, etc.). COMPLETION OF THIS FORM IS OPTIONAL. It is the faculty member's responsibility to print this form, fill it out, place it in a sealed envelope and deliver it to the Department Head's office by the end of the final exam period for the semester.

******************************************************************************

____________________________________
(Signature)

The Office of Technology supports the SOT program by working cooperatively with the Office of Institutional Research & Assessment, which is directly responsible for the overall implementation of this program. The Office of Technology provides the mainframe access and programming support necessary to perform the data manipulation and the resulting generation of reports for the SOT survey program. The Assistant Vice President of Technology is the chief administrative officer for this institutional unit. The Assistant Vice President for Institutional Planning, Research, and Policy Analysis is ultimately responsible for the effective implementation of electronic data management and storage of SOT data files and summary reports.

All activities involving the management and storage of SOT program data are implemented using standard, accepted procedures for handling confidential information. Any policy or procedural changes in the execution of the data management and storage activities must be requested in writing by the Assistant Vice President for Institutional Planning Research and Policy Analysis to the Assistant Vice President for Technology.

Two departments within the Office of Technology are directly involved in the implementation of the SOT program and the storage of SOT data files and summary reports. The Database/Web Team provides the programs necessary to gather and accurately report the results of SOT surveys each semester. Computer Operations controls the daily operations of the mainframe system. Specific categories of activities for each of these subunits are outlined in the sections that follow.

Responsibilities of the MIS department

The Coordinator of the Database/Web Team is directly responsible for maintaining the programs necessary to complete the following functions:

1. run procedures necessary to load the data into the main file.
   
   
2. produce the various summary reports using the main data file (e.g., individual faculty-, department-, college-, institution-level reports);

Responsibilities of the Operations department:

The Coordinator of Computer Operations is directly responsible for the day-to-day computer applications and activities necessary to complete the SOT program reports and support and archive of SOT data files and summary reports. These responsibilities include:

1. executing regular backups of the system;
   
   
2. printing data and summary reports;
   
   
3. mounting electronic data storage tapes (4 mm cassette tapes) and securely storing the tape for one calendar year until Institutional Research and Assessment claims the tape;
   
   
4. copying the Student Records Systems (SRS) data/master file into SOT account when requested.

Responsibility of the Assistant Vice President for Institutional Planning, Research and Policy Analysis

The Assistant Vice President for Institutional Planning, Research and Policy Analysis is directly responsible for maintaining confidential and fireproof archival storage of electronic files containing data files and summary reports by semester. Also, a paper copy of the institutional-level summary reports will be maintained.

Student Opinion of Teaching (SOT) program data are collected from students in designated classes using custom-designed questionnaire/scan forms. Since the initial implementation of the SOT program, these forms have been maintained in the following program archives to support a variety of University-wide activities that include the following: 1) verification of data processing results; 2) personnel evaluation and decisions (e.g., tenure, promotion, merit pay); and 3) appeals and grievance proceedings.

Once the University has completed its work with the SOT scan forms, original data (as individual student scan forms) may be made available to each faculty member for whom data were collected. Faculty may obtain SOT data forms only for the classes for which he or she is the instructor of record.

Scannable forms will be kept by Institutional Research and Assessment for one year starting with the end of each academic year. After one year, the forms which have not been obtained by faculty members will be shredded. For example, at the beginning of the 2000-2001 academic year, the forms from academic year 1999-2000 will be destroyed.

Faculty will be notified by several different means of communication of the availability of forms. These means may include e-mail notification, a notice in the By-Lion, and notification via the Faculty Senate. Faculty will be notified which forms are available, and the deadline date by which the forms must be claimed. It is the faculty member's responsibility to contact Institutional Research and Assessment if they want the scannable forms.

All reports will be distributed by the Office of Institutional Research and Assessment will be distributed by the 14th class day of the following semester.

Deans will receive the following reports:

• Summaries by College (Report 1A)
• Summaries by Department for all Colleges (Report 1C)
• Summaries by Instructor for each Department in their college (Report 2)
• Summaries by Class for Each Instructor in their college (Report 3)

Department Heads will receive the following reports:

• Summaries by Instructor for their Department (Report 2)
• Summaries by Class for Each Instructor in their department (Report 3)

Department Heads will receive the following reports to be distributed to individual faculty members:

• Summaries by Class for Each Instructor in their department (Report 3)
• Individual Item Analysis for each class (Supplemental Report)