Curricula assessment using the course diagnostic survey: a proposal.

Curricula assessment using the course diagnostic survey: a proposal. ABSTRACT The purpose of this paper was to develop a model of an alternativeapproach to assessing courses and ultimately curricula. Borrowing fromJob Characteristics Theory, a modified survey, the Course DiagnosticsSurvey (CDS) was developed. Using this instrument, a model is suggestedthat measures the attitudes and resulting outcomes at both the courseand overall program level. This model suggests a roadmap as to course orprogram components that directly impact desired outcomes. Hypotheses aresuggested to study the potential of the CDS as an appropriate tool forassessment. INTRODUCTION Assessment of student learning has moved to the forefront ofbusiness schools over the last decade (Palomba & Banta, 1999, 2001;Banta, Lund, Black & Oblander, 1996). Much of this new emphasis isdirectly attributable to AACSB International expectations. In fact, manyperceive this emphasis to increase with the new standards recentlyadopted that focus on assurance of learning (Black & Duhon, 2003;Mirchandani, Lynch & Hamilton, 2001; Michlitsch & Sidle si��dle?v. si��dled, si��dling, si��dlesv.intr.1. To move sideways: sidled through the narrow doorway.2. , 2002). Most schools seeking to assess student learning fall back onadministering standardized tests, imbedding measurements within coursesor conducting post surveys. An almost universal approach has been tosurvey students with either the periodic student evaluationsadministered each semester se��mes��ter?n.One of two divisions of 15 to 18 weeks each of an academic year.[German, from Latin (cursus) s or an instrument that is locally preparedthat asks a series of questions to provoke pro��voke?tr.v. pro��voked, pro��vok��ing, pro��vokes1. To incite to anger or resentment.2. To stir to action or feeling.3. To give rise to; evoke: provoke laughter. attitudinal responses. Manyof these instruments lack a high level of internal validity Internal validity is a form of experimental validity [1]. An experiment is said to possess internal validity if it properly demonstrates a causal relation between two variables [2] [3]. . The purpose of this research is to suggest a model using amodification of an instrument that has already proven to be valid andreliable at measuring motivational aspects of a job. The instrument tobe recommended, the Course Diagnostic Survey (CDS), is adapted from theJob Diagnostic Survey. This instrument addresses what many, such asCharles Duke Charles Moss Duke, Jr. (born October 3 1935), a retired USAF Brigadier General, was a United States astronaut for NASA. He is one of only twelve men who have walked on the moon. (2002), see as being as important as actual contentabsorption-student perceptions. As will be presented in subsequentsections of this paper, the CDS focuses on how the course design createsunique psychological states (student feeling toward their educationalenvironment) and thus creates affective affective/af��fec��tive/ (ah-fek��tiv) pertaining to affect. af��fec��tiveadj.1. Concerned with or arousing feelings or emotions; emotional.2. outcomes (satisfaction or lackthere of). BACKGROUND No longer content with just technical competence technical competence,n the ability of the practitioner, during the treatment phase of dental care and with respect to those procedures combining psychomotor and cognitive skills, consistently to provide services at a professionally acceptable level. from our businessschool graduates, employers are now demanding "... skills inleadership, problem solving problem solvingProcess involved in finding a solution to a problem. Many animals routinely solve problems of locomotion, food finding, and shelter through trial and error. , oral and written communication, along withattributes of motivation and assertiveness assertiveness/as��ser��tive��ness/ (ah-ser��tiv-nes) the quality or state of bold or confident self-expression, neither aggressive nor submissive. " (Fontenot, Haarhues& Hoffman, 1991, p. 56). However, the ability of our institutions ofhigher education to meet these changing demands has been severelyquestioned. Harvard President Emeritus e��mer��i��tus?adj.Retired but retaining an honorary title corresponding to that held immediately before retirement: a professor emeritus.n. pl. , Derek Bok Derek Curtis Bok (born March 22, 1930) is an American lawyer and educator, and the former president of Harvard University.Bok was born in Bryn Mawr, Pennsylvania, and graduated from Stanford University (B.A., 1951), Harvard Law School (J.D. (1992) has chided ouruniversities for their failure to even examine the effectiveness oftheir educational programs. Fortunately, one stream of research has begun to investigate theeffectiveness of selected programs using cognitive scales for thispurpose in business curriculums. Using their Skills/Career Usefulnessscale, Fontenot, et al. (1991) studied the effectiveness of SmallBusiness Institute (SBI SBI Special Background InvestigationSBI Subsidiary Body for ImplementationSBI State Bank of IndiaSBI Secure Border InitiativeSBI Small Business InstituteSBI Stockholm Brain InstituteSBI Serious Bacterial InfectionSBI Society of Breast Imaging ) courses and Business Policy courses indeveloping desired student skills. Using job analysis and designtechniques developed for work environments, Watts and Jackson (1995)investigated the applicability of Hackman and Oldham's (1976) JobCharacteristic Theory to course design. Job Characteristic Theory hasalso been used to assess an institution's student evaluation ofinstruction (Watts, 1992), and to analyze the effect of course redesign re��de��sign?tr.v. re��de��signed, re��de��sign��ing, re��de��signsTo make a revision in the appearance or function of.re on SBI student outcomes (Watts, Jackson & Box, 1995). The JDS See Java Desktop System. proposes that positive results will result in the workplace (high motivation, high satisfaction with the job and highperformance level) when three critical psychological states (experiencedmeaningfulness of the job, experienced responsibility for the outcomesof the job, and knowledge of the job results) exist. The theory goes onto suggest that the three critical psychological states are created byspecific core job characteristics being present. These core jobcharacteristics include: skill variety, task identity, tasksignificance, high levels of autonomy, and effective feedback. However,not all individuals will respond equally, but rather are influenced bytheir own growth need strength-how important is the job to each personindividually. This model is presented below in Figure 1. As can be seen in Table 1 below, there appears to be an intuitiverelationship between what occurs in the job setting to that in theacademic classroom. While this relationship may not be exact, it doesoffer promising possibilities. METHOD Subjects The subjects for this study were 586 undergraduate and graduatestudents in the school of business of a small southwest regionaluniversity. Students were represented across all academic disciplines,age distribution, sex, and ethnic background. This number representednearly 100 percent of all students enrolled in the school. Students wereasked to complete the Course Diagnostic Survey. No incentive or penaltywas provided for participation in the survey. During this initial phaseof the study and for statistical comparison, the results of allparticipants were combined into one group. Instrument As previously mentioned, the instrument used was a modified JobDiagnostic Survey (JDS) resulting is the Course Diagnostic Survey (CDS).The instrument was used to collect perceptions of core coursecharacteristics, critical psychological states, growth need strength,internal academic motivation and course satisfaction. Few modificationswere needed to apply the original instrument to the academic environmentbeing examined in this study. Seven point scales was used to maintainconsistency with the JDS. This approach has proven to be valid inseveral other studies involving students in the academic setting (Watts,1992; Watts, Jackson & Box, 1995; Watts & Jackson, 1995;Fontenot, Haarhues and Hoffman, 1991). Course Components Course components were measured with the seven point scale veryinaccurate to very accurate in response to "how much do you agreewith the statement". The course component skill variety wasmeasured by the items "the course requires me to use a number ofhigh and complex skills" and a reverse score of "the course isquite simple and repetitive". Task identity was measured by"the course provides me the chance to completely finish the piecesof work I begin" and a reverse score on the item "The courseis arranged so that I do not have the chance to do an entire piece ofwork from beginning to end". Task significance was indicated through responses on "Thiscourse is one where a lot of other people can be affected by how wellthe work gets" and a reverse response on "The course itself isnot very significant or important in the broader scheme of things".Autonomy was shown through the items "The course gives meconsiderable opportunity for independence and freedom in how I do thework" and "The course denies me any chance to use my personalinitiative or judgment in carrying out the work (reverse scored)". The final two components, feedback from the course and feedbackfrom the instructor, were measured respectively by "Just doing thework required by the course provides many chances for me to figure outhow well I am doing", "The course itself provides very fewclues about whether or not I am performing well (reversed)" and"The instructor often lets me know how well I am performing","The instructor and fellow students in this course almost nevergive me any feedback about how well I am doing in my work(reversed)". The table and figure below illustrate the means and standarddeviations of the sample population for the six course characteristicsi.e. skill variety, task identity, task significance, autonomy, feedbackfrom the course, feedback from the instructor. [FIGURE 2 OMITTED] Critical Psychological States The critical psychological states inspired by the course componentswere also measured in this study. As stated before, the CDS used a sevenpoint Likert scale Likert scaleA subjective scoring system that allows a person being surveyed to quantify likes and preferences on a 5-point scale, with 1 being the least important, relevant, interesting, most ho-hum, or other, and 5 being most excellent, yeehah important, etc . The scale was a measure of how well the studentagreed with the statement and scales ranged from very inaccurate to veryaccurate. The psychological state meaningfulness was indicated by two items,"The work I do in this course is very meaningful to me" and"Most of the things I have to do in this course seem useless ortrivial" which was reverse scored. Responsibility was measured by"I feel a high degree of personal responsibility for the work I doin this course", "I feel I should personally take the creditor blame for the results of my work in this course", "Whetheror not course work gets done right is clearly my responsibility"and the reversed item "It's hard, in this course, for me tocare very much about whether or not the work gets done right". Thelast psychological state measured, knowledge of results, was indicatedby "I usually know whether or not my work is satisfactory in thiscourse" and "I often have trouble figuring out whetherI'm doing well or poorly in this course" which was reversescored. The table and figure below indicate the means and standarddeviations of the three psychological states, meaningfulness,responsibility, and knowledge of results. [FIGURE 3 OMITTED] Student Outcomes The third element component of the CDS was a measure of two studentoutcomes. These included general satisfaction with the course andstudent motivation. The seven point Likert scale indicated how well thestudent agreed with the statement and scales ranged from very inaccurateto very accurate. General satisfaction with the course was measured by three items.These included "Generally speaking, I am very satisfied with thiscourse", "I am generally satisfied with the kind of workrequired in the course" and "I frequently think of droppingthis course" that was reverse scored. The outcome, studentmotivation, was determined by responses on "My opinion of myselfgoes up when I do the course work well", "I feel a great senseof personal satisfaction when I do the course work well", "Ifeel bad and unhappy when I discover that I have performed poorly inthis course", and the reverse scored item "My own feelingsgenerally are not affected much one way or the other by how well I do inthis course". The means and standard deviations of the student responses for thegeneral satisfaction and motivation outcomes are shown below. [FIGURE 4 OMITTED] Motivating Potential Score The Motivating Potential Score indicates the motivating potentialof a job or with the CDS, a course. It would be measured by theresponses of students in individual courses and calculated by theformula: PROCEDURE To capture the influences of course related activities, theinstrument was administered late in the semester. On a predetermined pre��de��ter��mine?v. pre��de��ter��mined, pre��de��ter��min��ing, pre��de��ter��minesv.tr.1. To determine, decide, or establish in advance: date, instructors announced in class that students had been asked toparticipate in an important study and read the following instructions: This questionnaire was developed as part of a study of course-related activities and how students react to them. The questionnaire helps to determine how courses can be better designed by obtaining information about how students react to different kinds of course-related activities. On the following pages you will find several different kinds of questions about your course. Specific instructions are given at the start of each section; please read them carefully. It should take no more than 15 minutes to complete the entire questionnaire. Please move through it quickly. The questions are designed to obtain your perceptions of course-related activities and your reactions to them. There are no trick questions. Your individual answers will be kept completely confidential. Please answer each item as honestly and frankly as possible. Thank you for you cooperation. DISCUSSION As stated in the introduction, the main purpose of this exploratorystudy was to propose the use of the Course Diagnostic Survey instrumentas a means of assessing students in an academic setting. As the originalinstrument was intended to assess the impact of redesign of jobs, themodification and resulting CDS could equally be as successful inassessing the impact of specific course components on the psychologicalstates and outcomes in the educational setting. While it appears thatthe instrument has potential in this area, additional study is needed tovalidate To prove something to be sound or logical. Also to certify conformance to a standard. Contrast with "verify," which means to prove something to be correct.For example, data entry validity checking determines whether the data make sense (numbers fall within a range, numeric data the instrument in an educational setting. Specifically, threehypotheses are proposed. H1: The CDS course components, i.e. skill variety, skill identity,task significance, autonomy, feedback from course, and feedback frominstructor; lead to the indicated psychological states. H2: The critical psychological states in the academic setting asmeasured by the CDS will be related to general satisfaction andmotivation. H3: The MPS as measured by the CDS will indicate the motivatingpotential of a specific course. Future studies using the study sample,as well as other samples, should attempt to show the relationshipsindicated in the hypotheses. This would assist in the validation See validate. validation - The stage in the software life-cycle at the end of the development process where software is evaluated to ensure that it complies with the requirements. of theCDS and its use in course assessment. It is also suggested thatadditional studies consider the use of the model in overall programdevelopment. REFERENCES Ater, E.C. & Coulter, K.L. (1980). Consumer internships:Encouraging consumer/business dialogue. Journal of BusinessCommunications, 17(2), 33-39. Banta, T.W., Lund, J.P., Black, K. & Oblander, F.W. (1996).Assessment in Practice. San Francisco San Francisco(săn frănsĭs`kō), city (1990 pop. 723,959), coextensive with San Francisco co., W Calif., on the tip of a peninsula between the Pacific Ocean and San Francisco Bay, which are connected by the strait known as the Golden : Jossey-Bass Publishers. Barnett, S.T., Dascher, P.E. & Nicholson, C.Y. (2004). Canschool oversight adequately assess department outcomes? A study ofmarketing curriculum content. Journal of Education for Business, 79(3),157-162. Black, H.T. & Duhon, D.L. (2003). Evaluation and improvingstudent achievement in business programs: The effective use ofstandardized assessment tests. Journal of Education for Business, 79(2),90-98. Bok, D. (1992). Reclaiming For the neopagan organization of this name, see Reclaiming (neopaganism). For the reclaiming of land, see land reclamation. To reclaim is to bring a word back to a more acceptable course. the public trust. Change, August, 23-29. Cordery, J.L. & Sevastos, P.P. (1993). Responses to theoriginal and revised job diagnostic survey. Journal of AppliedPsychology Journal of Applied Psychology is a publication of the APA. It has a high impact factor for its field. It typically publishes high quality empirical papers.www.apa. , 78(1), 141-143. Davis, M.A., Curtis, M.B. & Tschetter, J.D. (2003). Evaluatingcognitive training outcomes: Validity and utility of structuralknowledge assessment. Journal of Business & Psychology, 18(2), 191. Duke, C.R. (2002). Learning outcomes: Comparing student perceptionsof skill level and importance. Journal of Marketing Education, 24(3),203-217. Fontenot, G., Haarhues, M. & Hoffman, L. (1991). The benefitsof the SBI program: Perceptions of former students. Journal of SmallBusiness Strategy, 2(1), 56-71. Hackman, J.R. & Lawler, E.E. (1971). Employee reactions to jobcharacteristics. Journal of Applied Psychology Monograph, 55, 259-286. Hackman, J.R. & Oldham, G.R. (1975). Development of the jobdiagnostics survey. Journal of Applied Psychology, 60, 159-170. Hackman, J.R. & Oldham, G.R. (1976). Motivation through thedesign of work: Test of a theory. Organizational Behavior and HumanPerformance, 16, 250-279. Hackman, J.R. & Oldham, G.R. (1980). WorkRedesign. Reading, MA: Addison-Wesley. Hoffman, L., Fontenot, G. & Viswanathan, R. (1990). Anexploratory evaluation of the effectiveness of the SBI program asperceived by quantitative and non-quantitative majors. Proceedings ofthe 1990 SBIDA SBIDA Small Business Innovation Development Act National Conference, 80-85. Kent, T.W. & Davis, T.J. (2002). Using retranslation to developoperationally anchored scales to assess the motivational context ofjobs. International Journal of Management, 19(1), 10. Lawrence, E. (1990). Learning portfolio management by experience:University student investment funds Noun 1. investment funds - money that is invested with an expectation of profitinvestmentassets - anything of material value or usefulness that is owned by a person or company . The Financial Review, 25, 165+. Luthans, F. (1992). Organizational Behavior. New York New York, state, United StatesNew York,Middle Atlantic state of the United States. It is bordered by Vermont, Massachusetts, Connecticut, and the Atlantic Ocean (E), New Jersey and Pennsylvania (S), Lakes Erie and Ontario and the Canadian province of : McGraw-Hill,Inc. Marchese mar��che��se?n. pl. mar��che��si1. An Italian nobleman ranking above a count and below a prince.2. Used as the title for such a nobleman. , M.C. (1998). Some factors affecting the relationshipbetween job characteristics and job worth: A job-role interpretation.Journal of Organizational Analysis, 6(4), 355-369. Michlitsch, J.F. & Sidle, M.W. (2002). Assessing studentlearning outcomes: A comparative study of techniques used in businessschool disciplines. Journal of Education for Business, 77(3), 125-130. Mirchandani, D., Lynch, R. & Hamilton, D. (2001). Using the ETS ETS Educational Testing Service (nonprofit private educational testing and measurement organization)ETS Emergency Telecommunications ServiceETS Electronic Trading SystemETS Engineering (&)Technical Services major field test in business: Implications for assessment. Journal ofEducation for Business, 77(1), 51-57. Oldham, G.R., Hackman, J.R. & Stepins, L.P. (1978). Norms forthe job diagnostic survey. Technical Report No. 16, School ofOrganization and Management, Yale University Yale University,at New Haven, Conn.; coeducational. Chartered as a collegiate school for men in 1701 largely as a result of the efforts of James Pierpont, it opened at Killingworth (now Clinton) in 1702, moved (1707) to Saybrook (now Old Saybrook), and in 1716 was , 1-45. Palomba, C.A. & Banta, T.W. (2001). Assessing StudentCompetence in Accredited accreditedrecognition by an appropriate authority that the performance of a particular institution has satisfied a prestated set of criteria.accredited herdscattle herds which have achieved a low level of reactors to, e.g. Disciplines. Sterling, VA: Stylus stylus:see pen. (1) A pen-shaped instrument that is used to "draw" images or select from menus. Styli (the plural of stylus, pronounced "sty-lye") come with handheld devices that have touch screens, such as PDAs and video games. Publishing,Inc. Palomba, C.A. & Banta, T.W. (1999). Assessment Essentials. SanFrancisco: Jossey-Bass Publishers. Riggio, R.E., Mayes, B.T. & Schleicher, D.J. (2003). Usingassessment center methods for measuring undergraduate business studentoutcomes. Journal of Management Inquiry, 12(1), 68-78. Rungtusanatham, M. & Anderson, J.C. (1996). A clarification onconceptual and methodological issues related to the job characteristicsmodel. Journal of Operations Management Operations management is an area of business that is concerned with the production of goods and services, and involves the responsibility of ensuring that business operations are efficient and effective. , 14(4), 357-367. Serva, M.A. & Fuller, M.A. (2004). Aligning what we do and whatwe measure in business schools: Incorporating active learning andeffective media use in the assessment of instruction. Journal ofManagement Education, 28(1), 19. Spicer, D.P. (2004). The impact of approaches to learning andcognition cognitionAct or process of knowing. Cognition includes every mental process that may be described as an experience of knowing (including perceiving, recognizing, conceiving, and reasoning), as distinguished from an experience of feeling or of willing. on academic performance in business and management. Educationand Training, 46(4/5), 194. Tiegs, R.B., Tetrick, 1.E. & Yitzhak, F. (1992). Growth needstrength and context satisfactions as moderators of the relations of thejob characteristics model. Journal of Management, 18, 575-593. Watts. L.R. & Hudnall, J. (1991). Applying job characteristicstheory to course design. In G. James (Ed.), Proceedings of the MountainPlains Management Conference. Fort Collins, CO: Colorado StateUniversity. Watts, L.R. & Jackson, W.T. (1994). The effect of courseredesign on SBI student outcomes: An application of job characteristicsmodel. Journal of Business & Entrepreneurship, 6(1), 89-100. Watts, L.R., Jackson, W.T. & Box, T.M. (1995). Student relatedoutcomes and task design characteristics. Texas Business EducationAssociation Journal, 5(1), 127-141. Watts, L.R. & Jackson, W.T. (1995). The SBI program and studentoutcomes: A study of business policy classes. Journal of Small BusinessStrategy, 6(1), 93-103. William T. Jackson William T. Jackson (1876-1933) was an American politician. He served as mayor of Toledo, Ohio between 1928 and 1931. External linksFind-A-Grave profile for William T. Jackson , Dalton State College Dalton State College is one of six state colleges in the University System of Georgia. It is located in Dalton, Georgia.The institution was founded as Dalton Junior College in July of 1963 and opened in September of 1967 offering associate degrees. Mary Jo Jackson, Dalton State College Corbett F. Gaulden, Jr., University of Texas of the Permian Basin The University of Texas of the Permian Basin (commonly called UT Permian Basin or simply UTPB) is located in Odessa, Texas. It was authorized by the Texas Legislature in 1969 and founded in 1973. Its fall 2006 enrollment was 3,480. Table 1: Job Characteristics Compared to Course Characteristics COURSEVARIABLE GENERAL DESCRIPTION EQUIVALENCYSKILL VARIETY Usage of a wide Usage of a wide variety of skills variety of skillsTASK IDENTITY Task closure is Assignments tie evident together course concepts in a clear mannerTASK SIGNIFICANCE Outcomes are important Assignments are importantAUTONOMY Individuals have Students have impact impact and are able on course outcome to make a differenceFEEDBACK FROM JOB Job results are Grades are provided in evident a timely fashionFEEDBACK FROM AGENT Supervisor provides Instructor provides result information result information independent of gradesMEANINGFULNESS Work is meaningful Course is meaningfulRESPONSIBILITY Responsible for work Responsible for course outcomes outcomesKNOWLEDGE OF RESULTS Final outcomes are Final grades are known knownGENERAL SATISFACTION Overall satisfaction Overall satisfaction with job with courseINTERNAL WORK Job is stimulating and Course is stimulating MOTIVATION challenging and challengingMOTIVATING POTENTIAL (sk. var. + task id. + (sk. var. + task id. +STRENGTH (MPS) * task sign.)(autonomy) task sign.)(autonomy) (job feedback) (job feedback)Table 2: Course Component Means and Standard Deviations MEAN S.D.Skill variety 5.32 1.25Task identity 5.30 1.37Task significance 4.66 1.24Autonomy 4.48 .86Feedback from job 4.96 1.37Feedback from agents 5.05 1.45Table 3 Means and Standard Deviations of Psychological StatesExperienced meaningfulness 5.47 1.40Experienced responsibility 5.66 1.23Knowledge of results 5.13 1.50Table 4 Student Outcomes Means and Standard DeviationsGeneral satisfaction 5.48 1.54Internal motivation 5.01 1.01FIGURE 1: JOB CHARACTERISTIC THEORYCORE JOB [right CRITICAL [right PERSONAL ANDDIMENSIONS arrow] PSYCHOLOGICAL arrow] WORKOUTCOMES STATESSKILL [right EXPERIENCED [right HIGH INTERNALVARIETY arrow] MEANINGFULNESS arrow] WORKTASK MOTIVATIONIDENTITYAUTONOMY [right EXPERIENCED [right HIGH QUALITY arrow] RESPONSIBILITY arrow] WORK FOR OUTCOMES PERFORMANCE OF THE WORKFEEDBACK [right KNOWLEDGE OF [right HIGH arrow] THE ACTUAL arrow] SATISFACTION RESULTS OF THE WITH THE WORK WORK ACTIVITIES LOW TURNOVER AND ABSENTEEISM [up arrow] GROWTH NEED [up arrow] STRENGTH