Consistency and inconsistency in PhD thesis examination.

Consistency and inconsistency in PhD thesis examination. This is a mixed methods investigation of consistency in PhDexamination. At its core is the quantification quan��ti��fy?tr.v. quan��ti��fied, quan��ti��fy��ing, quan��ti��fies1. To determine or express the quantity of.2. of the content andconceptual analysis of examiner reports for 804 Australian Australianpertaining to or originating in Australia.Australian bat lyssavirus diseasesee Australian bat lyssavirus disease.Australian cattle doga medium-sized, compact working dog used for control of cattle. theses.First, the level of consistency between what examiners say in theirreports and the recommendation they provide for a thesis is explored,followed by an examination of the degree of discrepancy DISCREPANCY. A difference between one thing and another, between one writing and another; a variance. (q.v.) 2. Discrepancies are material and immaterial. between examinerrecommendations and university committee decisions on the theses. Twogroups of discrepant dis��crep��ant?adj.Marked by discrepancy; disagreeing.[Middle English discrepaunt, from Latin discrep recommendations are identified and analysed indepth. Finally the main sources of inconsistency in��con��sis��ten��cy?n. pl. in��con��sis��ten��cies1. The state or quality of being inconsistent.2. Something inconsistent: many inconsistencies in your proposal. are identified. It wasfound that the comments of a small minority of examiners wereinconsistent with each other or with the committee decision in asignificant way. Much more commonly the texts of examiner reports werehighly consistent and were closely reflected in the final committeedecision. Keywords doctoral examination higher education higher educationStudy beyond the level of secondary education. Institutions of higher education include not only colleges and universities but also professional schools in such fields as law, theology, medicine, business, music, and art. examiner consistency graduate education thesis assessment Introduction When a PhD thesis is examined, whatever the process, examinerstypically require something more from the student, whether it be thecorrection of typographical errors typographical error - (typo) An error while inputting text via keyboard, made despite the fact that the user knows exactly what to type in. This usually results from the operator's inexperience at keyboarding, rushing, not paying attention, or carelessness.Compare: mouso, thinko. in the written document or moresubstantial changes. It is also in the nature of doctoral candidaturethat there is the expectation that the new researcher can always learnmore, and that the thesis is but a step on this path (Mullins Mullins may refer to: Mullins, South Carolina, a US city Mullins (surname), people with the surname Mullins See alsoMullins River Mullins Center, Amherst, an arena Mullins effect Mullens &Kiley n. 1. An Australian boomerang, having one side flat and the other convex.Noun 1. kiley - an Australian boomerang; one side flat and the other convexkylie , 2002). As one commentator notes it is an 'apprenticeship inthe art of discovery' (Kwiram 2006, p. 141).This explains why mostexaminers provide some comment that is instructive in��struc��tive?adj.Conveying knowledge or information; enlightening.in��structive��ly adv. or formative formative/for��ma��tive/ (for��mah-tiv) concerned in the origination and development of an organism, part, or tissue. at thefinal point of a thesis examination (Holbrook, Bourke, Lovat Lovat may refer to: River Lovat, a river in England Lovat River, a river in Russia Lord Lovat, a title in the Peerage of Scotland Lovat Shinty Club, a shinty club from Kiltarlity, Scotland Lovat Inc, a Canadian company dealing in Tunnel Boring Machines &Dally, 2004a; Tinkler & Jackson Jackson.1 City (1990 pop. 37,446), seat of Jackson co., S Mich., on the Grand River; inc. 1857. It is an industrial and commercial center in a farm region. , 2004). Furthermore, there isevidence that the type, extent and tone of this comment reflect theoverall evaluation of the qualities of the thesis (Bourke, Hattie &Anderson Anderson, river, CanadaAnderson,river, c.465 mi (750 km) long, rising in several lakes in N central Northwest Territories, Canada. It meanders north and west before receiving the Carnwath River and flowing north to Liverpool Bay, an arm of the Arctic , 2004). This paper focuses on the standards that examinersapply and how consistently they apply them. The literature on examination and doctoral thesis quality hasmostly emerged from the UK and Australia and on the whole indicatesthat, except in all but general terms (Morley et al., 2002; Shaw &Green, 2002), doctoral 'qualities' and 'standards'have proved very difficult to 'tie down' (Tinkler &Jackson, 2004, p. 8). In the past when there were relatively fewcandidates destined des��tine?tr.v. des��tined, des��tin��ing, des��tines1. To determine beforehand; preordain: a foolish scheme destined to fail; a film destined to become a classic.2. for scholarly pursuits, thesis quality was not apublic or pressing issue, as evidenced by the lack of research andpublication on the issue.With the rapid 'massification' ofdoctoral degrees and the growing need for cross-disciplinary work totackle emerging social and scientific problems, there has come therealisation that not only is there an absence of benchmarks, but anabsence of information about the degree and its evaluation (Morley etal., 2002; Shaw & Green, 2002; Jackson & Tinkler, 2001; Tinkler& Jackson, 2004; Lovitts, 2007). A heavy reliance in Australia andelsewhere is placed on the collective experience of examiners andknowledge of 'peer review' (Hoddell et al., 2002 p. 64;Grabbe, 2003 p. 130; Tinkler & Jackson, 2004 p.119; Holbrook,Bourke, Fairbairn & Lovat, 2007). But what is the correspondencebetween peer review and thesis examination? Evidence is also emergingfrom the USA that few students publish in peer-reviewed journals peer-reviewed journalRefereed journal Academia A professional journal that only publishes articles subjected to a rigorous peer validity review process. Cf Throwaway journal. (Nettles net��tle?n.1. Any of numerous plants of the genus Urtica, having toothed leaves, unisexual apetalous flowers, and stinging hairs that cause skin irritation on contact.2. Any of various hairy, stinging, or prickly plants. & Millett, 2006). For most students, thesis examinationconstitutes the only instance of independent scholarly review of theircomplete project. While supervisors and examiners play a pivotal role in defining andshaping the practices in their disciplines, including how and whatcandidates need to learn to be successful, there is very little in theliterature that explores the connection between expectation, judgementand outcome (Mullins & Kiley 2002; Denicolo, 2003; Powell &McCauley, 2002, 2003). The lack of a formally articulated ar��tic��u��la��tedadj.Characterized by or having articulations; jointed. 'curriculum' in relation to assessment (Gilbert, 2004) promptsconcerns about consistency in procedures and judgements (Sloboda &Newstead, 1997; Tinkler & Jackson, 2000; Morley et al., 2002; Shaw& Green, 2002; Lawson et al., 2003; Denicolo, 2003). Althoughprocedures for adjudication The legal process of resolving a dispute. The formal giving or pronouncing of a judgment or decree in a court proceeding; also the judgment or decision given. The entry of a decree by a court in respect to the parties in a case. where examiners differ have been described(Tinkler & Jackson, 2004; Lawson et al., 2003), until the study onwhich this paper is based, there has been no sustained analysis ofexaminer recommendations on the same thesis and the relationshipsbetween examiner recommendations and the official committee decision. Some of the most powerful mythology mythology[Greek,=the telling of stories], the entire body of myths in a given tradition, and the study of myths. Students of anthropology, folklore, and religion study myths in different ways, distinguishing them from various other forms of popular, often orally in doctoral examination derivesfrom stories of substantial differences in judgement between examinersof the same doctoral thesis (Morley et al., 2002). High levels ofinconsistency would cast doubt on the reliability and quality ofdoctoral assessment and, in this current international climate ofquality assurance and research quality assessment, there is intensified in��ten��si��fy?v. in��ten��si��fied, in��ten��si��fy��ing, in��ten��si��fiesv.tr.1. To make intense or more intense: interest in the outcomes of doctoral research. So, are examinersfrequently inconsistent in their judgements as anecdote anecdote(ăn`ĭkdōt'), brief narrative of a particular incident. An anecdote differs from a short story in that it is unified in time and space, is uncomplicated, and deals with a single episode. would have usbelieve? The paper deals with three questions: * How consistent are PhD examiner comments and recommendationsacross institutions and disciplines? * If inconsistency is evident, what forms does it take? * What aspects of the written examiner reports influence thesubsequent decision of the committee when it is faced with conflictingrecommendations on a thesis? Most Australian universities (58 per cent) use three externalexaminers The external examiner plays an important role in all degree level examinations in higher education in the United Kingdom. The external examiner system was introduced into the UK during the 19th century, and it is therefore also found in countries whose higher education systems were , others use two; the process does not normally involve a vivavoce [Latin, With the living voice; by word of mouth.] Verbally; orally.When applied to the examination of witnesses, the term viva voce means oral testimony as opposed to testimony contained in depositions or affidavits. or oral examination. Almost half of Australian doctoral examinersare drawn from other countries, and the majority of these are located inthe USA and the UK. This information has been published elsewheretogether with other details related to supervision, and other individualinformation such as candidate and examiner gender (Bourke et al.,2007a). In this paper the level of consistency between what examiners sayin their reports and the recommendation they provide for a thesis areexplored, followed by an examination of the degree of discrepancybetween committee decision and examiner recommendation. Two groups ofdiscrepant recommendations are identified and analysed in depth. Approach This study arises out of a larger project investigating doctoralexamination through the use of examiner reports and recommendations, andcandidate information. This paper is concerned only with questions ofconsistency, but to reach the stage where we could deal with this issue,many other analyses were conducted. The collaborative mixed methodsdesign we employed has been published in full (Holbrook & Bourke,2004; Holbrook, Bourke, Lovat & Dally, 2004b) and utilises what hasbeen identified in the methods literature as a fully integrated mixedmodel design (Teddlie & Tashakkori, 2003). Data The researchers received ethics ethics,in philosophy, the study and evaluation of human conduct in the light of moral principles. Moral principles may be viewed either as the standard of conduct that individuals have constructed for themselves or as the body of obligations and duties that a approval to access candidaterecords (with identification removed) and examiner reports andrecommendations for up to nine Australian universities. During theperiod 2003-04 the researchers obtained the support of eightuniversities in providing a complete data set comprising examinerreports, examiner recommendations, institutional decisions and doctoralcandidate enrolment and supervision history on record for their 100 mostrecent doctoral completions. The most substantial source of data foreach candidate is the exam-iner reports that, in standardised Adj. 1. standardised - brought into conformity with a standard; "standardized education"standardizedstandard - conforming to or constituting a standard of measurement or value; or of the usual or regularized or accepted kind; "windows of standard width"; form,range on average between 100 and 150 lines. The text of these wasscanned and prepared for entry into QSR QSR Quick Service RestaurantQSR QoS (Quality of Service)Satisfaction RateQSR Quality System RegulationsQSR Quality Status ReportQSR Quality System ReviewQSR Quarterly Status ReportQSR Quality System Requirement N6 software. Sample The 804 theses and 2121 examiner reports included represent allbroad fields of study. In order to select institutions that reflect arange of research intensiveness, Australian universities were dividedinto three categories on the basis of research quantum: high (consistingof 8 universities), medium (14) and low (13). The final sample consistedof three institutions in each of the high and medium categories, and twoin the low. Given that low research-intensive universities had fewer PhDcandidates, the sample was reasonably representative on this dimension(Bourke et al., 2007a). Care was also taken to ensure representativenessby both geographic area and size of institution. Analysis In the larger study we coded and analysed the data over manyiterations. We undertook analysis institution by institution, comparingand combining them along the way. The core analyses are informed by andcontribute, in turn, to 'extended' analyses. The latterexplore examiner discourse and meta-themes of power, gender, disciplineand 'doctorateness' (see Holbrook, Bourke, Lovat & Dally,2004c; Lovat, Monfries & Morrison, 2004). In them we seek tounderstand how examination process is translated into action andexamination culture (Lovat, Holbrook & Bourke, 2007). These extendedanalyses are still in progress. At the heart of this study is the conversion of the core categoriesto quantitative data. The quantified coded text is then correlated cor��re��late?v. cor��re��lat��ed, cor��re��lat��ing, cor��re��latesv.tr.1. To put or bring into causal, complementary, parallel, or reciprocal relation.2. withexaminer and committee recommendation and candidate history such aswhether or not the candidate took leave, years of candidature, number ofsupervisors, and so on. The analyses are based on the proportions oftotal text units coded at 29 different categories, as well as instancesof code occurrence and text intersection intersection/in��ter��sec��tion/ (-sek��shun) a site at which one structure crosses another. intersectiona site at which one structure crosses another. . Examiner reports werestandardised to a particular format so that comparable measures (basedon line counts) could be produced for a range of features of the reportsincluding proportions of text units coded by category, and the number,pattern and sequence of instances of coded text. All of the text unitsassociated with the examination of the thesis are coded at least at onenode (i.e., coding category). The core coding categories wereestablished after an extended process of trialling to establish categorystability and coder consistency; detailed coding notes were generated(see Holbrook & Bourke (2004) for further detail of the processdeveloped and followed). The core coding categories capture all of thecharacteristics and content of the reports. The core categories referred to in this paper are * examiner and process: such comments capture examiners'approaches to examination, the interpretation of their role and theirexpectations * assessable areas covered: this category captures all commentabout the possible outcomes, subject matter and presentation of thethesis under examination--the substantive elements of the thesis and theproject at its core, e.g., scope and significance, literature, approach,analysis and reporting, communicative competence Communicative competence is a linguistic term which refers to a learner's L2 ability. It not only refers to a learner's ability to apply and use grammatical rules, but also to form correct utterances, and know how to use these utterances appropriately. , and publicationsarising from the study * dialogic di��a��log��ic? also di��a��log��i��caladj.Of, relating to, or written in dialogue.dia��log elements: there are specific features of examinerdiscourse that reflect on the nature of academic communication. Inparticular this category identifies the notion of 'active'dialogue--engagement with, and consciousness of, communicatingpersonally with the reader(s). * evaluative elements: this category captures all comment thatcontains evaluation and judgement, including different types ofinstructive/formative comment as well as text identifying positive andnegative summative Adj. 1. summative - of or relating to a summation or produced by summationsummationaladditive - characterized or produced by addition; "an additive process" judgements about various aspects of the thesis andthe candidate's capacity to conduct research. In order to answer the questions posed in this paper we drew on thequantified core data and examiner recommendation and committee decision. The examiner recommendation There is a strong similarity Similarity is some degree of symmetry in either analogy and resemblance between two or more concepts or objects. The notion of similarity rests either on exact or approximate repetitions of patterns in the compared items. between Australian universities in theway examiners are asked to provide a recommendation. Essentially thereare five categories ranging from (1) 'accept the thesis assubmitted', through (2) minor correction (invited or required), (3)requiring more substantial correction, (4) requiring the thesis to berevised and resubmitted for further examination, to (5)'fail', the latter without the possibility of resubmission forfurther examination. (In some cases there were additional categoriesthat could be readily collapsed into these five categories; in otheruniversities only four categories are used with categories 2 and 3 beingcollapsed, but none of the universities in our sample used only fourcategories.) As the examiners act independently, clearly there will betheses for which examiner recommendations vary widely across the fivecategories. The university committee is then faced with a decision thatis typically not straightforward. The content of the written reports isconsidered closely in determining their decision. When there are widely different recommendations made by examinerson a thesis, there are at least two types of discrepancies possible.First, an examiner may recommend acceptance or only minor change to athesis, while other examiners recommend that the thesis be revised andresubmitted, and the university committee may decide that the thesis isto be revised and resubmitted. It is the committee's view thatthese theses are marginal, at best. In this case it seems that theexaminer making the favourable recommendation has not influenced thedecision made by the university (categorised Adj. 1. categorised - arranged into categoriescategorizedclassified - arranged into classes as Group 1 theses, seeTable 1).Thus Group 1 theses are those where at least one examiner hasrecommended that the thesis be accepted (perhaps after minor amendment)while the university committee decided the thesis should be revised andresubmitted. Secondly, for Group 2 theses, there is the opposite case ofat least one examiner recommending that a thesis be revised andresubmitted or failed while the university committee decided that thethesis be accepted as submitted, perhaps with minor change. It is thecommittee's view that these theses are, at least, satisfactory. Inthis case the recommendation of the critical examiner or examiners seemsto have been disregarded dis��re��gard?tr.v. dis��re��gard��ed, dis��re��gard��ing, dis��re��gards1. To pay no attention or heed to; ignore.2. To treat without proper respect or attentiveness.n. (see Table 2). Given our earlier work indicatedthat examiner comments differed markedly between high-quality andmarginal theses (Bourke et al., 2005), the subsequent comparisonsbetween the examiner text categories for these two groups were doneseparately. Comparisons of the content of examiner reports Once the relevant theses in the two groups were identified, thecontents of the examiner reports were compared using the percentages oftext coded at each of 29 subcategories of text codes. The overall lengthof each report was also considered, making a total of 30 possibledetailed points of comparison for examiner reports within the twogroups. The proportions of comment that the 'discrepant'examiners devoted to different aspects of the thesis were compared withthe content of the 'consistent' examiner reports in order todetermine whether there were any significant differences in theproportions of comment provided by these two groups of examiners. Giventhe disparities in the examiner recommendations, it was anticipated thatthe proportions of comment in the subcategories in the reports ofexaminers requiring revision or resubmission, or recommending failure('dissatisfied' examiners) would be significantly differentfrom those of examiners who judged the thesis to be acceptable or torequire only minor corrections ('satisfied' examiners). Results As shown in Tables 1 and 2, only in 33 cases out of a total of 2121examiners (i.e., less than 2 per cent) made recommendations that wereinconsistent according to according toprep.1. As stated or indicated by; on the authority of: according to historians.2. In keeping with: according to instructions.3. our definitions. It is also worth noting thatthere were slightly higher proportions of theses with inconsistentrecommendations in the broad field of study of engineering (9 per cent),arts, humanities and social sciences (7 per cent) and education (6 percent) compared with health (3 per cent) and science (zero). For Group 1 (i.e., the theses that were considered marginal), of atotal of 64 examiner reports for 22 theses, 24 reports were discrepant.Of the 29 subcategories of report comment, only 8 differed between thesatisfied and dissatisfied dis��sat��is��fied?adj.Feeling or exhibiting a lack of contentment or satisfaction.dis��satis��fied examiners. Most categories did not differbetween these examiners. Four subcategories that differed related to theassessable areas covered. (i.e., significance and contribution; commentsabout publications that might arise; literature coverage and analysisand reporting). There was more comment from the satisfied examinersconcerning significance and contribution of the thesis and potentialpublications, and less substantive comment on the content of theliterature review and on the analysis and reporting of findings. Theother four subcategories that differed related to the evaluativeelements of the reports. As would be expected, there was more positivecomment (summative and other) from the discrepant examiners who weresatisfied with the thesis and less formative instruction andprescriptive pre��scrip��tive?adj.1. Sanctioned or authorized by long-standing custom or usage.2. Making or giving injunctions, directions, laws, or rules.3. Law Acquired by or based on uninterrupted possession. comment. Overall, the satisfied examiners wrote muchshorter reports than the dissatisfied examiners of the theses that wererequired to be revised and resubmitted (see Table 3). For Group 2 (i.e., theses that were considered at leastsatisfactory), of a total of 32 examiners for 11 theses, 13 reports werediscrepant. For these theses, only three of the 29 subcategories intotal differed between the satisfied and dissatisfied examiners. Theonly assessable areas subcategory sub��cat��e��go��ry?n. pl. sub��cat��e��go��riesA subdivision that has common differentiating characteristics within a larger category. that differed was 'publicationsarising' where the dissatisfied examiners wrote less. There weretwo evaluative elements that differed: both summative and other positivejudgements where the examiners who were dissatisfied wrote less (seeTable 3). It seems that, contrary to expectations, the quantitative analysis Quantitative AnalysisA security analysis that uses financial information derived from company annual reports and income statements to evaluate an investment decision.Notes: of the report content for both groups revealed that there were fewdifferences between the reports of satisfied and dissatisfied examinersfor theses where there was at least one inconsistent examiner report.The overall lack of significant differences in the comparison of reportsfrom satisfied and dissatisfied examiners of the same thesis suggeststhat, although these groups of examiners made substantially differentrecommendations, the content of their reports was generally synchronous Refers to events that are synchronized, or coordinated, in time. For example, the interval between transmitting A and B is the same as between B and C, and completing the current operation before the next one is started are considered synchronous operations. Contrast with asynchronous. . Cases with a major inconsistency between the committee and themajority of examiners There were four thesis examinations where the committee decisionwas inconsistent with the recommendations of a majority of examiners.These instances exhibit the extremes of inconsistency between thecommittee and examiners. Although clearly a rare occurrence, these caseshave been selected for individual description of individual examinerreport content because, in looking at the extreme cases, we hope tounderstand more about the examination process. Marginal theses We first turn to reports on two of theses where twoexaminers ticked the box 'accept as submitted' but thecommittee agreed with a third examiner who ticked the 'revise andresubmit' box (see Table 1). For candidate 4014, when the text ofexaminer reports is considered, we find agreement in the content of the'revise and resubmit' report (Examiner 2) and one of the'accept' reports (Examiner 3). Both referred very centrally toa lack of depth and analytical analytical, analyticpertaining to or emanating from analysis.analytical controlcontrol of confounding by analysis of the results of a trial or test. strength. The examiner who recommendedthe thesis be revised and resubmitted produced a detailed andauthoritative report that demonstrated how the level of originality o��rig��i��nal��i��ty?n. pl. o��rig��i��nal��i��ties1. The quality of being original.2. The capacity to act or think independently.3. Something original.Noun 1. andcontribution of the thesis was questionable. Moreover, the otherexaminer (Examiner 1), who also recommended 'accept' providedsome very tentative tentative,adj not final or definite, such as an experimental or clinical finding that has not been validated. statements about the substantiveness of the work,for example noting some incoherence incoherenceNot understandable; disordered; without logical connection. See Schizophrenia. , over-generalisation and lack ofunderstanding of important points. For candidate 7007, the two examiners who recommended'accept' (Examiners 1 and 2) presented the type of positivesummative comment that is typical of reports on high-quality theses. Thereports are of average length and are complimentary about all the keyaspects of the thesis. They note originality, effectiveness of thepositioning of the study within the field, effectiveness of the methodsand the care with which they are detailed, and also the up-to-date andcompetently executed treatment of the literature. The other examiner whorecommended 'revise and resubmit' (Examiner 3) was stronglycritical of every aspect of the study, including the fact that thecandidate has misread mis��read?tr.v. mis��read , mis��read��ing, mis��reads1. To read inaccurately.2. To misinterpret or misunderstand: misread our friendly concern as prying. Examiner 3's own work. It is a veryaggressive, detailed and authoritative report. The other two reports bycomparison provide fewer specifics and virtually no clear'warrants' of examiner expertise. Satisfactory theses There were two cases where the committeedecided the theses required only 'minor correction' althoughexaminer recommendations were much harsher, being either 'reviseand resubmit' or 'fail' (see Table 2). The committee decision for Candidate 7006 was to invite correctionsdespite the recommendations of two examiners that the thesis should berevised and resubmitted (Examiners 1 and 2). The committee followed the'minor correction' recommendation of Examiner 3 but all threeexaminers found the thesis lacked interpretative in��ter��pre��ta��tive?adj.Variant of interpretive.in��terpre��ta and analyticaltreatment of the material. One examiner commented that a'compendium does not equate e��quate?v. e��quat��ed, e��quat��ing, e��quatesv.tr.1. To make equal or equivalent.2. To reduce to a standard or an average; equalize.3. to a doctoral thesis' (Examiner2). One possible explanation for the recommendation of Examiner 3 andits acceptance by the committee was a greater willingness to accept anunconventional thesis--one examiner acknowledging that it had been adifficult thesis to examine because of its non-traditional form--buteven Examiner 3 expressed a desire to see the theoretical sectionexpanded and more rigorously argued and a greater degree of coherence In optics, correlation functions are used to characterize the statistical and coherence properties of an electromagnetic field. The degree of coherence is the normalized correlation of electric fields. In its simplest form, termed tobe given to the work. Candidate 2041 presents a truly anomalous a��nom��a��lous?adj.1. Deviating from the normal or common order, form, or rule.2. Equivocal, as in classification or nature. case where the committeedecision to invite corrections is inconsistent with all threeexaminers' reports. Two of the reports recommended revise andresubmit Verb 1. resubmit - submit (information) again to a program or automatic systemfeed backreturn, render - give back; "render money" (Examiners 1 and 3) and the third report (Examiner 2)recommended major corrections. All three were scathing about theliterary quality of the thesis and what they saw as serious problemswith editing and presentation. All examiners provided pages ofcorrections extending well beyond a list of typographical errors. Twoexaminers were highly critical of shortcomings A shortcoming is a character flaw.Shortcomings may also be: Shortcomings (SATC episode), an episode of the television series Sex and the City and limitations in theexperimental methodology: one commented that this constituted a majorlimitation of the study. While there were a few positive comments, suchas the contribution the thesis makes to the field, such comments weretentative and seemed to be insufficient to outweigh out��weigh?tr.v. out��weighed, out��weigh��ing, out��weighs1. To weigh more than.2. To be more significant than; exceed in value or importance: The benefits outweigh the risks. the advice of allthree examiners that the thesis was fraught fraught?adj.1. Filled with a specified element or elements; charged: an incident fraught with danger; an evening fraught with high drama.2. with content, methodologicaland presentation errors. Why the committee overruled the expert opinionof all three examiners and allowed the work to stand without anyrequired corrections is inexplicable in��ex��pli��ca��ble?adj.Difficult or impossible to explain or account for.in��expli��ca��bil . Discussion of results The paper has focused on 'consistency' in examinationprocess and is one of the first studies to undertake a sustainedanalysis of examiner recommendations on the same thesis, and of therelationships between examiner recommendations and the officialcommittee decision. For the same thesis it can be anticipated thatinconsistency may occur between the recommendations made by examinersand also between the examiners and the committee decision. We found anextremely high level of consistency in examiner recommendations. Only 33of the 804 theses (4 per cent) had one or more discrepant examinerreports according to our definition. Of the 2121 examiners'reports, only 37 examiners (less than 2 per cent) showed a markeddiscrepancy from the other examiners and the committee. It is also worthnoting that there were slightly higher proportions of theses withinconsistent recommendations in the fields of arts, humanities andsocial sciences, education and engineering compared with health andscience, and that one institution showed more discrepancy than theothers. A total of four cases showed a discrepancy between examinerrecommendations and the university committee decision. Where inconsistencies were detected between examiners, there was atleast one discrepant examiner for 11 satisfactory theses, and 22 forvery problematic, that is, marginal theses. Even in this very smallnumber of cases, the language of examiners and the content of theirwritten reports were generally synchronous even if their recommendationswere discrepant. In providing guidance for PhD supervisors andcandidates, we have noted this previously, and have suggested that thewritten report is one of the strengths of the examination process,together with the independence of examiners (Bourke et al., 2007b, pp.239-240). With the exception of the four cases showing major inconsistency,the committee that adjudicated on the differences between examiners drewon the examiner comments rather than the inconsistent recommendations,and the official outcome indicated this. Of the four cases, only one isinexplicable. In the other three cases at least two of the examinerscommented similarly in their reports, so there was majority agreement inthe text even if the recommendations were anomalous. The one notableexception is by its very nature one of those that goes to reinforce the'myth' that the process is faulty fault��y?adj. fault��i��er, fault��i��est1. Containing a fault or defect; imperfect or defective.2. Obsolete Deserving of blame; guilty. , whereas the many more thatare not discrepant suggest that the multiple reporting mechanism provesto be extremely robust in facilitating the achievement of equitable equitableadj. 1) just, based on fairness and not legal technicalities. 2) refers to positive remedies (orders to do something, not money damages) employed by the courts to solve disputes or give relief. (See: equity) EQUITABLE. outcomes. The one case in 804 theses that does seem inexplicable perhapsshould not be asked to carry too much weight in the debate aboutexaminers' consistency. This notwithstanding, a note of concernneeds to be raised in relation to all four cases where the examiners andcommittee did differ considerably. From the point of view of assessmentethics, equity and fairness, this situation is a worst-case scenario worst-case scenarion → Schlimmstfallszenario ntandhighlights the importance of there being in place a clear code ofpractice and transparent procedures. There is no mandated curriculum for the PhD in Australia orelsewhere; thesis examination is as subjective as each thesis is uniqueand yet the findings of this study demonstrate the innate robustness ofthe 'invisible' doctoral curriculum and evidence ofconsistently applied standards. Kwiram (2006) in an essay produced forthe Carnegie volume on the future of the doctorate in the USA, notesthat while there are differences in expectations, quality andperformance across candidates, faculties, disciplines, departments andnations 'there seems to be a tacit understanding of whatconstitutes a well-prepared Ph.D. student', and that 'in thecomplete absence any central repository (1) A database of information about applications software that includes author, data elements, inputs, processes, outputs and interrelationships. A repository is used in a CASE or application development system in order to identify objects and business rules for reuse. of rules or a cosmic cos��mic? also cos��mi��caladj.1. Of or relating to the universe, especially as distinct from Earth.2. Infinitely or inconceivably extended; vast: accreditingagency' there is 'extraordinary stability' (p. 142).Drawing on the empirical evidence reported here for Australia, weconcur CONCUR - ["CONCUR, A Language for Continuous Concurrent Processes", R.M. Salter et al, Comp Langs 5(3):163-189 (1981)]. . The consistency between examiners and the stability in what theyare looking for Looking forIn the context of general equities, this describing a buy interest in which a dealer is asked to offer stock, often involving a capital commitment. Antithesis of in touch with. are remarkable and present some exciting and heartening heart��en?tr.v. heart��ened, heart��en��ing, heart��ensTo give strength, courage, or hope to; encourage. See Synonyms at encourage.Adj. 1. possibilities. In the first instance this finding has implications forcross-disciplinary thesis projects. There has been concern expressed inmany forums including the Carnegie Initiative on the Doctorate (Golde etal., 2006) that there are significant disciplinary impediments IMPEDIMENTS, contracts. Legal objections to the making of a contract. Impediments which relate to the person are those of minority, want of reason, coverture, and the like; they are sometimes called disabilities. Vide Incapacity. 2. tocross-disciplinary work being undertaken because of the'silo-like' nature of the disciplines but, in fundamentalways, thesis examiners are privileging the same generic qualities intheir reports. This suggests there is a solid foundation of consistency,agreement and translatability across disciplines to build on. In thesecond instance, the assessment discourse of examiners shows they sharethe same expectations about thesis quality and that, in Australia, thereare processes in place (e.g., independent examiner reports that are readby committee) that ensure these understandings are considered whenmaking judgements. The examination process enables examiners to set thestandards for PhD theses and the examiners seem to be successful atdoing this. What this suggests is that there is no reason to try towrite standards from first principles or impose them from without,because they can be reliably identified from what examiners say in theirextensive reports. At least one of the outcomes of our work that appliesexaminer commentary to notions of thesis quality would be thedevelopment of a set of stable indicators that allows the academiccommunity and others to distinguish between theses of threshold qualityand those of higher and highest quality (see also Lovitts, 2007). As aconsequence, a framework for thesis quality could be articulated andused to improve learning processes and dispel the mystery that hasprevailed for so long. This is currently being explored by the authors. Acknowledgements The research reported here was supported by an Australian ResearchCouncil The Australian Research Council (ARC) is the Australian Government’s main agency for allocating research funding to academics and researchers in Australian universities. Discovery Grant. References Bourke, S., Hattie, J., & Anderson, L. (2004) Predictingexaminer recommendations on PhD theses. International Journal ofEducational Research, 41(2), 178-194. Bourke, S., Holbrook, A. & Lovat, T. (2005, June). Usingexaminer reports to identify quality in PhD theses. Paper presented atthe AARE Aare(är`ə)or Aar(är), longest river entirely in Switzerland, 183 mi (295 km) long, rising in the Bernese Alps and fed by several glaciers. Focus Conference: Quality in educational research, Cairns Cairns,city (1991 pop. 64,463), Queensland, NE Australia, on Trinity Bay. It is a principal sugar port of Australia; lumber and other agricultural products are also exported. The city's proximity to the Great Barrier Reef has made it a tourist center. .Retrieved 17 January 2008 fromhttp://www.aare.edu.au/05papc/bo05011y.pdf Bourke, S., Holbrook, A., & Lovat, T. (2007a). Relationships ofPhD candidate, candidature and examination characteristics with thesisoutcomes. Paper presented at the Annual Conference of the AARE,Adelaide, November 2006. Retrieved 17 January 2008 fromhttp://www.aare.edu.au/06pap/bou06655.pdf Bourke, S., Holbrook, A. & Lovat, T. (2007b). Examiners andexamination outcomes. In C. Denholm & T. Evans Ev��ans, Herbert McLean 1882-1971.American anatomist who isolated four pituitary hormones and discovered vitamin E (1922). (Eds), Supervisingdoctorates downunder. Melbourne: ACER Press: Denicolo, P. (2003). Assessing the PhD: A constructive view ofcriteria. Quality Assurance in Education, 11(2), 84-91. Gilbert, R. (2004). A framework for evaluating the doctoralcurriculum. Assessment and Evaluation in Higher Education, 29(3),299-309. Golde, C., Walker, G., & Associates (2006). Envisioning thefuture of doctoral education: Preparing stewards of the discipline.Carnegie Essays on the Doctorate. San Francisco San Francisco(săn frănsĭs`kō), city (1990 pop. 723,959), coextensive with San Francisco co., W Calif., on the tip of a peninsula between the Pacific Ocean and San Francisco Bay, which are connected by the strait known as the Golden : Jossey-Bass. Grabbe, L. (2003). The trials of being a PhD external examiner.Quality Assurance in Education, 11(2), 128-133. Hoddell, S., Street, D., & Wildblood, H. (2002). Doctorates:converging con��verge?v. con��verged, con��verg��ing, con��verg��esv.intr.1. a. To tend toward or approach an intersecting point: lines that converge.b. or diverging di��verge?v. di��verged, di��verg��ing, di��verg��esv.intr.1. To go or extend in different directions from a common point; branch out.2. To differ, as in opinion or manner.3. pattterns of provision. Quality assurance ineducation 10(2), 16-70. Holbrook, A., Bourke, S., Fairbairn, H., & Lovat, T. (2007).Examiner comment on the literature review in PhD theses. Studies inHigher Education, 32(3) ((in press: accepted June 14, 2006). Holbrook, A., & Bourke, S. (2004). An investigation of PhDexamination outcome in Australia using a mixed method approach.Australian Journal of Educational and Developmental Psychology developmental psychologyBranch of psychology concerned with changes in cognitive, motivational, psychophysiological, and social functioning that occur throughout the human life span. , 4,153-169. Holbrook, A., Bourke, S., Lovat, T., & Dally, K. (2004a)Investigating PhD thesis e xamination reports. International Journal ofEducational Research, 41(2), 178-194. Holbrook, A. Bourke, S., Lovat, T., & Dally, K. (2004b).Qualities and characteristics in the written reports of doctoral thesisexaminers. Australian Journal of Educational and DevelopmentalPsychology, 4, 126-145. Retrieved 17 January 2008 from http://www.newcastle.edu.au/group/ajedp/Archive/Volume_4/v4-holbrook-et-al.pdf Holbrook, A., Bourke, S., Lovat, T., & Dally, K. (2004c). PhDtheses at the margin: Examiner comment of re-examined theses. MelbourneStudies in Education, 45(1), 89-115. Jackson, C., & Tinkler, P. (2001). Back to basics: aconsideration of the purposes of the Ph.D. Assessment and Evaluation inHigher Education, 26(4), 355-366. Lawson, A., Marsh, H., & Tansley, T. (2003). Examining theexaminers. Australian Universities Review, 46(1), 32-36. Kwiram, A. (2006). Time for Reform? In C. Golde, G. Walker andAssociates (Eds), Envisioning the future of doctoral education:Preparing stewards of the discipline. Carnegie Essays on the Doctorate.San Franscisco: Jossey-Bass. Lovat, T., Monfries, M., & Morrison, K. (2004). Ways of knowingand power discourse in doctoral examination. International Journal ofEducational Research, 41(2), 163-177. Lovat, T., Holbrook, A., & Bourke, S., (2007). Ways of knowingin doctoral examination: How well is the doctoral regime? EducationalResearch Review. http://dx.doi.org/10. 2016/j.edurev.2007.06.002 Lovitts, B. (2007). Making the implicit explicit: Creatingperformance expectations for the dissertation dis��ser��ta��tion?n.A lengthy, formal treatise, especially one written by a candidate for the doctoral degree at a university; a thesis.dissertationNoun1. . Sterling, VA: Stylus stylus:see pen. (1) A pen-shaped instrument that is used to "draw" images or select from menus. Styli (the plural of stylus, pronounced "sty-lye") come with handheld devices that have touch screens, such as PDAs and video games. Publishing. Morley, L., Leonard, D., & David, M. (2002). Variations inVivas: quality and equality in British PhD assessments. Studies inHigher Education, 27(3), 263-272. Mullins, G., & Kiley, M. (2002). 'It's a Ph.D., not aNobel Prize': how experienced examiners assess research theses.Studies in Higher Education, 27(4), 369-386. Nettles, M., & Millett, C. (2006). Three magic letters: Gettingto Ph.D. Baltimore Baltimore,city (1990 pop. 736,014), N central Md., surrounded by but politically independent of Baltimore co., on the Patapsco River estuary, an arm of Chesapeake Bay; inc. 1745. , MD: Johns Hopkins University Johns Hopkins University,mainly at Baltimore, Md. Johns Hopkins in 1867 had a group of his associates incorporated as the trustees of a university and a hospital, endowing each with $3.5 million. Daniel C. Press. Powell, S., & McCauley, C. (2002). Research degreeexamining--common principles and divergent di��ver��gent?adj.1. Drawing apart from a common point; diverging.2. Departing from convention.3. Differing from another: a divergent opinion.4. practices. Quality Assurancein Education, 10(2), 104-115. Powell, S., & McCauley, C. (2003). The process of examiningresearch degrees: some issues of quality. Quality Assurance inEducation, 11(2), 73-83. Shaw, M.. & Green, D. (2002). Benchmarking the Ph.D.-atentative beginning. Quality Assurance in Education, 10(2), 116-124. Sloboda, J. A., & Newstead, S. E. (1997, September). Guidelines guidelines,n.pl a set of standards, criteria, or specifications to be used or followed in the performance of certain tasks. for examiners: An evaluation of impact. Psychologist psy��chol��o��gistn.A person trained and educated to perform psychological research, testing, and therapy.psychologist, 407-410. Teddlie, C., & Tashakkori, A. (2003). Major issues andcontroversies in the use of mixed methods in the social and behavioralsciences behavioral sciences,n.pl those sciences devoted to the study of human and animal behavior. . In A. Tashakkori & C. Teddlie (Eds), Handbook For the handbook about Wikipedia, see .This article is about reference works. For the subnotebook computer, see . "Pocket reference" redirects here. of mixedmethods in social and behavioral behavioralpertaining to behavior.behavioral disorderssee vice.behavioral seizuresee psychomotor seizure. research. Thousand Oaks Thousand Oaks,residential city (1990 pop. 104,352), Ventura co., S Calif., in a farm area; inc. 1964. Avocados, citrus, vegetables, strawberries, and nursery products are grown. , CA: SagePublications This article or section needs sourcesorreferences that appear in reliable, third-party publications. Alone, primary sources and sources affiliated with the subject of this article are not sufficient for an accurate encyclopedia article. . Tinkler, P., & Jackson, C. (2000). Examining the doctorate:Institutional policy and the Ph.D. examination process in Britain.Studies in Higher Education, 25(2), 67-180. Tinkler, P.. & Jackson, C. (2004). The doctoral examinationprocess: A handbook for students, examiners and supervisors. Berkshire Berkshire(bärk`shĭr, –shər, bûrk`–)or Berks(bärks, bûrks), former county, S central England. ,England: Society for Research into Higher Education The Society for Research into Higher Education (SRHE) is an independent United Kingdom-based international society which aims to improve the quality of higher education. and Open UniversityPress. Allyson Holbrook is an Associate Professor of Education, AssistantDean (Research Training), Faculty of Education and Arts, and theDirector of the Centre for the Study of Research Training and Impact(SORTI), University of Newcastle University of Newcastle can refer to: Newcastle University, a university in the United Kingdom. The University of Newcastle, a university in New South Wales, Australia . Sid (1) (Society for Information Display, Santa Ana, CA, www.sid.org) A membership organization founded in 1962 devoted to the information display industry. With chapters around the world, SID hosts conferences in the U.S. and abroad and publishes a monthly magazine. Bourke is Professor of Education and Assistant Dean (Research)for the Faculty of Education and Arts, University of Newcastle, and amember of the Centre for the Study of Research Training and Impact. Terry Lovat is Professor in Education in the Faculty of Educationand Arts and a member of the Centre for the Study of Research Trainingand Impact, University of Newcastle. Hedy Fairbairn is Project Manager for the Centre for the Study ofResearch Training and Impact, University of Newcastle. Email:hedy.fairbairn@newcastle.edu.auTable 1 Committee decision 'Revise & resubmit'where one or more examiners recommended 'Acceptas submitted' or 'Minor correction' (n = 22theses with 24 discrepant examiner reports initalics)Candidate * Committee Examiner 1 decision recommend1301 Revise & resubmit Revise & resubmit1509 Revise & resubmit Minor correction1708 Revise & resubmit Revise & resubmit1906 Revise & resubmit Accept as submitted4005 Revise & resubmit Fail4006 Revise & resubmit Accept as submitted4014 Revise & resubmit Accept as submitted5023 Revise & resubmit Fail5029 Revise & resubmit Minor correction5060 Revise & resubmit Fail6086 Revise & resubmit Minor correction6100 Revise & resubmit Accept as submitted7007 Revise & resubmit Accept as submitted7095 Revise & resubmit Revise & resubmit8004 Revise & resubmit Accept as submitted8020 Revise & resubmit Revise & resubmit8051 Revise & resubmit Major correction8056 Revise & resubmit Major correction8059 Revise & resubmit Revise & resubmit8067 Revise & resubmit Revise & resubmit8073 Revise & resubmit Accept as submitted8090 Revise & resubmit Revise & resubmitCandidate * Examiner 2 Examiner 3 recommend recommend1301 Minor correction Revise & resubmit1509 Revise & resubmit Fail1708 Revise & resubmit Accept as submitted1906 Revise & resubmit Revise & resubmit4005 Accept as submitted Revise & resubmit4006 Revise & resubmit Revise & resubmit4014 Revise & resubmit Accept as submitted5023 Minor correction No 3rd examiner5029 Revise & resubmit No 3rd examiner5060 Fail Minor correction6086 Revise & resubmit Revise & resubmit6100 Fail Revise & resubmit7007 Accept as submitted Revise & resubmit7095 Revise & resubmit Accept as submitted8004 Revise & resubmit Revise & resubmit8020 Minor correction Major correction8051 Revise & resubmit Accept as submitted8056 Minor correction Revise & resubmit8059 Revise & resubmit Accept as submitted8067 Minor correction Major correction8073 Revise & resubmit Revise & resubmit8090 Revise & resubmit Accept as submitted* The first digit of the candidate number indicates the university.Table 2 Committee decision 'Accept as submitted' or'Minor correction' where one or more examinersrecommended 'Revise & resubmit' or 'Fail' (n = 11theses with 13 discrepant examiner reports in italics) Committee Examiner 1Candidate * decision recommend2041 Minor correction Revise & resubmit2049 Minor correction Minor correction2080 Minor correction Revise & resubmit2090 Minor correction Minor correction3060 Minor correction Fail5008 Minor correction Minor correction5025 Minor correction Fail5026 Minor correction Fail7006 Minor correction Revise & resubmit8064 Accept as submitted Accept as submitted8087 Accept as submitted Revise & resubmit Examiner 2 Examiner 3Candidate * recommend recommend2041 Major correction Revise & resubmit2049 Revise & resubmit Accept as submitted2080 Accept as submitted Accept as submitted2090 Minor correction Fail3060 Minor correction No 3rd examiner5008 Fail Accept as submitted5025 Minor correction Accept as submitted5026 Minor correction Minor correction7006 Revise & resubmit Minor correction8064 Revise & resubmit Accept as submitted8087 Accept as submitted Accept as submitted* The first digit of the candidate number indicates the university.Table 3 Comparisons of comment by examiners with whom thecommittee agreed and disagreed (discrepant examiners):percentages of text where there was a significant difference(p<.05) between the two groups of examiners Satisfactory theses (Committee decision)Text sub-category code Examiners Discrepant& description agree (%) examiners (%)Significance and contributionPublications arising 4.0 0.1Literature coverageAnalysis & reportingSummative positive comment 16.0 3.0Formative instructionPrescriptive commentPositive other judgement 13.1 1.3Text units: report length (no. of lines) Marginal theses (Committee decision)Text sub-category code Examiners Discrepant& description agree (%) examiners (%)Significance and contribution 5.7 11.1Publications arising 0.4 4.9Literature coverage 11.6 5.8Analysis & reporting 43.7 28.1Summative positive comment 3.3 24.0Formative instruction 33.3 16.4Prescriptive comment 18.1 6.4Positive other judgement 2.8 9.1Text units: report length (no. of lines) 211 109