A study of inter-rater reliability among novice raters of the BEL120 writing course in a university in Malaysia / Nurrul Che Eme Embong

In the field of writing assessment, various factors have been identified to influence the validity of examinees' scores. One of the most prominent factors believed to threaten scoring validity in an assessment is the raters. Raters played an important role in any type of assessment particularly...

Full description

Saved in:

Bibliographic Details
Main Author:	Embong, Nurrul Che Eme
Format:	Thesis
Language:	English
Published:	2013
Subjects:	Terengganu Language
Online Access:	https://ir.uitm.edu.my/id/eprint/14622/1/TM_NURRUL%20CHE%20EME%20EMBONG%20ED%2013_5.pdf
Tags:	Add Tag No Tags, Be the first to tag this record!

id	my-uitm-ir.14622
record_format	uketd_dc
spelling	my-uitm-ir.146222016-09-05T03:53:24Z A study of inter-rater reliability among novice raters of the BEL120 writing course in a university in Malaysia / Nurrul Che Eme Embong 2013 Embong, Nurrul Che Eme Terengganu Language In the field of writing assessment, various factors have been identified to influence the validity of examinees' scores. One of the most prominent factors believed to threaten scoring validity in an assessment is the raters. Raters played an important role in any type of assessment particularly ones that involve writing. The raters need to be reliable in their rating ability and awarding marks which will be used to determine the examinees ability. Previous research has shown that the most concerned group of raters would be the novice raters. This study seeks to investigate the inter-rater reliability of novice raters in rating using holistic and analytic scoring rubrics for writing assessment, particularly in rating expository essays for BEL 120 course in UiTM Dungun, Terengganu. The three novice raters chosen for this study were selected based on the same characteristics. The raters were asked to rate 30 expository essays using Test of Written English (TWE) holistic scoring rubric first and after an interval of two days, they were asked to rate the same essays using BEL 120 analytic scoring rubric used by the faculty to mark BEL 120 final exam. The marks of the essays were computed and analyzed using SPSS Version 18.0 for Windows to generate the results. Intraclass Correlation Coefficient was used to determine interrater reliability since it involved more than two raters. The results showed that the novice raters have low inter-rater reliability level for both scoring rubrics used. The scores discrepancies among the raters also varied greatly thus the raters have low raters agreement when awarding marks using both scoring rubrics. Although the raters were familiar with analytic scoring rubric, the results showed that the inter-rater reliability level for analytic scoring is lower than holistic scoring. The findings provide an insight on the actual level of novice raters' inter-rater reliability and appropriate action is hoped to be taken by Academy of Language Studies in the institution to increase the raters' reliability in rating writing assessment as to increase scoring validity of the examinees. 2013 Thesis https://ir.uitm.edu.my/id/eprint/14622/ https://ir.uitm.edu.my/id/eprint/14622/1/TM_NURRUL%20CHE%20EME%20EMBONG%20ED%2013_5.pdf text en public mphil masters Universiti Teknologi MARA Faculty of Education
institution	Universiti Teknologi MARA
collection	UiTM Institutional Repository
language	English
topic	Terengganu Language
spellingShingle	Terengganu Language Embong, Nurrul Che Eme A study of inter-rater reliability among novice raters of the BEL120 writing course in a university in Malaysia / Nurrul Che Eme Embong
description	In the field of writing assessment, various factors have been identified to influence the validity of examinees' scores. One of the most prominent factors believed to threaten scoring validity in an assessment is the raters. Raters played an important role in any type of assessment particularly ones that involve writing. The raters need to be reliable in their rating ability and awarding marks which will be used to determine the examinees ability. Previous research has shown that the most concerned group of raters would be the novice raters. This study seeks to investigate the inter-rater reliability of novice raters in rating using holistic and analytic scoring rubrics for writing assessment, particularly in rating expository essays for BEL 120 course in UiTM Dungun, Terengganu. The three novice raters chosen for this study were selected based on the same characteristics. The raters were asked to rate 30 expository essays using Test of Written English (TWE) holistic scoring rubric first and after an interval of two days, they were asked to rate the same essays using BEL 120 analytic scoring rubric used by the faculty to mark BEL 120 final exam. The marks of the essays were computed and analyzed using SPSS Version 18.0 for Windows to generate the results. Intraclass Correlation Coefficient was used to determine interrater reliability since it involved more than two raters. The results showed that the novice raters have low inter-rater reliability level for both scoring rubrics used. The scores discrepancies among the raters also varied greatly thus the raters have low raters agreement when awarding marks using both scoring rubrics. Although the raters were familiar with analytic scoring rubric, the results showed that the inter-rater reliability level for analytic scoring is lower than holistic scoring. The findings provide an insight on the actual level of novice raters' inter-rater reliability and appropriate action is hoped to be taken by Academy of Language Studies in the institution to increase the raters' reliability in rating writing assessment as to increase scoring validity of the examinees.
format	Thesis
qualification_name	Master of Philosophy (M.Phil.)
qualification_level	Master's degree
author	Embong, Nurrul Che Eme
author_facet	Embong, Nurrul Che Eme
author_sort	Embong, Nurrul Che Eme
title	A study of inter-rater reliability among novice raters of the BEL120 writing course in a university in Malaysia / Nurrul Che Eme Embong
title_short	A study of inter-rater reliability among novice raters of the BEL120 writing course in a university in Malaysia / Nurrul Che Eme Embong
title_full	A study of inter-rater reliability among novice raters of the BEL120 writing course in a university in Malaysia / Nurrul Che Eme Embong
title_fullStr	A study of inter-rater reliability among novice raters of the BEL120 writing course in a university in Malaysia / Nurrul Che Eme Embong
title_full_unstemmed	A study of inter-rater reliability among novice raters of the BEL120 writing course in a university in Malaysia / Nurrul Che Eme Embong
title_sort	study of inter-rater reliability among novice raters of the bel120 writing course in a university in malaysia / nurrul che eme embong
granting_institution	Universiti Teknologi MARA
granting_department	Faculty of Education
publishDate	2013
url	https://ir.uitm.edu.my/id/eprint/14622/1/TM_NURRUL%20CHE%20EME%20EMBONG%20ED%2013_5.pdf
_version_	1783733303227973632

A study of inter-rater reliability among novice raters of the BEL120 writing course in a university in Malaysia / Nurrul Che Eme Embong

Similar Items