Validity and Reliability of Students Perceptions on OBE Approach in Malaysian VC Using Rasch Model

: This pilot study aims to produce empirical evidence regarding the validity and reliability of instrument of the perception of vocational college diploma students towards the Outcome Based Education (OBE) approach. Validity and reliability were analysed using Rasch Model Measurement assisted by Winsteps 3.72 software. This research instrument contained 26 items and was distributed to 60 diploma students of vocational college at the Northern Zone. Validity analysis of the instrument was done through four functional testings. For reliability and separation of respondents, it was found that the individual reliability value was 0.91, while that for items demonstrated an item value of 0.94 and item separation index of 3.90. Results from the analysis of polarity item found that 25 items had a positive PMC value between 0.52-0.83. Meanwhile, analysis on item fit found 23 items with an outfit mean-square value between 0.41 and 1.24. This situation suggested three items that require attention. In the analysis on local dependence that determines dependent items based on the standardised residual correlation value, it was shown that the correlation value for the items used did not overlap with that of other items. These findings provided the evidence that the instrument of perception of vocational college diploma students on OBE approach has a high level of validity and reliability to be used in actual studies.


INTRODUCTION
Outcome Based Education (OBE) is an educational approach that focuses on the empirical characteristics of performance and achievement. In 2007, OBE was compulsory in the Malaysian Institutions of Higher Learning. However, the OBE approach was only widely used in vocational colleges since 2016. At that time, the debate on issues in the implementation of OBE was seriously discussed by all parties, especially in the Vocational Technical Education and Training Division. The use of OBE is intended to meet the standards in ensuring that diploma programs in Vocational Colleges are recognised by the Malaysian Qualifications Agency (MQA). The accreditation from professional bodies offer significant recognition to the schools and/or programmes for their appropriate and effective assurance of learning process (Sun & Lee, 2020).
The OBE principles are adapted from the American Accreditation Board of Engineering Technology (ABET), which focuses on measuring learning outcome (LO) for each course with the results obtained act as the basis for continuous quality improvement (Jaafar et al., 2008). According to Sa'adah, et al., (2016), this paradigm shift is driven by the MQA at the Ministry of Education, Malaysia (MoE). This agency is responsible for developing and implementing the Malaysian education master plan, which outlines nine learning outcomes (LO). OBE gives students the freedom to learn using appropriate methods. In OBE, the key content needs to be identified first, which is followed by methods to study it (Borsoto et al., 2014). Therefore, this study was conducted to obtain an initial illustration of the perception of the Malaysian Vocational College diploma students on their understanding and readiness toward the OBE approach. Prior to the actual study, a pilot study was conducted to ensure that the instrument used has a high level of validity and reliability in measuring students' perception of OBE. Questionnaires that have been answered by the respondents in the pilot study were diagnosed through the process of purification and removal. Through this approach, function testing on items was done in depth through several analyses using Winsteps software to analyse quantitative data using the Rasch Model Measurement (RMM) method.
According to Kim and Helms (2016), OBE allows higher education institutions to improve students' learning experience and serves as a guide to enhance the quality and effectiveness of the programmes and modules. An effective OBE system allows institutions to plan, report and monitor study programmes and modules. Implementation of the OBE system may grant the required accreditation and give confidence to stakeholders on the quality of graduates and programmes in these institutions (Sun & Lee, 2020).
Outcome based education (OBE) is an educational approach that uses results in a curriculum driven by learning outcomes (LO) that need to be demonstrated by students at the end of the course (Davis, 2003). This explains that OBE focuses on the knowledge acquired by students at the end of a course and how students can produce something with what they have learned. Spady (1994) proposed three opinions related to the basic concepts of OBE namely i) all students can learn and succeed but not on the same day and not in the same way, ii) successful learning promotes even more successful learning, and iii) schools control the conditions that directly affect successful learning. Tucker (2004) argued that OBE is a process that involves reactions in the field of education to reflect student achievement and not just for the purpose of collecting credit in a course. Therefore, to evaluate OBE, accurate assessment methods need to be chosen to ensure that student achievement can be measured accurately. Therefore, course planning and evaluation methods designed by lecturers need to have the ability to support the learning outcomes that have been set.
Diploma students at the Malaysian vocational colleges are those who use the OBE approach in the teaching and learning process. This is because OBE serves as a benchmark to measure the success of an institution of higher learning (Mohayidin et al., 2008). OBE is able to make students i) able to analyse and synthesise information, ii) creative and innovative, iii) prone to entrepreneurship, iv) able to plan and organise assignments and v) able to work in a team. According to the fourth edition of Kamus Dewan Malaysia (2015), perception can be defined as a picture or shadow in the heart or mind regarding a point of view. Meanwhile, awareness can be defined as knowing and remembering the real situation.
Perception and awareness are two things that have a relationship in explaining a view or opinion. This study aims to examine students' perception as well as their awareness of the OBE approach that has been adopted in curriculum delivery in vocational colleges. The analysed aspect of perception was the result of preliminary studies conducted by researchers as well as literature review that has been conducted. According to Rhaffor et al. (2017), students in institutions of higher learning realise that the OBE approach is focused on student achievement besides knowing about the concepts of Program Learning Outcomes (PLO) and Program Educational Objectives (PEO). In addition, students have a view on the commitment and willingness of lecturers in implementing the OBE approach in curriculum delivery. In the context of this study, the researcher wants to study the perceptions of vocational college students in Malaysia from the aspect of awareness of OBE as well as their views on the commitment of lecturers in curriculum delivery.
Measurement is an important process in developing instruments because the methods used influence the form of research. According to Azrilah, et al., (2017), Rasch Model Measurement (RMM) refers to an idea, principle, guideline or technique that allows a measurement to be made against latent traits. The RMM does not only refer to the highest percentage or amount of score, but also allows the creation of a scientific measurement scale that in turn would allow weight or height to be made. The concept of RMM has been discussed in detail by Ewing, et al. (2005) as well as Nornazira (2016) in which measurement is referred to as a process of aiming to find ratios rather than giving numbers. Zhang and Zhing (2015) outlined RMM prescribing the use of Rasch measurements in the development and validation of instruments that emphasise the validity and reliability of the measured constructs.
According to the book entitled Basic Rasch Model Measurement written by Azrilah, et al. (2017), Rasch Model Measurement begins with a concept put forward by a Danish mathematician named Georg Rasch. He is a student of R.A. Fisher and Ragnar Frisch; both are well-known mathematicians in their respective fields namely Fisher Exact Test and Frisch Probability Theory. In the early stages of his involvement in his career, he had difficulties in finding a job as a mathematician, which later caused him to venture out as a statistical consultant. Rasch Model Measurement is one of the simplest response models since it has its own mathematical properties namely parameter model, item difficulties and examinee ability. RMM is also an adequate statistical method. Rasch pointed out that RMM can meet the measurement criteria that can be inferred just like the analysis done in the field of physical science.
Measurement, evaluation and analysis are branches in the field of statistics that are also important in the field of education. OBE has raised awareness among academicians that it is important to have a tool that can conduct accurate measurement. This situation is more conducive to the quantity and quality of learning and teaching services in addition to the results from both aspects. Most academicians become frustrated to find that most of the existing instruments are not suitable to be used in the tasks required as these instruments are unable to provide meaningful, accurate or authentic results. This opinion is in line with the study of Ismail and Mustaun (2019) who agreed that the concept of RMM allows all respondents to have a high possibility to answer all items correctly and simple items are more likely to be answered correctly by everyone.
In the context of this study, RMM was utilised to measure the construct of an instrument used for examining students' perception of the OBE approach in vocational colleges. The construction of instrument was the focus to be measured in this study. The items used to test the construct of this study comprised student perception, student awareness, views on the commitment and readiness of lecturers. This indicates that setting the correct construct will lead to proper analysis and evaluation. This situation in turn will lead to the right decision making. RMM was used to analyse the summary statistics for items, as well as statistics for item fit and dimensional testing (Ismail & Mustaun, 2019).
A pilot study is a small-scale study carried out before the actual study is done with the aim to observe the feasibility or reasonableness of a study to be done . Pilot study also serves to review instructions, test items in questionnaires and estimate response time (Lim, 2007;Bond & Fox, 2007). The objective of conducting the current pilot study is to examine the functionality of items from the aspects of (i) reliability and separation of item-respondent; (ii) polarity item; (iii) item fit and (iv) standardised residual correlation value in determining the dependent items.

METHODS
The study design involved in this study was a survey study using quantitative methods. Quantitative data were collected using a set of questionnaires reported by the RMM approach. This questionnaire has 26 questions that lead to students' perception of the OBE approach used in vocational colleges. In this study, two types of item were used in the research questionnaire. Researchers selected multiple-choice items and four-point scale items. Questions that used multiple-choice items consisted of gender, zone of vocational college, year of diploma and student category. Whereas to analyse the data from the study respondents, four-point scale items were used. Table 1 shows the distribution of questionnaires used in this study.

Population and Sample
For the purpose of this study, a pilot study was conducted with the aim of testing and building the effectiveness of the research tool in terms of language, comprehension of respondents and length of the statement used. In this study, a pilot study was conducted on 60 Diploma students of the Malaysian Vocational College in the northern zone. The selection of number of respondents has taken into account several opinions such as Linacre (2005) and Johanson and Brooks (2009) who suggested 30 respondents as well as Wolf (1997) who suggested a total of 30 to 50 respondents. This study used 60 respondents to ensure that the instruments used for the actual study are consistent to measure each item.

Research Procedure
Further discussion is related to the procedures in instrument construction and in conducting a pilot study. The first phase was related to the construction and measurement of instruments that have been carried out. Researchers also explored and analysed theories and models related to the study of OBE approach in the process of building research instruments. The instruments in this study were developed by the researcher based on Borich's (1980) needs assessment model to assess students' awareness and views on OBE. Students' perception of OBE was developed from previous research. The analysis was done by examining the relevant documents and interviews with Vocational College diploma students. This study used a set of questionnaires as a research instrument. Wolf (1988) asserted that a good questionnaire should be easy to read and understand by the respondents especially on the meaning of its items. This opinion is supported in Azhari's (2016) study, which stated that the use of questionnaires becomes very effective if the construction process is good with reliable and consistent items. After that, the researcher conducted the construction and validation of research instruments, population selection as well as a pilot study. During this process, the researcher applied for permission to conduct research to the Vocational Technical Education and Training Division. Permission from this party is important to allow all matters of the study to run smoothly.

Data Analysis
Winsteps Software Version 3.72 was used to analyse quantitative data using the Rasch Model Measurement method. This software was used to obtain accurate analysis information on the instrument formed. Statistical analysis conducted for the pilot stage is aimed at selecting the appropriate and best items (Cohen & Swerdilik, 2002). Rasch Model Measurement is able to identify the level of difficulty of the questionnaire items used in a study. In addition, this model can consider the ability and capability of the respondents who answered the instrument. In this study, Rasch Model Measurement was used as an approach to analyse the pilot study data for examining the functionality of items for each study construct through four aspects that can be explained through Table 2.

RESULT AND DISCUSSION
This section reports the findings of the analysis of pilot study related to the validity and reliability of the instruments used in this study.

Reliability and separation of item-respondent
Interpretation of Cronbach's Alpha values with a range between 0.00 to 1.0 is a value for the level of reliability in a study. The level of reliability of an item is said to be at a good, high and effective level when the range value approaches 1.0, whereas the range approaches the value of 0.00 to reflect a low level of reliability (Yusof, 2004). Table 3 shows the guidelines for analysing the data from a pilot study.  (2003) The reliability index of the respondents can be described as the ability of the individual to answer different sets of questionnaires in which each set measures the construct that is consistent. This means that all respondents have the same meaning of thinking about the same item. Meanwhile, the reliability index of an item refers to the similarity in terms of item difficulty compared to other samples with equivalent abilities. Therefore, in this study, the reliability of the items and respondents showed a compatible relationship between the items and the Rasch model.
The individual reliability value of 0.91 with individual separation index of 3.12. This shows that the individual abilities for all constructs were consistent. Linacre (2005) argued that a separation index exceeding 2.0 is good. The respondents' separation index showed the number of strata of respondents' ability to give the perceptions identified in the sample group, which were measured on 2 standard errors. This situation was based on the level of variables measured referring to differences in individual groups. Meanwhile, the reliability value of item demonstrated an item value of 0.94 and item separation index of 3.90. The item separation index, on the other hand, refers to the number of strata of item difficulty in 2 standard errors obtained on the set of tests used, which indicates the separation for the item difficulty level. Azrilah (2013) asserted that sufficient items to measure things to be measured are described as item reliability value.

Polarity item
Analysis of value of Point Measure Correlation (PTMEA CORR) is used to detect the polarity item in testing the extent to which the construction of a construct achieves its goal. If the PTMEA CORR value is positive (+), it indicates that the item has successfully measured the construct of interest (Bond & Fox, 2015). If the value obtained is otherwise negative (-), it means that the item developed has failed to measure the construct of interest. Therefore, it needs to be repaired or dropped as the item does not lead to a question or is difficult for the respondents to answer.
There is an item that has a negative value (-ve), which needs further action whether to be dropped or repaired. In the context of this study, the researcher decided to improve the item and ensure that it will move in parallel with other items.

Item fit statistics
Item fit to measure developed constructs is able to see item fit in measuring constructs as observations on study data. In the Rasch analysis report, the statistics used were known as (chi-square), which is infit and outfit mean square (MNSQ). MNSQ infit and outfit indices are always referred for item fit assessment (Azhari, 2016). Azrilah et al. (2017) defined MNSQ infit as sensitive or a match that fits the pattern of response to the targeted item and respondent. MNSQ infit will report an excess of matches from Guttman's response pattern. This outfit pattern is difficult to diagnose and overcome, so it can be completely ignored. MNSQ outfit, on the other hand, refers to an outlier match that works to help tracking the response to items that are too difficult or too easy for the respondent. Outfit reports an excess of matches to a given response, a lack of matches for stress and negligence that can be easily diagnosed and overcome. In the context of this study, the researcher decided to use the outfit since the results of this study found that outfit was easier to control than infit. This is because the outfit is a factor that bothers the respondents to answer the instrument. Therefore, the researcher thought of outfit as something that can be controlled. In this study, there was one item that was dropped and three items that were refined by looking at the needs of the study and views of experts. There were several references and opinions related to determining the value of MNSQ for fit value as shown in Table 7.

The standardised value of residual correlation in determining dependent items
The measurement of standardised residual correlation values aims to determine whether or not there are items that act on each other and are not singular. This is because the items have similar features to each other or both combine several shared dimensions. If the correlation value of (2) items exceeds 0.7, only one item is required and maintained for each pair of items involved. Item selection also refers to the MNSQ value, where a value close to 1.00 will be maintained (Linacre, 2010). Unidimensional measurement construct can be produced by items that are independent or singular (Wright & Masters, 1982).
The standardised residual correlation values. Findings showed that the correlation value for the item used detected no items overlapping with other items. Undetected items also have similar characteristics to each other. Once the data were analysed, item review was conducted based on the standard index as well as the conditions that need to be followed to achieve the standard validity and reliability of the instrument based on RMM. The process of removing and refining items were carried out with reference and taking into account the views and evaluations of experts. The results of this pilot study found one item to be removed while the other three items were purified.

CONCLUSION
Based on this pilot study, it can be explained that the validity and reliability of the instrument of students' perception of OBE approach has a quality that can be used for actual study. This pilot study is an initial step in assisting researchers to identify the perception of vocational college diploma students from the context of their readiness and views on the OBE approach. The implications of this analysis can help researchers in developing instruments that are truly capable of measuring the perception of vocational college diploma students towards the OBE approach. The use of RMM in analysing this instrument has been observed able to obtain a set of questionnaires that can measure the objectives in the actual study. Hence, it is crucial to conduct tests on the functions of item in terms of (i) reliability and separation of itemrespondent; (ii) polarity item; (iii) item fit and (iv) the standardised residual correlation value in determining the dependent item before conducting a study. Overall, the objective of this study has been achieved to produce a set of questionnaires with consistency to be applied. It is hoped that the actual study that will be implemented can measure the perception of VC diploma students on the OBE approach that has been adopted in vocational colleges. This study is very important to the Vocational Technical Education and Training Division, especially the vocational colleges. Through the findings of the actual study, VC can implement various appropriate programs to increase readiness and a good overview on the OBE approach.