Bias Geographic Location of Math National Examination in Junior High School: Analysis of Differential Item Functioning (DIF)

Indonesia is a country that is rich in human resources, natural resources, ethnic groups, regional languages, islands, and many others. In addition, Indonesia is also a country intensively making efforts to equalize development in various sectors such as infrastructure development and various quality improvement efforts such as education to compete with other countries. Indonesian students actively participate in assessment activities to find out Indonesia's position at the international level [1]. Such as the Program for International Students Assessment (PISA) and Trends in International Mathematics and Science Study (TIMSS) to determine the ability of Indonesian students nationally. NAs are carried out periodically, evaluated, and standardized every year.

Various countries use individual data to determine the progress of learning outcomes for each student [2, 3]. One way that is done by several countries is to focus on the results of NE such as Japan, Bangladesh, Uruguay, and several other countries [4]. Of course, each country has a different standard and focus [5, 6]. In Indonesia, a national exam evaluates student learning outcomes implemented since 2005, focusing on language, math, and social science scores [7].

One of the subjects that focus on the NE in Indonesia is mathematics; Indonesia actively participates in activities and the national exam. However, assessments such as PISA TIMSS show that the scores obtained by Indonesia are consistently below the average [8]. According to PISA data, Indonesia only got a math score of 386 out of an average of 490 in 2015 [9]. TIMSS data shows that Indonesia got a score of 386 for an average 500 in 2011 [10]. This figure is still far different from neighboring countries such as Malaysia, Vietnam, and Thailand. This fact is a question mark for us because the quality of Indonesian education, especially the results of mathematics exams, is very far from the target in terms of international scale.

Indonesia has many potentials exciting to study education in various conditions, such as geography, ethnicity, and religion. With situations that are unique and diverse. The position also affects education in Indonesia, one of the gaps in the quality of education in geography terms [11]. The location between western Indonesia is close to the center of government compared with the eastern part of Indonesia [3]. So the NE into the pros and cons by various circles in Indonesia, mainly when used as the sole criterion NE graduation for students and the determination of the quality of education a particular area so that impact a variety of fraud [12]. It could be due to one of them because there is inequality in education quality mapping of geo. It can be seen from the test results of the subjects tested, especially the NE in 2013/2014. it is shown that the difference value of mathematics courses is striking between the provinces of Daerah Istimewa Yogyakarta (DIY)=6.25 and Nusa Tenggara Timur (NTT)=5.02. This information is based on data from the education assessment center in 2014.

According to Muttaqin [13], the Indonesian government has made various efforts, such as establishing a 12-year compulsory education policy, raising the standard for national exam scores, and decentralizing education. This effort aims to achieve the target of universally equitable education, but the goal of increasing access and quality of education, while reducing inequality, is still far from being expected. Some forms of inequality that occur in Indonesia include:

First, inequality in the group of students from rich and poor economic groups in the age range of 13 to 15 years, 96 percent of students from rich families complete their education in 7 years, while only 80 percent for students from poor households [13]. Second, in terms of access, there are still more than half a million children aged 7 to 15 years who have never set foot in school in their entire life, and a total of more than 1.7 million children drop out of school before completing the nine-year compulsory education [13]. Third, in terms of competence, 55 percent of Indonesian students scored below the average according to the Program for International Student Assessment (PISA), 43 percent averaged, and only 2 percent scored above the average [14]. Fourth, in terms of quality, national examination scores of students in private schools are still lower than their counterparts in public schools, and there is also large variation within and between Islamic private schools.

Based on this explanation, it is clear that the inequality that occurs generally comes from the quality and competence of students, but data from the education assessment center in 2014 shows that several regions, especially those from the western provinces of Indonesia, have better scores than the eastern regions of Indonesia. This is something that is interesting to study considering that there has been no special study that discusses the geographical factor of the education gap.

Since there is an educational gap in Indonesia, one of them is the student competency factor which is generally taken on average based on the UN scores. In fact, some western regions, such as DIY, which have good access to education, have an average math exam score above the national average, while eastern regions like NTT have an average math exam score below the national average the education assessment center in 2014. The mathematics exam given consists of four materials, namely arithmetic operations, geometry, algebra, and statistics and probability. The researcher wants to investigate what items are the source of bias between the western and eastern regions of Indonesia with several DIF detection methods.

The gap results of NE in western and eastern regions of Indonesia can occur because of bias in items about the NA. Discrimination occurs when an imbalanced item is an item about favorable score a particular group [6]. The gap results of the NE because of the inequality score between regions can be caused by test items and, therefore, not test items [15-17]. Inequality is caused by the test item that's called biased items. Hambleton et al. [18] express understanding that item bias of individuals who have the same abilities of different groups do not have the same probability of replying with the correct item [18], Items that could potentially bias then analyzed with logic and why these items are relatively more difficult for one group [19]. When an item is somewhat more difficult for a group, and these difficulties are irrelevant to construct tests, that item is biased [20]. Some experts are replacing the term bias items with names Differential Item Function (DIF) [17, 20, 21]. We can express biased items mathematically in terms of probability equations.

Inequality score strived reduced, or at least a significant detected and direction [22, 23]. The detection results can be used for correction or repair items in the past about dating. Thus the necessary measurement and analysis more in some material mathematical possibility of bias when answered by the students [24]. Terms in measuring the test device need to be valid and reliable to obtain the measurement results following what is calculated. To determine the quality of a measuring instrument psychometric test should be conducted on the devices [17, 25]. Experts have established psychometric criteria for a psychological measuring instrument to be declared a good measuring tool [15]. They can provide information that is not misleading.

Invariant identification of interesting measurements to be discussed today. Consistent or measure bias an item known as the Differential Item Functioning (DIF) [26] with a variety of methods that can be performed to identify the functioning of the different items to two or more distinct groups in a test/exam, e.g., methods Item Response Theory (IRT) and a non-IRT [27].

Conceptually, the DIF is said to appear on an item problem. Suppose participants have the same capability to construct measured by the test but from a different group. In that case, they have a different probability of answering the question correctly [28]. For example, measure the same construct only one ability or unidimensional and diverse groups such as men and women. Next [17] argued that an item shows DIF if the test taker has the same ability in a different group and does not have the same probability of responding well.

Various studies on DIF focus on gender bias, a second language, age, and culture [16, 29, 30], but little has been discussed about the geographical conditions. This layout lies within the region with the central government. A regional grouping in Indonesia was divided into three groups Western, Central, and East. This problem becomes a probability for researchers to look at the gap NE by regional groups.

Generally, there are two types of DIF [15] namely 1) uniform DIF and non-uniform DIF. Uniform DIF appears if the advantage of one group over another occurs at every level of ability, and 2) non-uniform DIF appears if the benefit of one group over another does not happen in every capacity. Suppose it is associated with interaction in the statistical analysis of variance. In that case, uniform DIF occurs if there is no interaction between the ability level of participants and group membership, and non-uniform DIF occurs if there is an interaction between the test participant's ability level and group membership [14]. To determine whether a DIF indicates an item or not, a DIF index is needed, which is an index that shows as strong as a DIF indication is in the item. In the context of item response theory, whether or not DIF occurs on a question item lies in the item response function (Item Response). Part) for the item in the group in question. The response function's curve is called the item response curve or characteristic curve. Research detects the existence of DIF on test items, with divided population into two groups, namely the focal group and the reference group. A focal group is a group that is investigated whether there are items that contain DIF in that group. A reference group is a comparison group.

Based on the conditions in the field, which show that the score inequality causes the gap in the quality of education to be one of the most exciting probabilities to study, namely the existence of a bias in the results or DIF NE items based on regional groupings. Furthermore, it is hoped that some items of NE questions that are biased between the western region will be represented from the NE results data for SMP DIY students and the eastern part represented by NTT. Of course, to obtain more accurate results, a DIF detection method will be carried out using the IRT and non-IRT methods.

The gap in the realm of education is a challenge in itself experienced by Indonesia, despite various efforts made, there are still some challenges in its application. In this study, the researchers attempted to analyze the gap in the math UN scores between the western region of Indonesia represented by DIY and the eastern region represented by NTT. The National Examination Response data for the 2013/2014 academic year by students from each region were analyzed for each item using the DIF detection method, both using the IRT and classical/non-IRT approaches. The results of the analysis then produce items that contain DIF, provide information on the effect/magnitude of DIF contained in each item, and provide information on the group that benefits from each item. The items that contain the DIF are analyzed further so that a complete picture is obtained of what materials need to be considered for the Mathematics National Examination questions and what steps need to be taken by policy makers. More complete results can be seen in the results and discussion in this paper.

Table 1. DIF detection Method using the R

* Means it can be applied to the package "DifR."

This procedure begins by dividing the data into several groups with a range of specific abilities (ability groups). Then the researchers create a table with two rows and two columns for each group's ability. Focal group (f) is a group harmed by the presence of DIF. In contrast, the reference group is the group that became the basis of comparison for assessing the presence of DIF. If there are M group's abilities, the researchers will compile the Table 1 above as M. The method of detecting DIF on this occasion uses three methods for non-IRT. Classical methods include the Mantel-Heanzel method, Standardization Method, and Logistic Regression Methods [28, 37]. Modern methods for utilizing the IRT approach are the method of Lord and Raju [28].

DIF detection methods were performed using the R program using the difR package [27, 28]. On this occasion, the data used dichotomy result NE Mathematics by comparing two groups of respondents, a group of students from NTT as a focal group and a group of students of Yogyakarta as the reference group. Researchers took 1000 samples from each group. In this Article, we analyze the 39 items because item 21 did not meet the analysis criteria (could not be answered correctly by the sample used) so it was excluded from the data analysis. The results of DIF analysis are continued by doing analysis UN items that contain DIF.  Early analysis is to examine the DIF signification of each item in general and determine the category of difficulty based on the criteria presented in Table 2.

Table 2. Item difficulty category

Furthermore, DIF analysis is carried out specifically using five selected methods. The results of analysis that produce statistical values based on each DIF method are classified into three levels, namely A, B, and C. This level shows the effect / magnitude of the DIF loaded by each item. Table 3 describes the statistical value limits of the method used.

Table 3. DIF Level interpretation based on five methods

The item about the NE in the study is grouped into four subject matters: arithmetic operation, Algebra, Geometry, and Statistics / Probability. The analysis next is the categorization and count of the number of items that are experiencing DIF are based on the following criteria (Table 4):

Table 4. Categorization of items containing DIF by five methods

3.4 Step analysis using the R. program

(1) Prepare data in extension *. CSV on Microsoft excel or can also be in the form of a file extension *. sav in the SPSS program on a specific file.

(2) Open the Program Application and settings directory by clicking FILE → Change dir as Figure 1. Find the folder that contains the folder that contains the files to be analyzed, then click OK.

qqtu_pian_20230517100559.png

Figure 1. Change directory

(3) Choose a Package that supports analysis difR, namely mirt, ltm, lme4, deltaPlotR; by clicking Packages → load package, a menu like a Figure 2 will appear. Then select the package to use and click Ok.

2.png

Figure 2. Selectable package download

(4) The next step is to import data by writing the following syntax: >data<-read.csv(file="DATA UN.csv ", header=T) #(data csv)>data<-read.spss(file="DATA UN.sav", to.data.frame =TRUE)#(SPSS data).

(5) Displays data by writing the following syntax: >data. Then the display will appear as shown in Figure 3:

tu_pian_3.png

Figure 3. Data analysis

(6) DIF analysis. The first method used is an analysis using the Mantel -Heanzel method. The syntax used is as follows: MH<-difMH (data, group="Group", focal.name=2, purify=TRUE, nrIter=20, save.output=TRUE, output=c("MHresults", "default")).

(7) Output analysis can be viewed by writing the following syntax in the R Console.

>MH

(8) R console output is as follows: >MH

Detection of Differential Item Functioning using Coat- Haenszel method with continuity correction and with item purification.

Results based on asymptotic inference

Convergence reached after five iterations

Matching variable: test score

No set of anchor items was provided

No p-value adjustment for multiple comparisons

Coat- Haenszel Chi- square statistics:

     stats. P- value    

it1 1.1462 0.2843

it2 13.7604 0.0002 ***

it3 1.6933 0.1932

it4 44.6885 0.0000 ***

it5 21.8019 0.0000 ***

it6 162.9540 0.0000 ***

it7 0.4404 0.5069

it8 12.7616 0.0004 ***

it9 0.0451 0.8318

it10 25.2172 0.0000 ***

it11 44.4657 0.0000 ***

it12 40.8461 0.0000 ***

it13 207.1702 0.0000 ***

it14 113.9957 0.0000 ***

it15 18.4669 0.0000 ***

it16 70.7143 0.0000 ***

it17 45.0143 0.0000 ***

it18 21.0977 0.0000 ***

it19 1.1558 0.2823

it20 0.0002 0.9886

it21 12.7633 0.0004 ***

it22 37.5032 0.0000 ***

it23 0.7144 0.3980

it24 6.2192 0.0126 *

it25 6.4163 0.0113 *

it26 3.1205 0.0773.

it27 35.7052 0.0000 ***

it28 53.8681 0.0000 ***

it29 0.1717 0.6786

it30 9.6330 0.0019 **

it31 87.9321 0.0000 ***

it32 20.9872 0.0000 ***

it33 0.3241 0.5691

it34 181.3080 0.0000 ***

it35 23.7732 0.0000 ***

it36 63.0233 0.0000 ***

it37 0.0006 0.9803

it38 0.0000 0.9952

it39 10.7247 0.0011 **

Significant. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Detection threshold: 3.8415 (significance level: 0.05)

Items detected as DIF items:

it2

it4

it5

it6

it8

it10

it11

it12

it13

it14

it15

it16

it17

it18

it21

it22

it24

it25

it27

it28

it30

it31

it32

it34

it35

it36

it39

Effect size (ETS Delta scale):

Effect size code:

'A': negligible effect

'B': moderate effect

'C': large effect

alphaMH deltaMH

it1 0.8700 0.3271 A

it2 0.6586 0.9813 A

it3 0.8486 0.3859 A

it4 0.4546 1.8524 C

it5 0.5767 1.2936 B

it6 0.2004 3.7775 C

it7 0.9176 0.2020 A

it8 0.6290 1.0896 B

it9 1.0307 -0.0711 A

it10 0.5474 1.4160 B

it11 0.4532 1.8600 C

it12 0.4558 1.8465 C

it13 0.1541 4.3941 C

it14 0.2633 3.1357 C

it15 0.5882 1.2471 B

it16 0.4045 2.1271 C

it17 0.4482 1.8860 C

it18 0.5965 1.2140 B

it19 0.8793 0.3023 A

it20 1.0085 -0.0200 A

it21 0.6523 1.0042 B

it22 1.9418 -1.5595 C

it23 1.1029 -0.2302 A

it24 0.7253 0.7548 A

it25 1.3416 -0.6907 A

it26 1.2263 -0.4795 A

it27 2.0019 -1.6311 C

it28 0.4218 2.0288 C

it29 1.0563 -0.1288 A

it30 1.4417 -0.8596 A

it31 3.2969 -2.8035 C

it32 1.6514 -1.1788 B

it33 1.0710 -0.1612 A

it34 0.12012 3.7683 C

it35 1.7805 -1.3556 B

it36 0.4083 2.1048 C

it37 1.0036 -0.0084 A

it38 0.9945 0.0130 A

it39 1.4475 -0.8692 A

Effect size codes: 0 'A' 1.0 'B' 1.5 'C'

(for absolute values of 'deltaMH')

The first paragraph shows the DIF detection method used, using the Mantel -Heanzel method, and purification carried out for up to 5 iterations. Its output is the statistical value chi-square, p-value, and the significance level of the item containing the DIF. The second output shows the items containing the DIF. Moreover lastly, it shows the estimated value of alphaMH, deltaMH, and the effect category of DIF for each item.

(9) Visually, items containing DIF using the Mantel-Heanzel method can be observed using the following syntax: Plot (MH). So that the output appears as in Figure 4:

tu_pian_4.jpg

Figure 4. Coat– Haenszel statistics and detection threshold

The same way can be done to specify DIF and Graph values for other methods with the following syntax:

# DIF detection using standardization method

STD<-difStd (data, group="Group", focal.name=2, purify =TRUE, nrIter=20, save.output=TRUE, output=c("Stdresults", "default"))

# Graphics devices 

plot (STD)

#DIF detection using logistics regression method

LOG<- difLogistic (data, group = "Group", focal.name = 2, purify = TRUE, nrIter = 20, save.output = TRUE, output = c(" Stdresults "," default "))

# Graphics devices

plot (LOG)

# DIF detection using Lord's chi-squared method

LORD<- difLord (data, group = "Group", focal.name = 2, model = "1PL", purify = TRUE,engine = "lme4", save.output = TRUE, output = c(" LordResults "," defaults "))

# Graphics devices

plot (LORD)

# DIF detection using Raju's area method.

RAJU1<- difRaju (data, group = "Group", focal.name = 2, model = "1PL", purify = TRUE,signed = TRUEsave.output = TRUE, output = c(" RAJUresults "," default "))

# Graphics devices

plot (RAJU)

Briefly, the steps of DIF analysis using Program R are presented in the flow chart in Figure 5.

tu_pian_4.png

Figure 5. DIF analysis using Program R

Based on the results obtained, it can be seen that there is a very significant bias in the 2013/2014 NE item items. More than half of the NE questions contain DIF, which indicates that the NE questions tend to benefit the focal group [15, 38], in this case, for groups of students in NTT (representing the eastern region). Furthermore, disadvantaged students in DIY (representing the western region of Indonesia). This condition indicates that the NE questions contain a relative bias significant by location within [11]. Racial and ethnic group differences may reflect actual differences rather than measurement bias [38] know that ethnicity may influence learning culture and learning opportunities [38, 39].

Several things cause differences in the ability of DIY and NTT students to solve the NE questions in terms of this location, such as differences in educational facilities and facilities and infrastructure of the two regions [5, 40]. This situation can be supported by the fact that NTT is one of the target areas for the Bachelor of Education program in the Outermost, Frontier, and Disadvantaged Regions (SM3T) to improve the quality of education in that area. In addition, several studies have shown that differences in student backgrounds such as language [41], socioeconomic, and geography have contributed to students' abilities [5]. Geographical differences in Indonesia are generally the things that affect the primary language used. It may result in differences in students' abilities to understand some of the instructions in the NE questions given [41].

Low ability focal group can be caused by facility factors and several things that might cause a lot of NTT students not to solve the NE questions [3, 5]. Another factor is the still weak ability of students in several learning materials, namely Algebra [42, 43], Geometry [44, 45], and Statistics/probability [46, 47] material. Of course, this needs to be a focus for the education office, schools, teachers, and all stakeholders [48, 49] to focus and concentrate on these materials so that students at NTT can solve problems related to these materials in the future [50]. The material with a reasonably high DIF in this analysis is the same as the TIMSS results in 2011, where Indonesia has a below-expected value for the three materials. Indonesia only has an increase in the material for arithmetic operations [10]. Another study stated that the primary thing that affects students' ability to solve math problems is the content ability [51, 52]. it is necessary to carry out several policies to improve the content abilities of NTT students, such as improving the quality of learning to improve students' knowledge of mathematical content.

The exciting thing in the study is that with existing limitations, the general focal group /NTT has more opportunities than group /DIY reference for items containing DIF at the same capability level. This situation shows that with good guidance and educational support facilities, the ability focal group /NTT has potency significantly could increase in the future [5]. This condition naturally needs good cooperation Among the government center and region to support progress in several areas in eastern Indonesia.

Some of the limitations in this study can be used by further researchers, such as the sample used only using two provinces, namely DIY and NTT. In the future, analysis can be carried out for several areas involving more than two groups. In addition, the analysis carried out by researchers is still shallow to reveal the items and materials that contain DIF. In addition to geographical factors, many exciting things can be done to understand the bias of the NE items, such as differences in the primary language, learning opportunities, socioeconomics, other factors that are more interesting to study [3, 21, 38].