THE IMPACT OF SETTABLE TEST ITEM EXPOSURE CONTROL INTERFACE FORMAT ON POSTSECONDARY BUSINESS STUDENT TEST PERFORMANCE

The purposes of this study were to determine if there is a significant difference in postsecondary business student scores and test completion time based on settable test item exposure control interface format, and to determine if there is a significant difference in student scores and test completion time based on settable test item exposure control interface format by gender. Results of the study indicate that there is no significant difference in postsecondary business student scores or test completion times based on settable test item exposure control interface format. When the variable gender was added, female postsecondary business students were found to achieve significantly higher test scores and to have significantly faster test completion times. Effect size and descriptive statistic analysis suggests that these differences by gender are too small to be of much practical difference.


INTRODUCTION
A search of the ERIC database reveals a keen interest in computer-based testing by researchers over the past 35 years.Indeed, a focused search of the ERIC database using the descriptor "computer assisted testing" from 1970 through 2003 returned 1,954 citations.More than half (55.6%, n = 1,105) of these 1,954 citations were dated from 1990 through 2003.This research interest in computer-based testing is likely a result of the many advantages associated with its use (Goldberg & Pedulla, 2002).A number of researchers have reported on the advantages of computer-based testing (e.g., Alderson, 2000;Alexander, Bartlett, Truell, & Ouwenga, 2001;Barkley, 2002;Bocij & Greasley, 1999;DeSouza & Fleming, 2003;Goldberg & Pedulla, 2002;Greenberg, 1998;Shermis & Lombard, 1998;Shermis, Mzumara, & Bublitz, 2001;Song, 1998;Stephens, 2001;Truell & Davis, 2003).Often cited advantages of computer-based testing include decreased testing costs, effective records management, increased assessment options, improved scoring precision, instant feedback to students, more instructional time, more test administration choices, and reduced testing time.Despite the many advantages associated with computer-based tests for student assessment purposes, there are several areas of concern associated with their use.Two areas of concern with computer-based test use are user interfaces and test item exposure control formats.

NEED FOR THE STUDY
In recent years there has been a growing use of computer-based testing systems in postsecondary education.This increased growth is associated with the many advantages of their use for assessing student performance.Despite this growth and reported advantages, researches have noted several issues of concern.Specifically, this concern has focused on the user interface and test item exposure control formats.Thus, the results of this study fill a gap in the literature by addressing research recommendation put forward in the literature.

PURPOSE OF THE STUDY
The purposes of this study were (a) to determine if there is a significant difference in postsecondary business student test scores and test completion times based on settable test item exposure control interface format (i.e., all at once, one at a time-backing up, and one at a time-no backing up) and (b) to determine if there is a significant difference in postsecondary business student test score and test completion time based on settable test item exposure control interface format (i.e., all at once, one at a time-backing up, and one at a time-no backing up) by gender.Thus, the following research questions were investigated.

Research Design
The counterbalanced, Latin square quasi-experimental design was used in this study.Specifically, the counterbalanced Latin square design was selected because ". . .experimental control is achieved or precision enhanced by entering all respondents (or setting) into all treatments" (Campbell & Stanley, 1963, p. 50).Additionally, this design controls for the majority of threats to internal validity (Campbell & Stanley, 1963).
Treatment order was determined by random assignment.The specific counterbalanced, Latin square design used in this study is illustrated in Table 1.

Participants
Participants were those postsecondary business students enrolled in three, intact sections of the same college of business core course at a Midwestern university.More specifically, 90 students participated in the study.The number of students participating in each class was 34, 32, and 24, respectively.

Data Collection Procedures
The commercially available computer-based testing system used during this study automatically recorded postsecondary business student test score and test completion time data.The three classes were taught by the same instructor, met in the same classroom, and were provided with the same instructional materials.Classes met on a three day per week schedule.All computer-based tests were completed in a computer lab located near the classroom.All tests were proctored by the instructor.Students were allotted 50 minutes to complete each 50-item multiple choice test regardless which settable test item exposure control interface format.

Data Analysis
To answer research questions one, two, three, and four, MANOVAs and post hoc ANOVAs were used to analyze the data.There were 34, 32, and 24 postsecondary business students enrolled in the three, intact classes involved in this study, respectively.The Latin square design assumes an equal number of participants in each class so data from 24 postsecondary business students in each of the classes enrolling more than 24 students was randomly selected for inclusion in the data analysis.In order to form each of the 24 Latin squares, postsecondary business students were randomly matched across the three classes.Since each Latin square contained four observations and there were 24 replications, the data set had 72 observations.Effect size and observed power are reported in the findings section.As Kotrlik and Williams (2003) noted "It is almost always necessary to include some index of effect size or strength of relationship in your results section . .." (p. 1).Effect size magnitude in this study was determined using Omega square (ω2) values.Kirk's (1996) procedure for interpreting ω2 effect size magnitude is used in this study.Tests of statistical significance were conducted at α = .05.

Research Question One
Research question one sought to determine if there was a significant difference in postsecondary business student scores based on settable test item exposure control interface format.Results of the MANOVA-Hotelling's Trace-analysis indicated that there was no significant difference in postsecondary business student test scores based on settable test item exposure control interface format.MANOVA and ANOVA analyses for research question one and their associated descriptive statistics appear in Tables 2 and  4, respectively.

Research Question Two
Research question two sought to determine if there was a significant difference in student test completion time based on settable test item exposure control interface format.MANOVA-Hotelling's Trace-analysis indicated there was no significant difference in postsecondary business student test completion time based on settable test item exposure control interface format.MANOVA and ANOVA analyses for research question two and their associated descriptive statistics appear in Tables 2 and 4, respectively.

Research Question Three
Research question three sought to determine if there was a significant difference by gender in student scores based on settable test item exposure control interface format.MANOVA-Hotelling's Trace-analysis indicated either a significant difference in postsecondary business student test score or test completion time by gender.Post-hoc ANOVA analysis F(1, 185) = 11.164,p = 0.001 indicated that there was a significant difference by gender in student scores based on settable test item exposure control interface format.Specifically, postsecondary business female students scored significantly higher than did male students based on settable test item exposure control interface format.The means and standard deviations for female and male postsecondary business students were 43.87 (SD = 3.74) and 41.56 (SD = 4.85), respectively.These means and standard deviation differences are too small to be of much practical significance, however.This lack of practical differences by gender in postsecondary business student scores is supported by the effect size for the analysis.The effect size for this analysis is ω2 = 0.036.A ω2 of <0.05 is considered a small effect size (Kirk, 1996).settable test item exposure control interface format.Specifically, female postsecondary business students achieved significantly faster test completion times than did male postsecondary business students based on settable test item exposure control interface format.The means and standard deviations for female and male postsecondary business students were 1269.31(SD = 282.006)and 1415.10 (SD = 363.452)seconds, respectively.These means and standard deviation differences are too small to be of much practical significance, however.This lack of practical differences by gender in student scores is supported by the effect size for the analysis.The effect size for this analysis is ω2 = 0.013.A ω2 of <0.05 is considered a small effect size (Kirk, 1996).MANOVA and ANOVA analyses for research question four and their associated descriptive statistics appear in Tables 3 and 4, respectively.

CONCLUSIONS AND DISCUSSION
The results of this study offer several conclusions.These conclusions, however, are offered with the caveat that this study appears to be among the first to examine the impact of various settable test item exposure control interface formats and that additional investigation is needed.First, there is no significant difference in postsecondary business student performance based on settable test item exposure control interface format.Specifically, postsecondary business student test scores and test completion times did not differ significantly regardless of settable test item exposure control interface format.Second, female postsecondary business student performance on both test score and test completion time were significantly different from their postsecondary business student male counterparts.This significant difference for both test scores and test completion time is likely of little practical difference.These conclusions are supported by data in Tables 1, 2, 3, and 4. The results of this study are consistent with the earlier work of Truell (2005) who examined if differences existed in student scores and test completion time based on two computer-based user interface and paper and pencil formats.Truell (2005) reported that there was no significant difference in student scores based on test presentation format.In addition, there was no significant difference in test completion times between the two computer-based user interface test formats.Interesting, when gender was included in the analysis, female students scored significantly higher and achieved significantly faster test completion times than did their male counterparts.Truell (2005), after examining the effect size and descriptive statistics for each analysis, noted that these significant differences by gender were likely of little practical difference.The practice implication resulting from this study is that postsecondary business faculty can proceed with using the various settable test item exposure control interface formats.This use of various settable test item exposure control interface formats should be done with caution until more research has been conducted into their potential impact on test performance, however.

RECOMMENDATIONS FOR FURTHER RESEARCH
Based on a review of the literature and the findings of this study, the following recommendations for further research are put forward.
1.This study should be replicated.Given that relatively few studies have examined test item exposure control interface procedures, it would be prudent to conduct additional research in a variety of settings.Such studies would provide additional insight into the impact of settable test item exposure control interface features available with the various commercially available computer-based testing systems.2. As new settable testing features become available, research should be conducted to determine their potential impact on postsecondary business student test performance.Such studies will provide insight as to the impact of evolving technology on postsecondary business student computer-based test performance.

Table 1 .
Illustration of the 3 x 3 Counterbalanced, Latin Square Design

Table 2 .
Analysis of Latin Square Design

Table 3 .
Analysis of Latin Square Design with Gender Added

score time) = Class X Test Item Exposure Control Format X Test X Replication X Gender Multivariate Tests
Maximum possible test score was 50 regardless of settable test item exposure control interface format; maximum possible test completion time was 50 minutes regardless of settable test item exposure control interface format; time recorded and analyzed in seconds.