Homepage LogiPass articles

Online employee screening

Internet-Based Employee Assessment Tests - Key Issues

In recent years, many organizations have begun implementing internet-based assessment systems, where job candidates are tested via the internet through assessment tests.

Internet-based assessment tests were preceded by computerized assessment tests, which began several decades ago, and with technological advancement - enabled the creation of various psychometric tests compared to paper and pencil tests. Internet-based psychometric assessment systems began to appear mainly in the last decade. They look similar to regular computerized applications and from the user's perspective, the experience is identical, but internet-based systems have two main advantages: First, internet-based systems have an advantage in maintenance. System maintenance is easier, as it is installed in only one location (on an internet server) and not on a large number of local computers. The system can be implemented remotely, and there is no need to visit the organization for maintenance, updates, etc. The second main advantage is enormous cost savings: reducing costs associated with maintaining separate systems at different sites, costs of different operating systems and hardware, specialization in different systems, etc. No special technological adaptation is required from the organization interested in using the system.

In addition to their technological advantages, internet-based assessment systems allow organizations to streamline employee selection processes in several ways. First, organizations that perform initial screening through the internet save significant money on lengthy selection processes for candidates who do not meet the minimum threshold requirements. The cost of internet-based assessment systems is generally lower than the cost of selection processes conducted in the organization itself or in dedicated employee assessment institutes. The economic cost of bringing an unsuitable candidate to an expensive selection process, when this cost can be saved through early internet screening, can reach hundreds of shekels per candidate. Second, through internet screening, a very large number of job candidates can be assessed in a relatively short time. Unlike selection processes that require the involvement of a human assessor (such as an interviewer or examiner), an internet-based psychometric assessment system can test dozens, hundreds and thousands of employees simultaneously. Test results arrive within seconds of completing the test to the screening authority in the organization, and there is no need to waste precious time waiting for results. A third advantage of internet tests is the high level of accessibility to candidates. Today's work world is characterized by globalization as well as job seekers who are employed (as opposed to unemployed job seekers). Their job search needs are different than in the past. The testing process coordination is simple, the candidate can do it at a time and place convenient for them, and does not need to be absent from their current job or travel a significant geographic distance (whether within Israel or for work outside Israel) in order to go through the selection process. An additional advantage that is growing is the opportunities now given to candidates to practice and improve their abilities before the test. With these characteristics of internet-based assessment systems, the chances that qualified candidates will give up on applying due to difficulties involved in attending internal assessments and/or psychometric testing at an assessment center are reduced, and therefore the organization also benefits from the high accessibility of the assessment system to the job candidate.

Finally, an internet-based psychometric test has advantages for test developers, such as rapid updating of test items immediately for all test users, immediate calculation of psychometric data, and constantly updated norms.

Despite these advantages, internet-based assessment systems also have disadvantages that are important to note. First, these systems are technology-dependent, and it is necessary to ensure that the technology in the candidate's home (or the place where they are tested) meets the technical requirements of the system. Second, there is concern about biases and discrimination against candidates who are not accustomed to using the internet in particular or computers in general. These biases may have a cost such as rejecting a person suitable for the position, due to poor performance in the internet-based assessment system, a level of performance that does not reflect their abilities in the measured field, but rather the level of computer operation skills and internet proficiency. Third, there are populations that do not have the means required for internet testing from their homes. It is known that there is a statistical connection between socio-economic level and the presence of a computer connected to the internet at home.

Perhaps because of these disadvantages, or perhaps for other reasons, until recently there were no differences between internet-based psychometric test systems and computerized test systems (applications) in terms of how they were actually used - in both cases, the physical presence of the examinee at the business and testing on a local computer was required. In the last two years, internet-based systems have begun to look for solutions that would allow testing outside the organization, in order to maximize the advantages inherent in using internet-based psychometric test systems. The difficulty in testing outside the organization (in the candidate's home) stems from three main reasons:

System copying: Developing a psychometric assessment test system involves lengthy time and heavy costs. The concern that the system may be copied, distributed, learned and "burned" in a place where there is no operator or supervisor is very high, and makes the investment involved unprofitable.
Risks of cheating and fraud: In the absence of any supervision, there is a reasonable concern that the examinee will not take the test themselves or will use prohibited aids such as a calculator in quantitative tests, writing tools during memory tests, a dictionary in verbal tests, etc. To address this issue, it was proposed that internet-based selection processes would constitute the first stage in a two-stage employee selection process (Segall, 2001, quoted in Nye et. al, 2008). In the first stage, a psychometric test is conducted via the internet for many job candidates, and in the second stage, a "verification test" is conducted for the most successful candidates. In addition, job candidates being tested via the internet are required to commit to performing the test without cheating.
Technological complexity: Complex technological adaptation is required, so that all examinees, on all computers and existing operating systems, receive the same appearance and behavior of the computer during the test.

Despite the disadvantages mentioned above and the problems mentioned here, it appears that the advantages inherent in psychometric assessment systems via the internet outweigh the potential disadvantages of these systems, and therefore, as stated, the use of internet-based assessment systems for employee selection is increasing, while performing the testing process outside the business. Nevertheless, several questions arise regarding these systems, which are worth considering. This article will present some of these questions, and research that has addressed these issues.

Candidate performance levels in paper and pencil tests versus computerized and internet-based tests

One of the central questions that must be answered before deciding to use a computerized/internet-based employee assessment system is whether the performance level of employees is identical in paper and pencil tests compared to computerized/internet-based tests. Potosky & Bobko (2004) examined this issue in their research. The researchers review several studies that examined the degree of equivalence of non-internet computerized tests versus paper and pencil tests, and reported that when it comes to untimed tests, such as personality tests, no significant differences were found between groups tested using paper and pencil or a computer in terms of the resulting factor structure and test reliabilities, but there may be differences in group averages and the distribution of scores across different personality dimensions.

The researchers report on a single study from 2003 (Salgado & Moscoco, 2003) that examined the degree of equivalence of a personality test in paper and pencil format versus an internet-based personality test, where a very high correlation was found in each of the five personality dimensions of the Big Five model (OCEAN) between internet delivery and paper and pencil delivery, for a group of subjects tested in both media. It was also found in Salgado & Moscoco's (2003) study that the means and standard deviations of personality tests in different formats were similar, the reliabilities of the different dimensions were similar between the two administrations, as well as the resulting factor structure. Examinees' attitudes toward the internet test were more positive, and no significant relationships were found between the measured personality variables and perceptions and responses to internet-based tests.

Potosky & Bobko (2004) examined this issue in the context of cognitive tests. They note that studies examining differences between paper and pencil cognitive tests and non-internet computerized tests found that when the tests are similar in terms of test administration procedures, test results are similar between different administration methods.

The researchers report that they did not find a study that examined the degree of equivalence of paper and pencil cognitive tests and internet-based tests, and the purpose of their research was to fill this gap. They conducted a study in which they administered different types of cognitive tests (verbal, numerical, and figural) to subjects, both in paper and pencil format and in internet format. The research was conducted in a computer laboratory, where the researchers could control technological factors, such as adapting the computer to system requirements, page loading speed (especially relevant in figural tests), etc.

Their research found that the correlation between paper and pencil tests and internet tests is only 0.60 (p<0.001) for cognitive tests, a correlation considered too low to indicate sufficient equivalence of these two administration formats. However, there were significant differences between the different types of tests, such that the numerical test (r=0.74) yielded a significantly higher correlation than the figural test (r=0.44) [z=2.70, p<0.01]. The correlation of the two administration forms for the verbal test (r=0.58) was not significantly different from the numerical test [z=1.59, n.s.] or the figural test [z=1.02, n.s.]. Unlike cognitive tests (time-limited), for an untimed test (situational judgment questionnaire), a higher and sufficient correlation was found (r=0.84, p<0.001). These findings are similar to the correlations of paper and pencil tests with non-internet computerized tests.

Also in a paired t-test, it was found that the internet version of cognitive tests yielded lower scores than the paper and pencil version, compared to the situational judgment test where no differences were found in scores yielded by the two test media. It was also found that scores on cognitive tests were higher the second time examinees performed them on the internet, compared to their first internet experience with them. These findings are especially important in situations where an organization combines different administration methods, so that the candidate has a choice regarding the method of testing.

In addition to comparing the different test formats to each other, the researchers examined to what extent variables related to computer literacy and internet literacy influenced subjects' performance on the different tests. It was found that computer experience, computer understanding, and computer self-efficacy were positively related to scores on cognitive tests, so that examinees with a higher level of computer understanding and/or computer experience and/or computer self-efficacy received higher scores on cognitive tests, both in paper and pencil format and in internet format. The age variable was found to be negatively correlated with performance on the internet cognitive test, so that younger people performed better on the test; it was found that age is a predictor variable in itself for the internet cognitive test score, regardless of the level of understanding and experience with computers. This finding may encourage organizations to use varied testing methods, in order to avoid bias and discrimination against older candidates, who tend to achieve lower scores on internet tests. Overall, most subjects enjoyed performing the tests in the internet format more, although they felt that time pressure was higher in this format.

The researchers concluded that in transitioning from paper and pencil-based tests to internet tests, caution should be exercised in applying norms from one medium (e.g., paper and pencil) in administration in the other medium (e.g., internet), especially in figural tests. They also concluded that it is necessary to examine how to deal with technological limitations related to computer processing speed, internet bandwidth, test loading time, etc.

Cheating and fraud in internet-based tests

A very important question in the context of internet tests is their resistance to response distortions by examinees. Response distortion is a general term referring to cheating, social desirability, faking good, etc. (Nye, Do, Drasgow & Fine, 2008). When a job candidate comes to an organization or assessment center for selection tests, they are under supervision when performing the tests. Response distortions are possible, but more difficult to perform. However, when the examinee sits at home and performs the tests without supervision from the assessing organization, how can the assessors know that they performed the test themselves, alone, and in a straightforward and fair manner?

First, the question arises regarding the extent of the response distortion phenomenon - to what extent are there differences in test results between test administration under supervision (in the assessing organization) and testing conducted without supervision (via the internet, in the candidate's home). This question is complex to examine. In laboratory studies, the motivation to cheat is low, and it is not possible to determine the true extent of the phenomenon or its implications in high-risk selection processes. Field studies may also prove problematic, because of the nature of the employee selection process: only those with the highest scores advance to each stage, so the phenomenon of regression to the mean may lower candidates' scores regardless of cheating in the home test. On the other hand, the training effect involved in performing a test a second time may raise subjects' scores, so that their score in the test at the assessing organization will be higher than the score they achieved at home.

Despite the problems mentioned, several studies have been conducted on the subject. Nye et al. (2008) review several studies on unsupervised personality tests, raising varied findings. In some of the reported studies, differences were found in score averages, the resulting factor structure, and more, while in other studies no significant differences were found. These findings may be explained by differences in response distortions common in personality tests: these tests invite response distortions of the social desirability or faking good type, distortions which are likely not to be affected by the physical presence of a supervising factor.

In contrast, unsupervised cognitive tests are sensitive to cheating, which the physical presence of a supervisor may significantly reduce. Nye et al. (2008) conducted a large-scale study in which they examined the potential extent of the cheating phenomenon in a cognitive test of attention to details (also called a "clerical test"). They examined the differences in scores in a first unsupervised test, conducted via the internet, and a second test conducted in the assessing organization under supervision, and did not find evidence of cheating in the first test at the overall level, although there were individual examinees whose score pattern raises suspicion of this.

The major weakness of Nye et al.'s (2008) study is in the type of test chosen. This test examines a cognitive structure which is difficult to improve performance on with the help of aids prohibited for use, such as a calculator or dictionary. In addition, it is difficult for an ordinary person to assess whether another person has a higher level of attention to detail, making it worthwhile to ask the other person to perform the test in their place. It seems there is room to examine the extent of the cheating phenomenon in tests more sensitive to cheating, such as arithmetic calculation tests, vocabulary tests, and even memory tests. Beyond the study described above, no studies were found that examined the issue of cheating in unsupervised internet cognitive tests.

A second question concerns the validity of the selection tests. If there are indeed differences in the way of responding in the supervised test compared to the unsupervised test - to what extent do these differences affect the criterion validity of employee selection tests? Beaty, Nye, Borneman, Kantrowitz, Drasgow & Graur (2011) report that no studies were found that addressed this question, and they tried to fill this gap in the research field. The researchers conducted a meta-analytic study in order to draw conclusions on the subject. They examined information regarding the criterion validity of a personal questionnaire (biodata), the dimension of conscientiousness, and sales potential against manager evaluations of job performance and actual sales level. In general, the researchers found that the criterion validity of the three tests (personal questionnaire, conscientiousness, sales potential) was similar between supervised and unsupervised tests.

Beaty et al. (2011) offer several explanations for their findings. First, it is possible that phenomena such as cheating, fraud, or biases resulting from different test conditions are not common phenomena, and therefore do not affect the criterion validity of the various tests. Second, it is possible that at the level of positions for which the selections were made (junior management), candidates do not have high enough motivation to cheat or falsify in these tests. In addition, it is possible that response changes in non-cognitive tests do not affect criterion validity, as emerges from similar studies on the topic of faking good in personality tests in the context of employee selection.

As a result, the researchers conclude that the use of non-cognitive internet-based selection tests is justified, for the jobs examined. The validity of internet tests is similar to the validity of supervised tests, and their cost is significantly lower.

No studies were found that addressed this issue for cognitive internet selection tests, and therefore it is not known to what extent the predictive validity of cognitive selection tools changes when the tests are conducted under supervision or without supervision. It seems that there is room to investigate this important issue, in order to provide justification for the common use of unsupervised cognitive internet selection tests.

Summary

The purpose of this article was to review central issues in the implementation of internet-based test systems, both personality tests and cognitive tests. It is evident that there is much room for additional research on the subject, but from the few studies reviewed, several tentative conclusions can be drawn:

There are differences between paper and pencil tests and internet tests, and therefore the use of internet-based tests should be implemented in an adapted manner. At the most basic level - these tests require separate norms, and it is not possible to use the norms of paper and pencil tests in the equivalent internet versions.
Internet-based tests, especially cognitive tests, are sensitive to cheating; it is very difficult to examine the extent of this phenomenon in practice, but it is advisable to employ two-stage selection processes in order to reduce the possible implications of cheating by job candidates.
The criterion validity of internet-based personality selection tests is not compromised as a result of the media change, and therefore their use is appropriate.
No studies were found regarding the criterion validity of internet-based cognitive selection tests. This fact, combined with higher sensitivity to cheating in the test, requires test users to exercise extra caution in implementing internet-based selection methods.

By: Merav Hami, Psychologist

Bibliography:

Potosky, D. & Bobko, P. (2004). Selection Testing via the Internet: Practical Considerations and Exploratory Empirical Findings.Personnel Psychology, 57 (4), pg. 1003-1034.

Salgado, J.F. & Moscoso, S. (2003). Internet-Based Personality Testing: Equivalence of Measures and Assessees' Perceptions and Reactions.International Journal of Selection and Assessment, 11, pg. 194-203.

Beaty, J.C., Nye, C.D., Borneman, M.J., Kantrowitz, T.M., Drasgow, F. & Graur, E. (2011). Proctored versus Unproctored Internet Tests: Are Unproctored Noncognitive Tests as Predictive of Job Performance?International Journal of Selection and Assessment, 19 (1), pg. 1-10.

Nye, C.D., Do, B.D., Drasgow, F. & Fine, S. (2008). Two-Step Testing in Employee Selection: Is Score Inflation a Problem?International Journal of Selection and Assessment, 16 (2), pg. 112-120.

We would be happy to hear your opinion!

Interested in reading more?

LogiPass has dozens of additional interesting articles dealing
with issues of screening tests and career guidance.

Additional articles