New Insights on Technology and Assessment: Introduction to JWOP Special Issue

Introduction

In this Special Issue we have depicted an approach to technology and assessment counting on some scholars who have previously participated in the chair of the Psychometric Models & Applications Conference Series organized by the Universidad Autónoma de Madrid (UAM) with the wise chairing of Vicente Ponsoda and Julio Olea over the last two decades. The idea and initial works were initially proposed at the Third Spanish National Congress of Psychology held in Oviedo in 2017.

The Psychometric Models & Applications chair is being conjointly sponsored by the UAM and the UAM’s Instituto de Ingeniería del Conocimiento [Institute of Knowledge Engineering]. Its main aim is to disseminate assessment and testing innovations among personnel selection practitioners and organizational psychologists. To achieve this goal, the Chair’s central activity is an annual seminar. We choose a central topic and asked a few well-known experts to participate in the selected topic every year. Some invited speakers of previous seminars accepted to submit a paper to this JWOP Special Issue as well. Their names and seminar topics they contributed are listed below: Faking in Personality Testing (Jesús Salgado), Psychometric Methods to Detect Faking (Vicente Ponsoda), Situational Judgment Testing (Filip Lievens), Future Directions in Selection Assessment (Ann Marie Ryan), Contributions of Meta-Analytical Techniques to the Advancement in Personnel Selection (Silvia Moscoso), Advances in Adaptive Measurement (Francisco José Abad), Personnel Selection through Social Media (David Aguado), and Personnel Recruitment through Web Resources (Antonio León García-Izquierdo).

The goal of this Special Issue of Journal of Work and Organizational Psychology was to provide academics and professionals with some reflections and the state of the art in the technology and assessment field with the contribution of scholars who accepted the challenge of updating knowledge in this field.

Technology and Employment Issues

Information technology has become one the most important disruptive phenomena in the society during the last decades in the employment market (Cascio y Montealegre, 2016). Personal and work environments are increasingly in connection with some of such technologies, leading behavior to a virtual area where intangibility and the interaction between people and informational displays are a key point in daily and work life. This facilitate accelerated operations, which instead of being a way for more leisure has brought more intense work, multitasking, teamwork nets, continuous availability, hiperconnectivity, nomad jobs, and the need for immediacy in communication. All of this has great implications because of the perception of continuous availability and difficulties for detachment from work. In this context, psychological measurement has to cope with great challenges. Traditionally, measure developers tried to adapt to changes in the social environment looking for validity. Nowadays, they are investing a great effort in developing new methods, instruments, devices, and software to take advantage of the many possibilities technologies are bringing to the measurement arena. Accordingly, technology presents opportunities for the assessment of aspects considered impossible to deal with some time ago, and for new behaviors that emerge in a globalized, diverse, uncertain, complex, and changing environment. Thus, we need to look at robots, surveillance, machine learning, big data, and artificial intelligence as tools for people to change tasks and new procedures. This makes us wonder, for example, about to what extent jobs are demanding new or different competencies. Are those competencies increasing working hours? Are technology-based devices involved in the development of new human skills? Systems theory taught us that when automation transforms workers’ procedures, cognitive and behavior operations change into new tasks, so continuous job redesign and job crafting is a must in this context.

One of the main caveats of information technology stems from its intrinsic attractiveness that can easily give an apparent surface of quality. For instance, video-based interview technology, which allows rating simultaneously applicants while they are interviewed, does not necessarily mean the quality of responses needs to be better than those in a face to face interview. Even more, some advantages (e.g., travel cost savings) could turn into problems (e.g., connectivity failures, quality of the information gathered, etc.). This means that technology is not enough for ensuring good measurement, that is, reliable, sensitive, valid, accurate, and fair, so we must be strict before directly adopting and/or extrapolating scientific results of some measures to other settings without thoroughly testing them before. Moreover, cohort effects must be taken into account. In this sense, the “millennial” generation is probably more attracted by a mobile developed questionnaire than for a paper and pencil one, whilst “baby boomers” and other people over fifty actually feel micro-management overwhelmed, struggling with “technocrazy demanding tasks” (e.g., creating and storing passwords, downloading plug-ins, updating software, playing serious games, etc.), that is, the digital breach could unfairly discriminate juniors from seniors and leaving them out of job market skilled and qualified individuals while the western countries’ elder people are dramatically increasing and retirement age is extended some more years. So, we see a sort of mixed of generations working together and struggling to understand different values and norms from each other. Youth has become a social value versus elder people, avoiding the wisdom of experience. Lipovetsky (2000) has outlined some characteristics of postmodern era and the hypermodern society (individualism, consumerism, technocratic revolution), where we can see apathetic narcissism, consumerism, hyperindividualism, desertion of traditional values, abolition of the tragic, instant hedonism, loss of historical consciousness, and the discrediting of future, the ephemeral, cult of leisure, anxiety, and youth. These characteristics match fairly well with social networks, which could be seen as a delimited set of actors linked to each other through social relationships, but also a set of concepts and procedures of a methodological or analytical nature for data collection. That is, it would be a set in which social units are linked to each other through various interactions. As a consequence, Social Media Websites (SMW) have been programmed. Initially, Internet was a site for sharing knowledge, but then it turned into a great market environment where SMW found a very artificial “place” to be developed. We can see SMW as applications based on the ideology and technology of Web 2.0 (Lai & To, 2015), which facilitate information exchange and a cyberspace and a particular cybertime (both in synchronicity and asynchronicity.) This context is intangible, virtual, created and supported by pools of groups (McFarland & Ployhart, 2015), which can be better understood by the following seven functional blocks framework (Kietzmann, Hermkens, McCarthy, & Silvestre, 2011): identity, conversations, share (in terms of content), presence (accessibility in terms of location and availability), relations, reputation (in terms of status of others and themselves in a social network), and groups (in terms of created virtual communities). Following Stone, Deadrick, Lukasiewicz, and Johnson (2015), employees prefer interactive, interpersonal, information-rich approaches, so information technology management in personnel assessment should enhance these aspects, putting the focus more on the quality of information gathered (and to be analyzed and interpreted) than on the speed and the crave for data.

Simultaneously, it is increasingly common to remove the line dividing personal from professional life. Private information is now more open, accessible to employees, employers, colleagues, recruiters, and customers, as well as to other people. Private information conveyed through technology is easily transferable and very difficult to control, which can have effects both on privacy and reputation, where both types of networks, personal and professional, offer different levels of privacy, so that information on profiles can be exposed total or partially to public. The intrusion into personal life through any mechanisms threatens both applicants’/employees’ freedom, as well as their dignity and privacy and, in addition, the increase of the probability of opting for discriminatory practices. The use of data stored in social networks for labor purposes would require authorization due to data protection regulations and the conditions of use of SMW, which limits the use of the information to the social network itself. However, it is difficult to prove that a negative selection decision has been based on publications in social networks, but a job applicant who proves that s/he was rejected with these methods could sue that organization. Consequently, organizations should protect their reputation, intellectual property, know-how, and confidential information. Having facilitated their information, the digital era can be a very powerful environment for those disgruntled employees as a way to bring up employers’ dirty rags or their products, so companies need to be aware of this and implementing employees’ protocols in social networks, analysis of the communication with job seekers in personnel selection processes, and managing the information of those employees in social networks once they are out of the organization. Bruning and Ledingham (1999) identified six values that should orient any type of organization that has direct communication with public and customers: awareness of their own influence, dialogue, sincerity, trust, understanding and the desire to negotiate, collaborate, and provide solutions to any aspect that requires it. These premises announced two decades ago have to be translated and adapted to the current times on the realm of new technologies. In line with the above in this special issue of JWOP we address some of the main insights that this technological state generates in the field of recruitment, selection, and assessment.

JWOP’s Special Issue on New insights on Technology and Assessment

The content of the manuscripts falls into three major categories: (a) the reflection on new technological issues that impact recruitment and selection field; (b) the use of technology in the development and design of new assessment instruments; and (c) the evaluation of new key constructs in the prediction of performance in VUCA (volatility, uncertainty, complexity, and ambiguity) environments in which the organizations operate. Regarding the first issue, Ryan and Derous’ (2019) study explores with systematic and critical vision the improvements promised by the use of technology in the field that have not yet come true and, of great interest to academics and professionals, what are the aspects that must be developed.

Regarding the second issue, the paper by Aguado, Andrés, García-Izquierdo, and Rodríguez (2019) addresses a trending topic for bridging the gap between science and practice: the use of LinkedIn in the recruitment and selection processes. Their study, focused on the dimensionality of the information contained in LinkedIn and its connection with professional performance, provides important insights in both the applied and the academic fields. Also positioned in this second question, the paper by Morillo et al. (2019) addresses the issue of the development of multidimensional forced choice questionnaires (MFCQ). Their manuscript uses modern developments within the framework of the item response theory (IRT) to the construction of MFCQ and to obtain non-ipsatives scores.

A third group of articles deals with the measurement of relevant constructs in the current labor arena. Herde, Lievens, Solberg, Harbaugh, Strong, and Burkholder (2019) deals with the measurement of 21^st century skills by means of situational judgments tests (SJTs) in a context of enormous relevance due to globalization which is their assessment in a cross-region environment. In their article, the authors develop a set of STJs for the measurement of five fundamental competences: achieving objectives, adapting to change, analyzing and solving problems, learning and self-development, and working well with others. For this, and from a combined emic-etic approach, the authors show the configural and metric measurement invariance across regions.

Finally, Salgado, Banco, and Moscoso (2019) revisit the subjective well-being (SWB) construct as an important predictor of professional performance. In a four-year longitudinal study, they connect this bi-dimensional construct (cognitive and emotional) with professional performance.

Applied Reflections about Technology in Recruitment and Selection

Ryan and Derous (2019) explore how technological advances in assessment have radically changed the landscape of employee selection. However, as the authors point out, these changes have not been translated into reality with all their potential. The first issue addressed by the authors is related to the new constructs. Technological development has undoubtedly made (and continues to do so at a good pace) the development of new forms of measurement possible: new instruments, new methodologies, and new data. However, these technological changes also make it possible to measure new constructs that, on the contrary, have not been explored. The second aspect developed by the authors has to do with the type of improvement produced by technology: while the improvement produced in terms of efficiency is undoubted, little has been done in terms of effectiveness and fairness. The technological context generated by big data puts this question in the foreground: for example, what impact does the use of more data have on making decisions? What effects does the progressive substitution of human judgment for artificial intelligence have? The third and last question developed by the authors deals with the insufficient attention given to the changes that technology has produced in the nature of work and the consequent re-analysis that should occur in the strategic design of recruitment and selection (R&S) processes. Aspects such as virtuality, contingent work, automation, transparency, or globalization are necessarily modifying the strategies of recruitment and selection and a deep look is required to put into value the best designs in these contexts.

New Methods and Instrument for Recruitment and Selection

The article by Aguado et al., (2019) analyzes LinkedIn platform from a psychometric perspective. The authors point out that the use of LinkedIn is breaking the traditional frontier between the two phases of R&S process: the recruitment phase, that seeks to obtain a good number of potential candidates, and the selection phase, oriented to perform an assessment process candidates’ KSAOs. Increasingly, professionals make evaluative decisions about the data contained in LinkedIn profiles. Thus, it is not uncommon to make inferences about the personality of individuals and about the degree of adjustment to the position and the organization. However, these inferences are not supported on a study of the psychometric properties of data contained in LinkedIn. In fact, most of the studies developed about LinkedIn have addressed the question of how professionals and candidates use it, leaving aside the inevitable question of the validity of such information for decision making in personnel selection processes. The authors have obtained initial evidence about its criterion oriented validity in a sample of information and communication technologies incumbents. This is the first characterization of the dimensionality of LinkedIn, presenting a concept analogous to the Big Five in the field of personality, that is, the LinkedIn Big Four. Additionally, the authors offer initial evidence on how these LinkedIn Big Four relate to job performance.

Additionally, Morillo et al. (2019) propose a development based on the IRT to obtain parameters in MFCQ able to obtain non-ipsative scores and normatively comparable also. MFCQ are widely regarded in the personnel selection literature for their ability to control response biases. In spite of its undeniable advantages, its use has been intensely criticized due to the ipsativity of the scores generated with this type of instruments that, in fact, produce an important distortion of its reliability and validity. In this context, the development of strategies that lead to the acquisition of non-ipsative scores remains in continuous exploration. Different strategies have been proposed as quasi-ipsative scores, which have proven to be more valid predictors of performance. The authors drive this issue in a novel way starting from the application of the developments of IRT (using the MUPP-2PL model). Central in their contribution is the idea that the parameters of item blocks in the MFCQ can be obtained departing from the acquisition of parameters in traditional items in a Likert type format. This entails an important advantage in the applied field in order to simplify parameter acquisition in the construction of MFCQs. The authors test this assumption of empirically comparing the parameter estimates of the forced-choice format to their graded-scale equivalent on a Big Five personality instrument. The results seem to reasonably support this idea of invariance, especially for discrimination parameters, and they rigorously discuss the practical implications of their results by offering some guidelines for the design of forced-choice questionnaires based on the invariance assumption.

Assessment of New Constructs in the Recruitment and Selection Field

Herde et al.’s (2019) manuscript deals with a doubly important question: the measurement of the 21^st century skills and its measurement in cross-regional contexts. Regarding the first one, they focus on five critical competences: achieving objectives, adapting to change, analyzing and solving problems, learning and self-development, and working well with others. Regarding the second, the authors develop their measurement through the use of SJTs. They clearly identify the limitations that the use of SJTs can present in cross-cultural evaluation: differences in the representativeness of the critical situations presented to the evaluated ones and differences in the relevance, effectiveness, and construct linkage that each answer option represents. To limit the potential negative consequences of these effects they develop a combined emic-etic approach, in which both universality and indigenous constructs are incorporated. In it, the inclusion of specific aspects of each culture allows the presented situations to have cultural relevance, while the use of universal concepts facilitates cross-cultural comparison. Results obtained indicate an acceptable reliability of the five developed SJTs and the existence of configural and metric measurement invariance across regions. In sum, the authors contribute with their work to an effective instrument for the cross-cultural measurement of five critical competences in the context of 21^st century skills. In addition, professionals and academics obtain an extraordinary guide for the development of SJTs from a combined emic-etic approach.

Finally, Salgado et al. (2019) revisit a current construct in the prediction of professional performance: subjective well-being (SWB). In an extraordinary effort, the authors develop a four-year longitudinal study in which they relate the cognitive and emotional components of SWB with job performance. As Ryan and Derous (2019) indicate in their article in this Special Issue, there is an important need to explore new constructs that help improve the prediction of professional performance. The recall is also valid, as authors do, of a construct with a long psychological tradition that is especially relevant today. In their paper, the authors provide a validated and specifically developed measure to measure the two components of the SWB and, thus, to be able to study their relationship with performance jointly. The results indicate that SWB is a relevant predictor of professional performance. This has important consequences at a practical level since the results presented suggest that organizations can improve their effectiveness (through individual performance) by increasing employee well-being. In doing so, organizations can implement two mechanisms: selecting employees measuring SWB as a predictor and creating conditions that allow employees to activate positive emotions.

Conclusion

These manuscripts in this Special Issue cover some of the main topics in the field of technology, trying to answer a central question: how does technology contribute to personnel selection and assessment? To answer this, the authors have delved into different theoretical, methodological, and instrumental aspects, presenting significant contributions in which the technological issue is exploited effectively beyond efficiency and effectiveness.