Studying Within-Person Changes in Work Motivation in the Short and MediumTerm: You Will Likely Need More Measurement Points than You Think!

Introduction

Over the last years, several researchers have focused their attention on the issue of within-person changes in work motivation. Within-person research designs are essential to capture these temporal changes that are substantive. These designs allow us to observe motivation processes as they unfold in workers over time (Hamaker, 2012). Typically, the strongest research makes use of intensive longitudinal designs (e.g., experience sampling methods or ecological momentary assessment; Bolger & Laurenceau, 2013) in which several repeated assessments¹ are obtained for the same participants (e.g., Bakker & Xanthopoulou, 2009; Binnewies et al., 2010; Fisher & Noble, 2004).

By using these kinds of designs, researchers are able to consider both between-participants and within-participant variance in a focal outcome at the same time. Moreover, such designs allow researchers to corroborate or improve upon previous findings both through alternative explanations obtained by more detailed and ecologically-valid data and analyses, as well as to expand the amount of variance to be explained that in other cases (i.e., in between-participants designs) is confused with measurement errors. In this sense, to consider both levels (i.e., within- and between-participants) could provide a deeper understanding of the study of work motivation (e.g., Dalal & Hulin, 2008). Moreover, the addition of within-participant approaches to the research agenda is promoting the inclusion of more proximal causes of work motivation. For instance, research by Fisher and Noble (2004) revealed clear relationships among task interest, task difficulty, skills, and effort at the within-person level. Finally, these kinds of intensive research designs also frequently reveal a sizeable amount of variance attributable to the within-person level. For example, in recent reviews (i.e., McCormick et al., 2018; Podsakoff et al., 2019), the reported percentage of within-person variance in several examples of organizational behaviour ranged from 37.65% (for job satisfaction) to 61.24% (for vigour) over the total amount of explained variance.

At this point, one important question that is yet to be answered is how the amount of within-person variance in work motivation can be related to key properties of the research designs. Indeed, the amount of measurement points collected per participant, and its proportion in relation to the number of participants, can influence the amount of within-person variance that emerges. For instance, we can expect that a design using 100 participants with 3 measurement points per participant will obtain a lower value of within-person variance than a design using 20 participants with 100 measurement points each. Founded in this reasoning, the recent literature (e.g., Bakker & Xanthopoulou, 2009; Clinton et al., 2016; Ilies et al., 2007) has promoted a ‘rule of thumb’ of between 10 to 15 repeated assessments per participant as satisfactory to study both levels of variance (between- and within-person). The veracity of this ‘rule of thumb’, however, is yet to be tested and must be further examined.

Recently, two publications have researched this issue (i.e., McCormick et al., 2018; Podsakoff et al., 2019). These previous studies have, in our view, two important limitations that we are interested in overcome in this paper. First, they have not focused specifically on work motivation. Instead, they have reviewed several topics in organizational behaviour without paying special attention to the different motivational constructs that traditionally conform the work motivation literature (e.g., needs, expectancies, goals, justice perceptions, self-efficacy beliefs). For this reason, we have the purpose of going beyond this previous research by looking into these specific motivational constructs. Second, these two previous studies are reviews of published literature without collecting new original data. Again, we have the purpose of going beyond by combining a literature review focused on work motivation jointly with new empirical data that was collected specifically for this purpose.

The clarification of the needed amount of measurement points to study within-person variance in work motivation continues to be relevant for theory development. To study processes such as work motivation in organizational behaviour, we should clarify, based on empirical evidence, a recommended number of repeated assessments for a proper accounting of within-person variance rather than following rules founded in tacit practice. By doing so, we will provide normative knowledge to be followed in future research about work motivation. Moreover, this clarification is also needed in order to understand if previous existent knowledge about the relationships between work motivation and other relevant phenomena is supported or should instead be questioned. In order to advance knowledge in this subject, the objective of the present research is to examine empirically the number of measurement points needed to capture with precision within-person variance in work motivation. In other words, we aim to clarify what is a reasonable amount of measurement points per participant to allow for a complete unfolding of work motivation.

One might consider that studying within-participant variance per se is not an important issue considering that predominant research is usually focused on the study of the relationships between variables. However, if the within-participant variance is underestimated, it can also affect the pattern of relationships found. For example, if we underestimate the variance of work motivation at the within-participant level, we can overestimate the potential relationship between work motivation and other phenomena in a higher level (e.g., team or organizational climate) because these higher level phenomena are more stable and change less over time (Zaheer et al., 1999). Moreover, an underestimation of variance components in multilevel design can be related to the shrinkage in these designs, having, as a consequence, an influence in the estimated parameters in these models (e.g., Greenland, 2000). In consequence, the specification of the time-scales or time-frames would be a requirement to improve our research.

Short- and Medium-Term Changes in Work Motivation

By work motivation, we refer to the psychological processes that determine or energize the direction, intensity, and persistence of behaviour in the work setting (Kanfer et al., 2008). Needs, goals, expectancies, self-efficacy beliefs, perceptions of justice or engagement have predominated, over others, in the recent literature of work motivation. By within-person changes in work motivation, we refer to the variability that can be seen in each employee’s work motivation over a determined period of time. With an emphasis placed at the within-person level, the literature recently has put the person, and his/her possible temporal changes, at the centre of inquiry. Moreover, this research trend has followed the advice of studying the third ‘C’, i.e., change, proposed by Kanfer et al. (2008), in an attempt to go beyond more traditional research interests in the “content” of motivation (the first ‘C’ reflecting interests, expectancies, beliefs, etc.) and the “context” within which motivation evolves (the second ‘C’ comprising jobs, social relationships at work, team and organizational climate/culture, etc.).

As different authors have proposed (e.g., Kanfer, 2012; Lord et al., 2010), work motivation is changing constantly, since the various underlying processes are changing as well. However, not all of these processes involve change on the same temporal scale. For example, we can agree that the intensity dimension of work motivation changes over minutes or hours due to the fact that attentional effort changes as well or we can agree that the persistence dimension can also change over days or weeks since allocation of resources is also moving; finally, one’s motives, considered similar to motivational traits, can also change over decades and along the lifespan. Therefore, motivation should be considered as a state that is always in flux (Kanfer, 2012).

Work motivation theory has been sensitive to the study of changes in motivation over time. At this point, different time scales have been considered. Thinking about work motivation as a self-regulatory process allows us to consider work motivation changes in the short-, medium-, and long-term (Lord et al., 2010). Starting with a long-term scale, changes in work motivation can happen over the course of years, particularly when an employee’s role changes as a result of promotion, new assignments or demands, or as a mere consequence of changes in personal values. Kanfer and Ackerman (2004), for example, have proposed a theoretical model to understand these longer-term changes in relation to changes in skills, personality, affect, vocational interests, personal values, or one’s self-concept. Moreover, these changes can happen in the form of losses (e.g., decline in cognitive abilities), gains (e.g., acquisition of new skills), reorganizations (e.g., establishing new priorities in personal values), or exchanges (e.g., the primacy of motives) across the lifespan. Changes in work motivation can also take place in the medium-term (i.e., months, weeks, days). Lord et al. (2010), for example, proposed that automatic and conscious processes such as habits or goal-setting practices influence goal striving and resource allocations. Finally, short-term (i.e., minutes, hours) changes in work motivation can appear when connected with related phenomena such as attentional effort that can clearly influence behaviour intensity.

Focusing specifically on short and medium-term changes (i.e., hours and days), there has been a remarkable increment in the work motivation literature exploring change over this time scale, with methods such as intensive longitudinal designs and experience sampling, both comprising several repeated assessments per participant. This body of research is showing why it is important to consider short and medium-term within-person changes in work motivation. Using these research designs, we can account for changes that allow us to understand when the typical worker reports higher versus lower levels of work motivation over the course of hours or days, going beyond the traditional between-persons approach that only allows us to know who is more or less motivated at a certain point in time. Indeed, research based on between-persons designs is unable to explain variance that appears at within-individual level because: 1) it only makes comparisons between individuals and 2) work motivation is assumed to be a stable process that does not change over time in a significant way. However, results from within-person designs have consistently shown that motivation appears to be unstable, and a substantial amount of variance is attributable to this level (i.e., within-participant). For example, in a study by Bidee et al. (2017), the authors found that the proportion of within-person (compared to between-person) variance ranged from 43% to 54% across different motivational variables (i.e., need satisfaction of autonomy, competence and relatedness, and intrinsic motivation). Other studies focusing on similar motivational constructs, such as work engagement, vigour, flow, and self-efficacy, have found similar results (e.g., Bakker & Xanthopoulou, 2009; Casper et al., 2017; Debus et al., 2014; van Woerkom et al., 2016).

Moreover, these newer intensive longitudinal research designs (i.e., within-person approaches) allow us to explore different kinds of research questions, compared to the more classic between-participants designs. First, they can reveal different relationship patterns among variables at different-levels that are not always congruent. In other words, the processes are not always homologous across levels (Chen et al., 2005). For example, focusing on affect at work as a related motivational topic, Miralles et al. (2015) found that positive and negative affect, measured by the Scale of Feelings at Work (Warr & Clapperton, 2010), showed a different magnitude of relationship depending on the level of analysis. At the between-persons levels, positive and negative affect were uncorrelated (r = -.12, ns), consistent with the well-known findings of Watson (2000) and others. In contrast, at the within-person level, positive and negative affect were significantly and negatively related (r = -.51, p < .01) as other authors have also shown (e.g., Green & Salovey, 1999). In the past, this non-homology has created intense debates in the field of motivation as, for example, the one between Vancouver and Bandura about the self-efficacy and performance relationship (see Sitzmann & Yeo, 2013).

Second, within-person designs simply contend with different research questions in comparison with the corresponding between-persons designs. As we anticipated previously, in within-participant designs the questions are not about whether the employee is motivated or not, because the same worker can present significant changes in his/her motivation from time to time. Because these within-person changes can (and do) happen, research questions are now more focused on “when” workers are motivated and “why” motivation changes occur in the same employees.

And third, within-participant designs are the best equipped to study the influence of more proximal variables on work motivation. For example, Fisher and Noble (2004) found evidence that task interest is a significant and proximal cause of effort and that others task characteristics, such as task skill and task difficulty, have a more distal influence on effort. At this point, intensive within-person designs better contextualize the influence of target variables in the specific work setting and embed in real-time the behavioural processes we study (e.g., Schwarz, 2012).

Considering all of these arguments, jointly with the fact that within-participant designs present a better substantive-methodological synergy due the dynamic nature of work motivation and its appearance at within-person level, it is justified why these types of designs are the preferred ones in the current literature.

Since a greater emphasis on short- and medium-term within-person changes is being incorporated in the research agenda of work motivation, some critical aspects of the intensive designs that are frequently used have not yet been clarified (e.g., Gabriel et al., 2018). For example, it is not yet clear what temporal frame we need in order to reliably capture the within-person variance that exists in work motivation. This temporal frame has not been addressed by any theoretical development. Similarly, the size of time lags to consider for the repeated assessments again have not been clarified by theoretical guides. Moreover, it is unclear whether different dimensions of motivation (e.g., expectancies, needs, etc.) would require different temporal frames to capture them.

Research Objectives

In this paper, we focus on the following broad question: what temporal frame is needed to reliably capture within-person variance in work motivation? At this point, a rule of thumb seems to have emerged in the specialized literature: a 2-week period for the daily surveys was chosen on the basis of the recommendation of Reis and Wheeler (1991), who stated that “the 2-week record-keeping period is assumed to represent a stable and generalizable estimate of social life” (p. 287). In a similar way, considering the use of two or more levels of analysis (e.g., occasions nested in persons), Nezlek (2012) has proposed that “15 [measurements] are probably adequate for the most purposes” (p. 371), considering at the same time that “to [his] knowledge, there are no clear or firm guidelines for this” (p. 370). However, there is no empirical evidence to support these propositions and, for the sake of reliable scientific production, we ought to generate it. We believe that generating this guideline would be valuable and welcomed by the scientific community.

There is some evidence about the influence of the total number of repeated assessments per participant on the within-person variance in the more general organizational behaviour field. For example, in a recent review by McCormick et al. (2018), it was found that studies collecting 11 or more occasions showed higher within-person variance in comparison to studies with 10 occasions or less. Therefore, the main objective of this research is to clarify, based on empirical support, the temporal frame needed to capture within-person variance in work motivation.

In our opinion, clarifying how many measurement points are needed to study within-person work motivation changes would be valuable mainly for two reasons. First, it would provide a clear guideline for future intensive longitudinal designs in the area. Diary studies, experience sampling methods, and similar procedures are very time consuming for both participants and researchers. Consequently, having clear guidelines on how many measurement points are needed to capture within-person variability in work motivation would prevent both researchers and participants from wasting time collecting data that would not support a robust investigation on motivation dynamics. Second, it would provide a threshold value to be considered in the assessment of previous research and findings. This threshold value would speak to the validity of previous studies focusing on within-person variability in work motivation, its dynamics and changes, and its proximal antecedents and consequences. Consequently, we propose the following research questions (RQs):

RQ1: To what degree is the rule-of-thumb that ‘10-15 measurement points assessments are enough to capture within-person variance’ adequate in the case of work motivation?

RQ2: How many measurement points are needed to capture well the within-person variance of work motivation?

RQ3: To what degree is the existing literature capturing well the within-person variance of work motivation?

RQ4: Are there differences in terms of within-person variability across different motivational constructs (e.g., some motivational constructs are more temporally stable than others)?

By mentioning the terms “well” or “adequate” in our RQ’s, we are referring to uncover the maximum degree of within-person variance that could possibly be identified. To identify this maximum degree is important for the reasons explained before: going beyond the tacit practice used in the field and providing a guideline to be used in future research. Moreover, it would be interesting to clarify if different motivational constructs (e.g., engagement or flow) would require the same amount of measurement points to capture their within-person variability. For example, in the case of engagement some authors have proposed that it should be considered as an enduring process (e.g., Macey & Schneider, 2008). This means that it would be more stable than other motivational constructs and, in coherence, it would not need so many measurement occasions in order to capture the maximum possible variance. On the opposite, flow has been traditionally considered to be a very fluctuant and peak experience (e.g., Ceja & Navarro, 2012), therefore it would require the collection of more data points to capture it well.

To address these RQs, we have conducted two different studies. In the first study, we conducted a targeted literature review considering the previous research on work motivation and related topics (e.g., engagement, needs, self-efficacy) that have used intensive within-person designs. In the second study, we conducted an intensive longitudinal study measuring within-person work motivation for a sample of participants where we varied the amount of repeated assessments per each participant. Based on the findings from both studies, we sought to clarify the possible relationship between time frames and within-person variance in the specific case of work motivation.

Study 1

Method

Procedure. For this first study, we conducted a targeted literature review. As we started the literature selection, we decided to follow the approach of a systematic review as an interactive process, in the sense that we underwent consecutive refinement of the search strategy in terms of keywords and databases used, following a series of preliminary searches. The results from the exploratory searches were reviewed by the researchers and, from the quality of the retrieved articles, the most efficient search terms were identified.

The inclusion and exclusion criteria were defined so we were able to draw clear boundaries defining what articles could and could not be used. This strategy enabled us to identify all relevant literature regarding the key aims of the review. The studies included in this review required 1) the use of a longitudinal design (i.e., minimum three time points), 2) the inclusion of explicit information about the variance decomposition (i.e., ICC or similar) of the work motivation variables, and 3) a focus on the study of work motivation. We searched on PsycINFO, PsycARTICLES, and PsyCRITIQUES databases. We did not specify a publishing time-frame because, in general, studies using longitudinal designs are recent. The list of employed keywords (i.e., subject terms) to search for work motivation research consisted of: “work motivation”, “work engagement”, “flow”, “mood”, “self-efficacy”, “affect”, “goal setting”, “goal striving”, “psychological needs”, and “expectancies”. In the search, we combined these terms with “daily-diary”, “daily survey”, “daily diary”, “diary study”, “longitudinal design”, “ecological momentary assessment”, or “experience sampling” as keywords as well.

The reference list of the papers was examined and two studies were added to further complement the research findings. The search process was finalized on the 10th of March, 2018, and initially returned 110 references of potentially relevant studies. After duplicates were removed and the titles were reviewed for relevance, the abstracts and full documents were assessed against the general and particular inclusion/exclusion criteria. Fifty-nine studies were excluded, leaving a total of 51 studies eligible for full-text retrieval.

Data analysis. Once the final 51 studies were identified, each of them was read by the research team in order to extract the information needed to conduct the analysis. The information of interest comprised constructs, instruments, temporal framing, sampling strategy, number of participants, total number of measurement points per participant, number of measurement points per day and/or per week, duration of the studies, and finally the amount of within-person variance founded. More information on this procedure can be found in the Appendix. In order to conduct the analysis, we culled each study in terms of the variables under scrutiny, keeping the ones related to work motivation. Thus, the unit of analysis are the individual variables (e.g., intrinsic motivation, competence need, self-efficacy) and not each paper. In total, the final sample comprised 85 observations (M = 1.66 variables per paper) involving a total of 8,077 participants and 118,552 measurement points. All the analyses were conducted using R (R Core Team, 2018).

Results

As we can see in Table 1, the variables that we encountered more often in the studies were engagement (20%), positive affect (14%), negative affect (10%), and vigour (10%). All other 16 variables were found in less than 10% in the studies (please refer to Table 1 for detailed information on the frequency of all 20 constructs). Regarding the sample size used, the average was 96 participants (range = 27 to 256, SD = 54.08), with an average of 14.54 repeated assessments per participant (range = 2.37 to 73.59, SD = 14.98). The average number of measurement points collected per day/week² was 2.02 (range = 1 to 6, SD = 1.50) with the average total duration of the data collection over 10.62 days/weeks (range = 3 to 50 days/weeks, SD = 8.07). Regarding sampling strategy, 57.1% used signal contingent (i.e., collecting the measurement points randomly over time), 36.9% used interval contingent (i.e., collecting the measurement points at specific times, usually at the end of the day), and only 6% used event contingent (i.e., collecting the measurement points once some specified event happened). Finally, the average proportion of within-person variance for the work motivation variables was 51.9% (range = 13.6% to 88%, SD = 16.24) weighting by sample-size of each study.

Table 1 also shows the percentage of within-person variance found per each construct of interest. All the constructs with several observations (i.e., several research studies) show a very high dispersion of values; for instance, the case of engagement is illustrative, with 17 studies and values ranging from 22% as minimum value (in the study of Cullinane et al., 2017) to 88% as maximum value (in the study of Reina-Tamayo et al., 2017). This high dispersion happens despite the fact that the majority of these studies use the same measurement instruments (in the case of engagement, 15 out of 17 studies used the UWES by Schaufeli et al., 2006; see Appendix for the details). There are also some differences of interest among the constructs. For example, job satisfaction shows the lowest values of the within-participant variance in comparison with the rest of constructs (Mann-Whitney U = 57, p < .01), suggesting that job satisfaction has more stable components (e.g., cognitive dimensions such as beliefs about jobs). Comparing job satisfaction only with affect constructs (positive and negative affect and emotions), job satisfaction also shows less variability at within-participant level (Mann-Whitney U = 138, p < .01).

Considering the possible relationship among within-participant variance and some characteristics of the research designs applied (e.g., number of measurement points per participant, duration of the study), a correlation analysis yielded no significant relations: N of participants (r = -.11, p = .31), N of measurement points per participant (r = .14, p = .19), N of measurement points per day/week (r = .13, p = .24), and duration in days/weeks (r = .06, p = .61). Regarding the sampling strategy, the Kruskal-Wallis test was conducted to examine the differences. No significant differences, H(2) = 4.92, p = .08, were found among the three types (i.e., event, interval, and signal contingent), though a slight difference could be appreciated among event contingent sampling (Mdn = 67) and interval and signal contingent (Mdn_interval = 49 and Mdn_signal = 51). Moreover, in order to disentangle the potential influence of any daily patterns (e.g., circadian rhythm), we conducted a final test comparing the studies with only one assessment per day with the studies with several assessments per day, finding no statistical differences (Mann-Whitney U = 583, p = .08).

To look in more detail for any possible relationship between the within-person variance and the duration of the studies in terms of number of repeated assessments obtained from each participant, Figure 1 shows a scatterplot between these two variables with a fitted smoothed curve. It can be seen that the amount of variance increases gradually until reaching a maximum around of 25-30 measurement points per participant. Specifically, applying an extremum surface estimator (ESE) to detect the inflection point the result is between the values of 24-28. Noting the absence of studies in the scatterplot around the value of 20 occasions, we proceeded to split the sample into two corresponding groups: those with more than 20 occasions per participant and those with fewer than 20. After conducting an inferential variance difference test, we found the proportion of within-participant variance to be significantly higher for the group with more repeated measurements (Mann-Whitney U = 480, p = .02), representing a non-trivial difference of around 10% (Mdn_{less20registers} = 48.5 vs. Mdn_{more20registers} = 60). In consequence, having more measurement points (around 30 per participant) led to more within-person variability in work motivational constructs.

Discussion

In spite of having used a wide variety of terms for motivation-related constructs in our search of the literature (i.e., work motivation, work engagement, flow, mood, self-efficacy, affect, goal setting, goal striving, psychological needs, and expectancies), it is noteworthy that only some of these constructs have been the focus of attention in studies designed to estimate within-participant variance. For example, classic constructs in the field of motivation such as expectancies and goal-setting did not emerge in any of the studies we found. However, other motivational constructs such as engagement, affect (i.e., mood and emotions), vigour, and flow, for example, have received more attention in published work seeking to understand within-participant changes in motivation.

As we detailed above, the average study of within-person variance in motivation consisted of designs using a signal contingent approach, with an average of 14.64 repeated measurements (n) taken from an average of 96.15 participants (N). This suggests that, on average, scholars have been following the rules of thumb established in recent literature that recommend around 15 registers per participant (e.g., Bakker & Xanthopoulou, 2009; Clinton et al., 2016; Ilies et al., 2007). Moreover, these quantities translate into an average total sample size of 1,407 observations per study (N * n), a considerable amount compared to the usual sample size reached in standard designs (e.g., cross-sectional designs) due, in part, to the high-involvement and time-consuming characteristics of longitudinal studies.

Our finding on the proportion of within-participant variance across these studies is also highly relevant: more than 50% of the variance found in these studies resided at the within-participant level (M = 51.9, SD = 16.24). Despite the differences in the proportion of within-person variance that emerged across the different motivational constructs (RQ4), there was a considerable dispersion in the proportions of variance found and in the high values of this variance that are attributable to the within-participant level. We therefore corroborate that variability in motivation outcomes at the within-person level should be considered when conducting research in motivational research in order to not lose an important part of the variance that has to be explained. It should also be noted that not all motivational constructs showed the same degree of change over time. We observed that the constructs with a stronger cognitive component (e.g., job satisfaction) demonstrated lower within-participant variability in comparison to other constructs with a stronger valence component (e.g., positive and negative affect/emotion). Consequently, researchers should consider these differences when planning future studies. For instance, in a study proposing to examine the relationship between vigour and job satisfaction, researchers would be advised to base the number of repeated measurements per participant on measurement of within-person variability in vigour rather than job satisfaction, which would require less measurement points.

Regarding the possible influence of the number of repeated assessments on the percentage of within-participant variance in motivation, results revealed a relationship such that the number of measurement points collected per participant was linked to the proportion of within-participant variance. More precisely, findings from Study 1 suggest that 24-28 is a threshold number of measurement points per participant required to capture well the proportion of variance that exists in motivational variables (RQ2). It should also be noted, however, that studies with 30 or more measurement points per participant are more the exception than the rule (i.e., only 19% of studies; RQ3). Future research focusing on within-subject variability in motivation should aim to have, at least, this amount of repeated measurements in order to clarify whether this threshold value holds.

In order to support the results yielded from the reviewed literature, we conducted an empirical research studying again the possible links between the number of repeated assessments and the proportion of within-person variance found. This would be useful as well to overcome the limitation of working with secondary research, as we have done in Study 1.

Study 2

Method

Participants and procedure. Participants were 105 employees from different organizations. We recruited this sample using personal contacts and word of mouth in an attempt to obtain a diverse sample in terms of gender, age and occupation. Initially, 105 workers agreed to participate, but three of them withdrew from the study. The final sample consisted of 102 employees (51% women) with an average age of 34.2 years (SD = 8.39). Over two-thirds of the sample (68%) attended college/university, with an average of 8.1 years (SD = 8.6) of experience working in an organizational setting and an average tenure of 4.4 years (SD = 4.5). Employees came from different sectors, including jobs as architects, waiters, professors, accountants, tennis coaches, postwomen, hairdressers, etc. All participants voluntarily agreed to be involved in the study and did not receive any kind of financial compensation. However, we provided participants with personal feedback about their results at the end of the study (e.g., how their motivation evolved during the study, what kind of tasks motivated them more, etc.). Informed consent was obtained from all participants.

Participants were asked to complete a diary focusing on motivational constructs several times (i.e., from 3 to 6) per day at random intervals using a signal contingent design (Bolger et al., 2003). Data was collected during working hours using electronic devices (i.e., personal digital assistants and smartphones). Considering the research objective of studying the possible relationship between the number of measurement points per participant and the proportion of within-person variability, we randomly assigned a number of measurement points to be collected (i.e., from a minimum of 10 to a maximum of 120) by each participant. Taking into account that previous research has generally comprised a small-to-medium number of measurement points per participant (14.64 was the average obtained in Study 1), we aimed to increase this amount. In total, we obtained 6,375 repeated assessments from these 102 participants, with an average of 62.5 measurement points per participant (SD = 35.31). The distribution of the number of measurement points obtained per participant is detailed in Figure 2. Finally, we also considered a large range of options (from 0 to 100; see next section, Measures) in the response scale due the fact that the majority of research studies sampled in Study 1 used a relatively narrow range of response options (i.e., 1-to-5, 1-to-7). This was done to examine whether the range of response scale could influence the proportion of variance accounted for at the within-participant level.

Measures. First, we asked participants to briefly describe the work task being performed at the moment. After that, we used three items to measure work motivation (“How much does this task motivate me?”, “To what extent do I consider myself capable of performing the task well?”, and “If I perform the task well, will I achieve my personal goals?”) using a slide bar with two ends (“a little”, “a lot”). Participants responded using the slide bar directly and the device converted the response mark into a score ranging between 0 and 100, going beyond the traditional 5- or 7-point range usually used in published work. Considering the nested structure of the data, we followed the guidelines proposed by Shrout and Lane (2012) in order to calculate the multilevel reliability of this scale; the measure showed an excellent between-persons reliability (R_KF = .99) and a moderate reliability of within-person change (R_C = .66).

In order to study within-participant variability, we created two measures per each participant: the variance and the mean square of successive differences (MSSD; see von Neumann et al., 1941; or Revelle & Wilt, 2017). The MSSD is the average of the squared difference between successive observations at time i + 1 and i. Both measures, variance and MSSD, can account for variability over time and have been used extensively among researchers. However, variance only reflects dispersion of the data without considering its temporal nature. MSSD is just able to consider this temporal dependency taking into account the serial correlations (e.g., Jahng et al., 2008).

Data analysis. Data were analysed using descriptive, correlational, and graphical procedures similar to those used in Study 1. Despite the fact that the data collected could be considered as a three level nested data (i.e., occasions nested in days, and days nested in participants), for the sake of parsimony and based on findings that there is evidence of changes in work motivation at momentary level (i.e., within-day measurements) but not at daily level (i.e., across different days of the week), we opted to model the work motivation measure with two-levels (occasions and participants), reflecting what has been done in the majority of published studies sampled and analysed in Study 1. All analyses were conducted using R packages (R Core Team, 2018).

Results

The 6,375 measurement points from the 102 participants reflected an estimated ICC(1) of .46. This means that 54% of the total variance in work motivation resides at the within-participant level (see Bliese, 2000). This value is very similar to the 51.9% that appeared, on average, in the results of Study 1 and to the value of 54.18% obtained in the review of McCormick et al. (2018).

Focusing on a participant level, descriptive statistics and correlations can be found in Table 2. Results indicate that our sample consisted of medium-high motivated workers (M = 75.02) with a substantial amount of variability over time in their motivation levels (values of 332.49 and 202.26 for MSSD and variance, respectively). The two measures of variability (i.e., MSSD and variance) were strongly and positively correlated with each other (r = .93, p < .01). However, the number of repeated measures per participant did not correlate with either MSSD (r = .05, p = .63) or variance (r = .11, p = .25).

Because the correlation analysis considers the full range of values of each variable involved, there is no relationship between the number of measurement points and the variability measures (i.e., MSSD and variance). For this, scatter plots in Figure 3 (A and B) show plain lines indicating the absence of any relationship. However, going a step further and trying to better clarify the situation when we pay attention to the values around 30 measurement points per participants (Figure 3, C and D) we can see how variability increases from the beginning and appears to reach its maximum at approximately 30 measurement points³. Applying an extremum distance estimator (EDE)⁴ to detect the inflection point, the results are between the values 24 and 31. These results are largely consistent with those obtained in Study 1.

Discussion

Having a wide variety of occupations in Study 2 sample was important mainly for one reason: the majority of studies sampled in Study 1 came from homogeneous occupations (e.g., Almeida et al., 2016; Clinton et al., 2017; Debus et al., 2014) and from employees who often worked for the same company and/or department (e.g., Huang et al., 2015; Ilies et al., 2007; Rudolph et al., 2016). This sample homogeneity in the studies analysed in Study 1 may be reflective of a threat to the ecological validity of the results obtained. In Study 2, we addressed this potential threat by collecting a varied sample in terms of gender, age, and occupation. At this point, the results from Study 2 cannot be explained by socio-demographic characteristics, the type of occupation, or the organization involved.

Moreover, another strength of this study is reflected in the number of repeated assessments collected per participant. More precisely, we had an average of 60 measurement points per participant, going far beyond the average 14 measurement points found in the studies considered in Study 1. Of course it is more convenient, in practical terms, to collect around 10-15 measurements per participant. However, one question that remained unanswered was whether this number was large enough to capture all the existing within-participant variability. In fact, the most intuitive expectation would be that the higher the number of repeated assessments collected per participant, the higher the within-participant variance. However, the results yielded from Study 2 do not support this supposition. Instead of finding a linear and positive slope between the number of measurement points and the variance accounted at the within-participant level (i.e., variance and MSSD measures), we found that within-participant variance increased slightly until reaching what appeared to be an asymptotic value around 24-31 repeated measurements per participant. Consequently, the results of Study 2 suggest that 24-31 measurement points are what is needed to estimate well within-person variability in work motivation (RQ2).

Finally, it is important to note again that a substantial proportion of variability in work motivation (54%) resides at the within-participant level. Employees’ levels of motivation vary considerably from time to time, and this within-person variability is at least as important as the differences among workers usually of sole consideration in between-participant designs.

General Discussion

In this paper, we sought to answer four different, but related, research questions. The first two questions were “To what degree is the rule of ‘10-15 measurement points are enough to capture within-person variance’ adequate in the case of work motivation” (RQ1) and “How many repeated assessments are needed to capture well within-person variance in work motivation?” (RQ2). To address these questions, we conducted two studies to analyse 1) the temporal designs of existing studies in the work motivation literature and 2) an independent data collection of a heterogeneous sample where the number of measurement points per participant varied in order to examine its possible effects on within-person variance.

In doing so, we have gone beyond the typical reasoning based on intuition, chance, convenience, and tradition, usually used to support research decisions in temporal designs (Mitchell & James, 2001). Consequently, the present findings offer a clear guideline both to orient future research and to contextualize past work: to capture well the within-person variance of work motivation one would ideally need a minimum of 25-30 repeated measures.

Studying within-person variance means that our focus was on the persistence facet of work motivation. Persistence has been the less studied aspect in this field, in comparison with election and intensity (see Ployhart, 2008). This focus on persistence (i.e., the temporal stability of work motivation) shows us the importance of conceptualizing work motivation as reflecting a dynamic, within-person process that is always in flux. Work motivation shows significant changes from time to time as demonstrated by the amount of within-participant variance revealed in both a review of past work (51.9% in Study 1) and our independent data collection (54% in Study 2).

If over half of the variance of work motivation resides at the within-person level, the focus of research in the field should shift to emphasize new research questions. The first would be to better “describe” these temporal changes in order to clarify possible patterns (e.g., try to describe the presence of possible tipping points in the dynamics, the appearance of bifurcation points, or the presence of ceiling effect, to mention but a few). The second one would be to “explain” these short and medium-term changes. This second issue, which is especially important to continue expanding the frontiers in the field, was far from our objectives. Nevertheless, several proposals have been made, such as the influence of changes in the most proximal antecedents (e.g., Xanthopoulou & Bakker, 2013), the need to consider developmental and maturation processes (e.g., Molenaar & Campbell, 2009), and the importance of learning processes as well. Moreover, although we have not found influence of the amount of repeated assessments obtained per day in Study 1, this should be considered with caution because some possible influence of daily patterns could be present (e.g., circadian rhythm). We were unable to conduct this kind of refined research in Study 1 because the studies reviewed lacked information about the time when the assessment was collected. In addition, there is already evidence of this potential daily pattern. For instance, Curioso and Navarro (2019) found a double-reversed U shape in daily patterns, and based on previous research (e.g., Sonnentag, 2003) attributed the decay of daily work motivation to a fatigue process and the recovery of daily work motivation to the lunch break at midday and the switch off at evening-night.

The interest in the within-person variance is increasing as shown in the literature reviewed in Study 1. Much of the focus has been on exploring new research questions having to do with “when” a worker may be more or less motivated (rather than “which” workers are more or less motivated than others). Additionally, the central role of within-person variability in work motivation that has emerged from this growing line of research also allows us to revisit previous results and determine whether findings based on between-participant designs should be supported (in some cases) or questioned (in others). In either case, this new research line is vital in order to continue advancing the field now that it has been demonstrated that work motivation fluctuates significantly from time to time.

Another relevant lesson that emerges from the present research is based on our finding that different motivation-related variables have different degrees of fluctuation within-person over time (RQ4: Are there differences in terms of within-person variance across different motivational constructs?). At this point, it is true that the results obtained in Study 1 should be treated with some caution if the goal is to compare degrees of within-person volatility across different types of motivational variables. This caution is based on the fact that the number of studies that have examined within-participant variance in a particular variable can vary quite a lot (e.g., from 1 study on task interest to 17 studies on work engagement). There are, however, some general trends that should be considered. As reported in the results of Study 1, job satisfaction is a phenomenon that demonstrates more stability in comparison to other motivational constructs, such as engagement, vigour, or flow, to name but a few. On the other side of the continuum, flow experience appears to be among the most varying variables of the motivational phenomena. It is also worth mentioning how in the case of work engagement, which is the motivational phenomena that received most empirical interest (n = 17), the average proportion of within-participant variance fell at around 50%. This result speaks to the ongoing debate about whether the field should emphasize the enduring nature of work engagement (e.g., Macey & Schneider, 2008) versus its transient nature (e.g., Sonnentag et al., 2010). We found that approximately half of its total variance resided at the between-persons level while the other half resided at the within-participant level. Consequently, this debate will continue.

We now move to the third research question (RQ3): “To what degree does the existing literature capture well the within-person variance of work motivation?”. According to the results obtained in Study 1, current published empirical studies have generally followed the informal rule of 15 measurement points per participant. Therefore, we believe it is safe to infer that the within-person variance in work motivation (or related constructs) is not being properly captured in the existing literature; in fact, it has been underestimated. This underestimation could also influence the results regarding the relationships among the variables investigated in the research included in Study 1. For example, if we underestimate work motivation variability and we study its relationship with other phenomena at higher level (e.g., leadership; Tims et al., 2017) we are overestimating that relationship because this higher level phenomena usually varies less.

Our results indicated that within-person variance is better captured with 25-30 repeated measurements per participant. We can agree that obtaining 30 repeated assessments is not an easy task in empirical research. As it is well known, among the main challenges of intensive longitudinal data collection are how to avoid dropout (i.e., mortality) during the data collection process and how to persuade potential participants to engage in what can be a very time-consuming task. Therefore, we empathize with the fact that our recommendation to increase the previous informal rule of 10-15 repeated assessments to 25-30 will not be joyfully welcomed. This result clearly represents an increase in time, consequently more effort, and more potential challenges. We believe that the high participation (low mortality) we got in Study 2, where only three people withdrew, was due to the fact that we used a very short-form questionnaire that took one minute or less to answer. Other procedures can also be applied, such as using parallel forms of the questionnaire, using behavioural data by the observation analysis or extending the period to obtain the data.

Summing up, if researchers wish to fully capture he unfolding cycles of work motivational processes, more than 10-15 repeated measures will be required. In short, we as researchers of work motivation will have to go further. The reward will be more rigorously-designed studies with more consistent results. After the amount of evidence already generated in the field of within-person work motivation processes, we firmly believe that we should move to expand the field by doing more and better-designed research. Doing better means, in this case, that we should avoid following the aforementioned informal rules.

Limitations and Future Research

We wish to highlight some study limitations. First, the majority of the research considered in Study 1 (62%) consists of intensive longitudinal designs where repeated measurements were collected on a daily basis (i.e., one measurement obtained per day). Taking this feature into account, we mainly considered such studies as being short-term focused, which will probably not generate consensus since there is also research focusing on changes in motivation over minutes or hours (e.g., Lord et al., 2010). However, we did not include these investigations as they did not fit all our inclusion criteria (e.g., include explicit information about variance decomposition in this case). Therefore, we have to bear in mind that by short- and medium-term changes we are essentially considering day-to-day changes in work motivation.

Second, all the research reviewed and summarized in Study 1, as well as our original data collection in Study 2, used self-report questionnaires. This characteristic can also influence the amount of within-person variability obtained. The use of other types of measures is, however, uncommon in our discipline. For example, the use of psychophysiological measures as indicators of work motivation is rarely (if ever) seen (Ployhart, 2008).

Third, a related issue would be the potential interaction between the signalling to the participant by having increased the number of measurement points and the participant awareness of being measured. The increment of measurement points could increase participant self-awareness influencing exactly in the measured obtained. Individual differences such as emotional intelligence, personality traits (e.g., emotional stability) or self-monitoring can moderate this relationship, which should be clarified in future research.

Of course, within-participant variations can also change over time. We mean, it is possible to expect that these fluctuations can be higher at some moments (e.g., in the case of new employees, employees who take new roles). Moreover, it would be important to identify if data is being collected during routine handling periods or exceptional work contexts such as following a perturbation, as different scenarios are potentially strong influences in this level of volatility on work motivation. These issues should be addressed in future research. Continuing with future research directions, we have to say that we have concentrated our attention on work motivation; however, other important organizational behaviour processes (e.g., work attitudes such as commitment, performance, stress, etc.) could also be studied in order to clarify the number of measurement points needed to capture well within-participant variance. At this point, our view of the most interesting outcome of this study would be to demonstrate that different organizational phenomena required different temporal windows to fully unfold.

Conclusion

Work motivation has received a lot of attention in the research literature on Work and Organizational Psychology. Currently, much of said research makes use of longitudinal designs in order to study the persistence facet of work motivation. From that line of research, an informal rule emerged whereby 10-15 repeated assessments per participant was considered sufficient to capture within-person motivational dynamics in a reliable way. However, we were able to ascertain across two studies that we very likely will have to increase this number to 25-30 repeated measurements per participant if we are interested in fully capturing the unfolding cycle of work motivation. With only 10-15 repeated measurements, researchers are very likely underestimating the amount of within-person variability in work motivation. We are well aware that obtaining 30 measurement points per participant is not easy; however, our findings suggest that this is the number needed (or at least 25) in order to reveal a more complete and well-founded picture of work motivation dynamics. Finally, in our opinion, current research directions in the field of work motivation are just now starting to explore the central proximal causes and consequences of these short and medium-term changes. To assume this challenge and expand the frontiers of knowledge on the dynamics of work motivation, we would have to increase the rigour of our temporal designs in the assessment of within-participant variance by going beyond the use of informal rules.