Telehealth Competency Questionnaire-Consumer: Psychometric Validation of a Client-centered Measure

To effectively access telehealth services, individuals must possess certain competencies; yet, telehealth consumer focused measures are limited. The purpose of this study was to describe the development and validation of the Telehealth Competency Questionnaire – Consumer (TCQ-C). Among a sample of adults with chronic health conditions (n=134), findings showed that the TCQ-C is comprised of one factor that accounts for 66.6% of the variance, and internal consistency of subscales are good (range α = 0.80–0.87) and may be used for clinical or research purposes. The TCQ-C demonstrated moderate concurrent validity with the Telehealth Usability Questionnaire-Usability subscale (r = 0.728, p<.001), and significantly discriminates between adults >65 years and those younger as well as those with and without previous telehealth experience. The TCQ-C is a psychometrically sound instrument to evaluate baseline competencies among telehealth consumers so that education, research, and clinical practices are tailored to increase effective engagement between clients and providers.


International Journal of Telerehabilitation
To this end, researchers have developed measures to assess specific aspects of consumer experiences of telehealth.In a recent review of consumer-focused questionnaires used to evaluate telehealth services, Hajesmaeel-Gohari and Bahaadinbeigy (2021) found that the most commonly used measures evaluate satisfaction, usability, acceptance, and implementation.Authors concluded that a questionnaire with few questions and more comprehensiveness is needed.Therefore, the purpose of this article is to describe the development and psychometric validation of the Telehealth Competency Questionnaire -Consumer (TCQ-C).Specific research questions included: (1) What is the factor structure of the TCQ-C?; (2) What is the internal consistency of proposed subscales of the TCQ-C?; (3) What is the concurrent validity of the TCQ-C with the Telehealth Usability Questionnaire? (Parmanto et al., 2016); and (4) What is the discriminant validity of the TCQ-C based on age and previous use of telehealth?

Method Measure Development
We developed the TCQ-C using a 5-point Likert scale (1=highly agree to 5=highly disagree) following the guidelines of DeVellis and Thorpe (2021).To guide item development, we consulted research, existing measures, and recommendations about the knowledge, skills, and attitudes (i.e., competencies) necessary to engage in telehealth (e.g., Association of American Medical Colleges [AAMC], 2021; American Academy of Ambulatory Care Nursing [AAACN], 2018;van Houwelingen et al., 2016).After a review of recurrent themes among available literature, we outlined five areas of competency as most relevant for telehealth consumers (see Table 1).

Telehealth Usability Fundamentals
Telehealth concepts; considerations for use; safety issues; privacy; client rights AAACN, 2011;AAMC, 2021;Hollander et al., 2018;Luxton et al., 2014;van Houwelingen et al., 2016 Troubleshooting Telehealth hardware and software problem solving; requesting accommodations AAMC, 2021;Hollander et al., 2018;Luxton et al., 2014;van Houwelingen et al., 2016 Virtual Rapport Environmental and in-session strategies to develop rapport through telehealth; selfadvocacy AAACN, 2011;AAMC, 2021;Agha et al., 2009;Elliott et al., 2020;Henry et al., 2017;Hollander et al., 2018;Luxton et al., 2014;van Houwelingen et al., 2016 Care Planning Virtual assessment types; shared-decision making; post-session follow-up AAACN, 2011;AAMC, 2021;Elliott et al., 2020;Hollander et al., 2018;Luxton et al., 2014;van Houwelingen et al., 2016 Participation in Team-Based Care Telehealth providers, roles and responsibilities; effective communication; team-based decision making AAACN, 2011;Hollander et al., 2018;Milani et al., 2014 After competency areas were defined, the authors developed a preliminary list of 15 items.An interdisciplinary team of healthcare providers (n=5), identified as content experts in telehealth at Rush University Medical Center, reviewed the list of competencies.Based on feedback, the authors refined the list of competencies, reduced the number of items to 13, and revised language to ensure a 7 th grade reading level to align with average-difficulty health literacy accessibility recommendations from the U.S. Department of Health and Human Services (2010).

Participants
We electronically administered the TCQ-C to a convenience sample of adults with one or more chronic health conditions (n=134) via REDCap (Harris et al., 2019).We recruited the sample from a university registry of older adults that participate in interprofessional education activities.To be eligible, participants needed to be 18 years or older, community dwelling, and have one or more chronic health condition.We included those with chronic health conditions, as they would necessitate telehealth visits beyond preventative care and often interact with a team of healthcare professionals.Demographics of our sample are provided in Table 2.

Data Analysis
We used SPSS 28.0 to analyze data.To address the factor structure of the TCQ-C, we used principal components analysis (PCA).To determine the number of factors, we considered eigenvalues over 1.0 and item loadings (Guttman, 1954).We considered items with loadings >.32 (Tabachnick et al., 2013) and considered items as cross-loading if they loaded within 0.1 on more than one factor (Ferguson & Cox, 1993).To address the internal consistency of proposed TCQ-C, we used Cronbach's Alpha, and we used Pearson correlations to evaluate the concurrent validity of the TCQ-C with the TUQ Usefulness subscale, which has good internal consistency (standardized α = 0.85) (Parmanto et al., 2016).Lastly, we used ttests to determine discriminant validity between adults <65 years and >65 years as well as those that had previously used telehealth versus those that had not.

Results
Factor structure of the TCQ-C.We ran principal component analysis (PCA) on the 13-item TCQ-C questionnaire.Prior to PCA, we assessed suitability of the data for factor analysis and found many coefficients were greater than 0.3.The Kaiser-Meyer-Oklin value was 0.945, exceeding the 0.6 cutoff of Kaiser (1974) and Bartlett's Test of Sphericity (Bartlett, 1954) was significant at p>0.001; supporting factorability of the correlation matrix.The analysis revealed the presence of two factors with eigenvalues exceeding 1.0; however, inspection of the scree plot revealed a clear break after the first component (See Figure 1), with the second factor accounting for only 8.6% of the variance.

Principal Component Analysis Scree Plot
Parallel analysis confirmed one component with eigenvalues exceeding the criterion values from a randomly generated data matrix of the same size (13 items, 134 participants).The factor solution explained a total of 66.7% of the variance.We performed Oblim rotation to aid in the interpretation of the components.The rotated solution resulted in strong item loadings on one factor, and no item cross-loaded on two factors >0.3.Range of item loadings was 0.71-0.88(see Table 3).Note.The TCQ-C uses a 5-level Likert scale for all items ('1': strongly agree; '2': agree; '3' neutral; '4': disagree; '5': strongly disagree).Internal Consistency of TCQ-C Subscales.While PCA showed a one factor solution, we wanted to investigate the internal consistency of proposed subscales for clinical, education, and research purposes.Results showed the following values: Telehealth Usability Fundamentals subscale α = 0.81; Troubleshooting subscale α = 0.80; Virtual Rapport subscale α = 0.81; Care Planning subscale α = 0.87; and Participation in Team-based Care subscale α = 0.85.

International Journal of Telerehabilitation
Concurrent Validity of the TCQ-C.Pearson product-moment correlation coefficient between the TCQ-C and TUQ Usability subscale showed a moderate, positive correlation between these two variables, r = 0.728, p<.001.
Discriminant Validity of the TCQ-C.To investigate the discriminant validity of the TCQ-C based on participant age, we divided the sample between those 65 years and older (n=77) versus 64 years and younger (n=57).T-test results showed significant differences between age groups in total TCQ-C score (t= -1.61, 132, p<.001), with older adults reporting worse skills and abilities in using telehealth (M=2.82;SD=0.86) as compared to younger adults (M=2.40;SD=.0.82).T-test results showed significant differences between those that had not previously used telehealth (n=70; M=2.20, SD=0.87) versus those that had (n=64; M=2.94, SD=0.68) (t=2.66,132, p<.001).

Discussion
In alignment with recommendations of NACT, consumer-oriented trainings may be one approach to address inequities in telehealth access.However, it is first necessary to understand perceptions of telehealth competency where disparities presently exist, and how disparities may be influenced by the intersectionality of systemic barriers.Therefore, the purpose of this study was to describe the development of a measure assessing telehealth consumers' perceptions of competency (i.e., the TCQ-C).If we can reliably capture the areas in which consumers report variable levels of competency, we can design education, clinical, and research programs to best target their needs.
The TCQ-C was comprised of one factor, which we describe as overall consumer telehealth competency.All items loaded within the moderate to high range on one factor, and when we investigated the two-factor solution, no items loaded within 0.3 within one another, offering further support for the one factor solution.While the measure shows evidence of one factor, we did find high internal consistency values for the proposed subscales of the TCQ-C, which include Telehealth Usability Fundamentals, Troubleshooting, Virtual Rapport, Care Planning, and Participation in Team-based Care.We argue that these subscales may be important clinically, so that a full range of skills associated with effective telehealth access may be assessed.
The majority of consumer-focused measures of telehealth assess satisfaction, usability, and acceptance (Hajesmaeel-Gohari & Bahaadinbeigy, 2021).Therefore, the TCQ-C provides a more comprehensive picture of the range of skills needed to access telehealth.While there are fundamental skills needed for a consumer to participate in telehealth sessions (e.g., logging on to the virtual platform), the TCQ-C also captures the interpersonal skills needed for those with chronic conditions to effectively manage their care, such as being an active member of their healthcare team and communicating with multiple healthcare providers.
Lastly, the TCQ-C showed moderate concurrent validity with the TUQ-Usefulness Subscale, which we would expect given that the TCQ-C assesses a wider range of behaviors and perceptions associated with telehealth use.The TCQ-C also demonstrated the ability to discriminate between adults >65 years and those younger and those with and without previous telehealth experience.Given that the digital divide disproportionately impacts older adults (Faverio, 2022), such evidence of discriminant validity shows that the TCQ-C may be particularly helpful in assessing consumer baseline skills to develop tailored training programs for specific needs and specific individuals.

Limitations
The current study presents with limitations.First, the TCQ-C represents a consumer's perception of their competencies and may not align with objective or observational measures of skills demonstrated by a consumer during a telehealth encounter.Second, as telehealth continues to expand and new virtual platforms are developed, consumers may become increasingly competent in accessing services or new areas of competency may arise.Lastly, our sample was limited to those with chronic conditions in one geographical area and samples of those without chronic health conditions or from other geographical areas may present with different competencies and/or needs within telehealth encounters.

Conclusion
Telehealth demonstrates potential to increase access to health services where disparities exist, yet consumer-oriented training is necessary to promote equity among prospective users.The TCQ-C is a measure with demonstrated validity to assess user perceived telehealth competency.It may be used to increase our understanding of the unique telehealth training needs among populations for whom perception of competency is a barrier.

Table 1
Telehealth Consumer Competency Domains

Table 3
Telehealth Competency Questionnaire -Consumer (TCQ-C) Factor Loadings and Subscale Internal Consistency