Abstract
Risky sexual behaviors in adolescents and young adults often result in negative outcomes such as sexually transmitted infections (STIs) and unwanted pregnancies. This case study compares the experience of implementing a digital app-based platform aimed at reducing adolescent sexual risk behavior in a rural medical clinic and a university-based mental health clinic in an effort to better understand how the cliexa-OPTIONS mobile app can inform care and enhance the overall quality of treatment in very different clinical settings. Clinicians in charge of implementation of the digital app report that it opens up more transparent discussion with patients/clients, allowing them to provide better care in both medical and mental health settings. The differences in context provides different demographic-related challenges at the two sites. Challenges in online recruitment and workflow were evident only at the university-based mental health clinic as a result of the COVID-19 pandemic. Implementers at both sites indicated a highly positive overall experience with cliexa-OPTIONS and a desire to use the digital app in the future contexts. Patient/client satisfaction surveys indicate that patients/clients enjoy using the digital app and find it easy to use.
A Comparative Case Study of the Implementation of a Digital App-Based
Platform Aimed at Reducing Sexual Risk in Adolescents and Young Adults
Risky sexual behaviors often increase the likelihood of adverse health outcomes and long-term consequences such as unplanned pregnancy and sexually transmitted infections (STIs) (Cederbaum et al., 2017). Given the widespread increase in technology use, mobile applications have become a popular medium for disseminating information and augmenting traditional healthcare. App-based platforms targeting sexual and reproductive health show promising results in the promotion of comprehensive sexual education for the adolescent and young adult populations (Gannon et al., 2020; Steinberg et al., 2018; Brayboy et al., 2017). The purpose of this case study is to compare the experience of implementing a digital app-based platform aimed at reducing adolescent sexual risk behavior in both a rural medical clinic and a university-based mental health clinic in an effort to better understand how the cliexa-OPTIONS mobile app can inform care and enhance the overall quality of treatment. An overview of approaches to reduce adolescent sexual risk behavior, characteristics in different treatment settings, and components of app-based platforms are discussed.
Adolescent Sexual Risk Behavior
Since 2014, sexually transmitted infection (STI) rates have steadily increased among adolescents (Shannon & Klausner, 2018). It is estimated that approximately 50% of the 20 million new cases of STIs diagnosed in the United States each year occur in adolescents aged 15-24 (Workowski & Bolan, 2015). This includes approximately two-thirds of chlamydia infections and an overwhelming 70% of gonorrhea infections (Centers for Disease Control and Prevention, 2013). A clear consensus regarding the definition of sexual risk appears to be lacking in the literature. The term generally refers to inconsistent or lack of contraception, a high number of lifetime partners, substance use related to intercourse, and sexual activity with a partner who either has an infection or is at a higher risk of contracting one (O’Connor et al., 2014). Research shows that, unlike adults, most adolescents have not developed the cognitive processing skills that are necessary to identify risky situations. This population is more susceptible to peer pressure and emotionally-charged decision-making (Reyna & Farley, 2006). Given that adolescents and young adults are at a higher risk for long-term consequences of sexual risk behavior, there is a clear need for more research that identifies techniques and resources that can help adolescents make better, more informed decisions.
Approaches to Reducing Adolescent Sexual Risk Behavior
Most teens get their information about sex from school, parents, friends, and the media (Bleakley et al., 2009; Gray et al., 2005). Various approaches have been used in an attempt to reduce adolescent sexual risk behavior. The theory of reasoned action, a social cognitive model of behavior, posits that attitudes and perceived norms have a direct influence on intention to engage in a certain behavior and that an individual’s level of intention can predict subsequent engagement (Cederbaum et al., 2017; Fishbein & Azjen, 1975). Research has illustrated that parent-adolescent communication around sexuality leads to safer sexual behaviors (Widman et al., 2015). One study utilized a mental model approach for decision making and found that an interactive video intervention helped to reduce risky sexual behavior among young women (Downs et al., 2015).
Parents have been shown to be just as effective as health experts in delivering messages related to sexual abstinence or risk-reduction strategies (Dancy et al., 2006). However, some studies showed mixed results in terms of parental effectiveness, as messages from parents typically occur after engaging in sexual activity and are more likely to contain outdated information. Programs were found to be more effective in reducing sexual behavior when abstinence was not the predominant focus. Condom use was increased when teens were provided with information on how to use condoms or motivational training (Johnson et al., 2011). Specific interventions using videos were also shown to be associated with higher condom-related behaviors and overall clinical outcomes (O’Donnell et al., 1998; Neumann et al., 2011).
In school-based sex education programs, the use of concrete standards has been found to reduce sexual risk behavior, but many programs still lack funding for the proper implementation of these interventions (Kirby, 2007; Kirby 2006). One major concern that is apparent throughout the literature is a lack of fidelity in the delivery of sex education programs, which may contribute to a limited understanding of the factors influencing the adolescent sexual decision-making process (Cohen et al., 2008). Less effective interventions often lack a strong theoretical approach and a clear connection to change techniques (Michie & Abraham, 2004). While some approaches have been useful, further research is warranted to determine additional techniques that can enhance sexual education programs and interventions.
Adolescent Sexual Risk Behavior in Rural Medical Clinics
In a recent study utilizing a cognitive behavioral therapy (CBT)-based intervention to reduce sexual risk and behaviors in rural Georgia, researchers found that diverse teaching and recruitment strategies, expansion in education, and risk reduction efforts are needed to better equip rural clinics (Hallum-Montes et al., 2016). In rural Southern counties, there are various discrepancies impacting the healthcare of adolescents and young adults. These areas face barriers such as limited access to sexual and reproductive health services, more conservative community attitudes, increased sexual risk behaviors, confidentiality issues, lack of transportation, and limited sex education programs (CDC, 2014). Across the literature, most sex education and intervention programs appear to be originally developed for youth in larger metropolitan areas (Ott et al., 2011; Stanton et al., 2006; Stanton et al., 2005). This highlights the need for enhanced evidence-based practices that can be disseminated and effectively utilized in diverse clinical settings.
Adolescent Sexual Risk Behavior in Mental Health Clinics
Adolescents diagnosed with psychiatric disorders are at an increased risk for HIV and other sexually transmitted infections (Cunningham et al., 2017). Mental health problems may exacerbate sexual risk directly as a result of irrational thought processes, decreased affect regulation, and impulsive behavior, or indirectly through poor family, peer, and romantic relationships (Donenberg & Pao, 2005; Brown et al., 1997). A meta-analysis of 14 studies of adolescents in both inpatient and outpatient mental health clinics found that 69% of adolescents reported having sex, compared to 41% who endorsed sexual behavior within the general adolescent population (Kann et al., 2016).
Outpatient settings tend to have higher rates of reported condom use (80%) when compared to inpatient settings (41%), which may be attributed to the higher number of resources and accessibility to contraceptive methods found in outpatient clinics (DiClemente & Ponton, 1993). There is some evidence to suggest that peer support of substance use mediates the pattern of externalizing problems and subsequent sexual risk in African American adolescent girls, however, additional research is needed to determine associations within other ethnic and racial backgrounds (Donenberg et al., 2018). Current recommendations for treating adolescent sexual risk behavior in mental health clinics include using evidence-based practices (Cunningham et al., 2017), careful screening during psychiatric treatment (Brown et al., 2010), and addressing mental health and familial contexts in early adolescence (Donenberg et al., 2018).
App-Based Platforms
Mobile applications have revolutionized the American healthcare system and have allowed for the expansion of treatment for various mental and general health concerns. A recent study looked at the nature of self-efficacy and behavioral changes using health mobile apps and found that self-efficacy influences methods and behavioral change models, with mobile apps improving personalized care plans and goal-setting (McKay et al., 2019). Mobile users utilizing general health applications have better health outcomes and app-based health interventions have demonstrated effectiveness in the improvement of health behaviors (i.e., physical activity, weight control) among the general population (Lee et al., 2018). App-based platforms have also been effective in health promotion for specific issues, such as the provision of education to parents during the postnatal period (Shorey et al., 2018) and increasing knowledge and awareness of childhood burn injuries (Burgess et al., 2016). Mobile apps for health-related information are receiving increasing global awareness, but further research is needed to determine their effectiveness within developing countries (Lee et al., 2018).
Mental health applications can serve to provide psychoeducation, monitor symptoms, teach management and coping skills, and facilitate treatment adherence (World Health Organization, 2012). They also benefit the existing healthcare systems by increasing accessibility to treatment services, disseminating evidence-based interventions, and destigmatizing the treatment of mental health concerns (Reger et al., 2013). These apps provide specialized education and care for a variety of populations. For example, IntelliCare was created to improve the mental health of college students (Lattie et al., 2020). However, mental health apps are typically abandoned by individuals in a short time span. The IntelliCare app provided users with feedback regarding their symptoms and discovered that personally relevant observations were related to an increase in cognitive and behavioral coping strategies (Lattie et al., 2020). Some apps have also been shown to significantly decrease depressive symptoms after use (Ha & Kim, 2020). Results of a systematic review within the Veteran Affairs healthcare system found strong support for the overall feasibility and acceptability of mental health applications with users (Gould et al., 2019).
App-Based Platforms and Behavior Change
The number of behavior change techniques found in mobile apps appears to be lacking, which suggests improvements can be made in-app designs that would promote lifestyle behavior changes and a move towards better overall health (McKay et al., 2019). A recent systematic review found no strong evidence to support improvement in health behaviors or outcomes via the use of a mobile app (Milne-Ives et al., 2020). Additional research is needed in this area to identify specific techniques that are effective for behavioral change. Motivational interviewing through an app platform has shown promise, but the interactive and feedback features need further investigation and longitudinal studies to determine how effective this strategy is within a virtual medium (Nurmi et al., 2020). Overall, the research is limited regarding how apps can assist with behavior change and what specific features may be most effective. If additional information is gained about how apps can promote behavior change, these platforms may become a viable tool to help adolescents and young adults make healthy decisions.
App-Based Platforms and Sexual Health
Mobile apps show potential to advance the field of sexual and reproductive health (Logie et al., 2020). Some sexual education apps even feature a locator search option for seeking healthcare providers, promoting a wide array of sexual health services and contraceptive methods (Steinberg et al., 2018). Others provide comprehensive information related to anatomy and physiology, STI prevention, and sexuality in relationships, with users reportedly consuming more information than they would in a typical health class (Brayboy et al., 2017). A mobile sexual health app (MyPEEPS Mobile), initially developed for adolescent men in NYC, shows promising results for promoting HIV prevention, and individuals noted the ease of use, functionality, and inclusivity of diverse identities as strengths of the app (Gannon et al., 2020). While the research base is growing for sexual and reproductive health apps, a global survey of healthcare providers found that approximately 40% feel informed, almost half of providers indicated needing more information, and 15% expressed interest in additional training (Logie et al., 2020). This evidence suggests that sexual health interventions could become more effective via mobile apps with increased provider training and structured implementation.
App-Based Platforms with Adolescents
Some apps developed for use with adolescents have shown mixed results. One study focusing on migraine treatment only found significant improvement in adherence within the first month of the intervention and noted that parents might need to provide behavioral incentives to promote sustained treatment adherence (Ramsey et al., 2018). Another qualitative study of smartphone apps for the improvement of sleep behaviors discovered barriers to sleep hygiene that included adolescents straying away from routines on the weekends and feeling unsure of their abilities to follow through with healthy sleep practices (Quante et al., 2019). Mental health applications, such as CopeSmart, aim to promote emotional monitoring and positive coping strategies. However, a recent study revealed this particular smartphone intervention was ineffective in improving mental health outcomes (Kenny et al., 2020). Barriers to achieving effective use of apps with adolescents include lower levels of engagement and psychological distress, which may be improved with provider check-ins to discuss difficulties adolescents may experience with various app interventions (Kenny et al., 2020; Ramsey et al., 2018).
Current Study
Current evidence indicates promising results for app-based platforms as a medium of sexual education and reproductive health information for adolescents and young adults. However, Cohen et al. (2008) noted a lack of fidelity in delivery approaches, possibly due to educators’ limited understanding of the factors that influence adolescents’ decisions. Examining the use of cliexa-OPTIONS across a mental health clinic and a rural medical clinic may aid in the development of culturally tailored sexual health interventions, which could reduce unplanned pregnancies and STIs within the adolescent and young adult population. Researchers have noted the need for future research in the area of mobile app platforms for sexual and reproductive health, specifically for underserved groups such as adolescents (Logie et al., 2020). By comparing the use of this app across two diverse settings, healthcare providers and educators may gain a crucial understanding of the factors that influence the implementation of a digitalized platform to assess adolescent sexual risk behavior. Successful implementation of the cliexa-OPTIONS app in multiple clinics can subsequently provide additional information about adolescents’ decision-making processes and aid in the cultivation of individualized treatment plans. The current case study is guided by the following question: How have a rural medical clinic and a mental health clinic experienced implementing a digital app-based platform aimed at reducing adolescent sexual risk behavior? The results may provide valuable information that can help increase the capacity for diverse clinical settings to provide integrated follow-up care for adolescents and young adults.
Methods
Methodological Framework
The goal of a case study is to understand a particular context rather than attempting to discover general truth (Merriam, 2009; Stake, 1995). Through investigating a particular case, concrete universals (Erickson (1986) can emerge, bringing insights that illuminate larger truths about the phenomenon under consideration. Flyyvberg (2011) points to the strength of case study as “depth—detail, richness, completeness, and within-case variance” (p. 314). It also offers “an in-depth exploration from multiple perspectives of the complexity and uniqueness of a particular project, policy, institution, program or system in a ‘real life’ context” (Simon, 2009). The case study is particularly useful for “how” and “why” questions, for complex settings (Yin, 2014).
Digital App-Based Platform Description
This comparative case study examined the implementation of a digital app-based platform in two distinct settings: a rural medical clinic and a university-based mental health training clinic. The digital app-based platform aimed to reduce adolescent sexual risk behavior. This study defines adolescents as a person ages 15 to 24 years in alignment with ages defined by the U.S. Department of Health and Human Services. The app-based platform is for individual participant use with an electronic report that is generated and shared with providers of consenting participants at the respective site. The app integrates assessments for sexual risk and can include additional measures for anxiety, depression, substance use, and resilience. After an adolescent entered their information and responses to the screening tools, the medical or mental health provider was given a report that summarized data from the adolescent participant. This report was used to provide more individually informed follow-up care. Participants also received a report for their own knowledge.
Site Descriptions
For this comparative case study, two sites with different goals and patient populations were selected. Both sites were in the same western state; however, one site was in a rural area of the state and the other in a suburban area. Both sites were new to integrating the digital app-based platform, meaning they were not using the platform before the study began. Notable differences between the two sites include that the rural site was medically-focused with an emphasis on family planning while the other site was a university-based mental health training clinic serving clients who were primarily seeking services for mental health needs rather than reproductive and sexual health needs. Further details about each site are provided below.
Rural Medical Clinic
This family planning clinic served a rural county in the western part of the state with a population of about 43,000 spread over 2,340 square miles. About 76% of the population is White with another 20% identifying as Hispanic of Latino. Just over 50% is female. The median household income is $50,489 dollars per year and 13.2% of the population consists of those living in poverty. While the clinic closed its operations in the summer of 2020 after its Title X funding expired, it previously offered a full range of family planning services, including Pap tests, comprehensive birth control counseling and supplies, pregnancy testing and referrals, individual counseling, testing, and treatment for STIs, emergency contraception, and free and low-cost mammograms.
Site Supervisor Description. Christine (a pseudonym) was previously the nurse practitioner working at the rural family planning clinic, which closed its operations earlier this year, with Title X funding expiring in the summer of 2020. She served as the primary contact at this location for the implementation of the cliexa OPTIONS digital screening program and the associated Department of Health and Human Services grant-funded research. Christine has worked in family planning for twenty-eight years.
Site Patient Demographics. Participants from the rural health clinic completed a patient satisfaction survey after using the digital app-based platform. The sample (N = 35) was made up of 30 females and 5 males. The average age was 19.5. The sexual risk category, total partners, undesired pregnancies, and last STD check are summarized in Table 1.
Mental Health Training Clinic
This mental health training clinic served a university campus, including students, faculty, staff, and their families, as well as local community members. Mental health practitioners in training had consistent contact with their clients aside from this study through individual, family, and group counseling or psychological assessments. Given the wide-reaching nature of services offered through this clinic, potential participants presented with a variety of mental health needs, including but not limited to anxiety, depression, adjustment, grief and loss, trauma, and stress. All clients or patients were offered mental health services regardless of ability to pay. The mental health clinic has provided services within the area for over 50 years.
Site Supervisor Description. For this study, the site was provided personnel from the research team to recruit and meet with participants. The site supervisor, Karen (a pseudonym), met individually or with groups of persons already receiving mental health services when recruiting participants. Karen was a doctoral student trained in the digital app-based platform as well as a mental health practitioner who was able to conduct follow-up contact as needed with study participants based on study protocol. She was in her second year as a research assistant and site supervisor for this study.
Site Patient Demographics. Participants from the mental health training clinic completed a patient satisfaction survey after using the digital app-based platform. The sample (N = 25) was made up of 16 females, 2 males, and 7 individuals who did not indicate gender. The average age was 18.6. Sexual risk category, total partners, undesired pregnancies, and last STD check are summarized in Table 1.
Procedure
As part of a larger study aimed at reducing adolescent sexual risk behavior, this comparative case study received approval from the lead investigators’ Institutional Review Board (IRB). Sites described above opted into the study to learn about and have access to the digital app-based platform. No financial compensation was provided to the site or the medical or mental health practitioners at the site. Research personnel, however, were provided to sites to assist with recruitment and implementation of the platform. One individual with the research team was provided to each site part-time for the duration of approximately one year. Each site was provided between one and three tablets with the digital app-based platform loaded on the tablet. Tablets also had electronic consents that were provided before being able to enter data into the platform. Sites has access to use the digital app-based platform for medical and mental health purposes regardless of whether participants or their parents, depending on age, consented to the study. Study personnel was present at time of consent to respond to any questions participants or their parents had about the study or platform. Once consent was obtained or declined, adolescents completed the measures independently without study personnel present.
Data entered by participants in the platform resulted in a report used by study personnel to conduct follow up check ins regarding sexual health risks. These follow-ups, while not the focus of this study, were informed by what the adolescent participants identified they wanted to change, reduce, increase, or discuss with their provider or study personnel. After participants entered data in the digital app-based platform, they were sent a patient satisfaction survey to better understand the platform itself from the participant perspective and experience. Descriptive statistics from these surveys served as one source of data for this comparative case study. All adolescents who participated in this study were provided a $10 Target gift card regardless of completion of the patient satisfaction survey.
After each site implemented the digital app-based platform for approximately one year, each site supervisor was interviewed regarding implementation, benefits, and barriers to the platform. A semi-structured interview protocol was used with both site supervisors who were uniquely positioned to provide insight into implementation, benefits, and barriers when integrating the platform at their respective site. One of the researchers who had minimal to no prior contact with site supervisors conducted the semi-structured interviews. Site supervisors were not provided any incentives for participation.
Data Analysis
Patient satisfaction surveys were analyzed for mean scores from the Likert scale questions about the platform itself. Patient satisfaction surveys did not include open ended, qualitative responses. Thirty-five participants were provided patient satisfaction surveys at the rural medical clinic and 32 responded. Twenty-five participants were provided patient satisfaction surveys at the mental health training clinic and 3 responded. Mean scores for each of the survey questions are reported by site in results.
Interviews with site supervisors were semi-structured and lasted about 20-30 minutes. Each site has one site supervisor, and both participated in this study. The interviews were conducted in an informal conversational style but covered the nine questions summarized in Table 1. Interviews were transcribed verbatim for coding. Open coding occurred independently from two researchers, then thematic coding was conducted by both researchers. One researcher was the one who conducted the interviews and the other was a clinical supervisor on the research team. MAXQDA qualitative coding software was used and consensus was reached by both researchers for all thematic codes reported. Results reflect which thematic codes were endorsed by both sites, one site, or the other site in implementing the digital app-based platform. Themes evident across sites as well as those specific to one site only are summarized in Figure 1.
Table 1
Qualitative Interview Questions
| 1. Please describe your overall experience running the cliexa-OPTIONS digital screening at Montrose County Family Planning. What was the process like?
2. Describe how you integrated the digital screening into your normal workflow? 3. What barriers (if any) did you faced in the initial implementation of the digital screening tool? 4. How important was an additional member of staff in allowing the clinic to use the digital screening? 5. Describe your experience using the digital screening was in terms of providing additional information about patients. 6. Describe the type of information you received from your patients. Was it about the same or different? If it was different, how? 7. What improvements, if any, did you notice in the quality of interactions or services that were provided in your clinic using cliexa? 8. If you had another opportunity to use the digital screening at another site, would you choose to use it? Why or why not? 9. Is there anything about the cliexa-OPTIONS digital screening that might be improved to make it easier to implement or more effective in terms of patient outcomes? |
Results
Despite the differences in context between the two sites, there appear to be common experiences for clinicians in terms of the positive benefits of using the digital app-based sexual risk assessment, with both sites reporting both ease of use and increased information the digital app allows them to obtain from patients and clients, improving the level of care provided in both a medical and mental health care setting. Differences arise in terms of implementation and certain challenges primarily related to the specific contextual differences in the sites—such as staffing structure and demographic characteristics of patients and clients. Available patient satisfaction data also indicates an overall positive experience using the digital platform. The following sections summarize 1) the results of interviews with the clinicians responsible for implementing the digital app use at each site and 2) the patient/client satisfaction survey results.
Clinician Experience
Common Themes Across Sites. Both participants reported generally positive experiences with cliexa-OPTIONS, specifically in terms of ease of use, the value of the additional information provided by the digital assessment, and willingness to use the digital app in the future. Both agree that an additional member of staff is crucial to making the digital application work effectively.
Ease of Use. Christine says, “Getting the assessment up and running was a fairly simple process for us because Laura [a cliexa representative] did most of the footwork and so that was simple from our point of view. I don’t know what we would have done without her, especially with only two of us in the clinic (our staff went from six to two). Laura was very instrumental in taking care of the technical details.”
The digital screening appears to have been integrated quite seamlessly into the normal workflow of the clinic, primarily with the help of a medical/research assistant who incorporated the iPad questionnaire into the workflow—regular assessment and vitals—enabling Christine to have “a wealth of information” available to her by the time she saw the patient. Christine notes, regarding the assistant, “It seemed very easy for her to incorporate the questionnaire into the time period before I would see the patient.”
Karen notes that the digital app “makes the screening process more efficient.” While Christine had specific help from cliexa in implementing the digital app, particularly because of the clinic’s staffing situation, the mental health clinic is “fully staffed by master’s and doctoral level students who are taking practicum classes.” Karen describes implementation mostly in terms of talking to the members of these classes, explaining the project and “asking counselors to introduce it to their clients.” There were initially three people working on the implementation, which streamlined the process.
Additional Information Provided by Digital App. Christine says, “Once the digital platform was up and running, it just added so much depth. Being able to have young people have access to it gave us so much valuable information, and I was able to increase their care to a level I would have not been able to achieve otherwise. I found that often the young person was more transparent and felt more comfortable expressing their needs and their fears and their anxiety using the iPad, therefore having that extra assessment gave us more information to increase level of care in the family planning setting.” Christine further recalls the experience of a young girl who came into the clinic seeking family planning, initially without her mother but later accompanied by her mother (as recommended by the state if possible). During the consultation, Christine discovered that the girl had had contact with an individual who had syphilis. The girl told Christine that being able to express that contact initially on an iPad made it so much easier for her to talk about it later. She felt comfortable in a way she would not have felt otherwise. This was a common thread for all the teenage patients; they felt uncomfortable expressing their fears or problems in a face-to-face interview and having the option of using the iPad made it so much easier. They knew they would eventually have to talk about the problem but having first expressed it digitally helped them overcome a barrier. “In most cases, they would not have told me exactly how they felt without first giving me something in writing. It really increases transparency. It eases the kind of fears that they have and any kind of insecurities or anything that a young person might experience. I’ve been doing this for 28 year and the iPad has been the most valuable tool that I’ve had for reaching the young people and their parents. I was very happy with the tool.”
Karen also appreciates the additional information the application provides. She says, “One of my favorite things about the tools is that they fill out an outcomes questionnaire before every session and so that gives information about any symptoms that they’ve been experiencing over the past week, like depression or anxiety. With the cliexa-OPTIONS questionnaire we got more around sexual risk, and substance use. They also completed a resiliency assessment and that was really helpful. I heard from counselors that having that additional, as well as recommendations of possible things that they can do really helped them figure out with their treatment plans and being able to incorporate any new concerns that came up during that assessment. Since clients knew ahead of time that the questionnaire results would be shared with their counselor, that made it more of an integrated approach where they were able to disclose things on the questionnaire that hadn’t come up in the sessions yet. Maybe they were more comfortable disclosing some things on the questionnaire versus in session.”
Future Use. Christine has mostly praise for the cliexa-OPTIONS digital assessment, noting that “whoever decided that using iPads was the good thing was just really on top of their game.” She hopes to be able to implement the assessment in another clinic soon. The COVID-19 pandemic has caused many things to shut down in terms of hiring, but Christine hopes to be able to use it a nearly non-profit clinic that seems interested. “I just think it is a wonderful program and hopefully at some point we’ll be able to implement it at a clinic, so I would just say keep up the good work and thank you for giving me the opportunity to experience a very valuable tool.”
Karen feels that using the digital app has been an overall positive experience. “In terms of using the actual screening, everything has been good,” she says. She would also use it in future contexts, “because not only does it make the screening process more efficient, but it gives you a holistic view of the client. You’re getting more information about their different problem areas or presenting concerns. I think being able to provide more recommendations around treatment is a really good tool for counselors to have. Getting that information is just going to enhance their quality of care.”
Additional Member of Staff. Christine maintains the clinic would not have been able to incorporate the screening tool without the help of a dedicated assistant, given the extremely limited staffing situation at the rural family planning clinic. The assistant took care of everything related to the digital screening, making the information readily available to the clinician. Karen also notes the importance of having an additional person to manage the digital app, saying, “I don’t think it would be feasible to have the counselor do this for their individual clients. They do need that outside person to kind of be the mediator. It would be difficult to get the information and then also coordinate with all of the different counselors and share that follow-up information.”
Differences Between Sites. Differences between the sites include the challenges of distinct demographic groups, workflow, and online recruitment during the COVID-19 pandemic. It is important to note that most challenges faced by the two sites related more to the logistics of completing a research study and to the unique circumstances of COVID-19 than to difficulties with the digital application itself. A clinical site not conducting a research study would not have to gain informed consent from participants or their parents, in the case of those under the age of 18. Similarly, providing the digital screening in a normal face-to-face clinical setting would be quite different from rapidly moving to telehealth amid a global health crisis.
Difference in Demographics and Focus. The demographic characteristics of the two sites, with one located in a small rural community with a predominantly Latin American population and another at a public university made up of university students from various backgrounds, are quite different and raise distinct challenges at the two sites. The focus of the two sites is also different, with the rural family planning clinic focusing primarily on contraception and sexual health, while the university-based mental health clinic looks at sexual health as part of a spectrum of issues that include resiliency, substance abuse, eating disorders, and anxiety and depression. Christine reported among some members of the Spanish-speaking population at the rural clinic, underage women, perhaps the age of fifteen, might already be married and it is difficult, under those circumstances, to get their parents to sign a parental consent form allowing them to be part of the study. To help mitigate this problem and make the parents more comfortable, the clinic incorporated vaccinations into its treatment program. Christine says, “That really made a world of a difference. We were vaccinating against HPV (human papilloma virus) which should be included in the family planning process. The vaccinations actually increased our Title X program and increased the cliexa process as well.”
Karen pointed out that one problem the mental health clinic had not anticipated is “that as part of the screening clients answer questions about depression and if they’re experiencing any suicidal thoughts. We had one participant who indicated that they were experiencing suicidal thoughts and I didn’t get that information until after they had left. I ended up having to go to that counselor and their supervisor and ask them to contact the client to make sure that everything was OK. That’s when we decided we should do this before the sessions because if something does come up, I can go to the counselor right away and say, ‘Hey, you need to check-in on this during the session.’”
Workflow. While Christine did not have specific problems with the workflow and felt the digital application fit nicely within the routine of the clinic, Karen notes early complications with the workflow at the mental health clinic. She says, “It was a little complicated at first because we would give clients the questionnaire after their sessions and so I would sometimes have five or 10 people at a time who were interested, and we only had two iPads and there was only one of me. The clients would be able to complete the questionnaire but then I couldn’t do all the follow-ups at that time I would have to get the client’s phone number and then I would give them a call to do the follow-ups. That’s where it got complicated because I wasn’t able to contact everyone after that, so I couldn’t complete all the follow-ups. We eventually shifted to giving the questionnaire before their session and that was a lot better. I was able to get that information, do the quick follow-up with them, and they would show up about 20 minutes before their session and then I could share the information with their counselor in case there was anything that needed to be addressed during that session.”
Online Recruitment. The rural family planning clinic shut down when lockdowns related to the pandemic went into effect in March 2020. Conversely, the mental health clinic, after an initial closure lasting a few weeks, began to provide counseling services online and to recruit clients to use the digital assessment online as well. Karen notes that this was a challenging process. “Once we moved online, I had to wait for counselors to indicate that their clients were interested in participating and then I would follow up with them via email, sending them links to download the apps so they could fill out the questionnaires online. It’s a lot harder to do this kind of study online because I’m recruiting through emails and so sometimes people aren’t seeing them, or they just forget about them. I’ve been sending multiple reminder emails and not getting responses and that slowed down our recruitment a little bit. Having enough staff to be able to enroll more participants would make this easier.”
Figure 1
Qualitative Themes
| Rural Family Planning Site | Suburban Mental Healthcare Site |
| Ease of Use | |
| Additional Information Provided | |
| Willingness to Use in Future | |
| Sexual Health Focus
Parental Consent |
Mental Health Focus
Screening |
| Workflow | |
| Online Recruitment | |
Recommendations. Each clinician had recommendations for improving the digital assessment app in ways that would be meaningful to their specific context. Christine, coming from the perspective of someone providing services in a family planning clinic in a rural setting, felt that expanding immunization as part of the process, particularly for HPV, would be helpful. She also noted the language barrier for Spanish-speaking patients. cliexa completed the translation of the assessment items into Spanish.
Karen thought that including a brief risk assessment following that the PHQ9 would be useful. “That way, if clients answer that question positively, we get a little bit more information around what that looks like for them. If there is severe risk, we know what to do right away with that. So that’s one recommendation I would make for changing the screening.” Karen also thought some of the questioning regarding contraception could be framed in such a way to provide more information. “Getting more details around what they are using specifically or maybe what they have used that hasn’t worked or what is working well for them. That could help facilitate some of those follow-up conversations.”
Patient/Client Satisfaction
The patient/client satisfaction survey for the sexual risk assessment (SRAM) consists of eight questions with five-point Likert scale response options. Participants at the rural family planning clinic spent an average of 255.86 seconds (4.26 minutes) completing the various assessments. Participants at the university-based mental health services clinic spent an average of 165 seconds (2.75 minutes). A summary of risk factors for the two sites is provided in Table 2. Questions and mean responses for the patient/client satisfaction survey for each site are summarized in Table 3. Thirty-two participants from the rural family planning site completed patient satisfaction surveys. An obvious limitation in the university-based mental health clinic patient satisfaction data is the low number of responses. It appears that of the 25 participants, only 3 were given the patient satisfaction portion of the survey to complete. The mean responses from the two samples are similar, indicating that most participants enjoyed using the cliexa-OPTIONS sexual risk assessment and found it easy to use.
Table 2
Summary of Risk Factors
| Family Planning Site (N = 35) | Mental Health Site (N = 25) | |
| SRAM
Category |
Primary Low = 4
Primary Moderate = 1 Primary High = 1 Tertiary Moderate = 1 Tertiary High = 22 Tertiary High – Secondary Moderate = 5 Tertiary High – Secondary High = 1 Total = 35 |
Primary High = 1
Tertiary High = 11 Tertiary High – Secondary Moderate = 3 No response = 10 Total = 25
|
| Total Partners | Mean = 8.4 | Mean = 4.9 |
| Undesired Pregnancy | No = 26
Yes = 3 No response = 6
|
No = 14
No response = 11
|
| Last STD Check | Never = 10
Last six months = 6 Last year = 10 Last two years = 2 Last five years = 1 No response = 6
|
Never = 7
Last six months = 7 No response = 11 |
Table 3
Patient/Client Satisfaction Survey Results
| Survey Item | Montrose Mean St. Dev. |
UNC Mean St. Dev. |
||
| 1. It was simple to use the cliexa system
2. It was easy to learn to use the cliexa system 3. The way I interact with the cliexa system is pleasant 4. I like using the cliexa system 5. The cliexa system is simple and easy to understand 6. I feel comfortable using the cliexa system 7. cliexa is an acceptable way to document healthcare services 8. I would use cliexa services again Total |
3.97
4.06 3.94 3.97 3.97 4 4.03 4.22 32.16 |
1.23
1.05 1.02 1.03 1.06 1.02 1.03 0.83 7.67 |
4.00
4.33 4.33 4.33 4.33 4.33 4.33 4.33 34.33 |
1.00
0.58 0.58 0.58 0.58 0.58 0.58 0.58 4.93 |
Response Options
Strongly Disagree = 1; Disagree = 2; Neutral =3; Agree = 4; Strongly Agree = 5
Discussion
While the two sites serve distinct populations and have different areas of focus, they both emphasize helping adolescents and young adults to make healthy, informed, and positive decisions about their lives, including their sexual behavior. The cliexa-OPTIONS digital sexual risk assessment app appears to have been successful in each of these sites in terms of helping them to open up critical conversations with patients/clients and to provide avenues for improving the quality of care they receive. Furthermore, patients appear to enjoy using the digital app and find it easy to use.
Author Note: We have no known conflict of interest to disclose. Correspondence concerning this article should be addressed to
Becky De Oliveira, Department of Applied Statistics and Research Methods, Campus Box 28, 501 20th St, Greeley, CO 80639
Becky A. De Oliveira1, Kaitlin Dent2, Jennifer A. Smith2, William Merchant1, Stephen Wright2
1Department of Applied Statistics and Research Methods, University of Northern Colorado
2Department of Applied Psychology and Counselor Education, University of Northern Colorado
References
Bleakley, A., Hennessy, M., Fishbein, M., Coles, H. C. Jr., & Jordan, A. (2009). How sources of sexual information relate to adolescents’ beliefs about sex. Am J Health Behavior, 33(1), 37.
Brayboy, L. M., Sepolen, A., Mezoian, T., Schultz, L., Landgren-Mills, B. S., Spencer, N., Wheeler, C., & Clark, M. A. (2017). Girl talk: A smartphone application to teach sexual health education to adolescent girls. J Pediatr Adolesc Gynecol, 30, 23-28.
Brown, L. K., Danovsky, M. B., Lourie, K. J., DiClemente, R. J., & Ponton, L. E. (1997). Adolescents with psychiatric disorders and the risk of HIV. Journal of the American Academy of Child & Adolescent Psychiatry, 36, 1609-1617.
Burgess, J. D., Cameron, C. M., Watt, K., & Kimble, R. M. (2016). Cool runnings – an app-based intervention for reducing hot drink scalds: Study protocol for a randomized controlled trial. Trials, 17(1), 388.
Cederbaum, J. A., Rodriguez, A. J., Sullivan, K., & Gray, K. (2017). Attitudes, norms, and the effect of social connectedness on adolescent sexual risk intention. Journal of School Health, 87(8), 575-583.
Centers for Disease Control and Prevention. (2014). Sexually transmitted disease surveillance 2013. Atlanta, GA: National Center for HIV/AIDS, Viral Hepatitis, STD, and TB Prevention, Division of Sexually Transmitted Disease Prevention.
Centers for Disease Control and Prevention. (2013). Incidence, prevalence, and cost of sexually transmitted infections in the United States. Morbidity and Mortality Weekly Report (MMWR).
Chao, D. Y., Lin, T. M., & Ma, W. Y. (2019). Enhanced self-efficacy and behavioral changes among patients with diabetes: Cloud-based mobile health platform and mobile app service. JMIR Diabetes, 4(2).
Cohen, D. J., Crabtree, B. F., Etz, R. S., Balasubramanian, B. A., Donahue, K. E., Leviton, L. C., Clark , E. C., Isaacson, N. F., Stange, K. C., & Green, L. W. (2008). Fidelity versus flexibility: Translating evidence-based research into practice. American Journal of Preventive Medicine, 35(5), S381-389.
Cunningham, K., Martinez, D. A., Scott-Sheldon, L. A. J., Carey, K. B., Carey, M. P., & the MASH Research Team. (2017). Alcohol use and sexual risk behaviors among adolescents with psychiatric disorders: A systematic review and meta-analysis. J Child Adolesc Subst Abuse, 26(5), 353-366.
Dancy, B. L., Crittenden, K. S., & Talashek, M. L. (2006). Mother’s effectiveness as HIV risk reduction educators for adolescent daughters. J Health Care Poor Underserved., 17(1), 218-239.
DiClemente, R. J., & Ponton, L. E. (1993). HIV-related risk behaviors among psychiatrically hospitalized adolescents and school-based adolescents. American Journal of Psychiatry, 150(2), 324-325.
Donenberg, G., Emerson, E., Mackesy-Amiti, M. E., & Fletcher, F. (2018). Sexual risk among African American girls seeking psychiatric care: A social-personal framework. Journal of Consulting and Clinical Psychology, 86(1), 24-38.
Donenberg, G. R., & Pao, M. (2005). Youth and HIV/AIDS: Psychiatry’s role in a changing epidemic. Journal of the American Academy of Child & Adolescent Psychiatry, 44, 728-747.
Downs, J. S., Bruine de Bruin, W., Fischhoff, B., & Murrary, P. J. (2015). Behavioral decision research intervention reduces risky sexual behavior accepted for publication in current HIV research. Current HIV Research, 13(5), 439-446.
Erickson, F. (1986). Qualitative methods in research on teaching. In M. C. Wittrock (Ed.), Handbook of research on teaching (pp. 119-161). Old Tappan, NJ: Macmillan.
Fishbein, M., & Azjen, I. (1975). Belief, Attitude, Intention, and Behavior: An Introduction to Theory and Research. Addison-Wesley.
Flyvbjerg, B. (2011). Case Study. In Norman K. Denzin and Yvonna S. Lincoln, (Eds.), The Sage handbook of qualitative research (pp. 301-316). Sage.
Gannon, B., Davis, R., Kuhns, L. M., Rodriguez, R. G., Garofalo, R., & Schnall, R. (2020). A mobile sexual health app on empowerment, education, and prevention for young adult men (MyPEEPS Mobile): Acceptability and usability evaluation. JMIR Formative Research, 4(4), 1-11.
Gould, C. E., Kok, B. C., Ma, V. K., Zapata, A. M. L., Owen, J. E., & Kuhn, E. (2019). Veteran Affairs and the Department of Defense mental health apps: A systematic literature review. Psychological Services, 16(2), 196-207.
Gray, N. J., Klein, J. D., Noyce, P. R., Sesselberg, T. S., Cantrill, J. A. (2005). Health information-seeking behaviour in adolescence: The place of the internet. Soc Sci Med., 60 (7), 1467-1478.
Ha, S. W., & Kim, J. (2020). Designing a scalable, accessible, and effective mobile app based solution for common mental health problems. International Journal of Human-Computer Interaction, 36(14), 1354-1367.
Hallum-Montes, R., Rhodes, L., Malone, C., Martin, G., Senter, L., & Bunyasaranand, P. (2016). ‘Our only resource’: Perspectives and recommendations of rural African American youth on adapting sexual health and risk reduction interventions. Journal of Health Care for the Poor and Underserved, 27, 622-635.
Hutchinson, M. K., Jemmott, J. B., Jemmott, L. S., Braverman, P., & Fong, G. T. (2003). The role of mother-daughter sexual risk communication in reducing sexual risk behaviors among urban adolescent females: A prospective study. Journal of Adolescent Health, 33, 98-107.
Johnson, B. T., Scott-Sheldon, L. A., Huedo-Medina, T. B., Carey, M. P. (2011 Interventions to reduce sexual risk for human immunodeficiency virus in adolescents: A meta-analysis of trials, 1985-2008. Arch Pediat Adol Med, 165(1), 77.
Kann, L., McManus, T., Harris, W. A., Shanklin, S. L., Flint, K. H., Hawkins, Queen, B., Lowry, R., O’Malley Olsen, E., Chyen, D., Whittle, L., Thornton, J., Lim, C., Yamakawa, Y.,
Brener, N., & Zaza, S. (2016). Youth risk behavior surveillance – United States, 2015. Morbidity and Mortality Weekly Report, 65, 1-174.
Kenny, R., Fitzgerald, A., Segurado, R., & Dooley, B. (2020). Is there an app for that? A cluster randomised controlled trial of a mobile app-based mental health intervention. Health Informatics Journal, 26(3), 1538-1559.
Kirby, D. (2007). Abstinence, sex, and STD/HIV education programs for teens: Their impact on sexual behavior, pregnancy, and sexually transmitted disease. Annu Rev Sex Res, 18(1): 143-177.
Kirby, D. (2006). Comprehensive sex education: Strong public support and persuasive evidence of impact, but little funding. Arch Pediat Adol Med, 160(11), 1182.
Lattie, E., Cohen, K. A., Winquist, N., & Mohr, D. C. (2020). Examining an app-based mental health self-care program, intellicare for college students: Single-arm pilot study. JMIR Mental Health, 7(10).
Lee, M., Lee, H., Kim, Y., Kim, J., Cho, M., Jang, J., & Jang, H. (2018). Mobile app-based health promotion programs: A systematic review of the literature. International Journal of Environmental Research and Public Health, 15, 1-13.
Logie, C., Okumu, M., Abela, H., Wilson, D., & Narasimhan, M. 2020). Sexual and reproductive health mobile apps: Results from a cross-sectional values and preferences survey to inform World Health Organization normative guidance on self-care interventions. Global Health Action, 13(1), 1-7.
McKay, F. H., Wright, A., Shill, J., Stephens, H., & Uccellini, M. (2019). Using health and well-being apps for behavior change: A systematic search and rating of apps. JMIR mHealth and uHealth, 7(7), e11926.
Merriam, S. B. (2009). Qualitative research: A guide to design and implementation. Jossey-Bass.
Michie, S., & Abraham, C. (2004). Interventions to change health behaviours: Evidence-based or evidence-inspired? Psychology and Health, 1991), 29-49.
Milne-Ives, M., Lam, C., De Cock, C., Van Velthoven, M. H., & Meinert, E. (2020). Mobile apps for health behavior change in physical activity, diet, drug and alcohol use, and mental health: Systematic review. JMIR mHealth and uHealth, 8(3), e17046.
Neumann, M. S., O’Donnell, L., Doval, A. S., Schillinger, J., Blank, S., Ortiz-Rios, E., Garcia, T., O’Donnell, C. R. (2011). Effectiveness of the VOICES/VOCES sexually transmitted disease/human immunodeficiency virus prevention intervention when administered by health department staff: Does it work in the “real world”? Sexually Transmitted Diseases, 38(2), 133.
Nurmi, J., Knittle, K., Ginchev, T., Khattak, F., Helf, C., Zwickl, P., Castellano-Tejedor, C., Lusilla-Palacios, P., Costa-Requena, J., Ravaja, N., & Haukkala, A. (2020). Engaging users in the behavior change process with digitalized motivational interviewing and gamification: Development and feasibility testing of the precious app. JMIR mHealth and uHealth, 8(1), e 12884.
O’Connor, E., Lin, J.S., Burda, B. U., Henderson, J. T., Walsh, E. S., & Whitlock, E. P. (2014). Behavioral sexual risk reduction counseling in primary care to prevent sexually transmitted infections: A systematic review for the U.S. Preventive Services Task Force. Ann Intern Med, 161(12), 874-883.
O’Donnell, C. R., O’Donnell, L., San Doval, A., Duran, R., & Labes, K. (1998). Reductions in STD infections subsequent to an STD clinic visit using video-based patient education to supplement provider interactions. Sexually Transmitted Diseases, 25, 161-168.
Ott, M. A., Rouse, M., Resseguie, J., Smith, H., & Woodcox, S. (2011). Community-level successes and challenges to implementing adolescent sex education programs. Maternal and Child Health Journal, 15, 169-177.
Quante, M., Kandpur, N., Kontos, E. Z., Bakker, J. P., Owens, J. A., & Redline, S. (2019). A qualitative assessment of the acceptability of smartphone applications for improving sleep behaviors in low-income and minority adolescents. Behavioral Sleep Medicine, 17(5), 573-585.
Ramsey, R. R., Holbein, C. E., Powers, S. W., Hershey, A. D., Kabbouche, M. A., O’Brien, H. L., Kacperski, J., Shepard, J., & Hommel, K. A. (2018). A pilot investigation of a mobile phone application and progressive reminder system to improve adherence to daily prevention treatment in adolescents and young adults with migraine. Cephalagia, 38(14), 2035-2044.
Reger, G. M., Hoffan, J., Riggs, D., Rothbaum, B. O., Ruzek, J., Holloway, K. M., & Kuhn, E. (2013). The “PE coach” smartphone application: An innovative approach to improving implementation, fidelity, and homework adherence during prolonged exposure. Psychological Services, 10, 342.349.
Reyna, V. G. & Farley, F. (2006). Risk and rationality in adolescent decision making implications for theory, practice, and public policy. Psychol Sci Public Interest, 7(1), 1-44.
Shannon, C. L., & Klausner, J. D. (2018). The growing epidemic of sexually transmitted infections in adolescents: A neglected population. Curr Opin Pediatr, 30(1), 137-143.
Simons, H. (2009). Case study research in practice. Sage.
Shorey, S., Yang, Y. Y., & Dennis, C. L. (2018). A mobile health app-based postnatal educational program (home-but not alone): Descriptive qualitative study. Journal of Medical Internet Research, 20(4), 1.
Stake, R. E. (1995). The art of case study research. Sage.
Stanton, B., Harris, C., Cottrell, L., Ziaoming, L., Gibson, C., Guo, J., Pack, R., Galbraith, J., Pendleton, S., Wu, Y., Burns, J., Cole, M., & Marshall, S. (2006). Trial of an urban adolescent sexual risk-reduction intervention for rural youth: A promising but imperfect fit. Journal of Adolescent Health, 38(1), 55.e25-55.e36.
Stanton, B., Guo, J., Cottrell, L., Galbraith, J., Li, X., Gibson, C., Pack, R., Cole, M., Marshall, S., & Harris, C. (2005). The complex business of adapting effective interventions to new populations: An urban to rural transfer. Journal of Adolescent Health, 37(2), 163.e17- 163.e26.
Steinberg, A., Griffin-Tomas, M., Abu-Odeh, D., & Whitten, A. (2018). Evaluation of a mobile phone app for providing adolescents with sexual and reproductive health information, New York City, 2013-2016. Public Health Reports, 133(3), 234-239.
Widman, L., Choukas-Bradley, S., Noar, S. M., Nesi, J., & Garret, K. (2015). Parent-adolescent sexual communication and adolescent safer sexual behavior: A meta-analysis. JAMA Pediatrics, 170, 52-61.
Workowski, K. A., & Bolan, G. A. (2015). Sexually transmitted diseases treatment guidelines. MMWR, 64, 1-137.
World Health Organization. (2012). Mobile health (mHealth). http://www.whoint/goe/mobile_health/en/
Yin, R. K. (2014). Case study research: Design and methods (5th ed,). Sage.