Development and Evaluation of an mHealth App That Promotes Access to 3D Printable Assistive Devices

Abstract

Three-dimensional printing is an emerging service delivery method for on-demand access to customized assistive technology devices. However, barriers exist in locating and designing appropriate models and having the devices printed. The purpose of this work is to outline the development of an app, 3DAdapt, which allows users to overcome these issues by searching within a curated list of 3D printable assistive devices, customizing models that support it, and ordering the device to be printed by manufacturers linked within the app or shared with local 3D printing operators. The app integrates searching and filters based on the International Classification of Functioning, Disability, and Health, with the available devices including those developed from fieldwork collaborations with multiple professionals and students within clinical, community, and educational settings. It provides users the ability to customize select models to meet their needs. The model can then be shared, downloaded, or ordered from a third-party 3D printing service. This development and expert testing phase to assess feasibility and modify the app based on identified themes then prepared the team for the next phases of beta testing to reach the overall aim of 3DAdapt to connect individuals to affordable and customizable devices to increase independence and quality of life.

1. Introduction

Mobile health applications (mHealth apps) have become integral tools in healthcare, responding to the demand of users to have better access than ever before to personal medical records and condition-specific information, to more general wellness aims of on-demand tracking such as for vital sign monitoring and tracking fitness goals [1,2,3]. Although apps are useful tools and ubiquitous with everyday routines of technology integration, recommendations to improve the patient user experience of mHealth apps include designing apps to provide personalization and customizations to improve patient-centered services and health strategies [3,4]. More recent advocacy efforts include the web-based and app-based accessibility features of digital health information, including the user experience of persons with disabilities (PWDs), and to include the customizations required and the necessity of equal access required laborative design involving PWDs, developers, and healthcare professionals is essential for addressing these personalization and customization limitations and enhancing the overall utility of mHealth apps for PWDs [5].

Access to assistive technology (AT) is a significant concern for people with disabilities (PWDs), with an estimated one billion individuals globally needing AT devices; however, access is limited due to barriers including financial constraints and a lack of trained providers [6,7]. As the demand for AT rises and healthcare delivery evolves with peer support and online health communities [8], 3D printing is emerging as a viable solution, which involves the design of models with computer software, enhancing customizability and client-centered practice involving PWDs in the design [9,10]. For conditions such as rheumatoid arthritis, cerebral palsy, and amyotrophic lateral sclerosis, 3D-printed AT can decrease pain, improve functionality, and increase patient satisfaction and independence [11,12,13]. However, barriers to 3D printing include the need for more experience and training in operating CAD software and 3D printers [10,14]. One major challenge in this field is finding publicly available previously designed models to be printed. While there are numerous sites that host thousands of models, such as Thingiverse and MyMiniFactory, most of the available services are poorly curated with searches for AT devices that are “assistive” or “accessible” resulting in a majority of spurious, unrelated models (https://hingiverse.com, accessed on 30 April 2024; https://myminifactory.com, accessed on 30 April 2024).

The 3D printable models can be designed using numerous software packages, with each package having a different focus on style, features, and functionality. One feature that is available in some software packages is the idea of parametric, or parameterized, models. These models are defined by a set of parameters (i.e., named values) and then can generate numerous different 3D models based on the parameters. For example, a finger splint parametric model could have parameters for the proximal, middle, and distal segment lengths, the angle of each joint, and the diameter of each segment so it could be made to fit perfectly on any finger. The model can be customized by simply specifying a set of values to fit a particular need instead of needing an expert to redesign the model.

The downside to parametric models is that proper design must be completed by the expert from the start to be parametric. Then, software must also be available for the end user to specify the necessary values and perform the customization. Only one CAD software program is widely supported for this on both ends: OpenSCAD. However, OpenSCAD has low satisfaction (i.e., challenging to use) and market presence [15]. However, it is the only CAD software at all with easy-to-use end-user customization tools available, meaning Thingiverse, MakeWithTech, and CadHub all support generating customized models from OpenSCAD [16,17,18]. On the other hand, OnShape and Fusion 360 are CAD programs that support parametric models and have high satisfaction and medium–high market presence. Still, there are no easy-to-use end-user tools available for submitting values to obtain customized, 3D printable models out. Having easy-to-use tools that can access the inherent customizable nature of these models would increase access and suitability for non-experts.

The concept of expert volunteer crowdsourcing has shown much success in recent years, for example, on the extremely popular Stack Exchange question-and-answer network of websites where users obtain expert answers to their questions, and volunteer curators make sure to maintain a high level of quality for the questions, answers, and their organization with tags [19]. These forms of crowdsourcing are suggested as a powerful way to harness expert skills in information systems to optimize innovation [20]. Additionally, crowdsourcing can be used to harness the “wisdom of crowds” when regular users are allowed to participate and receive both user and social feedback [21]. When designing our proposed app, 3DAdapt, we aimed to harness this power of collecting feedback to encourage reviews, photos, and comments from a community of users. Among other design elements to address the barriers of 3D printing and facilitate user-centeredness of mHealth apps for PWDs through easy search-and-find filters, this paper will outline the design process and the initial feasibility testing processes with expert clinicians in 3D printing applications.

2. Materials and Methods

This section covers establishing the list of promoted accessible devices, development of the mobile app and server, and the content analysis of the application by field experts.

2.1. Pre-App Development: Establishing the Devices through Experiential Learning

Some investigations into increasing the acceptance of 3D printing have focused on and recommended enhancing training and education to address the barriers to 3D printing adoption [22,23,24]. Experiential learning of healthcare technologies, or the learning of technology applications through hands-on, problem-solving approaches to address a real-life problem, has increased positive perceptions, the intention to use, and improved performance in applying the learned technology [22,25]. Additionally, technology acceptance of 3D printing may improve when integrating a service-learning model within experiential learning when partnering with technology “expert” professionals to design customizations [23]. At a private Eastern Atlantic university, students complete a project through experiential service learning of problem solving a real-life clinical dilemma with customized 3D printable designs [23]. An ongoing list of devices was established for three consecutive years before 3DAdapt was developed. In addition, students in an occupational therapy course were tasked to search for and locate open-source designs to solve case study problems and provide free educational devices for low vision and blind support groups for children and adults as community service projects. At this institution, a 3D printing club that started in 2017 continuously added designs to this ongoing list of devices. Through 2020, as a pandemic response, the 3D printing club printed disposable devices for medical personnel support. Lastly, through a formal partnership collaboration with a local rehabilitation healthcare network and establishing a 3D printing program within their network, more devices were added to this ongoing list, along with collaborative research that focused on access to mobile technology devices within the network [26]. Through these collaborative efforts within clinical, community, and educational settings, this initial list of both original and publicly available AT designs included over 200 designed devices.

However, most of these devices were not categorized, named, or described so that PWDs could find and locate AT devices themselves. Within the app, we aimed to include additional information about each design that almost no other service provides. Thus, an occupational therapist (OT) on the 3DAdapt team (second author) added fundamental, user-centered, functional information on each imported design. The additional information included the following:

• Tags: Categorization of each design based on the World Health Organization’s International Classification of Functioning, Disability, and Health (ICF) categories [27] to ensure the reach of 3DAdapt would be accepted by international stakeholders and to communicate with all healthcare professionals and collaborative stakeholders within a common language. Each device was tagged into the three major ICF categories: body function, activities and participation, and environmental factors.

• Purpose: A short summary (1–2 sentences) of the AT design including its purpose and any special caveats (such as additional supplies or tools needed to assemble).

• Simple Name: A design name using the common practice language of a rehabilitation professional and what a PWD may understand to be helpful for them. For example, the original title of “Fork and spoon support for person with disabilities” (https://www.myminifactory.com/object/3d-print-fork-and-spoon-support-for-person-with-disabilities-5480, accessed on 30 April 2024) was renamed to “Universal cuff for utensils”. The original device name was still provided for search purposes.

• Printsets: To overcome the need to determine which of the necessary STL file(s), and potentially multiple STL files, to download and use, all necessary STL files to make up the entirety of a single AT device were uploaded as a new term titled “printset”, which is the collection of all necessary STL files, pre-scaled, in a “one-click” download.

2.2. Design and Development of the Mobile App

Development of a custom mobile application and server was required. For the testing phases, the application was only developed for Android phones with future plans for Apple devices. The server was designed to be deployed using standard cloud services so that it could be deployed easily and cost effectively.

2.2.1. Design of Mobile App

The mobile app was developed using Kotlin language using the Jetpack Compose v1.4.3 (https://developer.android.com/jetpack/compose, accessed on 30 April 2024) user interface (UI) toolkit, which is the recommended tool to use for Android App Development by Google. Additionally, Kotlin supports multi-platform development, meaning that parts of it can be reused within Android and iOS apps, so all of the non-UI code could be reused between these two platforms (https://kotlinlang.org/docs/multiplatform.html, accessed on 30 April 2024). Finally, a multi-platform version of Jetpack Compose is now available, although still in alpha testing [28]. It is hopeful that much of the UI will be directly portable from Android to iOS in the future. Still, during development, the current alpha status of the library needed to support more of the UI to be useful. However, large parts of the application are ready to transition to iOS as the Jetpack Compose library becomes more available and stable on iOS.

The mobile app was designed with multiple layers of abstraction following recommended design principles. As shown in Figure 1, the UI layer is kept separate from the application’s internals, allowing for the internals to be more readily used between platforms and updated independently from how things are displayed.

Besides the mobile app, there is also a web client with minimal functionality, mainly to support the sharing of links to people without the mobile app on their device. This is important so that the PWD can readily share links to their healthcare provider, care partner, or 3D printing service, who may still need to download the app. This communicates with the server using the same API as the mobile app. The web client is also where the advanced features of uploading new devices and moderation occur.

2.2.2. Design of the Server

The server portion of the application manages all of the persistent data. It provides an application programming interface (API) to be used by different clients such as a mobile app or website. The server is written in Python 3 and exposes a representational state transfer (REST) API utilizing JavaScript Object Notation (JSON) to transmit requests and responses. The development of a completely RESTful API is critical in that the server is made to be stateless, which allows for

• Availability: If the server experiences unforeseen problems, requests can be rerouted to alternative instances without noticeable interruption to the end user.

• Reliability: An instance of the server will give consistent responses regardless of how internet traffic is routed or loads are balanced across multiple servers.

• Scalability: The resources utilized by the server can be automatically scaled based on the demand for the service, reducing the cost to keep the service running by only having the resources needed at any given moment in use.

The API exposed by the server is available at https://app.swaggerhub.com/apis-docs/Coder-for-Life/3DAdapt/1.0.0 (accessed on 30 April 2024).

The popular web framework Flask v2.3.2 (https://flask.palletsprojects.com, accessed on 30 April 2024) was used because it has a large community support, many add-on libraries to support things such as robust security and emailing, and is supported by common cloud providers such as Amazon Web Services (AWS), Google Cloud, and Microsoft Azure. It runs popular, large websites such as Pinterest and LinkedIn.

The server utilizes MongoDB (https://www.mongodb.com, accessed on 30 April 2024) to store the data. MongoDB is a no-SQL database that provides better horizontal scalability than traditional relational (SQL) databases and is thus more suitable for cloud services with unknown or variable demands, reducing overall cost by allowing for an automatic reduction in resource usage (and thus cost) when there is a low demand for the service. MongoDB also has integrations with all common cloud providers.

Since databases are not optimal for storing large amounts of binary data such as images and 3D models, those files are stored separately from the database either within the local filesystem or on a cloud service such as Amazon Simple Storage Service (S3, https://aws.amazon.com/s3/, accessed on 30 April 2024). The S3 service allows storing gigabytes of data for pennies a month, is highly available, reliable, scalable, and requires exceptionally minimal set up. Within the database itself, the files are stored as URLs to the actual location of the files.

The database has the following collections of documents:

• Basic Information Collections:

• users—private information about users (emails, permissions, subscriptions, etc.)

• profiles—public information about users (bio, URLs to photos, etc.)

• designs, designs.files, and designs.printsets—information about the designs

• makes—the information about the design reviews (internally referred to as makes)

• collections and collections.designs—the information about the collections

• designs.comments and makes.comments—the comments posted on designs and reviews

• tags and categories—the available tags and categories used by the designs

• licenses—the available licenses for designs and their details

• Relational Collections that associate between other collections:

• likes—which designs, reviews, and collections are liked by which users

• watches—which designs, reviews, and collections are watched by which users

• followers—which users and following which other users

• Collections that store temporary or cached information:

• designs.files.customizations and designs.printsets.customizations—information about recently customized (or currently generating) models

• importers.cache—temporary information for importing designs from other sites

• email_verifications, password_reset, and token_blocklist—collections used to assist in verifying users, resetting passwords, and logging users out securely

• email_queue and updates—collections used to send users notifications of updates that they have requested

To maintain responsiveness, any long-running or resource-intensive tasks, such as generating image thumbnails, processing 3D model files, creating 3D renders of models, customizing models, and sending emails, are performed in the background utilizing Celery (https://docs.celeryq.dev/en/stable/, accessed on 30 April 2024) for managing background tasks using Redis (https://redis.io, accessed on 30 April 2024) as the broker and backend for the task queue. Celery allows multiple background tasks to run simultaneously and even distribute the work across multiple machines, which means that the server resources can be scaled appropriately for the current demand on the system at any given time.

2.2.3. Deployment of the Server

Figure 2 summarizes how these technologies are connected in the production server environment as deployed using the AWS cloud platform. They will be detailed in this section.

Flask is a Web Server Gateway Interface (WSGI) application. By design, it can only handle one request at a time and by itself cannot scale. However, using a WSGI server, such as gunicorn (https://gunicorn.org, accessed on 30 April 2024), scalability can be achieved by automatically increasing or decreasing the number of Flask application instances as required. This is achievable since each application instance of the web server is stateless. While both WSGI applications and servers can act as an HTTP server for development purposes, using an additional dedicated HTTP server, such as Nginx (https://www.f5.com/go/product/welcome-to-nginx, accessed on 30 April 2024), can provide additional robustness and features and is strongly recommended for production servers. To increase scalability, Nginx can also have multiple WSGI servers spread across multiple host machines.

There are numerous options to easily deploy servers in the cloud without owning hardware, including AWS, Google Cloud Platform, and Microsoft Azure. For this project, we chose AWS. To deploy unmanaged servers (these are unmanaged since the installation and configuration of the necessary software is on the developer instead of AWS installing and managing the software), one uses the AWS Elastic Cloud Compute (EC2) service. This allows running a virtual machine in the cloud and the installation of any server software on it. The developer selects the machine type, the machine RAM and CPU resources, and the amount of hard drive space. After that, the developer is charged an hourly rate to keep the machine running and a monthly rate for the amount of hard drive space. It is “elastic” since resource changes may occur as the demand fluctuates, either by adding additional EC2 instances or changing an individual machine’s resources.

For the web server, which runs Nginx, gunicorn, and the Flask server, we chose the t4g.small machine with 2 CPUs and 2 GB of RAM, which costs 0.0168 USD/h (about 12.30 USD/month, quoted for the eastern US region), although, with a savings plan, the charge can be as low as 0.0076 USD/h (about 5.60 USD/month) at the time of this writing. This set up can handle at least hundreds of requests per second. The minimum storage allowed is 8 GB, which is sufficient for this machine and costs 0.64 USD/month.

An additional EC2 instance, including the Celery task manager, is used for the background tasks. Separating this from the web server itself allows independent scaling and utilization of different resources. For this, we also chose a base of a t4g.small machine with 8 GB of storage as well.

To store static files, including uploaded photos, 3D models, and other uploaded files, the AWS S3 service is used. This allows the EC2 web server instance not to have to deal with these files, thus reducing its workload and costs, as S3 is a much cheaper service for serving static files. The first 200 designs contain about 2500 files, taking up 800 MB and 5000 photos taking up 900 M. The S3 service will cost about 0.05 USD/month for storage fees, USD 0.04 for the initial uploading of the data, and an estimated 0.92 USD/month for client retrieval costs (assuming around 50 k requests with a total of 10 GB data). It may be possible to use AWS CloudFront Content Delivery Network to reduce costs, but this has yet to be explored.

AWS also offers a managed service named ElastiCache as a replacement for Redis. This service costs about the same as EC2 instances but requires much fewer resources. An alternative solution is to use AWS Simple Queue Service (SQS), which Celery also supports. In this case, the charges are not continuous but instead per message (i.e., per background task initiated). This has other additional benefits, such as cost and scalability, by allowing the background task servers to only run when actually necessary instead of always on. The transient needs also allow the use of what are called EC2 spot instances, which can be up to 90% cheaper than regular EC2 instances. This will be an avenue of exploration in the future and was not examined during the initial deployment.

Another critical piece is the MongoDB database, which stores all the information that is not images and files. AWS does provide a service named DocumentDB that serves the same purpose as MongoDB and is advertised as being “MongoDB compatible” but is in fact only compatible with an older version of the database. Instead, MongoDB Inc. provides a service named Atlas that utilizes AWS. They provide a forever-free tier that is sufficient for running this application.

The final service needed is a small but essential service. The AWS Simple Email Service (SES) allows automated outgoing emails to be sent. It charges for every 1000 emails sent and the number of GB in those emails. We have a rough estimate of 3000/month with each email being 8 KB on average, totaling only 0.51 USD/month.

Ultimately, the sum of all of the necessary services through AWS will cost around 15 USD/month for the initial deployment but may grow as the application becomes more popular. The cost of the EC2 instances dominates the cost; thus, the use of a savings plan is important to keep costs down.

2.3. Beta Testing and App Evaluation

Based on Jake-Schoffman et al.’s [29] recommendations for evaluating the content, usability, and efficacy of commercial mobile health apps, the 3DAdapt app was tested in three phases, involving intended end users such as PWDs and rehabilitation professionals through a user-centered design approach [30] (see Table 1). This is conducted to establish how well an app serves its intended purpose and if there is an actual need for the app. This is accomplished with observational studies and efficacy testing. The goal is to increase the adoption of the app and increase its impact.

Expert Content Analysis

When conceptualizing the testing phases of 3DAdapt and projecting the international, public reach of the app, the team initially conducted an expert review of the feasibility of the app.

• Participants and App orientation

The expert review of the app content was conducted by three (3) specialists in the field who served as “expert reviewers”. Experts were defined as having produced 3D-printed devices, as part of research, programming, or clinical practice, and have documented outcomes or data on the user experience or usability of devices, for 2+ years. This person does not rely on the principles of 3D printing but rather connects understanding to the appropriate action through an interpretive approach [31].

Three experts were identified through purposeful sampling method, and these criteria were self-confirmed to serve as expert reviewers. The goal was to gather in-depth insights on how to enhance the app’s user-friendliness from this group of “information-rich cases” [32]. See Table 2 for more details about the interview participants.

Phones with the beta version of the 3DAdapt app were shipped to the three expert reviewers. A “how-to” guide with the code to unlock the phone was emailed to the reviewer 48 h before a scheduled Zoom interview. The tasks were aimed at orienting to the toolbar menu, ICF, and open text search functions, posting “makes” with photos and reviews, searching for devices (for low vision, meal preparation, playing music, ADHD, dressing, and opening bottles; reviewers were encouraged to search on their own as well), and potential customizations with printsets and how to share and print.

• Data Collection

Following familiarization with the app, the second and third authors conducted online interviews with the experts. The interviews lasted about 60 min and were recorded using the record option of the Zoom^® virtual platform (https://zoom.us/, v5.14.6, accessed on 30 April 2024) hosted by Moravian University. The second and third authors created a semi-structured interview guide, using the feasibility framework suggested by Bowen et al. [33] and American Psychiatric Association’s app evaluation model [34] (see Appendix A). The guide focused on the experts’ experiences and perceptions regarding the feasibility of the 3DAdapt app. Several key elements of feasibility were incorporated [33], including acceptability, demand, implementation, practicality, adaptation, and integration.

The third author conducted semi-structured interviews with participants for approximately 45 min. The semi-structured interviews aimed to gather experts’ opinions and feedback on the app’s usability, utility, acceptability, value, and ideas for improvement. The third author did not participate in the app’s design, which helped to limit bias resulting from app developers’ presence influencing subjects’ responses during interviews. The third author has expertise in qualitative research and is a rehabilitation provider specializing in assistive technology solutions for PWDs and children with disabilities. During the semi-structured interview, the second author turned off her camera and microphone. After the semi-structured interview, the second author turned on her camera and microphone and asked the interviewees structured questions. These questions were based on the interviewee’s responses to the open-ended questions and their written responses to the app familiarization tasks.

• Data Analysis

Data were transcribed verbatim by the third author and reviewed and confirmed by the second author. Qualitative content analysis [35,36], a research method for making replicable and valid inferences from data to their context to provide knowledge, new insights, a representation of facts, and a practical guide to action (integrating deductive and inductive strategies), was employed to analyze the interviews [37]. This approach was selected to address the primary aims of feasibility examinations and prioritizing feedback themes, whereby the constructs and subconstructs under study were integrated into the process of deductive thematic analysis while allowing for new ideas or themes to emerge from the qualitative data using inductive coding.

Data analysis was carried out using the following steps [38]. Familiarization—To begin the data analysis process, the team listened to the audio recording and reviewed the transcripts several times to become more acquainted with the data. To reduce bias and enhance reliability, the second and third authors coded at least 10% of the transcripts inductively. Identifying a thematic framework—A thematic framework was developed to guide data analysis based on Bowen et al.’s feasibility framework and the American Psychiatric Association’s app evaluation model. Any discrepancies in the codebook were resolved through discussion. Indexing and charting—The thematic framework or index was systematically applied to the entire dataset by the second and third authors. A priori categories were anticipated from the thematic framework, and new topics arising from data were merged with previously developed categories. The two authors held regular meetings to compare coding work, and intercoder agreement was subjectively assessed. MAXQDA, a qualitative data analysis software, was used for the coding and categorization process. Interpretation—The second and third authors collaboratively reviewed the codes to identify the major themes (overarching themes) and sub-themes (specific themes). Upon completion of the analysis of three interviews, a point of data saturation or redundancy was reached [39].

3. Results of Expert App Review

In this section, we present the results of the usability evaluation of the app through expert review and subsequent app modifications.

3.1. Results of Qualitative Content Analysis

While interacting with 3DAdapt, the three expert reviewers completed the self-paced “how-to” guide and then participated in the semi-structured interview. The qualitative content analysis of the semi-structured interviews with the three experts revealed three prominent themes (refer to Table 3): the utility and relevance of 3DAdapt; actionable steps for modification; and improvement in the future. Detailed results are presented below.

3.1.1. Theme 1: Utility and Relevance of 3DAdapt

The experts found various features of the 3DAdapt mobile application innovative and valuable to potential users. They attested to its feasibility through descriptions of acceptability, demand, implementation, practicality, adaptation, and integration, which were captured in this theme.

• Acceptability

Interviewees generally found 3DAdapt’s design appealing and functional. Expert A1 stated, “I thought that it was good. It looks really clean. To me. It looks really legible”. They appreciated the well-curated content of the 3DAdapt app, noting the good photos and concise descriptions. Expert A1 commented, “I really liked the photos and the descriptions like that, to me was, is really helpful to be able to just like, scroll through and see all the photos. I like being able to just like search bottle, or something like that”.

They saw appropriate app audiences as families and PWDs, novice-to-expert clinicians as those with 3DP knowledge, including rehabilitation students who are novices. Expert B3 commented, “...this would be great for a novice, someone who doesn’t know, or newer to 3d printing. So I should say, a novice practitioner”. They even thought the app included a wide variety of devices, which would be helpful even for experts. Expert B1 described, “For me, someone who thinks he knows this stuff. You all have found and curated more stuff than then I’ve been able to find on my own. So I would use it”.

• Demand

Experts acknowledged 3DAdapt’s potential to help end users like rehabilitation providers and PWDs discover necessary devices they might not know existed. A1 emphasized, “I’d like someone to be able to literally scroll through and be like, that’s actually something that would really help me,... because so often people are like, Whoa, I didn’t even know that existed, let alone a 3D printed”.

They also commented on 3DAdapt’s benefit in reducing the overwhelming nature of searching through extensive maker-sharing sites. A1 described, “maybe you could search Thingiverse for a toothbrush adapter and you would find it, but you wouldn’t necessarily know to even search that term. If you didn’t know that that thing existed. So I think I would want it to be like, limited enough, like, I don’t want it to be 1000s of things”. The experts also believed that the 3DAdapt app could offer a more efficient way to access 3D printing options compared to the current practice of browsing through a website. M3 described, “... right now, the therapist might be going through our online catalog, where they actually order it, … So the therapist might… pull up our website. And … will go through that with the patient, and say, I think this one’s going to work for you. Here’s a picture of it, because I don’t have the demo with me… So I kind of see that as kind of the way it works like, Okay, we’re going to look through the iPad here. And I think you might use you can use this, let’s look at some pictures of this”.

Experts opined that there is no demand for a “social site” for PWDs, and the app would not be helpful regarding social networking. A1 commented, “...disabled people are already like sharing a lot online on social media and sharing with each other and sharing ideas and problem-solving together, collaborating. So that’s something that’s already happening…”

• Implementation

While 3DAdapt shows promise, the experts stated that the success of 3D printing integration into rehabilitation practice will depend on dedicated education and specialist support. M3 noted, “I still consider it [3D printing] a hobby… the technology is still not plug and play”, indicating the current complexities and required expertise. A1 echoed this sentiment, stating that most clinicians lack the protected time to incorporate 3DP into their full caseloads, “Right now, it’s a little bit complicated and requires a lot of troubleshooting and hands on, it just it seems like clinicians just can’t, oftentimes unless they have very protected time, aren’t really going to be able to do 3D printing and have a full caseload”. The need for specialized therapists or professionals who focus on 3DP was emphasized, suggesting that widespread adoption in general clinical practice might be challenging without dedicated resources. M3 described, “you need a person who is the more or less of a dedicated specialist; can be an OT. That’s it. Specialists have a little extra setup time for that… we have more time than the, you know, average therapist to be able to read and work on these items, because of the way that we’re set up with being funded through the foundation”.

• Practicality

The 3DAdapt app was described as a “catalog” by all experts, which they found helpful for browsing and selecting 3D printable devices. M3 highlighted that the app could help bridge the gap for clinicians and push 3DP beyond its current “hobbyist” phase and said, “I do see value in an app where the therapist can look through some items and go, ‘Yeah, this makes sense’”. B2 appreciated the clarity of the app’s listings compared to other platforms like Thingiverse, where item descriptions and images can often be unclear. B2 described, “as an end user of a device I’m looking for… is the obvious use of that device. So do the name and the picture, make super clear to me what this thing does and how I might use it. Because all too often, if I’m flipping through, say, Thingiverse, I’ll see a thing and then title will say one thing, that picture will show something that is either zoomed in way too tight, or way too far back. And I can’t make heads or tails out of how that would work in my life. And I just scroll right past it”. A1 noted the empowerment and excitement users might feel seeing the possibilities of 3DP devices presented in a well-organized catalog and said, “And most people don’t have a 3d printer in their home, they’re still… kind of wowed by what’s possible. So I think it even just knowing that this exists, … there’s certain things they can get for very cheap or free through our program would be really empowering and exciting for people”.

• Adaptation

The app’s clean and legible design received positive feedback, with experts agreeing that the look was pleasing with visuals and text of appropriate size for all and intuitive for the intended users. M3 described 3DAdapt’s advantage in serving its intended users, “I love what you can do in Printables, but it’s not as helpful for the average person or the therapist who’s maybe dabbling in this a little or if they’re linked up with universities can print this stuff up. I think just having a really good catalog is gonna make a difference. …”, and A1 thought that 3DAdapt would make it easier for users to identify useful devices quickly, “it would maybe be more useful instead of saying to someone, go to Thingiverse, and let us know what you need, which is going to be very overwhelming and hard to kind of sort through, it might be more helpful to say check out this app and tell us what you need”. The visual presentation, including photos and descriptions, was particularly appreciated for its clarity and ease of use.

• Integration

Experts expressed that PWDs and care partners who may use 3DAdapt app will depend primarily on the technological savvy of the users. A1 described how she would share the app with tech-savvy patients, such as college students who are comfortable with smartphones and technology and said, “It depends. We have a range of [population] … someone who has a chronic illness and is a college student and is very active and participating in their care. And they’re very excited to get involved in finding solutions, and they would maybe young and are good at apps … Some folks we interact with are in their 80s and don’t have smartphone, so I won’t outwardly share … I would look through it myself and perhaps recommend things to them”.

3.1.2. Theme 2: Actionable Steps for Modification

This analysis of expert interviews identified specific themes that provided actionable steps to guide the team in improving the 3DAdapt app. These themes were used to prioritize modifications for the app, which the developers implemented. The identified themes and corresponding modifications made are detailed in Table 4.

3.1.3. Theme 3: Areas for Improvement in the Future

This theme consists of potential future improvements identified by experts for developers to consider. It includes the following four sub-themes: (1) Different levels of permissions for various users—this involves suggestions on describing the planned levels of permissions, such as providing more permissions and options for those in the roles of curator or moderator; (2) need for and ways to market the app—experts emphasized the importance of effective marketing, including through social media, word of mouth, or recommending the app for use in rehabilitation programs as a learning tool; (3) suggestions for enhancing user experience—recommendations included improving the home page, introducing videos, “screen grab” videos for the FAQ tab, and a “how to set up” tutorial. They also recommended raising awareness about where and how to 3D print an AT device by collaborating with existing organizations (e.g., https://makersmakingchange.com, accessed on 30 April 2024); and (4) sustaining user interest via community engagement—experts stressed the need to sustain interest by improving the algorithm for filtering popular or relevant assistive technology to the top of the feed.

The identified areas for future improvement have been carefully documented in a log. The team plans to address most of these in conjunction with promotional initiatives, such as producing brief YouTube video clips.

3.2. App Modifications Based on the Content Analysis

As described in Theme 2, following the content analysis, the app developers focused on addressing the identified actionable steps for modification. A list of all actionable steps identified and how they were addressed by the app developers is presented in Table 4.

Once the actionable steps were prioritized and the plans for the future solidified, the team then worked to improve the app within a 4-week time frame to meet the milestone time frame of the next usability testing phase (see Table 1) with the PWD end users; see Figure 3 for an example of an actionable step improvement.

4. Discussion and Conclusions

4.1. Overview

To our knowledge, this study is the first to utilize an evidence-driven approach, combined with expert user involvement, to develop a mobile app featuring a curated list of 3D printable assistive devices, providing searchable information related to the user’s body function, activities, and participation organized by category and ICF tags, along with descriptions, enabling PWDs to find and locate AT devices independently or with the collaboration of a clinician or care partner.

4.2. Implications

This phase employed a user-centered design approach in developing mHealth applications, which emphasized iterative testing and evaluative feedback from stakeholders, including the experts in the field early, to create effective and accessible 3D printable solutions. A small group of experts may identify usability issues and design flaws early in the development process, which can prevent costly revisions later on [40]. We found consistency with this method, as the experts assisted the developers to identify early the actionable step recommendations such as scrolling preferences and eliminating the potential of the app as a social media site. For the next phase, engaging end users during these still early phases of development helps designers understand real-world challenges and preferences, through focus groups and usability testing, which has been shown to improve the accessibility and functionality for PWDs [41]. After experts review and the developers have made the changes, the revised app can then be tested by end users, ensuring that it meets the needs and expectations of the PWD audience [42].

4.3. Strengths, Limitations, and Future Directions

The 3DAdapt Phase 1 evaluation of the development process demonstrated key strengths through its extensive engagement with end users, diverging from the prevalent trend of healthcare mobile application innovation driven by technology rather than user-centric approaches [43,44]. The incorporation of an interprofessional team, including an OT, in the app development process along with expert reviews of the app, and the integration of comprehensive functional descriptions of 3D devices based on a thorough review of available AT devices to meet clinical and end-user needs, all contributed significantly to the robustness of the app development endeavor.

Our study presents some limitations. Qualitative design research in a small sample is not meant to support formal hypothesis testing or make claims of causality that can be applied generally. Hence, our findings may not be applicable to a broader population due to self-selection bias. It is probable that our interview participants were more motivated to conduct 3D printing and had higher technical literacy levels compared to other rehabilitation providers who create AT devices using 3D printing. The effective utilization of the 3DAdapt app by end users necessitates significant resources, including initial training to identify 3D printing options and understand the printing process. Therefore, it is recommended that policymakers and rehabilitation facilities allocate protected time and logistic support for rehabilitation providers to acquaint themselves with 3DAdapt implementation [45]. Furthermore, it may be necessary to integrate training sessions within therapy programs to instruct PWDs and their care partners to utilize the app effectively.

Phases 2 and 3 of the app evaluation process involved a larger and more varied sample size of rehabilitation clinicians, PWDs, and their family members. A manuscript based on the data from these two app evaluation phases is currently under preparation.

4.4. Principal Findings

mHealth apps have the potential to revolutionize access to healthcare including the on-demand access to AT devices for PWDs. Still, limitations of 3D printing and accessibility concerns of the development of 3DAdapt must be addressed. This is achieved through collaborative efforts between PWD users, developers, and healthcare professionals including the initial step of responsiveness to expert clinical reviewer recommendations. By adopting a user-centered approach of real-life problem solving of design and subsequent feasibility testing, the team was able to develop and then address and continuously update improvements to 3DAdapt. Many improvements are simple word or presentation changes, leading to increased user satisfaction with the app and further the adoption of adaptive AT devices.

The analysis allowed the team to collect and first understand the app’s user-centeredness feature and what to continue to develop. Then, the team constructed a priority list of updates identified from Theme 2. Prioritizing the actionable steps and decision making between realistic, urgent changes to move to the next step of end-user testing and move some ideas to be possibilities for the future required communication.

With the changes and refinements completed through the content analysis of the beta app with the selected experts, the team was ready to move toward the final two phases (Phases 2 and 3) of usability and efficacy testing, focusing on the end-user PWDs and novice clinicians. Ultimately, the feedback and analysis allowed the creation of an app that brings AT into more hands, overcoming issues with technical expertise requirements and overall increasing awareness about 3D printing options for AT.

Author Contributions

Conceptualization, J.B. and S.B.; methodology, J.B., S.B. and M.K.; software, J.B.; validation, S.B. and M.K.; formal analysis, M.K.; investigation, S.B.; resources, J.B. and S.B.; data curation, M.K.; writing—original draft preparation, J.B., S.B. and M.K.; writing—review and editing, J.B., S.B. and M.K.; visualization, J.B., S.B. and M.K.; supervision, J.B. and S.B.; project administration, S.B.; funding acquisition, J.B and S.B. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

The expert reviewers consented to their participation in this content analysis phase through a non-disclosure agreement between parties.

Data Availability Statement

The data presented in this study are available on request from the corresponding author due to privacy reasons.

Acknowledgments

The authors thank students Owen Halliday, Zach Bingaman, Shane Houghton, Helen Ashbrook, and Seth Coleman for their contributions to the development of 3DAdapt and to occupational therapy students Ashley Oliva, Brianna Milstry, and Rebecca Bellino for their organization and editing efforts when inputting devices into the 3DAdapt database and with manuscript preparation.

Conflicts of Interest

The authors declare no conflicts of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. Diagram representing the application’s layers and organization, including which portions belong within the server and those that belong to the client applications. The green ovals indicate components that are primarily written for this application, while the other ovals indicate outside services.

Figure 2. The technologies are connected in the production server environment as deployed using the AWS cloud platform. The green ovals indicate components that are primarily written for this particular application while the other ovals are deployed services.

Figure 3. An example of an actionable step improvement. The screenshot on the left displays the horizontal scroll bars for the ICF tags at the top, and the experts indicated that the horizontal scroll was not intuitive, and the list of options was not easily viewable on the screen. The screenshot on the right displays the actionable improvement, as a “pop up” screen opens with vertical scrolling to display the ICF tags.

Table 1

App evaluation phases.

Phase 1: Expert feedback on initial usability and recommended app modifications	Overview: Three experts in the field of rehabilitation reviewed the 3DAdapt prototype to provide insights on the usability of the app and provide suggestions on any additional features that could benefit end users. Data Collection and Analysis: Qualitative content analysis of individual interviews.
App Modification: Developers addressed the issues and suggestions that emerged from the expert feedback.
Phase 2: Usability testing with care partners, PWDs, and clinicians	Overview: This phase assessed the feasibility of the 3DAdapt app, focusing on its acceptability among targeted users with disabilities and clinicians as end users of the app. Data Collection: Testing conditions included participants using the app to complete three standardized tasks, answering quantitative and qualitative questions about app acceptability and satisfaction, and reporting any unexpected events during app usage. Data Analysis: A mixed-methodology approach was adopted, incorporating usability scales and satisfaction Likert scale reporting, and interview/focus group data.
App Modification: Developers addressed the issues that emerged from the usability testing.
Phase 3: Efficacy testing of 3DAdapt with PWDs and novice clinician pairs	Overview: PWD and novice clinician users of 3D printing participated in an efficacy testing process. Data Collection: Testing conditions included participants completing a search and order of a device to print, receiving and using the printed assistive device, and then providing feedback within the app. This phase specifically focused on utilizing N-of-1 methodologies tailored to each user’s functional limitations.
App Modification: Developers addressed the issues identified from the efficacy testing.

For the purpose of this manuscript, only Phase 1 of the evaluation process is described in detail. A manuscript based on the data from evaluation Phases 2 and 3 is currently under preparation.

Table 2

Demographics of expert app reviewers.

Expert	Specialization and Employment Setting	Relevant Experience and Expertise
Expert A1	United States (US)-based licensed occupational therapist (OT) employed by their state’s designated AT Act Program in the northeast.	Meets with clients and healthcare providers on a consultant basis. Prints and distributes pre-designed and customized 3D devices; for example, fulfilling custom AAC keyguard requests. Responsible for managing the social media accounts to inform the public of AT options and disability advocacy topics, including 3D-printed devices that are freely available to anyone in the state who needs the device.
Expert B2	US-based licensed OT and a faculty member at a public R1 institution in the midwest.	Roles at the institution are primarily research with clinical program work at the associated medical center and AT research within a neurological clinic, primarily serving amyotrophic lateral sclerosis (ALS) patients.
Expert M3	A US-based licensed OT for over 25 years for a large non-profit healthcare system in the midwest operating 12 hospitals and 90 clinics.	Primary role is daily assignments of occupational therapy evaluations and treatments with productivity standards. Supervises the AT program; in their role, this expert fulfills 3D-printed device requests in the healthcare system, both pre-designed and customizable CAD AT devices.

Table 3

Identified themes and sub-themes.

Themes	Sub-Themes
1. Utility and relevance of 3DAdapt	Acceptability
	Demand
	Implementation
	Practicality
	Adaptation
	Integration
2. Actionable steps for modification	See Table 4 below for a detailed list of actionable steps
3. Areas for improvement in the future	Different levels of permissions for various usersNeed for and ways to market the app
	Suggestions for enhancing user experience
	Sustaining user interest via community engagement

Table 4

The actionable steps identified from the content analysis, along with how the app developers addressed them.

Actionable Steps for Modification	Efforts Made to Improve the App
Confusion between printset (a word invented for this mobile app) and files	Made it so that the average user is not even exposed to this level of detail but an advanced user will see definitions and purpose explanations when they look for the details.
Horizontal scroll is not intuitive in several places	Improved displays to not require horizontal scrolling at all.
Uniform style of description. The name and purpose of each design is not obvious in the user experience	When looking through designs, users need to see the curated title and purpose first rather than the original source title and description, which may even be in a different language. The original information can still shown but is initially hidden.
More pediatric options	Added more options for children and added pre-sized printsets with smaller sizing.
Liability concerns and not comfortable recommending	Enhanced the legal safeguards in the “terms and conditions” so healthcare personnel would not be responsible for users’ interactions in the app.
Fonts used are too large	Tested to make sure the phone’s accessibility settings for font size are used.
Account creation needs to be clarified with passwords	Added information about password requirements and clarity to error messages.
Customization	Added the ability to customize parametric models and clarified that sizing is typically in millimeters.
Q&A crowdsource	Encouraging comments with the description of how to use comments and added tab for “FAQ/Help” overall for the app.
Notification and email frequency	Added frequency options to user preferences.
Privacy protection concerns	Added prompt in photos not to include any personal, identifiable information, and checked for scrubbing of geotagging in photos.
Complex post-processing, filtering, and comments	Added search filters for customizable and more than one file to assemble. Added prompts in the reviews to write information if significant post-processing modifications were required.
ICF categories may not be helpful for non-professional users	Simplified keywords used for ICF categories but kept full descriptions when performing a “hard press”.
Private messaging is not a priority	Removed “messages” to encourage open public comments and reviews.

Appendix A. Interview Guide for the Experts

Please note that this guide only represents the main themes to be discussed with the participants and as such does not include the various prompts that may also be used (examples given for each question). Non-leading and general prompts will also be used, such as “Can you please tell me a little bit more about that?” and “What does that look like for you?”.

• Interview Protocol

• Permission: First of all, would you mind if we video record this zoom call because that would make it easier for me to summarize the results afterward?

• Questions for Experts

• Could you talk about your experience as an OT? What type of practice areas have you worked in? Could you describe your experiences and expertise with the application of 3D printing tech with AT provision?

• Do you have experience with evaluating health apps like this one? And if so, could you tell us something about what kind of rehabilitation apps you have evaluated?

• Quality of information

• What do/did you like about this app?

• Prompt: Does the app fulfill your needs? Why or why not? Do/did you enjoy sessions with your health app? Is/Was working with the app satisfying? Is the app worth recommending to others?

• How easy is/was using the app?

• Prompt: What makes the app information clear and understandable? How do/did you find the font size and representations?

• Who do you think could benefit from this app?

• Is app content correct, well-written, and relevant to the goal/topic of the app?

• Have you sometimes not known (did you sometimes not know) what to do next with this app?

• Prompt: Are there any parts that you do not use because they are complicated? What app features do/did you find intuitive? Do you sometimes wonder if you are using the app the right way? Who do you turn to help using the app?

• What medical or technical jargon have you seen in this app that a potential app user may not understand?

• Are you satisfied with the time taken to perform tasks on this app?

• Prompt: How about time to display graphs? Time to synchronize information?

• Acceptability/Benefit to users

• Could you discuss to what extent this app would be judged as suitable or attractive among the intended recipients of the app?

• Could you think about what benefits this app is going to provide the families, caregivers, and professionals working with users needing assistive technology? For example,

• Awareness: This app is likely to increase awareness of the importance of addressing…

• Knowledge: This app is likely to increase knowledge/understanding of…

• Attitudes: This app is likely to change attitudes toward improving…

• Intention to change: This app is likely to increase intentions/motivations to address…

• Help-seeking: Use of this app is likely to encourage further help-seeking for [insert target health behavior]…

• Behavior change: Use of this app is likely to increase/decrease…

• As you navigate the search-and-find features, are the devices you find practical for the end user?

• Demand

• Have any users expressed interest in ways of searching and finding 3D devices and needing a method that is more user-friendly?

• Prompt/probe: How much demand for this AT app is likely to exist among users who would benefit from it?

• Effective implementation

• Could you discuss the extent, likelihood, and manner in which we will be able to make this app available for users as planned and proposed? What barriers do you foresee?

• Prompt: What existing barriers do you foresee to implementation? [I.e., sots, access, user needs and preferences, and availability of similar apps.] What recommendations do you have on how to overcome them?

• How much and what type of resources will be needed to implement it?

• What factors can you foresee affecting the ease or difficulty of implementing the app?

• Based on your experiences, how can we, as rehabilitation professionals, help the users of this app better?

• Prompt: In your opinion, what is the best way to collaborate with users to train about this app? How can we spread awareness about this app?

• Practicality

• To what extent can this app be introduced among intended users using existing resources and current rehabilitation circumstances without outside support?

• What are your thoughts on the privacy protection concerns of using this app?

• Efficiency, speed, or quality of implementation

• Could you think about any positive/negative effects on the intended users?

• Ability of participants to use this app?

• Integration

• Could this app be integrated into the workday? Are there barriers?

• Research

• Clinical applicability

• Classroom instruction

• Thinking of your existing organizational setting where you provide 3D printing, how easy or difficult would it be to integrate this app into your available services to clients? Probe: What is the fit with your organization’s current infrastructure? What are your thoughts on the sustainability of this app for the future of how you design or deliver 3D-printed devices as part of your services or research? Do you foresee any costs (monetary or other perceived barrier) associated with integrating this app into your organization’s offerings?

• Adaptation

• Could you think about some changes/modifications we need to make to accommodate different users’ environmental and social contexts?

• Prompt: What about possible modifications considering potential users’ varying disabilities and accessing the information?

• Two Aspects: (1) Thoughts on the adaptive devices that are searchable and included within 3DAdapt? Does this consider adaptive devices for multiple different disabilities? (2) Thoughts on the user experience of the app itself? Does it lend itself to be accessible to users with various levels of abilities?

• Prompt: What makes the app information clear and understandable? How do/did you find the font size and representations?

• What customization features would you like to see in this health app?

• Wrap-up questions

• Is there anything I did not ask that we should consider for refining the app further?

• Thank you again for taking the time to meet with us. Please rest assured that your anonymity will be protected and that your name will not be mentioned in any of the reports or presentations.

References

1. Dawson, R.M.; Felder, T.M.; Donevant, S.B.; McDonnell, K.K.; Card, E.B., III; King, C.C.; Heiney, S.P. Identifying the strengths and limitations of existing mobile application evaluation tools. Nurs. Inq.; 2020; 27, pp. 1-8. [DOI: https://dx.doi.org/10.1111/nin.12333] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/31854055]

2. Vawdrey, D.K.; Wilcox, L.G.; Collins, S.A.; Bakken, S.; Feiner, S.; Boyer, A.; Restaino, S.W. A tablet computer application for patients to participate in their hospital care. Proceedings of the AMIA Annual Symposium Proceedings Archive; Washington DC, USA, 22–26 October 2011; pp. 1428-1435.

3. Higgins, J.P. Smartphone applications for patients’ health and fitness. Am. J. Med.; 2016; 129, pp. 11-19. [DOI: https://dx.doi.org/10.1016/j.amjmed.2015.05.038] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/26091764]

4. Paglialonga, A.; Patel, A.A.; Pinto, E.; Mugambi, D.; Keshavjee, K. The healthcare system perspective in mHealth. m_Health Current and Future Applications; Springer: Berlin/Heidelberg, Germany, 2019; pp. 127-142.

5. Roumeliotis, K.; Tselikas, N.D. Evaluating progressive web app accessibility for people with disabilities. Network; 2022; 2, pp. 350-369. [DOI: https://dx.doi.org/10.3390/network2020022]

6. Gupta, N.; Castillo-Laborde, C.; Landry, M.D. Health-related rehabilitation services: Assessing the global supply of and need for human resources. BMC Health Serv. Res.; 2011; 11, 276. [DOI: https://dx.doi.org/10.1186/1472-6963-11-276]

7. Tangcharoensathien, V.; Witthayapipopsaku, W.; Viriyathorn, S.; Patcharanarumol, W. Improving access to assistive technologies: Challenges and solutions in low-and middle-income countries. WHO South-East Asia J. Public Health; 2018; 7, pp. 84-89. [DOI: https://dx.doi.org/10.4103/2224-3151.239419] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/30136666]

8. Lu, Y.; Wu, Y.; Liu, J.; Li, J.; Zhang, P. Understanding health care social media use from different stakeholder perspectives: A content analysis of an online health community. J. Med. Internet Res.; 2017; 19, [DOI: https://dx.doi.org/10.2196/jmir.7087] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/28389418]

9. Schwartz, D.A.; Schofield, K.A. Utilization of 3D printed orthoses for musculoskeletal conditions of the upper extremity: A systematic review. J. Hand Ther.; 2021; 36, pp. 166-178. [DOI: https://dx.doi.org/10.1016/j.jht.2021.10.005]

10. Schwartz, J.K.; Fermin, A.; Fine, K.; Iglesias, N.; Pivarnik, D.; Struck, S.; Varela, N.; Janes, W.E. Methodology and feasibility of a 3D printed assistive technology intervention. Disabil. Rehabil. Assist. Technol.; 2020; 15, pp. 141-147. [DOI: https://dx.doi.org/10.1080/17483107.2018.1539877] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/30663439]

11. Janson, R.; Burkhart, K.; Firchau, C.; Hicks, K.; Pittman, M.; Yopps, M.; Hatfield, S.; Garabrant, A. Three-dimensional printed assistive devices for addressing occupational performance issues of the hand: A case report. J. Hand Ther.; 2020; 33, pp. 164-169. [DOI: https://dx.doi.org/10.1016/j.jht.2020.03.025]

12. Rasmussen, K.-A.M.; Stewart, B.C.; Janes, W.E. Feasibility of customized 3D-printed assistive technology within an existing multidisciplinary amyotrophic lateral sclerosis clinic. Disabil. Rehabil. Assist. Technol.; 2021; 18, pp. 1466-1472. [DOI: https://dx.doi.org/10.1080/17483107.2022.2034996]

13. Thorsen, R.; Cugnod, D.; Ramella, M.; Converti, R.; Ferrarin, M. A parametric 3D printed assistive device for people with cerebral palsy–assessment of outcomes and comparison with a commercial counterpart. Assist. Technol.; 2023; 36, pp. 16-21. [DOI: https://dx.doi.org/10.1080/10400435.2023.2202696] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/37083458]

14. Patterson, R.M.; Salatin, B.; Janson, R.; Salinas, S.; Mullins, M.J. A current snapshot of the state of 3D printing in hand rehabilitation. J. Hand Ther.; 2020; 33, pp. 156-163. [DOI: https://dx.doi.org/10.1016/j.jht.2019.12.018] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32156576]

15. Best 3D Printing Software. Available online: https://www.g2.com/categories/3d-printing?selected_view=grid&segment=all#grid (accessed on 15 September 2023).

16. Developer Documentation. Available online: https://customizer.makerbot.com/docs (accessed on 15 September 2023).

17. Getting Started. Available online: https://models.makewithtech.com/help (accessed on 15 September 2023).

18. Getting Started. Available online: https://learn.cadhub.xyz/docs/ (accessed on 15 September 2023).

19. Keller, J. Stack Overflow’s Crowdsourcing Model Guarantees Success. 2010; Available online: https://www.theatlantic.com/technology/archive/2010/11/stack-overflows-crowdsourcing-model-guarantees-success/66713/ (accessed on 15 September 2023).

20. Majchrzak, A.; Malhotra, A. Towards an information systems perspective and research agenda on crowdsourcing for innovation. J. Strateg. Inf. Syst.; 2013; 22, pp. 257-268. [DOI: https://dx.doi.org/10.1016/j.jsis.2013.07.004]

21. Karachiwalla, R.; Pinkow, F. Understanding crowdsourcing projects: A review on the key design elements of a crowdsourcing initiative. Creat. Innov. Manag.; 2021; 30, pp. 563-584. [DOI: https://dx.doi.org/10.1111/caim.12454]

22. Benham, S.; San, S. Student Technology Acceptance of 3D Printing in Occupational Therapy Education. Am. J. Occup. Ther.; 2020; 74, pp. 7403205060p1-7403205060p7. [DOI: https://dx.doi.org/10.5014/ajot.2020.035402] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32365312]

23. Benham, S.; Bush, J.; Curley, B. 3D printing applications through peer-assisted learning and interprofessional education approaches. Focus Health Prof. Educ. A Multi-Prof. J.; 2022; 23, pp. 81-95. [DOI: https://dx.doi.org/10.11157/fohpe.v23i3.591]

24. Slegers, K.; Krieg, A.M.; Lexis, M.A.S. Acceptance of 3D Printing by occupational therapists: An exploratory survey study. Occup. Ther. Int.; 2022; 2022, 4241907. [DOI: https://dx.doi.org/10.1155/2022/4241907] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/36636037]

25. Giesbrecht, E.; Carreiro, N.; Mack, C. Improvement and retention of wheelchair skills training for students in entry-level occupational therapy education. Am. J. Occup. Ther.; 2021; 75, pp. 7501205160p1-7501205160p9. [DOI: https://dx.doi.org/10.5014/ajot.2021.040428] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33399064]

26. Benham, S.; Milstrey, B.; Stemple, J.; Davis, J.; Scatena, D.; Bush, J.; Kolakowsky-Hayner, S.; Amy, K. Mobile device accessibility with 3D printed devices for individuals with physical disabilities. Disabil. Rehabil. Assist. Technol.; 2023; pp. 1-6. [DOI: https://dx.doi.org/10.1080/17483107.2023.2280244]

27. Geneva World Health Organization. International Classification of Functioning, Disability and Health ICF. 2001; Available online: https://icd.who.int/dev11/l-icf/en (accessed on 15 September 2023).

28. Aigner, S. Compose Multiplatform for iOS Is in Alpha. 18 May 2023. Available online: https://blog.jetbrains.com/kotlin/2023/05/compose-multiplatform-for-ios-is-in-alpha/ (accessed on 15 September 2023).

29. Jake-Schoffman, D.E.; Silfee, V.J.; Waring, M.E.; Boudreaux, E.D.; Sadasivam, R.S.; Mullen, S.P.; Carey, J.L.; Hayes, R.B.; Ding, E.Y.; Bennett, G.G. et al. Methods for Evaluating the Content, Usability, and Efficacy of Commercial Mobile Health Apps. JMIR Mhealth Uhealth; 2017; 5, e190. [DOI: https://dx.doi.org/10.2196/mhealth.8758]

30. Abras, C.; Maloney-Krichmar, D.; Preece, J. User-Centered Design. Encyclopedia of Human-Computer Interaction, Sage Publications; Bainbridge, W. Sage: Thousand Oaks, CA, USA, 2004; pp. 445-456.

31. Benner, P. From novice to expert. Menlo Park; 1984; 84, pp. 10-97. Available online: https://files.eric.ed.gov/fulltext/ED384695.pdf#page=130 (accessed on 30 April 2024). [DOI: https://dx.doi.org/10.1097/00000446-198412000-00027]

32. Patton, M.Q. Qualitative Research & Evaluation Methods: Integrating Theory and Practice; 4th ed. Sage: Thousand Oaks, CA, USA, 2014.

33. Bowen, D.J.; Kreuter, M.; Spring, B.; Cofta-Woerpel, L.; Linnan, L.; Weiner, D.; Bakken, S.; Kaplan, C.P.; Squiers, L.; Fabrizio, C. et al. How we design feasibility studies. Am. J. Prev. Med.; 2009; 36, pp. 452-457. [DOI: https://dx.doi.org/10.1016/j.amepre.2009.02.002] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/19362699]

34. Lagan, S.; Emerson, M.R.; King, D.; Matwin, S.; Chan, S.R.; Proctor, S.; Tartaglia, J.; Fortuna, K.L.; Aquino, P.; Walker, R. et al. Mental health app evaluation: Updating the American Psychiatric Association’s framework through a stakeholder-engaged workshop. Psychiatr. Serv.; 2021; 72, pp. 1095-1098. [DOI: https://dx.doi.org/10.1176/appi.ps.202000663] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33882716]

35. Erlingsson, C.; Brysiewicz, P. A hands-on guide to doing content analysis. Afr. J. Emerg. Med.; 2017; 7, pp. 93-99. [DOI: https://dx.doi.org/10.1016/j.afjem.2017.08.001] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/30456117]

36. Hsieh, H.-F.; Shannon, S.E. Three approaches to qualitative content analysis. Qual. Health Res.; 2005; 15, pp. 1277-1288. [DOI: https://dx.doi.org/10.1177/1049732305276687] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/16204405]

37. Elo, S.; Kyngäs, H. The qualitative content analysis process. J. Adv. Nurs.; 2008; 62, pp. 107-115. [DOI: https://dx.doi.org/10.1111/j.1365-2648.2007.04569.x] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/18352969]

38. Pope, C.; Ziebland, S.; Mays, N. Qualitative research in health care. Analysing qualitative data. BMJ; 2000; 320, pp. 114-116. [DOI: https://dx.doi.org/10.1136/bmj.320.7227.114]

39. Saunders, B.; Sim, J.; Kingstone, T.; Baker, S.; Waterfield, J.; Bartlam, B.; Burroughs, H.; Jinks, C. Saturation in qualitative research: Exploring its conceptualization and operationalization. Qual. Quant.; 2018; 52, pp. 1893-1907. [DOI: https://dx.doi.org/10.1007/s11135-017-0574-8]

40. Jaspers, MW A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. Int. J. Med. Inform.; 2009; 78, pp. 340-353. [DOI: https://dx.doi.org/10.1016/j.ijmedinf.2008.10.002]

41. Petrie, H.; Hamilton, F.; King, N.; Pavan, P. Remote usability evaluations with disabled people. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; Montreal, QC, Canada, 22–27 April 2006; pp. 1133-1141. [DOI: https://dx.doi.org/10.1145/1124772.1124942]

42. Kushniruk, A.W.; Patel, V.L. Cognitive and usability engineering methods for the evaluation of clinical information systems. J. Biomed. Inform.; 2004; 37, pp. 56-76. [DOI: https://dx.doi.org/10.1016/j.jbi.2004.01.003]

43. Principles for Digital Development. Available online: https://digitalprinciples.org/ (accessed on 30 April 2024).

44. Pereira-Azevedo, N.; Carrasquinho, E.; Cardoso de Oliveira, E.; Cavadas, V.; Osório, L.; Fraga, A.; Castelo-Branco, M.; Roobol, M.J. mHealth in Urology: A Review of Experts’ Involvement in App Development. PLoS ONE; 2015; 10, e0125547. [DOI: https://dx.doi.org/10.1371/journal.pone.0125547] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/25984916]

45. WHO Guideline Recommendations on Digital Interventions for Health System Strengthening; World Health Organization: Geneva, Switzerland, 2019; Available online: https://www.ncbi.nlm.nih.gov/books/NBK541902/ (accessed on 30 April 2024).