2.1. The Use of Mobile Health Apps

Mobile health apps can support continuous health monitoring at the individual and population level, encourage health behaviors to prevent or reduce health problems, support self-health management, increase knowledge, and reduce healthcare visits [3,4]. It has been proved to be beneficial to people's health in many aspects, in psychology, disease management, and healthcare [5,6].

In psychology, mental health apps have potentials in improving the monitoring and management of mental health symptoms or disorders [7]. For example, studies show that low-intensity mindfulness intervention through the use of mobile health apps can effectively improve the psychological distress (depression, anxiety, stress) and sleep dysfunction (subjective sleep quality, sleep latency, habitual sleep efficiency) of Chinese college students [8].

In disease management, mobile health apps can improve education and self-care for patients with complex conditions [9]. It has achieved good results in improving the metabolic index of diseases and strengthening self-management and shown overall positive results in improving physical deficiencies in hypertension, weight, tobacco smoke, diabetes, dyslipidemia, etc [10]. For example, brown demonstrated the feasibility and usability of an interactive, immersive 3d iPad health game for cancer patient disease management through experiments [11]. Baker developed a new medical non-contact control system, which greatly helped patients with physical disabilities. To realize the function of calling nurses and the effect of controlling hospital equipment [12].

In healthcare, satisfactory treatment compliance and subsequent health maintenance can be achieved with a high level of exposure to mobile health applications [13]. Some apps may have positive benefits when used to deliver exercise or gait training interventions, as self-management systems, or as measurement tools [14]. They are also effective adjuncts to the improvement and maintenance of weight loss after bariatric surgery [15].

Mobile health apps play an important role in improving or maintaining people's physical health, but the quality of mobile health apps on the market today varies greatly. Therefore, it is necessary to evaluate them and design applications that are more acceptable to users, thus bringing greater convenience to people.

2.2. Evaluation Index of Mobile Health Application

At present, many researches have been carried out in the field of mobile health application evaluation, and scholars have proposed different evaluation indexes.

We can use perceived usefulness to measure attitude, perceived ease of use to measure satisfaction, and behavioral intention to measure interest and importance to evaluate mobile health applications [16]. Awareness of devices and health information plays an important role in influencing how users rate mobile health applications [17]. Flexibility; performance; competency; learnability; completeness; information; other outcomes; error prevention; flexibility can be used to evaluate the usability of mobile health applications [18]. It has been proved that the quality of information and the credibility of sources played a great role in the evaluation of mobile health applications [19]. Privacy also plays an important role, patients want hospitals to assist them with high efficiency without revealing patients' identities [20].

We can evaluate applications by using the following criteria: cost (free or paid); content (medical information, symptom log, etc.); relevant public or private healthcare; target audiences healthcare and non-healthcare professionals; validity (citing medical information that has been peer-reviewed or otherwise verified); and average user ratings (from 1 to 5 stars) [9]. Georgsson puts forward a multilevel evaluation model: level 1 is the target system specification, which is used to understand user tasks for system development. Level 2 is a test of task performance to evaluate system validation and human-computer interaction in a laboratory environment. Level 3 aims to combine environmental factors to determine the impact of the system in the real environment [21]. According to the above research results, evaluation indexes are mostly specified from the application itself. Although they are important for evaluating mobile health applications, they mostly use indicators unrelated to the mobile health application service field. Therefore, we add indicators starting from users' experience, which can better reflect the “user-centered” concept.

2.3. Evaluation Method of Mobile Health Application

In the evaluation method of mobile health application, it can be roughly divided into two categories. One is to conduct relatively simple statistical analysis of the data by designing questionnaires, the other is to build a model and then analyze the data.

Brown uses Dedoose qualitative data analysis (QDA) software, a web-based qualitative and hybrid data analysis program, to analyze collected data and visualize the results to evaluate mobile health applications [18]. Battle uses a comprehensive analysis, which combine qualitative and quantitative methods to evaluate mobile health applications. In the qualitative part, data are collected by means of targeted sampling and Emi-structured interview, then they do thematic analysis. In the quantitative part, data are collected, calculated with SAS 9.2, and then analyzed and compared [22]. Liu estimated the influence of each quality dimension on continuous use intention, by statistical analysis of the survey responses of 191 users [23].

Meng combined health awareness with refined likelihood model to test people's daily willingness to use mobile health applications [19]. Mattias puts forward a method to enhance cognitive roaming (the Cognitive walkthrough (CW)), which means take the user as the center of the CW (UC—the CW) when solve defects that have been found in the original technology, and verify by thinking about the statement agreement. In order to assess the effectiveness, efficiency, and self-management of mobile health applications among users with diabetes [21]. Mattias tested the usability of mobile health applications by using the framework analysis (FA) approach and usability issues taxonomy (UPT). It is also shown that using this method can help identify usability problems, especially initial usability problems [24]. Paulo used a mixed method of partial least squares and fuzzy set qualitative comparative analysis to evaluate the number of causal condition combinations that promote mHealth adoption [25].

After the above summary, we can find that the current evaluation methods are mainly simple statistical analysis, or the establishment of simple qualitative and quantitative methods to analyze the obtained data, which are traditional and do not reflect intelligence, and do not consider the authenticity of the data.

Therefore, this paper proposes a mobile health application evaluation model based on user experience, which uses ontology language to describe the relationship between concepts in the evaluation model, makes different evaluation factors through domain experts, and calculates the weight distribution of evaluation factors, according to the knowledge collected by the user system, by machine learning algorithm meanwhile it supports optimize the weight by the prior knowledge of the domain experts through Delphi method.

4.1. Data Collection

870 young people were selected in this study for questionnaire survey, which includes 452 men and 418 women from 8 different provinces. Among them, there were undergraduates, postgraduates, teachers, civil servants and enterprise employees. At the same time, it is necessary to introduce the evaluation experts in Delphi method. It is composed of 20 experts with rich experience in mobile health application, including experts from Hefei university of technology (5 people), Hebei university (3 people), Beijing University of Technology (5 people), and employees from mobile health apps company (7 people). Some of the data used to support the findings of this study are included within the article.

4.2. Evaluation Indicators System

In order to facilitate readers to better understand the content of expert-level evaluation model in mobile health application evaluation model, this part will explain the definitions of the terms involved in the model and the relationship between them.

1. Evaluation Factor: Evaluation factor is a basic evaluation unit extracted from users' experience (including cognition and emotion), which is composed of a binary group: f=(attr,W), where attr represents the name of users' experience, w∈[0,1] represents the weight of evaluation factor, reflecting the importance of this evaluation factor in the evaluation of mobile health application.

   In this paper, reliability is an important aspect to describe users' experience and influence users' evaluation of APP, which can be defined as the evaluation factor of mobile health application. Domain experts can customize various types of evaluation factors in this model. To facilitate the use of prior knowledge to calculate the weight of evaluation factors, we give the concept of evaluation categories.

2. Evaluation Category: Evaluation category represents a conceptual clustering of evaluation factors, which is composed of triples: S=(Sn,Set,W), where Set={e1,e2,…,ek,en,1≤k≤n}, (ek is the evaluation factor or evaluation category); Sn is the name of the evaluation category; w∈[0,1] represents the weight of the evaluation category, reflecting the importance of the evaluation category in service evaluation.

   In this paper, cognition-based information quality evaluation can be used as an evaluation category S, the Sn item of which is “based on cognition,” the set item is {S(attr = “Cognitive Content”), S(attr = “Cognitive Interaction”)} including two secondary evaluation categories: the content information that mainly affects cognition and the interaction information that mainly affects cognition.

3. Evaluation Concept Tree: Evaluation concept tree T is a tree structure composed of evaluation factors and evaluation categories as nodes. Evaluation categories are internal nodes of evaluation concept trees and evaluation factors are leaf nodes of evaluation concept trees. The hierarchy of evaluation concept tree is composed as follows: Each element in the set item of evaluation category s is the child node of evaluation category s; all nodes have a unique parent except for the evaluation category at the root node. Meanwhile, the weights of nodes in the evaluation concept tree have the following constraints:

4. The weight of root node is 1.

5. The weight of any evaluation class node s is the sum of all the weight of its children.

In this paper, Figure 2 shows the evaluation concept tree of this study. The root node of the evaluation concept tree has two sub-evaluation categories (based on cognition and based on emotion), which contain different sub-evaluation categories. These two indicators were chosen because of what Christina Chick CF found in his research that clinical research across the developmental spectrum increasingly reveals the nuanced ways in which emotion and cognition can work to either support or derail rational (i.e., healthy or goal-consistent) decision making [26]. Weights start at the root node and are distributed layer by layer to the lower nodes and end at the leaf node. Different weight distribution of evaluation factors reflects different evaluation standards of users. Since different types of mobile health applications have different evaluation concept trees, it is necessary for domain experts to customize evaluation concept trees for mobile health applications in this field.

Figure 2

Evaluation concept tree.

4.3. Semantic Description Diagram of Evaluation Indicators System

Mobile health application evaluation model is composed of quintuples M=[T,SI,DI,ER,P] where T represents evaluation concept tree; SI represents the instance of the survey being evaluated; DI represents a data instance of SI, which contains multiple data items. Each data item represents the value of data instance DI on an evaluation factor. ER represents the output of evaluation model and records the quality evaluation result of DI. P is the relation set of the evaluation model, representing various constraint relations among concepts in the evaluation model.

Figure 3 shows the semantic description graph of the evaluation model of mobile health application in this study. Evaluation concept tree represents the tree of evaluation T, category represents the category of evaluation, factor represents the factor of evaluation, and predicate {has Root, consist of} jointly defines hierarchical relation of evaluation concept tree. Survey instance represents the instance of survey SI, data instance represents the instance of data DI, evaluation result represents the result of evaluation ER), predicate {from Tree, has Factor} represents the relationship between Data instance DI and evaluation concept tree T, while predicate from service represents data instance DI from service instance SI and evaluation result ER is evaluation result of service instance SI. O stands for name field and R for cardinality. A cardinality of 1 is numeric, and a cardinality of −1 is non-numeric.

Figure 3

Semantic description diagram of mobile health application evaluation model.

According to the above description of the evaluation model and its components. We assume that the evaluation concept tree T has n evaluation factors, and the weight of each evaluation factor is, and the value of DI, a data instance of the investigation instance SI, on each evaluation factor is V1,V2,…,Vn, in that way the evaluation of mobile health application by survey instance SI is the weighted sum of the values of DI on each evaluation factor, given in Equation (1)

Calculation formula of evaluation:

It can be seen from Equation (1) that the evaluation results of mobile health applications are determined by the value of evaluation factors and the weight distribution of evaluation factors. The value of the evaluation factor is derived from user experience, and the weight distribution of the evaluation factor can be calculated according to the user's feelings in the process of use and the prior knowledge of domain experts, which is obtained from the domain experts with the help of the evaluation concept tree. In the next section, we will calculate the weight distribution.

5.1. Machine Learning Algorithm

The machine learning algorithm can automatically learn the implicit weight distribution information in sample the data set to calculate the weight of the evaluation factors.

In this study, 870 young people were selected for the questionnaire survey, and the options were set for each evaluation factor and the acceptance level of the mobile health application. Because users rate mobile health apps based on their experience, the acceptance degree implies the information about the weight distribution of the evaluation factor. We represent a group of sample data as a binary group: S= <DI,d>; DI represents the evaluation data of a survey instance, and d has two values: d1 = True and d2 = False, thus indicating that respondents like (dislike) this candidate instance. In the machine learning algorithm, information gain represents the degree of influence of different evaluation factors on the service selection results, so this model uses information gain to calculate the weight of the evaluation factors.

Set S1 as a sample subset of d=d1 in sample dataset S and a sample set of candidate instances favored by the respondents, and set S2 as a sample subset of d=d2 in sample dataset S, representing the unselected sample set of service instances; thus, S=S1 and S2. We classify the sample data set according to the value of d term, and the expected information obtained is given by Equation (2):

where p1 is the probability that d is equal to d1 for any sample in the sample dataset S and P2 is the probability that d is equal to d2 in any sample of S. Use S1/S and S2/S to describe the sample subsets. Suppose SK1i is the sample set of subset Ski, where “d=d1,” and suppose Sk2i is the sample set of subset Ski where “d=d2”; then, the expected information is given by Equation (3): where, IS0,S1,…,Sn,fi can be calculated according to formula (2), P1=Sk1i/SKi and P2=Sk2i/SKi. Then the information gain obtained by the evaluation factor is given by Equation (4):

Equation (4) can be used to calculate the information gain of each evaluation factor, and Equation (5) calculates the weight of each evaluation factor based on Equation (4):

By calculating the information gain of the evaluation factors, we can learn the weight distribution of each evaluation factor from the sample set. After sorting out the data of 870 questionnaires, we divide the data set into two categories according to the attitude of the respondents toward the mobile health application: attitude ratings of 4 and 5 are placed in one category, and those from 1 to 3 are placed in another category. According to the machine learning algorithm, the weights of third-level indexes are calculated, and the results are shown in Figure 4. To make the picture look simpler. CC is short for cognitive content, CI is short for cognitive interaction, EC is short for emotional content, EI is short for emotional interaction.

Figure 4

Weight distribution of secondary indicators.

5.2. Delphi Method

Because the information used to construct the sample set is gained by using the knowledge generated from the data collected from the survey, the evaluation result depends heavily on the size of the sample set and the quality of the sample. Therefore, the evaluation effect obtained is somewhat accidental. In fact, the domain experience of domain experts also implies the information of the weight distribution of the evaluation factors; as a result, adding the domain expert's prior knowledge to the calculation not only accelerates the calculation of the weight but also improves the accuracy of the weight distribution. The prior knowledge referred to in this study is the empirical knowledge of domain experts of the importance of different evaluation categories, which can be used to calculate the weight distribution of the evaluation factors. Therefore, we adopt the Delphi method to introduce the knowledge of experts.

We score the indicators by sending emails to experts and then process the scoring results by using a statistical approach. In the expert questionnaire, the importance degree is assigned based on the five subscales method. Because the familiarity of the experts with the five-subscale index varies and because the bases for their judgments are inconsistent, we need to use the degree of familiarity and the basis of judgment to distinguish the authority of experts; namely, the authority coefficient is equal to the average degree of familiarity and determines the arithmetic coefficient. The authority coefficient of experts is generally required to be larger than 0.7, and the degree of expert authority in this study meets this requirement. See Table 1.

In this study, experts were consulted mainly on the weight of the first- and second-level index. Because of the small number of indicators and the low decision-making complexity, 20 experts are enough. The specific steps were as follows: the questionnaire was developed based on the literature research and evaluation indexes determined by experts in previous fields. After the Delphi questionnaire was collected, the questionnaire was analyzed. The Kendall coordination coefficient of the first-level index in the first round of the letter consultation was 0.716 (χ2=42.955, p<0.001), thus indicating a high degree of expert opinion coordination and no need for further letter consultation. The weights of the first- and second-level indicators are calculated; the weights are shown in Table 2. (For convenience, we simplify the representation of the indicators: “C” is an abbreviation for “based on cognition”; “CC” is an abbreviation for “Cognitive content”; “CI” is an abbreviation for “Cognitive interaction”; “E” is an abbreviation for “based on emotion”; “EC” is an abbreviation for “emotional content”; and “EI” is an abbreviation for “emotional interaction.”)

5.3. Interactive Algorithm Combining Machine Learning and the Delphi Method

To effectively use prior knowledge, we define the local weight of the nodes in the evaluation concept tree. The local weight W′N,M from node N to its ancestor node M is defined to represent the weight of node N in the evaluation concept subtree with node M as the root node. When node M is the parent node of node N, W′(N,M) is also called the nearest local weight of node N, denoted by the symbol W′(N). Meanwhile, the local weight of the root node and the nearest local weight of the evaluation concept tree are both 1. The local weight of a node is a measure of its importance in a local range. According to the definition of local weight, we can regard the weight of node N as the local weight W′(N,root) of node N in the evaluation concept tree. The weights of the nodes can be used to calculate the local weights.

In Definition 1, if the node N0 is the root node of the evaluation concept tree, then the node N0 is the weight of the node. From Definition 1 we get Definition 2.

Set the node sequence on the path from any evaluation factor to root of the root node of the evaluation concept as (S0,S1,…,Sn,fi) (where n ≥ 0 and S0 = root), and set the nearest local weight of S0,S1,…,Sn as W0′,W1′,…,Wn′ respectively, because S0 = root, W0′=1. From Definition 1, we can obtain the local weight of the node fi to a node Sk sequence:

From Definition 2, it can be concluded that the weight of the evaluation factor is

By substituting Equation (6) for Equation (7), a new formula for calculating the weight of the evaluation factors can be obtained:

Equation (8) gives a new method to calculate the weight of the evaluation factor: If the nearest local weight of the first k ancestor nodes of the evaluation factor and a local weight of the evaluation factor are calculated, the weight value of the evaluation factor can be obtained. The nearest local weight of the k ancestor nodes before the evaluation factor can be calculated by the prior knowledge of domain experts, while the local weight of the subtree with the ancestor node of the k layer as the root node can be calculated by machine learning algorithm.

A detailed description of this method is given below. In this model, by referring to the Delphi method, we determine the weights of the first- and second-level indexes by consulting 15 experts by letter. Importance was rated on a five-level scale: 1 means “very unimportant”; 2 means “not important”; 3 means “important”; 4 means “more important”; and 5 means “very important.” Under the condition that the degree of coordination of opinions of all items meets the requirement (Kendall coordination coefficient >0.7), the average value assigned by each expert for each indicator is calculated as the basis for determining the weight. The average score of each indicator is divided by the average score of all indicators at the same level as the weight. Set the child nodes of node N as N1,N2,…,Nm, the scores assigned by k experts to the nodes as Li1,Li2,…,Lik, the average scores of the nodes Ni as Zi, and the weight of the nodes Ni as Wi. According to the above description,

Algorithm implementation:

Step 1: Initialization, node variable v = root, current level counting variable c = 0, add to H;

Step 2: If ((c < k) inverted v (v is not an evaluation factor))

Domain experts set the importance level for each child node of v, and calculate the nearest local weight of each child node by Equations (9) and (10). Traverse each child node of V;

Repeat step 2;

Else {under the subtree T with v as the root node, machine learning algorithm is adopted to calculate the local weight of each evaluation factor on T'}

6.1. Experimental Results

According to the analysis, the weight distribution of indicators at all levels is obtained, as shown in the table below. See Table 3

To show the results of the weights more clearly and compare the weights of each index more intuitively, the results are expressed in the form of a rectangular tree graph. See Figure 5 for details.

Figure 5

The weight distribution represented by the rectangular tree graph.

6.2. Results Analysis

Different weights represent different degrees of importance. Next, we will analyze the results from the perspective of three levels of indicator results. The first section of the analysis identifies the importance of two primary indicators. Both cognitive and emotional information are important in influencing users' evaluation of mobile health apps. The cognitive level of information (which was 0.52) was only four percentage points higher than the emotional level (which was 0.48).

The mobile health application designed for the youth group shows that we should consider not only users' cognitive needs but also their emotional needs. Therefore, on the one hand, we need to be able to cognitively provide users with information that is as they would expect it to be: accurate, reliable, easy to understand, concise, and effective in problem solving. On the other hand, we need to provide users with information that is expressed, as they would expect, in various forms according to different needs. This is a chance to design an app according to users' preferences and that provides information that indeed provides real benefits to users.

In the second part of the analysis, the importance of four secondary indicators is determined. The effects of content information on cognition, interaction information on emotion, interaction information on cognition, and content information on emotion are, in order of importance, 0.31, 0.29, 0.21, and 0.19, respectively. At the cognitive level, content information is more important than interactive information. At the emotional level, interactive information is more important than content information. Therefore, when designing mobile health applications, different indicators should be emphasized at different levels.

In the third portion of the analysis, to better understand the results, we consider it necessary to explain each indicator in detail (see in Appendix A). The importance of 20 three-level indicators is determined. These indicators, in descending order of importance, are as follows: less disruptive (0.178), security (0.109), utility (0.093), reliability (0.081), navigational (0.067), sociality (0.061), personalized (0.058), profitable (0.058), accuracy (0.053), beauty (0.046), interesting (0.035), simplicity (0.034), understandability (0.034), fault tolerance (0.023), operational (0.019), fluency (0.015), multiple performance (0.012), correlation (0.009), generality (0.009), and learnability (0.006). The weight distribution is relatively concentrated, with the weight ratio of the first 10 indicators exceeding 0.8 and that of the first 5 indicators exceeding 0.5. Therefore, when designing mobile health applications, it is necessary to pay special attention to the key indicators.

Somewhat surprisingly, user's value having fewer distractions the most. However, because young people are very averse to being interrupted by useless information, it is reasonable that having fewer distractions is such a high priority for them. Therefore, efforts should be made to ensure that there is no distracting information (advertisements, ambiguous information, etc.) in the content provided. If such information must be added because of commercial interests, we should consider the form of the advertising and try our best to add only a small amount of soft advertising.

The second most important indicator is security. Therefore, developers need to take effective measures to protect the privacy of users. Developers should reduce system vulnerabilities, improve security, and prevent the system from being breached by criminals.

The third most important indicator is utility, which provides information to solve problems faced by users and provides reliable suggestions to users. Utility is the most basic requirement for mobile health apps. If a mobile health app does not have utility, it does not matter how well the app performs in other respects. Therefore, enterprises need to integrate internal and external high-quality resources to improve service quality.

The fourth most important indicator is reliability: the source of content must be authoritative and reliable and must clearly label who is the diagnostician and whether that person has the relevant qualifications. Reliability is the basis for achieving utility. Only information provided from a reliable source can instill in users a sense of security and help them find the right professional in case of an accident.

The fifth most important indicator is navigability: the system should have a simple navigation bar or other convenient navigation mode to help users quickly find the information they need.

In this study, we also found that the weights assigned to the four secondary indicators—sociality, personalized, beauty and interesting—accounted for a significant proportion of the total weights assigned, thus exceeding our expectations. The significance of these four secondary indicators is also an important finding of this study and provides a new direction for the design of mobile health applications aimed at young people. Concerning sociality, special modules can be set up in the app to provide opportunities for users to communicate through the app and bridge the communication between users, patients, and other relevant persons. “Personalized” indicates that the service and interface of the system can be personalized and optimized according to users' preferences so that users can participate in the design of the interface and information architecture and have some control over the system. Personalization is the embodiment of young people's pursuit of individuality. “Beauty,” indicates that the system interface and information presentation mode are aesthetically rich. Meanwhile, it is also of great importance that color, font, and graphics are used in the proper proportions; beautiful formats are designed so that information can be presented through pictures, cartoons, and other interesting forms. “Interesting” indicates that the design and operation mode of the system is interesting and that the makers of the app avoided creating a rigid design and an outdated interaction mode. Makers of interesting apps appropriately adjust and update the system promptly and follow development trends. Meanwhile, the research results also show that the importance of universality and learnability is very low, thus indicating that the first generation of young people to grow up with the Internet has no difficulty using mobile health apps. Young people do well in using different apps, so they pay less attention to versatility and learnability.

APPENDIX