Comprehensive Evaluation of Transit-Oriented Development (TOD) Guidance Systems: A Case Study of Shanghai TODs

Abstract

Transit-oriented development (TOD) is a leading urban planning paradigm whose success largely depends on the effectiveness of its guidance systems. Although guidance systems have been extensively researched in various contexts, there is a significant gap in evaluating these systems within TOD scenarios, particularly regarding quantitative assessment methods. This study introduces a novel quantitative evaluation methodology to assess the various dimensions of guidance systems in TOD contexts. It identifies four primary criteria: normativity, effectiveness, continuity, and individualization, alongside 13 specific secondary evaluation indicators. The contributions of these indicators are determined through expert scoring using the analytic hierarchy process (AHP) and are subsequently validated through on-site user assessments. Data collection for TOD guidance systems involved scoring using a system usability scale (SUS) questionnaire and objective data measurements across five routes within two representative TODs in Shanghai. The results reveal a strong linear regression relationship between AHP quantitative scores and SUS questionnaire scores, validating the effectiveness of the proposed indicator framework. In summary, this study addresses the gap in evaluating TOD guidance systems and provides a comprehensive evaluation system to guide future design and development efforts in TOD scenarios.

This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

1. Introduction

Transit-oriented development (TOD) is the most representative development model in modern urban planning. It is a comprehensive complex oriented and centered around public transportation, and it simultaneously integrates commercial, office, and residential functions [1]. TOD holds numerous irreplaceable advantages in urban land utilization, sustainable development of the environment, and human-centered transportation, a typical trinity of the “human–machine–environment.“

As a new trend in urban living scenarios, TOD is a complex system requiring comprehensive consideration of interrelationships between internal elements. Consequently, the guidance system is crucial to the usability and experience of TOD. More than 80% of human spatial perception comes from visual sense [2], which forms the basis of the importance of guidance systems. A sound guidance system can reduce the entropy of the system and the complexity of the environment, allowing the system to operate methodically [3]. Also, the importance of guidance systems is reflected not only in navigation but also in esthetics. The significance of graphical visual symbols for subway guidance should be noticed for their intuitiveness and attractiveness against simple text [4].

Moreover, the personalized esthetic design of the navigation system aids in users’ orientation awareness, as research has revealed that repetitive indoor designs can result in disorientation among older people [5]. In addition, guidance systems meet people’s regular guidance needs and emergency evacuation situations. For instance, well-designed guidance systems can reduce or even avoid casualties [6]. Last but not the least, people’s demands for inclusive guidance systems are increasing, which leads to expanding their scope, such as proposing a guidance system based on auditory interfaces [7] that provided new research ideas for accessible groups and multichannel guidance systems. From the perspective of research objects, there have been studies on the evaluation system of directional guidance systems in scenes such as roads and subway stations, but there is a lack of research on guidance systems in TOD. Besides, many studies focus on architecture, esthetics, and simulation, with little research on pedestrian needs and cognition [8], which highlights the importance of human-centered design in guidance systems and reveals that guidance systems are a complicated process that combines both subjective and objective factors from the perspective of pedestrian perception. This reflects the necessity of constructing an evaluation system for guidance systems in TOD. Meanwhile, research on guidance systems in TOD has focused chiefly on empirical verification of evaluation methods, while constructing a relatively complete evaluation system is lacking.

This article proposes a quantitative evaluation method to score different dimensions of guidance systems in TOD scenarios. This evaluation system is constructed through quantitative methods and expert scoring and is validated through empirical verification in actual scenarios. To some extent, this evaluation system addresses the limitations of the current evaluation of guidance systems in TOD and provides a reference for the design and development of it. Meanwhile, Shanghai’s metro network scale ranks first in the world. Additionally, this study collected extensive field data on guidance systems from representative TODs in Shanghai, providing valuable insights for general research on TOD guidance systems.

2. Literature Review

The literature on TOD signage systems can be broadly divided into two categories: evaluation of the content elements of individual signage and evaluation of the effectiveness of linkage and collaboration among individual signage.

2.1. Study of Individual Guidance Signage

The content element of the signage itself contains many dimensions. In 1980, Braaksma and Cook [9] introduced the visibility index (VI) as a metric to evaluate the ease of guidance and orientation, offering a quantitative approach to the complex issue of guidance systems. In 2007, new evaluation metrics, such as readable distance and visual angle, were proposed and quantified using the VCA model [6]. More recent studies have developed comprehensive indicators for evaluating guidance systems, summarizing design principles into three dimensions: functionality, systematization, and unity [10]. These studies highlight the importance of conveying information through symbols, colors (CLRs), words, and geometric shapes, emphasizing utility, recognition, prominence, artistry, and accuracy [11]. However, specific quantification methods for these indicators are often lacking. Additionally, as digital intelligence becomes a design requirement, new indicators, such as technological elements, must be incorporated into the evaluation system to enhance accessibility and inclusiveness [12].

In addition to a priori quantitative indicators, some studies use a posteriori indicators to measure the effectiveness of guidance systems. These include metrics such as the time required to complete tasks and deviations from the correct angle and distance of the path [13]. However, these indicators do not provide direct guidance for design. Guide signs are also categorized into different types, such as map ID, direction guide signs, location identification, and warning restriction marks, each requiring adherence to specific design criteria based on their functions [4].

2.2. Study of Connections and Collaborations Among Guidance Signages

Regarding the linkage and collaborative utility of individual signs, Passini [14] categorized directional signs into three broad areas: sign content, location, and form. This categorization underscores the complexity of directional systems and the impact of sign location on their utility. In 2002, Hegarty et al. [15] developed the SBSOD scale to evaluate individuals’ sense of direction. Addressing the complexity of guide sign layout, Qi et al. [16] proposed an optimization model for maximizing the VI. This model is enlightening as it compiles numerous VI values, but has limitations as it does not consider the facility’s service capacity or pedestrian flow direction. Shi and Duan [17] proposed evaluating guidance signage systems based on human guidance behavior regarding content and form. However, theoretical models and constraints proposed by Chinese scholars often appear too simple, limiting the applicability of the overall scheme. Davis, Ohman, and Weisbeck [18] identified elements contributing to indoor spatial wayfinding, including cues, CLR, familiarity, and location, but did not provide specific quantitative evaluation criteria. Li and Klippel [19] explained wayfinding behavior through environmental and human factors, integrating these into the environmental legibility (EL) framework, which considers architectural visibility, connectivity, and layout complexity. Human factors are assessed through self-assessment, offering insights into transitioning from wayfinding behavior to guidance system assessment. Wang et al. [20] focused on evaluating pedestrian safety in subway stations using real-time data and a new force model to simulate pedestrian behavior and assess risks. The findings on pedestrian dynamics and risk management can inform the design and evaluation of TOD guidance systems, especially regarding the layout and management of pedestrian flows. Additionally, Umar [21] proposed four key influencing factors for the built environment planning and their corresponding subfactors: technology, organization, environment, and cost. They also provided the respective impact contributions, offering a significant reference for the quantitative evaluation system of TOD guidance systems.

Research on TOD in situ guidance systems often focuses on empirical tests of evaluation methods, but lacks comprehensive evaluation systems [17]. A study on metro station signage systems suggested classifying and assigning different weights for quantitative evaluation, but did not clarify the relationship between different signage types [22]. However, some studies have provided valuable insights into macrolevel thinking and methodologies. For instance, Asah and Blahna [23] emphasized the importance of collaborative design, highlighting the necessity of incorporating stakeholder knowledge and integrating the expertise of both experts and stakeholders in the design and assessment of tools. This approach constructively contributes to establishing a quantitative assessment system and demonstrates the reciprocal influence of user knowledge on the indicator system. Lu et al. [24] provided a comprehensive method for evaluating public information guidance systems based on four critical criteria: system performance, user experience, operational efficiency, and impact assessment.

In summary, there is a notable lack of research on evaluating guidance systems in TOD scenarios, with most studies focusing on indoor environments like subway stations and shopping malls. Existing evaluation systems are often incomplete, missing key dimensions, and lacking systematization. Subjective evaluation indices reduce interpretability, while overly objective rating systems fail to account for dimensions that are difficult to quantify, such as esthetics. Few evaluation systems effectively combine both subjective and objective measures.

3. Materials and Methods

3.1. Evaluation of TOD Guidance Systems

Based on comprehensive desktop research, offline surveys, and expert investigations, we established and differentiated the evaluation criteria and dimensions for the TOD guidance system (Table 1).

Table 1

Evaluation framework for TOD guidance system.

Categories	Evaluation factors	Formula	Reference
Normativity	Graph size	$f_{\text{GS}}=H_i/H_{G0}{H}_{G0}=25L/1000$	GB/T 15566.1-2007 [25]
	Text size	$f_{\text{TS}}=H_j/H_{T0}{H}_{T0}=0.3\times H_{G0}$	GB/T 20501.1-2013 [26]
	Color	$f_{\text{CLR}}=\alpha \times \frac{\sum _n{C}_i}{N}+\beta \times \frac{\sum _n{C}_{Li}}{N}{C}_{Li}=R_1\times E_1+0.05/R_2\times E_2+0.05\alpha +\beta =1$	GB/T 20501.1-2013 [26]CIE S 017/E:2020 [27]ISO/IEC 40500:2012 [28]
	Visibility	$f_V=E_i/E_0{E}_0=200\text{lx}$	GB 50034-2013 [29]
Effectiveness	Information accuracy	$f_{\text{IA}}=\frac{\sum _{n}I{A}_i}{N}$	Zhang et al. [30]
	Information hierarchy	$f_{\text{IH}}=\frac{\sum _{n}I{H}_i}{N}$	Zhang et al. [30]
	Unity	$f_U=\frac{\sum _n{U}_i}{N}$	Zhang et al. [30]
Continuity	Intersection guidance	$f_{\text{IG}}=\alpha \times \frac{\sum _n{G}_i}{N}+\beta \times \frac{\sum _{m}G{F}_j}{M}\alpha +\beta =1$	Luet al. [24]
	Distance recognition	$f_{\text{DR}}=\frac{\sum _n\text{Roundup}\frac{D_0}{D_{i,i-1}},0+\sum _n\text{Roundup}\frac{D_0}{D_{i,i+1}},0}{2N}$	Lu et al. [24]
Individualization	Esthetics	$f_A=\frac{\sum _n{A}_i}{N}$	Yang and Tong [4]
	Accessibility care	$f_{AC}={AC}_s/{AC}_d$	Zhang et al. [30]
	Technology	$f_T=\frac{\sum _n{T}_i}{3N}$	Umar [21]
	Culture inclusiveness	$f_{CI}=\frac{\sum _n{CI}_i}{N}$	Cao [12]

This study initially identified a comprehensive set of factors influencing the evaluation of TOD guidance systems through a literature review and field observations. However, some of these factors may have limited impact. To develop a structured core indicator system for effective TOD guidance evaluation, expert opinions from the field of built environment design were sought. Five experts were recruited for structured interviews, selected based on criteria including a minimum of 5 years’ experience in built environment design, residence in a major city, and frequent use of TOD systems, allowing them to provide insights from both expert and user perspectives. The expert panel consisted of a PhD in architecture with 8 years of design experience, two environmental interaction designers with 7 years of academic study and 1 year of work experience, an expert in interior architectural research with 7 years of experience, and a Master’s degree holder in design with 6 years of experience in spatial interaction design. The findings from these interviews informed the refinement and synthesis of the initial set of factors.

Four primary dimensions were identified: normativity (primarily based on Zhang et al.’s [30] Text Location), effectiveness (based on Zhang et al.’s [30] Text Content), continuity (referencing Lu et al.’s [24] Continuity), and individualization (synthesized from various sources, detailed in Table 1). These dimensions encompass 13 secondary evaluation indicators: graph size (GS), text size (TS), CLR, visibility, information accuracy (IA), information hierarchy (IH), unity, intersection guidance (IG), distance recognition (DR), esthetics, accessibility care (AC), technology, and cultural inclusiveness (CI).

3.1.1. Normativity of TOD Guidance System

GS serves as a metric for gauging the size of graphic information within the guidance system. Graphics constitute a fundamental mode of information conveyance in guidance systems, distinct from textual elements. Insufficient graphic size might impede user discernment, while excessive dimensions could result in resource inefficiencies. The score for this indicator is contingent upon the standard graphic size, denoted as $H_{G0}$, as stipulated in GB/T 15566.1-2007 [25]. It is determined through the relationship between empirically measured graphic size, represented as $H_i$, and the standard graphic size. In this context, $L$ signifies the maximum observational distance for graphic signage.$$\begin{array}{}& f_{\text{GS}}=H_i/H_{G0}{H}_{G0}=25L/1000.\end{array}$$

TS serves as a metric for assessing the size of textual information within the guidance system. Text represents the most direct and natural means of conveying information within guidance systems and appropriately standardized TSs contribute to the more effective fulfillment of guidance functions. The score for this indicator is derived from the standard text size, denoted as $H_{T0}$, as outlined in GB/T 20501.1-2013 [26] (correlated with the standard graphic size $H_{G0}$). The determination involves the relationship between empirically measured TS, denoted as $H_j$, and the standard TS.$$\begin{array}{}& f_{\text{TS}}=H_j/H_{T0}{H}_{T0}=0.3\times H_{G0}.\end{array}$$

CLR is an indicator for assessing CLR normativity within the guidance system. The computation of this indicator is based on the international lighting standard CIE S 017/E:2020 [27], CLR contrast specifications from WCAG 2.0 (ISO/IEC 40500) [28], and CLR usage guidelines from GB/T 20501.1-2013 [26]. Herein, $C_i$ signifies the employment of standardized CLRs, taking values of either 0 or 1 ($C_i$ = 0 or $C_i$ = 1). $C_{Li}$ represents CLR contrast and is assigned a value of 1 when 100:1 > $C_{Li}$ > 4.5:1, otherwise $C_{Li}$ = 0. $C_{Li}$ is determined by the luminance of graphic and textual CLRs, denoted as $R_1\times E_1$, and the luminance of background CLRs, denoted as $R_2\times E_2$. $N$ denotes the quantity of guidance symbols within the guidance system. In the context of the overall assessment scope of the guidance system, the coefficients α and β, respectively, represent the proportion occupied by guidance signs evaluated based on the criteria “utilization of standardized CLRs” and “CLR contrast specifications.” Hence, CLR normativity comprises a proportional aggregation of the outcomes of these two distinct indicators.$$\begin{array}{}& f_{\text{CLR}}=\alpha \times \frac{{\sum }_n{C}_i}{N}+\beta \times \frac{{\sum }_n{C}_{Li}}{N}{C}_{Li}=R_1\times E_1+0.05/R_2\times E_2+0.05,\alpha +\beta =1.\end{array}$$

Visibility (V) is a metric for assessing the normativity of illuminance on guidance signage surfaces within the guidance system. $E_0$ represents the standard illuminance for signage surfaces stipulated in GB 50034-2013 [29] (the unit is lux). $E_i$ signifies the measured illuminance on the working surface of guidance signage.$$\begin{array}{}& f_V=\frac{E_i}{E_0}{E}_0=200lx.\end{array}$$

3.1.2. Effectiveness of TOD Guidance System

IA is a metric used to gauge the accuracy of conveying guidance information within the guidance system, including graphics, text, and semantics. ${\text{IA}}_i$ represents whether the information conveyed by the i-th guidance sign is accurate, taking values of ${\text{IA}}_i=0$ or ${\text{IA}}_i=1$. $N$ denotes the total number of guidance sign points. The IA of the system is defined as the ratio of accurately conveying guidance signs within the system to the overall total.$$\begin{array}{}& f_{\text{IA}}=\frac{{\sum }_n{\text{IA}}_i}{N}.\end{array}$$

IH is an indicator used to measure the presence of a clear IH within the guidance system. ${\text{IH}}_i$ represents whether the i-th guidance sign has a distinct IH, taking values of ${\text{IH}}_i$ = 0 or ${\text{IH}}_i$ = 1. $N$ denotes the total number of guidance sign points. The IH of the system is defined as the ratio of guidance signs within the system that possess a clear IH to the overall total.$$\begin{array}{}& f_{\text{IH}}=\frac{{\sum }_n{\text{IH}}_i}{N}.\end{array}$$

Unity (U) is a metric used to assess the degree of consistency among various guidance signs within the guidance system. $U_i$ represents whether the i-th sign maintains uniformity with other elements of the guidance system, taking values of $U_i$ = 0 or $U_i$ = 1. $N$ denotes the total number of guidance sign points. The uniformity of the system is defined as the ratio of guidance signs within the system that adhere to uniformity criteria to the overall total.$$\begin{array}{}& f_U=\frac{{\sum }_n{U}_i}{N}.\end{array}$$

3.1.3. Continuity of TOD Guidance System

IG is an indicator used to measure the guidance functionality at entry and exit nodes within the TOD guidance system. $G_i$ represents whether there is guiding information at regular entry and exit points, taking values of $G_i$ = 1 or $G_i$ = 0. N denotes the total number of regular entry and exit points. ${\text{GF}}_j$ signifies whether there is an overview map at high-traffic entry and exit points, with values of GF_j = 1 or GF_j = 0, representing the total number of high-traffic entry and exit points. In this context, the coefficients α and β, respectively, represent the proportion occupied by regular entry and exit points and high-traffic entry and exit points within the overall guidance system. Hence, multinode guidance involves a proportional aggregation of the outcomes of these two distinct indicators.$$\begin{array}{}& f_{\text{IG}}=\alpha \times \frac{{\sum }_n{G}_i}{N}+\beta \times \frac{{\sum }_m{\text{GF}}_j}{M}\alpha +\beta =1.\end{array}$$

DR is a metric that measures the distance between adjacent signs within the guidance system. $D_0$ represents the maximum distance between two adjacent signs. $D_{i,i-1}$ represents the distance between the i-th sign and the adjacent upstream sign, while $D_{i,i+1}$ represents the distance between the i-th sign and the adjacent downstream sign. $N$ denotes the total number of guidance sign points (Calculation is rounded to the nearest integer).$$\begin{array}{}& f_{\text{DR}}=\frac{{\sum }_n\text{Roundup}\frac{D_0}{D_{i,i-1}},0+{\sum }_n\text{Roundup}\frac{D_0}{D_{i,i+1}},0}{2N}.\end{array}$$

3.1.4. Individualization of TOD Guidance System

Esthetics (A) is a metric used to evaluate whether elements such as graphics, text, CLR, and materials within the guidance system are esthetically designed, in harmony with the environment, and responsive to the esthetic preferences of different user groups. $A_i$ represents whether the i-th guidance sign is esthetically pleasing, taking values of $A_i$ = 0 or $A_i$ = 1. $N$ denotes the total number of guidance sign points. The system’s esthetics is defined as the ratio of guidance signs within the system that meet esthetic criteria to the overall total.$$\begin{array}{}& f_A=\frac{{\sum }_n{A}_i}{N}.\end{array}$$

AC is an indicator used to assess the level of accessibility within the guidance system. ${\text{AC}}_d$ represents the expected quantity of accessibility signs to be provided, while ${\text{AC}}_s$ represents the actual quantity of accessibility signs provided.$$\begin{array}{}& f_{\text{AC}}={\text{AC}}_s/{\text{AC}}_d.\end{array}$$

Technology (T) is a metric used to assess the extent of technological and multimodal utilization within the guidance system. $T_i$ represents the value of the weight of scientific elements, such as visual, auditory, tactile, and digital, at the i-th node. $N$ denotes the total number of guidance sign points. The scientific elements of the system are defined as the ratio of guidance signs within the system that fulfill scientific element criteria to the overall total.$$\begin{array}{}& f_T=\frac{{\sum }_n{T}_i}{3N}.\end{array}$$

CI is a metric used to assess whether the language usage and elements, such as graphic symbols within the guidance system, take into account diverse cultural backgrounds. ${\text{CI}}_i$represents whether the i-th guidance sign possesses cultural inclusivity, taking values of ${\text{CI}}_i=0$ or ${\text{CI}}_i=1$. $N$ denotes the total number of guidance sign points. The cultural inclusivity of the system is defined as the ratio of guidance signs within the system that exhibit cultural inclusivity to the overall total.$$\begin{array}{}& f_{\text{CI}}=\frac{{\sum }_n{\text{CI}}_i}{N}.\end{array}$$

3.1.5. Method to Determine Contribution of Index in the Evaluation System

Based on the opinions gathered from participants in user interviews and expert surveys, we conducted a scoring evaluation of the proposed evaluation framework. We analyzed the expert ratings using the analytic hierarchy process (AHP) to determine the contributions of both primary indicators (Table 2) and secondary indicators (Table 3).

Table 2

Contribution of primary evaluation indicators.

Primary evaluation indicators	Contribution
Normativity	0.2677
Effectiveness	0.1495
Continuity	0.5198
Individualization	0.0630

Table 3

Contribution of secondary evaluation indicators.

Indicator serial number	Secondary evaluation indicators	Contribution
1	Graph size	0.0446
2	Text size	0.0446
3	Color	0.0892
4	Visibility	0.0892
5	Information accuracy	0.0912
6	Information hierarchy	0.0247
7	Unity	0.0336
8	Intersection guidance	0.1733
9	Distance recognition	0.3465
10	Esthetics	0.0120
11	Accessibility care	0.0294
12	Technology	0.0108
13	Culture inclusiveness	0.0108

The application of the AHP in this study is well-justified due to its ability to address the challenges posed by multidimensional and heterogeneous data across primary and secondary indicators, where single-dimensional analytical methods fall short. By enabling the integration of diverse dimensions, AHP provides a robust framework for analysis, although it does not entirely eliminate subjective bias. Furthermore, its widespread application in related fields, such as TOD and guidance system design, underscores its relevance and reliability. In this study, the AHP analysis was conducted with the involvement of domain experts, ensuring a high degree of authority and professional rigor. Notably, the study introduces innovations by quantifying the evaluation indicators to minimize subjectivity and by developing a comprehensive and highly structured indicator system that incorporates key influencing factors, enhancing both the objectivity and applicability of the findings.

Based on the AHP analysis results from Tables 2 and 3, we can draw the following preliminary conclusions:

1. Significance of continuity: Among the four primary dimensions, continuity is the most significant factor, accounting for over 50% of the overall importance. This underscores the necessity of prioritizing coherence between guidance elements in the design and planning of the TOD guidance system. For example, maintaining appropriate spacing distances and incorporating comprehensive guidance facilities at nodes are crucial considerations. Neglecting continuity and relying solely on the effectiveness and normativity of individual guidance elements within the system may not necessarily result in effective user guidance.

2. Key secondary indicators: Among the 13 secondary indicators, DR (34.65%) and IG (17.33%) are identified as the most pivotal factors, corroborating the credibility of the primary indicators’ contribution analysis. Following these, IA (9.12%), visibility (8.92%), and CLR (8.92%) are also significant. This highlights that the accuracy of information provided by individual guidance elements is a fundamental requirement for the guidance system. Additionally, the efficacy of guiding users depends not only on the accuracy of the information but also on objective physical factors such as visibility and CLR.

3.2. Data Collection

Based on the conventional scale evaluation approach, we devised a system usability scale (SUS) questionnaire tailored for the TOD guidance system to gather users’ subjective assessments. The SUS is a standard tool that allows users to assess the usability of a specific product or service, with strong flexibility, validity, and reliability, having inclusivity of users from different backgrounds [31, 32]. The SUS is designed to generate a scale score ranging from 0 to 100 and this score can explain the perceived usability of the system. The findings of existing studies have shown that SUS has excellent reliability and concurrent validity comparing with other measures of perceived and objective usability [31], which fulfills the specified requirements of our study.

This questionnaire comprises 12 questions, employing a 5-point scale, where 1 signifies strong disagreement with the statement and 5 indicates strong agreement. To enhance the objectivity of user assessments of the system, we have interspersed both positively and negatively phrased questions alternately. This approach allows for a more balanced comparison with our developed objective indicators, as shown in Table 4.

Table 4

SUS for TOD wayfinding.

Item number	Question	Corresponding secondary indicator
1	I believe that the graph and text sizes in this guidance system are either too large or too small	Graph/text size
2	I believe the color scheme of the guidance system is appropriately configured	Color
3	I perceive low visibility (working surface illuminance) in the wayfinding system	Visibility
4	I believe that the information conveyed by the guidance system is accurate	Information accuracy
5	I believe that the information hierarchy in this guidance system is clear and reasonable	Information hierarchy
6	I believe that there is a significant amount of inconsistency in this guidance system	Unity
7	I believe there is a lack of guidance information at the entrances and exits of the guidance system	Intersection guidance
8	I believe the distance between the guidance signs in the guidance system is very appropriate	Distance recognition
9	I find this guidance system to be very esthetically unpleasing	Esthetics
10	I believe the guidance system exhibits a high level of accessibility	Accessibility care
11	I believe there is poor coordination among various modalities in the guidance system	Technology
12	I believe the guidance system demonstrates good cultural inclusiveness	Culture inclusiveness

We selected two representative TODs in Shanghai: Xinzhuang TOD (specialized type) and Xujiahui TOD (urban type). Utilizing five distinct routes, we formulated five navigation tasks and constructed a travel chain matrix (Table 5). Subsequently, a cohort of 30 participants was recruited to individually complete the five tasks (Figure 1(a)), providing SUS ratings for each route after testing. Each participant was instructed to complete tasks on two routes, selected from five distinct routes. Simultaneously, objective indicator scores were obtained through on-site measurements as input to the objective indicator system (Figure 1(b)).To eliminate potential interference from the participant’s familiarity with the environment, we recruited participants with the requirement that they be as unfamiliar as possible with the Xinzhuang TOD and Xujiahui TOD. Most of them had not visited these two TODs before.

[figure(s) omitted; refer to PDF]

Table 5

Travel chain matrix.

Test identification number	TOD	Guidance task	Navigation type	Subject number
01	Xinzhuang	Take the subway from Xinzhuang commercial district to Xujiahui	Conventional	21
02	Xinzhuang	Take Line 1 and disembark at Xinzhuang station, requiring urgent evacuation	Evacuation	8
03	Xujiahui	Travel on Line 1 from Xinzhuang to Xujiahui and disembark, proceeding to the commercial district	Conventional	13
04	Xujiahui	Tourists visiting Xujiahui are required to make an emergency transfer to Line 11	Emergency	8
05	Xujiahui	Purchase a bottle of water from a convenience store in the ground-level commercial district	Recreational	8

To minimize potential bias in the results, this study recruited a diverse range of TOD users, with an equal gender ratio (1:1) and an age range of 18–50. The route design adhered to the following standards: First, two representative TODs were selected, one representing a city hub TOD and the other a suburban community TOD [33]. Second, the route design for the experiments incorporated widely applicable TOD scenarios, encompassing diverse environments and use cases, forming a comprehensive travel chain matrix.

4. Results and Discussion

4.1. Results

The average scores of the 12 questions within the SUS from participants across the five routes are as follows: It can be observed that the scores for Questions 1, 5, and 9 are generally low, while the scores for Questions 2 and 4 are consistently high (Figure 2).

[figure(s) omitted; refer to PDF]

The results of the 13 quantitative calculations based on the field measurements for each route are presented below. Visualizations indicate that the performance of the five routes is generally poor in indicators 2, 8, and 12, while it is generally good in indicators 3, 9, 10, and 13 (Figure 3).

[figure(s) omitted; refer to PDF]

4.2. Discussion

As shown in Figure 4, the overall scores for the five routes can be obtained after converting the scores from the SUS scale using the scoring formula and performing AHP-weighted calculations. The comparative table of SUS scores and AHP weighted scores for the five route guidance systems indicates a general consistency between the quantitative indicator framework and the SUS evaluation regarding the overall assessment outcomes.

[figure(s) omitted; refer to PDF]

As shown in Table 6 and Figure 5, a linear regression analysis was performed to fit the AHP quantitative indicators against the SUS scores for the five routes, yielding a correlation coefficient of 0.928, an R² value of 0.862, and a significance F value of 0.816. The linear regression relationship between AHP quantitative scores and SUS questionnaire scores demonstrates a strong fit, providing evidence that the indicator framework established in this study effectively evaluates the TOD wayfinding system.

[figure(s) omitted; refer to PDF]

Table 6

Linear regression statistics.

Multiple R	R2	Significance F
0.92837547	0.86188101	0.81584135

Figure 6 illustrates notable performance score discrepancies observed for visibility, IH, esthetics, and scientific elements. The above indicators exhibit substantial variations between the standard values and participants’ subjective perceptions. Illuminance might conform to the standard values within visibility, but during participants’ perceptual process, it can be influenced by factors such as surface reflectance coefficients. The quantified requirements for IH necessitate clear gradations for every signage point under ideal circumstances, yet participants may not pay equal attention to the hierarchical representation of each point. Perceptions of esthetics are inherently subjective and are subject to the same issues as IH. In the case of scientific elements, the quantified criteria stipulate the presence of visual, auditory, and digital elements at each signage point under ideal conditions; however, in practice, the absence of one element might not necessarily impact participants’ perception of wayfinding information.

[figure(s) omitted; refer to PDF]

4.3. Implications

The design and evaluation of TOD guidance systems, using Shanghai’s TODs as a case study, provide several valuable insights. First, the study emphasizes the critical role of continuity in TOD guidance systems, as demonstrated by the quantitative validation of the proposed indicator system. Among the four primary evaluation criteria, continuity holds the highest weight in the AHP analysis (0.5198), with IG and DR identified as the most influential secondary indicators. This finding suggests that continuity, more than normativity, effectiveness, or individualization, should be prioritized in the design of TOD guidance systems. This emphasis reflects the inherent complexity of TOD systems, where macrolevel integration and coordination significantly influence user wayfinding experiences, often outweighing the impact of isolated design elements. Nevertheless, the importance of addressing normativity, effectiveness, and individualization in design processes remains integral.

Second, the study highlights the comprehensiveness of the evaluation dimensions within the proposed indicator system, which captures a wide array of factors impacting user wayfinding experiences. The system incorporates both objectively measurable physical indicators (secondary indicators 1, 2, 3, 4, 8, and 9) and less quantifiable and experience-driven indicators (secondary indicators 5, 6, 7, 10, and 12). By integrating both subjective and objective dimensions, the system offers a holistic evaluation framework, surpassing conventional rating scales. Additionally, the inclusion and quantification of inclusive design principles (secondary indicators 11 and 13) ensure that the guidance systems address the accessibility and inclusiveness needs of diverse user groups, including those with special requirements.

Last, the study underscores the significance of adopting a combined subjective–objective evaluation approach. By integrating traditional subjective usability assessments with the validated quantitative indicator system, this approach enables a comprehensive understanding of the performance of TOD guidance systems. Such a dual-method evaluation framework provides more robust insights, facilitating the refinement of design strategies and enhancing the overall effectiveness of TOD guidance systems.

4.4. Limitations and Future Research Directions

While this study has made some progress in evaluating TOD guidance systems, certain limitations can be observed. For instance, this study did not delve deeply into the evaluation criteria for dynamic guidance systems. Although the secondary indicator, “technology,” includes considerations regarding the presence of dynamic wayfinding signage, it does not further elucidate the quality of their design. This is primarily because, despite the widespread use of dynamic guidance systems in large transportation hubs like airports and train stations, they are not yet ubiquitous in TODs primarily centered around subway transportation.

Furthermore, the generalizability of the indicator system in this study needs further validation. Due to experimental constraints, this study focused on two of the most representative TODs in Shanghai and selected five routes within them. Therefore, the effectiveness of this indicator system in other TOD scenarios could be further validated in future research, possibly employing cost-effective technologies such as VR to assess its generalizability in typical TODs. Besides, the study primarily involved the general population, tourists, and some members of the disabled community. Future research could include a more diverse range of participants, such as individuals with cognitive impairments, the elderly, and children, to make the evaluation results more inclusive.

Future research directions in this study could further enhance the development of TOD guidance systems and their evaluation methods. One potential direction is to adaptively adjust the weights of indicators within the system to reflect their contributions more accurately, tailoring the system to specific TOD scenarios. For example, in urban TOD environments characterized by high traffic flow and numerous nodes, it may be appropriate to increase the weights assigned to secondary indicators 8 and 9 to assess system performance in such contexts better.

Another potential avenue for advancing this research is providing route guidance strategies for specific TOD scenarios based on a quantified indicator system and user cognition. This may involve understanding users’ cognitive loads while navigating TOD environments. Future research could employ eye-tracking and cognitive psychology methodologies to gain deeper insights into how users process information and make decisions within TOD guidance systems. By integrating the indicator system with user cognition, a more comprehensive explanation of wayfinding mechanisms within TOD can be achieved, leading to the development of corresponding route guidance strategies.

5. Conclusions

To summarize, this study aims to establish a quantitative evaluation framework for the TOD guidance system and validate its effectiveness. Its theoretical value is reflected across a broad spectrum of performance indicators, providing valuable insights for the evaluation of guidance systems not only in TOD but also in other contexts. It furnishes a specific quantitative metric system at the application level, offering guidance system designers in the TOD domain a valuable reference and testing tool. Importantly, it transcends the limitations of relying primarily on subjective scales as the primary evaluation method, thus, contributing to an understanding of the effectiveness of TOD guidance systems.

Furthermore, this research indicates a need for further investigation into routing strategies and the generalizability of metric systems to other TOD scenarios. In conclusion, this study demonstrates that a quantitative metric system can effectively evaluate the design of TOD guidance systems and inspire further exploration into their underlying routing mechanisms.

Acknowledgments

The utmost gratitude is extended to Xujiahui TOD and Xinzhuang TOD for permitting us to conduct internal experiments and for the assistance of their staff during the experiments. The authors also thank Xiongjie Yang, Hui Yu, Ze Chen, Haowei Zhang, and Wei Gao for their contributions to this research. The original draft of this study was manually composed by the authors, subsequently refined through ChatGPT for English language polishing, and followed by additional manual verification. Acknowledgement is, hereby, extended to ChatGPT for its auxiliary support in enhancing the linguistic quality of this work.

References

[1] Q. Liu, Z. Kang, H. Zhang, R. Song, G. Li, "A Case Study on Annoyance Noise Caused by Metro Railway at a TOD Developed Depot," Advances in Civil Engineering, vol. 2022 no. 1,DOI: 10.1155/2022/3173567, 2022.

[2] E. H. Chapman, "The Image of the City by Kevin Lynch," The Journal of Aesthetics and Art Criticism, vol. 21 no. 1, pp. 91-91, DOI: 10.2307/427643, 1962.

[3] R. K. Dubey, T. Thrash, M. Kapadia, C. Hoelscher, V. R. Schinazi, "Information Theoretic Model to Simulate Agent-Signage Interaction for Wayfinding," Cognitive Computation, vol. 13 no. 1, pp. 189-206, DOI: 10.1007/s12559-019-09689-1, 2021.

[4] S. Yang, C. Tong, "Application of Graphic Language in Metro Visual Guidance System in Intelligent Environment," Application of Intelligent Systems in Multi-Modal Information Analytics, 2022.

[5] M. O’Malley, A. Innes, J. M. Wiener, "(Dis)orientation and Design Preferences Within an Unfamiliar Care Environment: A Content Analysis of Older Adults’ Qualitative Reports After Route Learning," Environment and Behavior, vol. 54 no. 1, pp. 116-142, DOI: 10.1177/0013916520953148, 2022.

[6] H. Xie, L. Filippidis, S. Gwynne, E. R. Galea, D. Blackshields, P. J. Lawrence, "Signage Legibility Distances as a Function of Observation Angle," Journal of Fire Protection Engineering, vol. 17 no. 1, pp. 41-64, DOI: 10.1177/1042391507064025, 2007.

[7] G. Uzan, P. Wagstaff, "Audio-Based Interface of Guidance Systems for the Visually Impaired in the Paris Metro," HCI in Mobility, Transport, and Automotive Systems,DOI: 10.1007/978-3-030-78358-7_39, 2021.

[8] B. Lei, J. Xu, M. Li, "Enhancing Role of Guiding Signs Setting in Metro Stations With Incorporation of Microscopic Behavior of Pedestrians," Sustainability, vol. 11 no. 21,DOI: 10.3390/su11216109, 2019.

[9] J. P. Braaksma, W. J. Cook, "Human Orientation in Transportation Terminals," Transportation Engineering Journal of ASCE, vol. 106 no. 2, pp. 189-203, DOI: 10.1061/TPEJAN.0000845, 1980.

[10] L. Wang, R. Wang, X. Jiang, "Analysis of Wayfinding System Design in Commercial Spaces," Packaging Engineering, vol. 31 no. 2, pp. 121-123, DOI: 10.19554/j.cnki.1001-3563.2010.02.040, 2010.

[11] W. Xiao, "Study on Symbolic Elements in Wayfinding Systems," Packaging Engineering, vol. 33 no. 20, pp. 136-139, DOI: 10.19554/j.cnki.1001-3563.2012.20.036, 2012.

[12] X. Cao, "Interactive Design Considerations in the Wayfinding System of the Beijing Subway," Packaging Engineering, vol. 35 no. 6, pp. 37-40, DOI: 10.19554/j.cnki.1001-3563.2014.06.010, 2014.

[13] J. Grübel, T. Thrash, C. Hölscher, V. R. Schinazi, "Evaluation of a Conceptual Framework for Predicting Navigation Performance in Virtual Reality," PLOS ONE, vol. 12 no. 9,DOI: 10.1371/journal.pone.0184682, 2017.

[14] R. Passini, "Wayfinding Design: Logic, Application and Some Thoughts on Universality," Design Studies, vol. 17 no. 3, pp. 319-331, DOI: 10.1016/0142-694X(96)00001-4, 1996.

[15] M. Hegarty, A. E. Richardson, D. R. Montello, K. Lovelace, I. Subbiah, "Development of a Self-Report Measure of Environmental Spatial Ability," Intelligence, vol. 30 no. 5, pp. 425-447, DOI: 10.1016/S0160-2896(02)00116-2, 2002.

[16] E. Qi, Q. Duan, Z. Bian, Y. Shi, "VI-Based Optimization Model of Layout of Guidance Sign in Multilayer Space," Computational and Mathematical Methods in Medicine, vol. 21, pp. 2528-2534, DOI: 10.13196/j.cims.2015.09.029, 2015.

[17] Y. Shi, Q. Duan, "Review of Research Progress on the Layout of Guidance Signage Systems Based on Human Wayfinding Behavior," China Transportation Review, vol. 03, pp. 47-56, 2016.

[18] R. Davis, J. M. Ohman, C. Weisbeck, "Salient Cues and Wayfinding in Alzheimer’s Disease Within a Virtual Senior Residence," Environment and Behavior, vol. 49 no. 9, pp. 1038-1065, DOI: 10.1177/0013916516677341, 2017.

[19] R. Li, A. Klippel, "Wayfinding Behaviors in Complex Buildings: The Impact of Environmental Legibility and Familiarity," Environment and Behavior, vol. 48 no. 3, pp. 482-510, DOI: 10.1177/0013916514550243, 2016.

[20] X. Wang, Z. Zhang, Y. Wang, J. Yang, L. Lu, "A Study on Safety Evaluation of Pedestrian Flows Based on Partial Impact Dynamics by Real-Time Data in Subway Stations," Sustainability, vol. 14 no. 16,DOI: 10.3390/su141610328, 2022.

[21] T. Umar, "Key Factors Influencing the Implementation of Three-Dimensional Printing in Construction," Proceedings of the Institution of Civil Engineers - Management, Procurement and Law, vol. 174 no. 3, pp. 104-117, DOI: 10.1680/jmapl.19.00029, 2021.

[22] C. Y. Zou, G. Wang, Y. Chen, "Study on Evaluation of Subway Passenger Wayfinding Systems Based on SEM," HCI in Mobility, Transport, and Automotive Systems. HCII 2019. Lecture Notes in Computer Science, vol. 11596,DOI: 10.1007/978-3-030-22666-4_34, 2019.

[23] S. T. Asah, D. J. Blahna, "Involving Stakeholders’ Knowledge in Co-Designing Social Valuations of Biodiversity and Ecosystem Services: Implications for Decision-Making," Ecosystems, vol. 23 no. 2, pp. 324-337, DOI: 10.1007/s10021-019-00405-6, 2020.

[24] L. Lu, Z. Gu, D. Huang, "An Evaluation Framework for the Public Information Guidance System," KSCE Journal of Civil Engineering, vol. 21 no. 5, pp. 1919-1928, DOI: 10.1007/s12205-016-0888-4, 2017.

[25] General Administration of Quality Supervision, Inspection and Quarantine of the People’s Republic of China (AQSIQ), Standardization Administration of China (SAC), "GB/T 15566.1-2007. Guidance System for Public Information—Setting Principles and Requirements—Part 1: General Principles," 2007.

[26] General Administration of Quality Supervision, Inspection and Quarantine of the People’s Republic of China (AQSIQ), Standardization Administration of China (SAC), "GB/T 20501.1-2013: Guidance System for Public Information—Setting Principles and Requirements—Part 1: General Principles," 2013.

[27] CIE, "CIE S 017/E:2020: Technical Report," ,DOI: 10.25039/S017.2020, 2020.

[28] ISO, "ISO/IEC 40500: 2012: Information Technology—W3C Web Content Accessibility Guidelines (WCAG) 2.0," 2012. https://www.iso.org/standard/58625.html

[29] Ministry of Construction of the People’s Republic of China, China Architecture & Building Press, "GB 50034-2013: Standard for Lighting Design of Buildings," 2013. http://www.hyopto.com/upfile/PDF/GB%2050034-2013%20%E5%BB%BA%E7%AD%91%E7%85%A7%E6%98%8E%E8%AE%BE%E8%AE%A1%E6%A0%87%E5%87%86.pdf

[30] W. Zhang, L. Zhu, Z. Zhang, Z. Zhang, L. Lu, "A Sustainable Evaluation Method for a Tourism Public Wayfinding System: A Case Study of Shanghai Disneyland Resort," Sustainability, vol. 12 no. 20,DOI: 10.3390/su12208489, 2020.

[31] J. R. Lewis, "The System Usability Scale: Past, Present, and Future," International Journal of Human–Computer Interaction, vol. 34 no. 7, pp. 577-590, DOI: 10.1080/10447318.2018.1455307, 2018.

[32] P. Kortum, C. Z. Acemyan, "The Impact of Geographic Location on the Subjective Assessment of System Usability," International Journal of Human–Computer Interaction, vol. 35 no. 2, pp. 123-130, DOI: 10.1080/10447318.2018.1437865, 2018.

[33] H. Pan, S. Sun, Q. Zhu, "Urban Rail Transit and TOD Construction Practice in China," Architectural Practice, vol. 09, pp. 178-187, 2019.