Performance of International Global Models and Official Tropical Cyclone Forecasts Over the Bien Dong Sea for the Period 2012–2019

Abstract

This research validates tropical cyclone (TC) forecasts from the National Centre for Hydro-Meteorological Forecasting of Vietnam, the Regional Specialized Meteorological Center (RSMC) Tokyo, and global models (the Japanese Meteorological Agency [JMA], global spectral model [GSM], the National Centers for Environmental Prediction [NCEP], Global Forecast System [GFS], and the European Centre for Medium-Range Weather Forecasts [ECMWF] Integrated Forecasting System [IFS]) for 72 activated TCs over the Bien Dong Sea. The best quality track forecasts come from the IFS and then GFS and GSM, while the GFS provides a good intensity forecast compared to IFS and GSM. The official guidance and model track forecast tend to be slower and leftward compared to observations (negative biases of along and cross-track errors). Regarding the ECMWF’s ensemble track forecast, the strike probability forecast is quite overforecast/overconfident, especially for a lead time of 72 h and at high probabilities.

This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

1. Introduction

With over 3000 km of coastline, Vietnam is strongly affected by tropical cyclones (TCs) from the Bien Dong Sea (also called the South China Sea) and the western North Pacific (WNP) [1–4]. The main effects of TCs on Vietnam include heavy rainfall, strong winds, and storm surges [5–7]. Annually, about 30 TCs originate in WNP, and approximately 10 move over the Bien Dong Sea; between 4 and 6 of these TCs have a direct impact on Vietnam’s coastline and significantly affect its society and economic activities [2, 4, 7]. The main season in which TCs directly affect or achieve landfall to the north of Vietnam starts in May or June; then, the effect of the TCs tends to shift to the south of Vietnam until December. August and September are the months when TCs most frequently occur over the Bien Dong Sea [2].

The TC track and intensity forecasts are mainly based on numerical weather predictions (NWPs), and the method has shown a significant improvement, especially for TC movement forecasts [8]. The improvements of NWPs come from the following: (i) the increase in computational power allows the global-scale NWPs to run at very high horizontal resolutions (5–10 km) and (ii) the increasing number of satellite observations and the ability to absorb these data using advanced data assimilation techniques. NWPs can now provide TC track forecasts that are highly consistent with observations, and the means of TC forecast positional errors for WNP, the North Atlantic, and the north Indian Ocean (NIO) are approximately 100–120 km at 48 h lead time in the last 10 years [1, 9]. Regarding TC intensity, forecasting abilities have developed slowly to date for many reasons, especially for the forecast range of 3–5 days (e.g., representing initial TC structures [10–12], ocean/atmosphere surface fluxes exchanges [13, 14], and predictability of NWPs [15]). According to recent validation with different global models, for WNP, the mean absolute error (MAE) of maximum wind speed at the lead times of 24 , 48 , and 72 h varies from 7–14 m/s, and compared to a reference climate model, the skill scores are very low for almost NWPs [16].

In operational applications, besides the dynamic approach that uses NWPs, the statistical approach and a combination of the statistical and dynamic approaches are used to forecast TC intensity [17–19], for example, for hurricane (namely, for strong TCs that occur in the Atlantic Ocean) forecasting at the National Hurricane Center (NHC) and the Central Pacific Hurricane Center (CPHC) of the US National Weather Service, besides using various NWPs from global scale (National Centers for Environmental Prediction [NCEP] and European Centre for Medium-Range Weather Forecasts [ECMWF]) to hurricane-specialized models (Hurricane Weather Research and Forecasting Model [HWRF]), many statistical-based guidance models for enhancing intensity forecast (Statistical Hurricane Intensity Prediction Scheme, Decay—Statistical Hurricane Intensity Prediction Scheme) and for probabilistic estimates of rapid intensification over the Atlantic and the Central Pacific regions [20]. The Regional Specialized Meteorological Center (RSMC) New Delhi–India Meteorological Department (IMD), responsible for TC monitoring and prediction over NIO, uses the IMD Global Forecast System [GFS] and HWRF in TC forecasting [21]. The RSMC Tokyo–Japanese Meteorological Agency (JMA) provides forecast information for WNP (including the Bien Dong Sea) based on the global spectral model (GSM), the Typhoon Ensemble Prediction System (TEPS) and Typhoon Intensity Forecasting scheme based on SHIPS (TIFS) [22].

In operational TC forecasting, skill information from both NWPs and the international TC forecasting guidance allows forecasters to make appropriate adjustments when making final decisions regarding official warnings. Official TC forecasts are an important initial basis for implementing disaster prevention activities. Given the importance of this information, in this research, we focus on TC forecasting, verifying the main NWPs used in the Vietnam Meteorological and Hydrological Administration (VNMHA). In addition, the forecasting errors of both JMA’s guidance and the VNMHA’s official bulletin are also evaluated. In Vietnam, the most important forecast lead time is up to 3 days, which is a suitable period for the timely implementation of activities related to natural disaster preparedness and response. Therefore, this study focuses only on the forecast ranges of NWPs and guidance for up to 3 days.

The remainder of the article is organized as follows. In the next section, we provide descriptions of the forecast and observation datasets, overviews of the global NWPs, official TC bulletins, and best tracks. Section 3 describes the methodology used to derive TC-related information from NWPs in VNHMA and the verification methodology. Section 4 shows the results of both the deterministic and ensemble verifications. Section 5 presents concluding remarks.

2. TC Guidance, the Best Track Dataset, and Global Model Forecasts

2.1. TC Guidance and the Best Track Dataset From RSMC Tokyo

Under the framework of the World Meteorological Organization, since 1989, the the Tokyo Typhoon Center has been chosen as an RSMC for issuing TC advisories in WNP [23–26]. The main warning information includes the TC’s center position, its direction and speed of movement, the maximum sustained wind speed, the maximum gust wind speed, and the accuracies of determination of the center position. All information can be found on the JMA’s website (https://www.jma.go.jp/jma/jma-eng/jma-center/rsmc-hp-pub-eg/RSMC_HP.htm). Since 2020, in addition to the TC advisories provided on the JMA’s website, VNMHA has been able to obtain the information in advance from RSMC via its communication function. More information can be found in the Annual Report on the Activities of the RSMC Tokyo Typhoon Center 2019. In this study, the TC forecast guidance from the RSMC is simply referred to as JMA.

The best track data for a TC are normally finalized 1 or 2 months after the TC’s dissipation and includes the TC’s center position, maximum sustained wind speed, and and some other related information about TCs. All additional available data, including information received later such as surface observations, products of geostationary and polar-orbiting satellites, and NWP output, will be fully utilized, and then reanalyzing procedures based on Dvorak analysis will be carried out for the whole TC’s life cycle [25]. The best track information is archived on the RSMC website (http://www.jma.go.jp/jma/jma-eng/jma-center/rsmc-hp-pub-eg/trackarchives.html).

2.2. TC Guidance From the National Centre for Hydro-Meteorological Forecasting (NCHMF)

Under the VNMHA, the NCHMF is responsible for TC monitoring and predictions over the Bien Dong Sea (west of 120°E; from 5°N to 23°N). The TC bulletins are issued by NCHMF when a TC is likely to form/genesis in the Bien Dong Sea or enter the Bien Dong Sea between 24 and 48 h in advance. Depending on each TC situation, the TC bulletin is issued either eight times per day or four times per day. In cases when tropical storms or tropical depressions make landfall within 48 h or 24 h, respectively, the TC bulletin is issued eight times per day; otherwise, it is issued four times per day.

In TC monitoring operations, the NCHMF mostly uses the ensemble forecasting products of the global medium-range ensemble forecasts of TC activity, including the genesis and the subsequent track. Moreover, the global NWP forecast for TC activity is also referred to at the NCHMF. Some of the main related procedures are as follows: (i) using the Dvorak technique to determine the TC’s intensity and comparing the estimates with those from other TC centers, including the Joint Typhoon Warning Center (JTWC), the RSMC Tokyo, the China Meteorological Administration (CMA), and the Hong Kong Observatory (HKO); (ii) adjusting the TC features in cases where they fall under Vietnamese radar coverage areas and conducting additional surface observations in coastal areas; (iii) conducting TC forecasts using both deterministic and ensemble forecasts from the GSM, GFS, and the ECMWF Integrated Forecasting System (IFS), see an example in Figure 1; and (iv) determining the warning areas based on the most recent 5 to 10 years of position errors from the NCHMF forecast, in combination with strike maps from ensemble forecasts. The first/last official VNMHA warnings, especially relating to TC’s intensity, can often be earlier/later than the JMA’s guidance, especially in coastal areas or landfall cases where there is good support from additional surface and radar observations and regional models. The details of how models and international center forecasts are used to generate the official TC forecasts of the NCHMF can be found in [27].

[figure(s) omitted; refer to PDF]

2.3. Global Model Forecast Data

Operational forecasting at VNMHA is mainly based on the JMA’s GSM, the NCEP’s GFS (hereinafter referred to as GFS), and the ECMWF IFS (https://www.ecmwf.int/en/publications/ifs-documentation). The VNMHA downloads GFS and GSM full-level forecast data with a resolution of 0.5° and forecast ranges up to 10 days from the NCEP’s FTP server (ftp://ftpprd.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/) and the JMA’s server (https://www.wis-jma.go.jp). Since 2011, the VNMHA has been licensed to use the ECMWF data for national severe weather hazard prevention. Every day, the ECMWF transfers the raw limited domain data (covering Southeast Asia) with forecast ranges up to 10 days for HRES (resolution ~9 km, the deterministic forecast, denoted as IFS in this study) and ensemble forecast (ENS) (resolution ~16 km, the ensemble forecast, 51 members, denoted as ENS_ECMWF in this study) products to the VNHMA’s servers for further postprocessing (e.g., making meteorological variable field plots or using the initial and lateral boundary conditions for limited area models).

3. Verification Methodology and TC Lists for Validation

3.1. Verification Methodology

To detect TC features, the track (TC’s centers) and intensity (maximum surface wind near the TC’s center, denoted as Vmax) forecasts are determined using the downhill simplex method [28] and multicheck procedures for the wind, geopotential, and pressure at the mean sea level (PMSL) fields from the NWPs. A short description of the multicheck procedure is as follows:

• The first guess TC’s centers are taken from TC warning reports (RSMC or NCHMF).

• The minimum of the geopotential field is detected at levels of 700 and 850 hPa around the first estimate, with a radius of 6–8°. These minimum TC centers are used as the first estimate of the next detection step.

• The TC centers are set as the average of TC centers at levels of 700 and 850 hPa.

• The TC centers at 850 hPa are updated with the minimum wind speed positions at this level around the first guesses with a radius of 6–8°.

• Tangential winds are checked at the 850 hPa level to ensure that they are higher than 1 m/s.

• The minimum PMSL value is determined, and the final TC centers are used as the average of the storm centers at 700, 850 hPa, and PMSL.

• The gradient of the PMSL is checked to ensure that it is higher than 0.1 hPa/°.

• The TC’s intensity or Vmax is determined by searching for the maximum wind speed value within a radius of ~100 km from the determined center.

In some particular cases, when TCs move too fast, the tracker algorithm does not work, and it is often recalibrated based on the forecaster’s analysis.

TC track errors are mainly validated using the average values of direct positional errors (DPEs), which are defined as the distance between the observed (from the best track data) and forecast positions [29]. TC track errors can be further analyzed using along track errors (ATEs) and cross-track errors (CTEs). The details of determining DPEs, ATEs, and CTEs are showed in Figure 2. Positive/negative ATEs correspond to a forecast TC that is too fast/slow relative to the observed TC. Positive/negative CTEs correspond to a forecast TC that is too far to the right/left of the observed TC. After having DPEs, ATEs, and CTEs of all forecast cycles at different lead times, the mean errors (MEs) or MAEs can be calculated as average values. The confidence intervals are constructed based on the statistics of the track forecast errors at each lead time for a 95% significance level, due to the limitations of the validation sample. For TC intensity errors, the MAEs will be calculated from the forecast values to the maximum sustained wind speed of the best track dataset. This study uses the “m/s” unit for Vmax.

[figure(s) omitted; refer to PDF]

The ensemble mean (ENS_MEAN) and ensemble spread are calculated from set of ensemble members. The ENS_MEAN is calculated as the mean forecast position from all ensemble member forecasts, and the ensemble spread is calculated as the mean of the distances of each member to the ENS_MEAN position. The verification of ENS_MEAN can be considered a deterministic forecast, and it is also validated using the DPEs, ATEs, and CTEs scores.

The spread is normally verified using DPEs of the control member or the ENS_MEAN forecast of ECMWF’s ENS system, which can provide information on uncertainty regarding the spread to its unperturbed control forecast member [30].

The probabilistic verification of the ensemble products is further analyzed using a reliability diagram [31]. In this study, this calculation is carried out by counting strike probabilities relative to fixed geographical points [30, 32, 33]. At different lead times, for the geography domain of 6–22°N and 106–120°E, the probability is calculated using a fraction of the total number of ensemble members that pass within 120 km of the center of each 2° × 2°-resolution grid. Rescanning with a radius of 250 km is carried out to reduce the number of points where there is no TC passing, thus minimizing the samples of both the model and the observations that have “no passing TC” events. Based on that, reliability diagrams are generated based on the degree to which the forecasts probabilities match the observation frequencies.

3.2. TC Lists for Validation

In this study, we consider only TCs that passed the 120°E line of longitude. Due to the availability of both deterministic and ensemble products from the ECMWF, only 00 Coordinated Universal Time (UTC) and 12 UTC times are selected for issuing official bulletins and model running. Depending on the VNMHA archive, 72 TCs were active over the Bien Dong Sea from 2012 to 2019. Of these TCs, 25 made landfall. Table 1 summarizes the TC forecast information, and its best tracks are shown in Figure 3. We woted that the sample for the 24 h forecast lead time is higher that those of 48 h and 72 h forecast lead times leading to the inability to reflect the comparison of errors between these forecast lead times.

[figure(s) omitted; refer to PDF]

Table 1

List of the selected tropical cyclones during the 2012–2019 seasons. The TCID and TC names are taken from the Regional Specialized Meteorological Center Tokyo Typhoon Center. Forecasting cycles are formatted as YearMonthDayHour (in UTC).

TCID	TC name	Year	Forecasting cycle		Number of forecast cycles for validating at different lead times
			Start	End	24 h	48 h	72 h
1213	Kai-tak	2012	2012081600	2012081600	1	1	0
1220	Gaemi	2012	2012100400	2012100512	4	4	2
1223	Son-Tinh	2012	2012102412	2012102812	9	6	3
1224	Bopha	2012	2012120512	2012120700	4	4	2
1225	Wukong	2012	2012122700	2012122700	1	0	0
1301	Sonamu	2013	2013010312	2013010712	9	8	4
1305	Bebinca	2013	2013062100	2013062300	5	3	1
1306	Rumbia	2013	2013062912	2013063000	2	2	0
1309	Jebi	2013	2013073100	2013080212	6	4	2
1310	Mangkhut	2013	2013080612	2013080700	2	0	0
1311	Utor	2013	2013081112	2013081500	8	6	4
1319	Usagi	2013	2013092012	2013092212	5	4	2
1321	Wutip	2013	2013092712	2013093000	6	3	1
1325	Nari	2013	2013101100	2013101412	8	7	5
1329	Krosa	2013	2013103100	2013110312	8	6	3
1330	Haiyan	2013	2013110812	2013111012	5	3	1
1407	Hagibis	2014	2014061412	2014061500	2	1	0
1409	Rammasun	2014	2014071512	2014071900	8	5	3
1415	Kalmaegi	2014	2014091400	2014091600	5	3	2
1416	Fung-wong	2014	2014091900	2014092100	5	5	1
1421	Sinlaku	2014	2014112800	2014112812	2	1	0
1422	Hagupit	2014	2014120700	2014121100	7	5	4
1423	Jangmi	2014	2014122912	2014123100	4	3	0
1504	Maysak	2015	2015040412	2015040500	2	2	0
1508	Kujira	2015	2015062100	2015062400	7	4	0
1510	Linfa	2015	2015070500	2015070900	9	7	4
1513	Soudelor	2015	2015080800	2015080800	1	0	0
1515	Goni	2015	2015082100	2015082200	2	1	1
1519	Vamco	2015	2015091400	2015091400	1	0	0
1521	Dujuan	2015	2015092800	2015092900	2	1	0
1522	Mujigae	2015	2015100200	2015100400	5	3	1
1524	Koppu	2015	2015101700	2015102000	6	2	1
1527	Melor	2015	2015121500	2015121600	2	1	0
1601	Nepartak	2016	2016070712	2016070900	4	2	0
1603	Mirinae	2016	2016072612	2016072612	1	0	0
1604	Nida	2016	2016073012	2016080200	5	3	0
1608	Dianmu	2016	2016081800	2016081812	2	0	0
1614	Meranti	2016	2016091312	2016091412	3	2	0
1619	Aere	2016	2016100600	2016100900	7	5	3
1621	Sarika	2016	2016101500	2016101812	8	6	4
1622	Haima	2016	2016101912	2016102012	3	3	0
1625	Tokage	2016	2016112500	2016112700	3	1	0
1704	Talas	2017	2017071612	2017071612	1	0	0
1708	Sonca	2017	2017072312	2017072500	4	2	0
1713	Hato	2017	2017082100	2017082312	6	4	3
1714	Pakhar	2017	2017082500	2017082700	5	2	1
1716	Mawar	2017	2017090100	2017090300	5	3	1
1717	Guchol	2017	2017090600	2017090600	1	0	0
1719	Doksuri	2017	2017091212	2017091412	5	4	1
1720	Khanun	2017	2017101212	2017101500	6	3	2
1723	Damrey	2017	2017110200	2017110312	4	1	0
1724	Haikui	2017	2017110912	2017111200	5	3	1
1725	Kirogi	2017	2017111800	2017111800	1	0	0
1726	Kai-tak	2017	2017121700	2017122112	8	7	1
1727	Tembin	2017	2017122212	2017122412	3	1	0
1801	Bolaven	2018	2018010300	2018010300	1	0	0
1804	Ewiniar	2018	2018060600	2018060712	4	2	1
1809	Sontinh	2018	2018071700	2018071812	4	1	0
1822	Mangkhut	2018	2018091412	2018091612	4	3	1
1823	Barijat	2018	2018091200	2018091200	0	0	0
1826	Yutu	2018	2018102912	2018110112	7	5	3
1901	Pabuk	2019	2019010112	2019010312	5	3	1
1904	Mun	2019	2019070212	2019070312	2	1	0
1905	Danas	2019	2019071612	2019071712	3	2	2
1907	Wipha	2019	2019073100	2019080212	6	4	2
1911	Bailu	2019	2019082312	2019082412	1	1	0
1912	Podul	2019	2019082800	2019082912	4	2	0
1922	Matmo	2019	2019103000	2019103012	2	0	0
1924	Nakri	2019	2019110512	2019111000	9	7	6
1926	Kalmaegi	2019	2019111812	2019111912	2	1	0
1928	Kammuri	2019	2019120212	2019120500	6	4	2
1929	Phanfone	2019	2019122500	2019122712	2	0	0
Total	72	—	—	—	291	182	77

4. Results

4.1. Deterministic Forecast Performance

Regarding to track forecast, the DPEs of the models and official guidance from 24 to 72 h are shown in Figure 4. On average, for the lead time of 24–72 h, the DPEs of GFS, GSM, IFS, and ENS_MEAN are 77.1–180.2 km, 85.1–193.8 km, 71.0–178.0 km, and 69.2–166.2 km, respectively. Among the deterministic forecasts, ENS_MEAN gives the best results and followed by the IFS, GFS, and GSM models.

[figure(s) omitted; refer to PDF]

The DPEs of JMA and NCHMF are 87.3–212.0 km and 93.0–213.1 km, respectively. The average standard deviations (STDs) of the DPEs of NCHMF and JMA are about 52 km for a 24 h lead time. For lead times of 48 h and 72 h, the STDs of NCHMF are 10% higher than those of JMA.

The results also show that, from 2012 to 2019, the DPEs tended to decrease slightly for the 24–48 h lead time for both the JMA and NCHMF models; this decrease was more pronounced for a 72 h lead time for ENS_MEAN. Notably, in Figure 4, the models have DPEs lower than those of JMA and NCHMF when compared at the same issuing time; in fact, the model products exhibit a delay in the product availability times (4–6 h for GFS, GSM, and IFS and 8–10 h for ENS_MEAN); therefore, the forecasters have to use the previous forecast cycles of the NWPs. In Table 2, the comparison of the DPEs of models from the 12 h of previous cycles and the DPEs of JMA and NCHMF shows that the DPEs of the models are higher by 5%–10% than those of JMA or NCHMF.

Table 2

Annual averages of the DPEs of the NCHMF and JMA at 24 h and 48 h lead times vs. model at 36 h vs. 60 h lead times, respectively.

Year					Lead time
	36 h				24 h		60 h				48 h
	GFS	GSM	IFS	ENS_MEAN	NCHMF	JMA	GFS	GSM	IFS	ENS_MEAN	NCHMF	JMA
2012	127.41	136.99	96.26	100.56	102.70	103.61	173.34	248.56	166.58	164.89	187.78	169.84
2013	125.30	135.49	92.98	91.89	112.64	104.56	203.87	245.43	156.01	159.19	184.76	187.56
2014	108.13	104.72	102.74	95.53	109.60	114.57	140.73	163.90	158.48	161.59	158.82	164.53
2015	110.26	125.64	105.86	109.42	92.02	93.42	188.86	191.76	183.53	191.21	145.58	165.52
2016	90.58	121.92	74.05	65.58	72.72	66.74	198.47	250.36	164.64	158.36	169.48	159.14
2017	93.06	114.83	79.51	70.01	74.55	68.54	117.43	137.09	140.18	97.56	129.83	113.00
2018	83.48	112.84	95.67	91.32	71.67	57.60	117.27	152.25	163.11	133.71	102.59	105.27
2019	107.10	118.89	87.68	93.66	108.79	89.53	151.52	161.65	142.10	137.43	176.43	130.32

Abbreviations: DPEs, direct positional errors; ENS_MEAN, ensemble mean; GFS, Global Forecast System; GSM, global spectral model; IFS, Integrated Forecasting System; JMA, Japanese Meteorological Agency; NCHMF, National Centre for Hydro-Meteorological Forecasting.

The above track forecast evaluation results show that there are qualitative similarities between models, centers in recent researches. Chen et al. [16] validated 21 TCs over the South China Sea for season 2021 from both global and regional NWPs. The results are quite similar, where IFS provides the best performance (DPEs were 63.5 km, 125.2 km, and 186.2 km for 24 h, 48 h, and 72 h lead times, respectively), after that are GFS and GSM. A comprehensive validation of Pieu et al. [34] for the Bay of Bengal (BoB) was assessed for 12 TCs of the period 1991–2019 based on the Met Office Unified Model (MOUM) at very high resolutions (4.5 and 1.5 km) and using time-lagged runnings for ensemble products [33]. The overall average track errors were ~150 km when compared with the best track data from JTWC and IMD. This similar result shows that the forecasting skills of the models are also quite stable for different sea areas.

Further assessment of the DPEs is related to the TC’s intensity for the TCs with a maximum sustained wind speed of over 24.4 m/s (severe tropical storm [STS], accounting for about 48% of the samples evaluated). Compared to overall DPEs, the DPEs of STS TCs tend to decrease by about 10%–15% for GFS, GSM, and IFS at all lead times; for ENS_MEAN, this value is about 15%–25%. For the JMA and NCHMF, the reduction is about 8%–12%. For the TCs weaker than STS (accounting for about 52% of the samples evaluated), the DPEs mostly increase: the value for the GFS model at 24–48 h is about 15%–25%, the value for GSM is about 10%–20%, and the value for IFS and ENS_MEAN is about 10%. At the 72 h lead time, the models show a slight increase of about 5%–8%. For the forecasts of the JMA and NCHMF, this value mainly increases by about 10%–12% in the 24 h lead time, while, in the 48–72 h period, the error increases but less so, by only about 5%–7% on average.

MEs or bias values for ATEs and CTEs of the models and official guidance from 24 to 72 h are shown in Figures 5 and 6. With a trend of negative bias in the ATE for all lead times, the forecast TCs moved more slowly than the observations. Among the models, ENS_MEAN and IFS provided better results; MAEs of ATEs are about 50–120 km for the lead time of 24–72 h. This is no coincidence, as the forecasts from the JMA and NCHMF have similar bias trends of ATEs to the models or show great dependence on the guidance of models. These evaluation results of ATEs are important in estimating the time of TC’s landfall, evaluating the early or late landfall corresponding to slower or faster bias. In the BoB in the work of Nahruma mentioned above [33], landfall times are often overestimated from ensemble forecasts which were equivalent to the fact that TCs were moving faster than they actually were and the use of ATEs can partly reflect this information.

[figure(s) omitted; refer to PDF]

The trend for CTEs is not as clear as that for ATEs; before 2016, CTEs mainly exhibited a positive trend (rightward to the observation), but after 2016, there is a negative trend (leftward to the observation), and a similar trend is seen in the official forecasts and the models. The MAEs of the CTEs are about 50–100 km for the models and 50–200 km for the JMA and NCHMF. Compared to the DPEs, from 2012 to 2019, there was no clear improvement in the ATEs and CTEs of the models or the JMA and NCHMF.

Regarding to TC’s intensity forecast, MAEs of Vmax of the modes from 24 to 72 h are shown in Figure 7. Except for 2016 with quite large errors from NCHMF and JMA forecasts compared to the model, on average, for the lead time of 24–72 h, the MAEs of GFS, GSM, IFS, and ENS_MEAN are 6.3–6.9, 8.4–9.6 m/s, 6.5–7.6 m/s, and 7.9–8.9 m/s, respectively. Among the deterministic intensity forecasts, GFS gives the best results and followed by the IFS, ENS_MEAN, and GSM models. For lead times of 24 h, 48 h, and 72 h, respectively, the MAEs of JMA are 4.3, 5.9, and 9.3 m/s and for NCHMF are 3.6, 6.4, and 8.6 m/s. In general, MAEs of JMA and NCHMF are lower than modes coming from the use of both information from NWPs, statistical schemes. With weak TCs (weaker than STS), model errors are reduced by 30%–40% compared to strong storms on STS; for example, MAEs of Vmax at 24 h forecast range from models are around 3.3–4.2 m/s. At short range forecast, the forecast of Vmax of NCHMF shows a better result compared to JMA coming from the uses of additional observations and regional models in making TC’s intensity assessments. For 48 h and 72 h forecast ranges, the intensity forecasts from JMA provided good guidance for NCHMF to release final forecasts. A comparison between models and official bulletins at the 72 h forecast range shows that TC’s intensity forecasting remains a challenge over Bien Dong Sea.

[figure(s) omitted; refer to PDF]

4.2. The ECMWF’s Ensemble Forecast Performance

In the practical application of the ensemble product of ECMWF to TC forecasting, the main track guidance is based on deterministic forecasts and the combined ENS_MEAN track; the affected areas depend on the error of the NCHMF itself and the spread of ensemble products (strike probability maps, Figure 1a) at each lead time forecast. In many cases, at the 24 h lead time, the spread is too narrow to capture the change in the TC’s tracks, especially for weak and slow-moving TCs. For high-intensity TCs, the overfitting issue (very small spread) sometimes leads to large errors in the ENS_MEAN forecast (e.g., the DPEs of TC Haiyan in 2013 at lead times of 24 and 48 h were 250–500 km in several forecast cycles).

For the ENS_MEAN forecast from ECMWF, the ENS_MEAN showed an advantage over the other deterministic forecasts. On average, the DPEs of the control member are higher than ENS_MEAN by 6%–8%. Regarding the uncertainties of this system, we investigated the relationship between the DPEs of both the control member and ENS_MEAN vs. the spread of ENS_ECMWF, as shown in Figure 8a (for the spread vs. DPEs of ENS_MEAN only, good spread–skill relationships are closer to the unit line). These relationships can be seen more clearly in the bidirectional scatter plots of Figure 8b [34], which are based on the typical percentile values (median, 25%, and 75%) of the DPEs of ENS_MEAN and the spread. There is a weak correlation between ENS_ECMWF’s spread and the DPEs of the control member/ENS_MEAN of ENS_ECMWF. The control member and ENS_MEAN have a large error, corresponding to small spreads (points located above the unit line), and this is especially clear for 48 h and 72 h lead times. There is a slight difference between the relationship of spread and DPEs of the control member to ENS_MEAN. The control member provides a relatively good relationship up to the lead time of 54 h (Figure 8c); meanwhile, the lower spread vs. ENS_MEAN’s DPEs occurs at the lead time of 42 h. Even with its higher DPEs, the spread represents the model’s good level of uncertainty for the lead times up to the first 2 days.

[figure(s) omitted; refer to PDF]

The reliability diagram for the lead times of 24, 48, and 72 h is shown in Figure 9. Here, the reliability of all lead times below the unit line indicates the overforecast/overconfidence trend of the ENS_ECMWF, especially at the 72 h lead time. The best confidence interval is at less than 50% forecast probability for the lead time of 48 h and less than 30% for the lead time of 72 h. When the reliability assessment of the probability forecast is based on the number of hit ensemble members in a specific area at a specific number of points, at the lead time after 48 h, the dispersion of a quality ensemble forecast system will expand quickly enough (typically 250–400 km), so the current reliability assessment for high strike probability rates above 80%–90% has little meaning. Cases of high strike probability (>90%) for the lead time after 2 days are very similar to track forecasts for all ensemble members. Therefore, they can invisibly reduce the uncertainty in an ensemble system. Therefore, around 50%–70% probability would be the most significant.

[figure(s) omitted; refer to PDF]

The difference can be seen when comparing a limited sea region (Bien Dong Sea) with a limited number of samples instead of evaluating a larger sample set and globally in Helen et al. [35]. This research validates a multimodel ensemble TC track probability forecast (combination of the ECMWF ensemble system (ECMWF ENS, 51 members), the Met Office (MOGREPS-G [the Met Office global and regional ensemble prediction system–global], 24 members), and the NCEP (GEFS [global ensemble forecast system], 21 members) in 2 years. At the 24 h and 48 h lead times, reliability diagrams for ECMWF show a similar to our results, although at high forecast probability thresholds (over 60%) for 72 h lead time there is a higher observation frequency, which could come from a larger sample and for a large region. Here, it can be seen that there will also be lower reliability for high probability prediction thresholds in a limited sea region.

5. Conclusions

The research presents verification results of TC forecasts from the official bulletins of JMA and NCHMF,and NWPs (IFS, ENS_ECMWF, GSM, and GFS) for the period 2012–2019, accounting for 72 activaved TCs in the Bien Dong Sea. The main conclusions are as follows:

i. The deterministic track forecasts from the NWPs and JMA provide good guidance for the official forecast of the NCHMF. A negative bias exists in ATEs and CTEs regarding slow-leftward-track forecasts.

ii. The trends in the DPEs, ATEs, and CTEs are very similar between deterministic track forecasts (including ENS_MEAN forecasts) and official track guidance, meaning that guidance is mainly dependent on.

iii. The track forecast errors are the largest for TCs with an intensity weaker than that of an STS (regarding the earlier development of the TC or the decay stages).

iv. For intensity forecast, a remarked difference of errors between models and official bulletins at 24 h and 48 h forecast ranges. Intensity forecasting shows limitations from the model, and with subjective adjustments, forecast official bulletins of NCHMF and JMA can reduce errors, especially for short-range forecasts.

v. The ENS_MEAN track forecasts show a better result compared to all of the deterministic models. However, the spread of ensemble forecasts is still limited in terms of covering real TC positions, and the strike probability forecast is quite overforecast/overconfident, especially for a lead time of 72 h and at forecasted areas with a high probabilities of occurrence.

In fact, regarding track forecast, the issues related to the higher DPEs of models to NCHMF/JMA forecasts can be explained by a few key points. The first of these is that the model is subjected to the latest observations. This forecast is independent of human subjectivity and objectively simulates the atmospheric state (e.g., a data assimilation technique). In operation, forecasters make a comparison between the model forecast and observation and try to take into account local or large-scale changes (e.g., subtropical high-pressure systems, intertropical convergence zones, the Fujiwara effect, and upper-/lower-level divergence) and surface conditions (sea surface temperature and surface latent heat fluxes) that are likely to affect the track and intensity (especially on the rapid intensification) of TCs [13, 36]. Because of objective forecast procedures, the ability to adapt information from model errors and the reliability of ensemble systems are based on the forecasters’ experience. Therefore, the error updates may exhibit a significant delay compared to the continuous objective updating of the model track forecast. Another important reason is that the forecasting of TCs depends on disaster management factors. Assessing of sudden changes in TCs in official forecasts requires careful analysis because these changes directly relate to the entire disaster management system. Severe TCs can even significant impact disaster prevention, ship banning, or residence relocation. This explains why the errors are updated slowly and the model’s output adjustment also proceeds step by step to assess the overall correlation between TC forecasting and prevention.

In a follow-up study, we will conduct a verification of TC forecasts, using enhanced approaches based on regional models and statistical models. How disaster prevention agencies take full advantages of the information provided by the NCHMF official bulletins will be discussed in more detail.

Acknowledgments

Hoa Vo Van was supported by the Ministry of Natural Resources and Environment’s project titled “Research on forecasting the abnormal change in track and intensity of tropical cyclones that approach to Vietnam sea shore” (code: TTMT.2022.06.01). Tien Du Duc was supported by the Ministry of Science and Technology’s project titled “Research and application of artificial intelligence technique to build the forecasting system of tropical cyclone activity over Bien Dong Sea and affecting Vietnam with lead time up to 3 days” (code: KC-4.0-46/19-25). Lars Robert Hole was sponsored by the Norwegian Agency for Development Cooperation and the Norwegian Ministry of Foreign Affairs. The authors would like to thank Ms. Nguyen Thi Nga and Ms. Chu Thi Huyen Trang for their help in preparing the figures.

References

[1] S. D. Aberson, C. R. Sampson, "On the Predictability of Tropical Cyclone Tracks in the Northwest Pacific Basin," Monthly Weather Review, vol. 131 no. 7, pp. 1491-1497, DOI: 10.1175/1520-0493(2003)131%3C1491:OTPOTC%3E2.0.CO;2, 2003.

[2] V. T. Phan, T. T. Long, B. H. Hai, C. Kieu, "Seasonal Forecasting of Tropical Cyclone Activity in the Coastal Region of Vietnam Using RegCM4.2," Climate Research, vol. 62 no. 2, pp. 115-129, DOI: 10.3354/cr01267, 2015.

[3] D. Q. Nguyen, J. Renwick, J. Mcgregor, "On the Presence of Tropical Vortices Over the Southeast Asian Sea-Maritime Continent Region," Journal of Climate, vol. 29 no. 13, pp. 4793-4800, DOI: 10.1175/JCLI-D-14-00468.1, 2016.

[4] C. Wang, J. Liang, K. I. Hodges, "Projections of Tropical Cyclones Affecting Vietnam Under Climate Change: Downscaled HadGEM2-ES Using PRECIS 2.1," Quarterly Journal of the Royal Meteorological Society, vol. 143 no. 705, pp. 1844-1859, DOI: 10.1002/qj.3046, 2017.

[5] M. K. Hung, D. D. Tien, D. D. Quan, "Assessments of Use of Blended Radar-Numerical Weather Prediction Product in Short-Range Warning of Intense Rainstorms in Localized Systems (SWIRLS) for Quantitative Precipitation Forecast of Tropical Cyclone Landfall on Vietnam’s Coast," Atmosphere, vol. 14 no. 8,DOI: 10.3390/atmos14081201, 2023.

[6] B. T. Nguyen, K. Sooyoul, D. D. Chien, "Assessment of Storm Surge along the Coast of Central Vietnam," Journal of Coastal Research, vol. 33 no. 3, pp. 518-530, DOI: 10.2112/JCOASTRES-D-15-00248.1, 2017.

[7] H. A. Nguyen-Thi, J. Matsumoto, T. Ngo-Duc, N. Endo, "A Climatological Study of Tropical Cyclone Rainfall in Vietnam," SOLA, vol. 8 no. 0, pp. 41-44, DOI: 10.2151/sola.2012-011, 2012.

[8] M. DeMaria, J. A. Knaff, C. Sampson, "Evaluation of Long-Term Trends in Tropical Cyclone Intensity Forecasts," Meteorology and Atmospheric Physics, vol. 97 no. 1–4, pp. 19-28, DOI: 10.1007/s00703-006-0241-4, 2007.

[9] L. A. Avila, P. Caroff, J. Callaghan, J. Franklin, M. DeMaria, "Track Forecasts," pp. 12-18, .

[10] Y. Kurihara, M. A. Bender, R. J. Ross, "An Initialization Scheme of Hurricane Models by Vortex Specification," Monthly Weather Review, vol. 121 no. 7, pp. 2030-2045, DOI: 10.1175/1520-0493(1993)121%3C2030:AISOHM%3E2.0.CO;2, 1993.

[11] H. Van Nguyen, Y.-L. Chen, "High-Resolution Initialization and Simulations of Typhoon Morakot (2009)," Monthly Weather Review, vol. 139 no. 5, pp. 1463-1491, DOI: 10.1175/2011MWR3505.1, 2011.

[12] H. Van Nguyen, Y.-L. Chen, "Improvements to a Tropical Cyclone Initialization Scheme and Impacts on Forecasts," Monthly Weather Review, vol. 142 no. 11, pp. 4340-4356, DOI: 10.1175/MWR-D-13-00326.1, 2014.

[13] P. R. Naskar, D. R. Pattanaik, "Variations of Central Pressure of Intense Tropical Cyclones Over the Bay of Bengal With Latent Heat Flux and Other Parameters," Journal of Earth System Science, vol. 132 no. 2,DOI: 10.1007/s12040-023-02065-6, 2023.

[14] E. D’Asaro, P. Black, L. Centurioni, "Typhoon-Ocean Interaction in the Western North Pacific: Part 1," Oceanography, vol. 24 no. 4, pp. 24-31, DOI: 10.5670/oceanog.2011.91, 2011.

[15] C. Q. Kieu, Z. Moon, "Hurricane Intensity Predictability," Bulletin of the American Meteorological Society, vol. 97 no. 10, pp. 1847-1857, DOI: 10.1175/BAMS-D-15-00168.1, 2016.

[16] G. Chen, X. Zhang, L. Bai, R. Wan, "Verification of Tropical Cyclone Operational Forecast in 2018, 51st Session ESCAP/WMO Typhoon Committee," 2018. http://www.typhooncommittee.org/51st/docs/item%206/6.2%20Verification%20of%20TC.pdf

[17] M. DeMaria, "A Simplified Dynamical System for Tropical Cyclone Intensity Prediction," Monthly Weather Review, vol. 137 no. 1, pp. 68-82, DOI: 10.1175/2008MWR2513.1, 2009.

[18] J. B. Courtney, S. Langlade, C. R. Sampson, "Operational Perspectives on Tropical Cyclone Intensity Change Part 1: Recent Advances in Intensity Guidance," Tropical Cyclone Research and Review, vol. 8 no. 3, pp. 123-133, DOI: 10.1016/j.tcrr.2019.10.002, 2019.

[19] S.-H. Kim, I.-J. Moon, P.-S. Chu, "Statistical-Dynamical Typhoon Intensity Predictions in the Western North Pacific Using Track Pattern Clustering and Ocean Coupling Predictors," Weather and Forecasting, vol. 33 no. 1, pp. 347-365, DOI: 10.1175/WAF-D-17-0082.1, 2018.

[20] M. DeMaria, J. L. Franklin, R. Zelinsky, "The National Hurricane Center Tropical Cyclone Model Guidance Suite," Weather and Forecasting, vol. 37 no. 11, pp. 2141-2159, DOI: 10.1175/WAF-D-22-0039.1, 2022.

[21] M. Mohapatra, B. K. Bandyopadhyay, D. P. Nayak, "Evaluation of Operational Tropical Cyclone Intensity Forecasts Over North Indian Ocean Issued by India Meteorological Department," Natural Hazards, vol. 68 no. 2, pp. 433-451, DOI: 10.1007/s11069-013-0624-z, 2013.

[22] M. Yamaguchi, T. Komori, "Outline of the Typhoon Ensemble Prediction System at the Japan Meteorological Agency," RSMC Tokyo-Typhoon Center Technical Review, vol. 11, pp. 14-24, 2009.

[23] K. Ito, "Errors in Tropical Cyclone Intensity Forecast by RSMC Tokyo and Statistical Correction Using Environmental Parameters," SOLA, vol. 12, pp. 247-252, DOI: 10.2151/sola.2016-049, 2016.

[24] C. Muroi, "Brief History and Recent Activities of RSMC Tokyo—Typhoon Centre," Tropical Cyclone Research and Review, vol. 7 no. 1, pp. 57-64, DOI: 10.6057/2018TCRR01.09, 2018.

[25] K. Masashi, "Tropical Cyclone Information Provided by the Rsmc Tokyo-Typhoon Center," Tropical Cyclone Research and Review, vol. 1, pp. 51-59, DOI: 10.6057/2012TCRR01.06, 2012.

[26] R. Oyama, "Algorithm and Validation of a Tropical Cyclone Central Pressure Estimation Method Based on Warm Core Intensity as Observed Using the Advanced Microwave Sounding Unit-A (AMSU-A)," Tropical Cyclone Research and Review, vol. 16, 2014.

[27] Q. N. Tran, "Analysis and Prediction of Tropical Storm Pakhar at National Center for Hydro-Meteorological Forecasting of Viet Nam," Tropical Cyclone Research and Review, vol. 1, pp. 177-183, DOI: 10.6057/2012tcrr02.10, 2012.

[28] H. William, A. Teukolsky Saul, T. Vetterling William, P. Flannery Brian, Numerical Recipes: The Art of Scientific Computing, . 978-0-521-88068-8

[29] J. T. Heming, "Tropical Cyclone Tracking and Verification Techniques for Met Office Numerical Weather Prediction Models," Meteorological Applications, vol. 24 no. 1,DOI: 10.1002/met.1599, 2017.

[30] G. Van der Grijn, "Tropical Cyclone Forecasting at ECMWF: New Products and Validation;," ECMWF Technical Memoranda, vol. 13,DOI: 10.21957/c8525o38f, 2022.

[31] D. S. Wilks, Statistical Methods in the Atmospheric Sciences, 2006.

[32] N. M. Leonardo, B. A. Colle, "Verification of Multimodel Ensemble Forecasts of North Atlantic Tropical Cyclones," Weather and Forecasting, vol. 32 no. 6, pp. 2083-2101, DOI: 10.1175/WAF-D-17-0058.1, 2017.

[33] A. Dube, R. Ashrit, S. Kumar, A. Mamgain, "Improvements in Tropical Cyclone Forecasting Through Ensemble Prediction System at NCMRWF in India," Tropical Cyclone Research and Review, vol. 9 no. 2, pp. 106-116, DOI: 10.1016/j.tcrr.2020.04.003, 2020.

[34] N. M. Pieu, A. S. Islam, M. K. Das, F. Abdullah, "Assessing Tropical Cyclone Tracks in the Bay of Bengal," ,DOI: 10.1109/OCEANS47191.2022.9977334, .

[35] H. A. Titley, R. L. Bowyer, H. L. Cloke, "A Global Evaluation of Multi-Model Ensemble Tropical Cyclone Track Probability Forecasts," Quarterly Journal of the Royal Meteorological Society, vol. 146 no. 726, pp. 531-545, DOI: 10.1002/qj.3712, 2020.

[36] K. Ito, S. Hirano, J.-D. Lee, J. C. L. Chan, "Three-Dimensional Fujiwhara Effect for Binary Tropical Cyclones in the Western North Pacific," Monthly Weather Review, vol. 151 no. 7, pp. 1779-1795, DOI: 10.1175/MWR-D-22-0239.1, 2023.