Enhancing sustainability in palm oil industry: reinforcement learning for renewable energy management considered climatic variability

Abstract

Energy sources are critical in the industrial sector, particularly as population growth intensifies the pressure on industries to scale up production to meet increasing demands. Integrating renewable energy sources such as biomass, biogas, and photovoltaic systems in the palm oil production process can be considered a pivotal strategy for mitigating carbon emissions. However, load fluctuations due to mill size and insufficient supply of biogas and photovoltaic are significant challenges in energy management for the palm oil industry. Adopting renewable energy solutions is hindered by process disturbances and climatic variability caused by weather, ambient temperature, relative humidity, and solar radiation. These factors pose constraints on the implementation of renewable energy solutions. Therefore, this study aims to fill the gap in terms of renewable energy management within the palm oil industry by applying reinforcement learning to achieve effective energy management and reduce carbon emissions under five climatic scenarios. The result shows that reinforcement learning effectively combines renewable energy sources to generate electricity and steam while efficiently managing energy storage without exceeding predefined levels. Additionally, deploying a photovoltaic system contributes to energy savings from biomass and biogas of 2.9% and reduces average daily carbon dioxide emissions by 3137 kg. The proposed method has proved beneficial across all climate scenarios for energy savings and emission reduction.

Introduction

Nowadays, energy sources have become indispensable in satisfying the energy demand from the industrial sector. It is because of the rapid increase in population and the growth of industries globally to encourage the progressive competition [1]. The palm oil industry significantly contributes to the global economy, particularly in Malaysia and Indonesia, accounting for approximately 85% of the global palm oil [2]. Among these, Thailand's palm oil industry is also one of the contributors, producing approximately 3.35 million tons annually and accounting for about 4.6% of the total market share [3]. This substantial production not only highlights the position of Southeast Asia as a significant global producer, but also draws attention to the palm oil industry as a major energy consumer in the sector.

The palm oil industry regularly produces a substantial quantity of biomass waste, which cogeneration systems effectively utilize as fuel to assist in power generation and the making of steam for use in the process [4]. The process involves the combustion of biomass wastes such as empty fruit bunches (EFB), palm pressed fibers (PPF), and palm kernel shells (PKS) [5] to fuel steam turbines in electricity generation [6]. Furthermore, the anaerobic digestion of palm oil mill effluent (POME) produces biogas that powers gas engine generators and supports waste heat recovery systems [7]. Adoption of biomass and biogas combustion makes the palm oil industry a renewable source since CO₂ emissions created during production and waste combustion are part of the carbon cycle and are absorbed during the growth period and palm plantation [8]. However, while biomass and biogas serve as the primary energy sources in the palm oil industry, there are still limitations to be addressed when optimizing these processes.

The scale of palm oil milling substantially influences biomass for energy generation, as the availability of biomass residues is directly proportional to the milling output [9]. Large mills in major producing regions can generate excessive biomass for energy production, whereas smaller mills may struggle with energy generation due to insufficient biomass waste [10]. Similar to the published work, Aziz et al. suggested that even if the mill has enough biomass to achieve high efficiency in energy generation, optimal operational conditions are required to be maintained consistently [11]. Analogously, biogas production is influenced by multiple factors. A study on the acidogenesis of POME found that the concentration of organic compounds and solid content significantly influences the growth of microorganisms [12]. Another important factor is that high concentrations of trace metals like iron were found to inhibit microbial activity and reduce biogas yield [13]. Variability in hydraulic retention time and agitation rates can also impact microorganism growth and biogas production, leading to suboptimal conditions for microbial growth and a lower biogas generation rate [14]. The fluctuations in energy supply caused by these disruptions could have led the palm oil industry to seek additional energy sources to support their processes.

Solar photovoltaic (PV) systems can provide a significant portion of the energy required for palm oil processing, especially in regions with high solar irradiance, such as the Southeast Asia region [15]. In Indonesia, the combination of solar PV with other renewable sources like biogas from palm oil biomass has been shown to be a technically and economically feasible option [16]. The results demonstrated that the hybrid system, comprising PV panels, a biogas-fueled generator, an inverter, and batteries, could produce around 67,216 kWh annually, showcasing both technical and economic viability as well as environmental benefits [17]. Such strategies not only decrease reliance on grid electricity but also reduce CO₂ emissions during production, contributing to the operational sustainability and economic viability of industrial settings by creating negative emission values on CO₂ releases [18] while ensuring that the energy supply is seamlessly matched with the production requirement.

Renewable energy integration is a sustainable step for the palm oil industry, but it also presents challenges and constraints, mainly due to the data characteristics that must be considered and appropriately managed [19]. Internally, the industry faces issues such as operational uncertainties [20], cyclic patterns of operation [21], multi-time scale operating conditions [22], and limited data [23]. These internal disturbances can complicate the consistent application of renewable energy solutions. To address these complexities, artificial intelligence (AI)-based management systems have been introduced, offering sophisticated approaches to energy optimization [24]. For example, a study by Oliveira, Silva, and Tostes discussed a methodology for renewable energy projects in the Amazon using palm oil biomass, which provides insights on integrating AI to enhance project viability through environmental, economic, and social sustainability indicators [25]. Additionally, Rana et al. explored advanced biotreatment technologies for palm oil mill effluent using improved bioreactor configurations to produce biogas, illustrating how AI can aid efficient waste management and energy production [26]. Moreover, Akhtar et al. investigated the use of remote sensing and AI to manage palm oil plantations, enhancing yield and sustainability through high-resolution data analysis [27].

Externally, climatic variability, including changes in wind speed, solar irradiation, and overall weather conditions, can significantly affect the performance of power generation for renewable energy systems [28]. Changes in external disturbances can drastically fluctuate generation stability, leading to significant inconsistencies in power output [29]. Managing these variations in renewable energy sources requires an effective storage solution to mitigate fluctuations and ensure a stable power output [30]. However, one of the primary challenges in integrating solar energy into the palm oil industry is its inherent unpredictability and variability [31]. Climatic variability can cause interruptions in the continuous operation of palm oil production due to the unstable energy supply required for its processes caused by the shortage or off-specification of biomass feedstock [32]. Additionally, extreme temperature fluctuations resulting from abrupt climatic changes can impact the efficiency of biogas production [33]. To handle these fluctuations and maintain generation stability, the industry can benefit significantly from adopting advanced technologies such as reinforcement learning (RL) [34].

RL is becoming an essential tool for optimizing energy management systems in industrial settings, particularly where there are high degrees of uncertainty and variability in supply [35]. For instance, Shouryadhar et al. proposed an enhanced reinforcement learning-based method (ERLBM) to optimize energy loads primarily from renewable resources like solar panels, a critical aspect for consistent energy supply in the palm oil industry [36]. Furthermore, Nakabi and Toivanen developed a microgrid model that uses deep reinforcement learning algorithms to prioritize resources and electricity pricing to coordinate multiple renewable energy sources in fluctuating conditions [37]. Ahrarinouri et al. used Q-learning to integrate demand response programs for different components in a multiagent RL for residential multicarrier energy management, which could be applied to efficiently manage energy across the diverse operations within the palm oil industry [38]. Despite these advances, climatic variability remains a critical challenge and unaddressed, causing frequent disruptions in the energy supply, which is pivotal for uninterrupted operations in the palm oil industry.

Therefore, this study aims to fill a research gap in energy management of renewable energy sources in the palm oil industry with climatic variability by using RL to achieve effective energy management. The proposed RL framework, utilizing Q-learning, is designed to dynamically adapt to fluctuations in energy management caused by operation noise, load fluctuation, and climatic variability. The system can learn and predict the optimal allocation of renewable energy sources, including biomass, biogas, and PV systems through Q-learning. The proposed approach ensures a stable and efficient energy supply, promoting renewable energy integration, while the interconnections between electricity and steam generation are effectively managed. The key contributions of this paper can be summarized as follows:

1. Propose an RL framework for energy management in the palm oil industry. This framework is designed to learn and adapt to varying energy outputs from solar, biomass, and biogas systems, ensuring efficient energy use throughout palm oil production.

2. Address the energy management in the palm oil industry under climatic variability. By focusing on the dynamic climatic conditions that influence renewable energy generation, this research seeks to establish a robust energy management system that can withstand and adapt to weather-induced fluctuations. This includes integrating predictive analytics to anticipate energy availability and demand, which is crucial for continuous operations.

3. Determine energy savings and CO₂ emission reduction potentials. Through the implementation of the proposed RL system, this study quantifies the energy savings achieved by more efficiently managing renewable resources. Furthermore, the study evaluates the impact on CO₂ emissions, highlighting the environmental benefits of shifting to a more sustainable energy management strategy within the industry.

Palm oil industry

The palm oil industry is currently experiencing substantial growth, characterized by intense competition and continuous development [39]. This expansion has coincided with an increasing interest in environmental protection within the industry [40], reflecting a broader commitment to sustainability. As a result, the industry has adopted the zero-waste principle, which involves using by-products and waste from the production process as raw materials in other continuous industries, thereby preventing loss and minimizing environmental impact. Figure 1 provides an overview of the palm oil industry. In line with these sustainability efforts, the smart complex concept has been implemented, consolidating all segments of the palm oil chain into one process. The comprehensive process includes palm oil milling, refining, biodiesel and glycerin production, and a power plant that leverages renewable energy sources such as biomass, biogas, and PV.

Fig. 1 [Images not available. See PDF.]

Overview of the palm oil industry

Palm oil milling process

In this process, the primary products obtained are crude palm oil (CPO) from the mesocarp and crude palm kernel oil (CPKO) from the endosperm [41]. The process involves feeding fresh fruit bunches (FFB) into a sterilization chamber where high-pressure steam is applied [42]. The sterilized FFBs are then moved to the stripping section, where fruits are separated from the EFB. The condensate generated in this part becomes heavily contaminated, resulting in POME. The separated fruits proceed to the digestion-pressing section, where steam aids in homogenization before the mixture is pressed, yielding pressed cake and crude oil, as depicted in Fig. 2a.

Fig. 2 [Images not available. See PDF.]

Process flow diagram of the palm oil industry production process

The crude oil is then transferred to the clarification section to remove any remaining solids, resulting in crude oil and sludge. Following this, crude oil undergoes further purification and drying to enhance the purity of the CPO and reduce moisture content. The crude oil is then separated from the sludge in the screening section, resulting in purified CPO. Meanwhile, the pressed cake from the digestion-pressing section is processed in the depricarping stage to separate palm nuts from the PPF. The palm nuts are then dried to reduce moisture before being moved to the cracking and winnowing section, where dry PKS are separated from the nuts [43]. The remaining mixture enters a hydrocyclone, where water is used to separate the wet kernels from the wet shells, generating additional POME. The wet kernels are then dried to reduce moisture content before being moved to the oil expression section, where the palm kernels are pressed to separate palm kernel cake (PKC) and CPKO.

Palm oil refinery process

The palm oil refinery process employs physical steam refining methods to remove palm fatty acid distillate (PFAD), encompassing several key steps: degumming, bleaching, filtration, and deodorization, as visualized in Fig. 2b [44]. Degumming begins as CPO enters the neutralization section, where the oil is preheated using an economizer. Phosphoric acid (H₃PO₄) is then added to neutralize the CPO, aiming to decompose non-hydrated phosphatides and form precipitates, making them insoluble and easily removable during the subsequent bleaching stage [45].

Following degumming, the oil proceeds to the bleaching section. In order to reduce color pigments and improve the oxidative stability of the oil, bleaching earth and steam are added to this process [46]. The mixture undergoes thorough agitation to ensure effective contact time and absorption by the bleaching earth. Subsequently, the bleached oil enters the filtration section, where steam assists in removing the bleaching earth, and precipitated substances are drained from the bottom of the filter. The filtered oil is then reheated as it passes through a heat exchanger before moving to the deodorization section, the final stage of the refining process. During deodorization, the oil is treated under vacuum conditions with steam to remove odorous substances and PFAD from the top of the deodorizer. The output of this process is refined, bleached, and deodorized palm oil (RBDPO), ready for further use or distribution.

Biodiesel and glycerin production process

The production of biodiesel and glycerin primarily involves a trans-esterification reaction [47], prominently displayed in Fig. 3. Among the methods available, trans-esterification is the most commonly utilized. Biodiesel emerges as the main product, while glycerin serves as the by-product. The initial step requires preparing RBDPO with a free fatty acid (FFA) content of less than 5%; higher levels could result in soap formation during the reaction, which occurs upon contact with sodium hydroxide (NaOH) as a catalyst [48].

Fig. 3 [Images not available. See PDF.]

Transesterification reaction

Once prepared, the RBDPO is heated and then fed to a reactor, where it is combined with methanol and NaOH to facilitate the transesterification reaction. The mixture is then sent to a separator, separating it into biodiesel and glycerol layers. The separation occurs due to the difference in densities between the biodiesel and glycerol, with the heavier glycerol settling at the bottom and the lighter biodiesel forming the top layer [49]. After the initial separation, the biodiesel still contains unreacted methanol, which needs to be recovered [50]. Employing a distillation column, the low-boiling-point methanol vaporizes and rises to the top, leaving the biodiesel stream behind.

The biodiesel is then subjected to a washing process to remove any remaining impurities and residual catalysts. During washing, the biodiesel is mixed with warm water, which helps dissolve and removes any soap, residual methanol, and catalyst. The mixture is then allowed to settle, while the water, now containing the impurities, is drained from the bottom. Following the washing process, the biodiesel undergoes a drying process using a drying tank where the biodiesel is heated to evaporate the water. Finally, the purified biodiesel is transferred to a biodiesel tank for storage.

The glycerin refinery process begins with crude glycerol stored in the crude glycerol tank, which is pumped by the feed pump through a basket filter to remove large impurities. The filtered glycerol is preheated in the heat recovery exchanger before entering the acidification vessel, where hydrochloric acid (HCl) is added to break down soaps. The mixture is then neutralized with sodium hydroxide in the neutralization vessel, which adjusts the pH to around 7 to facilitate easier operation, extend the remaining useful life of equipment, and prevent soap in the refinery unit. The neutralized glycerol is heated in the dryer heater and further dried in the dryer to remove excess water, with vapors condensed in the reflux condenser under vacuum conditions. The concentrated glycerol enters the glycerol distillation column, heated by the glycerol reboiler, for further purification. Distillate vapors are condensed in the condensation cooler and collected in the condensation receiver. The distilled glycerol is deodorized in the deodorizer and further concentrated in the evaporator, with residues removed by the residue pump. Process gases are then cleaned in the scrubber, and the refined glycerin is subsequently passed to the bleaching unit to make it clear before undergoing final filtration and packaging for commercial use. All of these steps, from the preparation of RBDPO to the final refined glycerin, are comprehensively illustrated in Fig. 2c.

Power plant

The palm oil industry relies heavily on both electrical and thermal energy for its operations. For a comprehensive overview of energy consumption across the palm oil industry, the energy consumption of palm oil milling, refining, biodiesel, and glycerin production is given in Table 1. To address this demand, power plants inside the palm oil industry incorporate cogeneration systems, which generate electricity and utilize thermal energy from the same fuel source [51]. Figure 4 demonstrates the comparison of energy usage between conventional generation and cogeneration systems.

Table 1. Energy consumption in each process of the palm oil industry

Process	Energy consumption
	Electricity (kWh)	Steam (MJ)
Palm oil milling	18.7/t FFB [9]	1090/t FFB
Palm oil refinery	6.3/t CPO [44]	1723/t CPO
Biodiesel and glycerin production	256.5/t RBDPO [52]	1980/t RBDPO

Fig. 4 [Images not available. See PDF.]

A systematic diagram of a conventional and b cogeneration systems

Renewable energy sources such as biomass, biogas, and PV play key roles in these cogeneration systems. Specifically, biomass from the palm oil milling process serves as a primary fuel through direct combustion. Biomass derived from the palm oil milling process is directly combusted in a traditional fire-tube steam boiler (Fig. 5a), whereas biogas powers a dedicated gas engine (model CAT CG170-20, 2000 kW, 480 V, 60 Hz, Fig. 5b). The detailed characteristics of the biomass properties used for combustion are enumerated in Table 2. This combustion triggers a rapid chemical reaction between biomass and oxygen, releasing significant heat based on the heating values of the biomass, as depicted in Fig. 5. This heat is used for producing high-temperature and high-pressure steam, which drives a steam turbine to generate electricity. After driving the turbine, the residual heat from the steam is recovered with a heat exchanger to generate the steam for production. Table 3 shows the component specifications for the cogeneration systems, including a boiler power output of 5 MW and a gas engine output of 2 MW, while the storage capacities for biomass, biogas, and batteries are 500 t, 10,000 m³, and 3 MW, respectively.

Fig. 5 [Images not available. See PDF.]

Process flow diagram of components in power plant of cogeneration system including a steam boiler and b gas engine generator

Table 2. Characteristics of biomass components

Biomass	EFB	PPF	PKS
Proximate analysis (wt.%)
Moisture	8.49	5.96	4.62
Volatile matter	70.54	68.81	69.24
Fixed carbon	15.36	15.79	15.66
Ash	5.61	9.44	10.48
Ultimate analysis (wt.%)
Carbon	40.93	43.19	41.33
Hydrogen	5.42	5.24	4.57
Nitrogen	1.56	1.59	0.99
Sulfur	0.31	0.19	0.09
Oxygen	51.78	49.79	53.02
Low heating value (MJ/kg)	16.50	19.00	16.80
Reference	[53]	[53]	[53]

Table 3. Specifications of power plant components

Specification	Unit	Value
Boiler power output	MW	5
Gas engine power output	MW	2
Biomass storage	t	500
Biogas storage	m3	10,000
Battery storage	MW	3

Biogas obtained from POME through anaerobic digestion achieves a high methane content after processing, where the specification of POME and biogas is shown in Table 4. The methane-rich biogas powers a gas engine, generating electricity for plant operations. The high-temperature exhaust gas produced by the engine is recovered by a waste heat boiler, transforming the residue heat into steam or hot water that is then utilized throughout the production process. The final source of energy is PV. As an alternative solar power solution, PV panels significantly reduce electricity costs for the palm oil industry by capturing solar energy, which contributes to a reduced environmental footprint.

Table 4. Specifications of POME and biogas

Specification	Unit	Value
POME density	kg/m3	980
Biogas produced	m3/t POME	23
Methane percentage	mol.%	50
Carbon dioxide percentage	mol.%	27
Nitrogen percentage	mol.%	23
Low heating value	MJ/kg	19.94

Climatic variability

Climate variation is one factor that significantly influences renewable energy production. The influence of climatic variability on renewable energy systems is comprehensively demonstrated by the data in Figs. 6 and 7. Figure 6 visualizes the subtleties of weather impacts on temperature, humidity, and solar radiation over time as they are essential factors for renewable energy production. Figure 7 illustrates scenario-based visual data across a spectrum of weather conditions, showing the extremes of precipitation, relative humidity, and ambient temperature for each defined scenario.

Fig. 6 [Images not available. See PDF.]

Impact of climatic variability on a solar radiation, b environmental parameters, and c cloud characteristics

Fig. 7 [Images not available. See PDF.]

Geospatial distribution of climatic factors influencing renewable energy production in scenarios of a-f-k clear and sunny, b-g-l partly cloudy, c-h-m overcast, d-i-n rain, and e-j-o thunderstorm

Clear and sunny conditions give the highest solar PV efficiency due to higher temperatures around 30 to 40 °C (Fig. 6a). The low and stable cloud cover of 20% substantially enhances the consistency of solar radiation, which is critical for maximizing PV output. In a partially cloudy scenario, the cloud cover is unstable and oscillates between 20 and 60% (Fig. 6b). This results in fluctuations in solar radiation reaching the solar panels at specific times compared to ideal cases such as sunny. Additionally, the variations in temperature and humidity associated with changing cloud cover can further impact system performance. Then, the overcast scenario is closely similar to the partially cloudy scenario. However, there are still distinctive differences. The temperature of the overcast scenario is lower than in previous weather conditions, while the peak cloud cover period is higher and longer, as observed by the amplitude and the area of cloud cover. This extended period of dense cloud cover significantly reduces solar radiation levels, leading to lower PV output. In the rainy-day scenario, despite persistent rainfall, the cloud cover amplitude is not as extreme as in a thunderstorm scenario, allowing for brief periods when solar radiation can still contact the panels. Although the temperature is low and the humidity is high, these conditions, combined with shorter periods of dense cloud cover, can result in slightly higher solar PV outputs than overcast scenarios. In contrast, the thunderstorm scenario involves even higher cloud cover amplitudes with frequent lightning and strong winds (Fig. 6c), leading to the lowest solar outputs.

Figure 8 represents the hourly variability and accumulation in solar PV generation under different weather conditions over a 24-h period. Under clear and sunny conditions, the solar PV output peaks impressively, reaching a total of 7247 kWh, with the highest hourly generation observed around midday. This peak indicates optimal solar panel performance due to direct sunlight and high temperatures. In partially cloudy conditions, the total generation decreases to 5325 kWh, with noticeable fluctuations throughout the day as clouds intermittently block the radiation. The overcast scenario results in a more significant drop in solar generation, totaling 2812 kWh, with a more uniform reduction across daylight hours. Rainy conditions bring about an even steeper decrease in output, accumulating a total of 2,908 kWh, as persistent rain and cloud cover disperse solar radiation. The most dramatic reduction is noted during thunderstorm conditions, where the PV generation plunges to a total of 810 kWh. Under extreme weather conditions, the power plant needs to generate more energy from biomass and biogas to compensate for the reduced output from solar panels during the thunderstorm scenario. This necessitates the integration of a resilient energy management system capable of swiftly switching between different energy sources in response to real-time weather conditions and energy demands.

Fig. 8 [Images not available. See PDF.]

Comparative analysis of a PV generation and b accumulation by weather conditions

Methodology

RL is one type of machine learning suitable for problem-solving that one will get the most benefit from, while there are many options without exact solutions [54]. Analogous to the way humans learn from Analogous to the way humans learn from trial and error, RL involves an agent that interacts with its environment, performing actions that result in states and associated rewards or penalties. RL features an agent interacting with an environment, carrying out actions that lead to various states accompanied by rewards or penalties [55]. The framework of RL is built on five fundamental elements: agents, environments, actions, states, and rewards. The primary objective of the RL framework is to find policies that maximize cumulative rewards over time. Achieving this involves a careful balance: the agent is tasked with exploring new actions that could potentially offer greater rewards and exploiting established actions known to provide consistent benefits.

Q-learning

Q-learning is a model-free reinforcement learning that operates independently of a model of the environment, utilizing the framework of Markov Decision Processes (MDP) to determine the optimal policy for decision-making [56]. The primary mechanism of Q-learning is an iterative update of the Q-values, which estimate the expected future rewards of taking certain actions in given states [57]. The Q-learning update rule can be expressed in a simplified form as follows:

1

$$Q(s_t,a_t)=Q(s_t,a_t)+\alpha [R_{t+1}+\gamma \underset{a}{max}Q(s_{t+1},a^{\prime })-Q(s_t,a_t)]$$where $s_t$ is the current state, $a_t$ is the action taken in the state $s_t$, $R_{t+1}$ is the instant reward after the agent takes action $a$ at the state $s$, $\gamma$ the discount factor, $\alpha$ is the learning rate, and $\underset{a}{max}Q(s_{t+1},a^{\prime })$ is the max Q-value at the next state $s_{t+1}$, considering all possible actions $a^{\prime }$.

The procedure for training an agent using Q-learning involves initially setting all Q-values to a baseline value, often zero. The agent then interacts with its environment in a series of episodes. In each episode, the agent makes decisions based on the current policy (which is derived from the Q-values, such as an ε-greedy policy), observes the outcome, and updates the Q-values according to Eq. (1). This process of exploration and learning continues, with the agent updating its Q-values at each timestep based on its experiences. Learning progresses through trial and error, allowing the agent to refine its policy and approach to achieving the highest possible long-term reward. The goal of Q-learning is to converge on stable Q-values for all state-action pairs, representing the optimal policy for maximizing cumulative future rewards. This convergence allows the agent to consistently make decisions that yield the highest expected rewards based on its learned experiences.

Deep Q-Networks (DQN)

DQN uses a deep neural network to approximate the Q-value function. The network takes the state of the environment as input and returns a vector of Q-values, one for each possible action, as visualized in Fig. 9. This setup substantially reduces the need for a tabular representation of Q-values, which becomes infeasible in environments with large or continuous state spaces. To stabilize the training process, DQN utilizes a replay buffer that stores transitions from the experiences of the agent. These transitions are randomly sampled to update the network weighting factors for avoiding local minima and reducing correlations between sequential transitions [58], where the target for updates, y, is defined in Eq. (2).

2

$$y=R_{t+1}+\gamma \underset{a}{max}{Q}_{\text{target}}(s_{t+1},a^{\prime })$$

Fig. 9 [Images not available. See PDF.]

Architecture of a DQN and throughput from state to Q-values

This target uses the discount factor and the reward, alongside the maximum Q-value from the next state, to calculate what the Q-value for the current state-action pair should ideally be. The training of the neural network involves minimizing the loss ($L$) between the predicted Q-values and target values ($y$) [59], calculated by the mean squared error given in Eq. (3).

3

$$L={(y-Q(s_t,a_t))}^2$$

By iteratively updating these values, the DQN learns to develop a policy under which the agent optimizes its decisions to maximize cumulative rewards. The overall framework for the DQN algorithm can be summarized in Fig. 10.

Fig. 10 [Images not available. See PDF.]

Illustration of interaction between the environment, Q-network, target Q-network, replay memory, and loss for DQN algorithms

Reinforcement learning for renewable energy management

A critical challenge in the energy resource management of the palm oil industry is meeting dynamic energy demands while maintaining efficient control over energy storage throughout various production stages [60]. Traditionally, this process relied heavily on control room operators who made decisions based on their expertise, experience, and analysis of historical data [61]. Operators monitor real-time data from industrial processes, interpret these observations, and interact with a Distributed Control System (DCS) to adjust process variables. Their actions aim to ensure optimal energy production and effective energy storage management, as depicted in Fig. 11a.

Fig. 11 [Images not available. See PDF.]

Comparison between methods of a conventional process optimization and b reinforcement learning optimization

The palm oil industry faces allocating complexity due to the presence of multiple energy sources, each with varying energy outputs and distinct environmental effects. Strategically combining the production capabilities of these diverse energy sources to fulfill fluctuating energy demands presents a significant logistical challenge. To address this complexity, RL offers a structured framework for decision-making, as illustrated in Fig. 11b. The framework shows the synergy between the state of the environment, the potential actions available from the agent, and the resulting rewards. The environment presents the agent with states that capture the current energy scenario, the action variables represent the possible interventions the agent can take, and the reward system evaluates the outcomes to guide the agent toward the most beneficial strategies.

Environment

Power plants and energy storage systems are key parameters in the allocation and control of energy resources in the palm oil industry. Therefore, power plants and energy storage systems are defined as environments for effective management of renewable energy sources. To accurately model this environment, a comprehensive neural network was developed and trained on approximately 2000 simulated data points that encapsulated the operational states and conditions of power plants and energy storage systems. The network architecture was constructed to predict the impact of various actions on load management on various types of power plants, including a biomass steam boiler and a gas engine generator. The input variables include key parameters such as ambient temperature, relative humidity, and specific mass flow rates for each power plant (flow rate of biomass for steam boiler and flow rate of biogas for gas engine), while the output variables provide evaluation on power output, steam output, and CO₂ emissions. Table 5 defines the specific input and output variables employed in constructing the environment model.

Table 5. List of input and output variables for the environment model

Power plant	Input variable	Output variable
Biomass steam boiler	X1 Ambient temperature (°C)	Y1 Biomass mass flow (kg/h)
	X2 Relative humidity (%)	Y2 Steam output (kg/h)
	X3 Power output (kWh)	Y3 CO2 emissions (kg/h)
Gas engine generator	X1 Ambient temperature (°C)	Y1 Biogas mass flow (kg/h)
	X2 Relative humidity (%)	Y2 Steam output (kg/h)
	X3 Power output (kWh)	Y3 CO2 emissions (kg/h)

State and action variables

Energy demand in the industrial sector is intrinsically related to production rates, which can vary by the hour. Accordingly, agent actions within the system are designed to correspond with the critical decisions required at each production hour. The agent dynamically adjusts the output from diverse energy sources, including biomass steam boilers, gas engine generators, and a PV battery storage, to support the production process and hourly energy requirements.

In the DQN model, the state represents a composite of variables such as current energy consumption, the status of energy storage, and prevailing weather conditions during a given period—all illustrated in Table 6. These state variables provide a comprehensive view of the current situation, enabling agents to gauge and execute the most appropriate actions. Action in palm oil industry management refers to the choice of agent at any given state—whether to increase, maintain, or decrease energy output from the available sources. The step sizes for these adjustments are specified as follows: the biomass steam boiler can be adjusted in increments of 5%, the gas engine generator in increments of 10%, and the PV battery storage in increments of 10%. These percentages represent the intervals of action within the RL model, dictating the thoroughness with which agents can modify energy outputs. The total number of possible actions in the DQN model can be calculated by considering the range of adjustments each energy source can make, divided by its respective step size, and then multiplying the number of options for each source. For this study, the possible action is 2541, where the multi-dimensional action space allows the RL agents to explore various action combinations to optimize overall energy efficiency and maintain stable production levels.

Table 6. A list of action and state variables for the environment model

Element	Variable	Range	Domain
Action	Capacity of the biomass steam boiler (%)	[0, 100]	Discrete
	Capacity of the gas engine generator (%)	[0, 100]	Discrete
	Capacity of the PV battery storage (%)	[0, 100]	Discrete
State	Ambient temperature (°C)	[0, 40]	Continuous
	Relative humidity (%)	[0, 100]	Continuous
	Electricity consumption (MWh)	[0, 10]	Continuous
	Steam consumption (t/h)	[0, 52]	Continuous
	Biomass storage (t)	[0, 500]	Continuous
	Biogas storage (m3)	[0, 10,000]	Continuous
	PV battery storage (kW)	[0, 3000]	Continuous

Reward system

In renewable energy management, the reward system focuses on three key points: generating enough energy to sustain present energy consumption, managing energy storage systems for energy generation, and reducing CO₂ emissions. The first term in reward is the Electricity Demand Reward (EDR) calculated using Eq. (4).

4

$$EDR=k_1·max(0,P_d-(P_{\mathit{bm}}+P_{\mathit{bg}}+P_{\mathit{PV}}))$$

EDR penalizes the DQN model proportionally to the scale of overproduction resulting from its actions, aligning the generated electricity ($P$)—with subscripts denoting the sources biomass ($\mathit{bm}$), biogas ($\mathit{bg}$), and photovoltaics ($\mathit{PV}$)—precisely with the consumption demand ($d$). Such a design discourages excessive energy generation, minimizing waste and avoiding the inefficiencies associated with surplus energy storage. The second term in the reward function is the Steam Demand Reward (SDR) shown in Eq. (5).

5

$$SDR=k_2·max(0,S{T}_d-(S{T}_{\mathit{bm}}+S{T}_{\mathit{bg}}))$$

Similar to the EDR, the SDR penalizes the DQN model for deviations in steam production relative to the demand for the palm oil industry. If the quantity of steam produced ($\mathit{ST}$) exceeds what is required for the production processes, the SDR applies a penalty to guide the model toward the desired balance. Also, the second term is interconnected to the first term. In the palm oil industry, the exhaust gas, or the residual heat from power generation through turbines is often repurposed to generate steam. Therefore, a reduction in power output can inadvertently lead to a shortage in steam production. Lastly, the final term in the reward function is the Sustainability Energy Reward (SER) calculated using Eq. (6), which weighs the actions of the DQN model by the sustainability of the energy sources utilized.

6

$$SER=(k_3·P_{\mathit{bm}})+(k_4·P_{\mathit{bg}})+(k_5·P_{\mathit{PV}})$$

The SER incentivizes the use of renewable resources such as solar power over biogas and biomass, encouraging the model to favor actions that reduce the carbon footprint and promote environmental conservation. This term reflects the long-term goal of achieving a greener energy mix and aligns the DQN model actions with the overarching aim of sustainable industry practices. By combining these three terms together using Eq. (7), the reward system forms a comprehensive approach that not only prioritizes immediate operational efficiency but also steers the DQN model towards long-term sustainability goals.

7

$$R=\left\{\begin{array}{c}if{S}_{\mathit{bm}}^{\mathit{lb}}<S_{\mathit{bm}}<S_{\mathit{bm}}^{\mathit{ub}}and{S}_{\mathit{bg}}^{\mathit{lb}}<S_{\mathit{bg}}<S_{\mathit{bg}}^{\mathit{ub}}and{S}_{\mathit{PV}}^{\mathit{lb}}<S_{\mathit{PV}}<S_{\mathit{PV}}^{\mathit{ub}}\\ R=-EDR-SDR-SER\\ else\\ R=-1,000\\ \end{array}\right\}$$where $R$ is the total reward given to the DQN model, while $S^{\mathit{ub}}$ and $S^{\mathit{lb}}$ are the upper and lower bounds for the energy production state, respectively. The aggregation of EDR, SDR, and SER into a single reward function ensures that the actions of the model are evaluated against a balanced scorecard, encapsulating efficiency, reliability, and sustainability. The weighting constant for EDR ($k_1$), SDR ($k_2$), and SER ($k_3$ to $k_5$) is set to 10^–2, 10^–2, 10^–3, 10^–4, and 10^–5, respectively. However, in scenarios where a shortage occurs, reflecting a significant mismatch between supply and demand, the DQN model is penalized with a -1,000 penalty. This severe punitive measure emphasizes the commitment of the system to avoiding energy shortages as critically as it does overutilization.

Results and discussion

DQN agent training result

The results of the DQN agent obtained from training through a training set of 800 episodes are shown as a learning curve in Fig. 12. The initial phase of training, spanning from 0 to 300 episodes, shows a reward pattern characterized by frequent occurrences of significant negative rewards, indicating the initial exploration phase of an agent within the simulation environment. This stage, which is marked by rewards as low as − 24,000, is crucial to the RL process, as it demonstrates the agent learning the complexities of managing and storing energy resources to efficiently fulfill the varying energy demands at each state.

Fig. 12 [Images not available. See PDF.]

DQN Agent learning curve: episode rewards and average episode rewards

A gradual increase in rewards between episodes 300 and 400 indicates that the agent is starting to enhance its policy, as evidenced by the positive shift in the learning curve. The upward trajectory in the learning curve implies that the agent is progressively refining its strategy to maximize the cumulative reward. Post the 400-episode mark, there seems to be a stabilization in the reward pattern, indicating that the agent has begun to employ more optimized decision-making strategies. By the end of the 800-episode training period, the frequency and magnitude of penalties have notably decreased, evidencing that the agent has enhanced its understanding of the task and is closer to achieving its objective of effective energy management.

The effectiveness of the training time and processing speed of the proposed model are also practical. The model can be efficiently processed, with a prediction speed of 8.4 ms and a total training time of 514 s. These metrics, quantified using industrial-grade computers (NISE 3900E-H310, Intel Core i7-8700 T CPU, 8 GB DDR4 RAM, and a 256 GB SSD), demonstrate not only the feasibility of deploying the model in real-world industrial settings but also the generality of the proposed model in terms of its ability to handle complex energy management tasks swiftly and effectively.

Energy management under climatic variability

Figure 13 presents the outcomes of energy management in the palm oil sector, achieved after 800 training sessions with reinforcement learning agents. Each process within the industry showcases hourly fluctuations in production rates, leading to variable electricity and steam demand and the corresponding generation of biomass and biogas from palm oil milling (Fig. 13a). The horizontal axis of the figure marks the passage of time across a 24-h production schedule, captured under five distinct climatic scenarios: clear and sunny, partially cloudy, overcast, rainy, and thunderstorms, respectively. These scenarios represent the range of environmental conditions to which the energy management system must adapt, accentuating the complexity of the task managed by the agent. PV generation depends on climatic variability. The agent aims to optimize the hourly utilization of various renewable energy sources while being mindful of their storage limits. The energy output of various renewable energy sources can be compared based on their electricity generation capabilities. Contrary to a gas engine generator that produces 0.58 t of steam per h, a 1 MWh biomass steam boiler produces approximately 7.25 t of steam per h, with the photovoltaic array contributing no thermal energy. It should be noted that regardless of generating the highest stream, the biomass steam boiler has CO₂ emissions of 828 kg/MWh, followed by a gas engine generator with CO₂ emissions of 740 kg/MWh and a PV with no emissions.

Fig. 13 [Images not available. See PDF.]

Operational scheduling in terms of a production rate and energy management results in terms of b electricity, c steam, d actions taken by RL, and e energy storage percentage under climatic variability

From the results, the agent decides to select each energy source to generate electricity and steam in quantities that meet the hourly energy consumption of the production process while minimizing CO₂ emissions (Fig. 13d). The decision includes managing energy storage effectively to prevent failures and accommodate the storage capacity of each energy source as observed by the storage level that consistently remains below 100% across five climatic scenarios (Fig. 13e). When energy storage is high, the agent proactively discharges excess energy, thereby preventing over-storage and optimizing the overall system efficiency. This action is particularly critical in scenario 4, where the agent tactically reduces energy storage levels by increasing the supply to the process.

Moreover, a notable operational insight is the action of an agent to generate an electrical power surplus to immediate demand (Fig. 13b). This decision is a direct response to the associated shortfall in thermal energy, as the steam output is intrinsically linked to the electricity generation process. These actions, informed by the adaptive learning process of the agent, ensure that electricity generation is not curtailed to the point where it would result in a proportional decrease in the residual heat, which is essential for steam production (Fig. 13c). Acknowledging this interdependence, the agent opts to produce and store excess electricity, thus guaranteeing adequate residual heat for steam generation to meet the demand. This preemptive approach prevents potential steam shortages, adhering closely to the dual objectives of the system and CO₂ emission reduction, as specified by the parameters of the reward function.

Based on this two-fold evidence, the reinforcement learning agent demonstrates an advanced operational decision-making level. By maintaining storage levels within optimal thresholds and generating a surplus of electricity to ensure sufficient thermal energy, the agent exhibits a nuanced understanding of the interconnected nature of the energy system. The outcomes highlight the ability of agents to engage in complex energy trade-offs, reflecting a sophisticated application of AI algorithms to real-world industrial challenges.

Energy savings and CO₂ emission reduction

According to the International Energy Agency's emission report, the industrial sector contributed about 24.2% of global carbon emissions, equivalent to around 8.42 billion tons of CO₂ [62]. The production of CPO involved significant greenhouse gas emissions, estimated between 637 and 1131 kg CO₂eq per ton of CPO [63]. Various sources contributed to these emissions, with the treatment of POME alone contributing between 637 and 1094 kg CO₂eq per ton. Boiler emissions added another 41.28–170.2 kg CO₂eq per ton, depending on the technology and efficiency of the boiler systems used. Oil palms also sequestered carbon annually, which contributed to reducing the net emissions ranges between 1915 and 2393 kg CO₂eq per ton of CPO [64]. More efficient energy management could promote further reductions in the net carbon footprint of the palm oil industry by optimizing the use of energy and reducing waste in the production process.

The results of energy savings from diverse sources across various weather conditions reveal a distinct pattern of contributions to overall efficiency, as shown in Fig. 14. The biomass steam boiler stands out as the predominant source for both electricity and steam generation, consistently delivering an average output of 81.50% and 85.5% across all weather scenarios. In contrast, the gas engine generator shows more variability, with its output peaking at 20.9% for electricity and 18.3% for steam during thunderstorm conditions, indicating a dependency on this source when the weather is less favorable. On average, its contribution is noted at 15.6% for electricity and 14.5% for steam.

Fig. 14 [Images not available. See PDF.]

Electricity and steam generation from each renewable energy source under five climatic scenarios

Despite being highly sensitive to weather changes, the PV system plays a crucial role in energy savings. Under optimal, clear, and sunny conditions, PV systems demonstrate their highest efficiency, offering a 5% contribution to the energy savings of both the biomass steam boiler and the gas engine generator. Solar panels are most effective under direct sunlight, which is abundant during clear and sunny days, allowing them to convert more sunlight into electrical energy. The efficiency of PV systems diminishes with increasing cloud coverage as less sunlight reaches the solar cells. This efficiency slightly decreases to 4% under partly cloudy skies. A more noticeable efficiency drop is observed during overcast conditions, descending to 3%. The trend continues downward in rainy weather, culminating in a minimal contribution of 0.8% amid thunderstorms. Averaging these conditions, the PV system contributes an overall energy savings of 2.9%.

The results obtained from representative energy management under various climatic variations using three energy sources produce different CO₂ emissions. PV is an emissions-free energy source that can be used to calculate CO₂ emission reductions when compared to total electricity per total CO₂ emissions generated, which is 0.853 kg CO₂eq/kWh. The overall CO₂ emission reduction from using PV is 15,688 kg CO₂eq, or an average of 3,137 per day, using the hourly CO₂ emission reduction shown in Fig. 15a.

Fig. 15 [Images not available. See PDF.]

Analysis of CO₂ emissions reduction through renewable energy integration under a hourly emission reduction and b in climatic scenarios

Figure 15b illustrates how climate variability affects CO₂ emission reduction. Under the clear and sunny conditions, the PV system accounted for a CO₂ emission reduction of 5116 kg CO₂eq, making up 32.6% of the total reduction. Partly cloudy conditions resulted in a slight reduction of 4604 kg CO₂eq, accounting for 29.3% of the total. When the cloud coverage increased to the overcast condition, the trend of reducing greenhouse gas emissions continued, with overcast contributing 3070 kg CO₂eq, or 19.6% of the total emission. The decrease in emissions savings was more pronounced during rainy weather, with a reduction of 2046 kg, or 13% of the total emission. The least reduction in CO₂ emissions was found during thunderstorms, reduced by 852 kg, accounting for 5% of the total emission. The results show the significance of the RL framework in adjusting energy production strategies in real-time to adapt to changing weather conditions. By effectively responding to the variability in weather patterns, the RL framework ensures that energy production is optimized to yield the highest possible CO₂ emission reductions, irrespective of the prevailing climatic conditions.

Conclusion

In this work, the RL framework models are deployed to manage renewable energy sources in the palm oil industry with variable climatic conditions. The multi-decision-making ability of RL is adopted to manage the uncertain outcome from renewable energy sources such as solar, biomass and biogas, which are the primary energy sources in palm oil production. The following main conclusions were derived from the study:

1. Energy management under climatic variability using RL ensures the generation of sufficient electricity and steam throughout the production process while optimizing hourly storage management with particular attention to avoiding over-storage of three renewable energy sources (the highest storage for PV is 82.2%) and effectively adapting to all climatic variations that impact renewable energy production.

2. Energy savings and CO₂ reductions from RL measures of PV impact applied to different climates. Under sunny and clear conditions, the PV system showed the highest efficiency, contributing to 5% energy savings and a 32.6% reduction in CO₂ emissions, but this efficiency decreased slightly to 4% energy savings and 29.3% for CO₂ reduction in a partly cloudy condition. Overcast conditions reduced efficiency, reduced energy savings by 3%, and reduced CO₂ emissions by 19.6%. The decrease in energy savings and CO₂ reductions occurred during the rainy weather, with energy savings of 1.9% and CO₂ savings of 13%. Minimal contributions were found during thunderstorms, with minimum energy savings of 0.8% and CO₂ emission reductions of 5.4%.

Overall, this RL framework as an integrated method of AI in energy management shows promise in optimizing energy efficiency and energy storage in the palm oil industry by considering the impact and sustainability of the environment.

Acknowledgements

The author would like to acknowledge the support of the Faculty of Engineering, Kasetsart University (Grant No.66/04/CHEM/D.Eng), Center for Advanced Studies in Industrial Technology, and the Center of Excellence on Petrochemical and Materials Technology. The authors are grateful to Honeywell and GRD Tech Co., Ltd. for providing the UniSim Design Suite simulation software used in this study.

Author contribution

CP: Conceptualization, Methodology, Validation, Supervision, Writing-Original draft preparation, Writing-Reviewing and Editing, Funding acquisition. TS: Conceptualization, Methodology, Investigation, Software, Data Curation, Writing-Original draft preparation, Visualization. SB: Conceptualization, Methodology, Supervision, Formal Analysis, Writing—Original draft preparation, Writing-Reviewing and Editing. MAH: Supervision, Formal Analysis, Writing-Reviewing and Editing.

Data availability

Due to confidentiality constraints, the detailed datasets involving the biodiesel production process information and historical records of climate variations discussed in this research are not openly accessible. Interested parties can request this data from the corresponding author, provided they obtain the required approvals.

Declarations

Competing interests

The authors declare no competing interests.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

References

1. Indrawan, N; Thapa, S; Rahman, SF; Park, J-H; Park, S-H; Wijaya, ME et al. Palm biodiesel prospect in the Indonesian power sector. Environ Technol Innov; 2017; 7, pp. 110-127. [DOI: https://dx.doi.org/10.1016/j.eti.2017.01.001]

2. Purnomo, H; Okarda, B; Dermawan, A; Ilham, QP; Pacheco, P; Nurfatriani, F et al. Reconciling oil palm economic development and environmental conservation in Indonesia: a value chain dynamic approach. Forest Policy Econ; 2020; 111, 102089. [DOI: https://dx.doi.org/10.1016/j.forpol.2020.102089]

3. Nutongkaew, P; Waewsak, J; Kiratiwibool, W; Gagnon, Y. Demand and supply of crude palm oil for biodiesel production towards food and energy security. Appl Mech Mater; 2016; 839, pp. 151-158. [DOI: https://dx.doi.org/10.4028/www.scientific.net/AMM.839.151]

4. Husain, Z; Zainal, ZA; Abdullah, MZ. Analysis of biomass-residue-based cogeneration system in palm oil mills. Biomass Bioenerg; 2003; 24, pp. 117-124. [DOI: https://dx.doi.org/10.1016/S0961-9534(02)00101-0]

5. Mehrpooya, M; Ghorbani, B; Abedi, H. Biodiesel production integrated with glycerol steam reforming process, solid oxide fuel cell (SOFC) power plant. Energy Convers Manage; 2020; 206, 112467.[COI: 1:CAS:528:DC%2BB3cXhtF2mtbc%3D] [DOI: https://dx.doi.org/10.1016/j.enconman.2020.112467]

6. Cheah, WY; Siti-Dina, RP; Leng, STK; Er, AC; Show, PL. Circular bioeconomy in palm oil industry: current practices and future perspectives. Environ Technol Innov; 2023; 30, 103050. [DOI: https://dx.doi.org/10.1016/j.eti.2023.103050]

7. Yoshizaki, T; Shirai, Y; Hassan, MA; Baharuddin, AS; Raja Abdullah, NM; Sulaiman, A et al. Improved economic viability of integrated biogas energy and compost production for sustainable palm oil mill management. J Clean Prod; 2013; 44, pp. 1-7.[COI: 1:CAS:528:DC%2BC3sXktFahsLc%3D] [DOI: https://dx.doi.org/10.1016/j.jclepro.2012.12.007]

8. Rasid, NSA; Syed-Hassan, SSA; Kadir, SASA; Asadullah, M. Life cycle assessment to evaluate the green house gas emission from oil palm bio-oil based power plant. Korean J Chem Eng; 2013; 30, pp. 1277-1283. [DOI: https://dx.doi.org/10.1007/s11814-013-0022-y]

9. Patthanaissaranukool, W; Polprasert, C; Englande, AJ. Potential reduction of carbon emissions from Crude Palm Oil production based on energy and carbon balances. Appl Energy; 2013; 102, pp. 710-717.[COI: 1:CAS:528:DC%2BC38Xhslygu77L] [DOI: https://dx.doi.org/10.1016/j.apenergy.2012.08.023]

10. Ohimain, EI; Izah, SC. Energy self-sufficiency of smallholder oil palm processing in Nigeria. Renew Energy; 2014; 63, pp. 426-431. [DOI: https://dx.doi.org/10.1016/j.renene.2013.10.007]

11. Aziz, M; Kurniawan, T; Oda, T; Kashiwagi, T. Advanced power generation using biomass wastes from palm oil mills. Appl Therm Eng; 2017; 114, pp. 1378-1386.[COI: 1:CAS:528:DC%2BC28XhvVels7jL] [DOI: https://dx.doi.org/10.1016/j.applthermaleng.2016.11.031]

12. Trisakti, B; Manalu, V; Taslim, I; Turmuzi, M. Acidogenesis of palm oil mill effluent to produce biogas: effect of hydraulic retention time and pH. Procedia Soc Behav Sci; 2015; 195, pp. 2466-2474. [DOI: https://dx.doi.org/10.1016/j.sbspro.2015.06.293]

13. Wang, S; Yuan, R; Liu, C; Zhou, B. Effect of Fe2+ adding period on the biogas production and microbial community distribution during the dry anaerobic digestion process. Process Saf Environ Prot; 2020; 136, pp. 234-241.[COI: 1:CAS:528:DC%2BB3cXisF2lu7s%3D] [DOI: https://dx.doi.org/10.1016/j.psep.2019.12.031]

14. Trisakti, B; Adipasah, H; Turmuzi, M. Effect of agitation on acidogenesis stage of two-stage anaerobic digestion of palm oil mill effluent POME into Biogas. IOP Conf Ser Mater Sci Eng; 2017; 180, 12127. [DOI: https://dx.doi.org/10.1088/1757-899X/180/1/012127]

15. Zahedi, A. Large scale solar PV generation for Southeast Asian countries. 2017 Australasian Universities Power Engineering Conference (AUPEC); 2017; Melbourne, IEEE: pp. 1-5.

16. Eka Putra, RR; Afriani, S; Miefthawati, NP; Jelita, M. Analisis teknis-ekonomi pembangkit listrik tenaga hybrid solar PV/biogas off grid system. SITEKIN; 2020; 18, 17. [DOI: https://dx.doi.org/10.24014/sitekin.v18i1.11695]

17. Hiendro, A; Husin, F; Junaidi, J; Khwee, KH. Assessing the utilization of palm oil mill effluent in photovoltaic and biogas hybrid energy system for off-grid village. IJPEDS; 2024; 15, 454. [DOI: https://dx.doi.org/10.11591/ijpeds.v15.i1.pp454-465]

18. Rodrigues, TO; Caldeira-Pires, A; Luz, S; Frate, CA. GHG balance of crude palm oil for biodiesel production in the northern region of Brazil. Renew Energy; 2014; 62, pp. 516-521.[COI: 1:CAS:528:DC%2BC3sXhsFOkurjK] [DOI: https://dx.doi.org/10.1016/j.renene.2013.08.006]

19. Ismail, MI; Yunus, NA; Hashim, H. The challenges and opportunities of solar thermal for palm oil industry in Malaysia. Chem Eng Trans; 2020; 78, pp. 601-606.

20. Panjapornpon, C; Bardeeniz, S; Hussain, MA; Chomchai, P. Explainable deep transfer learning for energy efficiency prediction based on uncertainty detection and identification. Energy AI; 2023; 12, 100224. [DOI: https://dx.doi.org/10.1016/j.egyai.2022.100224]

21. Ho, WS; Khor, CS; Hashim, H; Lim, JS; Ashina, S; Herran, DS. Optimal operation of a distributed energy generation system for a sustainable palm oil-based eco-community. Clean Techn Environ Policy; 2015; 17, pp. 1597-1617.[COI: 1:CAS:528:DC%2BC2cXitFKls73F] [DOI: https://dx.doi.org/10.1007/s10098-014-0893-6]

22. Jitchaiyapoom, T; Panjapornpon, C; Bardeeniz, S; Hussain, MA. Production capacity prediction and optimization in the glycerin purification process: a simulation-assisted few-shot learning approach. Processes; 2024; 12, 661.[COI: 1:CAS:528:DC%2BB2cXhtFagtr%2FM] [DOI: https://dx.doi.org/10.3390/pr12040661]

23. Bardeeniz, S; Panjapornpon, C; Fongsamut, C; Ngaotrakanwiwat, P; Azlan, HM. Digital twin-aided transfer learning for energy efficiency optimization of thermal spray dryers: leveraging shared drying characteristics across chemicals with limited data. Appl Therm Eng; 2024; 242, 122431.[COI: 1:CAS:528:DC%2BB2cXhslantrc%3D] [DOI: https://dx.doi.org/10.1016/j.applthermaleng.2024.122431]

24. Wu, Q; Qiang, TC; Zeng, G; Zhang, H; Huang, Y; Wang, Y. Sustainable and renewable energy from biomass wastes in palm oil industry: a case study in Malaysia. Int J Hydrogen Energy; 2017; 42, pp. 23871-23877.[COI: 1:CAS:528:DC%2BC2sXlvVSjsLY%3D] [DOI: https://dx.doi.org/10.1016/j.ijhydene.2017.03.147]

25. Oliveira, RCD; Silva, RDDSE; Tostes, MEDL. A methodology for analysis of cogeneration projects using oil palm biomass wastes as an energy source in the Amazon. DYNA; 2015; 82, pp. 105-112. [DOI: https://dx.doi.org/10.15446/dyna.v82n190.43298]

26. Rana, S; Singh, L; Wahid, Z; Liu, H. A recent overview of palm oil mill effluent management via bioreactor configurations. Curr Pollution Rep; 2017; 3, pp. 254-267.[COI: 1:CAS:528:DC%2BC2sXhtlymurvL] [DOI: https://dx.doi.org/10.1007/s40726-017-0068-2]

27. Akhtar, MN; Ansari, E; Alhady, SSN; Abu, BE. Leveraging on advanced remote sensing- and artificial intelligence-based technologies to manage palm oil plantation for current global scenario: a review. Agriculture; 2023; 13, 504. [DOI: https://dx.doi.org/10.3390/agriculture13020504]

28. Gnagbolou, Y; Agbomahena, MB; Azanandji, MR; Ngobi, GK. Correlation between photovoltaic energy production and certain climate parameters: case study in the plateau department in southern Benin. J Energy Res Rev; 2023; 14, pp. 39-47. [DOI: https://dx.doi.org/10.9734/jenrr/2023/v14i1277]

29. Jackson, ND; Gunda, T. Evaluation of extreme weather impacts on utility-scale photovoltaic plant performance in the United States. Appl Energy; 2021; 302, 117508. [DOI: https://dx.doi.org/10.1016/j.apenergy.2021.117508]

30. Beaudin, M; Zareipour, H; Schellenberglabe, A; Rosehart, W. Energy storage for mitigating the variability of renewable electricity sources: an updated review. Energy Sustain Dev; 2010; 14, pp. 302-314. [DOI: https://dx.doi.org/10.1016/j.esd.2010.09.007]

31. Azwan, MB; Norasikin, AL; Sopian, K; Abd Rahim, S; Norman, K; Ramdhan, K et al. Assessment of electric vehicle and photovoltaic integration for oil palm mechanisation practise. J Clean Prod; 2017; 140, pp. 1365-1375.[COI: 1:CAS:528:DC%2BC28Xhs1ygsrjP] [DOI: https://dx.doi.org/10.1016/j.jclepro.2016.10.016]

32. Freitas, E; Salgado, J; Alnoch, R; Contato, A; Habermann, E; Michelin, M et al. Challenges of biomass utilization for bioenergy in a climate change scenario. Biology; 2021; 10, 1277.[COI: 1:CAS:528:DC%2BB38Xksl2qsLg%3D] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/34943192][PubMedCentral: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8698859][DOI: https://dx.doi.org/10.3390/biology10121277]

33. Suksong, W; Tukanghan, W; Promnuan, K; Kongjan, P; Reungsang, A; Insam, H et al. Biogas production from palm oil mill effluent and empty fruit bunches by coupled liquid and solid-state anaerobic digestion. Biores Technol; 2020; 296,[COI: 1:CAS:528:DC%2BC1MXitFantrjN] [DOI: https://dx.doi.org/10.1016/j.biortech.2019.122304]

34. Mason, K; Grijalva, S. A review of reinforcement learning for autonomous building energy management. Comput Electr Eng; 2019; 78, pp. 300-312. [DOI: https://dx.doi.org/10.1016/j.compeleceng.2019.07.019]

35. Panjapornpon, C; Bardeeniz, S; Hussain, MA. Deep learning approach for energy efficiency prediction with signal monitoring reliability for a vinyl chloride monomer process. Reliab Eng Syst Saf; 2023; 231, 109008. [DOI: https://dx.doi.org/10.1016/j.ress.2022.109008]

36. Shouryadhar K, Chen C-N, Chen Y-C. An enhanced reinforcement learning based approach of energy management optimization for microgrids. 2022 IET International Conference on Engineering Technologies and Applications (IET-ICETA). 2022. 1–2. https://ieeexplore.ieee.org/document/9971705. Accessed 24 Apr 2024.

37. Nakabi, TA; Toivanen, P. Deep reinforcement learning for energy management in a Microgrid with flexible demand. Sustain Energy Grids Networks; 2021; 25, 100413. [DOI: https://dx.doi.org/10.1016/j.segan.2020.100413]

38. Ahrarinouri, M; Rastegar, M; Seifi, AR. Multiagent reinforcement learning for energy management in residential buildings. IEEE Trans Industr Inf; 2021; 17, pp. 659-666. [DOI: https://dx.doi.org/10.1109/TII.2020.2977104]

39. Ibrahim, HA; Zaidan, AA; Qahtan, S; Zaidan, BB. Sustainability assessment of palm oil industry 4.0 technologies in a circular economy applications based on interval-valued pythagorean fuzzy rough set-FWZIC and EDAS methods. Appl Soft Comput; 2023; 136, 110073. [DOI: https://dx.doi.org/10.1016/j.asoc.2023.110073]

40. Jiao, J; Yang, J; Ding, T. Factor allocation efficiency of environmental protection industry in China—Based on the perspective of micro enterprises. J Environ Manage; 2023; 348, 119385. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/37925981][DOI: https://dx.doi.org/10.1016/j.jenvman.2023.119385]

41. Faisal, MFM; Mahidin, M. Biomass residue from palm oil mills in Aceh province: a potential usage for sustainable energy. Int J Adv Sci Eng Inform Technol; 2013; 3, 222. [DOI: https://dx.doi.org/10.18517/ijaseit.3.3.324]

42. Chavalparit, O; Rulkens, WH; Mol, APJ; Khaodhair, S. Options for environmental sustainability of the crude palm oil industry in Thailand through enhancement of industrial ecosystems. Environ Dev Sustain; 2006; 8, pp. 271-287. [DOI: https://dx.doi.org/10.1007/s10668-005-9018-z]

43. Aziz, M; Oda, T; Kashiwagi, T. Design and analysis of energy-efficient integrated crude palm oil and palm kernel oil processes. J Jpn Inst Energy; 2015; 94, pp. 143-150.[COI: 1:CAS:528:DC%2BC2MXmt1aksrw%3D] [DOI: https://dx.doi.org/10.3775/jie.94.143]

44. Sulaiman, MA; Oni, AO; Fadare, DA. Energy and exergy analysis of a vegetable oil refinery. Energy Power Eng; 2012; 4, pp. 358-364. [DOI: https://dx.doi.org/10.4236/epe.2012.45047]

45. Lidu, SR; Mohamed, N; Klemeš, JJ; Varbanov, PS; Yusup, S. Evaluation of the energy saving opportunities for palm oil refining process: Sahabat Oil Products (SOP) in Lahad Datu, Malaysia. Clean Techn Environ Policy; 2016; 18, pp. 2453-2465.[COI: 1:CAS:528:DC%2BC28Xht1Gjtb%2FP] [DOI: https://dx.doi.org/10.1007/s10098-016-1252-6]

46. De Greyt, W. Edible oil refining: current and future technologies. Edible oil processing; 2013; Hoboken, John Wiley & Sons Ltd: pp. 127-151. [DOI: https://dx.doi.org/10.1002/9781118535202.ch5]

47. Trirahayu, DA; Abidin, AZ; Putra, RP; Hidayat, AS; Safitri, E; Perdana, MI. Process simulation and design considerations for biodiesel production from rubber seed oil. Fuels; 2022; 3, pp. 563-579.[COI: 1:CAS:528:DC%2BB3sXpvFels70%3D] [DOI: https://dx.doi.org/10.3390/fuels3040034]

48. Figueroa-Jimenez, S; Gamarra-Torres, J; Bonilla, D; Peralta-Ruiz, Y. Evaluation of biodiesel production process from palm oil (eleais guineensis) using exergy analysis methodology. Chem Eng Trans; 2015; 43, pp. 529-534.

49. Pirouzfar, V; Sakhaeinia, H; Su, C-H. Power generation using produced biodiesel from palm oil with GTG, STG and combined cycles; process simulation with economic consideration. Fuel; 2022; 314, 123084.[COI: 1:CAS:528:DC%2BB38XksFOntA%3D%3D] [DOI: https://dx.doi.org/10.1016/j.fuel.2021.123084]

50. Kianimanesh, H; Abbaspour-Aghdam, F; Valizadehderakhshan, M. Biodiesel production from vegetable oil: process design, evaluation and optimization. Polish J Chem Technol; 2017; [DOI: https://dx.doi.org/10.1515/pjct-2017-0048]

51. Choque Campero, LA; Wang, W; Martin, A. Thermodynamic and exergetic analyses of a biomass-fired Brayton-Stirling cogeneration cycle for decentralized, rural applications. Energy Convers Manage; 2023; 292, 117350. [DOI: https://dx.doi.org/10.1016/j.enconman.2023.117350]

52. Wahyono, Y; Hadiyanto, H; Budihardjo, M; Widayat, W. Energy balance calculation with life cycle assessment for production of palm biodiesel in indonesia. E3S Web Conf; 2019; 125, 10005.[COI: 1:CAS:528:DC%2BB3cXntFWlsLc%3D] [DOI: https://dx.doi.org/10.1051/e3sconf/201912510005]

53. Idris, SS; Rahman, NA; Ismail, K; Mohammed Yunus, MF; Mohd Hakimi, NIN. Microwave-assisted pyrolysis of oil palm biomass: multi-optimisation of solid char yield and its calorific value using response surface methodology. Front Chem Eng; 2022; [DOI: https://dx.doi.org/10.3389/fceng.2022.864589]

54. Panjapornpon, C; Chinchalongporn, P; Bardeeniz, S; Makkayatorn, R; Wongpunnawat, W. Reinforcement learning control with deep deterministic policy gradient algorithm for multivariable pH Process. Processes; 2022; 10, 2514. [DOI: https://dx.doi.org/10.3390/pr10122514]

55. Fayyazi, M; Abdoos, M; Phan, D; Golafrouz, M; Jalili, M; Jazar, RN et al. Real-time self-adaptive Q-learning controller for energy management of conventional autonomous vehicles. Expert Syst Appl; 2023; 222, 119770. [DOI: https://dx.doi.org/10.1016/j.eswa.2023.119770]

56. Guan, Y; Li, SE; Duan, J; Li, J; Ren, Y; Sun, Q et al. Direct and indirect reinforcement learning. Int J Intell Syst; 2021; 36, pp. 4439-4467. [DOI: https://dx.doi.org/10.1002/int.22466]

57. Pouyan M, Mousavi A, Golzari S, Hatam A. Improving the performance of Q-learning using simultanouse Q-values updating. 2014 International Congress on Technology, Communication and Knowledge (ICTCK). 2014. 1–6. https://ieeexplore.ieee.org/document/7033528. Accessed 30 Apr 2024.

58. Ohnishi, S; Uchibe, E; Yamaguchi, Y; Nakanishi, K; Yasui, Y; Ishii, S. Constrained deep Q-learning gradually approaching ordinary Q-learning. Front Neurorobot; 2019; 13, 103. [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/31920613][PubMedCentral: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6914867][DOI: https://dx.doi.org/10.3389/fnbot.2019.00103]

59. Nair A, Srinivasan P, Blackwell S, Alcicek C, Fearon R, De Maria A, et al. Massively parallel methods for deep reinforcement learning. arXiv. 2015. http://arxiv.org/abs/1507.04296. Accessed 26 Jul 2024.

60. Alarfaj, O; Bhattacharya, K. Material flow based power demand modeling of an oil refinery process for optimal energy management. IEEE Trans Power Syst; 2019; 34, pp. 2312-2321. [DOI: https://dx.doi.org/10.1109/TPWRS.2018.2889773]

61. Adams, D; Oh, D-H; Kim, D-W; Lee, C-H; Oh, M. Deep reinforcement learning optimization framework for a power generation plant considering performance and environmental issues. J Clean Prod; 2021; 291, 125915.[COI: 1:CAS:528:DC%2BB3MXisVyntLg%3D] [DOI: https://dx.doi.org/10.1016/j.jclepro.2021.125915]

62. International Energy Agency. Global energy review 2020; 2020; Paris, IEA: [DOI: https://dx.doi.org/10.1787/a60abbf2-en]

63. Hong WO. Review on the carbon footprint of the palm oil industry: insights into recent developments. 2022. https://www.preprints.org/manuscript/202211.0329/v1. Accessed 26 Jul 2024.

64. Sundaraja, CS; Hine, DW; Lykins, AD. Palm oil: understanding barriers to sustainable consumption. PLoS ONE; 2021; 16, e0254897.[COI: 1:CAS:528:DC%2BB3MXhvVOhtr3L] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/34407066][PubMedCentral: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8372893][DOI: https://dx.doi.org/10.1371/journal.pone.0254897]