In recent years, the reverse logistics framework allowing the unified analysis of manufacturing planning and the inventory management has gained a substantial interest among the researchers working in the field. In a book [1] author described the quantitative models to represent the activities of remanufacturing and recycling in the context of reverse logistics emphasizing three issues: namely: distribution planning, inventory management and production planning. In the survey [2] authors analyzed more than sixty case studies in reverse logistics published between 1984 and 2002 and discussed network structures and activities related to the recovery of products up to the end of life. Various optimization models for supply chains with a recovery of returned products have been proposed with special attention to the production control and inventory management using both, deterministic and stochastic approaches. In the majority of previous studies discrete time (as opposed to continuous time) settings is used. In [3] authors present an effective approach to determine the discrete policy of the optimal control for a system with product recovery, taking into account the uncertainty in the demand of new and returned products. They model the demand and the return as discrete independent random variables. In [4] a new discrete stochastic inventory model for a hybrid system is proposed: new and returned products are manufactured separately, the demands are independent but production policies are synchronized. Authors of [5] develop a periodic inventory model of on a finite planning horizon with consideration of production remanufacturing and disposal activities. In [6] authors propose a model of discrete time stochastic optimization for a hybrid system taking into account, the production, subcontracting, remanufacturing of returned products, return market of poor quality products the production line and disposal activities. The demand is a random variable normally distributed and the return of products depends on the demand. A continuous time optimization model is considered in [7] for the production, remanufacturing and disposal in a dynamic deterministic settings.

Manufacturing systems subject to random breakdowns and repairs were systematically analyzed in [8-10] in continuous time using stochastic optimization technique. Recently in [11] this methodology has been extended to address the global performances of the manufacturing system with the supply chain in closed loop. A stochastic dynamic system consisted of two machines dedicated respectively to manufacturing and remanufacturing; the random phenomena are breakdowns and repairs of the machines, the demand of new products was considered deterministic and known, the returned product was a portion of this demand.

The constant demand is a prevailing assumption in the large body of the research devoted to stochastic continuous time optimization of production management in failure-prone systems. Some papers develop optimality conditions and use them for searching numerical solutions [12]; others present analytical solutions as the recent article [13]. In much fewer studies where the random demand is analyzed—it is most often modeled as a Poisson process. This approach allows to keep the usual framework of random discrete events changing the state of the system for both machine breakdowns and demand jumps [14]. Poisson-type demand is used more systematically in inventory optimization problems [15]. A combined model: Poisson process coupled with the diffusion process has been recently proposed in [16] for modeling the demand in inventory problem. In fact diffusion-type processes were used for modeling the demand in the classical paper [8] were optimality conditions have been obtained, however it was the only source of random behavior since the machine breakdowns were not considered.

The system considered in this paper contains reverse logistics loop with manufacturing and remanufacturing branches revisiting the model proposed in [14]. We use continuous time stochastic control approach and adopt the diffusion-type component into the demand model merging this source of random behavior with random machine breakdowns described by Poisson process as in [9,10]. As a direct consequence of an adopted demand model the optimality conditions lead to the HamiltonJacobi-Bellman (HJB) equation of the second order. Second order HJB is often met in option price modeling, but for stochastic control in manufacturing systems the HJB is usually of the first order [9,10,14]. Analyzing the second order HJB we use the Kushner finite difference approximations and the policy improvement algorithm [17].

The paper is structured as follows. In Section 2 we describe the model of the hybrid system consisting of 2 machines. The first machine uses primary product, and the second—returned product; both are subject to breakdowns and repairs constituting the first source of uncertainty. We describe in details our demand model using diffusion type random processes constructed as an output of shaping filter excited by the white noise. We study 2 versions of such model simple Brownian motion and first order Markovian process. Latter version seems more realistically fit the real world situations. In Section 3 we derive optimality conditions in the form of Hamilton-Jacobi-Bellman (HJB) equations which are second order partial differential equations (PDEs) for the chosen demand model. In Section 4 we describe the numerical method based on finite difference approximations and policy improvement approach following the methodology proposed in [17] and also in [12]. In Section 5 we apply the developed methodology to the manufacturing system described in Section 2, compute the optimal production policy and show that it is of classical hedging point type. In conclusion we discuss the proposed methodology and obtained results, and outline the possible directions for future works.

We consider a hybrid manufacturing/remanufacturing system consisting of two parallel machines denoted M₁ and M₂ respectively, producing the same type of product. Stochastic phenomena are demand level and machine breakdowns/repairs. We take into account the activity of production in forward direction and the activity of reutilization of returned products in reverse logistics. The demand must be satisfied by inventory for serviceable items. This inventory will be built by the products manufactured or reused. The returned products will be in the second inventory namely recovery, they can be remanufactured, or be hold on stock for future remanufacturing. In our problem, we assume that the maximal production rates for each machine are known and the machine M₂ is producing at average supply for return rate, which is also its maximal rate. This situation is illustrated in Figure 1. State of the machine with is modeled as a Markov process in continuous time with discrete state (—machine is operational,—machine is out of order). We may the define

State transition diagram is shown in Figure 2.

Hybrid system is in production while in modes 1, 2 and 3. Transition probabilities from state to state for machine

With,

State transition (4 × 4) matrix is therefore given by

(2)

State equations can be written in the simplified form

Since the demand and return rates are considered as stochastic processes the more rigorous Ito form of Equations (3) will be used later. Namely let d(t) be a stationary Gaussian process with the constant mean and variance, where

. Below we further specify in one of two ways: either an increment of a standard Brownian motion, or an increment of the first order Markov process defined later using the shaping filter.

For the return (remanufacturing) rate, an assumption is made that it is proportional to the customer demand rate with r is a percentage of return.

Stochastic state differential Equations (3) can be rewritten in Ito form using notation

Equations (4) will be also used in the following generic form:

For the Case A the input to Equations (4) is specified as a standard Brownian motion increment .

For the Case B the input to (4) is specified as an increment of the shaping filter output (Ornstein-Uhlenbeck process)

where    (6)

Process is a first order Markovian, its correlation function is.

Additional insight to the proposed demand model can be given by considering the cumulative demand: where is a constant demand ramp, is a randomly varying portion of the demand. For the case A. (Wiener process), for the case B. (Ornstein-Uhlenbeck process). Also case A can be obtained from B setting.

Following constraints have to be added to (5)-(6)

Let the cost rate function to be defined as follows:

Here,

: inventory holding and backlog costs for manufactured product (per time unit);

: inventory holding cost for remanufactured product (per time unit);

: production costs for manufacturing and remanufacturing processes (per unit);

: maintenance cost for nonoperational state of the machines:

where

The objective is to determine the production rates and in order to minimize the expected discounted cost (is the discount rate):

The domain of admissible controls is defined as

Defined hybrid system is said to be meeting feasibility condition if

where et are limiting probabilities and maximal productions rates. We recall that the vector of limiting probabilities is defined as an eigenvector of the transition matrix Q(.)

  et      (12)

Let us define the value function is defined as a minimum (infimum) of expression (9) over all possible control inputs:

Let us briefly recall the guidelines for obtaining optimality conditions. Introducing time-dependant α-dependant cost function and value function we have:

According to Bellman optimality principle for cost function at we can write

(15)

Using Taylor expansion for the term and the value function over last 3 arguments, and keeping linear terms over and up to second order terms over we get:

(16)

Second order terms over are kept for further analysis because of diffusion-type processes affecting system dynamics. One more technical step consists of computing the value function using Markov chain-type machine dynamics (2) defined through transition probabilities

Merging Equations (16) and (17) we get

Averaging over random realizations of the demand driven by the Brownian input using Equations (5) and applying Ito’s lemma we get:

Now neglecting all terms of order higher than 1 over

taking, and considering the stationary regime, we finally get HJB equations in the following form:

A numerical approach proposed by Kushner in [17] and successfully used in the series of works [9,12] consists of introducing the grid in the state space for approximating the value function approximating the first derivatives by “up wind” finite differences, then use policy improvement—discrete analog of a gradient descent in policy (control) space. Use of “up-wind” derivatives results in conditional computations but greatly improve convergence of the numerical.

The first case we have analyzed corresponds to the hybrid system with manufacturing costs set relatively high in order to enforce production in remanufacturing loop. The (cumulative) demand is modeled as a Brownian process. The results are shown in Figures 3-6. Figure 3 illustrates the shape of the value function depending on the stock levels of manufactured and remanufactured products in mode 1. Value functions in other modes have similar shapes and are not shown. Figures 4 and 5 illustrate the optimal policy for the machine 1 (manufacturing) in mode 1 (both machines in operation) and mode 2 (remanufacturing machine 2 in failure) respectively. The optimal policies for the machine 1 are of hedging-point type, namely: maximal production if the stock level is below the threshold, zero production above the threshold and production “on demand” at the threshold-level. Comparing Figures 3 and 4 one can observe that the threshold level in mode 2 when machine 2 is in failure is higher than in mode 1.

Figure 6 illustrates the optimal policy for the machine

2 (remanufacturing) in modes 1 when both machines are in operation (optimal policy of the machine 2 in mode 3 is identical). Machine 2 is must produce at average supply (proportional to demand) rate—which is also its maximal rate as explained in the section 2.

Figures 7 and 8 show the realizations of Brownian and Markov-type (filtered) demand rates respectively (the variance is set to the same value). Comparing two graphs one can see that in Brownian case (Figure 7) the variation rate is much faster than in Markov case (Figure 8).

Figures 9 and 10 illustrate the optimal policy for the machine 1 (manufacturing) in mode 1 (both machines in operation) and mode 2 (remanufacturing machine 2 in failure) respectively. The results are to be compared with those shown in Figures 4 and 5. One can observe that the threshold values for the case of slower varying Markov demand are lower as compared to the case of Brownian demand. Parameters used for simulations are summarized in Table 1.

In the second case of Markov-type (filtered) demand the parameters of the filter may be used to fit the model

to the characteristics observed in the real life applications. A classical assumption of the constant demand in this context means that the variability of the demand is ignored and only its average rate is taken into account. A

Table 1. Parameters of the numerical example.

second order terms in HJB equations reflecting demand variability are in that case neglected and the optimal policy is found using the first order approximation of HJB equation. Optimal policy for the main (manufacturing) machine is of hedging point in both studied (Brownian and Markov) cases—as it is for the constant demand. According to Figures 4, 5, 9 and 10 one can see that more the demand varies rapidly in time, more the hedging stock level increases in order to respond to the demand variability. In addition, the average total cost also increases from 1939 (Markov) to 2236.5 (Brownian) as the demand variability increases.

We have shown that the problem of stochastic control corresponding to optimization of production planning in failure prone hybrid manufacturing/remanufacturing systems with random demand can be successfully analyzed for diffusion-type demands. We investigate this problem in continuous time which seems to be the most natural setting. We develop optimality conditions in the form of HJB equations and show that due to the Brownian component in the demand the HJB equations are the second order PDEs, contrary to the case of a constant demand where they are of the first order. We use finite difference approximations for HJB equations reducing a continuous time optimization problem to the discrete time, discrete state, infinite horizon dynamic programming problem, and use policy improvement technique [17] for solving it. Value functions of the stochastic optimization problems are usually non-smooth and corresponding HJB equations have to be addressed using generalized approaches such as viscosity solutions [18]. Theoretical studies of the convergence of discrete approximations to an exact (viscosity-type) solution of HJB equations when the size of the grid tends to zero is addressed in [19,20]. Such theoretical analysis is out of the scope of this paper where we propose a numerical approach targeting the new model for the uncertain demand that allows addressing more naturally the growing number of industrial applications. Considering possible extensions of the study presented in this paper we count to explore a compound demand model of Poisson and diffusion-type process thus allowing both the jumps and continuous random variation of the demand.