Learning Optimal Pricing with Reinforcement Learning

Project description

• Language: Python
• Working file: Jupyter Notebook
• Project type: Reinforcement Learning

Problem Statement

Dynamic pricing is a fundamental problem in economics and operations research. Sellers repeatedly interact with customers whose willingness-to-pay is uncertain. Each round:

1. The seller offers a price from a feasible set (1–100).
2. The customer either accepts or rejects.
3. The seller observes only binary feedback (buy/no-buy).

In our environment:

The theoretical optimal price is 43, yielding the maximum expected revenue given the mixture of valuations and probabilities.

The objective of this project is not only to find the optimal price but also to test whether reinforcement learning (RL) can discover optimal solutions in combinatorial problems. Pricing here serves as a tractable example:

• States and actions are finite and discrete.
• Feedback is stochastic but structured.
• There exists a ground-truth optimum against which learned results can be benchmarked.

This aligns with a broader scientific inquiry: can RL and similar adaptive methods consistently solve combinatorial optimization problems without explicit enumeration or full knowledge of the environment?

MDP Formulation

We formalize the pricing task as an MDP:

• States (S): previously offered price.
• Actions (A): next price to propose.
• Transitions: deterministic, next state is the chosen price.
• Reward: equal to price if accepted, else 0.
• Discount factor (γ): balances present and future outcomes.

The MDP lens enables us to treat pricing as a sequential decision problem where the agent incrementally improves its policy.

Actor-Critic Algorithm

The Actor-Critic framework merges two worlds:

• Actor: learns a parameterized policy π(a|s).
• Critic: estimates value function V(s) to provide feedback.

In our implementation:

• The Actor uses a softmax policy gradient with a temperature parameter τ.
• The Critic employs Temporal Difference (TD) learning.

But more generally, the Actor could be a neural policy trained with REINFORCE, PPO, or A2C, while the Critic could approximate values with deep regressors or Monte Carlo estimates. This generality is important: pricing is one instance, but the Actor-Critic template is applicable to much more complex combinatorial optimization tasks.

Implementation

Core Actor-Critic loop

# Initialize
theta = np.random.normal(0, 0.1, (num_states, num_actions))  # Actor parameters
V = np.zeros(num_states)                                     # Critic values

def softmax(logits, tau=1.0):
    exp_logits = np.exp(logits / tau)
    return exp_logits / np.sum(exp_logits)

for episode in range(N):
    s = 1  # initial state
    pi = softmax(theta[s])
    a = np.random.choice(range(num_actions), p=pi)
    
    reward, s_next = environment(a)
    
    # Critic update
    td_error = reward + gamma * V[s_next] - V[s]
    V[s] += alpha_c * td_error
    
    # Actor update
    grad_log_pi = np.eye(num_actions)[a] - pi
    theta[s] += alpha_a * td_error * grad_log_pi
    
    s = s_next

Explanation:

• The Actor generates a distribution over possible prices.
• A price is sampled, reflecting both exploitation of good options and exploration of uncertain ones.
• The Critic evaluates how good the observed reward was relative to expectations (TD error).
• This TD error informs both value updates (stabilization) and policy updates (exploration direction).

This dual mechanism makes learning more stable than pure policy gradient and more flexible than pure value-based methods.

Action selection

pi = softmax(theta[state], tau=1.0)
action = np.random.choice(range(num_actions), p=pi)

Here, τ (temperature) is crucial:

• At high τ, the policy explores broadly.
• At low τ, the policy becomes deterministic around the best options. In pricing, this ensures the agent samples from a wide set of prices initially but eventually concentrates on optimal values near 43.

Critic and Actor updates

# Critic update
td_error = reward + gamma * V[next_state] - V[state]
V[state] += alpha_c * td_error

# Actor update
grad_log_pi = np.eye(num_actions)[action] - pi
theta[state] += alpha_a * td_error * grad_log_pi

• The Critic nudges its value estimates to better match observed returns.
• The Actor adjusts probabilities so that actions leading to positive TD error become more likely in the future. This interplay is what allows convergence in relatively few episodes.

Simulation Setup

• Market: four customer segments with valuations {18, 43, 56, 81}.

• Shares: {0.4, 0.3, 0.2, 0.1}.

• Episodes: 100,000 per run.

• Simulations: 100 runs to average out stochasticity.

• Metrics:

  • Learned optimal price.
  • Final average reward (last 10% of episodes).

Results

Summary statistics

These values show that Actor-Critic not only converges to near-optimal solutions but does so with low variance, confirming the method’s reliability.

Figures and Interpretations

Figure 1: Histogram of learned optimal prices Most simulations converge near the optimal price of 43. Some spread remains due to exploration noise, but the concentration proves that Actor-Critic is capable of discovering the true optimum from scratch.

Figure 2: Mean smoothed rewards across episodes Average reward increases sharply and stabilizes after ~5,000 episodes. This demonstrates early convergence—a key strength of Actor-Critic relative to value-only methods.

Figure 3: Convergence to optimal price The convergence curve shows how exploration covers a wide price range early but gradually narrows around the global optimum at 43. The algorithm maintains occasional exploration, but the focus is clearly locked on the optimal solution.

Key Takeaways

• Scientific insight: RL methods such as Actor-Critic can indeed optimize combinatorial problems, confirming their potential beyond pricing.
• Performance: Actor-Critic consistently converges near the true optimal price (43) within ~5,000 episodes.
• Stability: Variance across runs is low, showing robust learning under stochastic feedback.
• Flexibility: Different Actor (policy networks, PPO) and Critic (deep value approximators, Monte Carlo) architectures can be applied in more complex versions of this problem.

Conclusion

This project confirms that reinforcement learning, specifically the Actor-Critic method, can successfully solve dynamic pricing problems without prior knowledge of customer valuations. Beyond pricing, this serves as evidence that learning-based approaches can tackle combinatorial optimization problems, converging to theoretical optima with high reliability.

The results support a broader conclusion: reinforcement learning is not only an engineering tool but also a scientific framework capable of discovering optimal strategies in environments characterized by uncertainty and discrete combinatorial structure.