🤖 RL Pricing Agent & Market Simulator

A reinforcement learning system that trains a PPO agent to dynamically price products in a simulated competitive retail market, calibrated on 26,342 real retail transactions, tested across 5 shock scenarios and 5 competitor archetypes.

---

📌 Overview

Can an AI learn to make smarter pricing decisions than a human expert and adapt those decisions in real time as market conditions change?

This project builds a complete answer to that question. Using real transaction data from a UK online retailer, we constructed a digital market simulator, trained a reinforcement learning agent to compete in that market, and measured exactly how much better the AI performs compared to expert-designed pricing rules across five market conditions and five competitor types.

Course: IS557 Applied Machine Learning — University of Illinois Urbana-Champaign
Timeline: January 2026 – May 2026

---

🏆 Key Results

Metric	Value
Training Steps	600,000 (across 5 independent random seeds)
Market Scenarios Won	5 / 5
Profit Lift vs. Expert Benchmark	+8% to +79%
Average Revenue Pool Captured	46.8% (vs. 33.3% equal-share benchmark)
Competitor Archetypes Beaten	4 / 5
Demand Model R²	0.516 (OLS, calibrated on real retail data)
Retail Transactions Processed	26,342
Validation Figures Produced	13 publication-ready plots
Price Volatility Reduction (Stability Penalty)	−90.8% with +2.1% profit gain

---

🏗️ Architecture

The project is organized into modular, reproducible components:

market_sim/
├── demand_engine/          # OLS demand model calibrated on 26,342 real transactions
├── shock_engine/           # Composable market shock simulator (5 shock types)
├── competitor_agents/      # 5 rule-based competitor archetypes + SmartHeuristic benchmark
├── gym_environment/        # Custom OpenAI Gymnasium environment (52-week episodes)
├── training/
│   └── train_rl_agent.py   # PPO training via Stable-Baselines3 (QUICK / hero run)
├── evaluation/             # Scenario benchmarking, multi-seed study, lambda ablation
├── scripts/
│   ├── output/             # 13 publication-ready figures + CSV summaries
│   └── cross_validate_demand.py
├── data/                   # Cleaned retail transaction dataset (26,342 records, 56 SKUs)
└── validate_simulator.py   # 49-check validation suite (run before training)

---

🔬 Methodology

1. Data Foundation — Real Retail Transactions

• Source: Online Retail dataset (UK retailer, Dec 2010–Dec 2011, 541,909 total transactions)
• Filtered to reusable shopping bag SKUs: 26,342 transactions across 56 SKUs
• Fit a calibrated OLS demand model:

D(t) = 33,743 − 17,641 × Price + 203 × t + ε     (σ = 2,386, R² = 0.516)

Parameter	Meaning
a = 33,743	Theoretical maximum weekly market demand
b = −17,641	Demand falls by 17,641 units per £1 price increase (high elasticity)
c = 203	Weekly growth trend in eco-friendly bag demand
σ = 2,386	Real market noise — genuine week-to-week unpredictability

Seasonal multipliers were extracted directly from the data (0.478 in February → 1.395 in November).

---

2. Market Simulator — Custom Gymnasium Environment

• 52-week episodes simulating one year of competitive retail pricing
• Action space: 29 discrete price points (£0.80 to £2.20 in 5p steps)
• Observation vector (6 dimensions): own price, average competitor price, demand level, seasonal factor, active shock type, price spread vs. competitors
• Demand allocation: Logit model — Share_i = exp(−α × Price_i) / Σ exp(−α × Price_j) with α = 2.5 (calibrated from data)
• Reward function:

  Reward = [(Price − Cost) × Units Sold − λ × |ΔPrice|] / 21,000
  

  The stability penalty (λ = 2,500) discourages erratic repricing — a real deployment constraint on platforms like Amazon Seller Central.

---

3. Shock Engine — Simulating Market Disruptions

Shocks work by modifying the structural parameters of the demand equation, not arbitrary multipliers. This is the economically correct approach and a key technical novelty of this project.

Shock Type	Active Period	Economic Effect	RL Lift vs. Benchmark
Normal	Baseline	Calibrated OLS parameters, no disruption	+24.4%
Inflation	Weeks 10–50	Price sensitivity +60%, cost floor +30%, market size −10%	+13.7%
Recession	Weeks 10–52	Market size −40%, demand volatility +50%, trend reverses	+8.1%
Demand Surge	Weeks 10–50	Market size +50%, price sensitivity −40%	+79.3%
Stagflation	Weeks 10–50	Inflation (0.8) + Recession (0.6) stacked simultaneously	+8.0%

Shocks are composable — multiple shocks can be stacked multiplicatively, matching how real economic disruptions work.

---

4. Competitor Archetypes — Who the Agent Fights Against

Competitor	Strategy	Real-World Parallel	RL Win Rate
Fixed Price Seller	Always charges £1.80	Traditional high-street brand pricing	25/25 (+70.6% avg lift)
Undercutter	Always 90–97% of agent's last price	Amazon loss-leader / discount repricers	0/25 (−5.3% avg lift)*
Premium Seller	Always 105–115% of agent's last price	John Lewis / Dyson premium anchoring	25/25 (+13.0% avg lift)
Chaotic Random	Random 70–130% of base price	Disorganised liquidation sellers	25/25 (+49.5% avg lift)
Reactive Agent	Raises/lowers based on sales velocity threshold	Basic Amazon repricers	25/25 (+70.3% avg lift)

*The Undercutter creates a structural race-to-the-bottom. This is treated as a genuine finding, it defines the boundary condition of the RL approach and mirrors the real challenge of competing with Amazon Warehouse pricing.

The SmartHeuristic benchmark is a sophisticated market-aware rule-based system (undercuts by 5p by default, adjusts +8–18p for surges and −8–12p for recessions). Beating it proves RL adds genuine value beyond expert rule design.

---

5. PPO Agent — Training Configuration

Parameter	Value
Algorithm	PPO (Proximal Policy Optimization) via Stable-Baselines3
Network Architecture	MLP — 6 inputs → 256 → 256 → 29 outputs (ReLU)
Training Steps	600,000 (hero run)
Parallel Environments	6 (rotating shock types for diversity)
Learning Rate	0.0003
Discount Factor (γ)	0.99
Stability Lambda (λ)	2,500
Random Seeds	5 independent runs for statistical validation
Training Time	~60–90 minutes on a standard laptop

---

📊 Full Results

Scenario Performance (Multi-Seed, 5 Seeds × 5 Scenarios)

Scenario	RL Profit (Mean ± SD)	Lift vs. SmartHeuristic	RL Revenue Pool Share
Normal	162,963 ± 1,420	+26.2% ± 1.0%	48.0% ± 1.7%
Inflation	18,782 ± 639	+15.6% ± 2.0%	45.2% ± 1.7%
Recession	21,938 ± 679	+8.7% ± 2.0%	43.3% ± 1.7%
Demand Surge	527,343 ± 63,971	+82.7% ± 6.4%	57.5% ± 3.1%
Stagflation	18,946 ± 637	+9.4% ± 2.1%	43.3% ± 1.7%

The 33.3% equal-share benchmark is the revenue each agent would capture if all three were equally effective. The RL agent consistently captures 43–58%, proving active market share capture.

Lambda Ablation (λ=0 vs λ=2,500)

Configuration	Avg Profit	Avg Pool Share	Avg Weekly Price Change
λ=0 (no penalty)	146,912 ± 8,746	25.5% ± 1.5%	0.055 ± 0.041
λ=2,500 (with penalty)	149,994 ± 12,844	28.5% ± 1.8%	0.005 ± 0.002
Improvement	+2.1% profit	+3.0 pp share	−90.8% volatility

The stability penalty is a win-win: the agent is both more profitable and more stable with it active.

---

📈 Output Figures (13 Publication-Ready)

Figure	Description
fig_A_learning_curve.png	PPO reward convergence over training episodes
fig_B_profit_comparison.png	RL vs. benchmarks: total profit per scenario with error bars
fig_C_revenue_pool.png	⭐ Revenue pool capture — the key result figure
fig_D_price_behaviour.png	Price trajectories and cumulative profit per scenario
fig_E_robustness_heatmap.png	Profit heatmap and pool share bar chart
fig_F_weekly_profit_grid.png	Week-by-week profit with profitable/loss fills
fig_G_competitor_breakdown.png	Three-panel RL vs. all 5 competitors overview
fig_H_competitor_lift_heatmap.png	Lift percentage grid — clearest wins/losses summary
fig_I_head_to_head_scorecards.png	Dumbbell charts: absolute profit per 1v1 matchup
fig_J_multiseed_scoreboard.png	Statistical reliability across 5 independent seeds
fig_K_lambda_ablation.png	Effect of stability penalty on profit, share, and volatility
fig_L_pricing_heatmap.png	Heatmap of the learned pricing policy (week × scenario)
fig_M_market_efficiency.png	RL pricing vs. theoretical optimum from demand model

---

🛠️ Tech Stack

Layer	Tools
Language	Python
RL Framework	Stable-Baselines3 (PPO)
Environment	OpenAI Gymnasium
Data Processing	Pandas, NumPy
Demand Modeling	Scikit-learn (OLS regression)
Visualization	Matplotlib, Seaborn
Validation	Custom 49-check suite (validate_simulator.py)

---

🚀 Getting Started

Prerequisites

pip install stable-baselines3 gymnasium pandas numpy scikit-learn matplotlib seaborn

Step 1 — Run Validation (always first)

python validate_simulator.py
# Expected: 49/49 checks passed ✅

Step 2 — Quick Test Run (~6–8 minutes)

# In training/train_rl_agent.py, set:
QUICK = True

python training/train_rl_agent.py

Step 3 — Full Hero Run (~60–90 minutes)

# In training/train_rl_agent.py, set:
QUICK = False

python training/train_rl_agent.py

Step 4 — Cross-Validation

python scripts/cross_validate_demand.py
# Requires: online_retail_II.xlsb or Retail_Transaction_Dataset.csv

All figures and summaries are saved to scripts/output/.

---

🔭 Future Work

• Undercutter counter-strategy — adversarial co-training to discover non-obvious differentiation responses to loss-leader competitors
• TitForTat competitor — test agent robustness against game-theoretic strategic imitation (already coded in competitors.py)
• Multi-product / multi-SKU pricing — extend the simulator to portfolio pricing
• Real-time API — wrap the trained PPO model into a live pricing recommendation endpoint
• Academic publication — three submission-ready angles: simulator architecture, RL result, and stability penalty contribution

---

📄 Research Contributions

This project has three publishable technical contributions:

1. Composable Parameter-Shift Shock Engine — structurally modifies demand equation parameters rather than using ad-hoc multipliers; shocks are stackable and economically grounded
2. Stability-Penalty Reward Shaping — reduces price volatility by 90.8% while improving profit, directly addressing the deployment barrier of platform repricing penalties on e-commerce marketplaces
3. Multi-Seed Statistical Validation — 5 independent training seeds with standard deviations reported across all 25 matchups, providing publication-grade reliability evidence

---

👤 Author

Tejas Jaggi
MS Information Management — University of Illinois Urbana-Champaign
LinkedIn · Portfolio · GitHub

---

Built with real data. Trained with discipline. Validated with rigour.