Poker Sandbox

1. The idea

Poker Sandbox became a strategy testing lab: a place to turn vague poker ideas into executable agents, run those agents through simulated hands, and diagnose which decisions were losing value. The project is part product and part ML system. The product layer makes strategy experimentation feel natural. The ML layer estimates expected value for decisions using supervised data generated by the simulator.

2. The full loop

The system is designed as one linear feedback loop. First, agents create behavior. Then the simulator records decisions. Feature engineering turns those decisions into training vectors. Random Forest models learn action value. Finally, a custom strategy is simulated, diagnosed, patched, and rerun.

3. Agent structures

The first problem was data. I did not have a giant hand-history dataset, so I created synthetic data through built-in agents. Each agent has a recognizable style: Tight-Aggressive plays fewer hands but applies pressure, Loose-Aggressive enters more pots and bluffs more, Calling Station over-calls, and Random creates noisy baseline behavior.

These agents are not meant to be perfect poker players. They are controlled behavior generators. By letting them play many hands, the simulator creates repeated examples of how different styles act across position, stack size, pot size, board texture, and betting pressure.

4. Simulation and decision logging

At every decision point, the simulator snapshots the table before the agent acts. That snapshot includes the legal actions, current street, board cards, pot size, amount to call, stack depth, prior betting behavior, position, and eventual result. This turns a poker hand into a sequence of decision records rather than one final win/loss outcome.

That matters because the model needs to learn from individual choices. A player might lose a hand but still make a good decision, or win a hand after making a bad one. Logging state-action examples gives the ML layer a more granular training signal.

5. Feature engineering

The Random Forest does not see raw cards the way a human does. It receives structured feature vectors. I engineered features that describe the strategic situation: hand strength, pot odds, stack-to-pot ratio, board wetness, position, facing-bet status, aggression history, continuation-bet opportunity, and action encoding.

For EV prediction, the important training example is a state-action pair. The same poker state can be copied across several legal actions; only the action feature changes. The model then learns which actions tend to produce better long-run value in similar situations.

6. Model selection

I chose Random Forest regression because the data is structured, tabular, and full of nonlinear interactions. Poker decisions are rarely explained by one feature alone. A call might be good with strong pot odds, bad with a weak hand, better in position, and worse against high aggression. Random Forests handle those feature interactions well without requiring a neural network or massive dataset.

Random Forests are also interpretable, robust to noisy synthetic data, and useful for both classification and regression. In this project, classifiers support hand-strength buckets, opponent action tendencies, and bluff detection. The EV model is a Random Forest regressor that predicts expected profit for each candidate action.

Each tree learns different splits over the feature vector. One tree might split first on hand strength, another on pot odds, another on whether the agent is facing a bet. The forest averages those outputs to produce a more stable EV estimate than a single decision tree.

7. Decision-time inference

When MLAgent reaches a decision, it does not ask the model for a single action directly. It enumerates all legal actions, creates a feature vector for each one, predicts EV for each candidate, and chooses the highest predicted value.

This makes the MLAgent a model-backed decision policy. The model supplies value estimates; the agent turns those estimates into actions.

8. Natural-language parsing

The strategy lab adds a product layer on top of the simulator. A user can write a strategy in plain English, such as "play loose aggressive, bluff rivers often, and do not overfold to small bets." The parser maps that sentence into a typed configuration: tightness, aggression, bluff frequency by street, call thresholds, continuation-bet frequency, and fold-to-pressure behavior.

Parsing matters because it turns subjective strategy language into parameters the simulator can execute. Instead of leaving a strategy as a note, the system converts it into a ConfigPolicyAgent that can play hands, produce metrics, and be compared against baseline opponents.

9. Diagnosis and patching

After a custom strategy runs through simulations, the diagnostic layer summarizes how it behaved: win rate, big blinds per 100 hands, aggression rate, fold rate, showdown frequency, and matchup performance against each baseline. The leak detector looks for patterns like over-folding to aggression, calling too wide, bluffing too frequently, or failing to apply pressure in profitable spots.

The patching step turns those findings into parameter changes. For example, if a strategy folds too often against small bets, the patch might lower fold-to-aggression and raise the call threshold. If it bluffs too much on later streets, the patch might reduce river bluff frequency. The system then reruns the diagnostic with the patched configuration so the recommendation is tested, not just suggested.

1. Run the custom strategy. Compile the natural-language strategy into policy parameters and simulate it against baseline agents.
2. Measure behavioral leaks. Compare aggression, fold rate, showdown rate, and EV-style performance against expected ranges.
3. Generate a patch. Modify the relevant parameters, such as bluff frequency, call threshold, or fold-to-pressure.
4. Rerun the experiment. Test whether the patch improves simulated performance instead of treating advice as static.

Why it matters

The project became more than a poker toy, even though it helped me a lot at home games. It used synthetic data generation, feature engineering, supervised learning, model-backed decisioning, natural-language configuration, simulation-based evaluation, and automated iteration. The same pattern applies to any domain where a user wants to describe a strategy, test it in a controlled environment, and improve it with evidence.