Kicking off this blog, I will be writing a series on poker bots and their threat to online poker. This is a vast topic, so I will be splitting it into a few parts. Topics I want to cover include an introduction to poker bots, a history and the current status of poker botting in online poker, an overview of different AI techniques and more. I may even teach you how to write a simple poker AI. Today will be the introductory post.

So why are you writing this series?

Poker bots are a hot topic right now, as evidenced by the many forum threads on the topic. What’s also evident is that many posters in those threads are completely unaware what bots precisely are, how they work, and how far ahead their development is. You see posts of the form “I’m not worried because…”

• “… bots can only be as good as their developer.”
• “… bots can’t beat humans.”
• “… bots can’t adjust.”
• “… I am better than the bots in my games.”

and many similar statements. These are all misguided, and this series will hopefully clear up these misconceptions. Bots are indeed a huge problem. They are taking millions of dollars out of the poker economy ((I should really have a citation here. Instead, I’ll refer you to one of the future posts when we’ll be studying the current state of poker bots.)), and unless something happens, it’s only going to get much, much worse.

Nevertheless, poker AI is an awesome field. Studying it is a great way to learn more about AI, game theory and mathematics & computer science in general. If I can get anyone interested in one of those fields, then that by itself will have made this series worth it.

And why should I believe anything you say?

I’ve been interested in poker AI for many years. I’ve read the main poker botting forums, I’ve studied the scientific poker AI papers and I’ve written my own poker AIs. No, I have never actually used poker bots online for money. Anyway, if you want to know more about me just go to the ”About“-page of this blog.

Some definitions

First, let’s define some terms so we know exactly what we’re talking about. As you probably know, a poker bot is a computer program that plays poker. It consists of two main parts:

1. A strategy engine, the poker AI.
2. An interaction engine, doing the communication with the poker client software.

So, keep in mind that throughout this series whenever I use the term poker bot, I mean a poker AI that plays online poker. Also, I’d like to preemptively clear up a common (or uncommon) misconception you probably don’t hold: Bots are 99% of the time completely software-based. This means that there is not some real robot physically controlling the computer. The image on the right, while creative, is not quite an accurate depiction of a poker bot.

Breaking it down: the AI

The AI of a poker bot is like the brain of a human player: it takes inputs from the game, processes it, and outputs a decision. In the beginning, AIs were extremely straight-forward: they consisted of rules such as ”if you have a hand with more than 90% equity, raise”. These days, there are much, much more complicated AIs, which can’t be expressed in such general rules. To be clear: poker AIs are legal, very interesting and totally awesome. Only when you make them play online, things get murky.

There are various types of AIs, which all fall more or less into one of the following categories (or a mix thereof):

• Rule-based
• Equilibrium-based
• Exploitative

Let’s look at them one-by-one.

Rule-based AIs

These guys adhere to some set of rules written by the creator, like the example in the previous paragraph. They contain exact guidelines on how to play pre-flop, they are told when they should check-raise their flush draws, etc. Such an AI consists of rules, every rule containing:

• A description of the situations the rule applies to.
• An action (Fold, call, raise).

The author has to write a set of rules covering the entire game of poker. Since there’s quite a lot of unique situations in poker (think 10⁵⁰, a ten with fifty zeroes, which really is quite a lot ((This is a very general approximation – the exact order of magnitude depends on the game type.))), and since most bot authors don’t have infinite time, this means that many diverse situations will be grouped into a single rule.

Also, since the author has to define every rule, this type of AI can never be much better at poker than its author. ((Except, of course, for the fact that the author can think long and hard about every rule he programs into his AI, while he has limited time at the tables. Also, bots don’t tilt.)) As far as I’m aware, there are no such AIs that can beat humans for most poker games. ((Whenever I say “as far as I’m aware”, take that with a grain of salt. The world of poker bots isn’t very transparant: there could be some group in Bulgaria making millions with some rule-based AI that nobody has ever never written about.)) However, the best poker bots most certainly do incorporate some rules.

Equilibrium-based AIs

Then there’s equilibrium-based AIs. These are AIs that attempt to play an equilibrium strategy. If you don’t know what that means, buy my book! ((Managed to sneak some self-promotion in here, sweet!)) While you wait for it to arrive, I’ll give a short summary of equilibrium strategies.

Game theory

Basically, in game theory (a field of mathematics) some guy named John Nash won a Nobel Memorial Prize in Economic Sciences for, amongst other things, proving that every game has an “optimal” strategy for every player. That is, there is some set of strategies (the equilibrium), one strategy for each player, such that no player can improve his expected value by switching to another strategy. In other words, these strategies are unexploitable. You can safely play this strategy and expect not to lose in the long run, except to the house (in rake). It is very hard to calculate equilibrium strategies in poker, due to the game size. There’s two things we can do to calculate equilibrium strategies in poker right now:

• Take really small abstractions (simplifications) of poker and solve those.
• Take slightly larger (but still extremely simplified) abstractions and approximate solutions for those.

There’s many ways of constructing abstractions: the most common things to do are called bucketing and imperfect recall. We’ll come back to these in some later post when we’ll look at equilibrium AIs in detail.

Exploitative AIs

Equilibrium strategies are not the most profitable though: if you want that, you need to adjust to your opponents: these are called exploitative strategies. There’s several ways a bot can implement this, the most common are probably:

• Have several pre-computed strategies, select the one deemed best against the current opponent.
• Use some tree-search algorithm in combination with opponent models.

A bot might have pre-computed strategies against very aggressive and against very passive players. It can then simply look up its opponent in a database to see if he’s aggressive or passive, and select the appropriate strategy. I’ll come back to the tree-search in some later post – these are probably the main force behind the strongest bots in existence right now.

The other half: the interaction engine

By itself, the poker AI can’t make money. Just like your brain needs eyes and hands, the AI needs to know and act too. This is where things get ugly: poker sites don’t allow bots on their sites. So, most of the things the interaction engine does are against some terms of service. Many sites actively try to detect this part of a poker bot in several ways. We’ll look at that in one of the next posts.

Anyway, the interaction engine logically consists of two parts:

1. The observer.
2. The actor.

Observers: spying on your own system

To learn the game state (e.g. its hole-cards, the opponents’ actions, etc), the bot needs to intercept that information in some way. There are several ways to do this, some are simple and straight-forward (but unreliable and slow), others are extremely complicated and reliable:

• Scraping: looking at the screen, and interpreting the images seen.
• Using some kind of information stream readily supplied by the poker client.
• Doing the same, but with a less accessible stream (e.g. memory reading, function hooking).
• Intercepting and decrypting the network traffic.

Scraping

A scraper works much like the human mind does. It looks at the screen (by taking a screenshot), and interprets the parts. It uses OCR to convert the on-screen text to machine-readable text, it looks for the position of the button, it reads the stack sizes, etcetera. This method can be applied on every poker site, even from outside the machine the poker client is running on. ((For example, you can run the poker client inside a virtual machine, and run the scraper outside it.)) It is quite hard to get these working reliably, for several reasons. One is the fact that graphics can change. If a scraper is unaware of the change, it can crash, or misinterpret the situation. Also, consider the following scenario (click for larger version):

A scraper is going to have trouble reading the pot size in the middle (240). There are many small, unexpected situations like that that make the life of a scraper a living nightmare.

Reading client-supplied information

Luckily, it’s often easy to get a perfectly usable, realtime data stream from the poker client. How? Look closely at the following screenshot:

Notice it? Click here for the close-up.

Many clients readily supply the complete hand history as it’s being played out. On many sites (I won’t name names), a poker bot can read this dealer chat with less than 10 lines of code. It’s almost as if it’s made for bots. (On other poker clients, the dealer chat is protected much better, but it can always be read, even by bots.)

Even if a site does not provide a dealer chat, there are possibilities. Strong programmers can write bots that read the memory of the poker client, directly reading the status of the game. Or they can hook functions in the poker client such that they are notified when something happens. Since the poker client runs on the user’s computer, it’s impossible to completely stop botters from finding the game state. There will always be ways around the protection.

Network interception

Another way to find the game status is to intercept the network traffic. Everything that happens needs to be sent from the server to the client (or the other way), so if you listen, you’ll find out what happens. The network stream is encrypted though, so this method does not generally work without modifications to the poker client. I will probably have a small separate post about something relating to this in the future.

Acting: pretending to be human

We know what the game state is, and we have decided what to do. Now, how do we do this? The answer brings us to a cat-and-mouse game, with the poker bot pretending to be the mouse.

At first, poker bots simply moved the mouse cursor to the right button and sent a click. This clearly works very well. However, the sites started logging two things: ((Note that these by themselves do not identify botters; players using certain scripts and hotkeys will show the same characteristics.))

1. Mouse movements.
2. Mouse click locations.

By searching for players who moved their mouse inhumanly fast and/or always clicked on the exact same spot of the action buttons, the sites were able to detect many bots. So, botters started randomizing the button click location, and writing programs that move the mouse cursor similar to how a human would move it. Such a weird way of making a living.

Anyway, here too the botters can directly inject their decisions into the poker client or the network stream, but unless they have already reverse-engineered those, they’ll probably just use the simpler mouse cursor method – it’s very reliable, since the buttons are always in the same place.

Conclusion

Today we’ve gone over a short introduction of poker bots and how they work. You also know some of the basic terminology. Before ending this post, let me mention again that bots are a huge threat to online poker: right now they are taking millions of dollars out of the poker economy per month, and unless something happens it’s only going to get worse.

What’s up next?

In the next post, we’ll dive into the history of poker AI, from IRC to Winholdem, from Openholdem to PokerAI. Depending on how much room that takes up, we might start looking at the current status of poker botting as well. I don’t have the series planned out perfectly yet, but I’m planning to cover at least the following:

• History of poker AI.
• Current state of poker botting and counter-measures taken by poker sites.
• Detailed description of various types of AI.
• A detailed statistical analysis of some hand histories to find some bots.

I’m also considering writing a tutorial of sorts into writing poker AIs (not bots). It’s extremely fun, and very challenging as well. You might learn computer programming along the way! Anyway, we’ll see. If you have any ideas, or if you have any other questions, suggestions or comments leave them in the comments below or send them on Twitter.

To stay informed, please follow me on Twitter, or subscribe to my RSS feed. If you like this series, please tweet about it, send it to friends, etc.