Frans Ekman's blog

View Frans Ekman's profile on LinkedIn

It's Alive, it's Alive! Programming Magic!

I recently had a conversation with a friend, who also has a programming background but has moved more into a managerial position like me. We talked about when programming is fun and when it is not. We have both had moments when there is nothing else in the world but the one thing you are working on 24/7. You have so much adrenaline in your blood that you couldn’t sleep if you wanted to, so you just keep on hacking the whole night. These are the peaks that make software development the most interesting profession in the world. This of course requires the right circumstances.

I think it all comes down to how much “magic” there is in the stuff that you implement. It may be something really Hi-Tech to you, that you perceived as magic at some point in your life. Or it may also be something really simple but has the potential to spread virally to millions of users. However, there must be some magic in it to be able to unlock that kind of a productivity boost. My biggest moment of magic was when I built my first chess engine in 2002.

From the day I played chess against a computer the first time, long before I took up chess for real, I had been wondering how it is possible that the computer can think. When I got a little bit older I understood that it has to search through the moves tree and check what results in the best position. But still, without proper understanding of algorithms and data structures, it felt to me like some kind of magic.

Before directly going to the story, I’ll give a quick background to how I got into chess and writing a chess engine. I have always been a fan of strategy games, ever since I got my first computer and played Civilization 1, Sim City and later C&C and similar games. However, as much as I would have wanted to become very good at any of them, I knew that the knowledge would become obsolete one day when a newer game was released. That put my view of chess into a completely different perspective, since that game is going to stay unchanged forever.

My cousin had taught me to play chess since I was small, but I never really enjoyed it at that age. But when I was studying for the first year in high school, surprisingly chess became very cool. We were about 15-16 years old. People were playing during the breaks and even girls played or watched. I am not talking about nerds and geeks. This was an all-embracing phenomenon. I immediately felt that this is my game and I must be best at it. I started playing a lot. I read a few chess books, joined a local club and after a while I played in tournaments.

When I took a beginners course in Java programming at TKK in spring 2002, the course had a bigger project assignment at the end with several different topics. One topic was to build a chess game, but it was without possibility to play against the computer. I couldn’t let the opportunity pass, so I just had to choose that assignment and also go for the artificial intelligence as extra. Students were allowed to do changes to the scope if they were reasonable.

This was a very interesting project. I still remember well the moment when I got to try the AI functionality for the first time. I had already implemented all the functionality necessary to make a list of all the legal moves in a position and to make moves on the board. There was only textual representation of the board at this time, but that was all that was needed. I was at a computer class in TKK, where I used to work to be able to concentrate. It was 5 am already and I had fully lost my sense of time.

I used the negamax algorithm, one of the simplest forms of search. I had implemented with a very simple static position evaluation function, only counting the score of the pieces at the board. When the first bugs had been fixed so that it did not crash immediately, it actually played. It was able to search to a certain depth and suggest a move that proved that it had some kind of intelligence. It was able to think. I immediately jumped up and said to myself: “It’s alive, it’s alive!”. Too bad there was nobody else listening.

These kinds of peeks came every now and then when I was improving the AI. Adding alpha-beta search and other improvements to the algorithm allowed the engine to search deeper and do more complex tactical combinations. Improving the evaluation function to take into account positional advantages, such as positions of pieces, king safety and pawn structure made it play a lot more like humans. I still remember the Wow-feeling when I realized it was thinking strategically and it was playing quite coordinated attacks against me.

When the school project was over, I decided to continue the development as a hobby. I rewrote the engine in C, while doing several other improvements as well. I was doing this for a year or so. My engine (Hoplite) implemented the Chess Engine Communication Protocol, so it was able to play against other engines. Hoplite was playing in several tournaments around the world. It was free and open source, so tournament organizers could easily enter it into a tournament.

I had a lot of fun watching how it performed in a tournament after I had done a new release. Also during development I had set up my own tournaments, where I had several engines playing against each other. I usually had 2-3 variants of my own engine and then a couple of other engines for reference. Tournaments were automatically played with a script, so I could have it running during the night and check the results later. If a new feature showed good results, I added it to the mainline. I almost took this process so far that I would have used evolutionary algorithms to get optimal configurations. This would have been overkill, since the biggest improvements would have come from completely different optimizations.

The last time I touched Hoplite was probably in 2005. There came other more important things and at those days I had already realized that making it play really well would require rewriting some parts and a lot of time, which I unfortunately did not have. Maybe one day I’ll give it a shot again.

These are the peaks that make programming so damn interesting. You need to implement stuff that has magic in it. Every time you do that, you raise the bar and it gets harder to find any problem with magic again. During the last 10 years I have been raising my bar all the time and nowadays it is getting very hard to find something that I could implement with as much magic in it for me as the chess engine once had.

This raises an interesting question: What can a software company do to let all developers experience these kind of moments? Should a company give a lot more freedom to developers to work on things they like even though they are not 1st priority? It’s always a tradeoff and it depends on what kind of software you are building. If there is development of complex algorithms or growth hacking involved, it may make sense to let a team operate outside the usual process. Product management can define the boundaries within which the team can make decisions themselves and then give green light to do whatever they want within those boundaries.

For those interested, I created a simple page for Hoplite, from which the source code or Windows binary can be downloaded. The original Hoplite page was removed from TKKs servers when I graduated. Follow this link: Hoplite Chess Engine