Reports | January 08, 2012 13:24

Controversy over Rybka's disqualification and ban (UPDATE)

Controversy over Rybka's disqualification and ban

Rybka's disqualification and ban from computer chess last summer by the International Computer Games Association (ICGA) is being disputed. In a 31-page article, a computer scientist working at London's Queen Mary University, supported by two chess programmers, argues that "the ICGA's findings were misleading" and the decision to punish Rybka and its programmer Vasik Rajlich "lacked any sense of proportion". Meanwhile, the ICGA has responded with a technical rebuttal.

Disqualification and ban

Last summer the International Computer Games Association (ICGA) disqualified and banned Rybka and its programmer Vasik Rajlich from previous and future World Computer Chess Championships. The ICGA accused Rajlich of plagiarizing two other programs, Crafty and Fruit, and demanded that he returns the trophies and prize money of the World Computer Chess Championships in 2007, 2008, 2009 and 2010.

Our report on the news made the mainstream media: it was picked up by the influential ExtremeTech and then copied by hundreds if not thousands of sites, including the New York Times and Der Spiegel.

Chessbase

It was the biggest computer chess story of recent years, but the editorial team of Chessbase didn't cover it. This is obviously related to the fact that the German company is in fact distributing and selling Rybka. (In their shop, it can be seen that on the Rybka 4 DVD the tag line 'Computer Chess World Champion' has now been traded for 'PC Chess Program by Vas Rajlich'.)

Last week however, Chessbase ended its silence. They published a lengthy article in four parts (you can download it in full in PDF here), dramatically entitled 'A Gross Miscarriage of Justice in Computer Chess' and written by Dr. Søren Riis.

Riis's article

The author, a computer scientist at Queen Mary University in London, makes the case that the ICGA’s allegation of "plagiarism", as well as the points they offered in support of their accusation, are without merit. The author, who is supported by an extensive technical report by Ed Schröder as well as support in the form of unpublished notes from chess programmer Sven Schüle, argues that the ICGA’s charges were based on false premises, tendentious conclusions and manipulated evidence.

Riis speculates that the chief motivating factor behind the persecution of Rajlich was his domination of computer chess programming in 2005-2010, during which time his program Rybka almost invariably annihilated other programs in public tournaments. The computer scientist and mathematician mentions that the people who voted for the Rajlich ban were direct beneficiaries and actually picked up his vacant titles, collateral which they now use to market their own chess engines. Riis further observes that, if anything, it is Rajlich’s program that has been systematically reverse-engineered and plagiarized.

According to Riis justice can only be served if the ICGA publishes a retraction of their accusations and restores Rybka’s world championship titles, concluding his defense by extolling Rajlich as

a great chess programmer, world champion and innocent man.

Biased?

We asked Mr Riis, who is not only a computer scientist in London but also a Rybka Forum moderator and therefore closely connected to the "Rybka family", for some more background. He replied to us:

There where strong arguments for publishing [the article at] ChessVibes as you would be seen as more neutral, while publishing with Chessbase would would open up for the most trivial counter-attack: “Cui Bono”. However, since I am a Rybka Forum moderator it was clear my article would always been seen as biased.

When I first read the ICGA report I thought they put a convincing case but is was only when I investigated the case I began to realise the full extent of the injustice. It was at that point that I decided to write my article which is written as a defense of Vasik Rajlich, and it is as much for him as it is to satify my own sense of justice and fair play.

I have no personal agenda against anyone in ICGA but they handled the Rybka case very poorly. I also don't have a particular personal agenda for Vas apart from the fact that I think he is exceptionally gifted, and I want to see justice done.

ICGA

We also asked the International Computer Games Association to comment. Dr. David Levy, head of the ICGA, sent us (and Chessbase) a lengthy rebuttal entitled 'No Miscarriage of Justice - Just Biased Reporting'. You can download it if full in PDF here but we'll quote from it:

As a historical review of progress in computer chess Riis’s article contains important and interesting information and comments. Unfortunately, however, his thesis lacks objectivity because it circles the core question and attempts to defend Rajlich by attacking the rule he was accused of breaking, attacking the investigative process in various ways and attacking some of those involved in that process.

When a defendant is brought before a court of Law, what is in question is whether or not (s)he broke the Law and not whether the Law itself is appropriate. And so it is with the ICGA rules. In considering the Rybka case the ICGA’s task was to decide the matter on the basis of its Tournament Rule 2, not to question the rule itself.

In his article. Levy tries to point out

the irrelevance to the ICGA rules of some of Riis's key arguments

and

correct some of his erroneous assumptions.

He then gives a number of examples that, according to Levy, point out that Riis's article at Chessbase is a case of biased reporting.

Technical rebuttal

We also received an article by Mark Watkins, a Research Fellow at the School of Mathematics and Statistics at the University of Sydney and member of the Computational Algebra Research Group. He was one of the investigators for the ICGA and his 'A critical analysis of the four parts of Riis' forms a technical rebuttal. You can download it if full in PDF here and we'll quote its summary:

In Part I, Riis suspects he has insights that the Panel ignored, and tries to argue that Rule #2 has become inapplicable. Neither of these assertions is borne out by his arguments in the later parts.

In Part II, among other problems, almost every relevant data point in his graph appears to be erroneous. Nothing in this first section holds water, in particular his intended conclusion that Rule #2 is no longer workable. On plagiarism, he asserts that Rajlich’s failure to cite Fruit in his ICGA entry is their problem (not his), and then returns to railing against Rule #2. He greatly minimizes the magnitude of Fruit/Rybka overlap here, perhaps as a corollary of erroneous conclusions made elsewhere. He then over-values the worth of indirect similarity testing as compared to direct code analysis. He finally mis-dates a quotation from Letouzey on originality, thus applying it to the wrong Fruit version.

In Part III, he gives 10 substantial Rybka/Fruit differences in evaluation, of which at most 4 seem viable, and all were noted by the ICGA. He does not discuss the remaining 20-30 evaluation elements, which were largely found to be substantially similar. He then misconstrues the “algorithmic” nature of the evaluation function, ignoring any creative aspects in its creation. He then monstrously exaggerates the impact of the floating-point zero issue on the ICGA decision, doing similarly with the PST issue in the next part. He fails to address a number of additional Fruit/Rybka congruences that were detailed by the investigation; together these helped to form a much broader picture than the 2 or 3 elements that he presents.

In Part IV, he misinterprets the question of PST copying. He seems to invent his own measure of comparison (raw numbers), while a proper metric would be the number of Fruit code changes necessary to replicate the Rybka output (and whether this number was abnormal). He then eccentrically proposes that one can skirt any “copying” issue by translating to a different programming language. He then suggests a timeline for Rybka development that erroneously states Rajlich went full-time in 2003, rather than in mid-2005. I will not address his defamation of Hyatt and the ICGA.

Riis omits any mention of the fact that Rajlich had previously plagiarized Crafty in private 2004 versions of Rybka, and furthermore that these versions had little internal similarity to the 2005 Rybka. The latter fact played a significant role in the Panel deliberations, strongly implying 2005 Rybka was a re-write, at the least.

Throughout, Riis displays little knowledge of programming, let alone that of computer chess therein. For instance, he claims that a one-time operation that takes less than a millisecond on Fruit startup (or during compile with Rybka) should be made fast for efficiency reasons. Furthermore, he is similarly lacking in any knowledge of the relevant aspects of copyright law, particularly the Abstraction- Filtration-Comparison Test that formed a basis for one part of the ICGA Panel analysis. He appears to apply a minimalist copy/paste standard to what “copying” might mean, ignoring any other creative aspects. Finally, he consistently refuses to apply any inferential capability regarding likely scenarios; combining this with an artificially impossible standard of proof, he is reduced to the pedantry of repeatedly asserting that no one can prove that Rajlich directly copy/pasted Fruit source code, when this was never the issue to start.

For the general chess audience one thing is clear: that this matter is very unclear. We're dealing with highly complicated issues related to programming and copyright. At several forums, heavy debates are ongoing and in fact we received several emails from our readers with a request to write about this subject. No doubt the last and definite word has not yet been spoken.

Update January 8, 2012 22:44 CET: We forgot to mention that last week we also asked Vasik Rajlich to comment on Riis's article. Only after we published today's article, we received an answer from him, in which Vasik refers us to the Rybka Forum, where he posted the following:

I want to give a really big thanks to Ed Schroder, Soren Riis, Chris Whittington, Nelson Hernandez, Nick Carlin, Jeroen Noomen and Alan Sassler for their superb efforts in defending me against the accusations that I have broken ICGA tournament rules. Soren did a great job detailing the shenanigans pulled during the ICGA's investigation, from stacking the jury to premature public accusations to a comprehensive fabrication of evidence.

I also appreciate the tenacity of Chris Whittington and especially Ed Schroder in digging through the mountains of documents and answering them point by point.

Finally, I greatly appreciate the support that I have received from countless others. This support has been touching and uplifting and I really appreciate it. Thanks guys! 
 
In other news, I'm working on Rybka.  Rybka 5 will be ready sometime this year and the remote Rybka renting will be launched sometime after that. We'll make announcements on our web site and in our forum when we have more information. It should be a fun year.
 
Best wishes to everyone for a great 2012!
 
Vas

In the same thread, Rajlich has now responded briefly to the main issue: the possible use of Crafty and Fruit code for creating Rybka. (We left out parts of the thread and only give Rajlich's key answers:)

Of course there is a clear influence of Fruit on Rybka. I haven't tried to quantify this influence or compare it to other engines from Rybka's generation. What I can say is that Rybka is original at the level of source code. In the context of source code, original means that the author either typed his own code or typed the code which generated his own code. For the super-geeks, yes, that can be applied recursively.

Peter Doggers's picture
Author: Peter Doggers

Founder and editor-in-chief of ChessVibes.com, Peter is responsible for most of the chess news and tournament reports. Often visiting top events, he also provides photos and videos for the site. He's a 1.e4 player himself, likes Thai food and the Stones.

Chess.com

Comments

choufleur's picture

indeed this is hardly understandable !

mishanp's picture

Actually, despite having written something myself about how complex an issue this is... I think it's pretty clear. You don't need to be an expert to see how poor Riis' arguments are, and the technical evidence for Rybka having broken the rule in question looks watertight.

noyb's picture

mishanp - Claiming something doesn't make it so. Quite to the contrary, Riis give clear facts to support his argument while Levy does not (Levy's 'rebuttal' contains not one fact to substantiate his or the ICGA's claims). Can you provide even one bit of proof that Vas broke a rule? If so, please provide it here.

mishanp's picture

noyb, this article neatly addresses anything technical in Riis' article: http://harveywilliamson.com/ICGA/Riis3.pdf (and follow the first link there for a recap of the whole situation so far) As I understand it Riis introduced nothing that hadn't already been posted by Ed Schroder and dismissed as (deliberately) misleading/irrelevant by a majority of his computer programming peers. Levy points out most of the absurdities obvious to any layman e.g. arguing a rule is outdated ("everyone now does it") obviously isn't a defence against breaking the rule, even if true. Just try that if you're charged with speeding... Other signs the document's shabby and cheap propaganda include the absence of any sources for most of the information and e.g. the chart of the frequency of an opponent's posts on a forum. That's something Riis and ChessBase should feel deeply embarrassed about.

Perhaps a coherent defence of Rybka against the charges can be written, but I certainly haven't seen one so far.

THS's picture

The ones that should be deeply embarassed, are the ICGA and its attackers. Not only did they use a fraudulent report containing falsified code to claim something that wasn't true at all and that has been debunked thoroughly (and not only by Schroder). They now submitted a rebuttal with outright lies.

mishanp's picture

Your claims were addressed in that rebuttal (convincingly, as far as I can tell), so by all means write a rebuttal of the rebuttal, but actually addressing the points made with logical arguments and none of Riis' junk "arguments" aimed at a gullible public ("it's 100 points better so nothing can have been copied", "Hyatt posts a lot so he must be a bitter loser who wants to overthrow someone more successful" etc.).

THS's picture

On the contrary. The ICGA report is very weak and contains a lot of self invented code. Read Ed Schroders website.

ca's picture

One thing that bothers me is, why rip Rybka from all the titles? It's pretty clear that the lastest versions of Rybka were original, the engine demolished every other engine that participed in the World Computer Chess Championships, it clearly shows that Rybka's creator made an engine better than all others and indicates that he made original progress, if not, how could he have breed an engine better than all others just plagiarizing? The hunt ghost against Rybka shows that other chess programmers were really envy of Rybka's strenght and success.

Rumpelstiltskin's picture

Thank you, as I am pretty sure that David Levy's Chessbase rebuttal will not be posted on the Chessbase website.

Ellie's picture
cavelorum's picture

Just because it's much stronger doesn't mean it's original. Stolen parts are still stolen parts...

What actually bothers me is the fact that Mr. Rajlich did never try to defend himself (not then, not now), apart from the video interview where he claimed that he didn't do anything wrong and admitted some code copying. Though, in his opinion those segments were within the scope of the public domain. However, the example that he gave clearly wasn't of a public domain code.

Finally, for Chessbase's sake I hope they'll publish this response as well. Otherwise they completely lose their credibility as an objective news source.

THS's picture

So why has Ed Schroder, former world champion and claiming Vas is guilty in the beginning, changed his mind and now claims that Rybka IS in fact original?

Merlinovich's picture

No Mr. Rajlich didn't admit to code copying. He admitted to idea copying. For copyright rules, there is a world of difference. You can't copyright ideas, it would stifle all fair use of new ideas, especially for use of new ideas that by chance was originating from several people on the same time.

cavelorum's picture

Oh yes he did... http://youtu.be/cQshTNJ4pSM?t=7m40s (see his comment @9:30, Nalimov code isn't public domain, also see his comment @16:40)

Anthony's picture

To be honest, Chessbase has zero credibility. It's just fine for your regular chessnews, but it is a business.

Their plugging of Kasparov's disgraceful (wall street backed) campaign in Russia without ever publishing retorts to Kasparov's useless statements, which should be basic journalistic practice, is another case in point.

TMM's picture

One thing is clear, which is that the ICGA did not act professionally. The outcome of the "investigation" was decided long before the "investigation" took place, and the report was just a matter of convincing the other board members with highly suggestive graphs that they were right.

Also, notice that Riis investigated this matter and published these results a long time after Rybka was banned. Then, only days later, Watkins and Levy have already "studied" his results and concluded he is wrong. This again sounds more like "our conclusions are fixed, and anyone suggesting otherwise is by definition wrong" than "ah, he raises some interesting points, let us see if we may have made mistakes, and respond with a thorough analysis later". They are not open for any discussions: they are right and the rest is therefore wrong.

cavelorum's picture

@TMM Nonsense. Are you seriously going to suggest that people like Ken Thompson cannot possibly review the evidences without prejudice and bias? By the way, Dr. Riis' analysis was at places deeply flawed and incorrect, no doubt about it.

TMM's picture

If you can prove that Ken Thompson agrees that reverse-engineered Rybka assembly-code, rewritten to match Fruit as much as possible, proves that Rajlich copied Fruit-code there, then I will eat my hat (and buy one).

Hans's picture

When Ken Thompson read the report, he was shown falsified and fruitified Rybka code. So his view was based on a deeply flawed and fraudulous report.

Ed Schroder also was convinced that Vas was guilty, until he found out that the report was fraudulous.

Harvey Williamson's picture

A total lie jeroen. Ken says Vas is guilty.

THS's picture

Ken has been shown a falsified report, on the basis of which he made a conclusion. But he didn't do any RE himself. Instead of blindly believing somebody, you could actually try to understand the REAL facts and evidence. Evidence that has sadly been debunked by Schroder and many others.

But it is clear that team Hiarcs had a lot to gain from a Vas' ban and it is not the first time team Hiarcs used its Levy connection to create favourable WCCC rules for itself.

THS's picture

If Ken Thompson didn't do his own reverse engineering, he cannot know if Rybka was copied or not.

S3's picture

" However, since I am a Rybka Forum moderator it was clear my article would always been seen as biased."

Always been seen? Come on..It IS biased.

Hans's picture

The people who voted 'Vas is guilty' in the ICGA panel were either gaining a world title or medal by doing so, or were hating Vas from the beginning (see the Computer Chess Club for proof). Speaking about biased....

Hans's picture

It is clear that none of the people here understand what has been going on. A team of people including former world champion Ed Schroder has taken a look at the 'evidence' given by the ICGA and as a matter of fact this has been a complete fraud. You can read the total debunking of the ICGA report here:

http://www.top-5000.nl/Rybka.htm

It is clear that the ICGA has used a report full of bogus code, which was falsified and fruitified to make it look like Rybka code, to convince others of 'code copying' This code copying hasn't been proved anywhere. The biggest fraud is in the PST section, where the ICGA panel has claimed that source codes are IDENTICAL. As you can view on Schroders website, they are definitely NOT the same. It is no surprise that Mark Watkins, the writer of the ICGA report, chose not to defend his pityful report when faced with the harsh truth.

Ironically dr Hyatt, one of the main accusers who never made a secret of his hate against Rajlich and Rybka, has been found copying Fruit material himself.

If you look at the FACTS and the claims by the ICGA, they are the biggest scandal in the history of computer chess. Especially dr Hyatt and David Levy should be ashamed of themselves for sentencing an innocent man.

Rumpelstiltskin's picture

The only FACT I can see is that Vas has never defended himself. Everything else is hearsay. My one question for Vas is if he is innocent, then why does he not defend himself? He (Vas) does not even defend his supporters.

Hans's picture

If you would be accused by 6 Hell's Angels that you stole one of their motor bikes and they invite you to come to defend yourself at their club house, would you do it?

Rajlich has clearly stated that the ICGA report is totally bogus and contains code that is not in Rybka.

Rumpelstiltskin's picture
Hans's picture

He HAS answered the ICGA: 'the report is bogus, it contains code that is not in Rybka'. And I am sure you would NOT go to the Hell's Angel club yourself ;-).

Rumpelstiltskin's picture

Really!? For all you know I could be a Hell's Angel member. I see you really do "Assume" allot don't you? ;-)

Hans's picture

I know a lot more than you do and 'knowing' is not 'asuming'... And it would surprise me if you are a Hell's Angel, as they are not noted for their interest in chess ;-).

Rumpelstiltskin's picture

Shows just how much you don't know! If you knew anything you would not be surprised!

Hans's picture

Well, in that case: what are you doing here! Get your motor bike out instead of snoozing around in boring chess forums!

Rumpelstiltskin's picture

Great idea, send me your address and I'll ride over there.

Ed Schroder's picture
Hans's picture

Here you can find evidence that the ICGA panel deliberately presented 'Rybka code' in such a way that it looked like Fruit:

http://www.top-5000.nl/fruitification.htm

Anthony's picture

All this would not happen if the absurd idea of 'intellectual property' would not exist.

Just imagine, just because you're the first to entertain a certain thought, you can call it your property.

More incredibly still: they are trying to make us believe, people are creative and think because they can claim intellectual property!

It stifles humanity's development, gets us into silly discussions like this and favors only megafirms and their banks, who monopolize knowledge.

THS's picture

Agreed. Bob Hyatt, one of the main Rybka accusers, has always stated that 'his program is 100% original'. Then they found out that he uses a lot of material of others and the stunner: without this material his program could not even play chess.... So far about 'originality'.

Hans's picture

Dr Levy's response is quite ridiculous, just to give you a few examples:

1) No court has made the finding of 'copyright infringement', still Levy is claiming this.

2) The claim of IDENTICAL formula's calculating Rybka PST's is totally wrong. See Ed Schroders website.

3) Levy claims 5 rule violations by Rybka, while the versions after 2.3.2a were not examined. This is just pure bogus.

The ICGA is currently in a very difficult situation: will they upheld a clearly bogus report that was favourable for a few programmers (suddenly getting world titles for free) and people who were hating Vas from the beginning, or are they REALLY interested in the truth?

noyb's picture

Excellent summary Hans! Nicely succinct.

Harvey Williamson's picture

It is not Hans it is Jeroen Nooman pretending to be Hans :-)

THS's picture

Who is Jeroen Nooman!?

Harvey Williamson's picture
Mark Lefler's picture

I find your arguments rather feeble:

>>1) No court has made the finding of 'copyright infringement', still Levy is claiming this.

Just because a court has not tried Rajlich (yet) does not mean the ICGA conclusions were wrong, and that Vas is innocent. Rybka was examined by a panel of peers and they agreed he was guilty.

>>2) The claim of IDENTICAL formula's calculating Rybka PST's is totally wrong. See Ed Schroders website.

The formulas and processes for generating the PST was identical, not the output of the formula since Rajlich used different constants. This is clearly described in Zach's and Mark's reports. I bet most of you have had algebra in school and understand how formulas work! If not, I suggest you pick up an algebra book.

>>3) Levy claims 5 rule violations by Rybka, while the versions after 2.3.2a were not examined. This is just pure bogus.

Versions 1 through 2.3.2a were shown to be derivatives. The ICGA Board decided a lifetime ban was appropriate. If 2.3.2a was found to be a derivative before Rajlich could enter future tournament, he would not have been allowed to compete. So it makes sense to take those trophies away as well.

Mike's picture

Ribka built proprietary stuff on top o free GPL community work. Rybka is strong due to the "bazaar" scheme, the same way Houdini is now the strongest program, because thousand or millions of integrated free minds will be always stronger than a dozen proprietary ones closed into a "cathedral" scheme.

Remco Gerlich's picture

Neither Rybka, Fruit or Houdini were ever built using the "bazaar" scheme, you vaguely recall something you don't really understand.

Mike's picture

Maybe I'm using new terms for you, so I recommend you to read "The Cathedral and the Bazaar" essay by Eric S. Raymond. And also note that the free version of Houdini, influenced by open source chess projects, had beaten Rybka. Always free community work will be stronger than the efforts of a few "copiers" of the community work. I think Chess programs are just too much complex to allow a few minds to beat thousand ones.

Hans's picture

'After 3 months of intensive research it's my final conclusion the accusers who investigated the Rybka chess program researched an original program doing all the same things other (good) programs also do. Rybka is an original chess program, the ICGA verdict and ban are uncalled for.'

Ed Schroder

Ellie's picture

and after how long did some take to believe the world was made in 7 days with species included DESPITE overwhelming evidence to the contrary

you dissemble the truth on your home page and deliberately mislead as soren riess attempted to

Pages

Latest articles