Scientists Create AI Program That Can Predict Human Rights Trials With 79 Percent Accuracy (theverge.com) 83
An anonymous reader quotes a report from The Verge: Computer scientists have created an AI program capable of predicting the outcome of human rights trials. The program was trained on data from nearly 600 cases brought before the European Court of Human Rights (ECHR), and was able to predict the court's final judgement with 79 percent accuracy. Its creators say it could be useful in identifying common patterns in court cases, but stress that they do not believe AI will be able to replace human judgement. As described in a study published in the journal PeerJ Computer Science, the AI program worked by analyzing descriptions of court cases submitted to the ECHR. These descriptions included summaries of legal arguments, a brief case history, and an outline of the relevant legislation. The cases were grouped into three main violations of human rights law, including the prohibition on torture and degrading treatment; the right to a fair trial; and the right to "respect for private and family life." (Used in a wide range of cases including illegal searches and surveillance.) The AI program then looked for patterns in this data, correlating the courts' final judgements with, for example, the type of evidence submitted, and the exact part of the European Convention on Human Rights the case was alleged to violate. Aletras says a number of patterns emerged. For example, cases concerning detention conditions (eg access to food, legal support, etc.) were more likely to end in a positive judgement that an individual's human rights had been violated; while cases involving sentencing issues (i.e., how long someone had been imprisoned) were more likely to end in acquittal. The researchers also found that the judgements of the court were more dependent on the facts of the case itself (that is to say, its history and its particulars) than the legal arguments (i.e., how exactly the Convention on Human Rights had or had not been violated).
Actually no. (Score:4, Informative)
You can't even tell if it's better than a coin toss. For this statistic to have any meaning at all you need to know the frequency the plaintiff wins. For example, let is suppose that the plaintiff wins in 79% of cases. Then an "AI" that merely always guess the plaintiff won would be correct in 79% of cases.
In fact given that it's unlikely the outcome is 50:50, then one would expect that such a dumb algorithm would be correct more often than not just by always guessing one side. It would therefore take very little extra "intelligence" so boost it over the top. In particular such intelligence could be simply an artifact of the data set. As an example suppost the data set contained 10% of plaintiffs whose names started with R. If this group of people won more often than the avergage, then simply learing to guess "win" anytime there was a plaintiff with an "R" name would improve the test. This is true even if you split the data up into cross validation sets, as the bias for "R" will persist on any randomly Chosen subset as well.
thus the results probably are meaningless. Certainly the article is.
Re: (Score:2)
You can't even tell if it's better than a coin toss. For this statistic to have any meaning at all you need to know the frequency the plaintiff wins.
Indeed. For instance, 99% of Japanese criminal cases end in conviction. So you could predict the result with 99% accuracy using the following algorithm:
1. Flip a coin
2. If it is heads, the defendant is guilty.
3. It it is tails, go to step 1.
Re: (Score:2)
That's the point.
Re: (Score:3)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
What in the world...?
It's trivially obvious that the way you would build such a model is to take a set of cases, subset them, identify the predictors of outcome (none of which is who won) in the subset, regress (in some way) the predictors with the outcomes in the subset, and then attempt to predict the outcome of the cases outside the subset using the
Re: (Score:3)
Sorry, but you're wrong. The remark is about the performance. 79% means nothing without knowing the baseline of an uninformed method. I think you can agree that a coin toss will produce the proper result in 50% of all cases. So if the performance of a system on a binary choice is 50%, it's as good as a coin toss, no matter how it's implemented. Suppose you make a system that always prints "plaintiff wins". Then its performance will be the actual win rate for the plaintiff. If that happens to be 79%, the sys
Re: (Score:2)
And considering "Refusal" is usually due to formal considerations that are well defined, the system should be able to predict refusal with a very high accuracy (...actually, the only inaccuracy would be human (clerical) error, when a case is wrongly passed or refused despite meeting or failing to meet the formal requirements) - and as result, with a system that has, say, 99.5% accuracy of determining between "Refused/Deliberated" (say, 0.5% of cases are wrongly refused or wrongly put under deliberation) the
Re: (Score:2)
Wrong. Read the article. It was a binary classifier. It doesn't matter if you Crossvalidate (subsetting) if the baseline frequencies are 79% for one of the classifications.
Re: (Score:1)
Re: (Score:2)
Dang it! (Score:1)
Here I am, only weeks away from completing my graduate degree in Predictive Human Rights Jurisprudence, and some jerkhead has built an artificial intelligence to take away my job!
I knew I should have stayed with medicine... we could have a cure for diabetic mice by now, if I hadn't listened to my advisor.
data analysis or "AI"? (Score:3)
why do people slap "AI" label on unnecessarily? publicity?
"The AI program then looked for patterns in this data, correlating the courts' final judgements with, for example, the type of evidence submitted ...a number of patterns emerged ...For example, cases concerning detention conditions ... more likely ...cases involving sentencing ...more likely"
this is mere data analysis. or is that what so called "AI" amount to?
and this,
"judgements of the court were more dependent on the facts of the case itself "
duh?!
how smart of so called "AI" to find that out?
Re: (Score:2)
Even if it is predictive, it relies on the 600 case training set being representative of future cases.
Human intelligence also relies on the future being similar to the past.
Re: (Score:2)
AI has always meant that in academy. Maybe you were using some other definition from, say, literature?
Re:data analysis or "AI"? (Score:4, Insightful)
this is mere data analysis. or is that what so called "AI" amount to?
"Mere" data analysis is when a human looks at the data and tries to find patterns. But it is "AI" when the algorithm is open ended, and finds it's own patterns and correlations. That is "machine learning" is certainly a branch of AI.
Re: (Score:1)
5 years ago we called that data mining.
I skimmed the paper. They converted the cases documents into word n-grams, did some clustering, made some type of n-gram by n-gram matrix that I've never learned about, did some more clustering, then fed those clusters into a SVM.
Data mining is a much better term, but I guess that's not fashionable anymore. In 5 more years is everything going to be about multidimensional awareness (MA)?
*All MA rights belong to Slashdot's ACs. Fear my pending patient.
Re: (Score:2)
... then fed those clusters into a SVM.
SVMs [wikipedia.org] are used in machine learning, which is a branch of AI.
Data mining is a much better term
"Data mining" and "machine learning" are two orthogonal concepts. You can do data mining with or without machine learning. Machine learning can be used for many things besides data mining.
Re: (Score:2)
"Mere" data analysis is when a human looks at the data and tries to find patterns. But it is "AI" when the algorithm is open ended, and finds it's own patterns and correlations.
I certainly agree that this falls under the classification of "AI" as a field. I'm guessing that part of the concern is also whether what was done here qualifies as "intelligence" rather than just a slightly more advanced algorithm for processing data.
Most research studies these days use fairly complex statistical computations -- often, lamentably, that the researchers themselves don't fully understand (or at least don't fully understand the limitations of). So, basically by the time many researchers ar
Al not A.I. (Score:2)
AI (Score:2)
What does AI stand for (Score:2)
Another Improbable program.
Comment removed (Score:5, Funny)
Re: (Score:2)
... What's the A stand for? [youtube.com]
Trial? (Score:2)
Most of them never come to trial
Terrorist like IISIS and Boko Haram will die rather than be captured
The leaders of large countries (eg Putin, GWB and the chinese) never submit to international tribunal
And regarding the remaining .21% ... (Score:2)
the endgame (Score:2)
Re: (Score:2)
My guess is that any forum you frequent will live down to that prediction...
Re: (Score:1)
Re: (Score:2)
Well I wrote a program to detect racism in Internet comments.
The first test, it returned notifications on ALL comment sections scanned.
There were no false positives.
Ummm (Score:2)
Not to be pissy or anything but let's face it. By the time a human rights violation has case has 'come to trial' it is nearly a forgone conclusion. Were 10K people massacred ? Was an entire countryside gassed ? Did the members of an entire tribe suddenly disappear ? Was a mushroom cloud, and nerve gas involved ? It doesn't take much of an AI or computer to come to a conclusion in those kinds of cases.
79% doesn't seem great (Score:1)
Re: (Score:2)
Conviction rate https://en.wikipedia.org/wiki/... [wikipedia.org]
"For 2012, the US Department of Justice reported a 93% conviction rate."
Re: (Score:1)
That's not really the best comparison though. ECtHR is more comparable to the US Supreme Court. Mostly dealing with appeals based on incompatibilities of legislation with fundamental law.
Re: (Score:2)
Re: (Score:2)
I hope one day it can predict... (Score:2)
The outcome of Bush, Obama, Cheney and Yoo's trials.
So they like facts more than legal arguments? Thats great, we have lots of those. Drone strikes targeted by cell phone data killing innocent people. A whole system of assasination based on paid informants and lies. A torture program that was swept under the rug rather than exposed and prosecuted....
Lots and lots of facts for them.
Shell script (Score:2)
I can predict Egyptian and Russian human rights trials with 100% accuracy with a simple shell script.
echo "Guilty!"
so is that good? (Score:2)
So I coin toss could predict it with 50% accuracy. How well can a human being perform the same task? Not sure that 20% better then a coin toss makes your AI very intelligent. Of coarse I'm sure it is just the beginning of a lot of cool work.
Assuming you have "standing" (Score:2)
If you don't, you have no rights.
BTW, there is no right to "standing" or for the government to reveal that you even potentially have "standing.