## Archive for the ‘Skepticism’ Category

### Dear “Skeptics,”… [Attn: All Data Scientists]

Tuesday, May 24th, 2016

Strings and multiverses can’t be experimentally detected. The theories aren’t falsifiable, which makes them pseudo-scientific, like astrology and Freudian psychoanalysis. Credit: parameter_bond/Flickr

The caption is from Horgan’s post. In case anyone asks, I retrieved and re-sized my own copy of the image.

From the post:

I hate preaching to the converted. If you were Buddhists, I’d bash Buddhism. But you’re skeptics, so I have to bash skepticism.

I’m a science journalist. I don’t celebrate science, I criticize it, because science needs critics more than cheerleaders. I point out gaps between scientific hype and reality. That keeps me busy, because, as you know, most peer-reviewed scientific claims are wrong.

So I’m a skeptic, but with a small S, not capital S. I don’t belong to skeptical societies. I don’t hang out with people who self-identify as capital-S Skeptics. Or Atheists. Or Rationalists.

When people like this get together, they become tribal. They pat each other on the back and tell each other how smart they are compared to those outside the tribe. But belonging to a tribe often makes you dumber.

Here’s an example involving two idols of Capital-S Skepticism: biologist Richard Dawkins and physicist Lawrence Krauss. Krauss recently wrote a book, A Universe from Nothing. He claims that physics is answering the old question, Why is there something rather than nothing?

Krauss’s book doesn’t come close to fulfilling the promise of its title, but Dawkins loved it. He writes in the book’s afterword: "If On the Origin of Species was biology’s deadliest blow to supernaturalism, we may come to see A Universe From Nothing as the equivalent from cosmology."

Just to be clear: Dawkins is comparing Lawrence Krauss to Charles Darwin. Why would Dawkins say something so foolish? Because he hates religion so much that it impairs his scientific judgment. He succumbs to what you might call “The Science Delusion.”

“The Science Delusion” is common among Capital-S Skeptics. You don’t apply your skepticism equally. You are extremely critical of belief in God, ghosts, heaven, ESP, astrology, homeopathy and Bigfoot. You also attack disbelief in global warming, vaccines and genetically modified food.

These beliefs and disbeliefs deserve criticism, but they are what I call “soft targets.” That’s because, for the most part, you’re bashing people outside your tribe, who ignore you. You end up preaching to the converted.

Meanwhile, you neglect what I call hard targets. These are dubious and even harmful claims promoted by major scientists and institutions. In the rest of this talk, I’ll give you examples of hard targets from physics, medicine and biology. I’ll wrap up with a rant about war, the hardest target of all.

To get the full flavor of what it means to be a skeptic, read this post and John’s accounts of the reactions to both his presentation and this post.

The “tell” of a target

Whether you are being skeptical of a popular (read “soft”) target like Bigfoot or skeptical of a “hard” target like psychiatric drugs, the reaction from believers is nearly universal: anger, denial and fairly rapidly, denunciation of yourself as unreasonable, etc.

Try being skeptical of a soft/hard target in your work.

Ask if there is racial bias in the algorithms you use day to day? Gender bias? If the answer is no, ask how do they know? Ask them to confirm it for you using data. What their hands closely during the demonstration.

After all, you are a data scientist and questions should be settled based on data and understanding the algorithms applied to them.

Yes?

Being a skeptic with a small “s” is a hard job. But your project, department, enterprise will be better for you being that skeptic.

Imagine one effective White House skeptic prior to the second war on Iraq. No $trillions spent, no countless lives lost, no instability in the region, etc. Skeptics with a small “s” can make all the difference in the world. ### Experts, Sources, Peer Review, Bad Poetry and Flint, Michigan. Sunday, January 31st, 2016 From the post: Officials behind the launch of a major initiative detailing lives of ordinary soldiers during the First World War were embarrassed by the discovery that they had mistakenly included the work of Blackadder character, Baldrick, in the achieve release. The work, entitled ‘The German Guns’ and attributed to Private S.O. Baldrick, was actually written by the sitcom’s writers Richard Curtis and Ben Elton some 70 years after the end of the conflict. Elton was reported to be “delighted at the news” and friends said he was already checking to see if royalty payments may be due. Although the archive release was scrutinised by experts, it is understood that the Baldrick poem was approved after a clerk recalled hearing Education Secretary Michael Gove referring to Baldrick in relation to the Great War, and assumed that he was of contemporary cultural significance. Another illustration that experts and peer review aren’t the gold standards of correctness. Or to put it differently: Mistakes happen, especially without sources. If the only surviving information was Education Secretary Michael Gove referring to Baldrick, not only would the mistake be perpetuated but it would be immune to correction. Citing and/or pointing to a digital resource that was the origin of the poem, would be more likely to trip warnings (by date of publication) or contain a currently recognizable reference, such as Blackadder. The same lesson should be applied to reports such as Michael Moore’s claim: 1. While the Children in Flint Were Given Poisoned Water to Drink, General Motors Was Given a Special Hookup to the Clean Water. A few months after Gov. Snyder removed Flint from the clean fresh water we had been drinking for decades, the brass from General Motors went to him and complained that the Flint River water was causing their car parts to corrode when being washed on the assembly line. The governor was appalled to hear that GM property was being damaged, so he jumped through a number of hoops and quietly spent$440,000 to hook GM back up to the Lake Huron water, while keeping the rest of Flint on the Flint River water. Which means that while the children in Flint were drinking lead-filled water, there was one—and only one—address in Flint that got clean water: the GM factory.

Verification is especially important for me because I think Michael Moore is right and that predisposes me to accept his statements, without evidence.

In no particular order:

• What “brass” from GM? Names, addresses, contact details. Links to statements?
• What evidence did the “brass” present? Documents? Minutes of the meeting? Date?
• What hoops did the Governor jump through? Who else in state government was aware of the request?

### Core Econ: a free economics textbook

Wednesday, November 5th, 2014

Core Econ: a free economics textbook by Cathy O’Neil.

From the post:

Today I want to tell you guys about core-econ.org, a free (although you do have to register) textbook my buddy Suresh Naidu is using this semester to teach out of and is also contributing to, along with a bunch of other economists.

(image omitted)

It’s super cool, and I wish a class like that had been available when I was an undergrad. In fact I took an economics course at UC Berkeley and it was a bad experience – I couldn’t figure out why anyone would think that people behaved according to arbitrary mathematical rules. There was no discussion of whether the assumptions were valid, no data to back it up. I decided that anybody who kept going had to be either religious or willing to say anything for money.

Not much has changed, and that means that Econ 101 is a terrible gateway for the subject, letting in people who are mostly kind of weird. This is a shame because, later on in graduate level economics, there really is no reason to use toy models of society without argument and without data; the sky’s the limit when you get through the bullshit at the beginning. The goal of the Core Econ project is to give students a taste for the good stuff early; the subtitle on the webpage is teaching economics as if the last three decades happened.

Skepticism of government economic forecasts and data requires knowledge of the lingo and assumptions of economics. This introduction won’t get you to that level but it is a good starting place.

Enjoy!

### Suppressing Authentic Information

Monday, November 3rd, 2014

In my continuing search for information on the authenticity of Dabiq (see: Dabiq, ISIS and Data Skepticism) I encountered Slick, agile and modern – the IS media machine by Mina Al-Lami.

Mina makes it clear that IS (ISIL/ISIS) has been the target of a campaign to shut down all authentic outlets for news from the group:

IS has always relied heavily on hordes of online supporters to amplify its message. But their role has become increasingly important in recent months as the group’s official presence on a variety of social media platforms has been shut down and moved underground.

The group’s ability to keep getting its message out in the face of intensive counter-measures is due to the agility, resilience and adaptability of this largely decentralized force.

Until July this year, IS, like most jihadist groups, had a very strong presence on Twitter, with all its central and regional media outlets officially active on the platform. However, its military successes on the ground in Iraq and Syria in June triggered a concerted and sustained clampdown on the group’s accounts.

IS was initially quick to replace these accounts, in what became a game of whack-a-mole between IS and the Twitter administration. But by July the group appeared to have abandoned any attempt to maintain an official open presence there.

Instead, IS began experimenting with a string of less known social media platforms. These included the obscure Friendica, Quitter and Diaspora – all of which promise better privacy and data-protection than Twitter – as well as the popular Russian VKontakte.

Underground channels

While accounts on Friendica and Quitter were shut down within days, the official IS presence on Diaspora and VKontakte lasted several weeks before their involvement in the distribution of high profile beheading videos caused them too to be shut down.

Since the accounts on VKontakte were closed in September, IS appears to have resorted to underground channels to surface its material, making no attempt to advertise an official social media presence. Perhaps surprisingly, this has not yet caused any problems for the group in terms of authenticating its output.

Once a message has surfaced – via channels that are currently difficult to pin down – it is disseminated by loosely affiliated media groups who are capable of mobilizing a vast network of individual supporters on social media to target specific audiences.

Unfortunately, Mina misses the irony of reporting that IS has no authentic outlets in one breath to relying in the next breath on non-authentic materials (such as Dabiq) to talk about the group’s social media prowess.

Suppression of authentic content outlets for IS leaves an interested reader at the mercy of governments, news organizations and others who have a variety of motives for attributing content to IS.

As I mentioned in my last post:

Debates about international and national policy should not be based on faked evidence (such as “yellow cake uranium“) or faked publications.

I have heard the argument that IS content recruits support for terrorism. I have read propaganda attributed to IS, the Khmer Rouge, the KKK and terrorists sponsored by Western governments. I can report not the slightest interest in supporting or participating with any of them.

The recruitment argument is a variation of the fear of allowing gays, drug use, drinking, etc., on television would result in children growing up to be gay drug addicts with drinking problems. I can report that no sane person credits that fear today. (If you have that fear, contact your local mental health service for an appointment.)

Why is IS attractive? Hard to say given the lack of authentic information on its goals and platform, perhaps its reported opposition to corrupt governments in the Middle East?

If I weren’t concerned with corrupt Western governments I might be more concerned with governments in the Middle East. But, as they say, best to start cleaning your own house before complaining about the state of another’s.

### Dabiq, ISIS and Data Skepticism

Wednesday, October 22nd, 2014

If you are following the Middle East, no doubt you have heard that ISIS/ISIL publishes Dabiq, a magazine that promotes its views. It isn’t hard to find articles quoting from Dabiq, but I wanted to find copies of Dabiq itself.

Clarion Project (Secondary Source for Dabiq)

After a bit of searching, I found that the Clarion Project is posting every issue of Dabiq as it appears.

The hosting site, Clarion Project, is a well known anti-Muslim hate group. The founders of the Clarion Project just happened to be full time employees of Aish Hatorah, a pro-Israel organization.

Coverage of Dabiq by Mother Jones (who should know better), ISIS Magazine Promotes Slavery, Rape, and Murder of Civilians in God’s Name relies on The Clarion Project “reprint” of Dabiq.

Internet Archive (Secondary Source for Dabiq)

All the issues at the Internet Archive claim to be from: “The Islamic State Al-Hayat Media Centre (HMC). I say “claim to be from” because uploading to the Internet Archive only requires an account with a verified email address. Anyone could have uploaded the documents.

Robert Mackey writes for the New York Times: Islamic State Propagandists Boast of Sexual Enslavement of Women and Girls and references Dabiq. I asked Robert for his source for Dabiq and he responded that it was the Internet Archive version.

Wall Street Journal

In Why the Islamic State Represents a Dangerous Turn in the Terror Threat, Gerald F. Seib writes:

It isn’t necessary to guess at what ISIS is up to. It declares its aims, tactics and religious rationales boldly, in multiple languages, for all the world to see. If you want to know, simply call up the first two editions of the organization’s remarkably sophisticated magazine, Dabiq, published this summer and conveniently offered in English online.

Gerald implies, at least to me, that Dabiq has a “official” website where it appears in multiple languages. But if you read Gerald’s article, there is no link to such a website.

I wrote to Gerald today to ask what site he meant when referring to Dabiq. I have not heard back from Gerald as of posting but will insert his response when it arrives.

The Jamestown Foundation

The Jamestown Foundation website featured: Hot Issue: Dabiq: What Islamic State’s New Magazine Tells Us about Their Strategic Direction, Recruitment Patterns and Guerrilla Doctrine by Michael W. S. Ryan, saying:

On the first day of Ramadan (June 28), the Islamic State in Iraq and Syria (ISIS) declared itself the new Islamic State and the new Caliphate (Khilafah). For the occasion, Abu Bakr al-Baghdadi, calling himself Caliph Ibrahim, broke with his customary secrecy to give a surprise khutbah (sermon) in Mosul before being rushed back into hiding. Al-Baghdadi’s khutbah addressed what to expect from the Islamic State. The publication of the first issue of the Islamic State’s official magazine, Dabiq, went into further detail about the Islamic State’s strategic direction, recruitment methods, political-military strategy, tribal alliances and why Saudi Arabia’s concerns that the Kingdom may be the Islamic State’s next target are well-founded.

Which featured a thumbnail of the cover of the first issue of Dabiq, with the following legend:

Dabiq Magazine (Source: Twitter user @umOmar246)

Well, that’s a problem because the Twitter user “@umOmar246” doesn’t exist.

Don’t take my word for it, go to Twitter, search for “umOmar246,” limit search results to people and you will see:

I took the screen shot today just in case the results change at some point in time.

Other Media

Other media carry the same stories but without even attempting to cite a source. For example:

Jerusalem Post: ISIS threatens to conquer the Vatican, ‘break the crosses of the infidels’. Source? None.

Global News: The twisted view of ISIS in latest issue of propaganda magazine Dabiq by Nick Logan.

I don’t think that Nick appreciates the irony of the title of his post. Yes, this is a twisted view of ISIS. The question is who is responsible for it?

Pick any issue of Dabiq and skim through it. What impressed me was the “over the top” presentation of cruelty. The hate literature I am familiar with (I grew up in the Deep South in the 1960’s) usually portrays the atrocities of others, not the group in question. Hate literature places its emphasis on the “other” group, the one to be targeted, not itself.

Analysis

First and foremost, the lack of any “official” site of origin for Dabiq makes me highly suspicious of the authenticity of the materials that claim to originate with ISIS.

Second, why would ISIS rely upon the Clarion Project as a distributor for its English language version of Dabiq, along with the Internet Archive?

Third, what are we to make of missing @umOmar246 from Twitter? Before you say that the account has closed, http://twittercounter.com/
doesn’t know that user either:

A different aspect of consistency on distributed data. The aspect of getting “caught” because distributed data is difficult to make consistent.

Fourth, the media coverage examined relies upon sites with questionable authenticity but cites the material found there as though authoritative. Is this a new practice in journalism? Some of the media outlets examined are hardly new and upcoming Internet news sites.

Finally, the content of the magazines themselves don’t ring true for hate literature.

Conclusion

Debates about international and national policy should not be based on faked evidence (such as “yellow cake uranium“) or faked publications.

Based on what I have uncovered so far, attributing Dabiq to ISIS is highly questionable.

It appears to be an attempt to discredit ISIS and to provide a basis for whipping up support for military action by the United States and its allies.

The United States destroyed the legitimate government of Iraq on the basis of lies and fabrications. If only for nationalistic reasons, not spending American funds and lives based on a tissue of lies, let’s not make the same mistake again.

Disclaimer: I am not a supporter of ISIS nor would I choose to live in their state should they establish one. However, it will be their state and I lack the arrogance to demand that others follow social, religious or political norms that I prefer.

PS: If you have suggestions for other factors that either confirm a link between ISIS and Dabiq or cast further doubt on such a link, please post them in comments. Thanks!

### Know Your Algorithms and Data!

Sunday, September 21st, 2014

If you let me pick the algorithm or the data, I can produce any result you want.

Something to keep in mind when listening to reports of “facts.”

Or as Nietzsche would say:

There are no facts, only interpretations.

There are people who are so naive that they don’t realize interpretations other than their are possible. Avoid them unless you have need of followers for some reason.

I first saw this in a tweet by Chris Arnold.

### Credulity Question for Interviewees

Tuesday, July 1st, 2014

Max Fisher authored: Map: The 193 foreign countries the NSA spies on and the 4 it doesn’t, which has the following map:

Max covers the history of the authority of the NSA to spy on governments, organizations, etc., so see his post for the details.

A credulity question for interviewees:

What countries are being spied upon by the NSA without permission? Color in those countries with a #2 pencil.

If they make no changes to the map, you can close the interview early. (The correct answer is six, including the United States.)

Clearly a candidate for phishing attacks, violation of security protocols, pass phrase/password sharing, frankly surprised they made it to the interview.

### The case for big cities, in 1 map

Thursday, February 20th, 2014

The case for big cities, in 1 map by Chris Cillizza.

From the post:

New Yorkers who don’t live in New York City hate the Big Apple. Missourians outside of St. Louis and Kansas City are skeptical about the people (and politicians) who come from the two biggest cities in the state. Politicians from the Chicago area (and inner suburbs) often meet skepticism when campaigning in downstate Illinois. You get the idea. People who don’t live in the big cities tend to resent those who do.

Fair enough. Growing up in semi-rural southeastern Connecticut, I always hated Hartford. (Not really.) But, this map built by Reddit user Alexandr Trubetskoy shows — in stark terms — how much of the country’s economic activity (as measured by the gross domestic product) is focused in a remarkably small number of major cities.

A great map, at least if you live in the greater metro area of any of these cities.

I could 21 red spots, although on the East coast they are so close together some were fused together.

It is also an illustration that a map doesn’t always tell the full story.

Say 21 or more cities produce have of the GDP.

Care to guess how many states are responsible for 50% of the agricultural production in the United States?

Thursday, February 20th, 2014

Mapping Twitter Topic Networks: From Polarized Crowds to Community Clusters by Marc A. Smith, Lee Rainie, Ben Shneiderman and Itai Himelboim.

From the post:

Conversations on Twitter create networks with identifiable contours as people reply to and mention one another in their tweets. These conversational structures differ, depending on the subject and the people driving the conversation. Six structures are regularly observed: divided, unified, fragmented, clustered, and inward and outward hub and spoke structures. These are created as individuals choose whom to reply to or mention in their Twitter messages and the structures tell a story about the nature of the conversation.

If a topic is political, it is common to see two separate, polarized crowds take shape. They form two distinct discussion groups that mostly do not interact with each other. Frequently these are recognizably liberal or conservative groups. The participants within each separate group commonly mention very different collections of website URLs and use distinct hashtags and words. The split is clearly evident in many highly controversial discussions: people in clusters that we identified as liberal used URLs for mainstream news websites, while groups we identified as conservative used links to conservative news websites and commentary sources. At the center of each group are discussion leaders, the prominent people who are widely replied to or mentioned in the discussion. In polarized discussions, each group links to a different set of influential people or organizations that can be found at the center of each conversation cluster.

While these polarized crowds are common in political conversations on Twitter, it is important to remember that the people who take the time to post and talk about political issues on Twitter are a special group. Unlike many other Twitter members, they pay attention to issues, politicians, and political news, so their conversations are not representative of the views of the full Twitterverse. Moreover, Twitter users are only 18% of internet users and 14% of the overall adult population. Their demographic profile is not reflective of the full population. Additionally, other work by the Pew Research Center has shown that tweeters’ reactions to events are often at odds with overall public opinion— sometimes being more liberal, but not always. Finally, forthcoming survey findings from Pew Research will explore the relatively modest size of the social networking population who exchange political content in their network.

Great study on political networks but all the more interesting for introducing an element of sanity into discussions about Twitter.

At a minimum, Twitter having 18% of all Internet users and 14% of the overall adult population casts serious doubt on metrics using Twitter to rate software popularity. (“It’s all we have” is a pretty lame excuse for using bad metrics.)

Not to say it isn’t important to mine Twitter data for what content it holds but at the same time to remember Twitter isn’t the world.

### On Being a Data Skeptic

Saturday, February 15th, 2014

On Being a Data Skeptic by Cathy O’Neil. (pdf)

From Skeptic, Not Cynic:

I’d like to set something straight right out of the gate. I’m not a data cynic, nor am I urging other people to be. Data is here, it’s growing, and it’s powerful. I’m not hiding behind the word “skeptic” the way climate change “skeptics” do, when they should call themselves deniers.

Instead, I urge the reader to cultivate their inner skeptic, which I define by the following characteristic behavior. A skeptic is someone who maintains a consistently inquisitive attitude toward facts, opinions, or (especially) beliefs stated as facts. A skeptic asks questions when confronted with a claim that has been taken for granted. That’s not to say a skeptic brow-beats someone for their beliefs, but rather that they set up reasonable experiments to test those beliefs. A really excellent skeptic puts the “science” into the term “data science.”

In this paper, I’ll make the case that the community of data practitioners needs more skepticism, or at least would benefit greatly from it, for the following reason: there’s a two-fold problem in this community. On the one hand, many of the people in it are overly enamored with data or data science tools. On the other hand, other people are overly pessimistic about those same tools.

I’m charging myself with making a case for data practitioners to engage in active, intelligent, and strategic data skepticism. I’m proposing a middle-of-the-road approach: don’t be blindly optimistic, don’t be blindly pessimistic. Most of all, don’t be awed. Realize there are nuanced considerations and plenty of context and that you don’t necessarily have to be a mathematician to understand the issues.
….

It’s a scant 26 pages, cover and all but “On Being a Data Skeptic” is well worth your time.

I particularly liked Cathy’s coverage of issues such as: People Get Addicted to Metrics, which ends with separate asides to “nerds,” and “business people.” Different cultures and different ways of “hearing” the same content. Rather than trying to straddle those communities, Cathy gave them separate messages.

You will find her predator/prey model particularly interesting.

On the whole, I would say her predator/prey analysis should not be limited to modeling. See what you think.

### Finding Occam’s razor in an era of information overload

Thursday, November 21st, 2013

Finding Occam’s razor in an era of information overload

From the post:

How can the actions and reactions of proteins so small or stars so distant they are invisible to the human eye be accurately predicted? How can blurry images be brought into focus and reconstructed?

A new study led by physicist Steve Pressé, Ph.D., of the School of Science at Indiana University-Purdue University Indianapolis, shows that there may be a preferred strategy for selecting mathematical models with the greatest predictive power. Picking the best model is about sticking to the simplest line of reasoning, according to Pressé. His paper explaining his theory is published online this month in Physical Review Letters, a preeminent international physics journal.

“Building mathematical models from observation is challenging, especially when there is, as is quite common, a ton of noisy data available,” said Pressé, an assistant professor of physics who specializes in statistical physics. “There are many models out there that may fit the data we do have. How do you pick the most effective model to ensure accurate predictions? Our study guides us towards a specific mathematical statement of Occam’s razor.”

Occam’s razor is an oft cited 14th century adage that “plurality should not be posited without necessity” sometimes translated as “entities should not be multiplied unnecessarily.” Today it is interpreted as meaning that all things being equal, the simpler theory is more likely to be correct.

Comforting that the principles of good modeling have not changed since the 14th century. (Occam’s Razor)

Bear in mind Occam’s Razor is guidance and not a hard and fast rule.

On the other hand, particularly with “big data,” be wary of complex models.

Especially the ones that retroactively “predict” unique events as a demonstration of their model.

If you are interested in the full “monty:”

Nonadditive Entropies Yield Probability Distributions with Biases not Warranted by the Data by Steve Pressé, Kingshuk Ghosh, Julian Lee, and Ken A. Dill. Phys. Rev. Lett. 111, 180604 (2013)

Abstract:

Different quantities that go by the name of entropy are used in variational principles to infer probability distributions from limited data. Shore and Johnson showed that maximizing the Boltzmann-Gibbs form of the entropy ensures that probability distributions inferred satisfy the multiplication rule of probability for independent events in the absence of data coupling such events. Other types of entropies that violate the Shore and Johnson axioms, including nonadditive entropies such as the Tsallis entropy, violate this basic consistency requirement. Here we use the axiomatic framework of Shore and Johnson to show how such nonadditive entropy functions generate biases in probability distributions that are not warranted by the underlying data.

### Statistics Done Wrong

Saturday, November 2nd, 2013

Statistics Done Wrong by Alex Reinhart.

From the post:

If you’re a practicing scientist, you probably use statistics to analyze your data. From basic t tests and standard error calculations to Cox proportional hazards models and geospatial kriging systems, we rely on statistics to give answers to scientific problems.

This is unfortunate, because most of us don’t know how to do statistics.

Statistics Done Wrong is a guide to the most popular statistical errors and slip-ups committed by scientists every day, in the lab and in peer-reviewed journals. Many of the errors are prevalent in vast swathes of the published literature, casting doubt on the findings of thousands of papers. Statistics Done Wrong assumes no prior knowledge of statistics, so you can read it before your first statistics course or after thirty years of scientific practice.

Dive in: the whole guide is available online!

As a matter of fact, a summary of warning signs for these problems would fit on 81/2 by 11 (or A4) paper.

Thinking when you show up to examine a data set, you have Statistic Done Wrong with the web address on the back of your laminated cheat sheets.

Part of being a data skeptic is intuiting where to push so that the data “as presented” unravels.

I first saw this in Nat Torkington’s Four short links: 30 October 2013.

### PubMed Commons

Sunday, October 27th, 2013

PubMed Commons

From the webpage:

PubMed Commons is a system that enables researchers to share their opinions about scientific publications. Researchers can comment on any publication indexed by PubMed, and read the comments of others. PubMed Commons is a forum for open and constructive criticism and discussion of scientific issues. It will thrive with high quality interchange from the scientific community. PubMed Commons is currently in a closed pilot testing phase, which means that only invited participants can add and view comments in PubMed.

Just in case you are looking for a place to practice your data skepticism skills.

In closed beta now but when it opens up…, pick an article in a field that interests you or at random.

Just my suggestion but try to do very high quality comments and check with others on your analysis.

A record of to the point, non-shrill, substantive comments might be a nice addition to your data skeptic resume. (Under papers re-written/retracted.)

### Lectures on scientific computing with Python

Sunday, October 27th, 2013

Lectures on scientific computing with Python by J.R. Johansson.

From the webpage:

A set of lectures on scientific computing with Python, using IPython notebooks.

Read only versions of the lectures:

To debunk pitches, proposals, articles, demos, etc., you will need to know, among other things, how scientific computing should be done.

Scientific computing is a very large field so take this as a starting point, not a destination.

### Trouble at the lab [Data Skepticism]

Sunday, October 27th, 2013

From the web page:

“I SEE a train wreck looming,” warned Daniel Kahneman, an eminent psychologist, in an open letter last year. The premonition concerned research on a phenomenon known as “priming”. Priming studies suggest that decisions can be influenced by apparently irrelevant actions or events that took place just before the cusp of choice. They have been a boom area in psychology over the past decade, and some of their insights have already made it out of the lab and into the toolkits of policy wonks keen on “nudging” the populace.

Dr Kahneman and a growing number of his colleagues fear that a lot of this priming research is poorly founded. Over the past few years various researchers have made systematic attempts to replicate some of the more widely cited priming experiments. Many of these replications have failed. In April, for instance, a paper in PLoS ONE, a journal, reported that nine separate experiments had not managed to reproduce the results of a famous study from 1998 purporting to show that thinking about a professor before taking an intelligence test leads to a higher score than imagining a football hooligan.

The idea that the same experiments always get the same results, no matter who performs them, is one of the cornerstones of science’s claim to objective truth. If a systematic campaign of replication does not lead to the same results, then either the original research is flawed (as the replicators claim) or the replications are (as many of the original researchers on priming contend). Either way, something is awry.

The numbers will make you a militant data skeptic:

• Original results could be duplicated for only 6 out of 53 landmark studies of cancer.
• Drug company could reproduce only 1/4 of 67 “seminal studies.”
• NIH official estimates at least three-quarters of publishing biomedical finding would be hard to reproduce.
• Three-quarter of published paper in machine learning are bunk due to overfitting.

Those and more examples await you in this article from The Economist.