扬声器 | 成绩单 |
| 斯坦杨 | 哈哈。 |
| 萨拉·杜特 | 允许你。 |
| 斯坦杨 | 现在看到一切。 |
| 好的，哦，不，我要从我身边运行幻灯片。 |
| 萨拉·杜特 | 是啊是啊就像你会一个正常的会议。 |
| 好了，冷静，确保我得到了一切，如果我近视的话。好的，我们正在录制我需要的我要我要关掉我的相机，仅供参考，然后。 |
| 好的。 |
| 斯坦杨 | 为了使这个全屏，所以现在只需排序。 |
| 如何做到这一点的第二。 |
| 萨拉·杜特 | 就是那个底部，上面写着他们对 34% 的行为的注释，它看起来像一个。 |
| 下船是的，对那个不好的人感到抱歉，在那个人的另一边。 |
| 一个是的，那个家伙。 |
| 斯坦杨 | 哈哈。 |
| 萨拉·杜特 | 你走了，我的意思是我要走了，我们马上就要开始了，但我要去见我自己，在我们开始之前你有什么问题吗？ |
| 斯坦杨 | 或者你真的不会像幻灯片一样录制我。 |
| 萨拉·杜特 | 是的，是的，他们问你有。 |
| 上面的相机。 |
| 过了就好了。 |
| 正确的。 |
| 好的。 |
| 斯坦杨 | 我准备好了。 |
| 萨拉·杜特 | 我要让自己静音，如果你有问题或需要停下来，请告诉我，否则我会让你知道的。 |
| 斯坦杨 | 去。 |
| 今天我将介绍用于评估多项研究的 JMP 插件和脚本。 或者你可以称之为如何抓住 p-hackers、用统计数据作弊的人，或者为什么大多数科学主张是错误的。 |
| 我将首先描述拼图部分以及它们如何组合在一起。 大多数声明实际上无法复制。 这些是科学主张。 |
| 我在争论，我的合著者也在争论，这是由于 p-hacking 造成的，这是一个主要问题。 我们将使用元分析和 P 值图来捕捉它们，所以这就是捕捉骗子的方法。 |
| JMP 插件和脚本是来自风险比率和置信限的 P 值。 它们来自元分析。 |
| 然后是Fisher的P值组合，一种类似于元分析技术的古老技术。 然后我们将展示一个 p 值图，我们有一个小脚本可以清理它并使其成为一个像样的 |
| 图片。 |
| 好吧，我们在天空中看到了一只兔子。 天空中有很多云，如果你在美好的一天坐在外面环顾四周，你可能会发现一只兔子。 |
| 所以这是一个随机事件。 当然，兔子实际上并不在天空中。 |
| Gelman 和 Loken 实际上发表了一篇小文章，说科学中的统计危机，所以人们使用统计数据是错误的，这就是我们今天要讨论的内容。 |
| 让我们进行一个流行病学实验。 |
| 我们有 10 面骰子，红色、白色和蓝色。 它们将变成数字（在这种特殊情况下为 0.046）。 让我们亲眼目睹这一切发生在我们眼前。 |
| 现在我们有一个 P 值。 这是随机的。 你自己做的。 这么好玩，不如我们再来一次？ |
| 红白， |
| 和蓝色。 |
| 现在，如果你这样做 60 次，你可以在这里填写表格。 这是我用 10 面骰子做的模拟。 您可以在四列和 15 行中看到 P 值。 所以这是 60 个 P 值。 我最小的 P 值是 0.004。 |
| 使用 60 个 P 值，您可以计算出概率。 大约 95% 的情况下，您至少有一个 P 值小于 0.05。 |
| 这是我女儿做的。 她的三个 P 值小于 0.05；他们被圈在这里。 然后进行流行病学家实验就这么简单 |
| 甚至我的妻子帕特也能做到，她在这里有三篇潜在的论文，她可以根据掷骰子和旋转一个好故事来写。 |
| P 值具有附加到它们的期望值。 所以在右边，我们的 P 值为 0.004、0.016 和 0.012。 附加到每个 P 值的是正常偏差。 您可以看到我的 .004 的正常偏差为 2.9，依此类推。 在左边，我们有 |
| 样本量、预期偏差和最小 P 值的预期 P 值，然后是偏差。 所以如果我们有 400 |
| 我们正在研究的问题，预期的 P 值为 0.00487，偏差为 2.968。 现在，正是这种偏差从基础实验带到了元分析的计算中，随着我们继续进行，我们将看到这一点。 |
| 流行病学中有多少说法是正确的？ 我在 2011 年发表了一篇论文，并接受了在观察性研究中提出的主张。 |
| 对于观察性研究中的每一项主张，我都发现了一项针对完全相同问题的随机临床试验。 所以在 12 项研究中，有 |
| 可以提出和测试的 52 项声明。 |
| 如果您查看正下方的列，则这 52 项索赔中有 0 项以正确的方向复制。 |
| 因此，就他们的主张而言，流行病学家的 0 比 52。 实际上有五项声明具有统计学意义，但它们与……观察性研究中声明的方向相反。 |
| 我们要看看一个 |
| 疯狂的问题。 谷物决定人类性别吗？ 所以如果你吃早餐麦片，你更有可能生男孩吗？ 嗯，这就是第一篇论文所说的。 |
| 这篇论文发表在皇家学会 B，这是他们的主要生物学期刊。 所以这三位合著者声称，如果你吃早餐麦片，你更有可能生男孩，在受孕期间和前后吃早餐麦片。 |
| 我和我的两个同伙看了这个，我们要求提供数据。 我们得到了数据，然后我们发表了对第一篇论文的反驳，说谷物引起的性别选择很可能是多重测试假阳性。 |
| 在受孕前的时间...预测受孕，就在受孕前后。 每份问卷中有 131 种食物，总共进行了 262 次统计测试。 |
| 您计算这些测试的 P 值，将它们从最小到最大排序，然后将它们与整数进行对比（这是底部的排名）， |
| 您会看到看起来很不错的 45 度线。 因此，我们正在研究均匀分布。 现在他们的主张来自左下角，他们说，嗯，这是一个小的 P 值。 小 P 值表示吃麦片...早餐麦片会导致更多 |
| 男孩婴儿。 |
| 很明显是统计上的误报。 |
| P 值图。 因此，我们将大量使用 P 值图。 在左侧，我们看到了一个可能为真零假设的 P 值图。 |
| 老年人长期运动训练不会导致死亡或死亡风险。 在右边，我们有吸烟和肺癌。 |
| 我们看到一大堆 P 值几乎接近于零，一直在整个页面上，右边还有一些散乱的人。 So this the right-hand picture is evidence for a real effect and the left-hand picture is support of the null hypothesis, no effect. |
| let's talk about a meta analysis because I am going to use those during the course of this lecture. On the left, we see a funnel and we see lots of papers dropping into the funnel. |
| The epidemiologist or whoever's doing the meta analysis picks what they think are high quality papers and they use those for further analysis. On the right, we see the...sort of the |
| evidence hierarchy. A meta analysis over many studies is considered |
| high-level information and then down at the bottom, expert opinion and so forth and so on. So the higher you go up the pyramid, |
| people contend the evidence is is better. |
| We're going to look at two example meta analysis papers. The first paper is by Orellano. |
| nitric oxide, ozone, |
| small particles and so forth. And it gathered data from all over the...all over the |
| world, and this was sponsored by the |
| WHO, so this is high-quality funding, high-quality paper. |
| And it's a meta analysis. We're going to see if the claims in that meta analysis hold up. |
| The bottom one was really funny. Patterns of red and processed meat consumption and risk for basically lung cancer and heart attacks. |
| There's been a lot of literature in the nutrition literature, saying that you really shouldn't be eating red meat. We're going to see if that...if that makes sense. |
| Let's go back and look at the puzzle parts and then see how they fit together. |
| We know from the open literature that most claims fail to replicate. This is called a crisis in science. P-hacking is a problem. P-hacking is running lots of test trying this and trying that and then, when you find a P-value less than .05, |
| you can write a paper. |
| We're going to use meta analysis and P-value plots to catch the people that are basically P-hacking, and I call P-hacking sort of cheating with statistics. |
| Others are, you know, little...described it a little differently. We're going to be using JMP add-ins. These JMP add-ins were written by Paul Fogle. |
| And they allow us to start with a meta analysis paper and quickly and easily produce a P-value plot. We're also going to describe Fisher's combining of P-values and then we're... |
| have a small script that will take a the two-way plot that comes out of JMP and then clean it up, so that the left and right margins are more...more attractive. |
| Well here's long-term exposure to air pollution and |
| causing asthma, so they say. On the left, we have what's called a tree diagram. That's the... |
| it looks sort of like a Christmas tree. |
| The mean values are given as the dots and then the confidence limits are the whiskers going out on either side. On the left are the names of the paper...papers that were selected by these authors and on the right are the |
| risk ratios and the confidence limits. Now you can often just scrape the risk ratios and confidence limits off and drop them into a |
| JMP data set, and so we see the JMP data set on the right. |
| Now the P-value plot. |
| thing written by Paul Fogle is going to convert those risk ratios and P... and confidence limits into P-values. |
| Here we see that it's been done. |
| The confidence limits can give you the standard error. |
| With the risk ratio and the standard error, you can compute a Z statistic. And then you can take the Z statistic and and get a P-value. On the right here, we see a P-value plot coming out, just as it does with with a couple of clicks in JMP. |
| Here again, on the left, we have the rough P-value plot, and on the right, after using a small script, we add the zero line, we add the dotted line for .05, and we clean up and expand the the numbers on the |
| X and Y axes. And so we can look now and judge all the studies that that were in that thing, but we see a rather strange thing. We see that the |
| all of this here, there are a few P-values under .05 and then a lot of P-values going up, and so we have an ambiguous situation. |
| Some of the P-values look like the the claim is correct, and others look like they're simply random findings. |
| Let's take a look at Fisher's combining of P-values. On the left, we have the formula. You take the P-value, take natural log, that sum them up times -2. |
| And that gives you a Chi squared. You look up the Chi squared in the table. So this...this meta analysis, we have the P-values. Under P-value, we have -2ln, and you see that there are few P-values that are small that add substantially to the summation. |
| If you think just a little bit, the summation is like all sums, it's subject to outliers manipulated...shifting the balance considerably. |
| And so the few small P-values (.0033, .0073) are adding dramatically to the summation of the Chi squared. In fact the summation is not robust. One outlier...an extreme outlier can tip the whole...the whole equation. |
| Now keep in mind that P-hacking can lead to small P-values and also, |
| scientists quite often, if if a study comes out not significant, they simply won't even publish it. |
| And if they find something significant, they typically will. So there's publication bias. The the whole literature is littered with small P-values |
| and this probably...that's the tip of the iceberg. Under the iceberg are all the publications that could have happened, but they were not statistically significant. |
| We're now going to look at a air pollution study. Mustafic in 2012 |
| published a paper in JAMA and he looked at the six typical air pollution |
| carbon monoxide, nitrous oxide, small particles (that's PM 2.5), sulfur dioxide, etc. |
| If you look at these P-value plots, all of them essentially look like hockey sticks. There are a number of P-values less than .05. |
| But then there are a substantial number of P-values that go up along a 45-degree line, indicating that there is no effect. |
| We're contending that we're looking at a mixture. We're looking at a mixture of significant studies and non significant studies. |
| And we're further contending that the significant studies largely come from P-hacking. There are other ways that can arise, but we think P-hacking is the thing. So if you look on the left-hand side here, |
| in the red box, we give the median number |
| of models or tests that could be conducted in a in a study, the studies that were in Mustafic's paper. |
| We counted the number of |
| outcomes, predictors, and covariates, and if you multiply that all out, the median number, which we call the search space, the median search space is 12,288. |
| That means that the authors, on the average, had 12...in the median, had 12,000 opportunities to get a P-value of less than .05. They had substantial leeway to get a statistically significant result. |
| We're going to look at the simple counting here for the nutrition studies. So we have here 15 studies. The authors, or the first authors on the paper. |
| And then across the top you see outcomes, predictors, covariates, tests, models, and search space. The outcomes, if they looked at three health outcomes, for example, Dixon, and if they had 51 |
| foods in their study, that will...those would be predictors. Now if they use covariates, |
| that would add substantially to the search space. And you can see for Dixon, theoretically, he had 20 million possible analyses that he could have done. |
| And if you look down the search space column, you can see substantial numbers of possible analyses that could be done. |
| The nutrition studies were done with cohorts. A cohort is a group of people that is collected up, |
| they're measured in questions, initially, then they wait over time and then they look at health effects. |
| Each of these cohorts has the name of a data set that the cohort uses. |
| And the cohorts, once they are assembled, can be used to ask, you know, a zillion questions. |
| If |
| the P-value for one of those questions comes out significant |
| and they feel like they can write a paper, quite often they do. |
| So, in the last two columns, we see the numbers of papers |
| that could arise. That are more...we used Google Scholar and we actually checked it out. So these are the number of papers that have appeared in the literature |
| associated with each of these cohorts. I'll add that we've looked at a lot of these papers and in none of these papers is there any adjustment for multiple testing or multiple modeling. |
| Nutritional epidemiology, environmental epidemiology, those are the ones we talked about here. Nutritional epidemiology uses a questionnaire, food frequency questionnaire (FFQ). |
| In an FFQ, you can have some number of foods. |
| Initially, they started off with 61 foods and I've seen some FFQ studies where there were 800 foods. I wouldn't want to be the one to fill out that questionnaire. |
| But given here are the number of papers that use FFQs over time. |
| Since 1985...the technique was invented in 1986...since 1985, there have been 74,000 FFQ papers, and based on our looking at them, none of these papers adjusted for multiple testing and all of them had substantially large statistical search spaces. |
| Environmental epidemiology, I simply did a Google search on the word "air pollution" in the title of the paper. |
| And here we see that over time, there've been 28-29,000 papers written about air pollution. |
| So far as I know, based on a lot of looking, none of these papers adjust for multiple testing or multiple modeling, and all of these papers have large...essentially all of these papers have very large search spaces. |
| Meta analysis goes into the name systematic review and meta analysis and, recently, starting in 2005, |
| journals have |
| used that term in the title of the paper. So starting in 2005, I asked if the words "systematic review" and "meta analysis" appeared in the title of the paper, and you can see that started off low, 1,500 papers a year. |
| 1,500 papers in that five-year period and finally in 2021, there |
| have been a total of 27,000 papers, so this is a cottage industry. These papers can be turned out relatively easily. |
| A team, often in China of 10 or five to 15 people, can turn out one meta analysis per week. |
| And their their pay is rated on how many papers they publish and so forth and so on. |
| So far as I know, these are all...half of these studies are observational studies and half are...come from randomized clinical trials. |
| Virtually none of these and in all the ones that we've looked at, so far, particularly in observational studies, have this hockey stick look, like there's some P-values that are small and there are a bunch of P-values that look completely random. |
| I will say that all the...essentially all of these studies are funded by your tax dollars or somebody's tax dollars. They're very lavishly funded by the public purse, is one way to say it. |
| Many claims have no statistical support. The base papers do not correct for multiple testing and multiple modeling. The base papers have large analysis search spaces. |
| And we've seen examples from environmental epidemiology and nutritional epidemiology that most people, I would say based on the evidence, are unreliable. |
| I say and others have said too, we have a science and statistical disaster and use of meta analysis and P-value plots, these claims can be either verified true or not. |
| Here we have four situations of smog. Upper left is London 1952; |
| the right is Los Angeles 1948. |
| The lower left is Singapore and I'm not sure...I don't remember the exact year for that. And then we have Beijing. Those are recent. |
| In the case of the London fog, statistical analysis of death...daily deaths and everything indicate that there upwards of 4,000 deaths that occurred in a three- or four-day period in London in 1952. That instigated the interest by epidemiologist in what was the killer. |
| The other three slides or other three pictures, there has been non reported increase in death during those time periods. |
| The ringer is that in London, they were burning coal |
| for heating and everything else. There was a temperature inversion and the contention is that acid in the air was carried by particles into the lower lungs and susceptible individuals, usually the old and weak, died in increased numbers, but that is not happening now |
| around the world. There is all kinds of pollution around the world and we don't see spikes in death rates associated with it. |
| Scams? Ha ha. I love scams. |
| Are these scams? Air pollution kills. Any claim from an FFQ study, for example, if you drink coffee, you're more likely to have pancreatic cancer. That's one of the claims that's probably not true. |
| Natural levels of ozone, do they kill? I think that's not true either. Environmental estrogens. Any claim coming from meta analysis using observational studies. |
| Keep in mind, the evidence in the public literature is that 80% of all, usually coming from university, science claims failed to replicate when tested rigorously, for example in randomized clinical trials. |
| Emotion and scare. The whole aim of practical politics is to keep the population menaced, scared and hence clamorous to be led to safety by menacing the population with an endless series of hobgoblins, all of them imaginary. |
| Flim flam, deception, confidence games involving...involving skillful persuasion or clever manipulation of the victim. So H.L. Mencken in the 1930s said that practical politics is largely scare politics. |
| We finish up with the the authors of this in this of this talk. I'm Stan Young and I can be reached at genetree@bellsouth.net. Warren Kindzierski is a Canadian epidemiologist and he's been working with me very closely for the last couple of years on papers that |
| are based on the things that we've seen here today. Paul Fogel is a very interesting statistician, lives in Paris and he was responsible for the writing of the |
| add-ins and scripts that we can use. I will say that the scripts will allow you to go from a meta analysis to an evaluation in probably a little bit less than an hour. |
| I'd like to recommend the National Association of Scholars report, Shifting Sands Report 1, this URL will get you there. |
| This report is a long and involved involved, but simple simple statistics report talking about air quality and health effects. |
| And with that I'll also make the following claim, if someone watching this wants to try out what's going on, we will give you the scripts and add-ins to JMP and then I will even help you |
| look at and interpret your particular analysis of a meta analysis. So with that I'll stop and I'm prepared to answer any questions. Thank you. |
