Speaker | Transcript |
| hey. |
| Jordan Hiller | Can you hear me. |
| Russ Wolfinger | hey Jordan, can you hear me. |
| Jordan Hiller | I can hey how are you. |
| Russ Wolfinger | Great oh shucks let me get my background is it working I'm done in my basement. |
| Jordan Hiller | All right, all right Take your time we got plenty of time where. |
| we're a little early and yeah. |
| Everything going. |
| Russ Wolfinger | really good um yeah I've been meaning to kind of want to catch up with you because got some got some good action going now on the genomics front. |
| Jordan Hiller | right on right on yeah I've heard a little bit I saw that some of the genetic stuff showing up in the EA. |
| First yeah. |
| Russ Wolfinger | Because. |
| it's really actually not ready yet, but johnson's been working on and Luciana Silva. |
| And they were joking is like Jordan got his hands on that. |
| Jordan Hiller | Well yeah you know my. |
| eyebrows went up when I when I say oh that's very interesting. |
| Russ Wolfinger | No yeah we're we're really excited about the new directions and just trying to take it a step at a time. |
| Jordan Hiller | right on. |
| Russ Wolfinger | And we're actually we've got a new, you know Jeff man moved RD to take over for kelsey. |
| Now we're now interviewing to backfill jeff's position. |
| In product management so that'll be the next piece of the puzzle to click in we get someone in there to help because. |
| we've got some questions around. |
| Marketing branding I mean they like we like we mentioned, I mean it's not even clear what the markets like these days and. |
| We got it we want to be careful and I did we really I really, really value I didn't read through all the stuff you wrote on our survey appreciate your insightful comments there. |
| Jordan Hiller | Well, you know I'm out of the I'm out of the market for for a while now, so you know take anything I say with a grain of salt. |
| Russ Wolfinger | Just kind of, but we want to get kind of going everybody's got a nice little insights from their their corner of the woods and going to put together a good game plan if we for listening to everybody. |
| Jordan Hiller | right on yeah. |
| Russ Wolfinger | You don't even know how do I. |
| How do I turn it background on now I don't see. |
| Jordan Hiller | yeah I can, I can tell you where that is so so by the camera icon where it says, probably says stop video right. |
| Russ Wolfinger | yeah. |
| Jordan Hiller | there's a little carrot that. |
| Russ Wolfinger | Okay, I got it virtual background, here we go. |
| Oh don't load. |
| load smart virtual hang on it's got me I don't know what happened cuz I usually got a standard background that comes up motorbike updated or something. |
| There we go. |
| yep. |
| Jordan Hiller | Nice. |
| Russ Wolfinger | yeah that's just I think it's building might be building a actually just took down the other end of it. |
| Jordan Hiller | yeah very nice. |
| Alright, so I have when you're ready, I have a little checklist for us to run through just to make sure everything's ready to go. |
| Russ Wolfinger | yeah How does my video and audio angles and lighting and everything look okay. |
| Jordan Hiller | It all looks good that looks fine. |
| let's see so we're in the right place at the right time, recording the right thing. |
| You know this is being recorded I don't have to worry about your verbal permission that's fine. |
| yeah your microphone sounds good, are you going to have any background noise, where you are seems pretty quiet right. |
| Russ Wolfinger | I should be good unless my. |
| it's actually kind of yucky day, so I don't think my neighbor will fire up a lawn mower. |
| Jordan Hiller | Okay let's see so cell phone and computer notification silenced, do you have outlook running or teams running or anything like that. |
| Russ Wolfinger | Let me I'm closing all that down now. |
| I'm about to have about 5 million. |
| browser tabs open I know right and that you have the cpu little bit. |
| yeah so I've got open as my PowerPoint in zoom right now. |
| Jordan Hiller | Okay that's great uh could you just confirm your display resolution to me for me. |
| Russ Wolfinger | Oh Okay, you want me to look on display settings. |
| Jordan Hiller | Yes, display settings should show that, where is it. |
| where's that gonna show show and display settings. |
| yeah. |
| Russ Wolfinger | it's on my laptop 1920 by 10 at. |
| Jordan Hiller | Perfect that's what should be all right, the other advice is, if you can is to auto hide the windows taskbar if you don't do that ordinarily. |
| Russ Wolfinger | Okay. |
| Jordan Hiller | hey why don't you start sharing your screen right now. |
| So that we can. |
| Russ Wolfinger | Okay yeah yeah that's good make sure share screen. |
| There we go screen. |
| I'm clicking it oh share okay there we go. |
| And then. |
| yeah. |
| This will be hopefully will be I've enjoyed putting together these five can have some fun Star Wars connections. |
| Jordan Hiller | that's great. |
| Okay uh let's see yeah. |
| Under taskbar settings I think you could have it a it's at the bottom there. |
| said it. |
| automatically yeah if you're I don't know how well that's what they want you to do, but I find that confusing when I do that because, because you know, then you have to go to the bottom of the screen to pull it up, but if you can get used to that, and if you can deal with it that's fine. |
| Russ Wolfinger | yeah I'm comfortable with that. |
| Jordan Hiller | Okay, great great. |
| uh what else. |
| Automatic. |
| Russ Wolfinger | Let me just try to MAC So when I go to present. |
| Jordan Hiller | yeah maximize. |
| slideshow. |
| Russ Wolfinger | I don't see that looks good. |
| I can just use my keyboard does, that is, that coming across okay. |
| Jordan Hiller | It is it's great. |
| Russ Wolfinger | And then, will I be there'll be a little instead of me personally somewhere on the screen. |
| Jordan Hiller | that's that's right, you know it records separately and they'll take care of that in post production, so you don't worry about that at all. |
| Russ Wolfinger | Okay let's go oh I don't know I didn't know if I should I do what I actually like it wasn't the talks I actually like to see the person. |
| Initially, just to get a feel for you know the body language and tone, but that, but if I'm actually will be showing the whole time that should be that's the optimal I think. |
| Jordan Hiller | it's up to you russ if you want to start by start with your screen already sharing, you can do that, and if you want to just say a few introductory remarks before you start sharing that that works too. |
| Russ Wolfinger | Oh okay I'm gonna but I mean can we, we could actually oh oh was recording already owned by the way. |
| Jordan Hiller | It is it a well yeah I guess it started that automatically but that's okay it's just going to add, you know, whenever. |
| In post production they'll start it at the at the at the right time don't worry about that. |
| Russ Wolfinger | Okay, and then, are you will you will you be involved at all, like in terms of saying it or anything at all. |
| Jordan Hiller | know when you're ready to go, I am going to turn off my camera and turn off my microphone and then we'll just say ready go. |
| Okay. |
| Russ Wolfinger | That sounds good, and then I've got I've got 20 slides so remind me what's is there a time limit. |
| Jordan Hiller | I don't think there really is you signed up for a 30 minute paper. |
| You know it's it tell virtual now, so the time is fungible. |
| Russ Wolfinger | But that'd be fine yeah I don't think I don't it's not like it's going to go for an hour anything to kind of just hit highlights. |
| yeah I think I think 30 minutes sounds about right. |
| Okay. |
| Jordan Hiller | All right, so what do you think you need to do anything else, before we get going, are you ready to go. |
| Russ Wolfinger | grab a slow drink. |
| yeah Oh, and then this will air, this will air, I think, on the fifth they were saying October five. |
| Jordan Hiller | Is way is when they go live. |
| Russ Wolfinger | yeah and I think I got I got some email about I might be maybe I'm one of the people that's going to be around live right after or something like that. |
| Jordan Hiller | very nice yeah good good. |
| Russ Wolfinger | So, should I like when I'm recording should I like allude to that or anything, or what what's the Convention. |
| Jordan Hiller | I'd say don't because this is going to live on as a resource later for people, even after the Conference for people to watch, so I take just record it straight and. |
| Russ Wolfinger | kind of time time invariant. |
| Jordan Hiller | yeah exactly. |
| exactly right. |
| Jordan Hiller | Okay. |
| So if you're ready okay great so I'm going to turn off camera and microphone and and I'll be watching here and yeah go ahead and start whenever you're ready. |
| Russ Wolfinger | Which when I finished what should I do. |
| Jordan Hiller | You could say. |
| that's the end Thank you good night. |
| And then I'll come back on and I'll. |
| Just wrap it up. |
| Russ Wolfinger | Okay, the editors will crop beginning an end as needed. |
| Absolutely okay. |
| Jordan Hiller | Okay, here I go I'm going away wrestling good luck I'm looking forward to watching this. |
| Russ Wolfinger | Thank you. |
| Okay. |
| Russ Wolfinger | p-values. |
| And as you may have heard, there's been some controversies around this this most infamous of statistics over the last several years and even recently here in 2021. |
| And I want to summarize some of the things that I've been seeing and thinking about them, and also for fun, |
| Return of the P-Values. |
| We've had periods, where they've kind of come into favor and I feel now there might be a bit of a resurgence, so let's see how that plays out. |
| To get started let's go back 100 years, almost...almost exactly 100 years from now, back in the early 1920s. |
| This was the heyday of arguably the most famous statistician of all time, R.A. Fisher, and he's the one credited with p-values, as well as many |
| very powerful and popular, to this day, concepts in statistics, for example, maximum likelihood and a frequentist style interpretation of probability. |
| Now one of Fisher's key peers of his day and even at the same institution was Jerzy Neyman and Egon Pearson. |
| And right at the very beginning, when Fisher was coming out with his p-values, they had some some fairly severe disagreements about their utility and how they should be applied in practice. |
| Neyman and Pearson, as you know, are the ones that came up with the famous hypothesis testing framework. It was more decision theoretic. |
| And they're also the ones credited with confidence intervals, data generating mechanisms, and other common things. And the statistics that we employ nowadays, it's kind of this weird hybrid of these two initial methods, but they were born in controversy, right from the beginning. |
| Now, as the years went on, |
| the debates continued and they tended to fall along the lines of the classic frequentist versus Bayesian |
| disagreements. And this often...this has many aspects to it, one of the key ones being just how do you interpret probability itself? |
| The two kind of popular views that distinguish the two camps often come down to do you view probability as kind of existing in the objective world and can be measured, say with real world repetitions? |
| Or is it more in our heads in terms of a degree of belief? And these can have very profound implications in terms of |
| statistical methods that you take, and even how you do probabilistic based modeling. |
| And, in particular, the Bayesians tend to usually...to get a Bayesian framework stood up, you need to specify prior distributions |
| on all parameters, which again would be interpreted as your often subjective degree of belief about the way this parameter...the values of parameter's likely to take on. |
| And they can be univariate, multivariate, etc., depending on the complexity of the model. You crank...you turn the crank through Bayes theorem |
| and you obtain posterior probabilities and densities upon which you make your inferences. Whereas frequentists tend to employ maybe in a sense, a little more... |
| little more distant methods or more kind of objective in terms of this direct calculations, and here's where you get the p-value. It's certainly a frequentist style statistic. |
| And it's been it's been criticized and looked over just multiple times as time has gone on, and we're going to look at some of the aspects of p-values in more depth as we proceed here. But they...again just kind of setting them in context historically. |
| And then more recently they...since since around the year 2000, there's been a big hoopla around the topic of reproducibility. |
| And this has resulted...there's been hundreds of papers and topics around scientific conclusions that have not been reproducible. |
| One lab, say, produces some results, makes a claim, and another lab tries to do the similar experiment and just can't...just gets something different. And this often |
| can create big controversies, because they can be...some of these conclusions are quite |
| important and can result in policy decisions or health around...or health or even decisions in any field that you might think of. And so many folks will claim we're kind of in this state of a reproducibility crisis. |
| Now to connect this to our Star Wars theme, let's imagine, as you know, across the galaxy we often might have |
| some kind of...there's always conflict brewing and rebellious groups popping up. And with this reproducibility crisis, if you can imagine the Star Wars, this evil council convening, |
| concluding that they must find a scapegoat for this reproducibility. And what...who could we blame for it? What could we...what what narrative could we spin? |
| And why not...why not blame it on bad statistical methods? In fact, the p-value looks like a nice vulnerable candidate. And we see there that Emperor Palpatine is pleased with this plan and is willing to devote a lot of resources to pulling it off. |
| Now amidst this, in the early , kind of mid to late 2000s, we convened a society kind of in the outer realms convened around the topic of genomic reproducibility. |
| We were intrigued by this problem where we saw, indeed, a lack of reproducibility in lists of differentially expressed genes. |
| And you can imagine a venn diagram where one lab would give their top 100 most important genes from an experiment. Another lab goes to reproduce it, |
| and that venn diagram hardly overlaps at all. And this is a problem we were tackling, kind of a direct scientific reproducibility questions. |
| And we weny through a lot of hard work and research and rerunning a lot of controls in many labs. We published this paper here, where we |
| came up with what I would consider a fairly simple but easy resolution to some of the reproducibility questions, |
| specifically as they relate to p-values and other related statistics, say sensitivity specificity in a binary decision context. |
| And the conclusion was that p-values themselves are not designed to be reproducible. They...their strength and even intent as a statistic is to control error rates. |
| In particular, we have the classic type one error rate of a false positive, and this is these the realm that |
| p-values and their theory is developed around, whereas the reproducibility question is a different one. |
| And in fact, our mindset is almost one of viewing reproducibility as the third dimension beyond specificity and sensitivity, the classic type one, type two error rates. |
| And in the context of genomics, we can help resolve the question by making plots, like you see here, called a volcano plot, |
| where each point is a gene that we've measured and we're comparing two conditions. Along the X axis, we have a measure of effect size or log, in this case log2 fold change, and on the y axis, we plot negative log10 p-value. |
| And the plot tends to always have this characteristic V shape due to the relationships between those two statistics, where the genes that...where the most action are the ones in the upper left and right, kind of like little spewing rocks out of a volcano. |
| But it turns out, in terms of reproducibility, |
| that the key...the dimension is actually the X axis and it turns out that effect size is more reproducible |
| than the y axis, the p value, which is kind of a standardized measure of signal to noise, if you want to view it that way. |
| And so the way around the question is you don't you...actually both are valuable and so the key is to look for genes in the upper left and right corners of this plot, where you can satisfy both criteria. |
| And the plot is actually an intuitive way to do that, and there are there are variations on how you might draw the lines, say either rectangular or curved as you see here. |
| But those are more, kind of, details beyond once you...once you get the main concept, and this was the main message of that paper, which I think, still is a solid one, to this day, but it hasn't really |
| had much uptake across...across the board. In fact, reproducibility still a big crisis and folks think we're in big trouble and things need to change. |
| Now, |
| kind of, as a result of the of the p-value taking the heat, as it were, for reproducibility, the American Statistical Association, which is arguably one of the leading |
| statistical societies in the world, did issue a statement in 2016 around p-values. They were, kind of, feeling the heat and thought it was a good idea to put some messaging out there, helping out to |
| allay some of the concerns and maybe talk about a few of the things I mentioned. However, the six statements, and we won't go in into them for detail for sake of time, |
| while they did present some interesting points, I think they kind of missed the mark on some things. |
| And now, these statements themselves took some heat, and so the the battle lines were drawn even even even more distinctly, and it's questionable whether this statement helped or hurt |
| in the whole debate, but at least it was...it was a progress forward and there are a lot of good things in these statements. |
| And I think some of the criticisms were maybe more nitpicky than needed to be, but at least...at least, |
| it's trying to address things. And these are...I do view these as valuable...as a valuable contribution in terms of understanding the controversies and making sure that we can...we can kind of make good decisions about how and when to use p-values, if at all. |
| Now the debates, I would say, kind of came to a head in 2019 in this issue of The American Statistician. And I, kind of, just amusingly, if you will, call this ANTIPA, |
| kind of an anti p-value riot, were nearly every paper in this journal...in this issue was kind of pejorative or bashing of p-values. |
| And the entire issue was led by Ron Wasserstein and is the head of...the executive director of the American Statistical Association. |
| And so, with that position of prominence, it's kind of brought a lot of the...lot of the disagreements, right to the front. And |
| the thing about this issue, which I do recommend, although don't expect going to this issue to resolve the questions for you, in fact you're likely to come away |
| with more with more questions and scratching your head than you did going in, because the papers in there are all over the place. |
| There's no kind of clear message in terms of what to do. The only clear message is that p-values are bad and then what should we do in response to that. |
| And so I would refer you to this issue to get more details, but you can see by some of the titles listed here and get a feel for some of the things that are being proposed, |
| including a fairly radical proposal to completely abandon the phrase and usage of statistical significance. |
| And I would include, as you know, in JMP software, sometimes we'll put...we'll make bars of log worth, negative log p-values, or put stars on output. |
| But this proposal would say that that's bad, shouldn't be doing that anymore, and it leads to bad practice. And so this is an actually an interesting issue for us |
| in the JMP division as we've been, in both JMP and I would say, before the year 2000, SAS has perhaps |
| produced more p-values than any other software company in the world, given its history. And so it's a natural topic for us to research, and if we have been wrong it would be good good to fess up for that and move forward, but if not...it's it's...I think it behooves us to present some reason, |
| some debates and actual rationale for why it might be a good...a good reason to continue to use the p-values. |
| So there's a kind of a burgeoning literature and this this really got things flaming. And there were quite a quite a few responses to this, some a blog posts, etc. |
| But again, if you want to kind of dive right into the big...the big thing, this is the issue to start with. And there have been a few follow ups, so many things since then. |
| Now even within this issue, though, it wasn't all bad. And I would point you to this article by Sander Greenland, who, I feel, has been a voice of reason |
| throughout the controversy. And he's made also several blog posts here and there, across the web, which you can search and find. And he proposed |
| staying on the log scale, which which we like, remember we're using on the volcano plots, and there's a way to view the p, we take the log2 |
| p- value, there's direct connections to bits of information and the classic Shannon information entropy criterion. And, of course, if you take the log, it's very easy to switch bases, you can go log base 10 just by multiplying or dividing. And so |
| I, personally, I tend to agree with this assessment here that there actually is some value in this...just looking at log p-values. |
| And also, I would refer you to the two papers listed at the bottom here in the green box. Benjamini gave a nice formal Rietz Lecture at the JSM in 2019, and then from the pharmaceutical industry, Eric Gibson's paper is also a really a great read with a lot of nice insight. |
| So let's take a look here. Are p-values really the problem? And you can start...star...if we can back away a little bit, |
| the thing is that they are used so frequently, we might could make the analogy of |
| driving an automobile. And you can imagine, someone putting up horrible pictures and accidents and just saying we need to...we need to stop driving all together; it's just too dangerous. |
| People can't handle their cars. You will...you'll be kind of be ignoring the fact that millions of folks drive safely every day and are able to make very good use for their car, for their families, and work needs. And so |
| some of these criticisms, I think, tend to be one-sided or biased and so, as with any controversy, it helps to kind of try to look for both sides and seek resolution. Or if something that is claimed really isn't resonating true, kind of put it in its place, as it were. |
| There's also this issue of there's no clear alternatives that have emerged to replace the the role of the p-value. No question that they do get misused and they are kind of somewhat of a tricky statistic to interpret at times, |
| but the question is, if we're not going to use them at all, let's say we made a decision in JMP to abandon them all together, what would we put in their place? |
| And there's been a lot of proposals, of course, confidence intervals, Bayesian measures posterior probabilities, etc. |
| But a lot of them fall to the same criticisms that p-values themselves have. And so the question, if we loop back to the issue of reproducibility, |
| you know, what...what is at the...what is at the heart of when a study does not reproduce well, what happened? And there's a lot of things we can consider if we back away from...from the evil...the evil narrative, as it were. |
| Thinking about things like the experimental design itself, and we know there's a lot of...there could be a lot of trouble there, and this is an area that |
| we take it really seriously at JMP in helping you design really solid experiments, avoid confounding and other factors that can often bias or throw...throw an experiment out of whack, as it were. |
| As well, there can be mistakes all along the board, in terms of data collection, cleaning. |
| And p-values indeed are vulnerable to hacking. You generate a lot of them, like we do, say in the field of genomics. |
| If you just want to cherry pick the most important ones, you're going to fall...fall victim to the classic problems of multiple testing, etc. Not to mention, our our publication framework, as solid...as strong as it is and helps with science, there are biases in there. |
| And also, of course, everyone likes...likes kind of a good fight. |
| And so to gain some perspective on this, I think it is important to realize that just looking back through the history of science, there's always been debates about topics. |
| When you get sharp people, really smart people in a room that are passionate about what they're pursuing |
| and if they disagree with another, one sparks are likely to fly. And that's kind of...kind of the rough and tumble of it. |
| And it's also good too, you know, just to make sure everyone listening, that you that you really do understand the definition of a p-value, |
| which is kind of in a way, I like to think, it's kind of Yoda-like; it's reverse logic. So if you have the idea is to first assume the null hypothesis is true, |
| and then ask the question, what's the probability of observing something more extreme than what we have observed, or that so-called tail area, |
| assuming that the null and the model, of course, the model itself is also true? So it's kind of a reverse, backwards logic like like Yoda grammar. |
| And so it often is not maybe not addressing the exact question that you want, which Bayesians will say, what you really want is like the probability...direct probability of a hypothesis being true, |
| which a p-value is not, so it's good to know that. If you do want that probability than a Bayesian approach is likely a good...a good way to go. And again, p-values are really there for error control not reproducibility. |
| And in a little connection I wanted to make, and I'd say the Bayesian versus frequentist debates maybe have reached what our good friend Steve Uber has labeled a detente, |
| and that there are connections that are often underappreciated, for example, this one, all the way back from 1987. |
| If you view...if we take a view of one-sided testing, where we're just looking at one side with distribution, it turns out in this paper that they presented at JASA(?), |
| that the the one-sided...one side of p-value can actually be directly related to a Baysean posterior probability (and this is illustrated by the red and blue |
| distributions that you see here) those two tail areas that are highlighted are exactly the same, and one's the p-value |
| and the other is a Bayesian posterior, both one-sided. And so we can get situations like this where you're going to get the same answer...you're going to get the same answer |
| whether you're taking a Bayesian or frequentist standpoint. I think these kind of resolutions are in fact quite nice and good to keep in mind. |
| And to me they...they are a reason actually to to use one-sided testing when it...when you can. I also want to give a plug for our colleague, |
| Brian Fricke, who's been working on a new add-in called confidence curves. |
| And I believe he's, if he hasn't already, he'll be putting that up on the JMP Community soon. So check that out, as he he actually |
| looks at p-values in connection with model screening results and is a nice way of comparing those results in a k-fold cross validation type framework. So please check that out when you get a chance. |
| Now, also again trying to gain perspective here. |
| I think another comparison or viewpoint to take is this nice distinction between predictive modeling prediction and explaining. |
| And this...this is a...this is more of a philosophical distinction, but relating it to statistical methods, I think makes a lot of sense. |
| And there's a nice paper by Galit Shmueli, back 10 years ago, 11 years now, in Statistical Sciences that outlines a lot of the key issues here. |
| But I raised them because we have...you know, machine learning data science is exploding these days. And most machine learners just like to live on the left side of this diagram. |
| And if you're in a prediction kind of mindset, things tend to simplify a little bit, |
| where the goal is just to create good predictions that are better, and a lot of the reproducibility questions start to go away. |
| Because we know if we do a good job with predicting, we can reasonably predict the future or new observations without overfitting. |
| And a lot of the questions just kind of don't become relevant, whereas I think if you want to shoot higher and really try to understand a scientific system, |
| that's where things can get more thorny. And I feel like the p-value tends to have more relevance on the right side of this, where you're trying to explain things. |
| You want to know if a statistical method is is producing errors and, if so, what what kind of errors, how unusual is the statistic that I'm looking at, |
| etc. And so it's important to keep these things in mind and apply them to this scenario that you're you're studying. |
| You may be in a situation where all you really do care about it is making good predictions, and if so, |
| take the p-values, don't...maybe have you don't have much relevance. But if you're in a classic design experiments setting that we know and love very well in JMP, |
| then all these more difficult questions come to play, but if you can address them, the payoff is also a much greater. If you can understand the system and explain it, you can certainly predict too, so I view the explanation, a goal with a higher one in terms of science. |
| And of course we're now coming...coming full circle with ANOVA, which also was something that Fisher popularized hundred years ago. We have tools with ANOVA |
| where we can really go after a scientific problem, break it apart and see what's going on, and achieve that explanation goal. |
| And I do feel p-values can be a key part of that. If you think of the classic ANOVA table, the final output of that is typically a p-value from an F test |
| indicating whether a certain effect is statistically significant over and above the observed noise in the experiment. |
| I think taking that...computing that statistic in its its tail area, it's often a very useful thing to do. And you can see some JMP output there on the left side, the p-values indicated out, |
| again, on a log scale. I think that kind of plot is quite valuable, very quick and intuitive visual summary of what's going on in an experiment. |
| Also, I want to refer you to this book by Deborah Mayo, as well as her blog series which she's very active. It's a great resource. I enjoy reading it. And |
| this actually...she's more of a statistical philosopher, although she knows a lot of...a lot of the key issues, but it is a great read in terms of |
| really diving into some of the deep issues and really further laying out why statistics like p-values actually do have good bearing within. She's more in the pro p-value camp and |
| her big thing is that what she calls severe testing, where you try and take a method and put it...really put it through the wringer, look for errors. |
| And this is just good...I think it's a good, solid science where you're always, if possible, taking the opposing view, |
| poking holes in your own, always looking for flaws, and kind of taking a humble view that something may not always be what it appears at first, when you really start peeling back the layers. And so it's a recommended book for sure. Check it out. |
| And then to kind of wrap up...kind of more takeaway advice from this talk, |
| for sure, I think some of the criticisms of p-values are people that just blindly look to see if they're below .05 or not, and make make a quick decision based on that. |
| Keep in mind that .05 is often not the best criteria for you. That's...that's another rule of thumb that dates all the way back to Fisher. |
| So it's kind of 100 years old and may not make sense for your setting. I know in genomics and say, genomewide association studies, we often want to go much, much more strict |
| to handle the multiple testing problem. But it is there and it's just, again if you know the definition of p-value, you can make wise decisions in terms of where to do these cut offs. |
| And again, I wanted to reiterate about taking one-sided testing into consideration when you can, as well as this nice dichotomy or kind of Hegelian dialectic between explain and predict mindsets. |
| Now here's a here's a little meme that I put together, just to poke a little fun at some folks. But I do feel like sometimes |
| the community actually likes to keep a...keep a controversy brewing and going in order to maybe drive their...their publications or or just just to keep the narrative going. |
| And so, p-values do remain vulnerable for this kind of thing, and probably will be in the future, but in the end, I do think they're actually quite valuable if and when you use them appropriately. |
| Now I wanted to mention one final thing. It turns out, in response to that 2019 ASA journal article that I mentioned, |
| the president of the ASA at that time Karen Kafadar, convened a task force to kind of put together what she was hoping it would be kind of a definitive reply to a lot of the key questions. |
| And she assembled what I consider a star-studded panel of top-tier statisticians and researchers. You know some names there, including our own beloved Dick De Veauz, a big JMP proponent, Brad Efron and others. |
| And they they they convened this task force, they got together and wrote a really nice tight response. |
| But it turned out the publication of it has been blocked, was blocked for almost two years. And I was following this and I actually was corresponding with Dick and Karen |
| about about when it's going to come out, and it turned out that they ended up deciding to publish it in a couple of journals. |
| A few journals didn't want to take it, so they had to bounce around a little bit and it's finally been released, just just a month or two ago. |
| And I would highly encourage you to take a look at it. It's very short. It's actually much shorter than I thought it would be. |
| But to me it really answers a lot of the key questions. You can see the text of it in its entirety at Deborah Mayo's blog |
| post and I give the link there at the bottom, if you want to check that out, or you can just search online. I believe it's being published in the Annals of Applied Statistics. |
| And I do like...I think that's actually kind of...kind of a nice reply and maybe it'll, you know, kind of settle down some of the controversies that have been around p-values. |
| To me, it looks very solid, but I would encourage you to check it out and we'll have to see how things play out within Stat Wars with time to come. Until then, I do want to promise it looks like order will be restored in the universe. We'lll just have to see when it happens. Thank you very much. |
