Podcast

Episode: 565 |
Ivan Oransky:
Co-founder of Retraction Watch:
Episode
565

HOW TO THRIVE AS AN
INDEPENDENT PROFESSIONAL

Ivan Oransky

Co-founder of Retraction Watch

Show Notes

In this conversation with Will Bachman, Ivan Oransky, a co-founder of Retraction Watch, shares his experience as a medical journalist and with Retraction Watch. Ivan explains that his friend and co-founder, Adam Marcus had uncovered a massive story about scientific fraud in Western Massachusetts, where an anesthesiologist had made up all clinical data. Adam, who was managing editor of an publication called Anesthesiology News got the scoop on the story, and Ivan, who was impressed with the story, suggested they start a blog about retraction notices, it turned out there were far more happening than previously thought.  13 and a half years later, Retraction Watch is still going strong and has a large audience. Adam and Ivan are volunteers but have four staff two of whom run a database of retractions that was recently acquired by CrossRef, a nonprofit that tracks scientific data and papers. The other two staff continue to contribute to the journalism work they started 13 and a half years ago, while Ivan and Adam still supervisor edit and direct it.

 

How to Evaluate an Article

Ivan shares his advice on how to evaluate an article in a medical journal or any published article. He emphasizes the importance of showing one’s work and examining the evidence used to reach a conclusion. He explains that, when looking at articles, it is crucial to consider the original sources, citations, and the journal’s track record of quality. He also emphasizes the importance of humility in making claims and not making pronouncements about things he or she doesn’t know anything about. He also warns against trusting credentials to suggest expertise, as it can be misleading, Ivan shares the example of a time when he was asked to peer review papers about COVID-19, simply because he had co-authored a letter about retractions of work. However, he is not an expert on the subject. Ivan believes that an expert should only be asked to peer review papers that they believe are likely to hold up or should not be published. 

 

Leading Causes of Retraction

Ivan explains that factors that commonly lead to a retraction. Two-thirds of retractions are for misconduct. This number is consistent across various works and he goes on to explain that there are several definitions of misconduct to take into consideration, including fabrication, falsification, and plagiarism. About 20 percent of the time, it’s due to a what’s known as honest error, and Ivan offers a few examples. The deeper cause is the requirement that researchers must publish in certain places to get a job in academia, tenure, promotion, and prizes. This drives people to do all sorts of things, and while this drives most people to work harder and try to work more efficiently, others may take a different approach. In fact, Ivan states that 2 percent of researchers admit to committing misconduct. 

 

The Replication Crisis

Ivan talks about the replication crisis, which has been a topic of interest in the social sciences and hard sciences. When Retraction Watch was first launched, there were about 400 retractions from journals a year. Last year, there were more than 10,000, a big increase despite the rising number of papers published. The root cause of this issue is the same problem: replications are not new research or findings and should be cherished and prized, but they are not. Big journals don’t like to publish replications, so they don’t reward new research. To get into a big journal, researchers need to publish new research, which is simple behavioral incentive economics. The discussion turns to incentives for people to write about scientific misconduct and fraud. Ivan states that, while there is more incentive not to write retractions, he cites a page on Retraction Watch that has dozens of stories from people committed to revealing issues with research, including well-known figures. These individuals face legal risks, such as lawsuits, and are usually not paid for this work. The conversation also touches on the potential negative repercussions of challenging senior professionals in their field, such as professors or presidents of universities. However, most of these individuals do not work in science anymore, or their career trajectory is not dependent on pleasing or failing to displease senior members of academia. These individuals often publish on sites like PubPeer, which allows users to leave comments on published studies. This helps expose the issues and claims in the media, helping to raise awareness and support for those who need help. Retraction Watch offers resources and social media platforms for those interested in learning more about the topic. They welcome feedback and story tips, and they are open to sharing more information about their work.

 

Timestamps:

01:03 Scientific fraud and retractions 

04:41 Evaluating credibility in scientific articles and peer review

09:10 Research retractions and the root causes

13:05 Replication crisis in science and the challenges faced by those uncovering fraud

17:18 Academic misconduct and whistleblowers

 

Links:

Website: https://retractionwatch.com/

 

One weekly email with bonus materials and summaries of each new episode:

 

SPEAKERS

Ivan Oransky, Will Bachman

 

Will Bachman  00:03

Hello, and welcome to Unleashed. I’m your host will Bachman and I’m excited to be here today with Ivan Urbanski who we were on the Harvard Crimson the other many moons ago, something like three decades ago, and Ivan is the founder of retraction watch. And I’m interested to hear more about that. Ivan, tell me, first just give us an overview of retraction watch.

 

Ivan Oransky  00:27

Well, well, it’s great to be here and to reconnect after all these years. That’s always fun. And thanks so much for having me. And I wanted to say, as it’ll become clearer in the story that I will tell you in a moment, I’m very much the co founder of retraction was one of the two co founders, you’ll hear more about my co founder in a second. So I’m a medical journalist. That’s what I’ve done with my career. I went to medical school, but then decided to be journalism full time. And one of my, at the time, I would call him a friend, but not necessarily a close friend yet, but someone I knew professionally. His name was Adam, Marcus and Adam had broken a massive story about some scientific fraud. In Western Massachusetts, someone named Scott Lubin who was an anesthesiologist studying pain medication and sort of, you know, the pain killing capacity of particular medication. And it turned out he had made up all the clinical data. In other words, all the trial participants in this studies, which were pretty well thought of at the time, Adam, because he was managing editor of a publication called anesthesiology news. Got the Beat on this that big scoop on it did a terrific job. This is back in like 2008 2009, I was really impressed with this. We started talking about it a little bit, obviously, a lot. And we’ve had these conversations about the attraction to that scientific fraud. This is something I had thought about. But you know, I mean, wasn’t necessarily a big bright spot on my radar. And then, at some point, I said to him, I remember the conversation. I remember exactly when it happened, but it would have been in summer of 2010. I said to him, What if we started a blog about retractions? You know, there are all these attractions happening. There were far more happening, it turned out than we thought and that anyone thought and there still are far more happening than many people think. But they’re interesting stories, more journalists, and like, it’s hiding in plain sight, let’s go do that. And by the way, the retraction notices are really not good. There isn’t good for science, somebody’s talking about this stuff. And they’re just, you know, we’re not learning what we should process is not as self correcting, and science scientists and researchers would like us to think. And so we did it, and really thought well, that this would be and it was actually quoted a couple months later, in the times, actually, you know, saying something effective? Well, we thought to refute attractions a month, or moms would read it would be fun, you know, maybe we accomplished something maybe not. Well, that like many of our predictions turned out to be way off the mark, and the interest and the the traffic to the site, the engagement, the number of tips, we get the number of times we’re asked to speak, you know, conferences, or to the media is just continues to grow and grow and grow. It’s very gratifying, although kind of a lot of effort. And so, you know, here we are now, that was summer of 2010. So here we are now 13 and a half, give or take years later. And it’s still very much going strong. We have a staff of four, Adam and I are volunteers have been some majority of the time we’ve been at protection watch, we had day jobs, it’s a very happy with. And but we have four staff who two of whom run a database of attractions that we created over time, really, that one of those staff members traded over time, and was recently acquired by another nonprofit called CrossRef. It’s just kind of a really, really big player in the space of, you know, metadata and keeping track of scientific data and papers and things like that. And then two who are on the journalism side, continuing to do the work that you know, Adam and I started 13 and a half years ago and Adam and I still contribute something that and certainly the supervisor can edit it and direct it. But we have two staff who are day to day doing the operations.

 

Will Bachman  04:41

As a physician, or even as a highly educated consumer, what sort of advice would you give to someone of if you’re looking at an article in a medical journal or more broadly, any sort of published article? Are there any sort of signs Does that give you kind of rays, the hairs on your back that make you think this seems like something that might be retracted someday? What are the things that kind of raise your your your suspicions a bit?

 

Ivan Oransky  05:13

Yeah, I mean, I think well, it gets to, and I know that education has changed since you are not you and I were in elementary school and what have you, but it gets to the old idea of showing your work, right? Because even if you get to the right answer, if you haven’t shown your work, I’m not sure if you know how to get the right answer again. And so I look for, whether it’s news stories, whether I’m reading studies, whether I’m, you know, reading claims by anyone, politicians, you know, people in industry academics, I look for, Show me the evidence, right, so how did you get to that conclusion? What, what did you draw on? And by the way, can you tell me how sure you are about how confident you are in it. In any the end, if you look at a continuum of, you know, what, for example, politicians tend to do, which is make big bold statements and promises, without any backup at all, or with this threat of backup that doesn’t necessarily hold up. And I’m not saying only politicians do that, by the way, but it’s how people typically think about, you know, claims that politicians make, to maybe the other extreme, if you will, the other end of the continuum, where you have, you know, a researcher publishing a fairly esoteric, and really detail and data driven paper article in a peer reviewed journal, right, that’s actually been reviewed by someone, although we can talk more about what their theory was all it’s cracked up to be. Those that sort of to continue. Now, I know which one I trust more, I don’t necessarily know, gonna be able to understand the winner trust more. And so if you’re looking, for example, just at a newspaper article, you know, or an online, anything doesn’t have to be in a newspaper. Did they link to original sources? You know, and when you’re looking at a study, do they cite relevant work? Is it in a journal that seems to have a good track record of actually doing quality charity, you know, some of those things are not that easy to spot, sometimes by design for a, you know, a late, you know, consumer audience, for example. But it really does get back to, you know, show me a word, show me how you got there. And also, what level of humility do you have about your claim? I am a, I’ve become more maybe I’ve always been a sort of stay in your lane guy. I don’t tell other people whether they should stay in their lanes. But I actually have come to, you know, every year more and more respect and appreciate expertise. And whether it’s my own expertise, whether it’s someone else’s expertise. And I don’t like to make pronouncements about things I don’t know anything about. And what you often see is people using whatever credential they have. And I think a misleading way to suggest they have expertise about things they don’t. And so for example, I’m asked to, I’ve been asked to like peer review, and maybe also define peer review for your listeners a little bit. That’s when people you know, an expert allegedly appear, looks at something and says, Yeah, this, this is likely to hold up or not, and or maybe says it’s not, and it shouldn’t get published. That’s the general what period you’re supposed to do. I’ve been asked to peer reviewed papers about COVID-19, for example, because I co authored a tiny letter about retractions of COVID-19 work, which, you know, I’m an expert in retractions, I’m gonna come to that an expert in COVID-19, I did go to medical school, that’s maybe 1% more expertise than the average person. But it’s not I don’t know anything about it, other than what I read myself. And let me

 

Will Bachman  08:59

let me ask you about retraction. So what I’m curious to get your sense of the main types of causes of retraction. So I imagine that, you know, one big one could be like, just outright fraud and other areas, someone made a mistake somewhere in their algorithm. And it’s like an honest mistake. In other cases, it might be, I don’t know if these get retracted, but ones where you just sort of happened to get non null results, but it was, you know, similar to just luck. I mean, if you have a 95% confidence interval, like, you know, 5% of time, that was just random luck that you got some positive or negative results. So what would you say are the big drivers of retractions and any kind of percentage breakdown just roughly.

 

Ivan Oransky  09:47

Right, so a couple of weeks. So the question what what is it about two thirds and retractions and that’s an environment there are now more than 50,000 Over the past couple of decades and goes back a little further than that, but about 50,000 are tracks Presents about point 2% of lyrics and it’s about one in 500k Plus, the real number should be much higher than that probably 10 times that. But that’s the number now. And that’s what we’re, that’s when I give percentages, that’s what I’m talking about. But two thirds of the time. And this is pretty consistent over various work that people have done like a real art database or earlier than that, but other work other databases. About two thirds of the time, retractions are for something, it’s considered misconduct. That breaks down a little bit, you have to look at the federal definition in the US of misconduct, just followed by a lot of people around the world fabrication that was making it up falsification, making it look better than it really is, or plagiarism, that’s actually part of the definition of misconduct, it’s about two thirds of the time. And that 20% of the time, it’s due to something that I think colloquially we would refer to as honest error, Somebody ordered the wrong chemicals, they, you know, ordered the wrong mouse, they mixed up, you know, some data in a way that you could walk through and see how they did it unintentionally, rather than intentionally where it sort of helped their claim, help their argument. And those are sort of the reasons that when you dig into these stories, and you look at attraction notices that run alongside attractions, those are the reasons that are quoted, but I think it’s always worthwhile looking at taking a step back and looking at the actual causes, right? So we think about a destination, we get the sort of proximate cause or um, you know, what really killed this person. The the deeper cause, pretty much universally is some version of Publisher Paris, the requirement that around the world in different ways that in order to get a job in academia, tenure, promotion, yada yada prizes, you need to publish in certain places generally, hold us in certain journals, that drives people to do all sorts of things. And I’m still confident that for most people, that drives them to work harder and try and work more efficiently and work better. Some people it drives them out of academia altogether. But some percentage and 2% of researchers admit to committing misconduct when you ask them that some percentage will go to the dark side will fake data will beautify data will be selective about their data, which was thinking a little bit what you were getting at with the question about, you know, 95% confidence in roles and things. But that’s, that’s the that’s more of the root cause.

 

Will Bachman  12:33

It seems that you were a little bit ahead of the curve, there was this entire kind of wave of at least attention being paid to the replication crisis and the social sciences, but I suppose also in the hard sciences. What do you think has driven that? Is there reasons other than this, publish or perish? Of? Why are we seeing so much issue with replication crisis? Now? What’s driving that beyond just the publisher perish issue? Yeah,

 

Ivan Oransky  13:05

I mean, I don’t know that we were ahead of the curve. I think if you look back, it’s really interesting to see when different fields had their sort of, you know, taking themselves out to the woodshed moment or, you know, Holy S moment, right. And so, again, anesthesiology goes back to 2008. That’s when Adam was writing about this, and some of the editors were realizing they had a problem in different journals, universities, medical schools work, cancer research was more like 2012, right, seminal paper that in nature, saying that most basic science in cancer was not reproducible, again, of a certain data set, I want to be clear psychology had around the same time they had Diedrich stopple, who was actually going out and making data up. That wasn’t just sort of a reproducibility replication problem. And over time, other fields have come to realize that they have issues too, broadly speaking. You know, so when we launched in 2010, when people weren’t really talking about the replication crisis. That’s true. And I still hate the word crisis. I think it’s been with us much longer than that. But in terms of retractions, I mean, certainly, when we launched, there were about 400, retractions from journals a year. Last year, they were 10,000. Well, actually more than 10,000. And that that’s a big increase, even if you account for the rising number of papers published, right. So it’s not just the linear relationship. I think so why, you know, why are we seeing all this? I think it still gets back to the same root causes. replications, by definition, are not new research. They are not new findings. They should be cherished and prized and rewarded, but they’re not. And that’s because, you know, frankly, I’m being a little bit over simplistic here. I sometimes do that. I’m a journalist, but, you know, big journals don’t like to publish, you know, sort of replication. They don’t I mean, they just don’t do it. If therefore, you You know, in order to get into a big journal, you need to publish new research. What are you going to do? That’s just the market. I mean, that is simple behavioral incentive economics. And that’s what you’re going to do. And so it all comes back really to the same problem.

 

Will Bachman  15:16

So what are the incentives for people? Or where are these retractions coming from? Who is uncovering them and doing all this work to find these fraud or errors and make them come to light? You know, what kind of incentives are there for people to do that?

 

Ivan Oransky  15:36

There’s every incentive for them not to write, starting with the fact that the vast majority of these people who we we refer to as sleuth so that people call them different things. But we’ve got a page on retraction watch, that is dozens of their stories, whether there are stories that we link to other people’s stories about them. These are the heroes and heroines of this work in this whole, you know, field of science. And so you have you know, we have dozens on our page. That’s a small number. I mean, there are far more. Some of them are quite well known people like Elizabeth Elizabeth pick, you know, as you and I are talking Sholto, David’s is in the UK in Wales, he’s been getting a lot of attention for his work, finding serious problems that has led to her attraction in Dana Farber Cancer Institute Research. And there are others. We talked to them all the time. Again, they’re not paid, the vast majority of men paid zero. And some of them are paid a pittance. While these huge publishing houses that make billions of dollars in profit every year, you know, some of them have started to hire people like that. Most of them have not. And so and by the way, even if you just sort of have a way to work for no money, which would be great. I mean, if you’ve got, you know, some suggestions on that, well, I’d love it. But I need to earn a living. And so these folks, you know, if you even if you figured out a way to do that, then you sit you face legal risks. You face lawsuits, Francesca Gino, who’s on leave from Harvard, for Harvard found, you know, misconducts, eat some tests that she’s suing the data colada, folks who really uncovered all those issues for defamation. She’s suing Harvard as well, for $25 million. And, you know, that isn’t something that people can or should take lightly. It’s rare still, for people to sue. But all it takes is one, and it and it really creates a chilling effect. So these are the people whose work we should be celebrating, we should be rewarding, we should be encouraging. And I think that’s starting to happen with all the press attention. I think it’s great. The fact that we have lots more people writing about this stuff, and even when we were kind of one of the few in 2010. Like it makes our lives a little harder as journalists, you don’t get as many scoots yourself, but you know what, it’s great for the world. And, you know, it’s but we need to be doing the same thing for the students.

 

Will Bachman  17:59

I can imagine that beyond the legal risk, there’d be some potential negative repercussions about challenging some senior professional in your field, right, of misconduct, you know, maybe you’re 99% Sure, or but it’s maybe not a slam dunk case. But it certainly seems like misconduct to challenge someone like the professor, you know, the president of Stanford, for example. And so, are the people doing this sort of who are not in the field? Or are people in the field? And it’s like a young up and comer like what’s the typical profile of one of these sluice? Yeah,

 

Ivan Oransky  18:38

it’s a good question. So most of them and I don’t want to over generalize, but most of them one way or another, I would put it this way, their career trajectory is not dependent on pleasing or failing to displease, you know, you know, senior members of academia. And I think that, that that’s sort of consistent with, you know, me and Adam, right. We have, again, day jobs where we’re accountable to people we can be hired, fired, you know, demoted, promoted, but none of them, you know, we’re not writing about their work. I mean, partly for calcium Institute, but literally, it’s different arenas. And that’s what’s true. So a lot of these people had trained in science, but they’re not working in science anymore. What this is a very much a sideline of what they do, or there are a few, a handful of very senior academics who’ve taken this on. I’m thinking in particular of two, but there are more. One, David vo who’s on our board of directors actually at the Center for scientific integrity. He very prominent cancer researcher in Australia. He’s recently retired, but for many years was against loosing out people who, you know, were also working in cancer. He did not have to worry about he won’t have to worry about certain kinds of repercussions, but not sort of, you know, he was he was quite senior and their version of tenure and everything. David Allison was The Dean of the Indiana University School of Public Health, quite prominent academic, has been working with his grad students and postdocs for years to find out, you know, again, find problems and claims. And so those are the rare cases, most of them. And by the way, a lot of them work anonymously because of this. Those two, but they use a site, many of them called pub pier, which is an excellent Board of Directors volunteer there. So full disclosure, but that’s where you can leave comments on any study that’s been published and anonymously. And that starts to show up in the media where people need someone like shelter, David, Elizabeth pic, others are using clip here’s the platform, it gets it out there, makes it public, and then people can look at it.

 

Will Bachman  20:48

Ivan, just to wrap up, thank you for joining Share, share the links of where people can find out more about retraction watch.

 

Ivan Oransky  20:55

Bestival bestplaces, retraction watch.com, which will take you as deep a dive as you’d like to take into our our journalism, linked you to our database. It’ll show you some other pages that we’ve come up with. We have, you know, sort of resources there for people who want to learn more. We’re on sort of some of the social media and who knows how long that will last but we we welcome feedback. We welcome more story tips. And really, it’s been a pleasure to speak with you today. Well,

 

Will Bachman  21:25

Ivan, thank you so much for joining today.

Related Episodes

Episode
569

Automating Tax Accounting for Solopreneurs

Ran Harpaz

Episode
568

Integrating AI into a 100-year-old Media Business

Salah Zalatimo

Episode
567

Author of Second Act, on The Secrets of Late Bloomers

Henry Oliver

Episode
566

Third Party Risk Management and Cyber Security

Craig Callé