Paper's Podcast theme collection: Assessment and evaluation

Start

[music]

Jason: Welcome back to the Papers Podcast where the number needs to listen is one. The gang is all here. We’ve got literature for you. We have insights and we have Lara’s dental pain. So like you can’t go wrong for an episode like this. So Jon’s here.

Jonathan: Hi Jason. Hi Lara and your nitrous oxide.

Jason: Lara may be a little bit more giggly than usual. I thought it was just my jokes, but it’s apparently the nitrous oxide.

Lara: oh no.

Jason: Lara, how is your tooth?

Lara: I had a root canal and honestly, I think my dentist was really excellent but I’m just so not emotionally prepared for this reality.

Jason: Remember what I said, I do discount dentistry in case you

[Laughs]

Lara: discount dental, thats what I dont need

[Multiple voices talking at once]

want an alternative way of getting things done.

Linda: Her face is already crooked and swollen up on one side. You’re gonna have her and it’s swollen.

Jason: Yikes.

[Laughs]

Lara: It is swollen though and if we’re on video, it’s not a good day.

Jason: Awesome. Okay, so, but Linda’s got a paper. So, Linda?

Linda: This is a paper about mistakes. Now, when I first said this paper around, Jon said, we don’t need to do another paper on clinical reasoning, but that’s not what I think it’s about here. I think it’s about mistakes and whether we learn from them and how we learn from them and whether we learn from them in practice. I’ll tell you, the authors are Kotwal, Howell, Zwaan, and Wright.

The latter two names I know, and Scott was trained with us and has become an outstanding clinician educator. Laura Zwaan, think, is a guru of clinical reasoning, if I understand correctly.

Jason: Scott Wright is a man with many clever papers. You got to give him credit.

Linda: sure.

Jason: He’s the guy that did… He got a New England Journal paper, I think, as a chief resident or something.

Linda: Yeah, that was about our context. So the idea here is that we talk about mistakes and how we learn from them. So I’m going to ask you all in a minute to describe for me or reflect on a mistake that you might have made clinically, preferably or otherwise, and a success that you’ve had and what was the takeaway from it. But just as we’re thinking about mistakes or errors, we know that in medicine, errors are distressingly common. And many of these errors are diagnostic errors, not all like that. But if we look at errors as opportunities to learn and improve and not to assign blame, we should be increasing the competence of people because we’re adopting a growth mindset. And clinicians should continuously grow with feedback so they get feedback on their clinical experience errors or successes.

Now, before we get into some of the details of reasoning why this paper, let me ask you, I’ll go Jon, Lara, Jason, to reflect on a mistake or a success and the takeaway.

Jonathan: There’s a mistake that I led as a trauma attending very early in my career. And I delegated a diagnostic act to a part of our team member. If you know anything about trauma medicine, it really is something that’s done as a collaboration. You have nurses, have respiratory therapists, you have clerical workers, you have porters, and then radiation technologists.

Then you have a surgeon and then you have an anethetist and et cetera, et cetera.

Jason: You mean an x -ray tech? I don’t think you mean a radiation tech because that’s different kind of trauma.

Jonathan: Okay. An x -ray tech, but you know, what’s the CT tech, guess. You need a bunch of people. And I delegate an act and that person worked within their bounds and made the mistake. And I never had a system that checked it. And ultimately a young person died and I didn’t sleep for weeks.

And what I learned from that is how people can come around and support you. And I certainly have a slightly different trauma practice, but man, I’d love to not have made that mistake. I’m pretty sure I can learn that lesson a lot easier. And there’s nothing victorious in my answer, except to say medicine’s really, really, really hard. And if you go through your entire practice imagining you’re not gonna make a mistake, you just haven’t paid attention enough. There’s a bunch trailing behind you.

Lara: The mistakes that I’ve made over the course of my career, I’ve talked about publicly many times. So instead of talking about that, what I’d like to do though is, because I’m a professor, I don’t have a clinical practice. I’m not a clinician. So professionally often the mistakes that I make are just different in scope and scale. But what I would want to point out is that the extent of my errors or failures is long and illustrious.

There’s a scholar, think is at Princeton, who started a CV, a curriculum vitae, a living document of all of the things he tried and didn’t do, he wasn’t successful at. And that has inspired me. So I started putting my own together a few years ago. It is full of grants I applied for that I didn’t get. Papers that I thought were really fantastic and not a single reviewer agreed. It’s full of people, you know…things I tried to do in an effort to make things better and it didn’t work at all. And one of the things that has helped me is to think about learning from failure, is to recognize that the list of my failures is always longer than the list of my successes. But the lessons that I’ve learned are invaluable. Like the only way I’ve learned how to write a really good introduction to a grant is because I have failed so miserably before, so often, that I now have a bit of a better sense of what makes a compelling argument in a grant introduction.

Jason: What am I going to say about this? I actually have a really big problem with the premise of this whole topic, this idea of mistakes, because it’s just such a nonspecific word. It’s poorly defined or even medical error. I know there’s a definition. There’s several operationalized definitions of medical error. I think sometimes we make a diagnosis and someone else makes a different diagnosis. Sometimes they make a diagnosis and had we made a different one, the patient might have done better. I think that’s true. So in my career, there’s lots of times because we had metrics back, know, and other people have different diagnoses or people bounce back, get admitted. There’s endless times where people have made different diagnoses and I reflect on them all. Sometimes it’s because new information becomes available.

As an emergency physician like Jon, we are always making best possible plans on incomplete information. So that’s a really prevalent thing. Thank goodness, what’s tapering in my life so far, knock on wood, is there’s fewer and fewer things where I just had no idea about that topic. Like I had not seen that key feature before. I definitely have a mental list of the scary versions of those. You know, I’ve had patients come in with some horrible diagnosis that I had never seen and didn’t recognize the key features for. It may or may not have made a difference, but they do haunt me. So I do think about them a lot, me in the intellectual sense. So this topic bothers me because there’s so much pejorative language around it. It’s a barrier. gets in the way of us all doing better. And I think we need to be more precise. Sometimes we make a diagnosis that other people disagree with, blah, blah, blah, blah. Did it harm the patient? Did we learn from it? All those kinds of things. And I know that this is the paper’s about, so I’m setting you up.

Linda: So thanks you all. I’ll just add very briefly, I think I certainly have made mistakes, but my approach to having made a mistake has changed over time. I remember how defensive I was when somebody pointed out that I’d missed a low serum potassium that probably contributed to somebody’s arrhythmia. When I was a resident, couldn’t be me and did I deny deny. And now I want to know about it. I want to do something about it. I want to learn from it. So there probably is an evolution. Their title is Exploring Clinical Lessons Learned by Experienced Hospitalists from Diagnostic Errors and Successes, and it’s in a clinical journal. It’s in the Journal of General Internal Medicine in January 2024.

Jason: It’s a great journal.

Linda: Excellent journal, and we’ll maybe come back to that a little bit. These authors point out that there is such a thing as diagnostic excellence. And part of it is we need to learn from our errors and our successes. And they introduced me to a concept or an approach I hadn’t heard of called the safety one and safety two approach. Safety one being let’s identify the cause of the error and try and fix it and prevent harm. Safety two being let’s focus on the successes and we’re trying to repeat them as best practices for safety. The other point to make here is that in the US, many, many hospitalized patients are cared for by hospitalists who play a key role in not only the care, but in the diagnosis of patient safety. And hospitalists often are busy promoting patient safety as well.

So what we don’t know is the lessons that have been learned by experienced hospitalists from their errors and successes. So these authors say, since it’s not being described how hospitalists learn from their clinical experiences over time, specifically the clinical insights attained from errors and successes in patient care, we conducted this study to identify and characterize clinical lessons learned by seasoned hospitalists from diagnostic errors and successes. So, very briefly, I’ll go Jason, Lara, Jon, your thoughts on the premise, something that you’d like to follow up on?

Jason: So, I’m to be grouchy on this one. So, I’m going to reiterate, I have trouble with the language used. It’s pejorative. I’m not sure the logic of what they’re saying is they’re going to research mistakes, it’s in the title. And they talk about diagnostic error and they’re asking people how they learn from it. But we’re gonna get into some of their findings and I’m not sure that people are answering that question. So I’m worried about whether I’m gonna be grouchy through this whole episode. I like the premise, I like the idea of helping us all get smarter and talking about the lived experience for this topic, but I would prefer some more precise language.

Lara: So from my perspective, I’m fascinated by the literature that addresses mistakes, failures, I often frame them as surprises, things I hadn’t anticipated would happen. In fact, had short -tangent, had a great opportunity to do a lovely collaboration with Perspective of Medical Education once co -edited with Alisa Nagler. And we wanted to call it learning from mistakes and then cross out the word mistakes and write the word surprises. And we wanted to leave the word failure crossed out.

Can I just tell you how many hours of emails I spent with the publisher saying, don’t delete the word that’s been crossed out. Leave it crossed out. anyway, I’m really interested in the topic. I really am. And so I’m looking forward to this conversation. In terms of the premise, there was nothing in there that gave me cause for pause. There wasn’t anything that I was worried about. They make a bit of a comment about there is no other research on this topic. And I never think that’s a good justification for a study.

This topic is important to me, so I’m in.

Jonathan: So you all know that I do have a whole program of research around clinical reasoning and diagnostic error. I hate, just like Lara, any one that’s justification of, we are the first ever, and then they choose niche, niche, niche, condition, condition, condition. And the punch line is we’ve done this study. It’s called I Made a Mistake. And the senior author was [inaudible].

She did as part of her PhD. It was, I think, really nice study that in parallel would help inform what this study was trying to do. And we used narrative inquiry and built a meta story. And we had very different findings from this. And not that every study that I ever do needs to be referenced by someone else’s paper, but it’s interesting that when I go through and look at the reference list, the situation of their problem doesn’t become obvious to me in hindsight. It’s only after I kind of got down into the results, I was like, this feels very different than the other stuff I know. That I was like, I’m a little worried about it. Now, Laura Zwaan is a colleague. She and I have been part of a number of projects together. And so I respect her deeply. And so when you sent this paper along, I was like, I’m interested to read it. But I’m foreshadowing that I was a little surprised where they went or how they arrived at that place.

Linda: OK. So we’ve heard some concerns and yet it’s piqued the interest, I think, of most people here. So let’s dive into the methods very briefly. It’s a qualitative paper. They used an interpretivist brackets constructivist paradigm, which, and I quote from them, “holds reality as multiple and subjective related to how individuals understand and create their own meanings influenced by specific social contexts”. They did semi -structured interviews of hospitalists with more than five years of experience in six hospitals, three communities, three academic in the US Northeast. They found 91 eligible people. They invited 30 for interviews and 24 were eventually interviewed. They used a guide to understand participants’ perspective of lived experiences in this case with mistakes and to generate rich descriptive data. And if you look at the interview guide, it certainly has some questions that would do that. They well describe the interview guide and the piloting. Have the questions focused on diagnostic errors, sorry, on experiences and reasoning with challenging diagnoses and half focused on diagnostic errors.

So back to something that Jon said on a recent podcast, if you’re going to ask questions about a topic, don’t get distracted by asking questions about something else as well in the same interview. They also elaborated on the lessons learned from errors and successes, and that’s in fact what the whole point of this study was.

They did a reflexive thematic analysis. They described it well. And now I have a question for Lara before I ask the rest of you for your comments on the methods section. There is a sentence which says, to assess for data sufficiency, we relied on information power and judged the sample to be adequate to answer the research questions. So I asked myself, what the heck is information power? I know saturation is now out, but is theoretical sufficiency also gone out the door?

Lara: OK, so saturation is not out, theoretical sufficiency is not out. They’re just different concepts and often work in different paradigms. But let me focus in on information power, how it is, how to use it, those sorts of things. The concept comes from a 2015 paper by Malterud et al. And the paper is called Sample Size and Qualitative Interviews.

I’ll make sure that that manuscript, a site, a link to that paper is in our show notes. So problematically, they don’t actually give like a hard and fast definition in the manuscript. But the basic premise is that in qualitative research, there are no hard and fast rules about how you know you have enough data to answer your question. There’s no equivalent to a power calculation. So instead, you have to take into consideration several different factors as you design and conduct your study. Together, these different factors help you to understand if you have enough data to answer your question. So do you have enough power in your data, enough information power to answer your question?

There are five factors to consider in terms of information power, right? So I’ll give those five to you. One is study aim. A really broad question will need more data than a really narrow question. This makes sense. If you have a broad question you have a lot of space to cover if you have a much more narrow question, your phenomenon is smaller. Sample specificity. This is about the specificity of experience. The more specific the experience you’re studying, the smaller sample you will need. If you’re using established theory, that’s number three. If you have a theory you’re using to shape your study in your analysis, you will need less data than if there’s no theory guiding your work. Four is quality of dialogue.

If you’ve done interviews, you know this to be true. Some participants are really good at talking. They’re really eloquent. And if you have really eloquent participants who are able to describe and articulate their thoughts well, you’ll just need fewer participants. Finally is the analysis strategy. If you’re trying to explore a phenomenon across multiple cases, you need more participants than if you’re doing in-depth narrative analysis. So there’s a little picture, actually, in that paper, that we’ve recreated and that I’ll make sure is in the show notes. It’s totally worth a look. It is those five elements and it shows you the arrows and information. It’s really good. Even Jon would like this picture. So it’s in the show notes.

Linda: Even though there aren’t straight lines, there are curves in the picture.

Lara: High quality stuff.

[music]

Linda: So, Jon, Jason, thoughts on methods?

Jonathan: I think overall, I understand how they got there and I understand what they did. I appreciated Lara’s explanation for information power and I actually did a bit of reading around it and I would say I’m not sure if they had strong information power. They didn’t have a theory to inform it. And then the quality of the dialogue, I wonder about that.

If you look at their interview guide, this is my newest thing. I seem to be going to the supplementary material every time. A third of their questions are about their opinion, not their experience. So I want to, if you’re going to tell me a story about when you made a mistake and what you learned from it, that’s valuable. But if you say, what should everybody know? What should we do different in the system? How should we fix all this? That’s not your experience. That’s just, that’s just pontification without, not informed by your actual experience. And that kind of data, I don’t think adds to information power.

You might have lots of words, but I’m not sure it’s grounded in something that’s lived or experienced. And so just want to be cautious about that.

There’s a couple of things I thought were interesting. I’ve never seen this. They pilot tested their interview guide. What the heck is that?

Lara: I was going to have the same point. Things that make you go, huh?

Jonathan: mean, there’s no right answer with what you don’t want stationization with your interview. You want to make sure that the words make sense. You want to run them by some people, but you don’t want to say, hey, let’s go get some data and make sure that we have.

Jason: But they talked about ambiguities and so on. I thought that was okay.

Jonathan: It is funny. That’s fine.

Lara: no, that’s not the part that made me go, huh? Hold on. Maybe you’ll get there.

Jonathan: I’ll leave it for you. This is the last part I wanted to say. The reflexive statement is a non sequitur. I rarely talk with reflexive stuff because I think people are getting what this is. This is really a non sequitur. I don’t think they understood what they needed to say there. And I’m prepared to say it’s because of the journal.

When we get to the limitations section, they’re gonna do some other things for their non -sequiturs. And I wonder if the journal said, hey, we’re not comfortable with this methodological stance. So just so it all holds together and wow, this might be weird maybe the journal made them phrase it this way. They said stuff like, we didn’t do a sample size because we’re doing qualitative methodology, which is like a, yeah, okay. And we didn’t test any hypothesis. I was like, yeah, we got that.

Jason: You’re you’re doing qualitative methodology.

Jonathan: On the whole, their methodology made sense.

Linda: Lara, did you want to just go back and say where you were going huh before Jason?

Lara: Yeah, just at one point, there’s one point in the manuscript, there are several points in the manuscript that made me go huh, and I agree with you completely, Jon. I think it’s a factor of where they published in different expectations. Place where I went huh was the line that read.

So they’re talking about their interview guide and how they created it using literature. Great. And expert input from co -authors. Great. But then here’s where I went, quote, “we presented preliminary versions of the guide at multiple division meetings in general internal medicine, hospital medicine, our institute, and made changes per the feedback we received”. End of quote. Huh. I have never done that. Now, it never occurred to me to do that. And then I thought, well, that’s a good idea. But then I’m just.

Not sure how I think about it. And so it made me go, huh? And I wanted to ask you guys because I’m not sure.

Jason: I had scratched on that one, too. I thought it was kind of like in a month, I’m going to invite you to the interview and here’s all the 10 questions going to ask you. So you might want to think about some answers. no, no, we want to change that question. That’s kind of a weird dialogue.

Linda: It’s funny. I viewed it as they were presenting it to people who were going to be part of their of their cohort of people interviewed because you want to make sure that it’s understandable and not ambiguous and all of that. Anyway.

Jonathan: The questions are not ambiguous. If you go to the actual questions are, describe an important lesson you learned from an episode of Diagnostic Reasoning.

Linda: Yeah.

Jonathan: How do you like, how do you try to improve your diagnostic?

Lara: But they took it to a meeting. I think that’s kind of cool and at the same point, I’m also like, I’m in that meeting. I’m the person going, I got other things to do.

Jonathan: But it’s good on them for a culture where you can discuss research design and integrate into all the other things. So, let’s not beat up on them from that. It’s just a kind of a head scratcher. If you can integrate your scholarship into the administration and the finances and all that other things of everything you do, you may have a really good culture.

Jason: Is one of them the division head? Just checking, just asking.

[Laughs]

And now I’d like to tell you all about my research.

Linda: No, in fairness, okay, not all meetings are a day and meeting could be anything. They could have had the opportunity like many people to present their research in progress. It could be that kind of a meeting. Doesn’t have to be a finance.

Jason: Could be diagnostic errors anonymous who knows?

Linda: Jason, why don’t you go ahead with some brief comments about the methods?

Jason: There are lots of funnies in this. And like Jon, there’s all these funny lines that we could pull out. Funny in the sense of they’re kind of like don’t quite match like the stance. We should coin a new term like they’re qual apologies. Something like that. Like little things that say, we didn’t have a hypothesis. I noticed all those too.

My challenge with this paper is it just seems like it’s poorly, it’s wrongly packaged. Not poorly, wrongly. I thought the intro was all about mistakes and how we can all get better. But really what they’re asking is, what have you learned in your practice to be a better doctor?

And like Jon, I was really struck by most of the anecdotes that they printed are not about their personal experiences, like their personal choices as a clinician. They’re about team and system things and other teams things with a patient that they related to. Like, hey, I had a patient, they got admitted to ICU and they changed the med and then they came back to me and they weren’t doing so well. And that made me think, right, there’s one of them though. One of them was, we missed, one of the quotes was we missed CNS lymphoma or something like that. And that one was closer to what I thought the whole paper was going to be about. Here’s my experience. Here’s what I learned from that, because that has all of those lived experiences have value. I could bring that to senior residents and say, hey, save yourself 20 years of practice. All of these people were in a study and here’s their pearls. But instead, I didn’t take that away from these results or these methods. I thought these methods started one place and then started asking questions about another thing and then analyzed another way. They lost me in the logic and I really wanted what they proposed and asked you that.

Linda: All right. So, let’s hear what they actually found. First thing I’ll say is they actually had a very broad range of demographics for the people they interviewed, in terms of age, experience, race, practice, academic rank, you name it.

They had five very broad themes about lessons learned from successes and failures, let’s say. The first is excellence in clinical reasoning is a core skill. The important part of that to me is that that includes foundational skills. You can’t make a diagnosis unless you gather the data, and you can’t gather the data unless you do a history and physical and order the right test and have uncertainty and clinical humility. So that was theme number one.

Theme number two was it really helps to talk to patients and to your colleagues, particularly when you’re stuck. That includes for patients, knowing the patients beyond their actual illness and what they’re here for today, and respecting all your colleagues. When the nurse says the patient doesn’t look well, listen to the nurse.

The third is thinking about the diagnostic process, having it somewhere above your brainstem when you’re actually making a diagnosis. Thinking about your assumptions, revisiting things when they don’t make sense, don’t blow it off, slow down when you have to. Go to the primary data, I’m always telling my residents that. Fourth, adopting a growth mindset, commitment to growth, they call it.

Learning from them by following up. What happened to that patient who I sent to the ward yesterday? Did I make the right diagnosis? And if not, why not? And finally, and this one had a little bit of trouble with initially, it’s called prioritizing self -care. Wellness and activities outside of medicine are important. And then I realized what they really wanted was if you’re…

If you’re well in yourself, you’re much more likely to have improved performance. So, they then talk a little bit about how does this compare to what is actually known? The authors say the findings contribute to the literature on diagnosis education, which is a field involved in improving the diagnostic processes and promoting a cultural shift to the growth mindset. That sounds reasonable.

I didn’t know there was a specific field of that. And then they say two things that confuse me. First, they say the results are consistent with what has been reported in the literature on diagnosis education. And the second, this adds credibility and specificity to the dimensions of quality embraced by diagnostic excellence. So kind of what I veered is they’re saying, you know, this is consistent with what we know. And then they say…

Well, actually, no, it adds a bit of credibility to it. And I wasn’t quite sure what to make of that. They do conclude key lessons were learned when dealing with errors and successes in patient care by clinically experienced hospitalists. And they suggest that the findings could serve as a guide to developing priorities for helping clinicians continue to learn.

Thoughts on the results sections? Any issues with them? I’ll go Lara, Jon, Jason.

Lara: The point that I really want to make about the results is this, and I’m going to offer it as a tip to our listeners. And it truly is just a piece of advice. In a qualitative study, you should be able to read the results section without having to read the quotes and still understand the significance and the meaning of your themes. The quotes from participants are illustrations. They’re examples. But you can’t rely on them to tell the story, the importance, the interpretations. And this should also give you a sense of how much word count to dedicate to your quotes. If you write up your results section and it’s mostly participant quotes and very little explanation or description from you as an author, you’re likely relying too heavily on the participants’ words to explain your findings. unfortunately for me, this was a bit of a situation I found myself thinking on as I was reading the results.

So I just want to suggest that that’s a really, it’s one of the things I talk to my learners that take all the quotes out and if it still makes sense, you’re doing fine. If it doesn’t, then you’re relying on your quotes to do the work for you. The only other comment I want to make about those results is that one of the things that I found really interesting and I’ll be interested in what the others think. For me, what was interesting about these findings was how the place where I would have put emphasis is about how Diagnostic reasoning isn’t just within the mind, the knowledge, the skills, the attitudes of the individual physician. This study shows us that it is that to be sure, but it’s also about the team you collaborate with and the fullness of the quality of their life. so diagnostic reasoning then is not just that moment that you were making the diagnosis. It’s bigger, it’s broader, it’s more. For me, that’s the takeaway message from this paper.

Jonathan: The findings all make sense to me. I’m not surprised to see any of them. I do wonder the way that the questions were posed led to a story like I led with. I don’t see my story of where I didn’t sleep for two weeks. I really had nightmares for two weeks where that kind of story would get into this paper. And so the other paper that I was a part of, the senior author, let me just mention her again, is Kandasamy, has some similar findings. And the punchline there was, if you have support and you have a reflexive ability, it will lead to growth. But here they say things like, you should have a growth mindset and you should have a system of feedback and you should embrace mistakes as opportunities for learning. I just wonder if some of the if there was a bit of virtue signaling in their responses. And then the part about prioritizing self -care is exercise, sleep, healthy eating and meditation are believed to be important. And reading fiction can help you with diagnostic skills. It doesn’t speak to the information power, the richness, what’s coming there. All that to say is I agree with all of their findings. I think it’s all there. But then the part that kind of makes me wonder, did they get it, is I don’t see any of the other big author groups like [inaudible].

I don’t see our own paper. I don’t see it situated in the other literature. And when I go to the reference list, as I mentioned before, where is it feeding off of what is already known, what’s already been reported and already discovered around that? How does this move things forward? It almost feels like a bit of a lonely island. It’s connected to the other literatures, but not the ones that advances forward. And so I think there’s good stuff here, but it could be great if we could see how you connect or how you see the difference or how you see it builds. I don’t see that happening. And you talked about the results section, Lava. I’m talking about the discussion section now about, and the so what of it. Here’s our findings and now so what? That part didn’t get there for me. Okay.

Jason: I don’t want to pile on because I want Scott to still be my friend when I see him at conferences.

I read this paper with the lens of a frontline teacher who really cares about the next generation and wanting to save them time and I want to be that guy that gives them pearls and I was looking for this paper to be one of those sources that might help us. I didn’t find it. I’m really sorry. You know, there’s a logic here and I follow what the authors are telling us, but I didn’t find any of it actionable. If they’re telling me to be more well and help me make more diagnoses.

It’s just not something I can bring to my fellows and my most senior residents as they transition to practice. By contrast, know, our group every now and then holds this panel of docs of various vintages and they all talk about like hard lessons they learned that changed their practice. I wanted more of that and I didn’t find it here in this academic paper.

Linda: So I’m hearing the need for more specificity in terms of what Jason’s calling pearls, and I’m also hearing the so what of what’s actionable is not in here. They do say this should act as a guide, but they don’t exactly say how. I got to say what I got out of it, and maybe this is a couple of paper clips, is one of the paper clips is that we still have to teach, it goes back to the first theme that they had of…you have to have good diagnostic reasoning and you have to have good data for that. So we still have to teach and you still have to learn how to do a history, how to do a physical, how to select labs, that sort of thing. So that’s one of the big things I took out of it. The other thing I took out of it is this is a clinical journal. And two or three of you mentioned, well, maybe it’s because of the journal. But I think if I were a clinician reading this, you know, good on them. This is a qualitative education study, and I think it’s reasonably understandable for clinicians who are reading a clinical journal. Maybe that explains some of the apologies. Apologies, we don’t have a sample size. Apologies, we don’t have whatever else it was. We’re not producing testing hypothesis.

They may have been asked to put that in because it was this kind of a thing, kind of a journal, but frankly, I think it does make it understandable for a clinician. I think so, yeah. And good on them, as I always say, well, we have a clinical journal for publishing education stuff. Don’t forget to use my term, the qual apologies. I like that. Hashtag. All right. Let’s go to our assessment.

[music]

In terms of the methods, I’ll go Jason, Jon, Lara.

Jason: I’ll make the grouchy guy go first. Scott, remember we’re friends, I had trouble with the logic, not with what they did. I trouble a little bit about how things were analyzed and illustrated. So I’m to give this a three.

Jonathan: I’m going to give it a three as well. I think for me on the scale, three is this is representative of the state of the art. And I think it’s right, right down the middle.

Lara: The same for me. I’m giving it a three, no fatal flaws. Absolutely. It’s solid. It would be whatever, what I would expect.

Linda: You’re not going to add anything for the paradigmatic approach, which they have.

Jonathan: Come on now, stop, stop with this.

Linda: you usually do.

Lara: Linda, I am in so much, like honestly, hun, I’ve had so many painkillers. I’m just thrilled to be sitting upright still. So let’s keep going, friend.

Jonathan: We just need you to be high and then you’ll play by the nose.

[laughter]

Is that what I just discovered?

Lara: I will, right now, like there’s reasons my husband won’t give me pens to sign anything right now. Cause I’ll disagree to anything.

Linda: All right. And I’m actually going to give it a three, four methods. I thought they were very clearly explained. They’re sort of straight down the middle.

[music]

Linda: How about the usefulness, the education impact of this? We will reverse it so Lara can go to sleep more quickly. Lara, Jon, Jason.

Lara: That’s a harder one for me because I’m not sure. I’ll give it a three. This would be among what I would see most of the literature as a three.

Jonathan: Same for me. It’s a study that brings some of the same themes back into the literature and so supports in a different context. And so it adds to the transferability of these findings that we’re seeing from multiple studies.

Jason: Two for me, because I read it looking for those pearls and I there’s a few in there. I really I hope that somebody listening to this study listens to this podcast and says, you know what, I can do it and take care of all these flaws or concerns that these hosts had. think another study should be coming that would build off this conversation.

So it gets a two for me.

Linda: So I gave this one a four. So I guess we all even out of three. I thought there were some useful concepts in here, maybe because I’m an internist and I can sort of see where they’re going with some of this. I thought it was, it made me think and it confirmed to me some of the things that we have to do in terms of teaching and learning the basic data gathering. So.

I think we’ve got an almost down the middle three for methods, and if you take an average, it’s three for impact. So we’ll call it a down the middle solid paper here. All right.

And that’s it for the Papers Podcast for this week. I would like to remind you that we’re always interested in what you have to say. You can write to us at thepaperspodcast at gmail .com, or you can check out our website at paperspodcast .com. And I got those right.

Jason: Well done.

Linda: Yay.

Jason: Three in a row.

Jonathan: Well, it took us two years, just saying.

Jason: 15 years.

Linda: So from my perspective, see you next week. And bye -bye.

Lara: Talk to you later.

Jonathan: Thanks for listening.

Jason: Take care, everybody, especially Lara’s tooth.

Jason: you’ve been listening to the Papers Podcast, we hope we made you just slightly smarter. The podcast is a production of the Unit for Teaching and Learning at the Karolinska Institutet. The executive producer today was my friend, Teresa Sörö. The technical producer today was Samuel Lundberg. You can learn more about the Papers Podcast and contact us at www .thepaperspodcast .com. Thank you for listening everybody and thank you for all you do. Take care.

Start

[music]

Jason: Hey, welcome back to the Papers podcast. We scan the health professions, education, literature to find cool papers to enhance your practice. Well, three of us do, but Jon has a paper today. We’re gonna go back to him in a second. So welcome, Lara.

Lara: Hi everybody!

Welcome, Linda. You have to unmute. It’s technology, Linda.

Linda: How are y ‘all?

Jason: Fantastic. Jon, I’m already picking on you. You just got back from a beautiful Algonquin park where you’re getting dad points for conduing down the Mississippi River through polar bear breeding dens and giant pterodactysl, whatever it was out in the nature there. How was that trip?

Jon: It was great. There was no pterodactyls. Jason, we’re gonna give you a little bit of an instruction on the Neolithic and the Mesolithic periods. We did see moose or mesas. I’m not sure what the term is. We saw some bald eagles, but the most… No, there’s no chocolate. That’d be delicious though. Linda, I did have a concession to civilization. I did bring wine, but it was boxed wine. So, But…

Linda: Sorry, where the mousse, chocolate mousse?

Jon: No, that’s not chocolate.

Linda: Aaw.

Jon: That would be delicious, though. Oh, Linda, I did have a confession to civilisation, I did bring wine, but it was box wine, so uuh. But, for all of you who are outdoor enthusiasts, there is nothing like paddling out into the middle of a lake late, late at night where there’s zero light pollution and seeing the Milky Way. It is something spectacular. For those of you in the Southern Hemisphere, you have a very different kind of set of constellations to look at here in the North. It was just fantastic. It’s highly worth doing it, even for all the effort, even for all the boxed wine.

Jason: Boxed wine, fantastic. All right, Jon, you got a paper. It’s a long one. I’ve already griped about it, but I have more gripes to come. Take it away.

Jon: Alright, so I’ve entitled our episode today as Assessing without “distressing”, factors Influencing assessors in low and middle income countries. this is one of my favourite topics, assessment. You know that this is where I like to spend my reading time, I like to debate ideas, and I’m hoping we’re not going to get into too fine a debate that distracts from a more general conversation. But it also has a second favourite and perhaps parallel topic for me, which is challenging the assumptions about the global north. And so here’s the set up. We’re in this era of CBME. One of the strong critiques of this competency -based medical education movement is the operational cost, the expense in both time and dashboards or logistics in operating programmatic assessment. Multiple assessment from multiple people, longitudinally aggregated together for a summative decision. That’s programmatic assessment.

So to complete all these assessments, to collate them, and then to confer by bringing together a committee or a series of people to try to provide a general gestalt of what all the data points require, that’s a lot of technological and human resources cost. In the global north, where resources are probably less constrained than in the global south, there has been ongoing and legitimate concern about that ability to take the theory and translate it into practice.

So imagine when you move into a more resource constrained context, like low and middle income countries, what’s the issue? So we’re going to tackle a paper. It’s by Sims et al. The official title is “Factors influencing clinician educators assessment practice in varied Southern contexts, a health behavior theory perspective. And it’s published in Advances in Health Sciences Education. All the cool kids call it AZ. It came out in May of this year. Now, before we jump into the paper.

Laura, Linda, Jason, I’d love to hear your experience been with programmatic assessment? Is it working? Is it have opportunities for challenge? What do you think?

Lara: So first I’m going to say that all the cool kids call it Advances, not AZ. But that’s…

Jonathan: Personal friend with the founding editor. I’m go with AZ as my answer.

Lara: Personal friend with the current editor says Advances. But anyway, I actually, you know, yeah. Okay. AZ advances, whatever you call it. Great journal. Happy. So glad you picked an article from there because I really am a fan of the journal. When it comes to programmatic assessment. You know, it like in most things in life, I’m an extrovert. So I’m going to think a crowd is better than being alone. Cause that’s kind of my, my, my sweet spot of life.

But when it comes to programmatic assessment, I believe that even more so because, you know, our collaboration and our evaluation teams are full of different personalities, different perspectives, different priorities. We might all be looking for a marker of excellence, but we’re all going to be looking for signs and signals of that in different ways. And we need those different perspectives, for me too, I also believe that there’s a hawk dove phenomenon. You know, some of us are a little harsher as evaluators and some of us are a little softer.

And I think that’s, you know, that’s part of the reason we have multiple evaluators on many different fronts so that we can try to balance hawks and doves. So I think it’s going to be really hard for any one person to hold all that variety, all of those hawk and dove orientations. So yeah, I’m in favor of a crowd.

Jonathan: I love it. You’re foreshadowing maybe a punchline from this paper. All right, Linda, what do you think?

Linda: I agree with you, Lara. I’ll say as a past clerkship director that even before CBME and I’d say as a past residency program director too, those who were the directors of assessment always went out to actually get opinions from different people, whether it be written opinions or hallway conversations or whatever, to get that diverse perspective. So if the question is, do you think a crowd is superior to a loner? The answer is yes, but there’s a but to it. And the but is that those, crowd members, the individuals in the crowd have to be educated individuals who A, actually see the learner so they can assess them and B, do it in some sort of an authentic way without bias.

Jonathan: All right, I like that. Jason, you’ve built a lot of programmatic assessment systems. Is that even appropriate for UGME?

Jason: Sure, yeah. It’s just a principle, right? Just an approach that has some theoretical basis to it. So I live, like many of you, in a resource -intensive, well -developed, highly structured system where programmatic assessment’s been used for a number of years, and it works. I see it around me every day, and I’m pretty happy to be in that environment. I also work around the world in a variety of settings where programmatic assessment is something talked about. Sometimes it’s something that people are working to implement and they run up against two themes. One of them is resource constraints, just people and systems. And then the other is like infrastructure. There are lots of places that don’t have infrastructure that can do programmatic assessment. And maybe that’s apropos to your paper too.

Jonathan: Okay, so if you’re following along at home, here’s the setup. We have this era of competency -based medical education, which has one of its tiers or pillars, the idea of programmatic assessment. Short hand for that is multiple assessors. Now let’s contextualize it into the global South, into low and middle income countries, where end of clerkship rotations are often supervised and with somewhat of assessment performed by dun dun dun, a single assessor. So what is the weight? What are the consequences when you move from a crowd to the lone wolf? Here’s the purpose as spoken by the authors, which probably sounds a lot better than my short hand back of the envelope version.

Our research question was what factors influenced the assessment practice of individual clerkship conveners in exit level medical programs in diverse Southern settings? Okay.

Here’s the methods. It was a qualitative study which adopted a constructivist paradigm with a model of health behavior developed by the senior author serving as a theoretical lens. So they’re going to do a qualitative study. Their informing framework, their theory, their lens that gives them the ability to see their data and to interpret it is using a model that the senior author has produced. Now, health behavior theories include a whole bundle so it’s not just one thing. There examples would be theory of planned behavior, trans theoretical model of change. We’ve talked before about social cognitive theory, self -determination theory, et cetera, et cetera.

Interestingly, health behavior theories, HBT’s describe the behavior of an individual in relation to how they seek health and how they maintain their health. But in this context, they’ve pivoted and to try to say, okay, there is a strong parallel or strong correlation with how an assessor who is in charge of assessment program, what are their educational behaviors? So they’re taking a theory and applying it in a different context. And it’s a theory that they have their own unique theory built by their senior author. So I’ll describe the model very briefly, but I’m going to direct the listener to paperspodcast .com where you can find the abstract where we have taken all the important bits and put it there for you. So if you’re like, “hey, I’m having a hard time in my working memory holding all these ideas together.” You can go see it. You can see the figure. You can dig into it. You can spin it around and say, okay, now I see how they’re trying to interpret and what’s the lens through which they’re looking at their data. So this is the model by Cillier and he has a number or she, I’m not sure if it’s he or she, has a number of factors, but I’m to break it down in a very simple sentence.

Your intention to perform an action is shaped by your personal factors, things like your attitude towards the task or your self -efficacy, whether you have the ability or the perception of ability, as well as contextual influences. What are the interpersonal dynamics? Who is part of your team? What is the leadership overseeing what you’re doing? And environmental conditions. What does the system look like? What are the roadblocks? What are the resources available or unavailable to you? And so informing an intention to action does not guarantee that action. The move really depends on your skills and your opportunities or constraints in your context. So best intentions are not sufficient. What they did is they did semi -structured interviews last anywhere from three quarters to an hour with a convenient sampling of three centers. Two of them were public universities in South Africa. And the third was a private nonprofit Mexico university.

And interviews were conducted in the language of choice for the participants. So there wasn’t a gap in terms of understanding and articulation of perception and experience. And they asked them three questions. They’re pretty straightforward. I’ll read them for you. How do assessment in your clerkship? What personal factors influence your assessment practice? What contextual factors influence your assessment practice? Getting back to the model of Cilliers. I think they did. They don’t actually say it out loud.

So I’m going to stretch a bit and squint a little bit to try to make sense of it. I think they did a generic thematic analysis. And then once they completed, they had an independent review of the findings was conducted by the Mexican author so that there could be a comparison in terms of the analysis as completed by the two South African authors. So let me pause there. What do you think of the methods?

Let’s turn off the podcast and continue with our drive or a run. Or are there things that you might want to, in a perfect world, critique and improve slightly? Jason, Linda, Lara.

Jason: So first of all, thank you for that summary, Jon, because for readers, this is not a paper for the light of heart. This is like take a weekend. It’s a long paper. It’s very detailed. actually thought it was very well written. It’s very detailed. The model of health behaviors theory got 25 elements. For an ER doc, my hippocampus was completely offline after that page.

And I couldn’t remember all those moving parts. But your three -part thing, that makes sense. That’s a lot of theories that we have. The individual, the context, and the infrastructure or the environment or whatever. And then everything’s a sub-bullet underneath that. But that was fine. I thought their methods were fine. I didn’t have major problems with them. I actually thought this was pretty well-written.

Linda: So yeah, it was pretty well written, although I didn’t find it that accessible. I was busy trying to get with their, they labeled parts of it using numbers and letters and I got totally confused in there. So as Jason said, not for the faint of heart. A couple of things. First of all, the research question, what factors influence the assessment practice of clerkship conveners? That’s…

not necessarily a question that’s solely for the global south. I mean you could ask the same question of others and I think a nice question might have been a comparator. The second thing is these diverse southern contexts are really maybe not so diverse. It’s two countries. I would not call them low-income. They’re probably middle -income countries. I’m not sure about the South African universities but the Mexican one, Tecnológico, is pretty high end when it comes to teaching and assessment and curriculum. I’m not sure it’s a representative thing of the global South. Methods, I think, were fine. There was an interesting reflexivity statement which really describes where people are coming from.

I’m not sure it really helps with understanding their perspective though on things. And finally, I’ll say, as Jason alluded to, adopting this theory, which is for individuals and their health behaviors into health professions, education, assessor behaviors, I don’t really see that there’s a link. They haven’t made that link for me.

Jonathan: Okay, Laura, before you jump in, think I can be friendly to all those critiques. I would say that this is the whole reason for the podcast. This paper is 28 pages, and what we’re trying to do is summarize it into 30 minutes. And so, hey, dear listener, look at all the work we’ve done for you. We’ve saved you hours of your time. Jason took a weekend to read 28 pages, so if you figure out that, Jason can essentially read one page every three hours.

We’re gonna work on that. We’re gonna get your crayons out. We’re gonna help you a little bit on

Lara: I have two points that I want to make about the methods. One critique and one compliment. So I’m going to start with a critique and it’s not really a critique as much as a consequence because when the authors don’t tell us what methodology they’re using, and as Jon noted, it kind of looks like thematic analysis, but I don’t know which form because remember thematic analysis is a bit of an umbrella concept under which a number of different approaches to thematic analysis can sit. And also looking at the methods, I could imagine that it could also classified as a form of content analysis or maybe a constructivist grounded theory orientation. And the reason that has a consequence, if you don’t tell me what it is and I’m left to figure it out, each one of those methodologies is aiming for a different outcome. And the challenge becomes for me as a reader is I’m trying to figure out which outcome you were aiming for and did your work achieve that goal.

So I agree with you, Jon. I do think they’re doing thematic analysis and the purpose of thematic analysis. Your end point is the generation of themes. And I think they achieve that. So I think it’s fine. But, you know, a word of caution to our listeners, always try to be really explicit and label what you’re doing so that your reader isn’t making decisions and guessing and doing those sorts of things. Always better to direct me than to leave me to just go down a rabbit hole.

Speaking of rabbit holes though, and just to pick up on something that Linda said, I read the statement and there were a few words in there I had never seen before and that led me to a gold mine paper. So the authors had this little phrase and in their reflexivity section it says, “reflexive preparation for data collection took place, blah, blah, blah”, through these different activities. I have never heard of reflexive preparation. So they had a citation, you know, I went and read the citation and I’m so glad I did because the citation was to a paper entitled “Reimagining reflexivity through a critical theoryfFramework”. And the first author is Danica Sims, who’s the first author of this paper. And this citation is all about engaging in reflexivity to recognize the position that you as a researcher have taken when conducting a study. And this paper describes how auto -ethnography, and I know that’s a long word, Jason, but you know, just deal with it. How auto -ethnography narratives can be harnessed to help the researcher think about their personal position of power. So your identity and how your positions of privilege are part of that intersectional set of identities. And also about your epistemological position of power. So the way you do research, the way you think about knowledge that reflects assumptions and ideologies. So for personal reflexivity, this paper, this one that they cited, suggests using theories of intersectionality and for epistemological reflexivity. And it uses deep coloniality theories and southern theories. Anyway, all this to say, friends, yes, the paper they cited is dense. So if you thought this one was thick, the other one’s thicker. But please, take the time, take the weekend, one day to pull it out if the only thing you do is pull out table one from that additional paper and you pin that on your electronic desk because that table has incredibly useful questions to help you write your reflexivity section of your paper. So for me now, I have table one from this paper from Sims and I have the Olmos-Vega guide on reflexivity as my two go -tos.

And so I really do want to encourage you to go to the article and dig it up because it’s gorge and so relevant for this study that’s taking a critical view on assessment, examining how it’s different in the global South. Total gush worthy. I can’t say enough great things about it. Congratulations to Sims for writing it. Just thank you.

[chicken sound]

Jason: I just want to introduce a new feature to the podcast.

Lara:
What the flying flip was that?

Jason: This was suggested to me from our listener, Ben Kinnear, friend of the podcast,

Lara: He gave you a duck?

Jason: A chicken. This is the multi -syllabic chicken, and it’s kind of an alarm that goes off when Lara uses too many big words in a single paragraph. You’re allowed big words, you can spread them around, but too many big words in a single paragraph, you get a multi -syllabic chicken alarm. Thank you, Ben.

Jonathan: So we have the two ends of the spectrum. have Uber nerd and we have four year old boy. So welcome back to the podcast.

[chicken sound]

Lara: Because you know, it resonates, doesn’t it? I don’t mind being an uber nerd, unless I’m the four year old boy, but anyway.

Jonathan: Let’s, let’s jump into the results. They interviewed 31 Kirk ship conveners in a different context. That’d be the Kirk ship rotation director.

Jonathan: And although they use a convenient sampling, there was a diversity of gender, career stage, clinical discipline, and social and cultural context. In brief, I’m to summarize it and I’m let all of you tackle a portion of the results. Here’s my big summary.

An intention to act is informed by personal contextual factors. If the intention is negative as a function of impaired personal or contextual factors, then inaction happens. The opposite is true if a positive intention is adopted by the assessor. But these interplay of factors between the environment and the individual are very contextual and very complex. So I’ll give you just a highlight of big level factors and let you dig in after this. What’s your personal attitude towards the assessment practice? What are the consequences or the impact on you or your students, whether you perform or do not perform the assessment? What’s the opportunity cost? How much work is going to be required or not?

And then what’s your self -efficacy? Do you feel prepared or able to do this? So that’s the personal factors. The contextual factors are interpersonal. Do you have strong leadership that’s inspiring what you can do or inhibiting it? What’s the organizational environment? Are there resources or rules or policies that promote or inhibit? What’s your training in it? Have you been prepared or adequately developed as a faculty member to complete what is expected of you in this role? What’s the outlying assessment expertise? Can you draw on others to help inform or change or modify or improve your practice? And what’s the socio cultural influence? What’s the historical context? What is the expectation from the health sciences community, from the larger community on terms of what should be expected? So what say you, Linda, Lara, Jason.

Linda: So I was looking at the mediators of assessment practices and thinking about them and thinking if they were the same or different than the kind of mediators we have. And they talk about personal competency, whether the assessors, these individual assessors felt that they actually were competent to assess, and also the environmental constraints, which is really context, which may come up on a, which will come up in another paper that’s coming up soon on Papers Podcast. And these environmental constraints include things that we’re all familiar with, clinical workload, education workload, class size, that’s maybe something that’s more significant in an LMIC, where class sizes, I suspect, tend to be quite a bit larger. Help.

HR help, so we say human resources help from assessors, but also from patients. And the big one, of course, is financial. Are these different from what we face? No. Are they different in relative amounts? Yes, I would say.

Lara: I want to highlight one of the contextual factors, the identity, the authors identified, and it has to do with the influence of social, historical, economic and political forces that the individual evaluator feels. And this is an area of particular like this is a topic, a way of thinking that’s personally important. And I’m just going to highlight the data that came out of the South Africa context today for the sake of time. And the paper knows that South Africa’s post-colonial and post-apartheid context imposed a pressure on these assessors. And there was a, and I’m going to quote here, “a perceived pressure to pass to maintain a throughput of historically excluded students, despite personal assessment beliefs as pass rates were linked to public institutions receiving funding subsidies from the government” end quotation. knowing that and then getting the data, the excerpts from the participants, you really start to hear how a country’s history and their political ideas and social factors are landing on the shoulders of individuals in individual moments trying to do something. Just a little bit, I want to share just a few excerpts. One participant says, if they fail a significant portion of students, they, institutions, are financially embarrassed.

Another participant talked about passing students to deal with a growing student population. Again, quote, “you can’t fail 20 students. Only 10 are allowed to fail or no more than five are allowed to fail. And it’s not about color or the language of the students in this case. It is about volume. What do you do when you suddenly have minus 20 or plus 20 in the next year? That political pressure is greater than the social political pressures of color and language.” if you, if you listen to these words of these participants, you can hear this history of all kinds of economic and social and cultural pressures on their shoulders, having to make these decisions about protecting a whole bunch of different things. How do you navigate a path that respects all these contextual considerations and do what you feel is the best thing for everybody involved in the moment? Talk about needing some serious policies, right? Like these are pressures and responsibilities that that assessor is feeling. So I really appreciate this paper because it demands that we look broadly when we think about specific actions that individuals take, it is really easy to say that we need immediate guidance for assessors because they do. Because these kinds of pressures, these social political pressures that land on the assessors, we can’t affect change at the individual level. One person is not enough. We need lots of people doing lots of choices so that we protect patient safety, we protect the profession.

We do the work of uplift. We do the work of social change, but asking one person to bear the brunt of all of that is just a lot.

Jason: I agree that I think that’s what makes this paper actually delicious to read It I’m really I do gripe about this my clinician educator lands on but how dense this paper is But it is so rich it is filled with lots and lots of quotes So what I’ll just highlight is how relatable I found elements of this educational ecosystem Are to the context I live in so at some point not too long ago in the Canadian system I live in.

There were individual assessors making individual rotation decisions that had enormous, magnified impact across people’s careers. And they did it with very little training, support, policy, infrastructure. And, that relates to this paper, the things that these people are dealing with still resonate today in almost any setting. And many of them resonate with a setting that existed when I was a medical student.

I could always live in this paper through some of the quotes. So here’s an example. Somebody saying, you know, that like this is you have to make an assessment tool that’s really light. Otherwise you give up part of your life to, to Mark all everything. It’s too elaborate. Somebody else said it was griping about their colleagues who just everybody that they rate is born a 10. So it’s not discriminating. It’s not useful to this poor clerkship director who’s trying to, have a useful assessment system. So there’s lots of really, really yummy elements in here.

I encourage you to it, actually go through it, because you’ll find so many things that you could use for your future assessment papers.

Jonathan: I wish we’d had this conversation before I’d come up with a title.

Linda: Could I just say I agree with Jason and with Lara, but you probably can ignore the front part of the paper where they do this lit review which takes up a good half of the paper. I think that’s what overwhelmed me, found, I started to get sort of negative feelings because I didn’t find that part accessible.

Lara: Yeah, if you’re looking for a tidbit for practice, I think you have a good point there. But Linda, the Cilliers theory, Francois Cilliers theory, and the framework and the way it’s mapped out, like from my perspective, I loved it. I was there every step of the way. In fact, I’m thinking, okay, people I know who work in assessment, those sorts of things, maybe this is a take home. Maybe for them, the figure in there that breaks it down, maybe that’s a thing.

Jason: Okay, so it’s a PhD thesis. Jon, what’s a better title?

Jonathan: Well, I was struck with the metaphor of the weight on the shoulder of the individual, which is the counterfactual to what a programmatic assessment is trying to do is saying, can we distribute that weight in a different way? And so I think that’s a better metaphor than this idea of I need to assess or I’m in distress.

Lara: And whose metaphor was that? Just question, huh?

[inaudible]

Jonathan: Well, every now and then even a blind squirrel gets a nut. So we’ll go with that. All right. Let’s go to our conclusion from the authors. Let them the last word and then we’ll go to our round of votes. Personal competencies and conducive environments support intention to action. While previous research has typically explored factors in isolation, human behavior theory framing enabled a systematic and coherent account of assessor behavior. These findings added particular contextual perspectives to understanding assessment practice.

It also resonate with and extend existing work that predominantly emanates from high income context in the global north. These findings foundation for the planning of assessment change initiatives. I do have a small little paper clip, it gets back to the debate that Linda and Lara having, which is this is a great example of where there is not a word count. And so advances, thank you, Rachel, Az, thank you, Jeff, doesn’t have a word count for authors. Now they require you to be very parsimonious and to have clarity in your writing. And I think the authors really did a good work here. I think if you are typical, or if you are more practiced in reading the medical education literature, you might find that the long introduction is not typical of what you see. But if you read social sciences and humanities, a big introduction that dumps all of the literature in upfront is much more typical with a very short discussion. I think that’s kind of what we see.

Jonathan: Let’s go to our round of votes. One to five in terms of methods. We’ll go Linda, Jason, Lara.

Linda: So I’m going to give this a three plus. The plus is for, as Lara would say, we have the paradigm upfront. So it gets an extra point for that. It’s reasonably good. As you mentioned, we had to struggle to figure out what kind of analysis was done. So three is middle of the road, solid average.

Jonathan: I don’t know; this is a tour de force paper. it’s missing some parts, like we just alluded to. We assume it did thematic analysis. And it’s a bit dense. I actually think it’s a four. It’s an amazing paper.

Lara: So I’m giving it a four as well. The fact that they don’t signal explicitly their methodology, that really is a problem for me, but they signal their paradigmatic orientation and like the goldmine paper they shared. So yeah, four.

Jonathan: They did a ton of work. The only critique is it’s not clear what their analysis is, what the guidelines are, what are the boundaries for their analysis, so it gets a four from me.

Now let’s do reverse order in terms of educational impact, one to five.

Lara: So I’m going to give this paper a five. I think the findings are so important and so vital for us to think about it. So yeah, it’s a five.

Jason: I can’t give it a five because of the accessibility issue, because it’s not going to, not enough people are going to pick up this paper, but I am going to endorse that more people should. If you write about assessment, I think that this is a cool paper to pull quotes from and talk about themes, talk about how there are parallels and what the implications are, despite the differences in setting and ecosystem. So I’m to give it a four.

Linda: I too am going to give it a four, similar reasons to Jason. The results are useful and I can see people who are trying to develop a programmatic assessment system and haven’t quite got there using it to make the life of those solo assessors a lot easier by knowing what’s going on here.

Jonathan: I’m going to give this a five. Complex problems lead to complex results. I think too long in the assessment game, we focused on the tool. I’m going to make my scale better. I’m going to make my tool better. Or we think about the system. I’m going to make my dashboard better. Or I’m going to make my incentives better. And we forget that really it comes down to human judgment on some kind of piece of data. And this paper unpacks it in ways that are really broad. And although they situate it in an LMIC, I think the takes on it really can transfer to the global north. So I think that’s a five. So there you have it. Listeners, we’d love to hear from you. What’d you think of the paper? Is this something you’re in line with? Are you intimidated by 28 pages or can you read faster than one page every three hours? We’d love to hear from you at our website where you can download the abstract. Everything is compartmentalized and abbreviated for you at Papers Podcast.com or you can send us an email at thepaperspodcast@gmail.com. Thanks for listening.

Lara: Talk to you later.

Linda: Bye bye.

Lara: Jason, say take care everybody. I’ll say it for him. Take care, everybody.

Jason: Take care everybody.

Lara: I sound so much like you!

Jonathan: I like Lara’s better.

Jason: True. That was good. That’s good.

All right. I also want to say a big thanks to our friend Merchior who is in Mexico City. He’s a giant of medical education in Mexico and I saw him at AMEE and he had all of these anecdotes from papers. He didn’t just listen to us he’s got little quotes. Interestingly, they had quotes from Lara, quotes from Linda, quotes from me, and then he stopped there. He’s like, yeah, lots of good quotes.

Jonathan: But he’s a giant, and he and I think very similarly. And so he’s learning from you, but he and I are simpatico and see the world in very similar ways, because we’re both giants.

[inaudible]

Jonathan: Honestly, it was it was all from the three of you. He didn’t mention me at all I don’t much even knows I’m on it actually

Jason: you’ve been listening to the Papers Podcast, we hope we made you just slightly smarter. The podcast is a production of the Unit for Teaching and Learning at the Karolinska Institutet. The executive producer today was my friend, Teresa Sörö. The technical producer today was Samuel Lundberg. You can learn more about the Papers Podcast and contact us at www .thepaperspodcast .com. Thank you for listening everybody and thank you for all you do. Take care.