16 Nov This week you will begin the Method section of your Program Evaluation Report; see page 221 for more information on this section. At this point, you should b

4

Week Three Program

Name

Institution

Course

Tutor

Date

            Various research methods could help assess the capabilities of Mental Health America programs aimed at reducing stigma and ensuring more people have access to mental health support. A mixed methods strategy can be suitable based on the complicated nature of mental health solutions and the necessity to capture quantitative and qualitative results.

            A quasi-experimental design would be helpful for a program on reduction of stigma. This design will enable a comparison between places where MHA implements such projects and places where it does not, recognizing that randomization might not be possible in real life. Attitudes regarding mental illness could also be recorded before and after the intervention with the help of surveys and interviews (Fink, 2015). Those data would offer a mix of quantitative and qualitative findings on the change process. Such an approach is consistent with the evaluation question: "How effectively does MHA employ the strategies in reducing the public's stigma against persons with mental illness and discrimination against them?"

            The program focused on improving the accessibility of mental healthcare requires the employment of the longitudinal study method. Many of the changes to the access algorithms would be assessed while measuring instant and long-term effects associated with the MHA’s solutions (Torjesen, 2022). The design would help incorporate regular data collection points to help monitor changes in the utilization of the services, waiting times, and demographic reach. Such a strategy helps address the assessment question regarding the MHA’s contribution to the improvement of access for hard-to-reach populations.

            Stratified random sampling is the most functional technique in terms of sampling procedures. It helps ensure a balanced representation across all social strata, weighting more even marginalized ones, which has become the primary objective of the MHA (Fink, 2015). It could include stratification based on factors like age, income, location, and previous contact with mental health facilities to ensure the inclusion of all target population groups. The method helps ensure that the collected data represents different communities served by the MHA while helping maintain statistical validity.

            For instance, in the evaluation of the stigma reduction program. The population is put under strata such as community features like urban/rural, socioeconomic status, and any previous exposure to mental health. It helps ensure the evaluation demonstrates the program's effects across the different community setups (Fink, 2015). Equally, in the case of a healthcare access program, stratifying groups could invite various geographical distances from healthcare services, having insurance or not, and how these factors vary with individuals to access care.

            There are several benefits to stratified random sampling within this research. Most importantly, it offers accurate and representative data regarding how diverse subgroups respond to MHA’s interventions, permitting more targeted program improvements. It is aligned with evidence-based strategy and supports the organizational objective of serving different populations successfully while at the same time ensuring the maintenance of ethical standards of assessment.

References

Fink, A. (2015). Evaluation fundamentals: Insights into program effectiveness, quality, and value (3rd ed.). Thousand Oaks, CA: Sage.

Torjesen, I. (2022). Access to community mental health services continues to deteriorate, survey finds. BMJ: British Medical Journal (Online), 379, o2585. https://doi.org/10.1136/bmj.o2585

,

And I can confirm and we can jump right in. I'm happy to say that this week's work is going to be among the most brief of the semester, and the actual assignment may be the shortest graduate school assignment you have. I think it's about two to four sentences long. That is the good news for this week. I am going to prep you, as I think I mentioned last week that Week five is a lot of work. Unfortunately, that falls right during Thanksgiving week. Just brace yourself. Understand that's what's coming that week. It's our data week, and it's going to be a lot of work. It's a heavy lift. Hopefully this week we'll give you a little reprieve so you can be ready to go in Weeks four and five. Okay. Let's get started. Here we are in Week three. We're going to start by talking about the discussion first. If this lets me move my page, why won't it there we go. We're going to talk this week about the method section of your report. So similar to how we've done it each week, you're going to think through in your discussion what the questions are around methodology. And then in the assignment, you'll write what you would write in report to the agency Board of Directors, to the agency, just a typical report. It's going to look a lot like a method section would for an APA paper, except, again, we have very little that we need to write. But it's going to have a centered, bolded methods, then you're going to move to the left, and you're going to I'm sorry, left justify the next sections of the methods portion of your report. What else did I want to say about that? This is going to be the work that's done at the agency. Whereas the introduction to the report was a lot about evidence based research and what kind of things agencies like this have been able to do, and so on. Now we're talking specifically at the agency. What would you do with your evaluation? How would you set out to look at the program and its particular results? Not broadly what evidence based research has found before, but what you're going to look at at this agency in this particular evaluation. Okay. When it comes to the discussion, we're going to talk both about the methods and the sampling. I know you haven't probably read Chapter three yet. It's Monday night. We jump in. Hopefully some of you did it this weekend, but I can't expect that every week. We're going to jump in and talk about the material, even though you may not have done that chapter yet. I want to start with the sampling. You'll see that the discussion, the first little section says, consider the various research designs presented in Chapter three, given the goals and objectives of your social services agency identified in week two. Discuss possible research designs that could help that could be used to evaluate outcomes at your agency. What would be the best design to answer your evaluation questions? Then I ask, what would be the best sampling method for your population? I want to start with that sampling question. Because it should be pretty straightforward, given the size of most agencies you're working with, you should be able to sample the entire population. That is to say, you would sample those people who are intake. I know that's a strange word, but we use it in clinical and evaluation work. Those who experience intake by a certain date and before a certain end date. Everyone who's taken in between let's say January 2025 and June 2025, that would be your population, but it would probably also be your sample. You're not going to engage in random sampling, like you put in a research design project. You're not going to say these folks get this treatment and these folks get this treatment, and we have this control group and so on. There may be a natural control group, but there's not one that you're going to create. When it comes to sampling, you just probably need to point out that you're going to look at the entirety of the sample, I'm sorry, the entirety of the population as your sample. If they're taken in, if they go through the intake process, if they're intake between the dates that you set up. It's pretty rare beyond that sampling to do anything different. If it's a big, big agency, you might want to shorten your time frame and say, anybody who's intake in January of 2025, that's pretty small just one month, but maybe January 1, 2025 to March 30, 2025. So you might have to reduce the time period that you accept each of your sample members because it will be the entirety of the population during the tide that you set. So it's a very big agency, shorten it to less than six months. If it's a typical sized agency, anybody intake during that six month period would be good. Again, no random assignment. That isn't going to occur. If you are using a sample that you're not looking at treatment outcomes, but you're looking at something like housing outcomes, and you have a homeless population, for example, you may have wildly different intakes in the winter than in the summer months or vice versa if you're in a very hot climate. Being in the Illinois region and near Chicago. We have a great deal more intakes during the winter for a homeless population, an unhoused population than we do in the summer. So you might have to take that into account, think through, should you do two or three months in the winter and two or three months in the summer, so you get a full sampling. But again, you're going to use everybody who comes in during the time frame that you set. Let's see what else I wanted to say. The main thing you're going to look at then is going to be the intake data versus the post treatment data, right? So most of you are working with organizations where you're looking for decreases in symptoms or increases in quality of life or decrease in pain or substance abuse. So you're going to do an intake test, and then at the end of the period you set a post treatment test. So now we get into the issue of design. Most of you are looking at a pre post test design, a pre post test design. Minimum, most of you are going to have what's called a self controlled pre post evaluation, a self controlled pre post evaluation. That's the minimum design. That's what's typical for this class, given this eight week period, given what agencies you're typically looking at, and so on. You can do more. If, for example, you can look at between group findings. Perhaps you work for an agency or you're working for an agency where some of the clients come in and get cognitive behavioral therapy, and other clients come in and get dialectical behavioral therapy, or maybe motivational interviewing MI. You could look between those groups, how they do it pre and post test. But for most of you, the agencies have the same treatment for everybody who comes into it. It might be a multi modal treatment where everybody gets cognitive behavioral therapy, and everybody gets family therapy, and everybody gets some stress reduction work. Then you would say, how does everybody do pretreatment and post treatment at intake and at post treatment. But if you have within the agency the opportunity to look between groups, those who are put into cognitive therapy, and those are put into MI or something like that. You could do it. You might also have in a situation, say when you're working with foster children, Some family therapy that's offered and individual therapy that's offered. Many foster children will unfortunately not have family who can or will engage in the family therapy. Then they only get the individual therapy. You could look between groups at those who get the combination of family and individual therapy versus those who just get the individual therapy. But the key here is you don't set up the groups. The agency is already doing the work They've designed as clinicians as therapists. We are the evaluation team. We're going in to say, how's it going? No, here's what you should do. You go in and you look at the naturally occurring population, the naturally occurring groups and the design you're going to employ, and you work from there. You don't create a control group. You don't do what we would do in typical say, dissertation style research or academic style res program evaluation, naturally occurring and works from there. There's one potential for a control group that might come up. That is an agency that has a wait list. If you have a wait list, you have a control group of people who have the same symptoms as the population. Maybe they need mental health reduction, symptom reduction, maybe they need substance abuse, symptom reduction, et cetera. Because they're on a wait list, they don't get treatment. And so you could compare those who go on the wait list in that six months with those who get put in treatment in those six months, and you'd be comparing a treatment group to a control group. To big butts, two big caveats. You may not create the wait list, that would be unethical. You may not say, these people will get treatment and these people won't. It's only if there's a naturally occurring already in place wait list that you could have a control experiment design. The other thing is, if somebody gets off the wait list during the period you're studying the agency, you must let them get treatment. They must be able to leave the control group and become part of the experimental group. You can't hold anybody back from getting a treatment, that would be unethical. Again, most of you are going to have as a design the self controlled pre post evaluation. Some of you, in rare circumstances, might have a weightless situation where you'd have a control experimental between group comparison, or you'd have naturally occurring between group comparisons between say those who get cognitive therapy and those who get dialectical therapy or MI or something like that. Again, you cannot randomly select. You cannot say you get the cognitive treatment, you get the dialectical, et cetera. Note, whatever the agency would naturally do according to their standards and intake and treatment system, you simply evaluated. You simply evaluated. Does anyone have any questions about design or sampling? And it's okay if there's no questions. I did note that this was a pretty straightforward week, but I don't always want to put one student on the spot, but this is a good time for questions. You can always e mail me if you have a question once you get started and I can respond to you before you're writing your discussion and so forth. So we're building our method section. Correct. Based off of what the agency is currently doing, and you're reading in Chapter three. Okay. Uh. That's exactly right. You're building your method section off what your agency can offer you, what they're already doing, and the designs that are presented in Chapter three, in all likelihood, your design will be this self controlled pre post evaluation. And in all likelihood, your population will be the entire your sample will be your entire population. Okay. Okay. Thank you. Sure. After you've worked this out in your discussion, after you've thought this through, you're really only going to do a very brief assignment. This week, you'll begin the method section of your program evaluation report. See Page 221 in Chapter nine for more information on this section. At this point, you should be able to complete those subsections that address the procedures of the design, including what's called the program definitions and design. The rest of the subsections of the method section will be completed in future weeks. Once we've talked about instruments next week and data in week five. Again, if you're just tuning in or you didn't hear me, Week five will be a lot of work, right smack dab in the middle of the holiday week. It's just how the course is designed, and it couldn't be avoided. So you'll notice that the subsections program definitions and design include program. And we already had program in the introduction section of our paper, but they mean two different things. So in the program section of your introduction, you were sort of talking about what treatment people were given at the agency are given at the agency. So you're really talking about what the treatment was, what the program was at your agency. Now you're going to be talking about what the program that you're looking at can do effectively. The program the agency again, and it's looking at whether you have been able to establish the effectiveness of what they're doing or not. So for this assignment, you will just be presenting about two to four sentences. It will be a new section of your report. Much like in APA style, you'll center the word method You'll bold it, and then you'll write justify and use the words the program as a subheading. Essentially, you're answering the question, what will the program evaluation do? What am I trying to do with my efforts here as a program evaluator? I'm going to give you the one sentence and you have to fill in some spaces with what your particular agency and program is doing. The sentence is, this program evaluation will measure whether this program significantly reduces fill in the blank? This program evaluation will measure whether this program significantly reduces mental health symptoms, substance abuse, stress symptoms, PTSD, whatever you might be dealing with in terms of your program. Next to the words, the program as a subheading, you'll say this program evaluation, we measure whether this program significantly reduces and then whatever you're studying, whatever you're studying, mental health symptoms, PTSD, substance abuse, whatever that might be. Now, the next section, definitions, is one that most of you can skip. You really won't need it in the case of the things that we're looking at. You've already described in your introduction the type of therapies that are being used. So you don't have to explain what cognitive therapy is or what MI is. You've already explained that in your introduction. If you haven't, go back and look at your introduction and improve it in that way. But really, this is for when we're talking about a variety of medications, what kind of medications do what work with which neurotransmitters, and so on. It's really not a section we need in this course for the type of program evaluations we do. So Skip that. Finally, you get to the design, which you're already going to have worked out in your discussion this week. This will be one or two sentences. If you do a really long sentence, it'll read something like this. After the word design, which will be a subheading, you'll say this program evaluation will incorporate a self contained pre design To determine if significant results are realized between intake and post treatment. Sorry, I've got a little bit of a snuffling DOG here. Hopefully, he's not going to be ARK, yes, I cannot say those words. I have to spell those words. I apologize. We're going to see. I'm going to repeat that sentence a couple of times. Of course, you can always review this recording. But for design after that subheading design under the bigger method section, this program evaluation will in a self contained pre post design to determine If significant results are realized between intake and post treatment. Essentially, between intake and post treatment, you found significant results. If you don't like the word realized, you can say results are between intake and post treatment. If you are going to do a between subjects question as well. Remember we talked about the fact that you might look at if it naturally occurred in your agency, the difference at pre and post treatment between those who had cognitive therapy, MI therapy, dialectical therapy, et cetera. Then you need another sentence. Right? So you need to talk about the between group finding you expect to find. Again, one sentence that states that you expect a significant difference between people who get cognitive therapy and MI therapy, or whatever you are going to look at. You only need to add that second sentence if you're doing a second question and you don't have to do a second question for this class. I just want to be clear, that's what would be followed up with there in the design section if you were doing a between groups exploration as well. Now, remember, do not turn in your introduction again. Whatever improvements I made, I commented on for you to make last week. You should make and start to create that in a file that's going to be glued together in this seventh week. So you're going to take your introduction that I've given you feedback on, improve it. And then after you get feedback on this section, you'll attach that to your intro and so on. But you don't return in the introductory section. We're going to really just create that entire document in Week seven from all the feedback hopefully you've gotten throughout the course. That's really all that I wanted to say about this week. Oh, I had, I'm sorry, additional slides, but they just talked about how last week we did the program, I'm sorry, the problem, the program and the evaluation in the introductory section. I was going to mention that when I was going to talk about the fact that we again bring up the program, but in a little bit different context. So it's not the program or agency or service that you're going to evaluate. But instead in the method section, the program is whether that treatment has been successfully differentiated in its outcomes between pre and post test. Again, here I have the method section. You're really only going to need the program and design. You will skip the definitions section. Okay. Any questions? Actually, we have do you have a in in our book. Do we have like the whole I I need, like a visual of like the whole template, just to Have you looked at Chapter nine yet? Have you looked at Chapter nine yet? For some reason, as when I turn the pages or even look I have both the hard copy and and the online version. You have both versions, and you don't see anything that is like a def of what you need? I'm probably confusing myself. Well, look back at these two slides, which I went through too quickly, unfortunately. Okay. This will sort of tell you in the introductory section, you'll have the name of the agency, a brief description of the agency. You'll have the name of the program to be evaluated. Okay. And then those things we went through last week, the problem, the program, and the evaluation, which I gave you feedback on through the rubric and some comments this week. Right. Then you'd go into the method section, which you'll have the program, excuse me, and design. We won't have a definition section. So as far as a template, this will at least give you an outline of what to expect. Okay. Okay. I know. It's a big project. It's a big project. I'm glad we have a lighter week. Yeah. It just kind of breaks it down for me. I appreciate it. Yeah. Yeah. So I'm sorry didn't go through these more slowly, but you can come back to the recording and we're at about 26 minutes in. And at 26 minutes in, you would see the introductory section review. And then you would see, Oh, it keeps jumping for me. My goodness. Then the method section. Yes. Okay. All right. The whole thing. Okay, great. Yeah. This is at least a short week. I know the next few weeks will probably go the full hour. Okay. Thank you so much. I appreciate it. Okay, Linda. Have a good night. You too. Thank you. Bye bye. Bye bye.