20 Nov The selection of measures is an incredibly important aspect of program evaluation.? Discuss possible measures that could be used in your program evaluat

Okay. Welcome to Week four in Program Evaluation. I'm actually restarting a recording, which you won't know because there wasn't anyone in attendance tonight, but I thought I'd mention it because everybody has scaffos, and I didn't like the way the recording was going. I even had my first interruption by my dog this semester. I decided to start fresh and go through the material again. Again, that isn't something that is all that relevant, but I thought that I'd mention this is maybe my second time through some of this material. Again, we're in Week four. This is going to be a very busy week with assignments, as I've promised next week, Week five, will be very heavy lifting work. We'll be using SPSS. Some of you will be using other programs you've talked to me about. We'll be entering our made up data and getting into assessment of it. But for this week, we're going to be continuing on with our methods section after doing a discussion. So we got started with the method section last week with a couple of the subsections within that, and now we're going to go through a couple of more, a few more actually. But as always, let's start with our discussion this week. Essentially, our work is our assignments. O assignments are our work. So we just go through that each week. For the discussion this week, you're going to be involved in selecting your measure for this program evaluation. Again, we're only selecting one measure. Some of you have gone as far as two measures, but in this eight week class, we're really completing the whole report between weeks two and seven. There's really no way to go too much further than that. The selection of measures or measure is an incredibly important aspect of program evaluation. So states our discussion. In your discussion, you should discuss possible measures that could be used in your program evaluation at your agency, including the pros and cons of the possible options as they relate to the following issues. Um The first thing we really want to get at here is how we want to assess the variable we're most interested in in our program evaluation. By going through the questions that are listed here and really evaluating the pros and cons of each measure you consider, I think you'll be able to narrow it down pretty quickly to the one or possibly two measures you want to use. Again, they're going to be pros and cons here. There's no perfect measure out there. That's why we always run multiple studies with multiple measures when we're doing academic research. In the real world, we're sometimes constrained with program evaluation. In this case, as I've already mentioned, we're really constrained in our quasi program evaluation or made up evaluation this semester. Some of the things to think about when you're thinking about the pros and cons of a measure, who are the participants that will be evaluated? Are you looking, for example, at children, at adolescents, at adults? Are you looking at people who have mental health symptoms, at people who have drug abuse or use symptoms, et cetera. You really need to pick a measure that's reliable and validated for the population, the sample you're looking at? In particular, again, you want to make sure that your instrument is reliable and valid for children if you're looking at children in your program, adolescents, if you're looking at adolescents in your program, and adults if you're looking at adults. The other thing to bear in mind when selecting a measure is who are the staff who are going to be administering the evaluations. We ourselves are not the researchers here. We don't go in and do this. We advise and then we deal with the data. But the folks who are going to administer and collect the evaluations are the people who work at the agency. So upon picking a measure, we have to be sure that it's appropriate to be used by the folks who will be administering it. So for example, make sure if there's a licensing requirement for the measure you select, that the people at your agency would be licensed to do that. Sometimes it can take three to four days or longer in training and then getting some supervision before you can use it. And a lot of people can't allow that time for agency employees to go get the training. So ideally you would pick a measure that doesn't have any licensing requirements or anything particular in terms of training for the agency folks to administer the evaluation. Another thing to keep in mind here is that self report measures are probably going to be your friend. You won't need a clinician for a self report measure, and a lot of our agencies only hire a few or maybe no clinicians at all. Self report is going to be really the direction you head with most of these measures. Yeah. Another thing that I made a note of was, if you use more than one agency representative to evaluate or administer rather these evaluations, you would then have to get an inter reader reliability. Again, we don't want to go very far with that in this class. You do want to have a sense of how many participants will be evaluated because price of the instrument of the measure is always very critical, and so bear that in mind as well. If you're going to be giving the evaluation, the instrument at both intake and discharge, now you have double the cost if there's a big cost involved. Bear in mind that knowing how many folks are going to be involved is very important, how many participants will be evaluated because that will affect the price of your instrument if there is one attached to your instrument. You also want to think about the assessment schedule. In other words, how often and when the participants will be evaluated. That's already what I was hinting at in the previous statement. That will impact how many of the instruments you actually need, and that again, might come back to pricing. Numbers five and six are very, very important and where you're probably mostly going to bring in your sources. First, when you mention your measures, in the discussion, you'll need to give a source from where you learned about these potential measures. But now when you describe their reliability and validity, you very much need to provide sourcing citations. In the discussion and later in the report. Sections we'll talk about for your assignment this week. You really need to show the evidence, show the citations for the instruments validity, and for the instruments reliability. A couple of times now I've mentioned price already. Mainly when thinking about who the participants would be, how many there would be, who would be administering it and how often. But do think about the cost of the instruments. Can your agency afford to pay for the instruments? Are there alternatives in the public domain that would work? Google is really going to be important this week. You need to take a look out there for the various measures, particularly that are in the public domain, for the variables that you want to assess. Evaluate alternatives if you're looking at something that's quite costly. Now we'll move into the assignment for the week, and essentially, you're going to be continuing the method section that you started last week, the method section that you just started last week. Just as a reminder, before we get into this assignment, here's where we've been with this report and where we're going this week. You won't be turning in these prior sections this week, I'm grading them week by week, and then in week seven, you'll glue them together in a final report. But two weeks ago and the one that you've already gotten feedback then on this week. You were discussing the introductory section of the report, and that would have included the name of the section, I'm sorry, the name of the agency, a brief description of the agency, the problem, the program, and the evaluation. And then last week, which you would have just turned in by yesterday, so I haven't graded them yet, but I'll get to them later in the week. You would have turned in the subsections of a new section called method, that would have included the program and the design. Most of you were to skip the definition section, and last week was a very brief assignment. Just a couple sentences long, especially because you probably did skip the definition section. Now where we're going this week, Is continuing, me, jumped ahead there, the method section with three new subsections, the setting, the participants and the outcomes and measures. All of this you will have thought through in the discussion already. Now you'll write the formal report for the setting, the participants, and the outcomes and measures. To look back at this in terms of how it's presented in the assignment, It says, based on the discussion of the pros and cons of various measures in the discussion thread this week, make a final decision on which measures should be used in your program evaluation. This week, you will be continuing the method section of your program evaluation report. Please again, refer to page 222 and complete the following subsections of the method section. Let's start with setting. First of all, you're going to flush left, as your other subsections have been under the method section, which was centered in bolted. Flush left, put the word setting in bold, and then continue on. Essentially, you're going to ask yourself, where at the agency is this assessment going to take place? You've already described the agency in the introduction. Again, here, you're just going to have one sentence describing where in the agency, you'll be conducting the evaluation. The sentence will read This evaluation will be conducted at fill in the blank for your program for your agency. For example, this evaluation will be conducted at the Adult outpatient building. Period. Again, you might have to make this up because you aren't actually interviewing in every case someone from the agency, you're gathering this information from the website. Maybe you would say, this evaluation will be conducted at the agency's main offices or at the agency's library or at the agency's conference center. There could be a variety of words you fill in at that end of the sentence, but give a sense of where this is. Again, you may have to make it up, but I want you to have a completed report, so you have a template at the end of this class for any future program evaluations you might engage in. Next will come participants. Again, flush left, bold the word participants, and also include the assessment schedule in this section. The participants may be everyone who gets admitted to the program between certain dates, and you should specify those dates. Again, the assessment schedule in this section. One or two sentences might suffice here. For example, all clients who are admitted to the program from June 1, 2025, to June 30, 2025, will be evaluated, right? I meant to say January 1, 2025 to June 30, 2025. If I didn't say that, that's what I meant to say. Let me repeat. All clients who are admitted to the program from January 1, 2025 to June th, 2025, will be assessed or will be evaluated. They will be assessed at intake and discharge or whatever other assessment schedule you decide is relevant. Okay. And now we get to this very important section, which will be longer than the first two. Again, your settings, just one sentence, your participants, just one or two sentences. Now we get to the outcomes and measures portion of your report. Be sure to include research evidence of the reliability and validity of the measure or measures you selected. This will already be in your discussion. Now you want to get it formally into your report for this assignment portion of what you should get done this week. Essentially, you want to make sure that you cite the validity and reliability sources you have for this measure, that you use APA style for those citations, and if you want your reference section that you include this week, could be a running document or page that includes last week's citations as well as this week's citations. So I know I don't want you to repeatedly turn in sections. But if you want your reference section to just be a running document, so you don't have to in week seven suddenly be alphabetizing things and so on. That'll be fine with me. I might make the occasional comment. I don't see this citation referenced in your paper, but I think I'll remember that it's because you're keeping a running reference page. So that will be fine. Otherwise, don't turn in anything you've previously turned in. Not this week. In Week seven, you'll turn in everything as one Amalgam report. Now, when it comes to outcomes and measurements, measures, you need to include the outcome and measure that you finally focused on that you decided after the pros and cons, did the best job of focusing on your symptoms that you're interested in. It might be mental health symptoms. It might be substance. It might be eating disorders. The symptoms associated with reading disorders. So here you put in the instruments and the reliability and validity to justify your choice of the instrument. You might sort of go over what you've thought about in the past in terms of what the problem is and so forth. But this week, you really want to focus on that outcomes and measurements. This could be a few paragraphs, which we haven't done yet in this report outside of the introduction to the agency. So this really could take time because you need to make sure you provide the evidence for the validity and reliability in the same population you're going to be using that you're going to be sampling, even though our sample, as we've already discussed in the report, probably be the entirety of the population. But that's very important. You want to make sure, for example, that you don't pull a measure that's only been validated in China, for example, when you're using a North American or US data population here, excuse me. Make sure that it's validated on folks from America, people of the age group that you mean to, both genders, and so on. This is normally where I would leave room for questions about specific projects. I'm going to throw out some ideas that I was thinking about and I have come up with former students before things that you could Google and look into. The main thing here is Google Google Google, public domain and making sure you match your population and making sure you have a reliable and valid source. Couple of things to keep in mind. As far as measures go, you may want to think about using the depression anxiety stress scale. This is the DA SS, the Academy of Pediatrics. Uses this depression anxiety, stress scale, DA, SS. If you're looking at children or adolescents, it's been u validated and is reliable for seven to 18 year olds. I know sometimes people come to me looking for measures that deal with that age range. Seven to 18 is the DAS, depression anxiety stress scale because it may be different than using the B depression or the B anxiety and so forth. For substance abuse, you want to turn to NDA as the experts, the National Institute of Drug Abuse We're always going to do with drug abuse, a urine or drug test first before we give the self report. The reason for this and you should document this in your outcomes and measure section is that the research shows that people are more honest on self report if they've just taken a urine drug test. Even though it's not going to be our major measure, it's not our primary outcome, we're still going to do it because it will increase honesty on the self report measure you then might select. Just mention that you're going to do A Cleo waved CLIA waived ten panel urine test. Again, I'm going to give you the exact wording for this. It'll be a leo waved ten panel urine test for drug use. Okay? And you can Google that again and you'll be able to put in the information and so on. But very important that you pay attention to that. And as you're looking for tests, look at the price because that'll be important to your agency. You'll want to report that. So when I Googled, for example, there were 25 tests for $118. But it's also possible that your agency already pays for these urine tests, and they would already have included that in their budget, and they won't mind. But I thought that $25 for 118 I'm sorry, 25 tests for $118 was a good A good benchmark. Now, for your primary outcome when you're studying substance abuse, you may want to use the ASI, the addiction severity index. Again, you'll do this at intake. This would be a case where you'd have to have some of the folks at the agency who are certified to give the ASI. So look into those details there. Addiction, severity test. If you don't use the whole scale, but you just use section or items, excuse me, D one through D. Then you wouldn't need a certified individual to evaluate the addiction severity. There's all these different sections on the ASI. If you just focused on items D one through D 13, that would help you with a good outcome there. Um If you go with that, the ASI and just that section d1d 13, still say that there's reliability and validity, but say that you're only using items D one through D 13 and the ten panel urine test. What else did I look at? I looked at some anxiety and depression scales. You might be interested in using your measurement. We've already mentioned the DAS. No, this one is the DA SS 21. This is a good one for anxiety and depression. It's a world health organization instrument. The UN has approved it and it's been re It's been shown to be reliable and valid through many countries. You could also use the symptom checklist, which is known as the SCL, the SCL 90. It'll have nine subscales and 90 items. Twice, you'd have to enter 90 items because you'd be doing it at pretest and post test or intake and discharge. Think about that because that's a lot of data entry. Maybe you're interested in self esteem. There's a Rosenberg ten items self esteem test in the public domain. That's that's for people ages 12 and up. Again, here you could Google self esteem Assessment for children. You can also use it for adults, Rosenberg, ten item Scale. As you're looking at your measurements this week, this is to look forward to next week a bit. Notice if there's any reverse scored items, you're going to have to think about that as you do data entry next week. As you're reading through the items, do you notice that most go in the direction of showing severity of, say, drug abuse, but some items make it seem like low severity? That would mean that you're going to have to reverse score those items. We'll talk more about that next week. But the more you've looked through your measures by next week, the clearer you'll be about that assignment, Gosh, this is where I would say, are there any other questions? Do people want to bring up their agencies as examples and so on? We're such a tiny class that, of course, just reach out to me if you have questions or if you want to run your measure by me, you do not have to. That's what the discussion for, and I can give you feedback there. Then you can write up your report. This section of the report, the setting, the participants, and the outcome and measures. But this is literally where I would say, are there any questions? So We won't have any. Given that you're not here this evening. But again, it's a tiny class. And so we're just going to work around that the best we can. I hope I gave some examples that hit home for some folks, but it'll really depend on what agency you're evaluating. Okay. Good bit of work this week, next week, a lot of work. Again, bear that in mind with the holiday coming up. Hopefully, you got my email about that as well, and I'm enjoying reading all the plans you have so far. Thank you for the hard work you've been doing. Take care. Okay.

,

4

Week Three Program

Name

Institution

Course

Tutor

Date

            Various research methods could help assess the capabilities of Mental Health America programs aimed at reducing stigma and ensuring more people have access to mental health support. A mixed methods strategy can be suitable based on the complicated nature of mental health solutions and the necessity to capture quantitative and qualitative results.

            A quasi-experimental design would be helpful for a program on reduction of stigma. This design will enable a comparison between places where MHA implements such projects and places where it does not, recognizing that randomization might not be possible in real life. Attitudes regarding mental illness could also be recorded before and after the intervention with the help of surveys and interviews (Fink, 2015). Those data would offer a mix of quantitative and qualitative findings on the change process. Such an approach is consistent with the evaluation question: "How effectively does MHA employ the strategies in reducing the public's stigma against persons with mental illness and discrimination against them?"

            The program focused on improving the accessibility of mental healthcare requires the employment of the longitudinal study method. Many of the changes to the access algorithms would be assessed while measuring instant and long-term effects associated with the MHA’s solutions (Torjesen, 2022). The design would help incorporate regular data collection points to help monitor changes in the utilization of the services, waiting times, and demographic reach. Such a strategy helps address the assessment question regarding the MHA’s contribution to the improvement of access for hard-to-reach populations.

            Stratified random sampling is the most functional technique in terms of sampling procedures. It helps ensure a balanced representation across all social strata, weighting more even marginalized ones, which has become the primary objective of the MHA (Fink, 2015). It could include stratification based on factors like age, income, location, and previous contact with mental health facilities to ensure the inclusion of all target population groups. The method helps ensure that the collected data represents different communities served by the MHA while helping maintain statistical validity.

            For instance, in the evaluation of the stigma reduction program. The population is put under strata such as community features like urban/rural, socioeconomic status, and any previous exposure to mental health. It helps ensure the evaluation demonstrates the program's effects across the different community setups (Fink, 2015). Equally, in the case of a healthcare access program, stratifying groups could invite various geographical distances from healthcare services, having insurance or not, and how these factors vary with individuals to access care.

            There are several benefits to stratified random sampling within this research. Most importantly, it offers accurate and representative data regarding how diverse subgroups respond to MHA’s interventions, permitting more targeted program improvements. It is aligned with evidence-based strategy and supports the organizational objective of serving different populations successfully while at the same time ensuring the maintenance of ethical standards of assessment.

References

Fink, A. (2015). Evaluation fundamentals: Insights into program effectiveness, quality, and value (3rd ed.). Thousand Oaks, CA: Sage.

Torjesen, I. (2022). Access to community mental health services continues to deteriorate, survey finds. BMJ: British Medical Journal (Online), 379, o2585. https://doi.org/10.1136/bmj.o2585

,

3

Week 2 Discussion Evaluation

Student Name

Institution

Course Name

Instructor

Date

Patients’ experiences in community mental health hospitals have continued to deteriorate, and many people in the community are reporting increased difficulties in accessing these services (Torjesen, 2022). Based on the information obtained from the interview with key personnel at the agency last week, Mental Health America offers various range of services to improve mental health involves numerous services such as promoting community support and well-being, reducing stigma, and increasing access to mental healthcare. This week, two areas of focus that I will evaluate for effectiveness in the agency will be increasing access to mental health and reducing stigma.

1. Reducing Stigma

Program Goal: The goal of this program is to ensure that the agency focuses on mental health stigma reduction, this will be achieved by fostering an inclusive community environment and educating members of the public.

Evaluation Question: To what degree do MHA’s educational initiatives impact the reduction of mental health stigma and discrimination in the community?

Evidence: According to Colizzi et al. (2020) prevention, promotion, and early intervention strategies on mental health have a significant impact on the well-being and health of the people in the community. Therefore, the success of this program will be demonstrated I post-intervention research shows that there is a significant reduction in stereotypes, and misconceptions about mental health when compared to the data before the program.

Data Collection: To effectively evaluate the success of the program, data on variables such as attitudes toward mental health patients and rates of stigma pre- and post-intervention will be key.

2. Increasing Mental Healthcare Access

Program Goal: The goal of this program will be to improve access to mental healthcare services for populations that have limited access by removing systemic, financial, and geographic barriers that prevent them from accessing the care.

Evaluation Question: How much does MHA contribute to improving access to mental health services for hard-to-reach populations?

Evidence: The program will measure improvement in mental health access, the success of this program will be demonstrated through the change in numbers, an increase in mental health access among the minority and underserved populations and a reduction in the amount of time they spend waiting will be a crucial measure of success.

Data Collection: Various variables will be crucial in highlighting the desired changes, these variables include several referrals made compared to those that were fulfilled within a specific time, rates of utilization of the service, and demographics of the mental health patients.

Generally, when collecting client data for both services, you should be mindful of maintaining confidentiality and obtaining informed consent as stated in the ethical standards. The evaluation process can be based on the guidelines for evidence-based practices in program evaluation provided by Fink (2015) and used in the current study. Stigma reduction and access enhancement are two of the main goals of MHA and it is important to appreciate the extent to which MHA is achieving these goals. The evaluation will be evidence-based and outcomes-based with case measurement of change in attitude within the target population or community and the effective rates of utilization of the services offered. Ethical standards such as the guarantee of confidentiality and informed consent will be used to ensure privacy.

References

Colizzi, M., Lasalvia, A., & Ruggeri, M. (2020). Prevention and early intervention in youth mental health: is it time for a multidisciplinary and trans-diagnostic model for care?  International journal of mental health systems,  14, 1-14. https://doi.org/10.1186/s13033-020-00356-9

Fink, A. (2015). Evaluation fundamentals: Insights into program effectiveness, quality, and value (3rd ed.). Thousand Oaks, CA: Sage.

Torjesen, I. (2022). Access to community mental health services continues to deteriorate, survey finds.  BMJ: British Medical Journal (Online),  379, o2585. https://doi.org/10.1136/bmj.o2585

,

Ethical Consulting and Confidentiality in Program Evaluation at Mental Health America

Student’s name

Instructor

Course

Date

Ethical Consulting and Confidentiality in Program Evaluation at Mental Health America

For this class, I will be using the organization Mental Health America for this agency evaluation. Its description indicates that Mental Health America is a nonprofit dedicated to cultivating mental health, preventing the causes of mental illness, and providing for social determinants which affect mental health (Mental Health America, 2023). The MHA works to build positive psychological and social outcomes through its emphasis on advocacy, community education, and supportive services such as mental health screening, peer support programs, support groups, and crisis counseling services. For more than a century, MHA has worked diligently to build a network that supports individuals in all states, focusing on three key areas: stigma reduction, increased access to mental health care, and building communities that support all dimensions of well-being.

Because of this, being a consultant assessing the program's effectiveness at MHA requires client confidentiality as the central rule. Data collected on clients through program evaluations needs to follow strict guidelines about the ethical treatment of that data. For instance, the APA guidelines clearly state that client information is to be treated as confidential (APA, 2017). All data reported from the clients must be de-identified; therefore, no personal details like names, addresses, and contact information should appear or be summarized in a code format that does not allow individual identification. The database for participant data will be kept in an encrypted and password-based database through the use of Research Electronic Data Capture (REDCap). In addition, all hard copies of documents will be stored away in locked drawers in a safe ro