วีดีโอ

In this example the meta-analysis is comparing the raw mean score for two separate groups at the "final" time point. I have never done a meta-analysis looking at pre- to post-intervention change as you're describing. I did a quick search and found two articles you may find interesting: pmc.ncbi.nlm.nih.gov/articles/PMC6998624/ journals.sagepub.com/doi/full/10.1177/25152459231217238

Interesting - I actually had not heard of either of these methods as they are not commonly used in my field. I found this paper: journals.lww.com/ijebh/abstract/2018/12000/a_new_improved_graphical_and_quantitative_method.3.aspx and this software that can do it: www.epigear.com/index_files/metaxl.html For doing this in R, I found search.r-project.org/CRAN/refmans/metasens/html/doiplot.html and the more complete documentation here: cran.r-project.org/web/packages/metasens/metasens.pdf I honestly don't know if i'll make a tutorial on this one, if I do it likely won't be anytime soon. The reason is just that I am really caught up on my current project and I don't have the time right now to do the reading/understanding of this plot and index that i'd want to do before teaching other people how to do it. I do appreciate the request - i will keep this on my radar for if i get any free time!

Glad to hear it was an efficient video, it is so hard for me not to ramble about unique use cases for it haha. Zotero is pretty easy to use once you experiment with a bit, but you may also find their bibliography creator useful if you need to create a bibliography fast and don't want to deal with Zotero as a program/app: zbib.org/

Do you mean the SE for the overall meta-analysis? If so, metafor gives you this in the results. Or do you mean the SE for a specific effect size? If so, this may be helpful for various situations where it is not as straight forward as calculating SE "normally": training.cochrane.org/handbook/current/chapter-06#section-6-3-1

also this may be helpful generally: library.soton.ac.uk/variance-standard-deviation-and-standard-error

Very cool, I had not heard of this but just skimmed their website quickly. I am glad people are working on this! It'd be great if it could connect to Ollama since a lot of people use that to chat locally w/ LLMs (i'm assuming it doesn't because it said the local models only work on Mac). I will read about this more in depth later, i very much appreciate the note about it!

Great question! PRISMA actually consists of more than just the flow chart, there's also a checklist (www.prisma-statement.org/). The flowchart, in my mind, achieves a few things: a) If done well, it transparently presents insights into how many studies were found and when/why they were excluded. b) Some fields use quite short journal articles and the flow chart is a way to present a lot of necessary information in a very brief format, and c) it helps you as an author ensure that you have provided enough information about your literature search and screening. That said, as noted the flowchart is just part of PRISMA - the checklist is also very helpful in ensuring your review is reported transparently.

transparency

I typed up a reply yesterday and I guess I forgot to hit reply since it isn't showing up here, but transparency as the other viewer mentioned is the primary reason :)

@@LearnMetaAnalysis I can see it. Thanks.

Great questions. If I'm understanding your question correctly than you have two different main ideas we should discuss: 1) multiple predictors, 2) multiple response variables. For 1) I think what you're looking for is multiple meta-regression. This isn't used much in my field, but the process is similar to what i call a moderator analysis. Here are some resources: Moderator analysis using a single variable as a moderator Video tutorial for conventional models w/ categorical variables: th-cam.com/video/LIzp2tgdW9c/w-d-xo.html Video tutorial for three-level models w/ categorical variables: th-cam.com/video/2Qd0NgK0Bkw/w-d-xo.html Video tutorial for three-level models with correlated and hierarchical effects and robust variance estimation: th-cam.com/video/bL1wSs6heoc/w-d-xo.html Note: You can do all three of these analyses with categorical or continuous moderators in Simple Meta-Analysis, and it will give you the R code to replicate the analysis. You can also do multiple meta-regression using SMA. SMA is open source and free, and can be run locally or on the web: learnmeta-analysis.com/pages/simple-meta-analysis-software Long story short(er): If you are doing multiple metaregression, you can set it up like a moderator analysis, and put a + between the different moderator variables. There are some examples in the documentation below. Written documentation on meta-regression: Meta-regression in metafor documentation regarding intercepts: www.metafor-project.org/doku.php/tips:models_with_or_without_intercept How-to example in metafor: cjvanlissa.github.io/Doing-Meta-Analysis-in-R/multiple-meta-regression.html Statistical explanation and example in different R package: bookdown.org/MathiasHarrer/Doing_Meta_Analysis_in_R/metareg.html#multiple-metareg In terms of 2) When you say response variables, do you mean the outcome? For example, I work in the field of education, so a typical outcome would be 'learning'. Within learning, we may have a moderator variable that differentiates between recall or transfer. Meanwhile, we may have a separate meta-analysis interested in the outcome of "motivation", and then we could have a moderator that differentiates between, for example, self-efficacy and intrinsic motivation. Typically speaking, in my field I do not prefer analyzing unlike outcomes in the same analysis. By which i mean for example, i see limited value in examining learning and motivation in the same meta-analysis. The reason is that you can't really draw a conclusion from that - what would the overall effect size mean? To make a broad generalization, it wouldn't mean much in my opinion - it would just mean some positive outcome, and that's generally not helpful in terms of advancing theory and practice in relation to either outcome. Accordingly, in my field I would recommend that only conceptually like measures be analyzed as outcomes in the same analysis, and the more similar the better. I hope that makes sense - if not please let me know and I'm happy to try and clarify.

Conceptually, yes. The terms subgroup analysis and moderator analysis are often used interchangeably in my experience (and that is also consistent with what Harrer et al. write: bookdown.org/MathiasHarrer/Doing_Meta_Analysis_in_R/subgroup.html). Please keep in mind that in this video we're dealing with a three-level model, so our code reflects that.

My background is in the field of education rather than medicine, so I was not familiar with diagnostic meta-analysis until you mentioned it (so i probably won't do a video about it, at least not anytime soon, as it's not something i've seen in my field). I found this article and Cochrane reference which might be helpful for you: pmc.ncbi.nlm.nih.gov/articles/PMC3681011/ training.cochrane.org/handbook-diagnostic-test-accuracy/current

Thanks for your note! I too am a newbie and was just following instructions and descriptions on the site (which uses pip rather than pip3, and describes it as python). Please feel free to explain the differences or how you know its python3 rather than python so i know for the future as i'm just learning python/coding in general for misc projects 😁

Correct, by default it does not create a shortcut on your desktop and you can initiate the software by using the command line prompt. However, there are ways around this where you can create your own shortcut (this is what i did). This is the process I used (not sure if it works on Mac): Open notepad and in a blank document, type in the same command as you would on commandline prompt and only that command: asreview lab Click file -> save as. Choose desktop as location if you want your shortcut there,and name it what you want followed with .bat (like Open ASReview.bat). IMPORTANT: under file name, there is a box that defaults to '.txt'. Click that and choose all files. This will make your file save as a .bat file instead of a .txt file. Note that .bat files run the text as command line prompt so be careful what you enter into the file in this file type. Save. You should see the icon appear on the desktop with two little gears on it. Now when you double click the .bat file it should automatically open cmd prompt, run the line, and open ASReview in a browser window.

Great question. I think it will only work with bibliographic type data in .csv files and such, but if you have a bunch of PDFs, I can think of a few different options that might work. The most viable and easiest to implement is you could add the pdfs to zotero, then export the bibliographic information from zotero. I think that should be efficient and easy. I just tried clicking and dragging a PDF into zotero and it automatically extracted the relevant citation information for me. So I think that will likely work for you as well, but please let me know if not!

@LearnMetaAnalysis Thank you for your reply my friend. But my materials are simple webpage articles and they does not have anything unless a URL. Thank you again🌷💐

Awesome! I'm so glad you found it helpful and congratulations on finishing your thesis! If you remember, after it is defended/published please circle back and let us know what topic you meta-analyzed, it's always fun to hear about what topics other people are investigating and some of the key findings 😁

I do not think I can give you a good answer to this based on the limited information. From what you have written, I think what you're asking is, should you reverse some effect sizes but not all effect sizes? And to that my answer is I don't really know because it depends on your outcome variable of focus and your analysis in general. For example, you stated a decrease in negative behavior; this makes me wonder if your outcome is positive behavior, negative behavior, or behavior change, as you may well have different ways of dealing with the data to account for each of those approaches. Regardless, the main idea is you want to ensure you know what a positive or negative effect size means across the entire sample of studies. For example, positive should likely always mean the same type of thing (hypothetically - an increase in positive behavior). If something like that is the case, a critical question I would consider is, is a decrease in negative behavior the same as an increase in positive behavior? [this last point gets away from your question a bit but I think its worth considering - no need to answer it here, it was rhetorical to prompt thought on the topic if its relevant to your data set]

@@LearnMetaAnalysis Thank you for your kind answer !

I think you may find this chapter on missing data helpful (i know this is a link to the older version of the book, but it is what I had saved on my computer and could access quickly): handbook-5-1.cochrane.org/chapter_16/16_1_missing_data.htm

My work schedule has been very busy so I have not been able to create it yet. If you want the code, you can use Simple Meta-Analysis to run a 3level MA with CHE RVE and it will show you the Rscript for each analysis: noahschroeder.shinyapps.io/SimpleMeta-Analysis/ If you prefer a text format, it is described in Harrer et al.'s open book: bookdown.org/MathiasHarrer/Doing_Meta_Analysis_in_R/multilevel-ma.html#rve however their example uses impute_covariance_matrix from clubSandwich which has been deprecated. That is why SMA uses vcalc from metafor instead. Unfortunately I don't think i'll have time to make a video about it for at least a few weeks as I have a few notable deadlines approaching.

I don't have a mac close by to try it, but for me on my PC, i had to close and reopen command line (in your case, terminal) after installing python (see ~2:55 in the video). Once i closed and reopened it, it worked properly. Outside of that and the instructions on their page, I am not sure I will be able to help: asreview.nl/download/

type python3 --version

So this will largely depend on what you're trying to analyze, but I'll give an example that will hopefully clarify (and if not please let me know!). Let's saying i'm running a meta-analysis examining the impact of virtual characters (VC) compared to non-VC learning conditions. If the study i'm analyzing is VC vs traditional teaching, i'd have that comparison written down somewhere in the row as my comparison. I would also have a column that is my 'control condition' column where i would have 'traditional teaching' as a categorical option in a drop down menu. Then i would also have additional columns that focus on aspects of the VC design, such as if it were animated, what gender it was, what kind of voice it used, etc. Then when i run my meta-analysis, i have categorical columns (control condition, animation, gender, voice in this example) I can examine as moderators. You can of course have additional columns to describe the control condition in more detail. It really depends on what you're trying to analyze, as you can, within certain constraints for meta-analysis to make sure you don't violate statistical assumptions, set up your coding form with a huge variety of different columns etc. Basically what I'm saying is, you can have whatever columns you want to describe your studies, as long as that is what you're interested in and you don't violate statistical assumptions. I hope this makes sense, if not please let me know!

@LearnMetaAnalysis Thank you for your reply! I will try to explain my problem. If I'm coding for an intervention in a school, and I want to code for instructional methods... let's say an experiment uses both meditation and discussion as a method. Should I code both methods in different columns, and then perform a meta regression later (instructional methods as the IV and effect sizes as the DV) on to see which methods had the most effect? ^^ Thank you so much for the reply. I think I'm struggling because it is not a case of this method versus that method, but combined methods

@@FC-os1mm Ah I think I understand the issue now. What I would likely do in this case is first code everything via just writing down the specific interventions so I can see what the 'combinations' of methods are. Then if there are enough similarities, you could create categories (e.g., meditation + discussion could be one category, lecture only could be one category, lecture + meditation could be a category, etc.). But I would probably take the approach of fine-grained, specific coding first, then reviewing what potential commonalities I had, then re-coding a new variable with the new categories. Example: let's say i have studies that have lecture (k=8), meditation + discussion(k=1), discussion (k=4), group work (k=12), group work + discussion (k=2). Then, I would likely (depending on my research questions) create the following groups: lecture only (k=8), discussion only (k=4), group work only (k=12), multiple interventions (k=3). Downside of this approach is that 'multiple interventions' group in this case is a mix of misc. interventions, so you would need to keep that in mind during interpretation; it would not be particularly meaningful in this case. But you may ultimately decide you want to keep them coded as fine-grained as they were initially, in which case you don't need to do any re-coding. It's all going to depend on your research question(s) and the data available. I apologize this seems like kind of a vague answer but 1) i don't want to tell you how to do your study because it's your study, 2) there are many ways this could be done, it really depends on what makes the most sense for what you need to know.

@@LearnMetaAnalysis No, don't worry! You have helped me so much! Thank you very much for your time and your explanation.

I use R because I like meta-analysis and the common meta-analysis packages use R. Outside of that, I have no real preference in programming languages :) I agree 100% on the competition in the AI coding precision space - there are a lot of different options and I certainly am not the one to say one is best as I am not a programmer. I meant this video as more for the "general" person working in an academic space who likes to use 1 service for many things and wants it to be 'good enough'. Perplexity is my favorite for that so far personally. If I was doing heavy coding, or really any one specific thing heavily, my current favorite LLM may change. If you have recommendations for others please feel free to share, i like testing out these different systems as it is fun to see what they can and can't do well. I know Google's notebookLM is getting a lot of social media attention at the moment and that's an interesting tool as well if you like to test out different LLMs!

@@LearnMetaAnalysis Have you tested SciSpace? It's much better than Perplexity imo. Only thing that's missing is the search engine functionality, otherwise I think it's superior in most other aspects.

It is on the section 6.2.1 Layout Types

@@alirezahekmati7632 Thanks for this. A standard format may be true in medicine. Generally speaking, in education I've seen a variety of layouts but the 'bare minimum' i've seen is study name and effect size information displayed in the 'plot' part. However I've also seen some that provide the means/SD as text. Metafor allows you to add/remove a number of different features in the plot. Links: www.metafor-project.org/doku.php/plots:forest_plot wviechtb.github.io/metafor/reference/forest.default.html

i *think* they will work well for every page in SMA except categorical moderators (but i haven't tested anything except SMD, because that's the data sets i have available), and only because on the categorical moderator page I have a custom-formatted table and the data needed for that table requires specific columns in the data set which i would expect for SMD, but may not be in the data set for OR, RR, HRs. Even if the table fails to load, I think it'll still give you the results if you click the results to download the intercept and no intercept models. I don't have any data sets with those effect sizes otherwise i'd test them against the R code. But like i said, i expect the results to the same, as long as you're not expecting any sort of transformations, because metafor does not specify the effect size in the meta-analysis code itself, it just specifies what column to look for for an effect size. For example, we specify yi = yi in the data set, rather than specifying something like yi = a specific type of effect size. I hope that makes sense, if not please let me know.

I opened a GitHub issue on the repo with the image of the Error.

Thank you so much for letting me know! I saw your note on github and I'd prefer if we can continue the conversation there as it will be easier for me to track. Here is the thread for those who want to follow it: github.com/noah-schroeder/runSMA/issues This should be an easy fix if it is a notable issue, metaSEM is only used on one page in the app. I look forward to discussing what you experienced on the github page and we'll get it sorted out so that the error doesn't occur anymore 🙂

I hope so! I am adding on as time allows. For example, the UI is now modernized compared to the version seen here. This video shows it: th-cam.com/video/29tPbQAPABU/w-d-xo.html as does the web interface: noahschroeder.shinyapps.io/SimpleMeta-Analysis/

I think you are correct. When i created this code template for myself I did not understand how loops worked and LLM's weren't available yet. I am confident this could work on a loop but I have never tried it. And now i just use Simple Meta-Analysis so I never have to mess around with editing the R code unless i'm modifying the app :)

​@@LearnMetaAnalysis Wow, so Simple Meta-Analysis is that good? I am going to test it for sure after the end of the playlist. Yes LLM's are a blessing!

@@alirezahekmati7632 I created SMA so I am biased :) But for perspective, I can run a 3level MA with CHE RVE and ~26 moderators in about 15 minutes there which would be ~5-6k lines of code in R :)

​ @LearnMetaAnalysis Wow, that's cool! I am a big fan of open-source software. I will try to contribute with feedback and maybe even code new features! Based on what I have seen so far it seems meta Analysis can be completely automated as long as you have data in good structure.

To be honest, so far i've submitted maybe 2-4 three-level meta-analyses and I have used these plots with some of them, and metafor plots when i used correlated and hierarchical effects and robust variance estimation. I have not had reviewers express a preference for one version or the other so long as i explain what this version is showing in the forest plot. So my feeling as of now is it may be field dependent, but in my field it seems more like it is the author's preference to express the data as they can rationalize.

@@LearnMetaAnalysis Thank you!

@@LearnMetaAnalysis Thank you!

Yes, in the context I gave in the video. There are 2 different tests of the moderator, with the no intercept model testing if they are significantly different than zero rather than the intercept model which tests if there are differences between levels (in the context of the example in the video)