Skip to main content
Download PDF

The effect of capacity building evidence-based medicine training on its implementation among healthcare professionals in Southwest Ethiopia: a controlled quasi-experimental outcome evaluation

BMC Medical Informatics and Decision Making volume 23, Article number: 172 (2023) Cite this article

Abstract

Background

Evidence-based medicine (EBM) bridges research and clinical practice to enhance medical knowledge and improve patient care. However, clinical decisions in many African countries don’t base on the best available scientific evidence. Hence, this study aimed to determine the effect of training interventions on background knowledge and awareness of EBM sources, attitude, competence, and practice of EBM among healthcare professionals.

Method

We designed a controlled group quasi-experimental pre-post test study to evaluate the effect of capacity-building EBM training. A total of 192 healthcare professionals were recruited in the study (96 from the intervention and 96 from the control group). We used a difference-in-differences (DID) analysis to determine the effect of the training. Along the way, we used a fixed effect panel-data regression model to assess variables that could affect healthcare professionals’ practice of EBM. The cut point to determine the significant effect of EBM training on healthcare professionals’ background knowledge and awareness of EBM sources, attitude, and competence was at a P-value < 0.05.

Result

The DID estimator showed a significant net change of 8.0%, 17.1%, and 11.4% at P < 0.01 on attitude, competence, and practice of EBM, respectively, whereas no significant increment in the background knowledge and awareness of EBM sources. The fixed effect regression model showed that the attitude [OR = 2.288, 95% CI: (1.049, 4.989)], competence [OR = 4.174, 95% CI: 1.984, 8.780)], technical support [OR = 2.222, 95% CI: (1.043, 3.401)], and internet access [OR = 1.984, 95% CI: (1.073, 4.048)] were significantly affected EBM practice.

Conclusion

The capacity-building training improved attitude, competence, and EBM practice. Policymakers, government, and other concerned bodies recommended focusing on a well-designed training strategy to enhance the attitude, competence, and practice towards EBM among healthcare professionals. It was also recommended to enhance internet access and set mechanisms to provide technical support at health facilities.

Peer Review reports

Background

Evidence-based decision-making is a strategic method of applying empirical knowledge and research findings [1, 2]. Evidence-based Decision-making at healthcare facilities has shown improvement over the past few decades due to the advancement of EBM [3, 4].EBM is a method for determining the safety and efficacy of medical therapies and public health interventions [5]. EBM avoids decisions based on gut feeling and enables it to rely on organized facts obtained from scientific literature, organizational data, professional expertise, values, and concerns of stakeholders [6].

Healthcare professionals should provide clinical services based on the best and rigorously tested evidence [7]. EBM is the ‘conscientious, explicit and judicious use of current best evidence in making decisions about individual patients’ [7, 8]. Despite this, the slow and haphazard process of promoting, translating, and implementing research findings into clinical compromises the potential benefits of clinical and public health research [9].

The practice of EBM involves the integration of three pillars such as up-to-date research evidence, clinical expertise, and predicaments, rights, and preferences of the patients in making clinical decisions about their care [10, 11]. We can summarize the concept of EBM into five-step models such as 1/ formulating answerable clinical questions, 2 / searching evidence, 3 / appraising the evidence, 4 / deciding to integrate evidence with your clinical expertise and the patient’s values, and 5/ evaluating the performance [12].

EBM has been used to guide policy formulation and implementation of preventive and curative interventions in Europe, North America, and Australia [13]. In this regard, around a hundred institutions have been established to synthesize research evidence, develop systematic reviews, policy briefings, and health technology assessments, and set guidelines for practice [14, 15]. Despite the aforementioned signs of progress, the European Centre for Disease Prevention and Control (ECDC) reports only 7% of the vaccination-based interventions achieved the highest grade evidence, which means clinical decisions with meta-analysis and randomized control trials [16]. In line with this, extant literature accentuated that most decisions in the world healthcare system are still done based on the low-quality evidence reported from case reports, case series, and observational studies [17].

When we come to Africa, many factors exacerbate the problems. Countries in this continent don’t always base their healthcare decisions on the best available scientific evidence [18,19,20,21]. A literature review signifies that talking about the level of evidence in the health systems of this continent was deemed luxurious. The inadequate background knowledge, competence, and behavior of the healthcare professionals are a challenge to implementing EBM in the continent [22, 23]. Healthcare professionals in Sub-Saharan Africa lack the competence to critically review and judge the quality of research [24]. The other problem that aggravated the situation is that only a little randomized research has been conducted for clinical conditions in this sub-continent [25, 26].

In Ethiopia, the Ministry of Health (MOH) has been working on building the healthcare human resource capacity by distributing clinical guidelines to all healthcare facilities [27, 28]. Nevertheless, those clinical guidelines haven’t been developed based on assessing high-grade evidence from systematic literature reviews and meta-analyses. Similar to other low-resource countries, clinical guidelines in Ethiopia are not timely updated to include the latest evidence from high-level medical research [29,30,31].

Additionally, teaching EBM is not yet a component of the undergraduate medical and health sciences curriculum, which results in a gap in background knowledge, behavior, competence, and practice of EBM [32]. Those situations lead most healthcare professionals to build their clinical decisions based on what they had learned from undergraduate or postgraduate classes; they didn’t use updated evidence [33, 34].

Previous studies revealed that 32.3–57.6% [27,28,29, 35,36,37,38,39,40] of healthcare professionals in Ethiopia integrate EBM into their medical practice. Few studies also determined background knowledge and awareness of EBM sources [27, 36, 41,42,43,44,45,46,47], attitude [27, 36, 39, 41, 43, 44, 46,47,48], and competence [49, 50] of healthcare professionals for EBM.

Literature also revealed that experience [51], age [52], educational level [53], workload or insufficient time [27, 36, 54, 55], sufficient hardware [55], poor internet access [27, 36], lack of technical support [54], knowledge of statistical terms [56], self-efficacy [52], lack of interest to find research reports [35, 39], and lack of patient cooperation [57, 58] were commonly depicted determinants of background knowledge and awareness of EBM sources, attitude, competence and practice of EBM.

Training has been the commonly recommended strategy to improve the background knowledge and awareness of EBM sources, attitude, competence, and practice of EBM. We argue that if we trained healthcare professionals to know the relevance of EBM, then it is more likely to integrate EBM into their daily medical practice and patient care. Previous studies reported training as one of the determinant factors of EBM [35, 36], but only a few studies examined its causal effect on healthcare professionals’ background knowledge and awareness of EBM sources, attitude, competence, and practice of EBM [59,60,61,62].

Examining the impact of training would enable us to identify the prospects to scale up the capacity of EBM. Therefore, we hypothesized that a well-designed training intervention could enhance background knowledge and awareness of EBM sources, attitude, competence, and practice.

Methods and materials

Study design and setting

We used a quasi-experimental study design using controlled before and after the study to evaluate the effect of capacity-building training on the implementation of EBM. A two-week capacity-building training from January 18 to February 2, 2022, was given to randomly selected healthcare professionals working at three hospitals in Ilu Ababa Bora and Buno Bedelle Zones, which were together called Ilubabor until recent times. Conversely, a controlled group was selected from two hospitals that didn’t take the training during the project period.

The measurement for both the intervention and control groups was taken at two points in time, in which the baseline data was taken, before the training, from January 10 to 17. End-line measurement for both groups was taken, after four months of the training days, from June 3 to 14, 2022. The control and the intervention groups were healthcare professionals in Ilu Aba Bora and Buno Bedele Zones, Oromia Region, Southwest Ethiopia (See Fig. 1).

Fig. 1
figure 1

Map of the study area for Ilubabor, Oromia Region, Southwest Ethiopia

Full size image

Study population

The source population of this study included all healthcare professionals working at public hospitals in Ilu Aba Bora and Buno Bedelle zones. Healthcare professionals who were on leave or retired were excluded from the study. Additionally, healthcare professionals who had previous exposure to EBM training, who had less than six months of experience, or who were not permanent employees were excluded from both the control and intervention groups of the study.

Sample size and sampling technique

We used the comparison of two population proportions formulas in the baseline and end-line components of the study. We took the largest after calculating the sample size for the primary outcome variable (the practice of EBM) and secondary outcome variables (background knowledge and awareness of EBM sources, attitude, and competence). Accordingly, a 55% prevalence of EBM practice from the previous study [35] was taken. Additionally, the probability score at 95% intervals, and a precision level of 5%, were considered.

$${\text{n}} = ({({{\text{Z}}_1}_{ - a} + {{\text{Z}}_1}_{ - b})^2}[{\text{P}}1{\text{ }}(1-{\text{P}}1)\, + \,{\text{P}}2{\text{ }}(1-{\text{P}}2)\left] {)/(} \right[{\text{P}}1-{\text{P}}2]2)$$

Z1-α = 1.96, standard normal deviate corresponding to the 95% confidence interval.

Z1-β = 0.840, standard normal deviate corresponding to 80% power of the study.

p1 = 0.55, the proportion of the outcome variable by reviewing the previous studies, whereas p2 is the expected level of practice at the end of the project in which an increase of 20% in EBM practice was expected. Accordingly, we got n = 91. Then we added a 5% non-response rate, and the final sample size became 96 for each group.

There were five hospitals in Ilu Aba Bora and Buno Bedelle zones. The total sample size obtained based on the above calculation (96) was proportionally allocated to three hospitals that were planned to be addressed in the capacity-building training. Then, healthcare professionals for the intervention group were selected with a simple random sampling technique from each hospital.

On the other hand, the total sample size obtained based on the above calculation (96) was proportionally allocated to two hospitals that weren’t addressed by the training intervention. Then, healthcare professionals for the control group were selected with a simple random sampling technique from each hospital.

The selections of those three hospitals for the intervention group were purposively based on the direction given by the Mettu University Office of community service. This was because the three hospitals were located at the community service site of Mettu University. The left two hospitals found in a similar zone but not in the community service site were considered control hospitals. The sampling frame was prepared based on human resource healthcare professionals’ profiles and the study objective in each hospital to select the participants.

Intervention

We designed a capacity-building training project, which aimed at enhancing the implementation of EBM by improving the background knowledge and awareness of EBM sources, attitude, competence, and practice of healthcare professionals at public health facilities in southwest Ethiopia. The training content and aim were evaluated by Mettu University community service coordinators and experts in the field.

Following the completion of the baseline data collection, training on EBM was given for a total of 2 weeks, from January 18 to February 2, 2022. The end-line survey was collected four months after the baseline survey (from June 3 to 14, 2022). The training was given only to the intervention groups. It took a total time of 41 h (8 h of discussion, 11 h of lectures, and 22 h of practical or EBM labs). The mode of delivery was face-to-face. The lab session was delivered at three digital labs of Mettu University. Trainees were guided by trainers during lab sessions. A discussion was planned to be at the end of lectures in which the trainer’s role was as a facilitator.

An EBM module was developed by reviewing different literature [9, 19, 22, 61, 63,64,65,66,67] and Johns Hopkins Nursing Evidence-Based Practice (JHNEBP) Model [68]. The module was distributed to trainers via soft or hard copy based on preference. Six health informatics and public health experts were involved in developing and reviewing the module as well as to deliver the training. The two trainers have master’s degrees in health informatics; the left 4 are an assistant professors and above in public health and health informatics.

The training module was classified into nine lessons. The training module mainly focused on the introduction and principles of EBM, formulating a focused clinical question, finding the current best evidence, evaluating the quality of the evidence, using an online database like PubMed and Cochrane Library, searching strategy, critical appraisal, interpreting research results, diagnostic test, validity appraisal, systematic review, meta-analysis and so on (See Additional file 2). A syllabus containing the detail of the training content was given before the commencement of the training. Case scenarios prepared by trainers were given to trainees during their practical sessions.

Outcome measure

The primary outcome variable of this study was the effect of capacity-building training on the EBM practice of healthcare professionals. The secondary outcomes variable included the change in background knowledge and awareness of EBM sources, attitude, and competence of EBM.

Operational definition

Healthcare professionals

For this study, healthcare professionals were operationalized as employees of health facilities who had at least a diploma certificate in the health profession and provide clinical care for patients or clients [69].

Workload

Healthcare professionals who worked more than 8 h per day were considered as having a workload or coded as ‘yes’ [70].

Instrument

The researchers developed questionnaires that met the study’s purpose. The items of the questionnaires focused on the healthcare professionals’ background knowledge and awareness of EBM sources, attitude, competence, and practice of EBM, and other individual factors such as age, sex, education level, and so on. The data collection tool was prepared in the English language. The instrument was assessed for content validity. A pre-test was done in Jimma referral hospital, which was not part of the actual data collection but has a similar study setting. Accordingly, the required corrections in language and content were done before the actual data collection commenced. Data was gathered by a self-administered questionnaire. Background knowledge and awareness of EBM sources: It was measured by 14 items with two response categories (1 = Yes and 2 = No). Finally, the responses were dichotomized into two “0” and “1”. If respondents responded as yes, it was recoded as “1” otherwise it was recoded as “0”. The normality test for background knowledge and awareness of EBM sources showed skewed distribution, so we computed the median, and healthcare professionals who responded correctly equal to or above the median were labeled as having a good background knowledge and awareness of EBM sources. On the other hand, healthcare professionals who responded below the median were considered as having poor background knowledge and awareness of EBM sources.

Attitude toward EBM

In implementation science, an attitude would be described as how positively one is predisposed towards using a particular evidence-based practice (EBP) [71]. The tool used to measure the attitude consisted of 11 Likert scale items ranging from 1 or strongly disagree to 5 or strongly agree. It was dichotomized into two favorable and unfavorable attitude based on the mean value.

Competence of EBM

Even though there are different commonly used EBM test tools to measure competence, such as the Fresno test [72] and Berlin test [64], we chose to use the assessing competence in EBM (ACE) tool [73]. The ACE tool is more comprehensive as it incorporates all EBM steps. The ACE tool also permits alternate scenarios to control the potential impact of recall bias during testing. The tool has a ‘yes’ or ‘no’ binary format, which could be answered based on provided patient case scenario. We used different scenarios for the pretest and posttest.

The total items of the questioner were 15, which mainly focuses on the five-step model of EBM [12], such as formulating answerable clinical questions (2 items), finding the evidence (2 items), critically appraising the evidence for its validity and usefulness (7 items), and applying the evidence and evaluating its performance (3 items). A high score means good competence in finding evidence for decisions in their daily practices.

EBM practice

It refers to the use of EBM in healthcare institutions. It was measured by 11 Likert scale questions ranging from 1 = never, 2 = rarely, 3 = sometimes, 4 = most of the time, and 5 = always. All individual answers to practice questions were computed to obtain total mean scores and categorized as good practice (if participants scored ≥ mean score) or poor practice (if participants scored < mean score) (See Additional file 3).

Statistical analysis

We used Epi-Data version 4.6 and STATA version 14.0 for data entry and analysis, respectively. Descriptive data were used to describe the demographic characteristics of participants. The baseline differences in the socio-demographic variables between the intervention and controlled groups were tested using either an independent-samples t-test for continuous variables or a chi-square for nominal/categorical variables.

To assess the effect of capacity-building community service training on the primary and secondary outcomes, DID analysis using a 5% level of significance was used. Our data was panel data set or repeated measurement of one variable a minimum of two times. Therefore, we used the Hausman test to select which model was appropriate to analyze our data among different models applicable for panel data analysis. Finally, the test showed that the fixed effect regression model was appropriate to study the effect of other variables on the primary outcome variable.

Both the adjusted R-squared (R2 = 0.736) and the F-test (F-statistic = 43.72, p < 0.001) showed that the fixed effect model was a fitted panel model for our data set and had high explanatory power than other panel models. According to its assumption, the fixed effect removes the effect of time-invariant characteristics (e.g. sex, age, and so on); thereby it controls type II error. This model assesses the net effect of the time-varying predictors on the outcome variable. The odds ratio (OR) with the corresponding 95% confidence intervals was reported to show the strength of the association. A P-value < 0.05 was considered as a cut point to indicate statistical significance in all analyses.

Result

Characteristics of study participants

The study comprised 192 healthcare professionals, with 96 for the intervention group and 96 of them for the control group. Among the total participants approached in the intervention group, 89(92.7%) and 83(86.5%) of them gave a response at baseline and the completion of the study, respectively. All 96(100%) eligible study subjects of the control group completed the self-administered questionnaires in the pre-intervention whereas only 87(90.6%) of this group gave a response in the post-intervention periods.

In addition, when the pretest of each group was analyzed, 59.2% of the intervention group and 71.6% of the control group participants were in the age group below 30 years. As the pretest data showed, males constituted 53.9% of the intervention group and 63.5% of the control group. In addition, among the intervention group study participants in the pretest period, the majority, 55.1% were nurse professionals. Whereas, among those participants in the control group during the pretest period and those nurse professionals were 38.5%. Details of individual characteristics of the study participants at baseline in both groups are reported in Table 1.

Table 1 Individual characteristics of healthcare professionals at baseline in Southwest Ethiopia, 2022
Full size table

Effects of the capacity-building training intervention on primary and secondary outcome variables

The level of EBM practice for the control group was 43.8% (95% CI 38.6%-47.2%) at baseline and 46.0% (95% CI 39.9%-51.5%) at the study endpoint. The level of EBM practice for the intervention group was 38.2% (95% CI 34.8%-43.1%) at baseline and 51.8% (95% CI 46.6%-54.1%) at the study endpoint. The DID analysis indicated that the intervention resulted in an 11.4% net increment in the level of EBM practice among the intervention group compared to the control group. This increment was significant at a P-value < 0.01.

The proportion of background knowledge and awareness of EBM sources among healthcare professionals increased by 7.7% (from 56.2 to 63.9%) in the intervention group while it increased by 2.6% (from 58.3 to 60.9%) in the control group. The average proportion of background knowledge and awareness of EBM sources net difference between the two groups was 5.1%.

The proportion of attitude toward EBM among healthcare professionals increased by 14.5% (from 67.4 to 81.9%) in the intervention group while it increased by 6.5% (from 62.5 to 69.0%) in the control group. The average proportion of attitude net difference between the two groups was 8.0%.

Additionally, the proportion of competence of EBM among healthcare professionals increased by 22.4% (from 42.7 to 65.1%) in the intervention group while it increased by 5.3% (from 51.0 to 56.3%) in the control group. The average proportion of competence net difference between the two groups was 17.1%. The DID analysis showed the effect of the intervention was significant for attitude (P-value < 0.01) and competence (P-value < 0.001), whereas the net effect on background knowledge and awareness of EBM sources wasn’t significant (See Table 2 for more detail).

Table 2 DID analysis of background knowledge and awareness of EBM sources, attitude, competence, and practices of EBM between intervention and control groups during baseline and end-line surveys (n1 = n2 = 96)
Full size table

Factors affecting EBM practice among healthcare professionals

To identify independent factors associated with the end-line-baseline difference of the differences in the mean EBM practice scores, we employed a fixed effect regression model. Variables included in the model were: sex, age, profession, educational level, experience, monthly salary, religion, workload, technical support, internet access, computer access, background knowledge and awareness of EBM sources, attitude, and competence of EBM.

The fixed effect was designed to study the cause of the change in practice within participants and time-varying characteristics deemed not to cause the change. Therefore, those time-invariant variables or variables that change at a constant rate over time were omitted. Finally, the model showed that attitude [OR = 2.288, 95% CI: (1.049, 4.989)], competence [OR = 4.174, 95% CI: 1.984, 8.780)], technical support [OR = 2.222, 95% CI: (1.043, 3.401)], and internet access [OR = 1.984, 95% CI: (1.073, 4.048)] showed a significant effect on EBM practice (See Table 3 for more detail).

Table 3 Fixed effect panel data modeling for independent predictors of EBM practice among healthcare professionals in Southwest Ethiopia, 2022
Full size table

Discussion

The main aim of this quasi-experimental evaluation was to investigate the effect of capacity-building training intervention on background knowledge and awareness of EBM sources, attitude, competence, and practice of EBM among healthcare professionals.

There was no significant difference in background knowledge and awareness of EBM sources, attitude, competence, and practice of EBM between the intervention and control groups at the start of the study. However, after the training intervention, there was a significant difference between the intervention and control groups for attitude, competence, and practice of EBM.

Improvements in background knowledge and awareness of EBM sources were also found, but the change from the DID analysis was not significant. The insignificant result in background knowledge and awareness of EBM sources might be due to the training content not being focused on it. Additionally, knowledge gained from the training might be lost after some point in time if not reinforced by practical follow-ups or assessments.

This finding was in line with a study of Monash University students, which reported a significant effect of learning on attitude and practice [62]. However, our finding contradicts the study of Monash, which reported an insignificant effect on the competence of EBM. This contradiction could be due to the variation in the teaching strategies and the study participants. The study participants in Monash were medical students, which might not enhance their competence by day-to-day practice, unlike healthcare professionals. The content of the course in our study wasn’t similar to the study of Monash, which might be another possible justification for this variation.

An interventional study conducted in China is similar to the current study in that they have a significant effect of EBM training courses on primary healthcare professionals’ attitude [61]. However, it contradicts our finding that reported a significant effect of the training intervention on background knowledge and awareness of EBM sources.

The discrepancy might be due to the difference in the dosage of the intervention and training content. The intervention study from China provided eight weeks of training with two hours every day, but our intervention was only for two weeks. Additionally, the delivery of the training in China was accompanied by conferences. But, the study in China did not assess the effect of the intervention on EBM competence and practice.

Our finding also differs from the study findings reported from South Africa, which reported a significant effect of an education intervention on Knowledge; but an insignificant effect on attitude towards EBM [21]. This variation could be due to the differences in the contents of the training and tools used to measure the attitude and background knowledge and awareness of EBM sources. On the other hand, the interval time between the pre-test and post-test in the previous study was three months, whereas it was four months in the current study.

Accordingly, the long duration between pre and post-training may cause the loss of newly acquired knowledge from the intervention if it isn’t reinforced by practical follow-ups or supportive supervision, which might be a possible justification for this discrepancy. However, further study is required to confirm this and to determine the optimal duration of an effective EBM educational intervention.

In line with our study findings, the study in Taiwan [59] and Portugal [60] reported a significant positive effect of educational intervention on the behavior and competence of EBM. However, unlike our study findings, background knowledge and awareness of EBM sources showed a significant improvement after training. The discrepancy might be due to the study in Taiwan, and Portugal didn’t include a controlled group. Additionally, the study participants in the study of Taiwan, and Portugal were medical students.

The fixed effect model in this study implied that attitude is a significant predictor of the EBM practice. Healthcare professionals who had a favorable attitude were highly practicing EBM. The finding was corresponding with previous studies conducted elsewhere in the World [37, 40, 54]. It could be justified as the beliefs of individuals toward using the EBM is necessary versus unnecessary, beneficial versus harmful predispose them to apply in clinical practice.

The study found that competence was a significant factor in EBM practice, in which healthcare professionals with high competence levels were more likely to practice EBM than their counterparts. In agreement with our finding, a study in Northwest Ethiopia implied that the ability to apprise evidence was a significant factor of EBM practice [40], and also it was consistent with the study of central Ethiopia [38].

The possible justification for this finding could be competent healthcare professionals can do EBM tasks effectively and efficiently. It is impossible to find the best available evidence to apply in clinical practice without the competence of formulating a clinical question, finding information or evidence, critically appraising the information/evidence, integrating appraised evidence with clinical expertise and patient preferences, and evaluating the evidence.

This study showed that healthcare professionals who got technical support are more likely to practice EBM. This finding was supported by a study in Northwest Ethiopia [37, 38]. The possible justification for this could be due to technical support paving the doorway to obtaining guidance from experienced professionals in the field. Providing such support for healthcare professionals could enable them to ask how to access, appraise, apply, and audit evidence.

The result of this study confirms that internet access was one of the predictors of EBM practice. This finding was in agreement with previous reports [35, 36]. The justification for this could be that internet-enabled individuals access sources of information or evidence from digital platforms.

Strengths and limitations of the study

The issue of omitted variable bias could also reduce due to the fixed effect regression model. However, it didn’t conclude causal associations between an intervention and an outcome since randomization couldn’t use in a quasi-experimental study. The data was collected based on self-reported information; this might cause an overestimation of participants’ practice. We relabeled a few variables after data collection because they needed modification to make the labeling representative of each item in the tool. This might be after participants’ responses were already affected due to confusion about its labeling (e.g. knowledge was relabeled to background knowledge and awareness of EBM sources).

The effect of the training was determined based on statistical significance, which might not represent public health or clinical significance. In this study, the baseline data was collected before the training. The end-line measurement was also taken after four months of the training, whereas the immediate and intermediate effect of the training wasn’t assessed. We based the statistical analysis plan only on practices of EBM, which not addressed the possible factors of background knowledge and awareness of EBM sources, attitude, and competence.

Conclusions

The capacity-building training improved attitude, competence, and EBM practice. Policymakers, government, and other concerned bodies recommended focusing on a well-designed training strategy to enhance the attitude, competence, and practice towards EBM among healthcare professionals. It was also recommended to enhance internet access and set mechanisms to provide technical support at health facilities.

Data Availability

Data supporting this study are not publicly available due to legal or ethical restrictions on sharing the data publicly. However, data are available upon a reasonable request from the corresponding author.

Abbreviations

OR :

Odds Ratio

ACE :

Assessing Competence in Evidence

CI :

Confidence Intervals

DID :

Difference in Difference

EBM :

Evidence-Based Medicine

EBP :

Evidence-Based Practice

ECDC :

European Centre for Disease Prevention and Control

JHNEBP :

Johns Hopkins Nursing Evidence-Based Practice

MeSH :

Medical Subject Headings

MOH :

Ministry of Health

Ph.D. :

Doctor of Philosophy

SE :

Standard Error

SD :

Standard Deviation

References

  1. Cvitanovic C, McDonald J, Hobday A. From science to action: principles for undertaking environmental research that enables knowledge exchange and evidence-based decision-making. J Environ Manage. 2016;183:864–74.

    CAS  PubMed  Google Scholar 

  2. Baba VV, HakemZadeh F. Toward a theory of evidence based decision making. Manag Decis. 2012.

  3. Djulbegovic B, Guyatt GH. Progress in evidence-based medicine: a quarter century on. The Lancet. 2017;390(10092):415–23.

    Google Scholar 

  4. Darlenski RB, Neykov NV, Vlahov VD, Tsankov NK. Evidence-based medicine: facts and controversies. Clin Dermatol. 2010;28(5):553–7.

    PubMed  Google Scholar 

  5. Fernandez A, Sturmberg J, Lukersmith S, Madden R, Torkfar G, Colagiuri R, et al. Evidence-based medicine: is it a bridge too far? Health Res Policy Syst. 2015;13:1–9.

    Google Scholar 

  6. Rousseau DM. Making evidence-based organizational decisions in an uncertain world. Organ Dyn. 2018;47(3):135–46.

    Google Scholar 

  7. Melnyk BM, Fineout-Overholt E. Evidence-based practice in nursing & healthcare: a guide to best practice. Lippincott Williams & Wilkins; 2011.

  8. LoBiondo-Wood G, Haber J. Nursing research E-Book: methods and critical Appraisal for evidence-based practice. Elsevier Health Sciences; 2021.

  9. Grol R, Grimshaw J. Evidence-based implementation of evidence-based medicine. Jt Comm J Qual Improv. 1999;25(10):503–13.

    CAS  PubMed  Google Scholar 

  10. Siminoff LA. Incorporating patient and family preferences into evidence-based medicine. BMC Med Inf Decis Mak. 2013;13(3):1–7.

    Google Scholar 

  11. Djulbegovic B, Guyatt GHJTl. Progress in evidence-based medicine: a quarter century on. The Lancet. 2017;390(10092):415–23.

    Google Scholar 

  12. Dawes M, Summerskill W, Glasziou P, Cartabellotta A, Martin J, Hopayian K, et al. Sicily statement on evidence-based practice. BMC Med Educ. 2005;5:1–7.

    PubMed  PubMed Central  Google Scholar 

  13. Bonner C, Fajardo MA, Doust J, McCaffery K, Trevena L. Implementing cardiovascular disease prevention guidelines to translate evidence-based medicine and shared decision making into general practice: theory-based intervention development, qualitative piloting and quantitative feasibility. Implement Sci. 2019;14(1):1–14.

    Google Scholar 

  14. Angelis A, Lange A, Kanavos PJTEJoHE. Using health technology assessment to assess the value of new medicines: results of a systematic review and expert consultation across eight european countries. Manage Hypertens. 2018;19(1):123–52.

    Google Scholar 

  15. Qaseem A, Forland F, Macbeth F, Ollenschläger G, Phillips S, van der Wees P, et al. Guidelines International Network: toward international standards for clinical practice guidelines. Ann Intern Med. 2012;156(7):525–31.

    PubMed  Google Scholar 

  16. (ECDC) ECfDPaC. Guide to public health measures to reduce the impact of pandemic influenza in Europe: The ECDC Menu. ECDC Technical Report. Stockholm: ECDC: ECDC Technical Report; 2010 [Available from: http://www.ecdc.europa.eu/en/publications/publications/0906_ter_public_health_measures_for_influenza_pandemics.pdf.

  17. Burns PB, Rohrich RJ, Chung KC. The levels of evidence and their role in evidence-based medicine. Plast Reconstr Surg. 2011;128(1):305.

    CAS  PubMed  PubMed Central  Google Scholar 

  18. Wiysonge CSJBWHO. Promoting evidence-based health care in Africa. Bull World Health Organ. 2017;95(9):616–7.

    Google Scholar 

  19. Brownson RC, Reis RS, Allen P, Duggan K, Fields R, Stamatakis KA, et al. Understanding administrative evidence-based practices: findings from a survey of local health department leaders. Am J Prev Med. 2014;46(1):49–57.

    PubMed  PubMed Central  Google Scholar 

  20. Puustjärvi J, Puustjärvi L. Practicing evidence-based medicine in developing countries. IEEE. 2015;15(3):1–9.

    Google Scholar 

  21. Buchanan H, Siegfried N, Jelsma J, Lombard CJT. Comparison of an interactive with a didactic educational intervention for improving the evidence-based practice knowledge of occupational therapists in the public health sector in South Africa: a randomised controlled trial. Trials. 2014;15(1):1–11.

    Google Scholar 

  22. Rudan I, Kapiriri L, Tomlinson M, Balliet M, Cohen B, Chopra, MJPm. Evidence-based priority setting for health care and research: tools to support policy in maternal, neonatal, and child health in Africa. PLoS Med. 2010;7(7):e1000308.

    PubMed  PubMed Central  Google Scholar 

  23. Wechsberg WM, Zule WA, Luseno WK, Kline TL, Browne FA, Novak SP, et al. Effectiveness of an adapted evidence-based woman-focused intervention for sex workers and non-sex workers: the women’s Health CoOp in South Africa. J Drug Issues. 2011;41(2):233–52.

    Google Scholar 

  24. Zeidan A, Behairy MJSJoMS. Knowledge attitudes and practices of evidence based medicine among residence doctors in Sudan. Sudan J Med Sci. 2010;5(3):207–12.

    Google Scholar 

  25. Iwelunmor J, Blackstone S, Veira D, Nwaozuru U, Airhihenbuwa C, Munodawafa D, et al. Toward the sustainability of health interventions implemented in sub-saharan Africa: a systematic review and conceptual framework. Implement Sci. 2015;11(1):1–27.

    Google Scholar 

  26. N’Goran E, Ouattara M, Huber E, Bezuidenhout D, Kourany-Lefoll E, parasitology PPCJAi. Challenges and lessons from conducting a paediatric clinical trial in sub-saharan Africa: the case of the praziquantel oral dispersible tablets phase II study in Côte d’Ivoire. Adv Parasitol. 2019;103:75–89.

    PubMed  Google Scholar 

  27. Abdulwadud O, Azazh A, Mekasha A, Heye TB, Nigatu B, Debebe F, et al. Cochrane, evidence-based medicine and associated factors: a cross-sectional study of the experiences and knowledge of Ethiopian specialists in training. Afr J Emerg Med. 2019;9(2):70–6.

    PubMed  PubMed Central  Google Scholar 

  28. Abdulwadud O, Tadesse F, Yilma G, Midekssa M, Ibraghimova IJTPAmj. Knowledge and experience with cochrane and evidence based medicine among health professionals in Debreberhan Referral Hospital in Ethiopia: a cross-sectional survey. Pan Afr Med Journal. 2018;30.

  29. Tessema GA, Gomersall JS, Laurence CO, Mahmood, MAJBo. Healthcare providers’ perspectives on use of the national guideline for family planning services in Amhara Region, Ethiopia: a qualitative study. bmJ Open. 2019;9(2):e023403.

    PubMed  PubMed Central  Google Scholar 

  30. Arega B, Agunie A, Minda A, Mersha A, Sitotaw A, Weldeyohhans G, et al. Guideline recommendations for empirical antimicrobial therapy: an appraisal of research evidence for clinical decision-making in Ethiopia. Infect Dis Therapy. 2020;9(3):451–65.

    Google Scholar 

  31. Seyoum T, Alemayehu M, Christensson K, Lindgren HJPo. Provider-perceived benefits and constraints of complete adherence to antenatal care guideline among public health facilities, Ethiopia: a qualitative study. PLoS ONE. 2021;16(8):e0255297.

    CAS  PubMed  PubMed Central  Google Scholar 

  32. Yodit Abraham AA. The new innovative medical education system in Ethiopia: Background and development 2013 [Available from: https://www.ajol.info.

  33. Manyazewal T, Oosthuizen MJ, Matlakala MCJBo. Proposing evidence-based strategies to strengthen implementation of healthcare reform in resource-limited settings: a summative analysis. BMJ Open. 2016;6(9):e012582.

    PubMed  PubMed Central  Google Scholar 

  34. Avan BI, Berhanu D, Umar N, Wickremasinghe D. Schellenberg JJHp, planning. District decision-making for health in low-income settings: a feasibility study of a data-informed platform for health in India, Nigeria and Ethiopia. Health Policy Plann. 2016;31(suppl2):ii3–ii11.

    Google Scholar 

  35. Aynalem ZB, Yazew KG, Gebrie MHJPo. Evidence-based practice utilization and associated factors among nurses working in Amhara Region Referral Hospitals, Ethiopia. Plose One. 2021;16(3):e0248834.

    CAS  Google Scholar 

  36. Beshir MA, Woreta SA. Kebede MJIjoe-bh. Evidence-based practice among health professionals in hospitals of Northwest Ethiopia: a cross-sectional study. Int J Evid Based Healthc. 2017;15(4):161–70.

    PubMed  Google Scholar 

  37. Degu AB, Yilma TM, Beshir MA, Inthiran AJPo. Evidence-based practice and its associated factors among point-of-care nurses working at the teaching and specialized hospitals of Northwest Ethiopia: a concurrent study. Plose One. 2022;17(5):e0267347.

    CAS  Google Scholar 

  38. Hadgu G, Almaz S, Tsehay S, Almaz S, Tsehay SJAJNS. Assessment of nurses’ perceptions and barriers on evidence based practice in Tikur Anbessa specialized hospital Addis Ababa Ethiopia. Am J Nurs Sci. 2015;4(3):73–83.

    Google Scholar 

  39. Worku T, Yeshitila M, Feto T, Leta S, Mesfin F, Mezmur, HJBmi, et al. Evidence-based medicine among physicians working in selected public hospitals in eastern Ethiopia: a cross-sectional study. BMC Med Inf Decis Mak. 2019;19(1):1–8.

    Google Scholar 

  40. Yehualashet DE, Yilma TM, Jemere AT, Gedlu NMJAiME. Practice. Factors Associated with practicing evidence-based medicine among Medical Interns in Amhara Regional State Teaching Hospitals, Northwest Ethiopia: a cross-sectional study. Adv Med Educ Pract. 2021;12:843.

    PubMed  PubMed Central  Google Scholar 

  41. Abeysena C, Jayawardana P, Wickremasinghe R. Wickramasinghe UJJoe-bm. Evidence-based medicine knowledge, attitudes, and practices among doctors in Sri Lanka. J Evidence-Based Med. 2010;3(2):83–7.

    Google Scholar 

  42. Lafuente-Lafuente C, Leitao C, Kilani I, Kacher Z, Engels C, Canouï-Poitrine F, et al. Knowledge and use of evidence-based medicine in daily practice by health professionals: a cross-sectional survey. bmJ Open. 2019;9(3):e025224.

    PubMed  PubMed Central  Google Scholar 

  43. Bin Briek A, Webair H, Al-Tuhaify MJJFM. Assessment of physicians’ attitude, awareness and knowledge of evidence based medicine: an observational study from Yemen. J Family Med. 2014;1(3):5.

    Google Scholar 

  44. Ghahremanfard F, Nassaji M, Mirmohammadkhani M, Tanha A, Mosavi M, Ghaemi A, et al. Knowledge and attitude toward evidence-based medicine among medical students in Semnan, Iran. J Evidence-Based Med. 2014;7(1):32–7.

    Google Scholar 

  45. Aldugieman TZ, Alanezi RS, Alshammari WM, Al-Shamary YW, Alqahtani MM, Alreshidi FSJJoFM, et al. Knowledge, attitude and perception toward evidence-based medicine among medical students in Saudi Arabia: analytic cross-sectional study. J Family Med Prim Care. 2018;7(5):1026.

    PubMed  PubMed Central  Google Scholar 

  46. Abdulwadud O, Nyareza S, Nyarango P, Moyo SJGJoP, Sciences A. Awareness, attitudes and knowledge of evidence-based medicine: a cross-sectional baseline survey among academic healthcare practitioners in Namibia. Global J Pure Appl Sci. 2017;23(2):347–53.

    Google Scholar 

  47. Abdulwadud O, Tadesse F, Yilma G, Midekssa M, Ibraghimova IJTPAmj. Knowledge and experience with cochrane and evidence based medicine among health professionals in Debreberhan Referral Hospital in Ethiopia: a cross-sectional survey. Pan Afr Med J. 2018;30.

  48. Abera T, Vijaya MNSVVRM, Dhanaraju MD, Gunasekaran TJJoPHSR. Assessment of awareness and attitude of healthcare professionals towards the use of evidence-based medicine in the regional referral hospital. J Pharm Health Serv Res. 2016;7(3):199–204.

    Google Scholar 

  49. Snow JE, Leach MJ, Clare BAJJoC, Medicine I. Attitudes, skill and use of evidence-based practice among US western herbal medicine providers: a national survey. J Complement Integr Med. 2017;14(1).

  50. Williamson KM, Almaskari M, Lester Z, Maguire DJJfnipd. Utilization of evidence-based practice knowledge, attitude, and skill of clinical nurses in the planning of professional development programming. J Nurses Prof Dev. 2015;31(2):73–80.

    PubMed  Google Scholar 

  51. Barends E, Villanueva J, Rousseau DM, Briner RB, Jepsen DM, Houghton E, et al. Managerial attitudes and perceived barriers regarding evidence-based practice: an international survey. PLoS ONE. 2017;12(10):e0184594.

    PubMed  PubMed Central  Google Scholar 

  52. Dagne AH, Beshah MH, Kassa BG, Dagnaw EHJRH. Implementation of evidence-based practice and associated factors among nurses and midwives working in Amhara Region government hospitals: a cross-sectional study. Reproductive Health Journal. 2021;18(1):1–10.

    Google Scholar 

  53. Ammouri AA, Raddaha AA, Dsouza P, Geethakrishnan R, Noronha JA, Obeidat AA, et al. Evidence-based practice: knowledge, attitudes, practice and perceived barriers among nurses in Oman. Sultan Qaboos University Medical Journal. 2014;14(4):537–45.

    Google Scholar 

  54. Loai A, Khamis E. Awareness, attitudes and knowledge about evidence based medicine among doctors in Gaza: a cross sectional survey. East Mediterr Health J. 2017;23(9):626–31.

    Google Scholar 

  55. Bahammam MA. Linjawi AIJSmj. Knowledge, attitude, and barriers towards the use of evidence based practice among senior dental and medical students in western Saudi Arabia. Saudi Med J. 2014;35(10):1250–6.

    PubMed  PubMed Central  Google Scholar 

  56. Chagnon F, Pouliot L, Malo C, Gervais M-J, Pigeon M-ÈJIS. Comparison of determinants of research knowledge utilization by practitioners and administrators in the field of child and family social services. Implement Sci. 2010;5(1):1–12.

    Google Scholar 

  57. Hong J, Chen JJIJoER, Health P. Clinical physicians’ attitudes towards evidence-based medicine (EBM) and their evidence-based practice (EBP) in Wuhan, China. Int J Environ Res Public Health. 2019;16(19):3758.

    PubMed  PubMed Central  Google Scholar 

  58. Taylor SL, Dy S, Foy R, Hempel S, McDonald KM, Øvretveit J, et al. What context features might be important determinants of the effectiveness of patient safety practice interventions? bmJ Open. 2011;20(7):611–7.

    Google Scholar 

  59. Weng Y-H, Kuo KN, Yang C-Y, Liao H-H, Chen C, Lo H-L, et al. Effectiveness of national evidence-based medicine competition in Taiwan. BMC Med Educ. 2013;13(1):1–8.

    Google Scholar 

  60. Macedo CR, Macedo EC, Torloni MR, Atallah ÁNJJoEiCP. Evaluation of a national evidence-based health care course via teleconference in a developing country. Wiley Online Library. 2013;19(4):713-9.

  61. Fei J, Li Y, Gao W, Li JJBme. Efficacy of evidence-based medicine training for primary healthcare professionals: a non-randomized controlled trial. BMC Med Educ. 2018;18(1):1–9.

    Google Scholar 

  62. Ilic D, Nordin RB, Glasziou P, Tilson JK, Villanueva, EJBme. A randomised controlled trial of a blended learning education intervention for teaching evidence-based medicine. BMC Med Educ. 2015;15(1):1–10.

    Google Scholar 

  63. Crowther H, Lipworth W, Kerridge I. Evidence-based medicine and epistemological imperialism: narrowing the divide between evidence and illness. J Eval Clin Pract. 2011;17(5):868–72.

    PubMed  Google Scholar 

  64. Fritsche L, Greenhalgh T, Falck-Ytter Y, Neumayer H, Kunz RJB. Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine. The BMJ. 2002;325(7376):1338–41.

    CAS  PubMed  PubMed Central  Google Scholar 

  65. Greenhalgh T, Howick J, Maskrey NJB. Evidence based medicine: a movement in crisis? BMJ. 2014;348.

  66. Heneghan C, Mahtani KR, Goldacre B, Godlee F, Macdonald H, Jarvies D. Evidence based medicine manifesto for better healthcare. BMJ. 2017;23:357.

    Google Scholar 

  67. Maggio LA, Cate Ot, Irby DM, O’Brien BC. Designing evidence-based medicine training to optimize the transfer of skills from the classroom to clinical practice: applying the four component instructional design model. Acad Med. 2015;90(11):1457–61.

    PubMed  Google Scholar 

  68. Hunt JJNM. Johns Hopkins nursing evidence-based practice. ProQuest. 2012;19(7):8.

    Google Scholar 

  69. Babiker A, El Husseini M, Al Nemri A, Al Frayh A, Al Juryyan N, Faki MO, et al. Health care professional development: Working as a team to improve patient care. Sudan J Paediatrics. 2014;14(2):9.

    Google Scholar 

  70. Birhanu M, Gebrekidan B, Tesefa G, Tareke M. Workload determines workplace stress among health professionals working in felege-hiwot referral Hospital, Bahir Dar, Northwest Ethiopia. Journal of environmental and public health. 2018;2018.

  71. Fishman J, Yang C, Mandell DJIS. Attitude theory and measurement in implementation science: a secondary review of empirical studies and opportunities for advancement. Implement Sci. 2021;16(1):1–10.

    Google Scholar 

  72. Ramos KD, Schafer S, Tracz SMJB. Validation of the Fresno test of competence in evidence based medicine. The BMJ. 2003;326(7384):319–21.

    PubMed  PubMed Central  Google Scholar 

  73. Ilic D, Nordin RB, Glasziou P, Tilson JK, Villanueva E. Development and validation of the ACE tool: assessing medical trainees’ competency in evidence based medicine. BMC Med Educ. 2014;14(1):1–6.

    Google Scholar 

Download references

Acknowledgements

The authors are indebted to Mettu University to permit the required budget for this capacity-building training project. Moreover, the authors would like to extend their heartfelt thanks to healthcare professionals, trainers, data collectors, and supervisors who participated in this study.

Funding

The author(s) received no specific funding for this work.

Author information

Authors and Affiliations

  1. Department of Health Informatics, School of Public Health, College of Medicine and Health Sciences, Woldia University, Woldia, Ethiopia

    Habtamu Setegn Ngusie

  2. Department of Health Informatics, College of Health Sciences, Mettu University, Mettu, Ethiopia

    Mohammadjud Hasen Ahmed, Shuma Gosha Kanfie & Sisay Yitayih Kassie

  3. University of South-Eastern Norway, Post office box 235, N-3603, Kongsberg, Norway

    Shegaw Anagaw Mengiste

  4. German Cancer Research Center (DKFZ), Heidelberg, Germany

    Mihretu M. Kebede

  5. Department of Public Health, College of Health Sciences, Mettu University, Mettu, Ethiopia

    Shuayib Shemsu

  6. Department of Health Informatics, School of Public Health, College of Medicine and Health Sciences, Wollo University, Dessie, Ethiopia

    Mulugeta Hayelom Kalayou

  7. Faculty of Health and Social Sciences, University of South-Eastern Norway, Drammen, Norway

    Monika Knudsen Gullslett

Authors
  1. Habtamu Setegn Ngusie
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohammadjud Hasen Ahmed
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shegaw Anagaw Mengiste
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mihretu M. Kebede
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shuayib Shemsu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Shuma Gosha Kanfie
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Sisay Yitayih Kassie
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Mulugeta Hayelom Kalayou
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Monika Knudsen Gullslett
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

HSN, MHA, SYK, and SGK made significant contributions to the conception, design, data collection, supervision, data curation, investigation, data analysis, interpretation, and write-up of the manuscript. SAM, MHK, SS, MKG, and MMK have contributed to developing the proposal, validation, revising the manuscript, preparing figures, analysis, visualization, and interpretation of data as well. Finally, all authors (HSN, MHA, SAM, SS, MMK, SYK, MHK, SGK, and MKG) reviewed and approved the final manuscript.

Corresponding author

Correspondence to Habtamu Setegn Ngusie.

Ethics declarations

Ethics approval and consent to participate

Ethical clearance was obtained from the ethical review board of Mettu University College of Health Science with ethical reference number: RPG/90/13. Written informed consent was obtained from all study participants. The unique identifier (ID) given during data collection was recorded and accompanied by the questionnaire to document the written consent. The data collection procedure was also anonymous. Additionally, this study was conducted following the Declaration of Helsinki (See Additional file1).

Consent for publication

Not applicable.

Competing of interests

The authors have declared that no competing interests exist.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Additional file 1

. GREET 2023 checklist for EBM training in Southwest Ethiopia, based upon the TIDieR guidance

Additional file 2.

Supportive Table 1: EBM module and leaning objectives to providing training for healthcare professionals in southwest Ethiopia, 2022: developed based on different models and literatures.

Additional file 3.

Appendix A: Data collection tool.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ngusie, H.S., Ahmed, M.H., Mengiste, S.A. et al. The effect of capacity building evidence-based medicine training on its implementation among healthcare professionals in Southwest Ethiopia: a controlled quasi-experimental outcome evaluation. BMC Med Inform Decis Mak 23, 172 (2023). https://0-doi-org.brum.beds.ac.uk/10.1186/s12911-023-02272-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://0-doi-org.brum.beds.ac.uk/10.1186/s12911-023-02272-7

Keywords

BMC Medical Informatics and Decision Making

ISSN: 1472-6947

Contact us