- Research article
- Open Access
A mixed-methods evaluation of the MOREOB program in Ontario hospitals: participant knowledge, organizational culture, and experiences
BMC Health Services Research volume 19, Article number: 460 (2019)
MOREOB (Managing Obstetrical Risk Efficiently) is a patient safety program for health care providers and administrators in hospital obstetric units. MOREOB has been implemented widely in Canada and gradually spread to the United States. The main goal of MOREOB is to build a patient safety culture and improve clinical outcomes. In 2013, 26 Ontario hospitals voluntarily accepted provincial funding to participate in MOREOB. The purpose of our study was to assess the effect of MOREOB on participant knowledge, organizational culture, and experiences implementing and participating in the program at these 26 Ontario hospitals.
A convergent parallel mixed-methods study in Ontario, Canada, with MOREOB participants from 26 hospitals. The quantitative component used a descriptive pre-post repeated measures design to assess participant knowledge and perception of culture, administered pre-MOREOB and after each of the three MOREOB modules. Changes in mean scores were assessed using mixed-effects regression. The qualitative component used a qualitative descriptive design with individual semi-structured interviews. We used content analysis to code, categorize, and thematically describe data. A convergent parallel design was used to triangulate findings from data sources.
308 participants completed the knowledge test, and 329 completed the culture assessment at all four time points. Between baseline and post-Module 3, statistically significant increases on both scores were observed, with an increase of 7.9% (95% CI: 7.1 to 8.8) on the knowledge test and an increase of 0.45 (on a scale of 1–5, 95% CI: 0.38 to 0.52) on the culture assessment. Interview participants (n = 15) described improvements in knowledge, interprofessional communication, ability to provide safe care, and confidence in skills. Facilitators and barriers to program implementation and sustainability were identified.
Participants were satisfied with their participation in the MOREOB program and perceived that it increased health care provider knowledge and confidence, improved safety for patients, and improved communication between team members. Additionally, mean scores on knowledge tests for obstetric content and culture assessment improved. The MOREOB program can help organizations and individuals improve care by concentrating on the human and organizational aspects of patient safety. Further work to improve program implementation and sustainability is required.
There are over 380,000 births in Canada each year  and the majority occur in hospitals . Improvements in women’s health and health care, as well as advancements in prenatal and obstetric care have led to decreases in maternal and neonatal mortality rates . However, recent trends in maternal characteristics, including older age of mothers and higher rates of obesity, have resulted in increased rates of some maternal and neonatal adverse events . In 1999, the Institute of Medicine’s landmark report To Err Is Human identified adverse events, many of which are preventable, as a common cause of morbidity and mortality in hospitals and called for a coordinated strategy to improve patient safety . Although the report set a goal of reducing adverse events by 50% over five years, there remains room for improvement. For example, in Canada in 2014–2015, 1 in 18 hospital stays (or 138,000 hospitalizations) involved at least one occurrence of potentially preventable harm . While obstetric and newborn patients have a lower rate of harm compared to other patient groups (with a harm rate of 4.2 per 100 patients, and 1.0 per 100 patients, respectively) , obstetrical departments are often a focus for patient safety and quality improvement. There is a need to prioritize patient safety in obstetrics due to the unique challenges of working in labour and delivery , the fact that an adverse outcome may affect more than one patient, and due to the high cost of liability insurance and claims [8, 9], with obstetrics accounting for nearly half of the Healthcare Insurance Reciprocal of Canada’s (HIROC) liability claims and patient compensation payments .
The MOREOB (Managing Obstetrical Risk Efficiently) program was developed by the Society of Obstetricians and Gynaecologists of Canada (SOGC), and is currently administered by Salus Global, a healthcare consulting and implementation firm established by the SOGC and HIROC in 2007 . The MOREOB program aims to create a culture of patient safety in obstetrical units by using high reliability organization (HRO) principles ,including awareness of systems that influence patient care and outcomes, focusing on near-misses as opportunities to improve processes, a culture that promotes open communication and teamwork, and a commitment to ongoing training and learning . The program contains three evidence-informed modules: Learning Together, Working Together, and Changing the Culture . The program requires that all obstetric team members at enrolled hospitals (e.g., nurses, midwives, family physicians, obstetricians, administrators) jointly participate in the three modules, each of which is implemented over approximately one year. MOREOB uses a “train-the-trainer” approach whereby the hospital recruits a “core team” of interdisciplinary team members, who are then trained and supported by Salus Global to implement the program at their respective hospitals. Standardization of implementation across sites is facilitated by detailed program manuals and implementation responsibilities for the core team, orientation sessions by MOREOB facilitators prior to implementation of each module, and ongoing support by program consultants . Additional information about the MOREOB program goals is provided in Additional file 1.
To date, over 300 North American hospitals have participated in the MOREOB program , but there are limited studies and variable outcomes. In Canada, the province of Alberta implemented the MOREOB program in all hospitals in 2004, and a subsequent evaluation found statistically significant reductions in perineal tears, length of stay, and neonatal severe morbidity rates . No significant changes were observed for postpartum infection, postpartum hemorrhage with caesarean section, or fetal mortality . One Ontario study reported that participation in the MOREOB program resulted in a significant decrease in the number of reportable events and associated costs for the healthcare liability insurer . Another Ontario study found that participation in the MOREOB program did not translate to an observable decrease in adverse maternal or neonatal birth outcomes . However, data collected from Canadian sites showed increases in participant knowledge and organizational safety culture .
In 2002 in the province of Ontario, hospitals providing obstetrical services began to voluntarily implement the program, and by 2012, 67 hospitals were involved with MOREOB. In 2013, the Ontario Ministry of Health and Long-Term Care (MOHLTC) offered to fund the program for the remaining 34 provincial hospitals that were not yet engaged with MOREOB, and 26 hospitals accepted and completed the program. The implementation of the MOREOB program at these Ontario hospitals provided an opportunity to conduct a mixed-methods evaluation of the program implementation. This evaluation aimed to assess the outcomes and local implementation process for the MOREOB program in the Ontario context, provide data to facilitate comparison between provincial settings, and contribute qualitative data for more in-depth information on participant experiences with the program.
This study used a convergent parallel mixed-methods design [19, 20]. Simultaneous data collection and subsequent analysis and integration of data from quantitative (i.e., surveys) and qualitative (i.e., semi-structured interviews) sources allowed us to compare and corroborate our findings from different data sources (Fig. 1). Our research team was not involved in implementing the MOREOB program in any of the sites but conducted the comprehensive evaluation of the program with two main goals, specifically to evaluate (1) the program outcomes and (2) the implementation process (Table 1).
This paper reports the methods and results for objectives 2a, 2b, and 2c only. The outcomes of knowledge and culture were selected as they were aligned with the MOREOB program goals (Additional file 1), and the data available from existing surveys administered routinely as part of participation in the MOREOB program. The evaluation of the implementation process aimed to identify barriers and facilitators to implementing and participating in the program to provide insight into why the program may or may not have worked in different settings and to identify areas for improvement.
We used a descriptive pre-post repeated measures design to assess the effect of the MOREOB program on participants’ knowledge about obstetrical core content contained in the MOREOB program and to assess the effect of the MOREOB program on participants’ perceptions of their organizational culture.
We used data from two existing surveys administered routinely to all participants by Salus Global as part of involvement in the MOREOB program:
An online, standardized, 75-item multiple-choice knowledge test previously developed by the MOREOB program which assesses core clinical knowledge covered in the program. The knowledge test had a maximum score of 93, however for ease of interpretation, we transformed the score to a percentage (D Walker, 2015, unpublished; ).
A previously developed and validated online 54-item Culture Assessment Survey (CAS)  to assess participants’ perceptions of the culture of patient safety in their organization. As designed, the CAS is divided into six categories (patient safety is everyone’s priority, teamwork, valuing individuals, open communication, learning, and empowering people), each of which measures aspects of patient safety in obstetrics. Nine questions are included in each category, with each question scored on a scale of 1 to 5. The mean response to the nine questions on the 1 to 5 scale is used as the score for each category. The overall score for the CAS is then the mean score on the 1 to 5 scale for the six data elements, and therefore the overall score for the CAS is presented on a scale from 1 to 5 with a maximum score of 5 . The total CAS score as a whole, as well as the scores on the six categories were found to have good internal consistency .
The knowledge test and CAS were administered at four time points; pre-MOREOB, post-Module 1, post-Module 2, and post-Module 3 (Fig. 1).
As a standard component of participation in the MOREOB program, all MOREOB participants from the 26 hospitals were asked to complete a knowledge test and the CAS at four time points while progressing through the program. In addition to participant scores, the database from Salus Global contained information about job title, hospital site, and a unique identifier which allowed for participant responses to be linked over time. Only those participants who completed the knowledge test and/or CAS at all four time points were included in the final analysis.
MOREOB participants at the 26 sites completed the standardized knowledge-based assessment tool and CAS at each of the four time points as they progressed through the program. Data were collected by Salus Global on their MOREOB secure on-line platform and released to the research team for analysis.
We summarized items from the baseline questionnaire and after each of the three modules using descriptive statistics (means and standard deviations, or frequencies and percentages, as appropriate). Changes in mean scores across the four time points were assessed using linear mixed-effects regression analysis. Repeated measures were accounted for using an unstructured covariance pattern matrix, and clustering at hospital sites was accounted for using a random intercept. Least square means and pairwise comparisons were assessed at the 1% level. All analyses were carried out using SAS v. 9.4.
We used a qualitative descriptive design [22, 23]. This design aligned with our evaluation objectives to provide an accurate and comprehensive summary of participants’ perceptions of the effect of the MOREOB program on health care provider knowledge, organizational culture, and barriers and facilitators to implementing and participating in the MOREOB program [22, 24].
A 14-question semi-structured interview guide (Additional file 2) was used to gather in-depth information about interprofessional team experiences with the MOREOB program. The research team reviewed the initial draft interview guide, with modifications made to increase comprehensiveness and clarity (e.g., adding additional probes to elicit information on more concepts, changing wording to make the questions more easily understood). Minor modifications to the guide were made throughout the qualitative data collection period to reflect the timing of the interview in relation to program implementation and to probe new themes emerging from previously conducted interviews.
We used a purposeful sampling approach to recruit interview participants . Specifically, individuals were eligible to participate in an interview if they were (1) health care providers (i.e., nurse, family physician, obstetrician, midwife) and/or administrators/leaders (i.e., nursing or medicine); and (2) participants in the MOREOB program and/or involved in implementing the program as a member of the core team at one of the 26 sites during the two interview recruitment periods (January–April 2015 and April–July 2016).
Permission from each of the 26 hospitals was obtained for Salus Global to release the names and contact information of the MOREOB core team chairs to the research team for the purpose of interview participant recruitment. Interviews were conducted at two time points (post-Module 2 and post-Module 3) to capture the variety of experiences as the MOREOB program progressed. At each of the two time points, an email with an attached recruitment poster was sent by the research team to the MOREOB core team chair at each of the 26 hospitals, with instructions to circulate the invitation to all team members participating in the MOREOB program at their site. Interested participants were instructed to contact the principal investigator to learn more about the study, ascertain eligibility, and schedule a time for an interview. Reminders were sent by the research team to the MOREOB core team chairs to maximize responses. Individuals could participate in interviews at one or both time points. We welcomed participants who volunteered to be interviewed a second time as an opportunity to learn about differences they experienced between modules.
While in qualitative research there are no standardized rules for sample size, we used the concept of information power for guidance . For example, our study aims were relatively narrow (we aimed to explore key dimensions and implementation of the MOREOB program), our purposive sample of participants were specific to our study aims (all had experience participating in and/or implementing the MOREOB program), and we anticipated rich dialogue with participants due to the use of interviewers experienced in both qualitative research and the clinical practice area. These factors, in combination with our experience from past provincial program evaluations, led us to predict that we would need up to 20 participants to provide sufficient information power to answer our research questions.
Two research staff (one midwife, one nurse) experienced in qualitative interviewing conducted the telephone interviews. Informed written consent was obtained prior to each interview. Interviews lasted an average of 34 min. The first round of interviews (n = 9) was conducted between March and April 2015 to learn about participants’ experience with modules 1 and 2 of the MOREOB program. The purpose of conducting the first round of interviews after module 2 was to allow sufficient time for participants to be able to experience and report on any perceived changes to culture and clinical processes in their setting. The second round of interviews (n = 10) was conducted between April and July 2016 to learn about participants’ cumulative experiences after completion of module 3 of the MOREOB program and plans for sustainability. Four health care providers were interviewed during both round one and two, meaning that there were a total of 15 unique interviewees. Interviewing, transcription, and analysis proceeded concurrently to permit follow-up of issues emerging from the data, and allow probing of emerging themes in subsequent interviews [22, 23, 26].
Digital recordings of the semi-structured interviews were transcribed verbatim. Data were imported into NVivo 11™ . In alignment with our qualitative descriptive study design, we used conventional content analysis to code and analyze our data [22, 28, 29]. Initially, interviews were analysed independently by two study team members. We used an inductive process whereby the codes emerged from the data itself, rather than a pre-existing framework. Specifically, the two study team members independently read each transcript, followed by assigning preliminary codes (i.e., labels that represent the content of words and excerpts in the transcripts ). The two coders met every three to five transcripts to compare their coding, reach consensus, and develop and revise the coding scheme. All discrepancies were resolved through discussion between the two coders. As analysis progressed, codes were continually reviewed to identify similarities and differences, and conceptually similar codes were grouped into broader categories . Lastly, we looked for common underlying meaning between categories and created broader themes and sub-themes, which contributed to answering “how” participants experienced the MOREOB program . The analysis of data collected in phase 1 was used to inform revisions to the interview guide for phase 2. For example, questions were added to elicit feedback on perceived differences between modules and on plans for sustainability of the program, and additional probes on key emergent themes such as health care provider confidence were included. Research team members met to discuss the coding scheme, emerging themes and sub-themes, and to build consensus regarding study findings.
In total, we analyzed data from 308 respondents for the knowledge test, 329 respondents for the CAS, and 19 semi-structured interviews (15 participants) (Fig. 2).
The number of respondents at each time point is shown in Table 2. Of the 962 respondents who completed the knowledge test at any time point, 308 (32%) completed it at all four time points. Of the 988 respondents who completed the CAS at any time point, 329 (33.3%) completed it at all four time points.
Knowledge scores increased by a mean of 7.9 percentage points (95% CI: 7.1 to 8.8) from baseline to post-Module 3 and CAS scores increased by a mean of 0.45 (on a scale of 1 to 5, 95% CI: 0.38 to 0.52). Both increases were statistically significant. Least square mean estimates with standard errors across the four time points are presented in Fig. 3, for the knowledge test and CAS scores.
Increases in knowledge test and CAS scores for each professional group are presented in Table 3 The one respondent in the knowledge test group and two in the CAS group who selected “other” for profession were excluded from this analysis due to low numbers. Increases in both scores were noted for the three professional groups presented, with nurses experiencing the largest increase in knowledge scores, with a mean increase of 9.06 percentage points (95% CI 8.05 to 10.07). Physicians and residents had the greatest increase in CAS scores, with a mean increase of 0.72 on a scale of 1 to 5 (95% CI: 0.57 to 0.88).
Between March 2015 and July 2016, 19 telephone interviews were conducted with a total of 15 health care providers from 11 sites (Table 4).
Overall, participants reported being satisfied with the MOREOB program. Five main themes and their respective sub-themes were identified: communication, knowledge, safer care, increased confidence, and program implementation (Fig. 4). Table 5 provides participant quotes to support our findings .
Theme 1: communication
An important focus of the MOREOB program was improving communication strategies. Participants identified specific communication strategies learned through the MOREOB program implemented in practice, including ‘Take Five’ and ‘SBAR’ (Situation, Background, Assessment, Recommendation), which were implemented during debriefing about cases.
Participants perceived increased interprofessional communication, resulting from bringing the different professions together to participate in the program, sometimes for the first time. These opportunities for interprofessional communication and teamwork led to participants perceiving a more familiar and collegial work environment.
Theme 2: knowledge
Participants valued the knowledge itself gained from the MOREOB program. MOREOB was described as a comprehensive, time efficient, and accessible resource as all information is contained in one place, both in-print and online. In addition, participants perceived that the content of MOREOB had been well-researched, was evidence-based, and relevant to the local context.
Participants described how participating in the MOREOB program allowed all team members to share the same knowledge and language, put everyone “on the same page,” and gave everyone a common reference point.
All participants described how MOREOB increased their knowledge base and that of their colleagues. For some, this increase in knowledge was a reminder of the importance of ongoing training to maintain current knowledge and skills. The MOREOB program was described as being particularly helpful in a small hospital setting with low birth volumes where it can be difficult for new obstetrical nurses to gain sufficient experience in obstetrics.
Theme 3: safer care
Participation in MOREOB resulted in the perception that the health care team was providing safer care to patients. Participants identified concrete changes made on their units due to their participation in MOREOB. Examples of concrete changes included development or updating of policies and procedures, creating disaster carts with appropriate equipment, changes to clinical practices, posting algorithms on walls for clinicians to view, conducting case audits, and utilizing debriefing methods.
Participants described how MOREOB changed the way they delivered care. First, participants described improved collaboration with their colleagues, which resulted in better ‘team work, openness, and transparency’, which ultimately improves safety for patients. For those sites who described pre-existing hierarchical relationships between obstetrical care providers, participants described a decrease in hierarchy within their team. However, a couple of sites described still struggling with hierarchy and were still working to change the culture at their sites. In addition, participants identified that participation in the MOREOB program improved the patients’ experience by improving their collaboration with women and emphasizing the importance of patient centred care.
Generally, participants could not speak definitively as to whether their hospital’s participation in the MOREOB program had led to improved outcomes such as decreased maternal and infant morbidity and mortality, as they did not have data to refer to during the interview. One site had these data and stated they observed a reduction in critical events since participating in MOREOB. Overall, participants believed that the increased knowledge and concrete changes were ultimately improving care, outcomes, and patient experience.
Theme 4: increased confidence
Participants described the impact MOREOB had on their self-confidence, particularly nurses who were described as being empowered through the MOREOB program by increasing their knowledge and skills and improving their team communication. In addition, participants described an increase in their confidence in the knowledge, skills, and judgments of their team members. As MOREOB participants improved communication, increased their knowledge base, and improved their ability to provide safer care, their confidence in themselves and others increased.
Theme 5: program implementation
Participants described different strategies used for implementing the MOREOB program, including use of huddle boards, staff emails and staff newsletters. In general, participants described following the MOREOB activities as intended, with sites modifying the activities to make it more relevant for their setting or to fit it into limited time frames as needed. Some participants stated that the MOREOB program plan was not clear, even after attending the orientation, and felt that additional detail on upcoming activities would be beneficial.
Participants described several influencing factors that affected how easy or difficult it was to implement the MOREOB program in their context. For example, buy-in was increased when team members were interested in the content of the module and “hungry for new knowledge.” Participants also identified organizational-level barriers including limited time of staff and lack of funding to pay staff for their time to participate in the program, and a baseline organizational culture that was more resistant to change. Organizational-level facilitators included engaged hospital administration and the presence of MOREOB champions who were highly committed to the implementation of the program.
Participants described varying levels of awareness regarding the sustainability of the MOREOB program at their hospital once their three-year funding ended. Participants were keen to keep MOREOB active at their hospital, and many sites were pursuing continued involvement, either formally with Salus Global, or when funding was not available, through internally-led education programs and skills drills. Regardless of the specific strategy, sustainability of the knowledge and skills gained through MOREOB was identified as a priority.
Staff turnover and new hospital priorities were identified as ongoing challenges to sustainability of the program. As new people joined the unit, they had not participated in the initial MOREOB program modules with the remainder of the team which presented challenges as the team was “storming and forming” all over again. A key challenge for sustainability of the program was cost, with participants perceiving that the program should be kept at a reasonable price to ensure institutions and providers can benefit from it.
Our mixed-methods study evaluated the effect of the MOREOB program on participant knowledge, perception of organizational culture, and experiences implementing and participating in the program. Findings from both the quantitative and qualitative components of this study demonstrated increases in knowledge and positive perceptions of organizational culture. Findings from the qualitative component revealed that participants spoke positively of the program and the benefits of participating. In addition, barriers and facilitators to implementation and sustainability were identified.
The increases in knowledge scores occurring for all professional groups resonate with findings from our qualitative analysis, with participants describing MOREOB benefits such as increased knowledge and sharing a common knowledge, as well as increased confidence in themselves and in their teammates as a result of this increased knowledge.
Data collected in a previous study revealed similar results on the knowledge tests in Canadian hospitals after MOREOB implementation, with all professional groups showing an increase in knowledge from baseline to post-Module 3 . As in our evaluation, nurses were the professional group with the largest increase in knowledge test scores from baseline to post-Module 3 . This may be partially explained by the small size of the hospitals included in our sample, with approximately 80% providing level 1 (low-risk) care and approximately 70% having a birth volume of less than 500 per year. At such small sites, many nurses practice in multiple areas (e.g., medical/surgical) and are cross-trained to provide obstetrical care as needed. Conversely, physicians and midwives practicing at these sites specialize in obstetrics and thus may have had a greater depth of baseline knowledge in this area.
Participant perception of organizational culture
Our study showed an overall significant increase in the safety culture from baseline to post-Module 3, with scores increasing almost half a point out of 5 from pre-MOREOB levels. Likewise, in the previous Canadian evaluation, there was a significant increase on all six elements of the CAS . Our qualitative findings were aligned with these results, with participants giving examples of improvements related to the safety culture in their obstetric units, including strengthened teamwork and communication.
Team work, which is one of six categories on the CAS, has been identified as an essential element for reducing maternal and infant morbidity and mortality, as well as improving patient experience in obstetrics . Characteristics of team training to improve obstetrical outcomes include use of in-house rehearsals and debriefing, interprofessional training, and access to and training on evidence-based clinical guidelines , all of which are consistent with features of the MOREOB program and may have facilitated the positive culture changes we saw in our evaluation.
From the qualitative data, we learned that despite participation in the MOREOB program, some sites still struggled to overcome issues with hierarchy on the interdisciplinary team, which is important as it can prevent health care providers from speaking up . The ongoing challenges with hierarchy we observed in our interviews may not be surprising, given that hierarchal gradients continue to exist in many health care settings, despite attempts to reduce these hierarchies within and between professions . One industry that has successfully implemented interventions to reduce hierarchy between team members and improve culture and safety is aviation . Specifically, the use of Crew Resource Management (CRM) training helps to flatten the hierarchy, minimize power differentials, and improve communication by training all crewmembers, regardless of role, seniority, or experience, to speak up with safety concerns [33, 34]. Parallels have been drawn between aviation and healthcare related to safety management [35, 36] and the application of CRM training in health care was recommended in the 1999 To Err is Human Report . Examples of CRM in the field of obstetrics specifically can be found in the literature [7, 37,38,39,40]. While the MOREOB program uses HRO principles  applied in the aviation industry, additional strategies and ongoing training opportunities beyond the 3-year MOREOB program may be required for sites to continue to decrease interprofessional hierarchy and improve their ability to work and communicate as a team.
The varying experiences in culture change found in our qualitative interviews speak to the importance of the context in which the MOREOB program is being implemented. The influence of site context on culture change has been observed in other healthcare settings, with a longitudinal mixed-methods study of 10 hospitals participating in a quality collaborative intervention reporting distinct differences between the hospitals with substantial culture change versus those without. Hospitals with positive culture change employed an interprofessional and inter-hierarchal approach, reported high levels of participation, and utilized strategies for managing conflict and maintaining engagement . Although the MOREOB program aims to improve the safety culture in birthing units, the extent to which this can be improved may be influenced by specific hospital and team characteristics.
While our quantitative and qualitative findings support the effectiveness of the MOREOB program in improving several dimensions of a culture of safety, it is important to acknowledge that other patient safety initiatives in obstetrics, such as CRM team training, the Comprehensive Unit-based Safety Program (CUSP), and other locally developed quality improvement initiatives, have also demonstrated improvements in health care provider safety attitudes [37, 38, 42, 43]. Examples of key elements of these other initiatives included implementing an obstetric patient safety nurse [37, 42] and an obstetrics patient safety committee [37, 42, 43], using interdisciplinary team training [37, 38, 42, 43], and implementing improved communication systems and strategies [37, 38, 42, 43]. All reported improvements in safety culture after implementation of their respective patient safety initiatives, with significant improvements in teamwork culture [37, 42], safety culture [37, 42], job satisfaction [42, 43], working conditions , and perceptions of management [37, 42, 43].
Lastly, there is evidence to suggest that positive workplace and organizational culture is associated with improved patient outcomes including decreased mortality rates, adverse events, readmission rates, and increased patient satisfaction across a variety of health care settings . While our study found a significant increase in the CAS which was largely supported by our qualitative findings, it remains unknown how this positive change in culture translates to clinical outcomes in our setting.
Barriers and facilitators to implementing and participating in MOREOB program
Overall, interview participants expressed satisfaction with their participation in the MOREOB program. Several individual- and organizational-level facilitators and barriers to implementation and maintenance of the MOREOB program were identified, some of which are consistent with factors reported in the literature, and align with existing theoretical frameworks . For example, participants indicated that social influences  such as the effect of the core team and champions, influenced their team’s success implementing the MOREOB program. This was similar to lessons learned in another Canadian evaluation, where the hospitals’ MOREOB core team was identified as essential to driving the participation of members . Second, our participants described the effect of the environmental context and resources  such as the availability of funding to support the MOREOB program and baseline organizational culture. With limited financial, physical, and human resources within health care organizations, running and sustaining the MOREOB program can present a challenge.
Previous research suggests potential strategies to facilitate implementation and uptake of the MOREOB program include increased leadership engagement, conducting a baseline barriers assessment, and use of ongoing monitoring and evaluation at a local level. First, leadership has been identified as critical for developing a culture of safety , and in the case of MOREOB, the support of senior administration is key to successful implementation of the program, through direct participation on the core team, and through supporting their work . Next, an assessment prior to MOREOB program implementation may assist hospitals to identify individual- and organization-level barriers, and identify existing strengths and resources that may be used to facilitate implementation . Lastly, participants in our study were largely unable to definitively state whether their team’s participation in the MOREOB program had translated to improved patient outcomes at their hospital. Evaluating key outcomes using local data is an important component of the knowledge translation process  to improve patient safety. Ongoing monitoring and evaluation of hospital-based patient safety outcomes may assist with keeping teams engaged with the MOREOB program by showing participants where they are (or are not) making a difference, and identifying areas of improvement for the MOREOB core team related to program implementation and participation.
While our evaluation showed positive findings from the participant surveys and qualitative interviews, understanding the actual impact on perinatal outcomes is important, because a patient safety program should really translate into improved outcomes. As part of the larger evaluation we reported that these findings did not translate into a statistically significant reduction in adverse maternal and neonatal outcomes at the provincial level . Despite the lack of reduction in adverse outcomes, it seems clear from the results we present in this paper that the MOREOB program increased knowledge, changed the patient safety culture, and improved the work environment at many participating hospitals. For example, participants in the interview component of our study identified other improvements that were not within the scope of this current study to assess quantitatively. Specifically, interview participants spoke of perceived improvements from the MOREOB program related to patient experience, health care provider confidence, and interprofessional communication. These additional benefits are important to individuals and health systems because: (1) improved patient experience is an important goal for health care systems , and (2) health care provider confidence and communication are important individual-level factors influencing “speaking up behavior” , an essential component of a patient safety culture . Given the potential importance of these additional health care provider and patient outcomes, future work to comprehensively assess the effect of the MOREOB program on these constructs using validated measures may be warranted.
Considering the overall positive results from the surveys and interviews, it is not surprising that most participants expressed a desire to keep the program active at their centers. Future work to create a program model that is affordable and sustainable for participating hospitals is needed. Since the completion of this evaluation, Salus Global has launched MOREOB 2.0 and MOREEX, which can support health care teams in acute settings beyond obstetrics . Future evaluations of the revised MOREOB 2.0 program may consider our lessons learned from the Alberta  and Ontario provincial evaluations as well as the Canadian evaluation , and eventually provide comparative data.
Strengths and limitations
The use of a mixed-methods approach is an important strength of this study. We used data from two surveys, carried out at four time points as participants progressed through the MOREOB program. As well, we included data from qualitative interviews carried out with 15 participants, including an administrator, nurses, midwives, and physicians, providing rich in-depth information on the MOREOB program. The fact that findings from both methods were convergent regarding improvements in knowledge and safety culture supports the effect of the program on these outcomes.
To our knowledge, ours is only the second paper to publish results of the knowledge test and CAS at baseline and after implementation of the three MOREOB modules. Our findings are similar to those published previously, and are an important addition to the literature due to the limited methodologic and sampling information provided in the earlier publication . Further, we have used a more robust analytic approach for analyzing repeated measures survey data.
A limitation is the inclusion of only those 26 hospitals who received provincial funding to implement the MOREOB program in 2013. These sites likely differ from the 67 hospitals that voluntarily initiated participation in the MOREOB program prior to 2013, independently of any funding. As well, for the knowledge test and CAS, all of the respondents were participants in the MOREOB program, meaning that we do not have data on non-participants. This is problematic, in that one would expect knowledge scores to increase over a four-year time period, with increased time in the workplace. Due to this limitation, we cannot attribute the increases we observed solely to participation in the MOREOB program. Additionally, the majority of respondents did not complete the CAS and knowledge test at all four time points. This is not surprising given staff turnover rates on obstetric units, however, our analyses were therefore limited to the one third of respondents who completed all of the assessments. Furthermore, it is difficult to know the clinical effect of the increases in knowledge and culture that were observed. Another limitation is the lack of data available on survey respondents, for example, we do not know their age or number of years in the workforce, which could influence survey scores. While our qualitative interviews included participants from diverse hospitals and professional backgrounds, only one obstetrician participated in an interview.
Lastly, it is important to acknowledge that our sample of 15 unique interviewees only included individuals from 11 out of the 26 potential hospitals. While efforts to maximize recruitment were made by communicating through the chair of the MOREOB core team at each site, by sending multiple email reminders, and by accommodating all participant schedules for interviews, we still only managed to capture participants from about 40% of eligible hospitals. Nonetheless, the information we obtained from these interviews provided valuable insight into the MOREOB program and its implementation to answer our research questions, although these findings may not be transferable to individuals and teams at the remaining sites.
Results of this mixed-methods study suggest that participants were satisfied with their participation in the MOREOB program. Barriers and facilitators to implementation and sustainability of the MOREOB program were identified. Strategies to address barriers, maximize facilitators, and enhance sustainability of the MOREOB program are needed. Our results also suggest that the program may have a positive effect on participant knowledge and organizational safety culture. Further work to understand how these improvements translate to outcomes for patients and health care providers is needed.
Availability of data and materials
The qualitative data analyzed during this study (i.e., interview transcripts) are not publicly available due to them containing information that could compromise research participant privacy/consent.
The data that support the findings from the survey component of this study are available from Salus Global but restrictions apply to the availability of these data, which were used under license for the current study, and so are not publicly available. Data are only available with permission from Salus Global and the participating hospitals.
The American College of Obstetricians and Gynecologists
Advances in Labour and Risk Management
Culture Assessment Survey
Crew Resource Management
Healthcare Insurance Reciprocal of Canada
High Reliability Organization
Ministry of Health and Long-Term Care
- MOREOB :
Managing Obstetrical Risk Efficiently
Situation, Background, Assessment, Recommendation
Society of Obstetricians and Gynaecologists of Canada
Statistics Canada. Births, 2015 and 2016. Ottawa; 2018. https://www150.statcan.gc.ca/n1/en/daily-quotidien/180430/dq180430f-eng.pdf?st=SN97LHtg. Accessed 12 June 2019.
Canadian Institute of Health Information (CIHI). Giving birth in Canada: a regional profile. Ottawa; 2004. https://secure.cihi.ca/free_products/GBC2004_regional_e.pdf. Accessed 8 Apr 2019
Public Health Agency of Canada. Perinatal health indicators for Canada 2017. Ottawa; 2017. http://publications.gc.ca/collections/collection_2018/aspc-phac/HP7-1-2017-eng.pdf. Accessed 8 Apr 2019.
Liu S, Joseph KS, Bartholomew S, Fahey J, Lee L, Allen AC, et al. Temporal trends and regional variations in severe maternal morbidity in Canada, 2003 to 2007. J Obs Gynaecol Can. 2010;32:847–55.
Kohn LT, Corrigan JM, Donaldson MS. To err is human: building a safer health system. Washington, DC: National Academy Press; 1999.
Canadian Institute for Health Information (CIHI) and Canadian Patient Safety Institute (CPSI). Measuring patient harm in Canadian hospitals. Ottawa, 2016. https://www.cihi.ca/sites/default/files/document/cihi_cpsi_hospital_harm_en.pdf. Accessed 8 Apr 2019.
Nielsen PE, Goldman MB, Mann S, Shapiro DE, Marcus RG, Pratt SD, et al. Effects of teamwork training on adverse outcomes and process of care in labor and delivery: a randomized controlled trial. Obstet Gynecol. 2007;109:48–55.
Pettker CM, Thung SF, Norwitz ER, Buhimschi CS, Raab CA, Copel JA, et al. Impact of a comprehensive patient safety strategy on obstetric adverse events. Am J Obstet Gynecol. 2009;200:492.e1–8.
Pettker CM, Thung SF, Lipkind HS, Illuzzi JL, Buhimschi CS, Raab CA, et al. A comprehensive obstetric patient safety program reduces liability claims and payments. Am J Obstet Gynecol. 2014;211:319–25.
The Healthcare Insurance Reciprocal of Canada (HIROC) and the Canadian Medical Protective Association (CMPA). Delivery in focus. Strengthening obstetrical care in Canada: Ottawa; 2018. https://www.cmpa-acpm.ca/static-assets/pdf/research-and-policy/system-and-practice-improvement/final_18_hiroc_cmpa_obstetrical_report-e.pdf. Accessed 8 Apr 2019
Salus Global Corporation. History. 2019. https://www.salusglobal.com/history. Accessed 6 May 2019.
Salus Global Corporation. MOREOB. 2018. https://www.moreob.com/. Accessed 21 Sep 2018.
Hines S, Luna K, Lofthus J, Lewin group. Becoming a high reliability organization: operational advice for hospital leaders. Rockville, MD; 2008. https://archive.ahrq.gov/professionals/quality-patient-safety/quality-resources/tools/hroadvice/hroadvice.pdf. Accessed 8 April 2019.
Salus Global Corporation. The MOREOB program overview 2012–2013. London; 2013.
Thanh NX, Jacobs P, Wanke MI, Hense A, Sauve R. Outcomes of the introduction of the MOREOB continuing education program in Alberta. J Obstet Gynaecol Can. 2010;32:749–55.
Geary M, Ruiter PJA, Yasseen AS. Examining the effects of an obstetrics interprofessional programme on reductions to reportable events and their related costs. J Interprof Care. 2018;1(9). https://0-doi-org.brum.beds.ac.uk/10.1080/13561820.2018.1543255 epub ahead of print.
Weiss D, Fell DB, Sprague AE, Walker MC, Dunn S, Reszel J, et al. Effect of implementation of the MOREOB program on adverse maternal and neonatal birth outcomes in Ontario, Canada: a retrospective cohort study. BMC Pregnancy Childbirth. 2019;19:151. https://0-doi-org.brum.beds.ac.uk/10.1186/s12884-019-2296-5.
Milne J, Walker D, Vlahaki D, Milne J, Walker D, Vlahaki D. Reflections on the Canadian MORE (OB) obstetrical risk management programme. Best Pr Res Clin Obs Gynaecol. 2013;27:563–9.
Fetters MD, Curry LA, Creswell JW. Achieving integration in mixed methods designs-principles and practices. Health Serv Res. 2013;48:2134–56.
Creswell JW. Research design - qualitative, quantitative, and mixed methods approaches. 4th ed. Thousand Oaks, California: SAGE Publications, Inc; 2014.
Milne JK, Bendaly N, Bendaly L, Worsley J, Fitzgerald J, Nisker J. A measurement tool to assess culture change regarding patient safety in hospital obstetrical units. J Obstet Gynaecol Can. 2010;32:590–7.
Sandelowski M. Whatever happened to qualitative description? Res Nurs Health. 2000;23:334–40.
Sandelowski M. What’s in a name? Qualitative description revisited. Res Nurs Health. 2010;33:77–84.
Kim H, Sefcik JS, Bradway C. Characteristics of qualitative descriptive studies: a systematic review. Res Nurs Health. 2017;40:23–42.
Malterud K, Siersma VD, Guassora AD. Sample size in qualitative interview studies. Qual Health Res. 2016;26:1753–60.
Thorne S. Interpretive description: qualitative research for applied practice. 2nd ed. New York: Routledge Taylor & Francis Group; 2016.
QSR International Pty Ltd. NVivo 11 for Windows. 2015.
Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15:1277–88.
Bradshaw C, Atkinson S, Doody O. Employing a qualitative description approach in health care research. Glob Qual Nurs Res. 2017;4:1–8.
Graneheim U, Lundman B. Qualitative content analysis in nursing research: concepts, procedures and measures to achieve trustworthiness. Nurse Educ Today. 2004;24:105–12.
Cornthwaite K, Edwards S, Siassakos D. Reducing risk in maternity by optimising teamwork and leadership: an evidence-based approach to save mothers and babies. Best Pract Res Clin Obstet Gynaecol. 2013;27:571–81.
Guise J-M, Segel S. Teamwork in obstetric critical care. Best Pract Res Clin Obstet Gynaecol. 2008;22:937–51.
Green B, Oeppen RS, Smith DW, Brennan PA. Challenging hierarchy in healthcare teams – ways to flatten gradients to improve teamwork and patient care. Br J Oral Maxillofac Surg. 2017;55:449–53.
Government of Canada. Chapter 7: Crew Resource Management (CRM). In: Development and implementation of an Advanced Qualification Program (AQP). Ottawa; 2010. https://www.tc.gc.ca/eng/civilaviation/standards/commerce-aqp-chapter7-menu-196.htm. Accessed 8 April 2019.
Kapur N, Parand A, Soukup T, Reader T, Sevdalis N. Aviation and healthcare: a comparative review with implications for patient safety. JRSM Open. 2015;7:1–10.
Skinner L, Tripp TR, Scouler D, Pechacek JM. Partnerships with aviation: promoting a culture of safety in health care. Creat Nurs. 2015;21:179–85.
Raab CA, Will SEB, Richards SL, O’Mara E. The effect of collaboration on obstetric patient safety in three academic facilities. J Obstet Gynecol Neonatal Nurs. 2013;42:606–16.
Pratt SD, Mann S, Salisbury M, Greenberg P, Marcus R, Stabile B, et al. Impact of CRM–based team training on obstetric outcomes and clinicians’ patient safety attitudes. Jt Comm J Qual Patient Saf. 2007;33:720–5.
Haller G, Morales M, Pfister R, Garnerin P, Chipp P, Guillemot V, et al. Improving interprofessional teamwork in obstetrics: a crew resource management based training programme. J Interprof Care. 2008;22:545–8.
Mancuso MP, Dziadkowiec O, Kleiner C, Halverson-Carpenter K, Link T, Barry J. Crew resource management for obstetric and neonatal teams to improve communication during cesarean births. J Obstet Gynecol Neonatal Nurs. 2016;45:502–14.
Bradley EH, Brewster AL, McNatt Z, Linnander EL, Cherlin E, Fosburgh H, et al. How guiding coalitions promote positive culture change in hospitals: a longitudinal mixed methods interventional study. BMJ Qual Saf. 2018;27:218–25.
Pettker CM, Thung SF, Raab CA, Donohue KP, Copel JA, Lockwood CJ, et al. A comprehensive obstetrics patient safety program improves safety climate and culture. Am J Obstet Gynecol. 2011;204:216.e1–6.
Simpson KR, Knox GE, Martin M, George C, Watson SR. Michigan health & hospital association keystone obstetrics: a statewide collaborative for perinatal patient safety in Michigan. Jt Comm J Qual Patient Saf. 2011;37:544–52.
Braithwaite J, Herkes J, Ludlow K, Testa L, Lamprell G. Association between organisational and workplace cultures, and patient outcomes: systematic review. BMJ Open. 2017;7:e017708.
Cane J, O’Connor D, Michie S. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci. 2012;7:37.
Sammer CE, Lykens K, Singh KP, Mains DA, Lackan NA. What is patient safety culture? A review of the literature. J Nurs Scholarsh. 2010;42:156–65.
Graham I, Logan J, Harrison M, Straus S, Tetroe J, Caswell W, et al. Lost in knowledge translation: time for a map? J Contin Educ Heal Prof. 2006;26:13–24.
Health Quality Ontario. Ontario patient experience measurement strategy: Toronto; 2016. https://www.hqontario.ca/Portals/0/documents/system-performance/patient-experience-measurement-strategy-1608-en.pdf. Accessed 8 Apr 2019
Okuyama A, Wagner C, Bijnen B. Speaking up for patient safety by hospital-based health care professionals: a literature review. BMC Health Serv Res. 2014;14:61.
Canadian Patient Safety Institute. Patient safety culture. 2016. https://www.patientsafetyinstitute.ca/en/toolsResources/PatientSafetyIncidentManagementToolkit/PatientSafetyManagement/pages/patient-safety-culture.aspx. Accessed 8 April 2019.
Salus Global Corporation. New look, more content. 2018. https://www.salusglobal.com/news/98-new-look-more-content. Accessed 8 Apr 2019.
The research group would like to acknowledge the contributions of the hospital teams and health care providers who volunteered to share their experiences with the implementation of the MOREOB program.
This study was funded by the Ontario Ministry of Health and Long-Term Care. The funding body had no role in the design of the study, nor did they have a role in data collection, analysis, interpretation of data, or publications.
Ethics approval and consent to participate
Ethical approval for this study was obtained from the Children’s Hospital of Eastern Ontario Research Ethics Board (13/135X). All individuals who participated in an interview signed an informed consent form prior to their participation. Written consent was obtained from administration at all participating hospitals for their de-identified MOREOB survey data to be released from Salus Global Corporation to BORN Ontario.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Reszel, J., Weiss, D., Sprague, A.E. et al. A mixed-methods evaluation of the MOREOB program in Ontario hospitals: participant knowledge, organizational culture, and experiences. BMC Health Serv Res 19, 460 (2019). https://0-doi-org.brum.beds.ac.uk/10.1186/s12913-019-4224-9
- Obstetrics and gynecology
- Patient safety
- Qualitative research
- Safety culture