case study on quality assurance

Making quality assurance smart

For decades, outside forces have dictated how pharmaceutical and medtech companies approach quality assurance. The most influential force remains regulatory requirements. Both individual interpretations of regulations and feedback received during regulatory inspections have shaped quality assurance systems and processes. At the same time, mergers and acquisitions, along with the proliferation of different IT solutions and quality software, have resulted in a diverse and complicated quality management system (QMS) landscape. Historically, the cost of consolidating and upgrading legacy IT systems has been prohibitively expensive. Further challenged by a scarcity of IT support, many quality teams have learned to rely on the processes and workflows provided by off-the-shelf software without questioning whether they actually fit their company’s needs and evolving regulatory requirements.

In recent years, however, several developments have enabled a better way. New digital and analytics technologies make it easier for quality teams to access data from different sources and in various formats, without replacing existing systems. Companies can now build dynamic user experiences in web applications at a fraction of the cost of traditional, enterprise desktop software; this development raises the prospect of more customized, user-friendly solutions. Moreover, regulators, such as the FDA, are increasingly focused on quality systems and process maturity. 1 MDIC Case for Quality program. The FDA also identified the enablement of innovative technologies as a strategic priority, thereby opening the door for constructive dialogue about potential changes. 2 Technology Modernization Action Plan, FDA.

Smart quality at a glance

“Smart quality” is a framework that pharma and medtech companies can apply to redesign key quality assurance processes and create value for the organization.

Smart quality has explicit objectives:

• to perceive and deliver on multifaceted and ever-changing customer needs
• to deploy user-friendly processes built organically into business workflows, reimagined with leading-edge technologies
• to leapfrog existing quality management systems with breakthrough innovation, naturally fulfilling the spirit—not just the letter—of the regulations

The new ways in which smart quality achieves its objectives can be categorized in five building blocks (exhibit).

To learn more about smart quality and how leading companies are reimagining the quality function, please see “ Smart quality: Reimagining the way quality works .”

The time has arrived for pharmaceutical and medtech companies to act boldly and reimagine the quality function. Through our work on large-scale quality transformation projects and our conversations with executives, we have developed a new approach we call “smart quality” (see sidebar, “Smart quality at a glance”). With this approach, companies can redesign key quality processes and enable design-thinking methodology (to make processes more efficient and user-friendly), automation and digitization (to deliver speed and transparency), and advanced analytics (to provide deep insights into process capability and product performance).

The quality assurance function thereby becomes a driver of value in the organization and a source of competitive advantage—improving patient safety and health outcomes while operating efficiently, effectively, and fully aligned with regulatory expectations. In our experience, companies applying smart quality principles to quality assurance can quickly generate returns that outweigh investments in new systems, including line-of-sight impact on profit; a 30 percent improvement in time to market; and a significant increase in manufacturing and supply chain reliability. Equally significant are improvements in customer satisfaction and employee engagement, along with reductions in compliance risk.

Revolutionizing quality assurance processes

The following four use cases illustrate how pharmaceutical and medtech companies can apply smart quality to transform core quality assurance processes—including complaints management, quality management review, deviations investigations, and supplier risk management, among others.

1. Complaint management

Responding swiftly and effectively to complaints is not only a compliance requirement but also a business necessity. Assessing and reacting to feedback from the market can have an immediate impact on patient safety and product performance. Today, a pharmaceutical or medtech company may believe it is handling complaints well if it has a single software deployed around the globe for complaint management, with some elements of automation (for example, flagging reportable malfunctions in medical devices) and several processing steps happening offshore (such as intake, triage, and regulatory reporting).

Yet, for most quality teams, the average investigation and closure cycle time hovers around 60 days—a few adverse events are reported late every month, and negative trends are addressed two or more months after the signals come in. It can take quality assurance teams even longer to identify complaints that collectively point to negative trends for a particular product or device. At the same time, less than 5 percent of incoming complaints are truly new events that have never been seen before. The remainder of complaints can usually be categorized into well-known issues, within expected limits; or previously investigated issues, in which root causes have been identified and are already being addressed.

The smart quality approach improves customer engagement and speed

By applying smart quality principles and the latest technologies, companies can reduce turnaround times and improve the customer experience. They can create an automated complaint management process that reduces costs yet applies the highest standards:

• For every complaint, the information required for a precise assessment is captured at intake, and the event is automatically categorized.
• High-risk issues are immediately escalated by the system, with autogenerated reports ready for submission.
• New types of complaints and out-of-trend problems are escalated and investigated quickly.
• Low-risk, known issues are automatically trended and closed if they are within expected limits or already being addressed.
• Customer responses and updates are automatically available.
• Trending reports are available in real time for any insights or analyses.

To transform the complaint management process, companies should start by defining a new process and ensuring it meets regulatory requirements. The foundation for the new process can lie in a structured event assessment that allows automated issue categorization based on the risk level defined in the company’s risk management documentation. A critical technological component is the automation of customer complaint intake; a dynamic front-end application can guide a customer through a series of questions (Exhibit 1). The application captures only information relevant to a specific complaint evaluation, investigation, and—if necessary—regulatory report. Real-time trending can quickly identify signals that indicate issues exceeding expected limits. In addition, companies can use machine learning to scan text and identify potential high-risk complaints. Finally, risk-tailored investigation pathways, automated reporting, and customer response solutions complete the smart quality process. Successful companies maintain robust procedures and documentation that clearly explain how the new process reliably meets specific regulatory requirements. Usually, a minimal viable product (MVP) for the new process can be built within two to four months for the first high-volume product family.

In our experience, companies that redesign the complaint management process can respond more swiftly—often within a few hours—to reduce patient risk and minimize the scale and impact of potential issues in the field. For example, one medtech company that adopted the new complaint management approach can now automatically assess all complaints and close more than 55 percent of them in 24 hours without human intervention. And few, if any, reportable events missed deadlines for submission. Now, subject matter experts are free to focus on investigating new or high-risk issues, understanding root causes, and developing the most effective corrective and preventive actions. The company also reports that its customers prefer digital interfaces to paper forms and are pleased to be updated promptly on their status and resolution of their complaints.

2. Quality management review

Real-time performance monitoring is crucial to executive decision making at pharmaceutical and medtech companies. During a 2019 McKinsey roundtable discussion, 62 percent of quality assurance executives rated it as a high priority for the company, exceeding all other options.

For many companies today, the quality review process involves significant manual data collection and chart creation. Often, performance metrics focus on quality compliance outcomes and quality systems—such as deviation cycle times—at the expense of leading indicators and connection to culture and cost. Managers and executives frequently find themselves engaged in lengthy discussions, trying to interpret individual metrics and often missing the big picture.

Although many existing QMS solutions offer automated data-pull and visualization features, the interpretation of complex metric systems and trends remains largely a manual process. A team may quickly address one performance metric or trend, only to learn several months later that the change negatively affected another metric.

The smart quality approach speeds up decision making and action

By applying smart quality principles and the latest digital technologies, companies can get a comprehensive view of quality management in real time. This approach to performance monitoring allows companies to do the following:

• automatically collect, analyze, and visualize relevant leading indicators and outcomes on a simple and intuitive dashboard
• quickly identify areas of potential risk and emerging trends, as well as review their underlying metrics and connections to different areas
• rapidly make decisions to address existing or emerging issues and monitor the results
• adjust metrics and targets to further improve performance as goals are achieved
• view the entire value chain and create transparency for all functions, not just quality

To transform the process, companies should start by reimagining the design of the process and settling on a set of metrics that balances leading and lagging indicators. A key technical enabler of the system is establishing an interconnected metrics structure that automates data pull and visualization and digitizes analysis and interpretation (Exhibit 2). Key business processes, such as regular quality management reviews, may require changes to include a wider range of functional stakeholders and to streamline the review cascade.

Healthcare companies can use smart quality to redesign the quality management review process and see results quickly. At one pharmaceutical and medtech company, smart visualization of connected, cross-functional metrics significantly improved the effectiveness and efficiency of quality management review at all levels. Functions throughout the organization reported feeling better positioned to ascertain the quality situation quickly, support decision making, and take necessary actions. Because of connected metrics, management can not only see alarming trends but also link them to other metrics and quickly align on targeted improvement actions. For example, during a quarterly quality management review, the executive team linked late regulatory reporting challenges to an increase in delayed complaint submissions in some geographic regions. Following the review, commercial leaders raised attention to this issue in their respective regions, and in less than three months, late regulatory reporting was reduced to zero. Although the company is still in the process of fully automating data collection, it has already noticed a significant shift in its work. The quality team no longer spends the majority of its time on data processing but has pivoted to understanding, interpreting, and addressing complex and interrelated trends to reduce risks associated with quality and compliance.

Healthcare companies can use smart quality to redesign the quality management review process and see results quickly.

3. Deviation or nonconformance investigations

Deviation or nonconformance management is a critical topic for companies today because unaddressed issues can lead to product recalls and reputational damage. More often, deviations or nonconformances can affect a company’s product-release process, capacity, and lead times. As many quality teams can attest, the most challenging and time-consuming part of a deviation or nonconformance investigation is often the root cause analysis. In the best of circumstances, investigators use a tracking and trending system to identify similar occurrences. However, more often than not, these systems lack good classification of root causes and similarities. The systems search can become another hurdle for quality teams, resulting in longer lead times and ineffective root cause assessment. Not meeting the standards defined by regulators regarding deviation or nonconformance categorization and root cause analysis is one of the main causes of warning letters or consent decrees.

The smart quality approach improves effectiveness and reduces lead times

Our research shows companies that use smart quality principles to revamp the investigation process may reap these benefits:

• all pertinent information related to processes and equipment is easily accessible in a continuously updated data lake
• self-learning algorithms predict the most likely root cause of new deviations, thereby automating the review of process data and statements

In our experience, advanced analytics is the linchpin of transforming the investigation process. The most successful companies start by building a real-time data model from local and global systems that continuously refreshes and improves the model over time. Natural language processing can generate additional classifications of deviations or nonconformances to improve the quality and accuracy of insights. Digitization ensures investigators can easily access graphical interfaces that are linked to all data sources. With these tools in place, companies can readily identify the most probable root cause for deviation or nonconformance and provide a fact base for the decision. Automation also frees quality assurance professionals to focus on corrective and preventive action (Exhibit 3).

Pharmaceutical and medtech companies that apply these innovative technologies and smart quality principles can see significant results. Our work with several companies shows that identifying, explaining, and eliminating the root causes of recurring deviations and nonconformances can reduce the overall volume of issues by 65 percent. Companies that use the data and models to determine which unexpected factors in processes and products influence the end quality are able to control for them, thereby achieving product and process mastery. What’s more, by predicting the most likely root causes and their underlying drivers, these companies can reduce the investigation cycle time for deviations and nonconformances by 90 percent.

4. Supplier quality risk management

Drug and medical device supply chains have become increasingly global, complex, and opaque as more pharmaceutical and medtech companies outsource major parts of production to suppliers and contract manufacturing organizations (CMOs). More recently, the introduction of new, complex modalities, such as cell therapy and gene editing, has further increased pressure to ensure the quality of supplier products. Against this backdrop, it is critical to have a robust supplier quality program that can proactively identify and mitigate supplier risks or vulnerabilities before they become material issues.

Today, many companies conduct supplier risk management manually and at one specific point in time, such as at the beginning of a contract or annually. Typically, risk assessments are done in silos across the organization; every function completes individual reports and rarely looks at supplier risk as a whole. Because the results are often rolled up and individual risk signals can become diluted, companies focus more on increasing controls than addressing underlying challenges.

The smart quality approach reduces quality issues and optimizes resources

Companies that break down silos and apply a more holistic risk lens across the organization have a better chance of proactively identifying supplier quality risks. With smart quality assurance, companies can do the following:

• identify vulnerabilities by utilizing advanced analytics on a holistic set of internal and external supplier and product data
• ensure real-time updates and reviews to signal improvements in supplier quality and any changes that may pose an additional risk
• optimize resource allocation and urgency of action, based on the importance and risk level of the supplier or CMO

Current technologies make it simpler than ever to automatically collect meaningful data. They also make it possible to analyze the data, identify risk signals, and present information in an actionable format. Internal and supplier data can include financials, productivity, and compliance metrics. Such information can be further enhanced by publicly available external sources—such as regulatory reporting, financial statements, and press releases—that provide additional insights into supplier quality risks. For example, using natural language processing to search the web for negative press releases is a simple yet powerful method to identify risks.

Would you like to learn more about our Life Sciences Practice ?

Once a company has identified quality risks, it must establish a robust process for managing these risks. Mitigation actions can include additional monitoring with digital tools, supporting the supplier to address the sources of issues, or deciding to switch to a different supplier. In our experience, companies that have a deep understanding of the level of quality risk, as well as the financial exposure, have an easier time identifying the appropriate mitigation action. Companies that identify risks and proactively mitigate them are less likely to experience potentially large supply disruptions or compliance findings.

Many pharmaceutical and medtech companies have taken steps to improve visibility into supplier quality risks by using smart quality principles. For example, a large pharmaceutical company that implemented this data-driven approach eliminated in less than two years major CMO and supplier findings that were identified during audits. In addition, during the COVID-19 pandemic, a global medtech company was able to proactively prevent supply chain disruptions by drawing on insights derived from smart quality supplier risk management.

Getting started

Pharmaceutical and medtech companies can approach quality assurance redesign in multiple ways. In our experience, starting with two or three processes, codifying the approach, and then rolling it out to more quality systems accelerates the overall transformation and time to value.

Smart quality assurance starts with clean-sheet design. By deploying modern design techniques, organizations can better understand user needs and overcome constraints. To define the solution space, we encourage companies to draw upon a range of potential process, IT, and analytics solutions from numerous industries. In cases where the new process is substantially different from the legacy process, we find it beneficial to engage regulators in an open dialogue and solicit their early feedback to support the future-state design.

Once we arrive at an MVP that includes digital and automation elements, companies can test and refine new solutions in targeted pilots. Throughout the process, we encourage companies to remain mindful of training and transition planning. Plans should include details on ensuring uninterrupted operations and maintaining compliance during the transition period.

The examples in this article are not exceptions. We believe that any quality assurance process can be significantly improved by applying a smart quality approach and the latest technologies. Pharmaceutical and medtech companies that are willing to make the organizational commitment to rethink quality assurance can significantly reduce quality risks, improve their speed and effectiveness in handling issues, and see long-term financial benefits.

Note: The insights and concepts presented here have not been validated or independently verified, and future results may differ materially from any statements of expectation, forecasts, or projections. Recipients are solely responsible for all of their decisions, use of these materials, and compliance with applicable laws, rules, and regulations. Consider seeking advice of legal and other relevant certified/licensed experts prior to taking any specific steps.

Explore a career with us

Related articles.

Smart quality: Reimagining the way quality works

Ready for launch: Reshaping pharma’s strategy in the next normal

Healthcare innovation: Building on gains made through the crisis

• Browse All Articles
• Newsletter Sign-Up

• 11 Apr 2023
• Cold Call Podcast

A Rose by Any Other Name: Supply Chains and Carbon Emissions in the Flower Industry

Headquartered in Kitengela, Kenya, Sian Flowers exports roses to Europe. Because cut flowers have a limited shelf life and consumers want them to retain their appearance for as long as possible, Sian and its distributors used international air cargo to transport them to Amsterdam, where they were sold at auction and trucked to markets across Europe. But when the Covid-19 pandemic caused huge increases in shipping costs, Sian launched experiments to ship roses by ocean using refrigerated containers. The company reduced its costs and cut its carbon emissions, but is a flower that travels halfway around the world truly a “low-carbon rose”? Harvard Business School professors Willy Shih and Mike Toffel debate these questions and more in their case, “Sian Flowers: Fresher by Sea?”

• 17 Sep 2019

How a New Leader Broke Through a Culture of Accuse, Blame, and Criticize

Children’s Hospital & Clinics COO Julie Morath sets out to change the culture by instituting a policy of blameless reporting, which encourages employees to report anything that goes wrong or seems substandard, without fear of reprisal. Professor Amy Edmondson discusses getting an organization into the “High Performance Zone.” Open for comment; 0 Comments.

• 27 Feb 2019
• Research & Ideas

The Hidden Cost of a Product Recall

Product failures create managerial challenges for companies but market opportunities for competitors, says Ariel Dora Stern. The stakes have only grown higher. Open for comment; 0 Comments.

• 31 Mar 2018
• Working Paper Summaries

Expected Stock Returns Worldwide: A Log-Linear Present-Value Approach

Over the last 20 years, shortcomings of classical asset-pricing models have motivated research in developing alternative methods for measuring ex ante expected stock returns. This study evaluates the main paradigms for deriving firm-level expected return proxies (ERPs) and proposes a new framework for estimating them.

• 26 Apr 2017

Assessing the Quality of Quality Assessment: The Role of Scheduling

Accurate inspections enable companies to assess the quality, safety, and environmental practices of their business partners, and enable regulators to protect consumers, workers, and the environment. This study finds that inspectors are less stringent later in their workday and after visiting workplaces with fewer problems. Managers and regulators can improve inspection accuracy by mitigating these biases and their consequences.

• 23 Sep 2013

Status: When and Why It Matters

Status plays a key role in everything from the things we buy to the partnerships we make. Professor Daniel Malter explores when status matters most. Closed for comment; 0 Comments.

• 16 May 2011

What Loyalty? High-End Customers are First to Flee

Companies offering top-drawer customer service might have a nasty surprise awaiting them when a new competitor comes to town. Their best customers might be the first to defect. Research by Harvard Business School's Ryan W. Buell, Dennis Campbell, and Frances X. Frei. Key concepts include: Companies that offer high levels of customer service can't expect too much loyalty if a new competitor offers even better service. High-end businesses must avoid complacency and continue to proactively increase relative service levels when they're faced with even the potential threat of increased service competition. Even though high-end customers can be fickle, a company that sustains a superior service position in its local market can attract and retain customers who are more valuable over time. Firms rated lower in service quality are more or less immune from the high-end challenger. Closed for comment; 0 Comments.

• 08 Dec 2008

Thinking Twice About Supply-Chain Layoffs

Cutting the wrong employees can be counterproductive for retailers, according to research from Zeynep Ton. One suggestion: Pay special attention to staff who handle mundane tasks such as stocking and labeling. Your customers do. Closed for comment; 0 Comments.

• 01 Dec 2006
• What Do You Think?

How Important Is Quality of Labor? And How Is It Achieved?

A new book by Gregory Clark identifies "labor quality" as the major enticement for capital flows that lead to economic prosperity. By defining labor quality in terms of discipline and attitudes toward work, this argument minimizes the long-term threat of outsourcing to developed economies. By understanding labor quality, can we better confront anxieties about outsourcing and immigration? Closed for comment; 0 Comments.

• 20 Sep 2004

How Consumers Value Global Brands

What do consumers expect of global brands? Does it hurt to be an American brand? This Harvard Business Review excerpt co-written by HBS professor John A. Quelch identifies the three characteristics consumers look for to make purchase decisions. Closed for comment; 0 Comments.

Article & Observations About Diverse Teams Working Better Together

Software Testing Client Project Case Study

Apr 21 • Case Studies

We are often asked what software testing is . The video below shares a solid definition of the term.

But we thought a software testing project case study might be helpful to better understand what software testers do on a typical day. This includes testing software, writing requirement documents for our clients, and creating user guides to ensure compliance for our clients to use for quality assurance and auditing purposes.

Iterators LLC was hired to complete accessibility testing for a few projects for the Library of Congress (LOC). Accessibility testing is required on all government websites, with Section 508 and WCAG 2.2 requirements used. To become a Trusted Tester an employee must complete the DHS Trusted Tester online training course requirements and pass the DHS Section 508 Trusted Tester Certification Exam of Homeland Security in Accessibility, and we are in a unique position to help on this project. We cross-train all our employees so that we can work on several projects at one time or several different aspects of a project to complete the work and reduce the cost to our clients.

Our first project assigned by LOC was testing their new braille feature on BARD Mobile for Android. We were tasked with testing the braille feature with several refreshable braille displays.

During our testing, we used the Orbit Reader 20 , and two different braille displays from Freedom Focus 14 and Freedom Focus 40 . There are plans to use other refreshable displays such as Humanware, but this testing has not occurred yet. We needed to test refreshable braille displays and their use in tandem with Google BrailleBack and Google TalkBack .

This work was to ensure that all hardware worked as expected with the apps we were testing. For this testing, we had to complete functional testing, smoke testing, exploratory testing and had a user panel to ensure we caught all issues that a visually impaired individual might experience while using the app.

Initially, our client was unsure if we would find any bugs and hesitant to have us enter bugs into Bugzilla as they stated the software was “complicated”. Bugzilla is a web-based general-purpose bug tracking system and not unlike other bug tracking systems we use every day such as Jira, TestRails, PractiTest, and Click-Up.

Testing was completed over several agile sprints with many significant software testing bugs found. Our testing had us test against the National Library Service requirements document. Next, we had to create an up-to-date user manual. While the manual had been updated several times, the testing had not been.

For example, when downloading a book or magazine from the Now Reading section of the mobile app, the download would end up at the bottom of the page. For years, the user guide had listed the download being at the top of the page once the document was downloaded.

Our testing team, on several occasions, said this was an error in the documentation and that the download ends up at the bottom of the page. This was corrected in the user document and sent to the development team to fix per the requirement document.

Over the next several months, we reported 30 high-priority bugs with about half fixed at this point. We have encouraged our client to test in an agile fashion because once the development team is finished, it’s harder to get these bugs fixed.

Our bugs were reported and based on the requirement document used to create the software. Lastly, the user guide had to be rewritten to reflect the app's behavior and general updates.

Once the app was tested and created, the user guide was sent to Communication Services to ensure the style used per other requirement documentation. This document had to be approved before being disseminated to the public. For example, how does the library determine what the Most Popular Books are, and over what period.

Once the document was returned from COS, this PDF document had to be remediated . The process of remediation is to make a PDF, create the heading for the document, alt text added to meaningful images, and decorative images either ignored or taken out of the digital document altogether.

Once the remediation process is complete and validated, the document becomes ADA-compliant. We then provide an accessible PDF that can be read with the use of a screen reader and create the HTML output so that the document can be added to the Library of Congress website.

You can find the current user guide we completed here: https://www.loc.gov/nls/braille-audio-reading-materials/bard-access/bard-mobile-android/#creatingfolders3.3

Case studies can be a great learning tool in software testing and project management. By looking at project case study examples, you can see how the project was planned and executed, as well as how certain tasks were managed. This can give a better understanding of what software testing involves on a daily basis. With the right software testing case studies example, software testers can hone their skills, improve project performance, and ultimately deliver better software testing results.

Related Resources:

• Crafting an Effective Test Plan: A Step-by-Step Guide
• Top Test Management Tools
• Mobile Application Functional and Performance Testing

About the Author

Jill Willcox has worked on accessibility issues for most of her professional career. Iterators is an inclusive women-owned small business (WOSB) certified by the Small Business Administration and WBENC. We provide software testing services for websites, mobile apps, enterprise software, and PDF remediation services, rendering PDFs ADA compliant.

Jill Willcox

Clutch names iterators llc as a top certified women-owned business for 2022.

Iterators LLC named Top Certified Women0owned Business Again 2022

Test Strategy vs Test Plan: What’s the Difference?

What is the difference between a test strategy and a test plan? Read this article to b...

May 05 • Reference

QA Case Study

• Industry Financial
• Solution E-mail platform build automation
• Company The customer needed an outsourced QA team to start manual testing of their products. As an established business Client required full-time QA engineers in-house so they needed a company that had experts in place that they could use.
• Tools & platforms Usually startups are less concerned

Software Quality Assurance Case Study

QA case study : Usually startups are less concerned about the quality and in many cases developers are doing QA part of work, we understand that it saves cost and gives chance to release product in a short period of time but on another hand clients will face poor quality this is exactly the situation we saw happened with our customer in Vancouver – financial technology company was looking for a way how to improve their quality and decided to find an outsourcing partner in order to set up QA team from scratch. After RFP process – GeeksForLess been chosen as a strategic partner in order to start manual testing and then move towards automation testing.

Solution we made

• SEPARATED TEST CASES PER APPLICATIONS AND PROJECTS
• COVERED 99% OF ALL REQUIREMENTS AND ALREADY EXISTING FUNCTIONALITY WITH TEST CASES
• ADDED BASIC TEST PLANS
• AUTOMATED ALL TEST CASES WHICH ARE STABLE AND COULD BE COVERED BY JS TOOL
• ESTABLISHED PROCESS – NO MORE RELEASES TO PRODUCTION WITHOUT QA CONFIRMATION
• SETUP DAILY CALLS WITH UPDATES ON THE PROGRESS
• MOVED TOWARDS CONTINUOUS INTEGRATION BY USING SPECIAL TOOLS, AUTOTESTS WERE RUNNING AUTOMATICALLY AFTER PRODUCTION RELEASES
• QA ENVIRONMENT HAS BEEN SET UP INTEGRATED QA TEAM INTO ALL PROJECT ASPECTS SO THEY BECAME A FULL PART OF CLIENT ECO-SYSTEM CUSTOMERS

or check our contact info

Sign up for the newsletter

Digital editions.

Total quality management: three case studies from around the world

With organisations to run and big orders to fill, it’s easy to see how some ceos inadvertently sacrifice quality for quantity. by integrating a system of total quality management it’s possible to have both.

Top 5 ways to manage the board during turbulent times Top 5 ways to create a family-friendly work culture Top 5 tips for a successful joint venture Top 5 ways managers can support ethnic minority workers Top 5 ways to encourage gender diversity in the workplace  Top 5 ways CEOs can create an ethical company culture Top 5 tips for going into business with your spouse Top 5 ways to promote a healthy workforce Top 5 ways to survive a recession Top 5 tips for avoiding the ‘conference vortex’ Top 5 ways to maximise new parents’ work-life balance with technology Top 5 ways to build psychological safety in the workplace Top 5 ways to prepare your workforce for the AI revolution Top 5 ways to tackle innovation stress in the workplace Top 5 tips for recruiting Millennials

There are few boardrooms in the world whose inhabitants don’t salivate at the thought of engaging in a little aggressive expansion. After all, there’s little room in a contemporary, fast-paced business environment for any firm whose leaders don’t subscribe to ambitions of bigger factories, healthier accounts and stronger turnarounds. Yet too often such tales of excess go hand-in-hand with complaints of a severe drop in quality.

Food and entertainment markets are riddled with cautionary tales, but service sectors such as health and education aren’t immune to the disappointing by-products of unsustainable growth either. As always, the first steps in avoiding a catastrophic forsaking of quality begins with good management.

There are plenty of methods and models geared at managing the quality of a particular company’s goods or services. Yet very few of those models take into consideration the widely held belief that any company is only as strong as its weakest link. With that in mind, management consultant William Deming developed an entirely new set of methods with which to address quality.

Deming, whose managerial work revolutionised the titanic Japanese manufacturing industry, perceived quality management to be more of a philosophy than anything else. Top-to-bottom improvement, he reckoned, required uninterrupted participation of all key employees and stakeholders. Thus, the total quality management (TQM) approach was born.

All in Similar to the Six Sigma improvement process, TQM ensures long-term success by enforcing all-encompassing internal guidelines and process standards to reduce errors. By way of serious, in-depth auditing – as well as some well-orchestrated soul-searching – TQM ensures firms meet stakeholder needs and expectations efficiently and effectively, without forsaking ethical values.

By opting to reframe the way employees think about the company’s goals and processes, TQM allows CEOs to make sure certain things are done right from day one. According to Teresa Whitacre, of international consulting firm ASQ , proper quality management also boosts a company’s profitability.

“Total quality management allows the company to look at their management system as a whole entity — not just an output of the quality department,” she says. “Total quality means the organisation looks at all inputs, human resources, engineering, production, service, distribution, sales, finance, all functions, and their impact on the quality of all products or services of the organisation. TQM can improve a company’s processes and bottom line.”

Embracing the entire process sees companies strive to improve in several core areas, including: customer focus, total employee involvement, process-centred thinking, systematic approaches, good communication and leadership and integrated systems. Yet Whitacre is quick to point out that companies stand to gain very little from TQM unless they’re willing to go all-in.

“Companies need to consider the inputs of each department and determine which inputs relate to its governance system. Then, the company needs to look at the same inputs and determine if those inputs are yielding the desired results,” she says. “For example, ISO 9001 requires management reviews occur at least annually. Aside from minimum standard requirements, the company is free to review what they feel is best for them. While implementing TQM, they can add to their management review the most critical metrics for their business, such as customer complaints, returns, cost of products, and more.”

The customer knows best: AtlantiCare TQM isn’t an easy management strategy to introduce into a business; in fact, many attempts tend to fall flat. More often than not, it’s because firms maintain natural barriers to full involvement. Middle managers, for example, tend to complain their authority is being challenged when boots on the ground are encouraged to speak up in the early stages of TQM. Yet in a culture of constant quality enhancement, the views of any given workforce are invaluable.

AtlantiCare in numbers

5,000 Employees

$280m Profits before quality improvement strategy was implemented

$650m Profits after quality improvement strategy

One firm that’s proven the merit of TQM is New Jersey-based healthcare provider AtlantiCare . Managing 5,000 employees at 25 locations, AtlantiCare is a serious business that’s boasted a respectable turnaround for nearly two decades. Yet in order to increase that margin further still, managers wanted to implement improvements across the board. Because patient satisfaction is the single-most important aspect of the healthcare industry, engaging in a renewed campaign of TQM proved a natural fit. The firm chose to adopt a ‘plan-do-check-act’ cycle, revealing gaps in staff communication – which subsequently meant longer patient waiting times and more complaints. To tackle this, managers explored a sideways method of internal communications. Instead of information trickling down from top-to-bottom, all of the company’s employees were given freedom to provide vital feedback at each and every level.

AtlantiCare decided to ensure all new employees understood this quality culture from the onset. At orientation, staff now receive a crash course in the company’s performance excellence framework – a management system that organises the firm’s processes into five key areas: quality, customer service, people and workplace, growth and financial performance. As employees rise through the ranks, this emphasis on improvement follows, so managers can operate within the company’s tight-loose-tight process management style.

After creating benchmark goals for employees to achieve at all levels – including better engagement at the point of delivery, increasing clinical communication and identifying and prioritising service opportunities – AtlantiCare was able to thrive. The number of repeat customers at the firm tripled, and its market share hit a six-year high. Profits unsurprisingly followed. The firm’s revenues shot up from $280m to $650m after implementing the quality improvement strategies, and the number of patients being serviced dwarfed state numbers.

Hitting the right notes: Santa Cruz Guitar Co For companies further removed from the long-term satisfaction of customers, it’s easier to let quality control slide. Yet there are plenty of ways in which growing manufacturers can pursue both quality and sales volumes simultaneously. Artisan instrument makers the Santa Cruz Guitar Co (SCGC) prove a salient example. Although the California-based company is still a small-scale manufacturing operation, SCGC has grown in recent years from a basement operation to a serious business.

SCGC in numbers

14 Craftsmen employed by SCGC

800 Custom guitars produced each year

Owner Dan Roberts now employs 14 expert craftsmen, who create over 800 custom guitars each year. In order to ensure the continued quality of his instruments, Roberts has created an environment that improves with each sale. To keep things efficient (as TQM must), the shop floor is divided into six workstations in which guitars are partially assembled and then moved to the next station. Each bench is manned by a senior craftsman, and no guitar leaves that builder’s station until he is 100 percent happy with its quality. This product quality is akin to a traditional assembly line; however, unlike a traditional, top-to-bottom factory, Roberts is intimately involved in all phases of instrument construction.

Utilising this doting method of quality management, it’s difficult to see how customers wouldn’t be satisfied with the artists’ work. Yet even if there were issues, Roberts and other senior management also spend much of their days personally answering web queries about the instruments. According to the managers, customers tend to be pleasantly surprised to find the company’s senior leaders are the ones answering their technical questions and concerns. While Roberts has no intentions of taking his manufacturing company to industrial heights, the quality of his instruments and high levels of customer satisfaction speak for themselves; the company currently boasts one lengthy backlog of orders.

A quality education: Ramaiah Institute of Management Studies Although it may appear easier to find success with TQM at a boutique-sized endeavour, the philosophy’s principles hold true in virtually every sector. Educational institutions, for example, have utilised quality management in much the same way – albeit to tackle decidedly different problems.

The global financial crisis hit higher education harder than many might have expected, and nowhere have the odds stacked higher than in India. The nation plays home to one of the world’s fastest-growing markets for business education. Yet over recent years, the relevance of business education in India has come into question. A report by one recruiter recently asserted just one in four Indian MBAs were adequately prepared for the business world.

RIMS in numbers

9% Increase in test scores post total quality management strategy

22% Increase in number of recruiters hiring from the school

20,000 Increase in the salary offered to graduates

50,000 Rise in placement revenue

At the Ramaiah Institute of Management Studies (RIMS) in Bangalore, recruiters and accreditation bodies specifically called into question the quality of students’ educations. Although the relatively small school has always struggled to compete with India’s renowned Xavier Labour Research Institute, the faculty finally began to notice clear hindrances in the success of graduates. The RIMS board decided it was time for a serious reassessment of quality management.

The school nominated Chief Academic Advisor Dr Krishnamurthy to head a volunteer team that would audit, analyse and implement process changes that would improve quality throughout (all in a particularly academic fashion). The team was tasked with looking at three key dimensions: assurance of learning, research and productivity, and quality of placements. Each member underwent extensive training to learn about action plans, quality auditing skills and continuous improvement tools – such as the ‘plan-do-study-act’ cycle.

Once faculty members were trained, the team’s first task was to identify the school’s key stakeholders, processes and their importance at the institute. Unsurprisingly, the most vital processes were identified as student intake, research, knowledge dissemination, outcomes evaluation and recruiter acceptance. From there, Krishnamurthy’s team used a fishbone diagram to help identify potential root causes of the issues plaguing these vital processes. To illustrate just how bad things were at the school, the team selected control groups and administered domain-based knowledge tests.

The deficits were disappointing. A RIMS students’ knowledge base was rated at just 36 percent, while students at Harvard rated 95 percent. Likewise, students’ critical thinking abilities rated nine percent, versus 93 percent at MIT. Worse yet, the mean salaries of graduating students averaged $36,000, versus $150,000 for students from Kellogg. Krishnamurthy’s team had their work cut out.

To tackle these issues, Krishnamurthy created an employability team, developed strategic architecture and designed pilot studies to improve the school’s curriculum and make it more competitive. In order to do so, he needed absolutely every employee and student on board – and there was some resistance at the onset. Yet the educator asserted it didn’t actually take long to convince the school’s stakeholders the changes were extremely beneficial.

“Once students started seeing the results, buy-in became complete and unconditional,” he says. Acceptance was also achieved by maintaining clearer levels of communication with stakeholders. The school actually started to provide shareholders with detailed plans and projections. Then, it proceeded with a variety of new methods, such as incorporating case studies into the curriculum, which increased general test scores by almost 10 percent. Administrators also introduced a mandate saying students must be certified in English by the British Council – increasing scores from 42 percent to 51 percent.

By improving those test scores, the perceived quality of RIMS skyrocketed. The number of top 100 businesses recruiting from the school shot up by 22 percent, while the average salary offers graduates were receiving increased by $20,000. Placement revenue rose by an impressive $50,000, and RIMS has since skyrocketed up domestic and international education tables.

No matter the business, total quality management can and will work. Yet this philosophical take on quality control will only impact firms that are in it for the long haul. Every employee must be in tune with the company’s ideologies and desires to improve, and customer satisfaction must reign supreme.

Contributors

• Industry Outlook

• Case Studies

Home » Case Studies

Below are synopses of actual Beta Breakers automation engagements.  In each synopsis however, customer and/or product identities have been scrubbed in order to protect Non-disclosure agreements. If you are interested in similar results for your QA needs, complete the form or call us to get started. 

Case Study #1 – Android Application Acceptance Test Suite

The Android Acceptance Test Suite covers all essential regression features and provides timely feedback and reporting to Android Application developers and QA. This suite is easily run within the CI process or in QA by pulling the most recent HockeyApp build on which to test. By providing constant feedback to QA and reporting regression bugs, the suite saves hours of QA testing per run and allows manual testers to focus on tighter feedback cycles for brand new features. Test results are provided via Cucumber Reports with attached debug information such as custom exception messages, screenshots at the time of the error, and a snapshot of the page XML. Reporting features useful metrics in addition to results, such as percentage of features passed, features failed, skipped, and pending implementation. Utilizing Selenium /Appium, we leverage a single codebase against Google Android and Amazon Fire OS mobile operating systems.  No third party vendors were involved.  The Beta Breakers team consisted of one full-time QE resource with occasional augmented and collaborative support from other Beta Breakers QE staff.  The completed automation suite was delivered on time and was very well accepted.

Case Study #2 – API Acceptance Test Suite

The API Acceptance Test Suite covers all of the exposed endpoints in the customer’s backend API. The test cases are written from the perspective of user stories as opposed to merely unit testing the endpoints. In addition to the backend API, the test suite leverages other dependent APIs. In order to easily facilitate the testing requirements of the Mosaic development team, an HTTP Client was designed and developed for testing REST and REST-like APIs. It currently resides in a Nexus repository and is used by other teams. The Test Suite is fully integrated into the API’s Continuous Integration pipeline on Jenkins. In addition to testing new builds at every deployment, the test suite runs twice a day on a timer. The test suite is configured to accept environmental variables as testing arguments allowing multiple Jenkins jobs to leverage the same test suite while testing multiple environments and features. The Jenkins jobs are configured to conditionally notify the development team of test failures based on the urgency around the features. Developers are notified via email and slack notifications. Test results are stored and displayed on the Jenkins job via the Jenkins Cucumber-JVM plugin. In addition to displaying feature coverage and test results, the reports contain step-by-step user flows with detailed drop downs. These step drop downs convert the HTTP requests made by the test suite into curl commands that the reader can copy and paste directly into their terminal. The reports also contain the REST responses (in this case JSON responses) that were returned via each API call. This approach allows for a high level look at Behavior Driven reports while still allowing readers of the reports to zoom to a lower level view of the flows and what may have gone wrong. This approach also allows many hours of manual QA to be replaced by automation, as well as supplemented by automation. The look and feel of the reports are perfect for giving management a high level look at the quality engineering product velocity while also maintaining low level use for developers and manual QA alike. Baked into the API test suite is a series of tests specifically for monitoring the status of various user-facing processes and service health checks. These test results are sent to an InfluxDB instance where they are read and displayed on another customer product, the “Quality Monitoring Dashboard”.  Again, no third party vendors were involved.  The Beta Breakers team consisted of two full-time QE resources with occasional augmented and collaborative support from other Beta Breakers QE staff.  The completed automation suite was delivered on time and was very well accepted.

Case Study #3 – Web Acceptance Test Suite

The Web Acceptance Test Suite covers all user-facing web features available on the customer’s web application. In addition to covering user features, compatibility testing with various browsers is baked in. Currently, the Web Acceptance test suite measures compatibility with Firefox, Chrome, Safari, Edge, and Internet Explorer. Within this compatibility testing, it also tests video playback and video player functionality. The core engine that drives the Web Acceptance Test suite is Selenium Webdriver and Cucumber. Cucumber allows all test plans to be written and expressed via user stories compatible with Behavior Driven Development. Given the many stages of the development process, and numerous environments, the test suite is parameterized to build and test environments, browsers, and features within the specifications of the user running the test suite. The suite itself manifests in many Jenkins jobs that run the specific tasks required for each environment and browser. The core purpose of the test suite is for complete integration within the development CI pipeline. When developers push new builds to their respective environments, this action triggers a test suite run against those specifications. The results are displayed in HTML Cucumber Reports and the development teams are notified via email and slack. The manual QA team also has access to a number of Jenkins jobs in order to supplement their regression testing, saving hours of manual QA time in the process. The team’s Selenium Grid instance allows for multiple jobs to be running simultaneously against different environments and browsers. This architecture means the manual QA team can run their own regression tests without fear of interfering with the development CI process. Embedded in the test reports are screenshots of test failures as well as links to video playback of the tests in question.

Case Study #4 – iOS Acceptance Test Suite

The iOS Mobile Acceptance Test Suite, similar to the Android Acceptance Test Suite, utilizes our in-house Selenium/Appium framework and covers all essential regression features as well as all other automatable feature tests. The test suite built as a Maven project is integrated into the development CI pipeline. The test suite is configured to test against real devices or iOS simulators; however, due to limitations of the applications requirements, key functionality is not possible on simulators. The test suite thereby will trigger test runs on physical devices located on campus. Test results are provided in the form of Cucumber Reports, via the Jenkins Cucumber-JVM plugin. Embedded in the reports are screenshots of key steps during failure as well as other exception messages, stack traces and other key debug information for developers. The reports themselves provide a good high level view of the user stories which are tested. The test suite is also configured to test against environments throughout the entire development process, including development, staging, QA, and production.

Case Study #5 – Product Acceptance Test Suite

This product (herein referred to as Product A) is available on Web and iOS applications. Our team integrated into the development process, attending sprint planning meetings, daily standups and weekly scrums. Our team developed automated test suites using our in-house Selenium wrapper for both Web and iOS applications. Test suites cover all essential regression features as well as all other automatable feature tests. The Product A iOS Mobile Acceptance Test Suite is configured to test against real devices or iOS simulators; however, due to limitations of the application’s requirements, key functionality is not possible on simulators. The test suite thereby will trigger test runs on physical devices located on campus. Both Web and iOS test suites are integrated within the development CI/CD pipeline. Test results are provided in the form of Cucumber Reports, via the Jenkins Cucumber-JVM plugin. Embedded in the reports are screenshots of key steps during failure as well as other exception messages, stack traces and other key debug information for developers. The reports themselves provide a good high level view of the user stories for which are tested. The test suite is also configured to test against environments throughout the entire development process, including development, staging, QA, and production. In addition to the standard automation of the product, a detailed cost analysis and recommendation was performed and given with regards to the Google App Engine configuration of the product.

Case Study #6 – Product A Performance/Cost Test Suite

In order to properly cost test Product A on the Google App Engine, a parameterized performance test suite needed to be made first in order to simulate the proper user behaviors at expected traffic count. The performance test suite leverages JMeter, using the Maven Blazemeter plugin. The JMeter test plan was written to include the developers’ unit test suite, written in JUnit, in order to simulate user interaction with the site. The JMeter test plan also included manual http requests against features that were not tested in the developers’ unit test suite as user stories. The test suite allows thread count to be passed as an argument by the test runner. A custom sandbox GAE instance was created for the front and back end of the application. Front and back end instances were then tested using the performance test suite, and GAE metrics were measured as the tests were run multiple times against various configurations. For the backend instance, the Cron jobs also had to be taken into consideration. The Cron buckets were filled with user activity and in swoop emptied to allow isolated data analysis of the GAE configuration’s behavior. Once each test was run with each instance type, configurations were further narrowed in on by fine tuning other test parameters. Eventually, an expected cost for each configuration was attained, and a recommendation was made based on the best user experience at the lowest cost.

Case Study #7 – API Acceptance Test Suite

A full regression test was developed for Product A, a collection of REST API endpoints for financial services. The test cases were developed by the automation engineer and finalized collaboratively with the client. A list of bugs and flaws in the behavior of the APIs was delivered. The client did not utilize a CI/CD process, and wished to run the test suite with a manual trigger. Thus, a solution was provided to run the suite via the Maven command line, with an integrated Cucumber HTML report to provide a convenient log of the results for each test run. Technical assistance was provided to configure the Maven project to run securely on a VM. A JMeter load testing script was also developed and provided to the client, with a demonstration of how to use the script on their own, as well as recommendations of load limits from a round of in-house testing.

Case Study #8 – New Product Integration Into A Software Ecosystem

Our customer’s project was to replace a crucial but outdated in-house software application with a customized third-party SAAS solution. The whole of the project included a web front-end SAAS, legacy desktop software, REST APIs, Amazon Web Services and PostgreSQL. Test cases were developed from ‘User Stories’ which documented the end-to-end flows of typical product usage. Since the User Stories could branch into different types of usage or into error conditions, multiple test cases were derived from a single User Story.  Generally, a test case would begin by creating assets via front-end Selenium automation through the third-party SAAS. Next, automated back-end validation of the downstream components was performed, testing the handling, storage and distribution of the information in the assets. Integration tests were also created to test individual components using mocked data. Our customer used an in-house Selenium solution for front-end automation, SmartBear ReadyAPI, and NodeJS with strict linting rules for AWS interactions. Test cases were tied together using Atlassian Bamboo and results were pushed to Practitest. Our automation engineer worked closely with the customer’s QA resources, and integrated into their AGILE workflow, participating in daily SCRUMS, Sprint planning and retrospectives. Our engineer was also responsible for providing live or recorded demonstrations of completed tasks, as well as extensive documentation of the testing behavior.

What Can We Test for You?

" * " indicates required fields

© 2024 Beta Breakers | Privacy Policy

Privacy Policy   |   Contact Us

• When to Use
• Criteria for Testing
• Why Automate with Us?
• Browser Compatibility
• Device Compatibility
• System Compatibility
• Localization Testing
• Usability Testing
• Test Plan Writing
• Load/Performance Testing
• Accessibility Testing
• Quality Engineering
• Agile Testing
• Continuous Testing
• E-Book Testing
• Exploratory Testing
• Game Testing
• Globalization Testing
• Manual Testing
• Regression Testing
• Branding Agencies
• Business & Finance
• Educational Software
• Entertainment
• Healthcare & Medical
• Retail & E-Commerce
• Web Developers
• Why Outsource Quality Assurance?
• The Process
• Mission Statement
• Testimonials

Quality Assurance in Higher Education: Selected Case Studies

Journal Title

Journal issn, volume title, corporate author.

This publication marks the first collection to emerge from the work of the COL/ICDE quality assurance research group, chaired jointly by Jocelyn Calvert (Deakin University) and Alan Tait (UK Open University). It consists of a set of case studies of quality assurance practices in distance teaching universities, along with an introductory chapter by the editor. These case studies represent a significant addition to existing work in the field, particularly at a time when universities in many parts of the world and distance teaching universities, both new and established, are placing great emphasis on reviewing and, where necessary, revising what they do in accordance with established practices and principles of quality assurance. (88 pages) // Series formerly called "Perspectives on Teacher Education." Renamed to "Perspectives on Open and Distance Learning."

Description

Collections.

Join ImpactQA at SAP Sapphire Orlando 2024 on June 3-5. Book a slot now to consult with our SAP experts!

• +1 (214) 347-9928
• [email protected]

Case Studies

• Let's Connect
• Book a Call

Home > Case Studies

• API Testing
• Automation Testing
• Cloud Testing
• Compatibility Testing
• Content QA Testing
• Embedded Testing
• Functional Testing
• Integration Testing
• Load Testing
• Mobile Application Testing
• Performance Testing
• Regression Testing
• Sanity testing
• Security Testing
• Software Testing
• System Testing
• Usability Testing
• Vulnerabilities Testing
• Electronics
• Health and Wellness
• Logistics and Supply Chain
• Manufacturing
• Retail and E-commerce

SAP CRM Performance Testing Solution for India’s Largest Natural Gas Distribution Company

Deployed advanced testing strategies to elevate product quality for a leading it management software provider, reduced software errors 75% faster with end-to-end testing for a leading north american real estate technology provider, revolutionizing restaurant operations: vita mojo's digital transformation with impactqa, supported india's second-largest two-wheeler company in eliminating compatibility and performance bottlenecks, successfully automated workflows for kpmg to ensure process optimization, efficiently managed end-to-end regression workflows for clubspark to achieve 100% automation coverage  , improve user experience by achieving 60% better app performance for a leading global beverages retail/online store, expert testing ability to improve website performance & load capability by 70% for new york road runners (nyrr), get started..

Experience a Free 30 Min Consultation with an Expert.

Schedule a Free Consultation

Subscribe to our newsletter

Get the latest industry news, case studies, blogs and updates directly to your inbox

Invalid Captcha!

Evolution of quality assurance practices in enhancing the quality of open and distance education in a developing nation: a case study

Asian Association of Open Universities Journal

ISSN : 2414-6994

Article publication date: 18 August 2022

Issue publication date: 5 October 2022

The purpose of this study is to look at the challenges and successes that the case institution has had in implementing and maintaining quality assurance (QA) processes. The study also looks into the role of QA techniques in improving an institution's performance in a developing country.

Design/methodology/approach

The research is qualitative. Nine significant persons were interviewed, including the institution's top administration, faculty members and related staff. Data were gathered to learn more about the background, incremental changes and numerous internal and external elements that influenced how QA was approached over time.

The findings revealed the challenges and experiences of the evolution of QA practices in the case institution. The results show the changes in QA practices regarding three aspects, including people, place and program of case institutions under three phases. During phases I and II, the adoption and upgradation of QA practices were very slow; phase III showed significant improvement in all three aspects. In addition, the positive impact of QA practices is evident in improving the performance of students and teaching and nonteaching staff of the case institution.

Practical implications

The findings of this research could aid open and distance learning (ODL) providers in other developing nations in understanding the challenges of such a system. It may also make it easier for other ODL providers to comprehend the QA-led success in stakeholder confidence, employability and reputation.

Originality/value

The research will give insights into how QA procedures are used and valued in ODL in developing nations.

• Quality assurance
• Developing economies
• Open and distance education
• Bangladesh Open University
• Performance

Ferdousi, F. , Ahmed, A. and Momen, M.A. (2022), "Evolution of quality assurance practices in enhancing the quality of open and distance education in a developing nation: a case study", Asian Association of Open Universities Journal , Vol. 17 No. 2, pp. 147-160. https://doi.org/10.1108/AAOUJ-02-2022-0025

Emerald Publishing Limited

Copyright © 2022, Farhana Ferdousi, Amir Ahmed and Md Abdul Momen

Published in the Asian Association of Open Universities Journal . Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) license. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this license may be seen at http://creativecommons.org/licences/by/4.0/legalcode

1. Introduction

Higher education faces numerous interconnected challenges, including limited access to tertiary level institutions, insufficient funding, inadequate information communication technology (ICT) infrastructure and low research output ( Mannan, 2015 ). Particularly in developing countries, educational systems face a variety of challenges. It includes rising costs due to fiscal constraints, a lack of curriculum renewal, a lack of appropriate learning materials, a shortage of trained teachers, an insufficient level of systemic compatibility and inadequate teaching aids, all of which contribute to low-quality and ineffective education ( Dhawan, 2020 ). In addition, natural disasters such as COVID-19 have wreaked havoc on these countries' traditional face-to-face education infrastructures ( Churiyah et al. , 2020 ). Additionally, many groups, particularly the rural poor and disadvantaged, including women, have limited access to quality education. However, in these circumstances, the requirement for large numbers of people resulted in the concept of open and distance learning (ODL), which has developed into a critical component of higher education in both developed and developing countries ( Alhumaid et al. , 2020 ).

The concept of ODL focuses on open access to education and training to make the learners free from the constraints of time and place and offer flexible learning opportunities to individuals and groups of learners. However, the central purpose of ODL is to provide education to those who lack the opportunity to obtain formal education. In open education, there are no formal classroom practices and students focus on independent study. In addition, learning content, including lectures, is disseminated by correspondence, broadcast using electronic media or provided through a learning management system here. Distance education (DE) is seen as a cost-effective and efficient means of increasing access to education. While DE with its utilization of advanced communication technology is a unique tool to enhance the quality of education in various groups and areas of the country ( Arthur-Nyarko et al ., 2020 ), contemporary communication technologies also facilitate the reach of education to a large number of people who can learn in their place, pace and time ( Tomasik et al. , 2021 ).

ODL was initially criticized for its poor quality, its lower standards of students who enroll and being detrimental to higher education planning in the country ( Stella and Gnanam, 2004 ). Even after introducing several changes in course materials, curriculum and teaching methods, there is considerable controversy throughout academia and among employers about what constitutes quality in ODL and how to ensure it ( Devkota, 2021 ). Assuring the quality of education is a fundamental aspect of gaining and maintaining credibility for programs, institutions and national higher-education systems worldwide. Despite a long and generally successful track record, ODL is still required to prove that the quality of student learning is at least equivalent to face-to-face teaching ( Arkorful and Abaidoo, 2015 ). A robust quality assurance (QA) system can assist in accomplishing this. However, the interest of stakeholders in ODL has increased interest in QA. While some argue that ODL QA practices are comparable to those used in traditional education, others argue that ODL tests conventional assumptions and thus that current QA mechanisms are insufficient to ensure the quality of ODL ( Palvia et al. , 2018 ). On the other hand, QA is not an attempt to create quality but a systematic and comprehensive effort to improve it. QA aims to demonstrate and improve the quality of an institution's methods, educational products and outcomes. It includes developing and producing instructional materials, academic programs, services, support and student learning standards in ODL ( Margaryan et al. , 2015 ).

While there are a large number of studies ( Hoecht, 2006 ; Ryan, 2015 ; Kahveci et al. , 2012 ) that have focused on the practices of QA in the traditional education system, some studies have looked into the same from ODL ( Perraton, 2012 ; Darojat et al. , 2015 ; Kihwelo, 2013 ; Scull et al ., 2011 ; Stella and Gnanam, 2004 ). However, while some studies have concentrated on the nature and importance of ODL, some other studies ( Darojat et al. , 2015 ; Kihwelo, 2013 ; Scull et al ., 2011 ; Stella and Gnanam, 2004 ) have shown the use of QA in the context of ODL. In this respect, while previous studies ( Stella and Gnanam, 2004 ) are mostly based on developed countries, only a few studies ( Darojat et al. , 2015 ; Belawati and Zuhairi, 2007 ) have examined the use of QA in ODL from a developing country's perspective. Available evidence indicates that the acceptance of ODL differs from country to country. In some developed countries, including the USA, Australia and the UK, stakeholders recognize the off-campus degrees considerably ( Stella and Gnanam, 2004 ), while the situation is different in some European countries where ODL promoters are struggling to gain acceptance of ODL. While the open university was introduced as an Apex Body in a developing country like India, Bangladesh is still concerned about the broader acceptance of ODL despite some positive changes being apparent in recent times during COVID-19 ( Churiyah et al. , 2020 ).

However, the focus on a developing country like Bangladesh is necessary due to its unique environment. It differs from developed countries in terms of opportunities, facilities, technological advancement, etc. ( Dhawan, 2020 ). Accordingly, a specific QA strategy is not universally adaptable because the DE “outfit” differs widely from institution to institution and from one country to another. This warrants that the pace of development and process of maintenance of QA in ODL be flexible ( Arthur-Nyarko et al ., 2020 ). This study, therefore, explores the experiences of incorporating QA practices in the Bangladesh Open University (BOU), the only open and distance education (ODE) provider in Bangladesh. In addition, this study further contributes to the literature by addressing the following research question:

What are the experiences of BOU in adopting and maintaining QA practices?

To explore the experiences of BOU in adopting QA practices;

To explore the experiences of BOU in maintaining QA practices;

To explore the impact of QA practices in enhancing the performance of BOU.

2. Context of QA practices in ODL and Bangladesh Open University (BOU)

While quality involves satisfying a defined standard in higher education, the quality of the inputs (e.g. staff, libraries, laboratory equipment and facilities) is generally assumed to determine the quality of its output (i.e. graduates and research).

However, recent studies argue that the quality inputs do not necessarily guarantee quality outputs; instead, it is critical to evaluate the outcome of the education, which is measured by students' learning achievement. A critical educational goal in higher education is to achieve high-quality teaching and learning or pedagogical techniques that result in student learning outcomes. Achieving high-quality teaching and learning requires a multidimensional approach that includes curriculum and course design, learning contexts, feedback, learning outcomes assessment, learning environments and student support services. In an ODL system, a comprehensive and integrated online student support system is mandatory to maintain the quality of ODL. ODL requires more endeavors in logistics, supply chain management and supporting activities, such as laboratories to support effective teaching and learning of science online at a distance. QA in the context of ODL requires a detailed examination of those characteristics that can give credibility to the various programs and products of ODL ( Tomasik et al. , 2021 ). With the increasing acceptance of ODL as widening access to education, it has become increasingly necessary that the QA process be developed and maintained if the ODL provision is relevant and more functional than the products recognized in the conventional higher learning in the emerging open learning environment ( Amin and Jumani, 2022 ).

Nevertheless, QA in ODL institutions has gained serious attention from institutions and stakeholders over a couple of decades. Accordingly, institutions began to redefine and revise their institutional missions, strategies and visions that address quality issues. While some studies have concentrated on exchanging ideas and experiences regarding the “best practices of QA implementation” in ODL contexts from around the world ( Tait, 1997 ), other studies examining the use of QA in Asian ODL have revealed significant variation in the level of QA practices ( Devkota, 2021 ). Therefore, considering the socioeconomic differences, this study focuses on exploring the experiences of BOU in adopting and maintaining the QA practices since its inception.

However, as students' backgrounds and needs diversify, BOU faces increasing pressure to accommodate the students' requests for flexibility. In this context, this institution has implemented QA practices as a cost-effective method and tool for addressing the ODL quality challenge. As in ODL, it is required to keep the quality concept in mind while developing and implementing systems, processes and procedures. In BOU, implementing QA principles in ODL was a monumental task as it required a great deal of effort, patience, socialization and training to ensure that innovation is a productive effort. However, BOU did not formulate a QA policy during its establishment, but the QA evolved along with its maturity. The formulation of the QA policy occurs at a later stage of its establishment based on the learners and institutional experience.

BOU was chosen as a case as it is the only university that provides both open and distance education (ODE) in Bangladesh and has experienced many challenges and changes from its inception. The institution was contacted via the registrar (hereafter “organization contact”). In examining the implementation of QA in BOU, 17 interviews were conducted over eight months with the university's key personnel ( Table 1 ) to explore the experiences in adopting and maintaining QA practices since its inception. Interviewees were selected based on their knowledge regarding the implementation of QA, and accordingly, two participants from top-level management, five administrative staff members, nine faculty members and one faculty member associated with the Institutional Quality Assurance Cell (IQAC) were interviewed. Interviewees were required to have had involvement with QA activities in the BOU, to have a minimum of five years' experience with the institution and to be willing to participate in the study. The case institution contact provided his support to find relevant respondents from different departments who met the aforementioned criteria.

A qualitative approach was used to better understand the experiences from the inception to date and the impact of using QA practices in enhancing the performance of the BOU. Using an interview schedule (open-ended questions), data were collected to obtain information concerning the context, gradual changes and various internal and external factors that brought such changes. The researchers collected data.

Meetings were held on the BOU campus using an interview guide. In order to gain an insight into the implementation process of QA, BOU's internal documents covering the last 10 years were also analyzed. The documents included BOU's meeting minutes, brochures and quality manuals. Prior consent was taken from the interviewees. Participants were offered the opportunity to interview in Bangla. Three participants chose to have the interviews conducted in Bangla, and the remaining participants chose the interviews to be conducted in English.

While participants were allowed to be interviewed in both Bangla and English, many respondents responded in English, and only a few respondents gave their responses in Bangla. Accordingly, the Bangla interviews were transcribed and translated. Data were analyzed using NVivo software (NVivo is a software program used for qualitative research, and it facilitates the analysis of text, audio, video and image data, including interviews, focus groups, surveys, social media and journal articles). While at the early stage of data analysis, the data were grouped into many codes. Finally, these codes were specifically categorized into different time phases such as phase I, phase II and phase III. The documents used included meeting minutes, bulletins and institutions' websites.

Furthermore, evidence was also collected from other records, including the University Grants Commission's (UGC's) instructions. The triangulation method was incorporated where the responses of the interviews and the documents collected from various sources and observations were compared ( McKinnon et al ., 2000 ). Using multiple data sources reduces the risk of a validity issue because it relates to different research contexts or information about the same comparable research. The reliability was ensured through the use of archival documents.

4. Case study

A case study is a standard research tool in the social sciences. Case studies are investigations into the causes of underlying concepts that focus on a specific person, organization or event. A case study is a detailed and in-depth examination of a person, group or event in contrast to a single vision of an individual obtained from a survey response or an interview; case studies capture a variety of viewpoints. The case study method allows for a better grasp of the subject at hand ( Bennett, 2004 ). It also reduces the possibility of prejudice by diluting the agenda of a single individual. The case study method is always preferable in education research for revealing a complex grasp of underlying phenomena ( Yazan, 2015 ). The research employs the case study technique for the study at hand, based on the preceding explanation. Here, for the underlying objective, the research works with BOU.

4.1 About the Bangladesh Open University

In 1992, BOU was the first and only university established to provide ODE in Bangladesh. As a public university initially, the BOU started with a minimal number of students and staff members, which eventually increased to 29,700 academic staff members and 574,000 students (website). This university uses a flexible mode of education delivery in providing education and training from anywhere at its own pace.

The focus of this university is to promote education through multimedia instruction at all levels of education in science, agriculture, humanities, social science, etc. It offers 56 formal academic programs, along with 19 nonformal programs (environmental protection, basic science, elementary mathematics, agriculture, bank service, marketing management, health nutrition, population and gender issues) where ICTs are the core to provide education. With a blended learning approach, the university's instructional system comprises mainly the development of instructional materials in modular form, delivery of the printed materials to the learners, face-to-face tutoring of students at selected tutorial centers (called study centers), online tutoring (for both national and international students) and delivery of learning materials through ICTs.

4.2 Phases of changes in QA practices in the BOU

4.2.1 phase i (embryonic): period from 1992 to 1995.

The first phase was the “embryonic” phase for the case institution [1] . From the inception to 1995, this period is considered as the phase I of BOU when QA practices were relatively weak compared to any other period (participant 1). This phase mostly emphasized the initial development of infrastructure including regional centers (RCs) [2] and subregional centers (SRCs) [3] in different parts of the country; study centers as well as administrative and academic building in the main campus. However, BOU faced numerous challenges in hiring desired academic and administrative staff (participant 1). Participants' views reveal that the low acceptance of ODL and lack of expertise in the management of the ODL system were the key challenges and impeding factors in ensuring quality at this phase. Moreover, initially, the tools, techniques and facilities to support every activity were in their infancy.

4.2.2 Phase II (evolution): initial QA framework of BOU (1996–2013)

While there was no QA framework found until the early stage of phase II, some extent of QA practices received attention at the later stage of this phase. However, during the second quarter of this phase, BOU adopted an “internal QA framework”. In this respect, BOU used to have a “needs analysis framework” to identify the needs and gaps regarding three components, that is, (1) people, (2) programs and (3) place. This framework aimed to identify the desired quality requirements and then analyze the gap between the expectation and reality. As mentioned by participant 1, the “needs analysis framework” includes identifying the QA practices needed for the people (academic staff, administrative staff and students); program (curriculum, module, teaching, learning other processes and procedures) and place (space and facilities).

… The critical challenge in quality improvement emerged from people with diverse backgrounds. The lack of adequate training to develop the required skill within the staff and academics also challenged the success of the ODL system.

…. classroom delivery is a crucial skill for a faculty member to enrich his[her] knowledge … …till 2013, the faculty members were more involved in preparing modules and undertaking exams and other duties only. It was a significant drawback in enhancing the faculty knowledge and skills.

… the previous selection process failed to ensure quality of staff, ……however, in many cases, staff were found inappropriate for the selected positions.

In addition, the diverse background of students with no admission test also led BOU to produce low-performing graduates. Participants believed that there was a lack of interactive communication between the students and tutors, which is considered one of the critical inhibiting factors against BOU's success (participant 9). However, until this phase, the “study centers” used to facilitate direct interaction between tutors and students only every week. Moreover, with respect to the people aspect, participants viewed that while there was a little arrangement for the development and incentive programs for the faculty members and staff (participants 1 and 8), phase II took a few initiatives, including faculty and staff development through allocating funds for both local and foreign training.

… though the development of a faculty member largely depends on research and training, these were received less priority.

Participants' views also suggest that the lack of research culture was a key to the demotivation of faculty involvement in research. Moreover, the easy promotion was also viewed as an inhibiting factor in this regard.

Moreover, some underminers operate in the shadows to demoralize QA issue processes by creating or exaggerating destructive features of the BOU-implemented change. Other teachers, particularly those who lack a firm understanding of the topic or the willpower to resist, may be readily persuaded to join the negative debate. The BOU's upper management takes the issue of challenging employees seriously. They discuss this conduct in private or public, as appropriate, and determine the underlying causes. Regarding QA practices, BOU established a tone and a standard of conduct. Peer pressures shape it so that tough people become accustomed to it (participants 5 and 9).

Nevertheless, along with the “program” aspect, the “place” also appears to have received moderate attention even though most participants recognized it as a key operational area that must be given attention in improving quality. As the participants viewed, while there were many “study centers” all over Bangladesh with inadequate facilities, the lack of a proper monitoring system was central to the quality problem. In addition, the “RCs” also received less attention from the authority. However, over time, the increased awareness regarding the necessity of physical facilities in increasing the quality of staff members, faculty members and students led the authority to improve the facilities of “RCs,” “subregional centers,” “study centers” and the main campus.

4.2.3 Phase III (matured): QA practices from the end of 2013 to present

… although the new selection and promotion processes were highly debated, these proved necessary for improving the quality of academic and administrative staff.

… the research grant was found as a motivating tool in increasing faculty involvement in research. The fund also motivated external researchers in conducting collaborative research.

Additionally, participants 9 and 16 emphasize the role of ranking in developing the institution's QA practices, notably in the growth of its people and processes. University rankings are growing increasingly popular and have extended globally. According to some scholars, rankings can help ensure that schools and universities are of high quality ( Pham, 2018 ). However, because the assessment tools are not as effective as they could be, mainstream university rankings do not increase the quality of higher education as much as anticipated ( Liu, 2020 ).

In addition, phase III showed significant attention in respect to “program” aspects. Participants' views expressed that while updating the curriculum did not get much priority in phase II, phase III brought remarkable changes. Participants revealed that the curriculum review received key priority where a committee comprising members from different departments used to discuss and thoroughly examine the curriculum for any change and improvements, including the content editing, the mode editing, the language editing, etc. The changes in the curriculum took place from 2014 to 2019 (participant 1) in different academic programs. Moreover, e-learning materials that are very popular with the learners were prepared to complement the self-learning material to enrich the learners' knowledge further. Participants' views suggest that the top management was inclined to continuously update the materials supplied to the distance learners, whether in print or electronic form. Participants' views also expressed that phase III shows considerable interest in equipping learners with better learning tools including high-speed internet, micro-SD cards containing video and audio programs inserted into mobile sets of students, which assist students even when they are off-line. The university also uses YouTube ( www.youtube.com/user/bdopenuniversity ) and BouTube ( www.boutube.com ) to help students by providing all necessary textbooks/modules.

Moreover, along with the previous system, a few more technology-based systems and application software programs were introduced to ensure smooth administrative services and provide better online services to students. In addition, at this phase, the expansion of academic programs has also been prioritized. At this stage, study centers in foreign countries received attention. As revealed from participant 3, centers were established in South Korea, four centers in Qatar and more are under consideration in the Middle East.

In addition, participant 7 brings up the issue of massive open online courses (MOOCs) as the key to accelerating the success of BOU's ongoing QA practices. MOOCs can give students access to the knowledge they might not have otherwise. They can also help people who cannot afford college ( Al-Rahmi et al. , 2019 ). MOOCs are an excellent way to get a nontraditional education online, and they can work well with traditional university education.

However, suitable and attractive infrastructure at phase III also got critical attention. Although the importance of the “place” aspect in the QA framework sparked various debates, BOU emphasized the development of good infrastructure and facilities to comply with the desired quality requirements. In line with that, BOU focused more on increasing the facilities of “study centers,” “RCs” and the main campus. However, in order to comply with the requirements of the government, during the later part of this phase, BOU established an IQAC with the financial and technical assistance from the UGC's Higher Education Quality Enhancement Project (see Figure 2 ).

The objective of IQAC is to promote a quality culture following national QA guidelines and international practices. Moreover, the focus was to ensure that the university's QA procedures were designed following the Quality Assurance Unit guidelines and national requirements. Additionally, the establishment of this cell focuses on developing, sustaining and enhancing the quality of education and public perception of the university through consistent QA practices and performance. It contributes to the development of the university's image in the eyes of stakeholders by ensuring transparency, accountability and sound management practices in all areas of administration and preparing the university to meet external QA and accreditation requirements. The cell also includes setting development goals and indicators of improvement in the educational and administrative functions of the organization. The cell develops the plan to achieve the government's desired goals, including identifying the problems by evaluating the overall features of the university's degree programs, curriculum, methods of teaching, teacher training, research activities and administrative activities. Based on these, BOU emphasized the following QA activities.

… the continuous QA practices may help BOU increase the ODL program's acceptance to a greater extent.

However, participants indicate that QA practices increased stakeholders' satisfaction, including the students, employers and regulatory bodies. Participants' views also mentioned that increased recognition is one of the critical indicators of the positive consequences of QA practices (see Figure 3 ).

5. Conclusion, implications and limitations of the study

This study aimed to analyze the experiences of implementing and maintaining the QA practices by BOU. Moreover, the study analyzes the impact of QA practices in enhancing BOU effectiveness. The findings show the importance of the role played by QA in increasing the effectiveness of the institution. The study contributes to two strands of literature: QM studies within the management literature and developing countries. It provides practical evidence on the challenges and experiences and how the emphasis on QA practices changed over time. Specifically, the findings provide an insight into the change that was made to the practices of BOU.

The analysis revealed that the changes in QA practices of the case institution mainly focused on three components, comprising people, program and place under three phases, including phase I, phase II and phase III. The result shows that the nature of ODL is different from that of traditional education, which causes unique challenges for BOU to manage quality. The QA practices in ODL and the traditional system are different in many ways, in terms of the qualification of academic and administrative staff, students' background, mode of education, teaching-learning methods, space and facilities, etc. However, the result reveals that there were no such QA practices during phase I; instead, this phase concentrated more on developing basic infrastructures. The emphasis was more on introducing the new model of the education system – the ODL approach to many students. The key challenges were attracting and recruiting desired candidates for the academic and administrative positions, the low acceptance of ODL and the lack of expertise in the management of the ODL system. The finding has also been supported by the scholars Kihwelo (2013) .

The study demonstrated that while the first stage of phase II was found to have no attention to QA practices, the later stage of phase II shows little to moderate attention to various aspects of QA practices. In this phase, a “needs analysis framework” was developed to identify the requirements of QA practices in terms of people (academic staff members, administrative staff members and students); program (curriculum, module, teaching, learning and other processes and procedures) and place (space and facilities). This framework helped the case institution in identifying the challenges, such as inexperienced academic and staff positions, diverse backgrounds of students with no admission test, lack of face-to-face interaction between teachers and students and rigorous recruitment and promotion process, which inhibit the desired quality.

Regarding the programming aspect, findings reveal that while the experienced faculty members from reputed universities were involved in preparing the modules, there was a lack of an initiative in regularly reviewing the curriculum and modules. However, the result shows that while there was little attention with respect to faculty training and research, a moderate level of improvement was mentioned at the later stage of this phase. Moreover, this phase pays moderate attention in increasing the facilities of its regional and study centers.

However, compared to phase I and phase II, phase III showed significant changes in QA practices in terms of all three aspects, including people, process and programs. The participants mentioned changes in terms of faculty recruitment and promotion; review of curriculum and modules; inclusion of modern tools and techniques in the mode of learning and teaching; rigorous training for faculty and staff development and availability of research grants. Finding suggests that following the aforementioned developments, a continuous improvement mechanism is developed after establishing IQAC. The IQAC led BOU in ensuring QA practices according to the national requirements. The analysis suggests that in an effort to enhance the acceptability of ODL to the stakeholders, BOU is preparing to apply for the accreditations. The findings indicate that any improvement is the result of awareness and intention of the top management. When top management understands the need and benefits of assuring quality, they can create a milestone. Therefore, it is imperative for the top management to create a systematic process to adopt QA practices and monitor the outcome on a regular basis.

The study extends the existing literature by exploring the experiences of implementing and maintaining the QA practices and the impact on the effectiveness of ODE in a developing country like Bangladesh. In particular, the analysis incorporates a qualitative approach to exploring the research objectives. In doing so, the findings assist the regulators and authorities of ODL providers in developing countries in understanding how to enhance the use and effectiveness of QA practices. The focus on a developing country was considered important as the use and effectiveness of QA in developing countries are different from those in developed countries due to the political, social and cultural environment. Moreover, while QA practices are prevalent in developed countries, institutions within developing economies experience difficulties incorporating such practices due to scant facilities, resistance and negligible investment in technologies. Similar results are observed in other developing nations, such as India. Even though institutions in underdeveloped nations endeavor to provide excellent education, each institution has its own contextual goals, norms and laws, budgetary constraints, security concerns and technical legacies ( Patra et al. , 2022 ). Consequently, the measures must consider the specific environment of the concerned institution. In addition, developing nations' governments must identify the issues that impede the institution's application of QA procedures and then develop the required measures to overcome these obstacles.

The study also extends the literature by providing an insight into the effectiveness of QA, in terms of increased recognition, employability and satisfaction of stakeholders. This implies that institutions should endeavor to make greater use of QA practices to provide better services, thereby enhancing the institution's image. This finding supports the previous studies by Tsekouras et al. (2003) .

However, the findings imply that QA is a never-ending process. If institutions simultaneously focus on improving three aspects, including people, processes and programs, it is possible to enhance the effectiveness of the institutions. Therefore, the institutions should endeavor to use QA practices to provide better services, thereby enhancing the institution's image, which also highlights the critical role of QA practices in enhancing performance.

The study has limitations. While the case study aimed to explore the changes in QA practices through a retrospective analysis of organizational participants' views of the changes, there is a possible risk of missing valuable information due to the inability of respondents to recall events. While such concerns were minimized by using multiple data sources, future research may extend the current study by investigating the findings in different research sites.

Internal needs analysis/gap analysis framework of BOU

QA framework during phase III

IQAC-induced QA practices

List of participants

The time period of each phase is on the basis of the discussion of the participants.

Regional centers (RCs) usually organize the admission tests and maintain linkage with study centers including local school, colleges and universities.

Here, the “people” refers to the internal and external stakeholders including governing council members, management team members, senior academic and students who are part of QA activities. The “program” refers to the processes, procedures and activities including curricular design, teaching and learning, governance systems, professional development of staff, research, student assessment, staff recruitment, student admission, institutional ceremonies and student support services. “Place” refers to the space and facilities which are required to support every activity.

Al-Rahmi , W. , Aldraiweesh , A. , Yahaya , N. , Kamin , Y.B. and Zeki , A.M. ( 2019 ), “ Massive open online courses (MOOCs): data on higher education ”, Data in Brief , Vol.  22 , pp.  118 - 125 .

Alhumaid , K. , Ali , S. , Waheed , A. , Zahid , E. and Habes , M. ( 2020 ), “ COVID-19 and e-learning: perceptions and attitudes of teachers towards e-learning acceptance in the developing countries ”, Multicultural Education , Vol.  6 No.  2 , pp.  100 - 115 .

Amin , S. and Jumani , Z. ( 2022 ), “ Analysis of existing policies and practices in distance education: empowering women education ”, International Research Journal of Education and Social Sciences , Vol.  1 No.  1 , pp.  60 - 67 .

Arkorful , V. and Abaidoo , N. ( 2015 ), “ The role of e-learning, advantages and disadvantages of its adoption in higher education ”, International Journal of Instructional Technology and Distance Learning , Vol.  12 No.  1 , pp.  29 - 42 .

Arthur-Nyarko , E. , Agyei , D.D. and Armah , J.K. ( 2020 ), “ Digitizing distance learning materials: measuring students' readiness and intended challenges ”, Education and Information Technologies , Vol.  25 No.  4 , pp.  2987 - 3002 .

Belawati , T. and Zuhairi , A. ( 2007 ), “ The practice of a quality assurance system in open and distance learning: a case study at Universitas Terbuka Indonesia ”, International Review of Research in Open and Distance Learning , The Indonesia Open University , Vol.  8 No.  1 , p. 340 .

Bennett , A. ( 2004 ), “ Case study methods: design, use, and comparative advantages ”, Models, Numbers, and Cases: Methods for Studying International Relations , Vol.  2 No.  1 , pp.  19 - 55 .

Churiyah , M. , Sholikhan , S. , Filianti , F. and Sakdiyyah , D.A. ( 2020 ), “ Indonesia education readiness conducting distance learning in Covid-19 pandemic situation ”, International Journal of Multicultural and Multireligious Understanding , Vol.  7 No.  6 , pp.  491 - 507 .

Darojat , O. , Nilson , M. and Kaufman , D. ( 2015 ), “ Quality assurance in Asian open and distance learning: policies and implementation ”, Journal of Learning for Development (JL4D) .

Devkota , K.R. ( 2021 ), “ Inequalities reinforced through online and distance education in the age of COVID-19: the case of higher education in Nepal ”, International Review of Education , Vol.  67 No.  1 , pp.  145 - 165 .

Dhawan , S. ( 2020 ), “ Online learning: a panacea in the time of COVID-19 crisis ”, Journal of Educational Technology Systems , Vol.  49 No.  1 , pp.  5 - 22 .

Hoecht , A. ( 2006 ), “ Quality assurance in UK higher education: issues of Trust, control, professional autonomy and accountability ”, High Education , Vol.  51 , pp.  541 - 563 .

Kahveci , T.C. , Uygun , Ö. , Yurtsever , U. and Ilyas , S. ( 2012 ), “ Quality assurance in higher education institutions using strategic information systems ”, Procedia – Social and Behavioral Sciences , Vol.  55 , pp.  161 - 167 .

Kihwelo , P.F. ( 2013 ), “ Quality assurance system in open and distance learning for normative judgment ”, HURIA , Vol.  4 No.  1 , pp.  1 - 21 .

Liu , S. ( 2020 ), “ Can ranking contribute to the quality assurance of higher education? An examination of the Chinese disciplinary ranking ”, Cambridge Journal of Education , Vol.  51 , pp.  1 - 19 , doi: 10.1080/0305764X.2020.1829548 .

Mannan , A. ( 2015 ), “ Higher Education: no crisis but problem ”, True and Impartial Daily Sun , available at: https://www.daily-sun.com/post/45817/Higher-Education:No-crisis-but-problem .

Margaryan , A. , Bianco , M. and Littlejohn , A. ( 2015 ), “ Instructional quality of massive open online courses (MOOCs) ”, Computers and Education , Vol.  80 , pp.  77 - 83 .

McKinnon , K.R. , Walker , S.H. and Davis , D. ( 2000 ), Benchmarking: A Manual for Australian Universities , Department of Education, Training and Youth Affairs , Canberra, ACT .

Palvia , S. , Aeron , P. , Gupta , P. , Mahapatra , D. , Parida , R. , Rosner , R. and Sindhi , S. ( 2018 ), “ Online education: worldwide status, challenges, trends, and implications ”, Journal of Global Information Technology Management , Vol.  21 No.  4 , pp.  233 - 241 .

Patra , A. , Asghar , A. , Chaudhary , P. and Ravi , K.S. ( 2022 ), “ Integration of innovative educational technologies in anatomy teaching: new normal in anatomy education ”, Surgical and Radiologic Anatomy , Vol.  44 No.  1 , pp. 25 - 32 .

Perraton , H. ( 2012 ), Open and Distance Learning in the Developing World , 2nd ed. , Routledge, Taylor and Francis Group , London and New York .

Pham , H.T. ( 2018 ), “ Impacts of higher education quality accreditation: a case study in Vietnam ”, Quality in Higher Education , Vol.  24 No.  2 , pp.  168 - 185 .

Ryan , T. ( 2015 ), “ Quality assurance in higher education: a review of literature ”, Higher Learning Research Communications , Vol.  5 No.  4 .

Scull , W.R. , Kendrick , D. , Shearer , R. and Offerman , D. ( 2011 ), “ The landscape of quality assurance in distance education ”, Continuing Higher Education Review , Vol.  75 , pp.  138 - 149 .

Stella , A. and Gnanam , A. ( 2004 ), “ Quality assurance in distance education: the challenges to be addressed ”, Higher Education , Vol.  47 No.  2 .

Tait , A. ( 1997 ), Quality Assurance in Higher Education: Selected Case Studies , The Commonwealth of Learning , Vancouver, British Columbia.

Tomasik , M.J. , Helbling , L.A. and Moser , U. ( 2021 ), “ Educational gains of in-person vs. distance learning in primary and secondary schools: a natural experiment during the COVID-19 pandemic school closures in Switzerland ”, International Journal of Psychology , Vol.  56 No.  4 , pp.  566 - 576 .

Tsekouras , J. , Stamboulis , M. and Litsardakis , M. ( 2003 ), The Financing of Vocational Education and Training in Greece , Panorama Series, 78 .

Yazan , B. ( 2015 ), “ Three approaches to case study methods in education: Yin, Merriam, and Stake ”, The Qualitative Report , Vol.  20 No.  2 , pp.  134 - 152 .

Corresponding author

Related articles, we’re listening — tell us what you think, something didn’t work….

Report bugs here

All feedback is valuable

Please share your general feedback

Join us on our journey

Platform update page.

Visit emeraldpublishing.com/platformupdate to discover the latest news and updates

Questions & More Information

Answers to the most commonly asked questions here

Quality Assurance: 407ETR

In accordance with SQM Group's Privacy Policy , you must check the box to demonstrate express consent.

407 ETR - Quality Assurance Best Practice

SQM Group Helped 407 ETR Achieve:

FCR/Csat Results

There has been a visible improvement in results since the implementation of the new CQA program . Results are positive, with 84% of CQA evaluations exceeding and meeting expectations. “The results are truly indicative of the perception of the customer interaction with our employees as well as our employee’s ability to provide First Call Resolution! I recommend the CQA program that SQM offers!” says Shantell Edwards, a call center team manager.

After implementing the new CQA Program, our end-of-year VoC improved in the following areas: world-class call results showed a 3 point improvement, call resolution improved from 93% to 95%, which increased the department’s FCR from 82% to 85%. This improvement was made within 6 months. Most of the improvement was driven by the top performing agents in the first and second quartile, improving by 7% points. The last quartile, as expected, showed a small decline. The CQA program further reinforced the importance of call resolution to our agents.

Our call center has been certified by SQM as a world-class FCR performing call center for the last five years. The criteria to be certified as a world-class FCR performing call center is your FCR rate must be 80% or greater. Of the 500 call centers SQM measures, only 5% of their clients perform at the world-class FCR standard of 80%.

407 ETR Company Description

407 ETR is the world’s first all-electronic toll road of its kind, and from the day it was opened up to the present day, it continues to fulfill its mission of relieving traffic on local highways and roads. Drivers take about 380,000 trips on the highway each workday. Customers tell 407 ETR that the number one reason they use the highway is to save time. Customers also report that they feel they make a good choice in using 407 ETR.

What is Customer Quality Assurance (CQA)?

Opportunity Statement

We recognized that to elevate 407 ETR to a World Class Call Center status, reduce operating expenses, and implement independent evaluation of our customer service interactions’ quality, there was a necessity to implement the Call Quality Assurance (CQA) best practice program.

Over the past 11 years, SQM has offered us invaluable feedback, benchmarking, and workshops, which provided 407 ETR employees with the tools to continuously improve our Voice of the Customer (VoC) results. To raise our standards, we looked at where we were, where we desired to be, and what actions we would need to get there. From an efficiency and budget perspective, it was easy to see why partnering with SQM would benefit us.

First and foremost, our mission was to increase Csat and improve FCR . It was also important that we balanced customer feedback with our agents’ work and provided an unbiased assessment of our work and compliance with our processes. To inspire employees daily to be committed to demonstrating company values, it is necessary to align the CQA program to our company values.

We compared the cost per survey and QA evaluations conducted internally versus by SQM and discovered a significant saving per CQA if SQM conducted the evaluations. This budgetary saving allowed us to explore further how we would prepare such a substantial change in moving from an internal to an external focus that would impact our agents, team managers, application audits and administration, training, and more.

Best Practices Used

A call list of customers who have called the call center is compiled and sent to SQM daily. Customers are surveyed within one day of their call to the call center. The SQM call quality evaluator reviews customer survey ratings, listens to the customer feedback and to the recording of the interaction between the agent and customer, and reviews screen captures to assess call compliance. The CQA evaluation form embeds customer survey results, customer audio feedback, and call compliance results. An agent has access to view VoC feedback, call compliance metrics, and links to the audio. Comments provided by the evaluator are very detailed. This is viewed positively both from a team manager’s perspective, as it helps them prepare for coaching sessions, and from the agent’s perspective, who views the evaluator comments in mySQM™ FCR Insights reporting tool.

A new evaluation form was created internally with Customer Service and SQM Group. The CQA evaluation form has 60 of the possible 100 points based on 3 parts of the customer surveys conducted by SQM to generate the score (30 for call resolution, 15 for Agent Csat, 15 for customer Continue to Use). The other 40 points are allocated for the call compliance metrics (broken down as 65% customer focus and 35% organization focus).

For the first month, we ran the new program alongside the old program, which provided valuable insights into the new program’s effectiveness. This also allowed us to determine the new program’s impact, refine the form and guidelines developed, and make any changes without negatively impacting our customers and employees.

The Customer Service Department, Applications Audits and Administration (previously responsible for call quality evaluations), and SQM’s Development Team created a combined program and form to further assist Agents in achieving their world-class certification goal through the new Call Quality Assurance program. Teamwork and collaboration were imperative in the planning stages with SQM’s team, our internal CQA team manager, the customer service manager, our training department manager, and the manager of technical services. This teamwork and collaboration resulted in developing the new CQA program, cultivating agent growth, and fostering agent development.

Our 407 ETR training facilitator travelled to SQM to conduct all necessary training for the SQM team of call quality evaluators, furthering our strength in a partnered endeavour. Ongoing calibration sessions were conducted to ensure consistency in scoring and still continue to take place today.

Through the change, supporting agents was an absolute necessity and number one on our list of priorities. A communication plan was developed, providing frequent small updates about the new program to allow our employees to absorb the information, ask questions, and incorporate the change.

Our performance improvement plan was reviewed to establish an accurate three-month rolling trend for the new call quality standards and Agent Csat. The new call quality three-month trend began four months following the new CQA program’s implementation, providing employees with the opportunity to get fully acquainted and comfortable with the new program.

There are critical components that comprise customer interactions that significantly impact our customers and organization and, in turn, FCR, such as failure to adhere to policies and workflows. The new CQA process includes a critical component section, and should a critical component be missed; the CQA evaluation receives a score of zero. This integral piece of the evaluation provides us with common themes where opportunities for improvement have been successfully identified. This information has enabled us to continuously work toward implementing changes so that agents continue their drive to own the call and create a world-class experience. Agents have online access to results immediately once a call quality evaluation is complete. The customer survey is conducted and then reviewed by a dedicated CQA evaluator. Once evaluated, it is available in the mySQM™ FCR Insights reporting tool and can be viewed by the agent. Having the ability to view feedback right away provides the agents with an understanding of what they are doing to create a positive customer experience. It also encourages the agents to continue their behavior and continue to learn what they can do differently in the future to create that memorable customer experience while following appropriate policies and procedures. Essentially, it creates a desire to exceed the purpose of one’s job and responsibilities.

It is very important that regular employee feedback is provided from agents on the CQA program. Focus groups were conducted to learn how agents viewed the new CQA Program and their recommendations for continuously improving the program. When employees feel they have been listened to, have a say in the program, and have contributed to improving the program, they believe in the program and its benefits to them. This keeps the agent engaged and working to strive for world-class results. A connection is made between what a customer is told and its completion. Previously, a call could be scored 100% by the customer, but processes may not have been followed through in reality. This current process ensures all calls are completed within recommended standards. Hearing such feedback reaffirms our agent’s place’s significant importance on the desire to own the call. By giving the customer survey the weight of 60% of the CQA score, the new call quality program serves as an additional tool that emphasizes the importance of ensuring that our customers receive the best customer service that our agents can provide. This supports our desire for agents to create a world-class experience.

SQM Awards Received

Call Center World Class FCR Certification (Commercial): 2016 - 2020 Best Practices - Customer Quality Assurance: 2016 - 2020

Case Study: How the Palestinian Association of Certified Public Accountants Established a QA Review Mechanism and Began Adoption of ISQM1 & ISQM2

Introduction to isqm 1 and 2.

International Standard on Quality Control 1 (ISQC 1) was introduced by the International Auditing and Assurance Standards Board (IAASB) and provides guidance on the quality control system for firms that audit and review financial statements. ISQC 1 has made provisions for a sound quality control system that ensures that the firm complies with ethical and professional standards and legal and regulatory requirements. But the audit environment is becoming more complex, and IAASB made the decision to develop and issue a new suite of quality assurance standards: International Standard on Quality Management (ISQM 1 and ISQM 2). These build on ISQC in order to provide more relevant and comprehensive guidance on the effective implementation of QA and the quality management system for firms that perform assurance engagements other than audits or reviews of financial statements.

The new standards are driven by the emphasis on a risk-based approach to quality assurance. Whilst ISQM 1 sets out the approaches to quality assurance for firms, ISQM 2 focuses on quality assurance at the engagement level and the appointment of an engagement quality reviewer. Both standards establish the requirements for a firm's system of quality management to ensure a robust and adaptive system based on risk assessment is implemented to manage quality risks in a constantly evolving environment. The standards also provide guidance on the responsibilities of the firm's leadership, the management of engagement quality, and the monitoring of the system's effectiveness.

The implementation of ISQM 1 and ISQM 2 is essential for ensuring the quality of audits and other assurance engagements. These standards facilitate the building of users’ confidence in the reliability of financial statements and other assurance reports issued by audit firms, improve the consistency of audits, enhance risk management processes and improve information exchange with stakeholders within and outside the firm.

ISQM 1, issued in 2019 and enforced in 2022, builds on the framework established by ISQC1. It expands its scope to cover various types of assurance services provided by accounting firms and, most importantly, places greater emphasis on the assessment of quality risks as a foundation for designing a quality management system. The standard presents an integrated approach to quality management. It requires firms to identify and assess the quality risks associated with their services, implement appropriate controls to mitigate them, and introduce monitoring and remediation procedures. It also requires firms to establish a quality culture within the organisation and promote continuous improvement as foundations for effective implementation.

ISQM 1 is complemented by ISQM2, which also came into force in December 2022. ISQM 2 focuses on the effective management of quality at the engagement level, providing guidance on how firms can ensure that individual engagements meet the required standards of quality and that the engagement quality reviewer appointed to the engagement has sufficient skills and competencies to perform their task.

Implementation of ISQM 1 and 2 shifted the approach from control of the quality to the management of quality, which will require audit firms to invest time and resources. The integrated QA approach benefits   audit firms and users of financial statements.   It will enable a more robust and effective approach to quality building trust and confidence in audit and assurance services during times of turbulence and uncertainty.

PACPA experience

One professional accountancy organization (PAO) that has demonstrated great commitment to ensuring high standards of quality of audits and assurance engagement and the benefits of a robust system is the Palestinian Association of Certified Public Accountants (PACPA).

PACPA was established in 1995 as a professional association for Palestinian accountants. PACPA’s jurisdictions have been reformed by the Palestinian Audit Practice Law for the year 2004. Since its establishment, PACPA has been committed to promoting the highest standards of professionalism and ethical conduct among its members. In 2010, PACPA started extensively working in adoption of the SMOs and became an associate member of IFAC in 2013. In 2015, PACPA built its manuals and internal regulations and provided the required awareness and trainings. In 2020, PACPA conducted its first round of the field review to the members and became one of the first professional accounting organisations in the Middle East to fully adopt ISQC 1. This decision was driven by a desire to enhance the quality of services provided by Palestinian accounting firms and to promote greater transparency and accountability in the profession. The adoption of ISQC 1 was a significant undertaking for PACPA, as it required a comprehensive review of its member firms' quality control systems and the implementation of a new standardised approach to quality assurance.

PACPA worked closely in coordination and support of IFAC and the World Bank to ensure that the adoption of ISQC 1 was tailored to the needs and circumstances of Palestinian accounting firms. This included developing guidance and training programs to support the implementation of the standard and providing ongoing support to member firms as they worked to comply with its requirements.

The proactive adoption of ISQC 1 has positively impacted the Palestinian accounting profession, enhancing the credibility and reputation of Palestinian accounting firms and increasing client confidence in the quality of their services. It has also helped promote greater transparency and accountability in the profession, which is essential for building trust with clients and stakeholders.

PACPA has also learned some valuable lessons from its experience of ISQC 1 adoption. One of the most important of these lessons is the importance of stakeholder engagement. PACPA has worked closely with its members, stakeholders, and regulators to ensure that the adoption of ISQC 1 is aligned with their needs and expectations. Another lesson learned by PACPA is the need for ongoing training and development. Implementing a quality assurance system requires a significant investment of time and resources. It is important to ensure that staff and members are equipped with the necessary skills and knowledge to implement the system effectively.

Implementing ISQC 1 helped PACPA:

• Become a sustainable PAO by enhancing the quality of audit services provided by its members;
• Establish policies and procedures for quality control, including a system for monitoring compliance with those policies and procedures;
• Promote consistency and reliability in the services, thereby increasing the credibility of the profession in Palestine;
• Strengthen its relationships with stakeholders, including regulators, investors, and the public; and
• Become better positioned to attract and retain members, which helped increase the organisation’s revenue and contribute to its long-term sustainability.

Overall, PACPA's adoption of ISQC 1 demonstrated the organisation’s commitment to promoting the highest standards of professionalism and ethical conduct among its members and its dedication to ensuring the sustainability and success of the Palestinian accounting profession. However, this journey is not static, and PACPA is now fully engaged with the adoption and implementation of the new suite of quality standards. In the middle of 2023, PACPA updated the quality assurance guidelines and testing procedures to align with ISQM 1 and 2.  Orientation of the reviewers and training of the members were conducted based on the new standards, in addition to considering the lessons learned from the previous round of quality assurance program.  While PACPA is conducting another round of the QA review on the members in 2023, PACPA members understand the positive impact of the quality assurance program on their personal and professional development.  The quality assurance program was one of the areas in which PACPA is achieving progress that supports PACPA’s application for the full membership in IFAC, which is expected to be finally approved in November 2023.

Muhammad Zarif Ludin

Head of Institutional Partnerships

Zarif Ludin is the Head of Institutional Partnership at ACCA and leads and develops strategic partnerships with regional and global accountancy organizations, donors and development agencies, and represents ACCA in external forums and consistently promotes ACCA’s global contribution to public value. Zarif Ludin serves ACCA’s Category “A” observer at the IFAC board and PAODAG as a Technical Assistant and is a member of the Ethics working group at CAPA.  

Zarif Ludin was the Chief Executive of CPA Afghanistan at the Ministry of Finance, Afghanistan, since June 2013, being responsible for the enactment of accountancy law, the establishment of the executive office, board and related technical committees and the capacity building of accountants and auditors. Zarif Ludin focused on quality education, well-defined regulations and CPA Afghanistan’s recognition on national, regional and international platforms. He played a significant role in obtaining South Asia Federation of Accountants (SAFA) and Confederation of Asia and Pacific Accountants (CAPA) associate memberships for CPA Afghanistan. He initiated the idea of Strategic Partnership and revenue sharing for CPA Afghanistan jointly with ACCA.

In addition, he has experience of more than 12 years of working with different Organizations like USAID and US embassy-funded programs, Financial Institutions and Banks, NGOs and Universities as a Finance Manager, Program Coordinator and part-time Lecturer. Zarif holds a master’s degree in Public Policy from the Graduate Institute of Geneva.

Jamal Milhem

Executive Director

Mr. Jamal Milhem has joined the IFAC PAO Development & Advisory Group (PAOD&AG) in January 2018 for the first term 2018-2020 and was reelected for the second term 2021-2023. 

Mr. Milhem is currently the Executive Director and a partner of TAGI in Palestine and a member of the TAGORG Executive Management Council from 2015-2022 (TAGI is an International Audit firm, Member of FOF and one of TAGORG firms with more than 100 offices worldwide). He has more than 30 years of experience in public accounting and advisory services in different countries in TAGI and TAGORG. In addition, he has more than 8 years of experience in higher education in different universities in Palestine related to teaching and development of accounting education programs.

Mr. Milhem has joined the Palestinian Association of Certified Public Accountants (PACPA) in 1997 (PACPA is a national PAO and an IFAC member), and he has been serving as a board member since 2003. He was elected as vice-president of PACPA from 2010 to 2016, reelected as a vice president since 2018-2020 and elected as PACPA president for the years 2021-2022. He also chaired different committees in the PACPA such as the Code of Ethics Committee and the Quality Assurance Committee. Furthermore, he has been elected as a board member of IASCA since 2011, (IASCA is an accounting organization working across the MENA region and an IFAC member). Mr. Milhem was a member of the American Institute of Certified Public Accountants, AICPA – Montana State (USA) from 2000 till 2011. Mr. Milhem also served as a member of many other national professional committees, e.g., National Team for AML and risk assessment, National Palestinian WTO Accession Team, Steering Committee for National Team for Transparency and licensing committees in the Palestinian Capital Market Authority i.e., financial analyst, financial consultant, investment consultant and IPO manager.

Mr. Milhem has earned his Bachelor’s degree in accounting and auditing in 1992, and his Master degree in business administration in 1994 from the University of Poona, Symbiosis College, India. He joined a PHD Accounting and Finance program in the Arab American University, Palestine in 2022.  He has obtained his CPA certification from the Montana State Board of Public Accountants – USA in 1998.