| dc.description.abstract | In an era where governments around the world invest heavily in data collection and data management, poor-quality data is expensive and has many direct and indirect costs. While there are different types of data quality challenges, some of the more complex data quality problems depend on the design and production processes involved in generating data. Therefore, it is important to design systems that support better data quality. This involves understanding what quality means in a specific context, understanding how it can be measured, and identifying ways to encourage better data quality behaviours.
Healthcare is not immune to the challenges of data quality and can be classified as a complex socio-technical system by virtue of its characteristics. As such, the study of healthcare data quality and its improvement is well suited for the domain of systems design and human factors engineering. Cognitive Work Analysis (CWA) is especially well suited for this task, as it can be used to better understand the context and workflow of users in complex socio-technical domains. It is a conceptual framework that facilitates the analysis of factors that shape human-information interaction and has been used in healthcare for over 20 years. The approach is work-centred, rather than user-centred, and it analyses the constraints and goals that shape information behaviour in the work environment. I used CWA as a framework to help me analyse the problem of data quality in healthcare.
My research uses an instrumental case study approach to understand data quality in primary care. My goal was to answer three questions: In primary care, how are individual users influenced by their environment to input high-quality data? What techniques could be used to design systems that persuade users to enter higher-quality data? Is it possible to improve data quality in primary care by persuading users with the user interface of information systems in these complex socio-technical systems? The scope of work included modelling data quality, defining and measuring data quality in a primary care system, establishing design concepts that could improve data quality through persuasion, and testing the viability of some design concepts.
I began analysing this problem by creating an abstraction hierarchy of patient treatment with medical records. This model can be used to represent patient treatment from a primary care perspective. The model helped explain the patient treatment ecosystem and how data is generated through patient encounters.
After creating my model to represent patient treatment, I incorporated it into two CWAs of data quality and data codification. The first model represented codification in the primary care ecosystem, whereas the second model represented codification in community hospitals. After developing abstraction hierarchies for both domains, I analysed similar tasks from each system with control task analysis, strategies analysis, and worker competencies analysis. The tasks that I analysed related specifically to data codification: in primary care, I modelled the record encounter task performed by clinicians at a Family Health Team (FHT), and in the community hospital, I modelled the abstract task performed by health information management professionals. I used the same record encounter task at the FHT as a continuing focus of my case study.
I used both models of codification to perform a comparison. My goal was to identify the differences between the ecosystems and tasks that were present in primary care and the community hospital. Comparing CWA models is not a well-defined process in the literature, and I developed an approach to conduct this comparison based on seminal works. I used the approach to systematically compare each phase of my CWA models. I found that the analysis of both system domains in parallel enabled a richer understanding of each environment that may not have been achieved independently. In addition, I discovered that a rich environment exists around data codification processes, and this context influences and distinguishes the actions of users. While the tasks in both domains were seemingly similar, they took place with different priorities and required different competencies.
After building and comparing models, I investigated the summarizing task in primary care more closely by analysing data within a FHT’s reporting database. The goal of this study was to understand data quality tradeoffs between timeliness, validity, completeness, and use in primary care users. Data quality measures and metrics were developed through interviews with a focus group of managers. After analysing data quality measures for 196,967 patient encounters, I created baselines, modelled each measure with logit binomial regression to show correlations, characterized tradeoffs, and investigated data quality interactions. Based on the analysis, I found a positive relationship between validity and completeness, and a negative relationship between timeliness and use. Use of data and reductions in entry delay were positively associated with completeness and validity. These results suggested that if users are not provided with sufficient time to record data as part of their regular workflow, they will prioritize their time to spend more time with patients. As a measurement of the effectiveness of a system, the negative correlation between use and timeliness points to a self-reinforcing data repository that provides users with little external value. These findings were consistent with the modelling work and also provided useful insight to study data quality improvements within the system.
I used my measures from the data analysis to select design priorities and behaviour changes that should, according to my ongoing case study, improve data quality. Then I developed several design concepts by combining CWA, a framework for behaviour change, and a design framework for persuasive systems. The design concepts adopted different persuasion principles to change specific behaviours.
To test the validity of my design concepts, I worked with a FHT to implement some of my proposed interventions during a field study. This involved the introduction of a non-invasive summary screen into the user workflow. After the summary screen had been deployed for eight weeks, I received secondary data from the FHT to analyse. First, I performed a pre-post measurement of several data quality measures by doing a simple paired t-test. To further understand the results, I borrowed from healthcare quality improvement methodologies and used statistical process control charts to understand the overall context of the measures. The average delay per entry was reduced by 3.35 days, and the percentage of same-day entries increased by 10.3%. The number of records that were complete dropped by 4.8%. Changes to entry accuracy and report generation were not significant. Several additional insights could be extracted by looking at each the XmR chart for each variable and discussing the trends with the FHT. Feedback was also collected from users through an online survey.
Through the use of a case study spanning several years, I was able to reach the following conclusions: data codification and data quality are manufactured within complex socio-technical systems and users are heavily influenced by a variety of factors within their ecosystem; persuasive design, informed with data from a CWA, is an effective technique for creating ecologically relevant persuasive designs; and data quality in primary care can be improved through the use of these designs in the system’s user interface. There are interesting opportunities to apply the results of my work to other jurisdictions. A strength of this work lies in its usefulness for international readers to draw comparisons between different systems and health care environments throughout the world. | en |
