Do No Harm Guide: A Behind-the-Scenes Look
Data practitioners, quantitative researchers by trade, often have little concept of what it takes to conduct qualitative research — the methods, skills, and practices necessary to gather and apply such data. But qualitative researchers, in our experience, do have quantitative skills: cleaning data, estimating summary statistics, and running regressions.
Last fall, we wrote a short, four-page report to update Urban’s data visualization style guide with guidelines on how to create charts and maps through a lens of diversity, equity, and inclusivity. This report drew from the few references that existed on this topic, but it was clear we had more to say. We thus proposed a longer paper to the Tableau Foundation that would, through qualitative research of data visualization field, provide a guide for encouraging thoughtfulness in how analysts work and present their data.
The resulting Do No Harm Guide, a 40+ page resource, emerged from dozens of interviews we conducted with nearly 20 people. Although neither of us are trained qualitative researchers, we believe the lessons we learned while writing the guide could provide valuable insight for other quantitative researchers and help bring the two fields closer together.
Conducting the Interviews
At the outset of the project, we identified about six people we wanted to interview for this work based on the references and citations included in our first paper. But our list of potential interviewees expanded quickly as we continued to research the issues associated with racial equity in data and as our interviewees connected us with more and more people doing relevant work.
After reaching out to our interviewees and setting a date and time for the discussion, we sent them an agenda for the interview that included the five main content areas we hoped to discuss along with our original four-page report. The agenda would allow our interviewees to understand our purpose and goals for the project as well as help frame the conversation prior to speaking with them.
Some of the conversations veered into different areas — topics we hadn’t considered or challenges that were specific to their particular domain or field. Other conversations were straightforward and moved through the interview guide one question at a time. For each interview, both of us took our own notes and uploaded them to a shared folder.
Overall, we learned three techniques that we felt were useful in these discussions:
1. Let the interviewee speak. We invited them for a reason and letting them tell their stories and share their experiences were key to the final product. Sometimes that meant we did not ask every single question we had prepared in our interview agenda; other times, it meant coming up with and asking entirely new questions that were not in the original agenda as we adapted to the way the conversation flowed.
2. Be familiar with the interviewees’ work. Our interviewees were a diverse group, from the types of institutions they worked for, to the kinds of roles they performed, to their areas of expertise and the methods they used for their work. Knowing what kinds of projects they worked on before the interview was not only helpful in crafting the actual interview, but also assuring the person that we truly valued their input and their personal experiences.
3. Fully record and transcribe interviews. At the beginning of each interview, we asked permission from our interviewees to record and transcribe the session. Having a recording and full transcription of our conversations was important for two reasons: they served as materials that we could refer to when we started writing a draft of the guide weeks after the interview concluded and they helped ensure we accurately quoted our interviewees.
Writing the Do No Harm Guide
Through this project, we wanted to create a practical guide that individuals and organizations can use in their data communication efforts on a daily basis. To meet that goal, we prioritized clearly organizing the content, writing short sections, and providing concrete examples, which would allow users to understand the guidelines and their application quickly.
With our existing notes, we focused on the key areas of the interviews we knew would be valuable for the final guide and for which we might want to include direct quotations. We each took turns building out a detailed outline for the entire guide, which enabled us to individually pull quotations and other references directly from the transcripts into a larger document.
Good writing is a process and an evolution. In this case, we were especially mindful of trying to live by what we were writing — for example, we argue that we should all try to use “people-first” language when describing people and communities. But in the first draft of the paper where we reference a bar chart example from Catherine D’Ignazio and Lauren Klein’s book, Data Feminism, we wrote “showing the rate of mental health diagnosis of inmates by race” when instead, a better approach (and which is in the final guide) would be, “showing the rate of mental health diagnosis of incarcerated people by race.” Thus, the writing took on a different level of detailed review and consideration.
Overall, we learned two techniques that we felt were useful in this stage of the work:
1. Work individually, then as a team. We found it very valuable for each of us to take separate notes, summarize our own thoughts, and then pull things together. This enabled each of us to highlight the sections and topics we felt were important for the work.
2. Build a detailed outline. Especially with this many interviews and outside references, it was incredibly valuable to have a detailed outline for the entire product. For each section, we could pull in separate references and quotes from our interviews, and from there start to write the full narrative.
Wrapping Up
We conclude this post with a final caveat about these kinds of qualitative methods and research: we are not suggesting that pursuing qualitative research is as simple as conducting a few interviews. Entire fields are devoted to qualitative methods, which require study, training, and practice to master. From our experience of writing this guide, we feel that understanding and utilizing qualitative methods is important and can very much complement quantitative research. Being able to probe deeper and ask open-ended questions can help researchers understand the how and the why of what they observe in the data. We hope that this guide — and our experience writing it — will help move qualitative and quantitative research fields closer to one another.
