City Data - Design Research Method

I developed a set of customized design research methods to identify and describe data needs of city employees in order to quickly deliver an inexpensive and easy-to-use data tool.
 

My research method was unique because I was able to apply qualitative research methods to improve our partner's use of quantitative data. In one meeting, I helped our stakeholders:

  • Surface the most actionable data points

  • Identify why staff would use these data points to achieve their goals

  • Ensure that people’s roles and responsibilities could support the final plan
     

Below you can learn about my process and how the research methods works.

 

To keep this case study streamlined, I wrote a blog post where I describe the partner staff and their data needs in more detail.

Role: Product Researcher
 

Tools: User research, product design, web analytics
 

Timeline: 3 weeks

My Role

This project took place during my time as a Product Apprentice at NYC Opportunity, a city agency working to reduce poverty and increase access.

 

While there, I helped the digital product team solve tough interaction design problems, run user sessions, compose product briefs, and build web analytics dashboards.

The awesome apprentice team at NYC Opportunity.

The Challenge

A partner city agency had asked us to provide a weekly data report of web analytics for a digital product we built for their staff. At the time, they already received similar data from us.

We could have immediately jumped into a data solution. But first we wanted to better understand the reasons behind the request. 
 

  • What was the reason behind a request for weekly data? 

  • Among dozens of web metric data points, which would be most useful? 

  • Lastly, how could we offer a data solution that was manageable to maintain for both teams?
     

With all of these challenges in mind, I tailored common user research methods to help the working group scope a new data report.

Stakeholder Interview activity. We completed user stories (left) for each staff role.

Process

Stage 1 - Research
 

  • I gathered baseline research by interviewing team members who knew the history of the city agency partnership.
     

  • At the end of those interviews, I tested out the research methods I was developing in order to get feedback from my colleagues.
     

  • I then used the final research methods in a meeting with our partner staff. The research methods included a Stakeholder Interview and Closed Card Sort.
     

    • The Stakeholder Interview took the form of a worksheet with partially blank user stories for each partner staff role and how they might use the data being requested, and why.
       

    • The Closed Card Sort included printed cards for each of the 40+ web analytics data points they requested. I asked them to sort the cards into three categories: "I can act on it now," "I can act on it later," and "I don't expect to act on it for a while."

      You can learn why I chose these methods further below.

 

Stage 2 - Synthesis
 

  • We did most of the synthesis together in the meeting! With this collaborative approach, we were able to arrive at a conclusion together in a manner that nurtured the partner relationship.
     

  • Together we generated insights from the patterns we noticed in the staff's answers to the research questions.
     

  • From these insights we could more clearly see our opportunities. In response, we came up with a few solutions:
     

    • We went from a request for 40+ data points to only three. The three data points were the ones that very closely matched their current goals, making them more actionable.
       

    • We also went from a weekly report for an unclear audience to just a monthly report for senior staff meetings. This change made sense, as it was primarily senior staff who could first act on the data.
       

Stage 3 - Design
 

  • I designed a report using the three data points in Microsoft Excel.
     

  • Excel was a perfect tool because it didn’t require costly technical development. Plus anyone with access to spreadsheet software could access the report. 
     

Stage 4 - Testing
 

  • Since we knew our partner staff so well, we were able to give them the Excel report for immediate use. While user testing would have been ideal, in this case the research we did together was enough to match the needs and available resources for the project.
     

  • When we checked in, we learned they were able to easily use the monthly reports and had no further requests.
     

Stage 5 - Handoff
 

  • Even better, we learned that our partners had  adopted the reporting process themselves. They no longer needed us to send a monthly report.
     

  • They were able to take the raw data and use Python scripting to complete an automated analysis in time for their senior staff meetings.
     

Closed card sort activity. We went from 40+ data points to the most actionable three.

How It Works

The methods I used are familiar to user researchers: a Stakeholder Interview and a Closed Card Sort.

 

What makes my approach unique is how I remixed these methods to apply to data needs.

 

I was able to address everything from the raw data to the team’s goals and team roles. Essentially, my research methods managed to ask a fairly pointed question without putting our partners on the spot: what were they planning to do differently with all the data they were requesting?

 

These research methods helped us arrive at the necessary answers, but in a collaborative way.

 

Stakeholder Interview and User Stories
 

I conducted a Stakeholder Interview that relied on user stories as interview prompts. We completed a worksheet with partially completed user stories for each partner staff role and their data needs.

 

This was an effective substitute for intensive research interviews that we didn’t have time for, nor were necessary because we knew the partner staff  well. However, we did need to get on the same page what they really needed, and why.

 

There were two steps:
 

  1. Ask the staff point person to start the user stories by describing people’s roles. 

    I’m a program manager.
    My role is _______.

     

  2. Then ask them to complete user stories by describing how each team member could use the data.

    I can use data to _______ so that ________.

These user stories helped us align around who would be using the requested data, and why. This alignment gave everyone in the meeting shared context before we dove into more detailed data points.


 

Closed Card Sort

Then I asked the partner staff to sort the data points they requested into the three categories. The categories I proposed were time-based: could they act on the data point now, later, or not for a while? The purpose was to surface which data points were most actionable now

There were three steps:

 

  1. Place 40+ web analytics data points, printed out on cards, onto the table.
     

  2. Ask the staff person to sort them into three buckets:

    I can act on this data now.
    I can act on this data later.
    I don't expect to act on this data for a while.
     

  3. One the cards are sorted, discuss how to handle the data points that are needed now. Who exactly could use them, how often, and what format they would need them in?

The Closed Card Sort helped naturally surface the current goals of the partner staff. By asking why certain data points were more actionable than others, we learned that their most important goal was to ensure their staff knew about and were using the digital product.

 

From this insight we were able to align around a final plan with a perfectly understandable justification: they wanted their staff to use the digital product we built them, and they wanted data to show whether they were meeting their goal.

What’s Happened Since?

  • Our partners gained a clearer understanding of their own needs which led to an easier to maintain solution that met their most important goals.
     

  • Later on they adopted the reporting themselves. They took the raw data we sent and used Python scripting to complete an automated analysis in time for their senior staff meetings.
     

  • I’ve since used this method at my current role on a data and product team at the NYC Taxi & Limousine Commission.
     

  • I’ve given a presentation on the principles behind using these research methods as applied to data purposes.