Improving Product & Service Emails for Salesforce Administrators

Research & Insights Intern @ Salesforce
June 2020 to Sept 2020

OVERVIEW

For 12 weeks, I focused on how small-to-mid business Salesforce Administrators engage, perceive, and interact with Salesforce product service email communications.

Prior research on the Administrator customer base showed that many were from small-to-midsized organizations and on the lowest level of Salesforce support. This study dived into the compounding effects on this user base when it comes to receiving Salesforce Product & Service email communications.

🙌

FYI: Insights from this research led to a revision of the email approval process for Admins worldwide!

Following my internship, one impact of this research work includes a revision of the email approval process within the Technology, Communications, and Readiness team to include additional checks and balances. This helped ensure understandability and actionability for these target Salesforce Administrators.

Role

UX Researcher

Timeline

12 Weeks (June to Sept 2020)

Stakeholders

Arun K. (Director of Marketing Communications), Robin C (Intern Manager), Technical Writers and Project Managers from the Technology Communications Readiness team

Methods

Interviews, Surveys, Jobs to be Done, Heuristic Evaluation, Literature Review, Stakeholder Interviews, Journey Mapping, Workshops

Tools

SurveyMonkey, Miro, Webex

BACKGROUND

Multiple times a year, Salesforce sends out Product & Service email notifications to update Salesforce Administrators on planned technology changes.

The Technology Communications & Readiness (TCR) team, made up of Technical Writers and Project Managers, is responsible for crafting and sending out communications to ensure that customers are prepared for planned technology changes. These communications can be categorized as Product & Service Notifications — and common examples of this include feature retirement emails, incident emails, infrastructure change emails, and more.

These emails are critical for Admins to avoid disruption and successfully do their jobs!

Salesforce Administrators manage and customize the Salesforce platform for their company or organization. As Admins are the point-of-contact for all Salesforce platform needs at their company, they need to be updated whenever there’s a change, update, or release so they can stay up to date and successfully manage the platform for their end-users.

PROBLEM STATEMENT

But the problem is...these emails are super complex!

This results in compounding challenges for the majority of the Admin user base — those who tend to be from SMB organizations and on the lowest level of Salesforce support.

In reality, these were lengthy and technical emails, ranging in complexity. This is problematic when looking at data on the Salesforce Administrator user base — 68% of Admins are on the most basic level of Salesforce support and from small-to-midsize companies. This means that most of these Admins are on the lower end of the proper Salesforce training and education needed digest the complexity of Salesforce, and lack the proper access to Salesforce customer service when wanting to have their questions answered.

RESEARCH OBJECTIVES

Thus, the burning question is...

How effective and understandable are Salesforce product & service email communications for small-to-mid-size business Salesforce Administrators* on the lowest level of Salesforce support*?

*SMB Administrator: defined by if their organization has less than <100k AOV
*Lowest level of Salesforce support: defined by Basic support tier

Here are some specific research questions.

💡  Identifying Opportunity Areas

Where is there opportunity to change the current email content or experience so it's most effective for these Admins?

PART 1: DIVING INTO EXISTING INSIGHTS

To kick off this study, I needed to fill in existing knowledge gaps.

After spending a week talking to various stakeholders to gather context on what I needed to know, I started off by conducting a Literature Review on existing insights at Salesforce to learn more about the following relevant spaces.

This lead to a series of initial learnings that supported the rest of my study, such as...

✍🏻

A writers note about handling a firehose of information —

I tend to be the type of person that wants to get into the details as much as I can — but quickly into this internship, I realized that was going to be impossible! Salesforce is incredibly complex, and after adjusting to my new normal of what would be 12 weeks of an insane amount of info, I had to learn to figure out quickly what was noise in order to effectively answer this problem statement within the time frame.

PART 2: DEFINING CONTENT EFFECTIVENESS

While I had found existing principles around content effectiveness when assessing Salesforce-branded content, I had to ask —“Are these the same principles we should use to define what makes Product & Service email content effective?”.

How do we define what makes an effective email? In this case, it depends on what the Salesforce Administrator, the our target user, defines an “effective” and “successful” email by. To start understanding this, I conducted a workshop with the TCR team utilizing a Jobs-to-be-Done framework to understand the goals and motivations Salesforce Administrators have when encountering these product service email communications. From the set of jobs created from this workshop, I revised the content effectiveness principles to better fit the context of Product & Service emails.

🤔

Q&A: Why conduct a stakeholder workshop around Jobs-to-be-Done?

This was a great way to quickly draw from the team's knowledge and think more about the mental models they use when writing these emails within a 1-hour timeframe. Jobs-to-be-Done helped get at the user goals for these email types, and overall, this workshop also helped introduce this framework to a non-researcher crowd.

PART 3: TALKING TO ADMINS

In a part-survey part screener, I kicked off this study with a qualitative survey conducted with 100+ Salesforce Administrators.

Acting as a part-survey and part-screener, I built a survey gathering details on participant criteria (What size is their company/organization? Are they an Admin? What support level are they on?) as well as asking initial insights about their behavior and past experience when interacting with Salesforce Product & Service emails.

This was followed up with 1:1 interviews with 10 Salesforce Administrators over the course of a week, leveraging a Heuristic Evaluation activity.

After screening through 100+ screener participants, I selected 10 Salesforce Administrators to talk to through 60-minute 1:1 remote interviews. Towards the latter half of each session, I conducted a Heuristic Evaluation activity to further understand and observe their perceptions and understandability of these emails over a series of metrics.

Step 1 of Heuristic Eval Activity

Over screen-share, each participant read two Salesforce product service emails that differed in style/content.

Step 2 of Heuristic Eval Activity
‍‍
Following each email, participants rated a series of statements created from the revised content effectiveness principles and “Jobs-to-be-Done” list. This aimed to quantitatively measure the effectiveness of the email content.

🤔

Q&A: Why design a 1:1 interview with a Heuristic Evaluation activity?

To answer the research question, I also needed to measure how "effective" these emails might be: leveraging the revised content effectiveness principles to build a Heuristic Evaluation activity allowed me to properly gather this data in a more structured way.

SAMPLE INSIGHT & DELIVERABLES

Over the last two weeks, I delivered various read-outs to different teams at Salesforce. Here's a sample insight from one of those read-out decks.

The last two weeks of the internship involved reading out a research deck to various stakeholders in differing forums. As I needed to deliver insights to differing teams (ex: the Technical Writing team vs. the UX team) I built different research decks depending on the audience. Here’s a sample from that deck I shared with my main stakeholders —

SAMPLE INSIGHT

Salesforce Administrators on Basic-level support find other ways to compensate for the 1:1 guidance they lack through Salesforce support.

Basic-level support misses the mark when it comes to delivering on quick and easy 1:1 support with Salesforce customer service reps. However, Admins have other means to gather that 1-1 support when they’re really stuck and need help — they have learned to leverage existing connections with support at Salesforce, hire outside Salesforce consultants and contractors, or even utilize social media to tweet or reach out to those related to the topic.

💬

“I have pre-existing relationships with customer support and customer success people, who are now newly employed by Salesforce... So I have standard [Basic] level support but also other means of communicating besides googling what we already know.” — SMB Salesforce Administrator

💡

So...what can we take away from this insight? First off, 1:1 support matters! Secondly, Basic-level support Admins need to be able to communicate Salesforce issues and insights to people who aren’t Salesforce support, and we have to help them get there.

RESEARCH IMPACT

One impact of this research work includes a revision of the email approval process within the team to include additional checks before email send-out to Admins.

Following my internship, one impact of this research work includes a revision of the email approval process within the team to include additional checks in order to ensure understandability and actionability for these target Salesforce Administrators. I also returned back to the Research & Insights team the following summer as a full-time researcher!

LEARNINGS & TAKEAWAYS

I actually kept a doc of learnings I had throughout the internship — every time something stood out, I jotted it down. Here's a few things I had written down at the time.

Conducting research can be ambiguous; it's important to learn how to embrace ambiguity.

Embracing ambiguity is important when conducting research. This project was nuanced and vague from the start, and having an open mind (instead of stubbornly going down one path) was critical, as the more we uncovered about the problem, the more we could figure out where to go next.

Narrow your research scope by cutting down on what you don't need to know.

And along with that — knowing which rabbit hole you shouldn't dive into is a skill to learn! Salesforce is complex. I was absorbing tons of new information every day, but with a limited amount of time, I had to learn to figure out quickly what was noise in order to effectively answer this problem statement within the 12-week time frame.