Dask Working Notes - Posts tagged User Survey

2021 Dask User Survey

2021-09-15T00:00:00+00:00

This post presents the results of the 2021 Dask User Survey, which ran earlier this year. Thanks to everyone who took the time to fill out the survey! These results help us better understand the Dask community and will guide future development efforts.

The raw data, as well as the start of an analysis, can be found in this binder:

Let us know if you find anything in the data.

Highlights
Who are Dask users?
How people like to use Dask
Diagnostics
Stability
User satisfaction, support, and documentation
Suggestions for improvement
Previous survey results

Highlights

We had 247 responses to the survey (roughly the same as last year, which had just under 240 responses). Overall, responses were similar to previous years.

We asked 43 questions in the survey (an increase of 18 questions compared to the year before). We asked a bunch of new questions about the types of datasets people work with, the stability of Dask, and what kinds of industries people work in.

Our community wants:

More documentation and examples
More intermediate level documentation
To improve the resiliency of Dask (i.e. do computations complete?)

Users also value these features:

Improved scaling
Ease of deployment
Better scikit-learn & machine learning support

The typical Dask user

The survey shows us there is a lot of diversity in our community, and there is no one way to use Dask. That said, our hypothetical “typical” Dask user:

Works with gigabyte sized datasets
Stored on a local filesystem
Has been using Dask between 1 and 3 years
Uses Dask occasionally, not every day
Uses Dask interactively at least part of the time
Uses a compute cluster (probably)
Likes to view the Dask dashboard with a web browser
For the most part, Dask is stable enough for their needs, but improving the Dask’s resiliancy would be helpful
Uses the Dask dataframe, delayed, and maybe the Dask Array API, alongside numpy/pandas and other python libraries
The most useful thing that would help this person is more documentation, and more examples using Dask in their field.
They likely work in a scientific field (perhaps geoscience, life science, physics, or astronomy), or alternatively they might work in accounting, finance, insurance, or as a tech worker.

You can read the survey results from previous years here: 2020 survey results, 2019 survey results.

# Let's load in the survey data...
%matplotlib inline

from pprint import pprint
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import textwrap
import re


df2019 = (
    pd.read_csv("data/2019-user-survey-results.csv.gz", parse_dates=["Timestamp"])
      .replace({"How often do you use Dask?": "I use Dask all the time, even when I sleep"}, "Every day")
)

df2020 = (
    pd.read_csv("data/2020-user-survey-results.csv.gz")
      .assign(Timestamp=lambda df: pd.to_datetime(df['Timestamp'], format="%Y/%m/%d %H:%M:%S %p %Z").astype('datetime64[ns]'))
      .replace({"How often do you use Dask?": "I use Dask all the time, even when I sleep"}, "Every day")
)

df2021 = (
    pd.read_csv("data/2021-user-survey-results.csv.gz")
      .assign(Timestamp=lambda df: pd.to_datetime(df['Timestamp']).astype('datetime64[ns]'))
      .replace({"How often do you use Dask?": "I use Dask all the time, even when I sleep"}, "Every day")
)

common = df2019.columns.intersection(df2020.columns).intersection(df2021.columns)
added = df2021.columns.difference(df2020.columns)
dropped = df2020.columns.difference(df2021.columns)

df = pd.concat([df2019, df2020, df2021])
df['Year'] = df.Timestamp.dt.year
df = df.set_index(['Year', 'Timestamp']).sort_index()

Who are Dask users?

Most people said they use Dask occasionally, while a smaller group use Dask every day. There is a wide variety in how long people have used Dask for, with the most common response being between one and three years.

q = "How often do you use Dask?"
ax = sns.countplot(y=q, data=df2021[q].dropna().str.split(";").explode().to_frame());
ax.set(ylabel="", title=q);

q = "How long have you used Dask?"  # New question in 2021
order = ["More than 3 years", "1 - 3 years", "3 months - 1 year", "Less than 3 months", "I've never used Dask"]
ax = sns.countplot(y=q, data=df2021[q].dropna().str.split(";").explode().to_frame(), order=order);
ax.set(ylabel="", title=q);

Just over half of respondants use Dask with other people (their team or organisation), and the other half use Dask on their own.

q = "Do you use Dask as part of a larger group?"
order = [
    'I use Dask mostly on my own',
    'My team or research group also use Dask',
    'Beyond my group, many people throughout my institution use Dask',
]
ax = sns.countplot(y=q, data=df2021[q].dropna().str.split(";").explode().to_frame(), order=order)
ax.set(ylabel="", title=q);

In the last year, there has been an increase in the number of people who say that many people throughout their institution use Dask (32 people said this in 2021, compared to 19 in 2020). Between 2019 and 2020, there was a drop in the number of people who said their immediate team also uses Dask (121 people said this in 2019, compared to 94 in 2020). It’s not clear why we saw either of these changes, so it will be interesting to see what happens in future years.

q = 'Do you use Dask as part of a larger group?'
ax = sns.countplot(y=q, hue="Year", data=df.reset_index());
ax.set(ylabel="", title=q);

What industry do you work in?

There was a wide variety of industries represented in the survey.

Almost half of responses were in an industry related to science, academia, or a governmant laboratory. Geoscicence had the most responses, while life sciences, physics, and astronomy were also popular fields.

Around 30 percent of responses were from people in businesss and tech. Of these, there was a roughly even split between people in accounting/finance/insurance vs other tech workers.

Around 10 percent of responses belonged to manufacturing, engineering, and other industry (energy, aerospace, etc). The remaining responses were difficult to categorise.

q = "What industry do you work in?"  # New question in 2021
data = df2021[q].dropna().str.split(";").explode().to_frame()
order = data.value_counts()[data.value_counts() > 1].keys().get_level_values(0)
ax = sns.countplot(y=q, data=data, order=order);
ax.set(ylabel="", title=q);

How easy is it for you to upgrade to newer versions of Python libraries?

The majority of users are able to easily upgrade to newer versoins of python libraries when they want.

q = "How easy is it for you to upgrade to newer versions of Python libraries"
sns.countplot(y=q, data=df2021[q].dropna().explode().to_frame()).set_ylabel('Scale from 1 (Difficult) to 4 (Easy)');

How people like to use Dask

People like to use Dask in conjunction with numpy and pandas, along with a range of other python libraries. The most popular Dask APIs are Dask Dataframes, Dask Delayed, and Dask Arrays.

The vast majority of people like to use Dask interactively with Jupyter or IPython at least part of the time, and most people view the Dask Dashboard with a web browser.

What are some other libraries that you often use with Dask?”

The ten most common libraries people use with Dask are: numpy, pandas, xarray, scikit-learn, scipy, statsmodels, matplotlib, xgboost, numba, and joblib.

q = "What are some other libraries that you often use with Dask?"
data = df2021[q].dropna().str.lower().str.split(", ").explode().to_frame()
labels = pd.value_counts(data[q]).iloc[:10].index
sns.countplot(y=q, data=data, order=labels).set_ylabel('');

Dask APIs

The three most popular Dask APIs people use are:

In 2021, we saw a small increase in the number of people who use dask delayed, compared with previous years. This might be a good thing, it’s possible that as people develop experience and confidence with Dask, they are more likely to start using more advanced features such as delayed. Besides this change, preferences were pretty simliar to the results from previous years.

apis = df2021['Dask APIs'].str.split(", ").explode()
top = apis.value_counts().loc[lambda x: x > 10]
apis = apis[apis.isin(top.index)].reset_index()

sns.countplot(y="Dask APIs", data=apis);

Interactive or Batch?

The vast majority of people like to use Dask interactively with Jupyter or IPython at least part of the time. Less than 15% of Dask users only use Dask in batch mode (submitting scripts that run in the future).

q = 'Interactive or Batch?'
data = df2021[q].dropna()
data = data.str.replace('Interactive:  I use Dask with Jupyter or IPython when playing with data, Batch: I submit scripts that run in the future', "Interactive and Batch")
data = data.str.replace('Interactive:  I use Dask with Jupyter or IPython when playing with data', "Interactive")
data = data.str.replace('Batch: I submit scripts that run in the future', "Batch")
order = ["Interactive and Batch", "Interactive", "Batch"]
sns.countplot(y=q, data=data.explode().to_frame(), order=order).set_ylabel('');

How do you view Dask’s dashboard?

Most people look at the Dask dashboard using a web browser. A smaller group use the dask jupyterlab extension.

A few people are still not sure what the dashboard is all about. If that’s you too, you might like to watch this 20 minute video that explains why the dashboard is super useful, or see the rest of the docs here.

q = "How do you view Dask's dashboard?"
ax = sns.countplot(y=q, data=df2021[q].dropna().str.split(", ").explode().to_frame());
ax.set(ylabel="", title=q);

Local machine or Cluster?

Roughly two thirds of respondants use a computing cluster at least part of the time.

q = 'Local machine or Cluster?'
df[q].dropna().str.contains("Cluster").astype(int).groupby("Year").mean()

Year
2019    0.654902
2020    0.666667
2021    0.630081
Name: Local machine or Cluster?, dtype: float64

q = 'Local machine or Cluster?'
order = [
    'Personal laptop',
    'Large workstation',
    'Cluster of 2-10 machines',
    'Cluster with 10-100 machines',
    'Cluster with 100+ machines'
]
ax = sns.countplot(y=q, data=df2021[q].dropna().str.split(", ").explode().to_frame(), order=order);
ax.set(ylabel="", title=q);

If you use a cluster, how do you launch Dask?

SSH is the most common way to launch Dask on a compute cluster, followed by a HPC resource manager, then Kubernetes.

q = "If you use a cluster, how do you launch Dask? "
data = df2021[q].dropna()
data = data.str.replace("HPC resource manager (SLURM, PBS, SGE, LSF or similar)", "HPC resource manager (SLURM PBS SGE LSF or similar)", regex=False)
data = data.str.replace("I don't know, someone else does this for me", "I don't know someone else does this for me", regex=False)
data = data.str.split(", ").explode().to_frame()
order = data.value_counts()[data.value_counts() > 1].keys().get_level_values(0)
ax = sns.countplot(y=q, data=data, order=order);
ax.set(ylabel="", title=q);

If you use a cluster, do you have a need for multiple worker types in the same cluster?

Of the people who use compute clusters, a little less than half have a need for multiple worker types in the same cluster. Examples of this might include mixed workers with GPU vs no GPU, mixed workers with low or high memory allocations, etc.

q = "If you use a cluster, do you have a need for multiple worker / machine types (e.g. GPU / no GPU, low / high memory) in the same cluster?"  # New question in 2021
ax = sns.countplot(y=q, data=df2021[q].dropna().str.split(";").explode().to_frame());
ax.set(ylabel="", title="Do you need multiple worker/machine types on a cluster?");

Datasets

How large are your datasets typically?

Dask users most commonly work with gigabyte sized datasets. Very few users work with petabyte sized datasets.

q = "How large are your datasets typically?"  # New question in 2021
ax = sns.countplot(y=q, data=df2021[q].dropna().str.split(", ").explode().to_frame());
ax.set(ylabel="", title=q);

Where are your datasets typically stored?

Most people store their data on a local filesystem.

q = "Where are your datasets typically stored?"  # New question in 2021
data = df2021[q].dropna().str.split(", ").explode().to_frame()
order = data.value_counts()[data.value_counts() > 1].keys().get_level_values(0)
ax = sns.countplot(y=q, data=data, order=order);
ax.set(ylabel="", title=q);

What file formats do you typically work with?

The two most common file formats (csv and parquet) are popular among Dask Dataframe users. The JSON file format is also very commonly used with Dask. The fourth and fifth most common filetypes (HDF5 and zarr) are popular among Dask Array users. This fits with what we know about the Dask Dataframe API being the most popular, with Dask Arrays close behind.

q = "What file formats do you typically work with?"  # New question in 2021
data = df2021[q].dropna().str.split(", ").explode().to_frame()
order = data.value_counts()[data.value_counts() > 1].keys().get_level_values(0)
ax = sns.countplot(y=q, data=data, order=order);
ax.set(ylabel="", title=q);

This survey question had a long tail: a very wide variety of specialized file formats were reported, most only being used by one or two individuals who replied to the survey.

A lot of these specialized file formats store image data, specific to particular fields (astronomy, geoscience, microscopy, etc.).

list(data.value_counts()[data.value_counts() == 1].keys().get_level_values(0))

['proprietary measurement format',
 'netCDF3',
 'czi',
 'specifically NetCDF4',
 'grib2',
 'in-house npy-like array format',
 'jpeg2000',
 'netCDF4 (based on HDF5)',
 'proprietary microscopy file types. Often I convert to Zarr with a loss of metadata.',
 'sas7bdat',
 'npy',
 'npy and pickle',
 'root with uproot',
 'root',
 'regular GeoTiff',
 '.npy',
 'Text',
 'VCF BAM CRAM',
 'UM',
 'CASA measurement sets',
 'Casa Tables (Radio Astronomy specific)',
 'Custom binary',
 'FITS',
 'FITS (astronomical images)',
 'FITS and a custom semi-relational table specification that I want to kill and replace with something better',
 'Feather (Arrow)',
 'GPKG',
 'GeoTIFF',
 'NetCDF4',
 'Netcdf',
 'Netcdf4',
 'PP',
 'SQL',
 'SQL query to remote DB',
 'SQL to Dataframe',
 'Seismic data (miniSEED)',
 'TFRecords',
 'TIFF',
 'Testing with all file formats. Just want it as a replacement for spark. ',
 '.raw image files',
 'ugh']

XKCD comic “Standards” https://xkcd.com/927/

Preferred Cloud?

The most popular cloud solution is Amazon Web Services (AWS), followed by Google Cloud Platform (GCP) and Microsoft Azure.

q = "Preferred Cloud?"
order = [
    "Amazon Web Services (AWS)",
    "Google Cloud Platform (GCP)",
    "Microsoft Azure",
    "Digital Ocean",
]
ax = sns.countplot(y=q, data=df2021[q].dropna().str.split(", ").explode().to_frame(), order=order);
ax.set(ylabel="", title=q);

Do you use Dask projects to deploy?

Among those who use dask projects to deploy, dask-jobqueue and dask helm chart are the two most popular options. There was a wide variety of projects people used for deployment.

q = "Do you use Dask projects to deploy?"
order = [
    "dask-jobqueue",
    "dask's helm chart",
    "dask-kubernetes",
    "dask's docker image at daskdev/dask",
    "dask-gateway",
    "dask-ssh",
    "dask-cloudprovider",
    "dask-yarn",
    "qhub",
    "dask-mpi",
]
ax = sns.countplot(y=q, data=df2021[q].dropna().str.lower().str.split(", ").explode().to_frame(), order=order);
ax.set(ylabel="", title=q);

Diagnostics

We saw earlier that most people like to view the Dask Dashboard using their web browser.

In the dashboard, people said the most useful diagnostics plots were:

The task stream plot
The progress plot, and
The memory useage per worker plot

q = "Which Diagnostic plots are most useful?"  # New question in 2021
ax = sns.countplot(y=q, data=df2021[q].dropna().str.split(', ').explode().to_frame());
ax.set(ylabel="", title=q);

We also asked some new questions about diagnostics in 2021.

We found that most people (65 percent) do not use Dask performance reports, which is a way to save the diagnostic dashboard to static HTML plots for later review.

q = "Do you use Dask's Performance reports?"  # New question in 2021
ax = sns.countplot(y=q, data=df2021[q].explode().to_frame(), order=["Yes", "No"]);
ax.set(ylabel="", title=q);

Very few people use Dask’s Prometheus metrics. Jacob Tomlinson has an excellent article on Monitoring Dask + RAPIDS with Prometheus + Grafana, if you’re interested in learning more about how to use this feature.

q = "Do you use Dask's Prometheus Metrics?"  # New question in 2021
ax = sns.countplot(y=q, data=df2021[q].explode().to_frame(), order=["Yes", "No"]);
ax.set(ylabel="", title=q);

Stability

We asked a number of questions around the stability of Dask, many of them new questions in 2021.

The majority of people said Dask was resiliant enough for them (eg: computations complete). However this is an area we could improve in, as 36 percent of people are not satisfied. This was a new question 2021, so we can’t say how people opinion of Dask’s resiliancy has changed over time.

q = "Is Dask resilient enough for you? (e.g. computations complete)."  # new question in 2021
ax = sns.countplot(y=q, data=df2021[q].dropna().explode().to_frame(), order=["Yes", "No"]);
ax.set(ylabel="", title="Is Dask resilient enough for you?");

Most people say Dask in general is stable enough for them (eg: between different version releases). This is similar to the survey results from previous years.

q = "Is Dask stable enough for you?"
ax = sns.countplot(y=q, data=df2021[q].dropna().explode().to_frame(), order=["Yes", "No"]);
ax.set(ylabel="", title=q);

People also say that the API of Dask is stable enough for them too.

q = "Is Dask's API stable enough for you?"
ax = sns.countplot(y=q, data=df2021[q].dropna().explode().to_frame(), order=["Yes", "No"]);
ax.set(ylabel="", title=q);

The vast majority of people are satisfied with the current release frequency (roughly once every two weeks).

q = "How is Dask's release frequency?"  # New question in 2021
ax = sns.countplot(y=q, data=df2021[q].dropna().explode().to_frame());
ax.set(ylabel="", title=q);

Most people say they would pin their code to a long term support release, if one was available for Dask.

q = "If Dask had Long-term support (LTS) releases, would you pin your code to use them?"  # New question in 2021
ax = sns.countplot(y=q, data=df2021[q].dropna().explode().to_frame(), order=["Yes", "No"]);
ax.set(ylabel="", title="Would you pin to a long term support release?");

User satisfaction, support, and documentation

We asked a bunch of new questions about user satisfaction in the 2021 survey.

How easy is Dask to use?

The majority of people say that Dask is moderately easy to use, the same as in previous surveys.

q = "On a scale of 1 - 5 (1 being hardest, 5 being easiest) how easy is Dask to use?"
ax = sns.countplot(y=q, data=df2021[q].dropna().explode().to_frame());
ax.set(ylabel="1 = Difficult, 5 = Easy", title="How easy is Dask to use?");

How is Dask’s documentation?

Most people think that Dask’s documentation is pretty good.

q = "How is Dask's documentation?"  # New question in 2021
ax = sns.countplot(y=q, data=df2021[q].dropna().explode().to_frame());
ax.set(ylabel="1 = Not good, 5 = Great", title=q);

How satisfied are you with maintainer responsiveness on GitHub?

Almost everybody who responded feels positively about Dask’s maintainer responsiveness on GitHub .

q = "How satisfied are you with maintainer responsiveness on GitHub?"  # New question in 2021
ax = sns.countplot(y=q, data=df2021[q].dropna().explode().to_frame());
ax.set(ylabel="1 = Not satisfied, 5 = Thrilled", title=q);

What Dask resources have you used for support in the last six months?

The documentation at dask.org is the first place most users look for help.

The breakdown of responses to this question in 2021 was very similar to previous years, with the exception that no-one seemed to know that the Dask YouTube channel or Gitter chat existed in 2019.

q = 'What Dask resources have you used for support in the last six months?'

resource_map = {
    "Tutorial": "Tutorial at tutorial.dask.org",
    "YouTube": "YouTube channel",
    "gitter": "Gitter chat"
}

df[q] = df[q].str.replace(';',', ')  # Make separator values consistent
d = df[q].str.split(', ').explode().replace(resource_map)
top = d.value_counts()[:8].index
d = d[d.isin(top)]

fig, ax = plt.subplots(figsize=(8, 8))
ax = sns.countplot(y=q, hue="Year", data=d.reset_index(), ax=ax);
ax.set(ylabel="", title=q);

Suggestions for improvement

Which would help you most right now?

The two top priorities people said would help most right now are both related to documentation. People want more documentation, and more examples in their field. Performance improvements were also commonly mentioned as something that would help the most right now.

q = "Which would help you most right now?"
order = [
    "More documentation",
    "More examples in my field",
    "Performance improvements",
    "New features",
    "Bug fixes",
]
ax = sns.countplot(y=q, data=df2021[q].explode().to_frame(), order=order)
ax.set(ylabel="", title=q);

How can Dask improve?

We also gave people the opportunity for a free text response to the question “How can Dask imporove?”

Matt has previously written an early anecdotes blogpost that dives into the responses to this question in more detail.

He found these recurring themes:

Intermediate Documentation
Documentation Organization
Functionality
High Level Optimization
Runtime Stability and Advanced Troubleshooting

Since more documentation and examples were the two most requested improvements, I’ll summarize some of the steps forward in that area here:

Regarding more intermediate documentation, Matt says:

There is a lot of good potential material that advanced users have around performance and debugging that could be fun to publish.
Matt points out that Dask has excellent reference documentation, but lacks a lot of good narrative documentation. To address this, Julia Signell is currently investigating how we could improve the organization of Dask’s documentation (you can subscribe to this issue thread if you want to follow that discussion)
Matt comments that it’s hard to have good narrative documentation when there are so many different user narratives (i.e. Dask is used by people from many different industries). This year, we added a new question to the survey asking for the industry people work in. We added this because “More examples in my field” has been one of the top two requests for the last three years. Now we can use that information to better target narrative documentation to the areas that need it most (geoscience, life science, and finance).

q = 'What industry do you work in?'
data = df2021[df2021["Which would help you most right now?"] == "More examples in my field"]
order = data[q].value_counts()[data[q].value_counts() > 1].keys()
ax = sns.countplot(y=q, data=data[q].dropna().str.split(', ').explode().to_frame(), order=order);
ax.set(ylabel="", title="What field do you want more documentation examples for?");

What common feature requests do you care about most?

Good support for numpy and pandas is critical for most users. Users also value:

Improved scaling
Ease of deployment
Resiliancy of Dask
Better scikit-learn & machine learning support

Most feature requests are similar to the survey results from previous years, although there was an increase in the number of people who say better scikit-learn/ML support is critical to them. We also added a new question about Dask’s resiliancy in 2021.

In the figure below you can see how people rated the importance of each feature request, for each of the three years we’ve run this survey.

common = (df[df.columns[df.columns.str.startswith("What common feature")]]
          .rename(columns=lambda x: x.lstrip("What common feature requests do you care about most?[").rstrip(r"]")))
a = common.loc[2019].apply(pd.value_counts).T.stack().reset_index().rename(columns={'level_0': 'Question', 'level_1': "Importance", 0: "count"}).assign(Year=2019)
b = common.loc[2020].apply(pd.value_counts).T.stack().reset_index().rename(columns={'level_0': 'Question', 'level_1': "Importance", 0: "count"}).assign(Year=2020)
c = common.loc[2021].apply(pd.value_counts).T.stack().reset_index().rename(columns={'level_0': 'Question', 'level_1': "Importance", 0: "count"}).assign(Year=2021)

counts = pd.concat([a, b, c], ignore_index=True)

d = common.stack().reset_index().rename(columns={"level_2": "Feature", 0: "Importance"})
order = ["Not relevant for me", "Somewhat useful", 'Critical to me']
sns.catplot(x='Importance', row="Feature", kind="count", col="Year", data=d, sharex=False, order=order);

Previous survey results

Thanks to everyone who took the survey!

If you want to read more about the 2021 Dask survey, the blogpost on early anecdotes from the Dask 2021 survey is available here.

You can read the survey results from previous years here:

The 2021 Dask User Survey is out now

2021-05-25T00:00:00+00:00

The Dask User Survey is out again! Tell us how you use Dask, and help us make it better for everyone.

Click this link to take the survey.

Feedback from users is very important. It helps give us a clear picture who our users are and what is important to them. Your responses will inform prioritization for Dask development and improve the experience for the Dask community.

We expect the survey to take no more than 5-10 minutes. It has the following short sections:

How do you use Dask?
How could Dask improve?
What other tools do you use with Dask?
Optional: What do you work on?

Survey results from previous years

We will also publish answers to non-sensitive questions in our annual survey review to help keep everyone informed.

You can see the results from previous user surveys here:

2020 Dask User Survey

2020-09-22T00:00:00+00:00

This post presents the results of the 2020 Dask User Survey, which ran earlier this summer. Thanks to everyone who took the time to fill out the survey! These results help us better understand the Dask community and will guide future development efforts.

The raw data, as well as the start of an analysis, can be found in this binder:

Let us know if you find anything in the data.

We had 240 responses to the survey (slightly fewer than last year, which had about 260).
Overall, results look mostly similar to last year’s.
Our documentation has probably improved relative to last year
Respondents care more about performance relative to last year.

New Questions

Most of the questions are the same as in 2019. We added a couple questions about deployment and dashboard usage. Let’s look at those first.

Among respondents who use a Dask package to deploy a cluster (about 53% of respondents), there’s a wide spread of methods.

Most people access the dashboard through a web browser. Those not using the dashboard are likely (hopefully) just using Dask on a single machine with the threaded scheduler (though the dashboard works fine on a single machine as well).

Learning Resources

Respondents’ learning material usage is farily similar to last year. The most notable differences are from our survey form providing more options (our YouTube channel and “Gitter chat”). Other than that, examples.dask.org might be relatively more popular.

Just like last year, we’ll look at resource usage grouped by how often they use Dask.

A few observations

GitHub issues are becoming relatively less popular, which perhaps reflects better documentation or stability (assuming people go to the issue tracker when they can’t find the answer in the docs or they hit a bug).
https://examples.dask.org is notably now more popular among occasinal users.
In response to last year’s survey, we invested time in making https://tutorial.dask.org better, which we previously felt was lacking. Its usage is still about the same as last year’s (pretty popular), so it’s unclear whether we should dedicate additional focus there.

How do you use Dask?

API usage remains about the same as last year (recall that about 20 fewer people took the survey and people can select multiple, so relative differences are most interesting). We added new choices for RAPIDS, Prefect, and XGBoost, each of which are somewhat popular (in the neighborhood of dask.Bag).

About 65% of our users are using Dask on a cluster at least some of the time, which is similar to last year.

How can Dask improve?

Respondents continue to say that more documentation and examples would be the most valuable improvements to the project.

One interesting change comes from looking at “Which would help you most right now?” split by API group (dask.dataframe, dask.array, etc.). Last year showed that “More examples” in my field was the most important for all API groups (first table below). But in 2020 there are some differences (second table below).

            <tr>
                    <th id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87level0_row0" class="row_heading level0 row0" >Array</th>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row0_col0" class="data row0 col0" >10</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row0_col1" class="data row0 col1" >24</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row0_col2" class="data row0 col2" >62</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row0_col3" class="data row0 col3" >15</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row0_col4" class="data row0 col4" >25</td>
        </tr>
        <tr>
                    <th id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87level0_row1" class="row_heading level0 row1" >Bag</th>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row1_col0" class="data row1 col0" >3</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row1_col1" class="data row1 col1" >11</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row1_col2" class="data row1 col2" >16</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row1_col3" class="data row1 col3" >10</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row1_col4" class="data row1 col4" >7</td>
        </tr>
        <tr>
                    <th id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87level0_row2" class="row_heading level0 row2" >DataFrame</th>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row2_col0" class="data row2 col0" >16</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row2_col1" class="data row2 col1" >32</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row2_col2" class="data row2 col2" >71</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row2_col3" class="data row2 col3" >39</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row2_col4" class="data row2 col4" >26</td>
        </tr>
        <tr>
                    <th id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87level0_row3" class="row_heading level0 row3" >Delayed</th>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row3_col0" class="data row3 col0" >16</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row3_col1" class="data row3 col1" >22</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row3_col2" class="data row3 col2" >55</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row3_col3" class="data row3 col3" >26</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row3_col4" class="data row3 col4" >27</td>
        </tr>
        <tr>
                    <th id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87level0_row4" class="row_heading level0 row4" >Futures</th>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row4_col0" class="data row4 col0" >12</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row4_col1" class="data row4 col1" >9</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row4_col2" class="data row4 col2" >25</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row4_col3" class="data row4 col3" >20</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row4_col4" class="data row4 col4" >17</td>
        </tr>
        <tr>
                    <th id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87level0_row5" class="row_heading level0 row5" >ML</th>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row5_col0" class="data row5 col0" >5</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row5_col1" class="data row5 col1" >11</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row5_col2" class="data row5 col2" >23</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row5_col3" class="data row5 col3" >11</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row5_col4" class="data row5 col4" >7</td>
        </tr>
        <tr>
                    <th id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87level0_row6" class="row_heading level0 row6" >Xarray</th>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row6_col0" class="data row6 col0" >8</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row6_col1" class="data row6 col1" >11</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row6_col2" class="data row6 col2" >34</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row6_col3" class="data row6 col3" >7</td>
                    <td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row6_col4" class="data row6 col4" >9</td>
        </tr>
</tbody></table>

2019 normalized by row. Darker means that a higher proporiton of users of that API prefer that priority.
Which would help you most right now?	Bug fixes	More documentation	More examples in my field	New features	Performance improvements
Dask APIs

            <tr>
                    <th id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87level0_row0" class="row_heading level0 row0" >Array</th>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row0_col0" class="data row0 col0" >12</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row0_col1" class="data row0 col1" >16</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row0_col2" class="data row0 col2" >56</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row0_col3" class="data row0 col3" >15</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row0_col4" class="data row0 col4" >23</td>
        </tr>
        <tr>
                    <th id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87level0_row1" class="row_heading level0 row1" >Bag</th>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row1_col0" class="data row1 col0" >7</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row1_col1" class="data row1 col1" >5</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row1_col2" class="data row1 col2" >24</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row1_col3" class="data row1 col3" >7</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row1_col4" class="data row1 col4" >16</td>
        </tr>
        <tr>
                    <th id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87level0_row2" class="row_heading level0 row2" >DataFrame</th>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row2_col0" class="data row2 col0" >24</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row2_col1" class="data row2 col1" >21</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row2_col2" class="data row2 col2" >67</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row2_col3" class="data row2 col3" >22</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row2_col4" class="data row2 col4" >41</td>
        </tr>
        <tr>
                    <th id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87level0_row3" class="row_heading level0 row3" >Delayed</th>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row3_col0" class="data row3 col0" >15</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row3_col1" class="data row3 col1" >19</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row3_col2" class="data row3 col2" >46</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row3_col3" class="data row3 col3" >17</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row3_col4" class="data row3 col4" >34</td>
        </tr>
        <tr>
                    <th id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87level0_row4" class="row_heading level0 row4" >Futures</th>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row4_col0" class="data row4 col0" >9</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row4_col1" class="data row4 col1" >10</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row4_col2" class="data row4 col2" >21</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row4_col3" class="data row4 col3" >13</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row4_col4" class="data row4 col4" >24</td>
        </tr>
        <tr>
                    <th id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87level0_row5" class="row_heading level0 row5" >ML</th>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row5_col0" class="data row5 col0" >6</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row5_col1" class="data row5 col1" >4</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row5_col2" class="data row5 col2" >21</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row5_col3" class="data row5 col3" >9</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row5_col4" class="data row5 col4" >12</td>
        </tr>
        <tr>
                    <th id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87level0_row6" class="row_heading level0 row6" >Xarray</th>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row6_col0" class="data row6 col0" >3</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row6_col1" class="data row6 col1" >4</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row6_col2" class="data row6 col2" >25</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row6_col3" class="data row6 col3" >9</td>
                    <td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row6_col4" class="data row6 col4" >13</td>
        </tr>
</tbody></table>

Examples are again the most important (for all API groups except Futures). But “Performance improvements” is now the second-most important improvement (except for Futures where it’s most important). How should we interpret this? A charitable interpretation is that Dask’s users are scaling to larger problems and are running into new scaling challenges. A less charitable interpretation is that our user’s workflows are the same but Dask is getting slower!

What other systems do you use?

SSH continues to be the most popular “cluster resource mananger”. This was the big surprise last year, so we put in some work to make it nicer. Aside from that, not much has changed.

And Dask users are about as happy with its stability as last year.

Takeaways

Overall, most things are similar to last year.
Documentation, especially domain-specific examples, continues to be important. That said, our documentation is probably better than it was last year.
More users are pushing Dask further. Investing in performance is likely to be valuable.

Thanks again to all the respondents!

2019 Dask User Survey

2019-08-05T00:00:00+00:00

This notebook presents the results of the 2019 Dask User Survey, which ran earlier this summer. Thanks to everyone who took the time to fill out the survey! These results help us better understand the Dask community and will guide future development efforts.

The raw data, as well as the start of an analysis, can be found in this binder:

Let us know if you find anything in the data.

Highlights

We had 259 responses to the survey. Overall, we found that the survey respondents really care about improved documentation, and ease of use (including ease of deployment), and scaling. While Dask brings together many different communities (big arrays versus big dataframes, traditional HPC users versus cloud-native resource managers), there was general agreement in what is most important for Dask.

Now we’ll go through some individual items questions, highlighting particularly interesting results.

How do you use Dask?

For learning resources, almost every respondent uses the documentation.

Most respondents use Dask at least occasionally. Fortunately we had a decent number of respondents who are just looking into Dask, yet still spent the time to take the survey.

I’m curiuos about how learning resource usage changes as users become more experienced. We might expect those just looking into Dask to start with examples.dask.org, where they can try out Dask without installing anything.

Overall, documentation is still the leader across user user groups.

The usage of the Dask tutorial and the dask examples are relatively consistent across groups. The primary difference between regular and new users is that regular users are more likely to engage on GitHub.

From StackOverflow questions and GitHub issues, we have a vague idea about which parts of the library are used. The survey shows that (for our respondents at least) DataFrame and Delayed are the most commonly used APIs.

About 65.49% of our respondests are using Dask on a Cluster.

But the majority of respondents also use Dask on their laptop. This highlights the importance of Dask scaling down, either for prototyping with a LocalCluster, or for out-of-core analysis using LocalCluster or one of the single-machine schedulers.

Most respondents use Dask interactively, at least some of the time.

Most repondents thought that more documentation and examples would be the most valuable improvements to the project. This is especially pronounced among new users. But even among those using Dask everyday more people thought that “More examples” is more valuable than “New features” or “Performance improvements”.

2020 normalized by row. Darker means that a higher proporiton of users of that API prefer that priority.
Which would help you most right now?	Bug fixes	More documentation	More examples in my field	New features	Performance improvements
Dask APIs

            <tr>
                    <th id="T_820ef326_b488_11e9_ad41_186590cd1c87level0_row0" class="row_heading level0 row0" >Every day</th>
                    <td id="T_820ef326_b488_11e9_ad41_186590cd1c87row0_col0" class="data row0 col0" >9</td>
                    <td id="T_820ef326_b488_11e9_ad41_186590cd1c87row0_col1" class="data row0 col1" >11</td>
                    <td id="T_820ef326_b488_11e9_ad41_186590cd1c87row0_col2" class="data row0 col2" >25</td>
                    <td id="T_820ef326_b488_11e9_ad41_186590cd1c87row0_col3" class="data row0 col3" >22</td>
                    <td id="T_820ef326_b488_11e9_ad41_186590cd1c87row0_col4" class="data row0 col4" >23</td>
        </tr>
        <tr>
                    <th id="T_820ef326_b488_11e9_ad41_186590cd1c87level0_row1" class="row_heading level0 row1" >Just looking for now</th>
                    <td id="T_820ef326_b488_11e9_ad41_186590cd1c87row1_col0" class="data row1 col0" >1</td>
                    <td id="T_820ef326_b488_11e9_ad41_186590cd1c87row1_col1" class="data row1 col1" >3</td>
                    <td id="T_820ef326_b488_11e9_ad41_186590cd1c87row1_col2" class="data row1 col2" >18</td>
                    <td id="T_820ef326_b488_11e9_ad41_186590cd1c87row1_col3" class="data row1 col3" >9</td>
                    <td id="T_820ef326_b488_11e9_ad41_186590cd1c87row1_col4" class="data row1 col4" >5</td>
        </tr>
        <tr>
                    <th id="T_820ef326_b488_11e9_ad41_186590cd1c87level0_row2" class="row_heading level0 row2" >Occasionally</th>
                    <td id="T_820ef326_b488_11e9_ad41_186590cd1c87row2_col0" class="data row2 col0" >14</td>
                    <td id="T_820ef326_b488_11e9_ad41_186590cd1c87row2_col1" class="data row2 col1" >27</td>
                    <td id="T_820ef326_b488_11e9_ad41_186590cd1c87row2_col2" class="data row2 col2" >52</td>
                    <td id="T_820ef326_b488_11e9_ad41_186590cd1c87row2_col3" class="data row2 col3" >18</td>
                    <td id="T_820ef326_b488_11e9_ad41_186590cd1c87row2_col4" class="data row2 col4" >15</td>
        </tr>
</tbody></table>

Perhaps users of certain dask APIs feel differenlty from the group as a whole? We perform a similar analysis grouped by API use, rather than frequency of use.

Normalized by row. Darker means that a higher proporiton of users with that usage frequency prefer that priority.
Which would help you most right now?	Bug fixes	More documentation	More examples in my field	New features	Performance improvements
How often do you use Dask?

            <tr>
                    <th id="T_821479f4_b488_11e9_ad41_186590cd1c87level0_row0" class="row_heading level0 row0" >Array</th>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row0_col0" class="data row0 col0" >10</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row0_col1" class="data row0 col1" >24</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row0_col2" class="data row0 col2" >62</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row0_col3" class="data row0 col3" >15</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row0_col4" class="data row0 col4" >25</td>
        </tr>
        <tr>
                    <th id="T_821479f4_b488_11e9_ad41_186590cd1c87level0_row1" class="row_heading level0 row1" >Bag</th>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row1_col0" class="data row1 col0" >3</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row1_col1" class="data row1 col1" >11</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row1_col2" class="data row1 col2" >16</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row1_col3" class="data row1 col3" >10</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row1_col4" class="data row1 col4" >7</td>
        </tr>
        <tr>
                    <th id="T_821479f4_b488_11e9_ad41_186590cd1c87level0_row2" class="row_heading level0 row2" >DataFrame</th>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row2_col0" class="data row2 col0" >16</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row2_col1" class="data row2 col1" >32</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row2_col2" class="data row2 col2" >71</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row2_col3" class="data row2 col3" >39</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row2_col4" class="data row2 col4" >26</td>
        </tr>
        <tr>
                    <th id="T_821479f4_b488_11e9_ad41_186590cd1c87level0_row3" class="row_heading level0 row3" >Delayed</th>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row3_col0" class="data row3 col0" >16</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row3_col1" class="data row3 col1" >22</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row3_col2" class="data row3 col2" >55</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row3_col3" class="data row3 col3" >26</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row3_col4" class="data row3 col4" >27</td>
        </tr>
        <tr>
                    <th id="T_821479f4_b488_11e9_ad41_186590cd1c87level0_row4" class="row_heading level0 row4" >Futures</th>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row4_col0" class="data row4 col0" >12</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row4_col1" class="data row4 col1" >9</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row4_col2" class="data row4 col2" >25</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row4_col3" class="data row4 col3" >20</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row4_col4" class="data row4 col4" >17</td>
        </tr>
        <tr>
                    <th id="T_821479f4_b488_11e9_ad41_186590cd1c87level0_row5" class="row_heading level0 row5" >ML</th>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row5_col0" class="data row5 col0" >5</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row5_col1" class="data row5 col1" >11</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row5_col2" class="data row5 col2" >23</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row5_col3" class="data row5 col3" >11</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row5_col4" class="data row5 col4" >7</td>
        </tr>
        <tr>
                    <th id="T_821479f4_b488_11e9_ad41_186590cd1c87level0_row6" class="row_heading level0 row6" >Xarray</th>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row6_col0" class="data row6 col0" >8</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row6_col1" class="data row6 col1" >11</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row6_col2" class="data row6 col2" >34</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row6_col3" class="data row6 col3" >7</td>
                    <td id="T_821479f4_b488_11e9_ad41_186590cd1c87row6_col4" class="data row6 col4" >9</td>
        </tr>
</tbody></table>

Nothing really stands out. The “futures” users (who we expect to be relatively advanced) may prioritize features and performance over documentation. But everyone agrees that more examples are the highest priority.

Common Feature Requests

For specific features, we made a list of things that we (as developers) thought might be important.

The clearest standout is how many people thought “Better NumPy/Pandas support” was “most critical”. In hindsight, it’d be good to have a followup fill-in field to undertand what each respondent meant by that. The parsimonious interpretion is “cover more of the NumPy / pandas API”.

“Ease of deployment” had a high proportion of “critical to me”. Again in hindsight, I notice a bit of ambiguity. Does this mean people want Dask to be easier to deploy? Or does this mean that Dask, which they currently find easy to deploy, is critically important? Regardless, we can prioritize simplicity in deployment.

Relatively few respondents care about things like “Managing many users”, though we expect that this would be relatively popular among system administartors, who are a smaller population.

And of course, we have people pushing Dask to its limits for whom “Improving scaling” is critically important.

What other systems do you use?

A relatively high proportion of respondents use Python 3 (97% compared to 84% in the most recent Python Developers Survey).

3    97.29%
2     2.71%
Name: Python 2 or 3?, dtype: object

We were a bit surprised to see that SSH is the most popular “cluster resource manager”.

SSH                                                       98
Kubernetes                                                73
HPC resource manager (SLURM, PBS, SGE, LSF or similar)    61
My workplace has a custom solution for this               23
I don't know, someone else does this for me               16
Hadoop / Yarn / EMR                                       14
Name: If you use a cluster, how do you launch Dask? , dtype: int64

How does cluster-resource manager compare with API usage?

Normalized by row. Darker means that a higher proporiton of users of that API prefer that priority.
Which would help you most right now?	Bug fixes	More documentation	More examples in my field	New features	Performance improvements
Dask APIs

            <tr>
                    <th id="T_8326d0f8_b488_11e9_ad41_186590cd1c87level0_row0" class="row_heading level0 row0" >Custom</th>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row0_col0" class="data row0 col0" >15</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row0_col1" class="data row0 col1" >6</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row0_col2" class="data row0 col2" >18</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row0_col3" class="data row0 col3" >17</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row0_col4" class="data row0 col4" >14</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row0_col5" class="data row0 col5" >6</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row0_col6" class="data row0 col6" >7</td>
        </tr>
        <tr>
                    <th id="T_8326d0f8_b488_11e9_ad41_186590cd1c87level0_row1" class="row_heading level0 row1" >HPC</th>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row1_col0" class="data row1 col0" >50</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row1_col1" class="data row1 col1" >13</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row1_col2" class="data row1 col2" >40</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row1_col3" class="data row1 col3" >40</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row1_col4" class="data row1 col4" >22</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row1_col5" class="data row1 col5" >11</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row1_col6" class="data row1 col6" >30</td>
        </tr>
        <tr>
                    <th id="T_8326d0f8_b488_11e9_ad41_186590cd1c87level0_row2" class="row_heading level0 row2" >Hadoop / Yarn / EMR</th>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row2_col0" class="data row2 col0" >7</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row2_col1" class="data row2 col1" >6</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row2_col2" class="data row2 col2" >12</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row2_col3" class="data row2 col3" >8</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row2_col4" class="data row2 col4" >4</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row2_col5" class="data row2 col5" >7</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row2_col6" class="data row2 col6" >3</td>
        </tr>
        <tr>
                    <th id="T_8326d0f8_b488_11e9_ad41_186590cd1c87level0_row3" class="row_heading level0 row3" >Kubernetes</th>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row3_col0" class="data row3 col0" >40</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row3_col1" class="data row3 col1" >18</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row3_col2" class="data row3 col2" >56</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row3_col3" class="data row3 col3" >47</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row3_col4" class="data row3 col4" >37</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row3_col5" class="data row3 col5" >26</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row3_col6" class="data row3 col6" >21</td>
        </tr>
        <tr>
                    <th id="T_8326d0f8_b488_11e9_ad41_186590cd1c87level0_row4" class="row_heading level0 row4" >SSH</th>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row4_col0" class="data row4 col0" >61</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row4_col1" class="data row4 col1" >23</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row4_col2" class="data row4 col2" >72</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row4_col3" class="data row4 col3" >58</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row4_col4" class="data row4 col4" >32</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row4_col5" class="data row4 col5" >30</td>
                    <td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row4_col6" class="data row4 col6" >25</td>
        </tr>
</tbody></table>

HPC users are relatively heavy users of dask.array and xarray.

Somewhat surprisingly, Dask’s heaviest users find dask stable enough. Perhaps they’ve pushed past the bugs and found workarounds (percentages are normalized by row).

Takeaways

We should prioritize improving and expanding our documentation and examples. This may be accomplished by Dask maintainers seeking examples from the community. Many of the examples on https://examples.dask.org were developed by domain specialist who use Dask.
Improved scaling to larger problems is important, but we shouldn’t sacrifice the single-machine usecase to get there.
Both interactive and batch workflows are important.
Dask’s various sub-communities are more similar than they are different.

Thanks again to all the respondents. We look forward to repeating this process to identify trends over time.

Dask APIs	Array	Bag	DataFrame	Delayed	Futures	ML	Xarray
If you use a cluster, how do you launch Dask?