https://blog.dask.orgDask Working Notes - Posts by Tom Augspurger2026-03-05T15:05:21.039368+00:00ABloghttps://blog.dask.org/2020/09/22/user_survey/2020 Dask User Survey2020-09-22T00:00:00+00:00Tom Augspurger<style type="text/css">
table td {
background: none;
}
table tr.even td {
background: none;
}
table {
text-shadow: none;
}
table tr:hover td {
background: none;
}
</style>
<p>This post presents the results of the 2020 Dask User Survey,
which ran earlier this summer. Thanks to everyone who took the time to fill out the survey!
These results help us better understand the Dask community and will guide future development efforts.</p>
<p>The raw data, as well as the start of an analysis, can be found in this binder:</p>
<p><a class="reference external" href="https://mybinder.org/v2/gh/dask/dask-examples/main?urlpath=%2Ftree%2Fsurveys%2F2020.ipynb"><img alt="Binder" src="https://mybinder.org/badge_logo.svg" /></a></p>
<p>Let us know if you find anything in the data.</p>
<aside class="system-message">
<p class="system-message-title">System Message: WARNING/2 (<span class="docutils literal">/opt/build/repo/2020/09/22/user_survey.md</span>, line 38)</p>
<p>Document headings start at H2, not H1 [myst.header]</p>
</aside>
<section id="highlights">
<ul class="simple">
<li><p>We had 240 responses to the survey (slightly fewer than last year, which had about 260).</p></li>
<li><p>Overall, results look mostly similar to last year’s.</p></li>
<li><p>Our documentation has probably improved relative to last year</p></li>
<li><p>Respondents care more about performance relative to last year.</p></li>
</ul>
<aside class="system-message">
<p class="system-message-title">System Message: WARNING/2 (<span class="docutils literal">/opt/build/repo/2020/09/22/user_survey.md</span>, line 45)</p>
<p>Document headings start at H2, not H1 [myst.header]</p>
</aside>
</section>
<section id="new-questions">
<h1>New Questions</h1>
<p>Most of the questions are the same as in 2019. We added a couple questions about deployment and dashboard usage. Let’s look at those first.</p>
<p>Among respondents who use a Dask package to deploy a cluster (about 53% of respondents), there’s a wide spread of methods.</p>
<img src="/images/2020_survey/2020_3_0.png">
<p>Most people access the dashboard through a web browser. Those not using the dashboard are likely (hopefully) just using Dask on a single machine with the threaded scheduler (though the dashboard works fine on a single machine as well).</p>
<img src="/images/2020_survey/2020_5_0.png">
<aside class="system-message">
<p class="system-message-title">System Message: WARNING/2 (<span class="docutils literal">/opt/build/repo/2020/09/22/user_survey.md</span>, line 57)</p>
<p>Document headings start at H2, not H1 [myst.header]</p>
</aside>
</section>
<section id="learning-resources">
<h1>Learning Resources</h1>
<p>Respondents’ learning material usage is farily similar to last year. The most notable differences are from
our survey form providing more options (our <a class="reference external" href="https://www.youtube.com/channel/UCj9eavqmvwaCyKhIlu2GaoA">YouTube channel</a> and “Gitter chat”). Other than that, <a class="reference external" href="https://examples.dask.org">examples.dask.org</a> might be relatively more popular.</p>
<img src="/images/2020_survey/2020_7_0.png">
<p>Just like last year, we’ll look at resource usage grouped by how often they use Dask.</p>
<img src="/images/2020_survey/2020_10_0.png">
<p>A few observations</p>
<ul class="simple">
<li><p>GitHub issues are becoming relatively less popular, which perhaps reflects better documentation or stability (assuming people go to the issue tracker when they can’t find the answer in the docs or they hit a bug).</p></li>
<li><p><a class="reference external" href="https://examples.dask.org">https://examples.dask.org</a> is notably now more popular among occasinal users.</p></li>
<li><p>In response to last year’s survey, we invested time in making <a class="reference external" href="https://tutorial.dask.org">https://tutorial.dask.org</a> better, which we previously felt was lacking. Its usage is still about the same as last year’s (pretty popular), so it’s unclear whether we should dedicate additional focus there.</p></li>
</ul>
<aside class="system-message">
<p class="system-message-title">System Message: WARNING/2 (<span class="docutils literal">/opt/build/repo/2020/09/22/user_survey.md</span>, line 74)</p>
<p>Document headings start at H2, not H1 [myst.header]</p>
</aside>
</section>
<section id="how-do-you-use-dask">
<h1>How do you use Dask?</h1>
<p>API usage remains about the same as last year (recall that about 20 fewer people took the survey and people can select multiple, so relative differences are most interesting). We added new choices for RAPIDS, Prefect, and XGBoost, each of which are somewhat popular (in the neighborhood of <code class="docutils literal notranslate"><span class="pre">dask.Bag</span></code>).</p>
<img src="/images/2020_survey/2020_12_0.png">
<p>About 65% of our users are using Dask on a cluster at least some of the time, which is similar to last year.</p>
<aside class="system-message">
<p class="system-message-title">System Message: WARNING/2 (<span class="docutils literal">/opt/build/repo/2020/09/22/user_survey.md</span>, line 82)</p>
<p>Document headings start at H2, not H1 [myst.header]</p>
</aside>
</section>
<section id="how-can-dask-improve">
<h1>How can Dask improve?</h1>
<p>Respondents continue to say that more documentation and examples would be the most valuable improvements to the project.</p>
<p>One interesting change comes from looking at “Which would help you most right now?” split by API group (<code class="docutils literal notranslate"><span class="pre">dask.dataframe</span></code>, <code class="docutils literal notranslate"><span class="pre">dask.array</span></code>, etc.). Last year showed that “More examples” in my field was the most important for all API groups (first table below). But in 2020 there are some differences (second table below).</p>
<style type="text/css" >
#T_0a8701b8_e96b_11ea_9e95_186590cd1c87row0_col0 {
background-color: #fff7fb;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row0_col1 {
background-color: #cacee5;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row0_col2 {
background-color: #023858;
color: #f1f1f1;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row0_col3 {
background-color: #f1ebf4;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row0_col4 {
background-color: #c4cbe3;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row1_col0 {
background-color: #fff7fb;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row1_col1 {
background-color: #3b92c1;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row1_col2 {
background-color: #023858;
color: #f1f1f1;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row1_col3 {
background-color: #62a2cb;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row1_col4 {
background-color: #bdc8e1;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row2_col0 {
background-color: #fff7fb;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row2_col1 {
background-color: #c2cbe2;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row2_col2 {
background-color: #023858;
color: #f1f1f1;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row2_col3 {
background-color: #94b6d7;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row2_col4 {
background-color: #e0dded;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row3_col0 {
background-color: #fff7fb;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row3_col1 {
background-color: #e6e2ef;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row3_col2 {
background-color: #023858;
color: #f1f1f1;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row3_col3 {
background-color: #ced0e6;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row3_col4 {
background-color: #c5cce3;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row4_col0 {
background-color: #dedcec;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row4_col1 {
background-color: #fff7fb;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row4_col2 {
background-color: #023858;
color: #f1f1f1;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row4_col3 {
background-color: #1c7fb8;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row4_col4 {
background-color: #73a9cf;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row5_col0 {
background-color: #fff7fb;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row5_col1 {
background-color: #b4c4df;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row5_col2 {
background-color: #023858;
color: #f1f1f1;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row5_col3 {
background-color: #b4c4df;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row5_col4 {
background-color: #eee9f3;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row6_col0 {
background-color: #faf2f8;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row6_col1 {
background-color: #e7e3f0;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row6_col2 {
background-color: #023858;
color: #f1f1f1;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row6_col3 {
background-color: #fff7fb;
color: #000000;
} #T_0a8701b8_e96b_11ea_9e95_186590cd1c87row6_col4 {
background-color: #f4eef6;
color: #000000;
}</style><table id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87" ><caption>2019 normalized by row. Darker means that a higher proporiton of users of that API prefer that priority.</caption><thead> <tr> <th class="index_name level0" >Which would help you most right now?</th> <th class="col_heading level0 col0" >Bug fixes</th> <th class="col_heading level0 col1" >More documentation</th> <th class="col_heading level0 col2" >More examples in my field</th> <th class="col_heading level0 col3" >New features</th> <th class="col_heading level0 col4" >Performance improvements</th> </tr> <tr> <th class="index_name level0" >Dask APIs</th> <th class="blank" ></th> <th class="blank" ></th> <th class="blank" ></th> <th class="blank" ></th> <th class="blank" ></th> </tr></thead><tbody>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span> <tr>
<th id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87level0_row0" class="row_heading level0 row0" >Array</th>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row0_col0" class="data row0 col0" >10</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row0_col1" class="data row0 col1" >24</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row0_col2" class="data row0 col2" >62</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row0_col3" class="data row0 col3" >15</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row0_col4" class="data row0 col4" >25</td>
</tr>
<tr>
<th id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87level0_row1" class="row_heading level0 row1" >Bag</th>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row1_col0" class="data row1 col0" >3</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row1_col1" class="data row1 col1" >11</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row1_col2" class="data row1 col2" >16</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row1_col3" class="data row1 col3" >10</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row1_col4" class="data row1 col4" >7</td>
</tr>
<tr>
<th id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87level0_row2" class="row_heading level0 row2" >DataFrame</th>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row2_col0" class="data row2 col0" >16</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row2_col1" class="data row2 col1" >32</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row2_col2" class="data row2 col2" >71</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row2_col3" class="data row2 col3" >39</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row2_col4" class="data row2 col4" >26</td>
</tr>
<tr>
<th id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87level0_row3" class="row_heading level0 row3" >Delayed</th>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row3_col0" class="data row3 col0" >16</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row3_col1" class="data row3 col1" >22</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row3_col2" class="data row3 col2" >55</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row3_col3" class="data row3 col3" >26</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row3_col4" class="data row3 col4" >27</td>
</tr>
<tr>
<th id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87level0_row4" class="row_heading level0 row4" >Futures</th>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row4_col0" class="data row4 col0" >12</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row4_col1" class="data row4 col1" >9</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row4_col2" class="data row4 col2" >25</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row4_col3" class="data row4 col3" >20</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row4_col4" class="data row4 col4" >17</td>
</tr>
<tr>
<th id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87level0_row5" class="row_heading level0 row5" >ML</th>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row5_col0" class="data row5 col0" >5</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row5_col1" class="data row5 col1" >11</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row5_col2" class="data row5 col2" >23</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row5_col3" class="data row5 col3" >11</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row5_col4" class="data row5 col4" >7</td>
</tr>
<tr>
<th id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87level0_row6" class="row_heading level0 row6" >Xarray</th>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row6_col0" class="data row6 col0" >8</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row6_col1" class="data row6 col1" >11</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row6_col2" class="data row6 col2" >34</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row6_col3" class="data row6 col3" >7</td>
<td id="T_0a8701b8_e96b_11ea_9e95_186590cd1c87row6_col4" class="data row6 col4" >9</td>
</tr>
</tbody></table>
</pre></div>
</div>
<style type="text/css" >
#T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row0_col0 {
background-color: #fff7fb;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row0_col1 {
background-color: #f1ebf5;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row0_col2 {
background-color: #023858;
color: #f1f1f1;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row0_col3 {
background-color: #f5eef6;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row0_col4 {
background-color: #d0d1e6;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row1_col0 {
background-color: #f0eaf4;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row1_col1 {
background-color: #fff7fb;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row1_col2 {
background-color: #023858;
color: #f1f1f1;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row1_col3 {
background-color: #f0eaf4;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row1_col4 {
background-color: #4c99c5;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row2_col0 {
background-color: #f5eff6;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row2_col1 {
background-color: #fff7fb;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row2_col2 {
background-color: #023858;
color: #f1f1f1;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row2_col3 {
background-color: #fcf4fa;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row2_col4 {
background-color: #8eb3d5;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row3_col0 {
background-color: #fff7fb;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row3_col1 {
background-color: #ebe6f2;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row3_col2 {
background-color: #023858;
color: #f1f1f1;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row3_col3 {
background-color: #f5eff6;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row3_col4 {
background-color: #3d93c2;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row4_col0 {
background-color: #fff7fb;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row4_col1 {
background-color: #f5eef6;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row4_col2 {
background-color: #0567a2;
color: #f1f1f1;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row4_col3 {
background-color: #cacee5;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row4_col4 {
background-color: #023858;
color: #f1f1f1;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row5_col0 {
background-color: #ede8f3;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row5_col1 {
background-color: #fff7fb;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row5_col2 {
background-color: #023858;
color: #f1f1f1;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row5_col3 {
background-color: #c1cae2;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row5_col4 {
background-color: #80aed2;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row6_col0 {
background-color: #fff7fb;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row6_col1 {
background-color: #f8f1f8;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row6_col2 {
background-color: #023858;
color: #f1f1f1;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row6_col3 {
background-color: #c9cee4;
color: #000000;
} #T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row6_col4 {
background-color: #86b0d3;
color: #000000;
}</style><table id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87" ><caption>2020 normalized by row. Darker means that a higher proporiton of users of that API prefer that priority.</caption><thead> <tr> <th class="index_name level0" >Which would help you most right now?</th> <th class="col_heading level0 col0" >Bug fixes</th> <th class="col_heading level0 col1" >More documentation</th> <th class="col_heading level0 col2" >More examples in my field</th> <th class="col_heading level0 col3" >New features</th> <th class="col_heading level0 col4" >Performance improvements</th> </tr> <tr> <th class="index_name level0" >Dask APIs</th> <th class="blank" ></th> <th class="blank" ></th> <th class="blank" ></th> <th class="blank" ></th> <th class="blank" ></th> </tr></thead><tbody>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span> <tr>
<th id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87level0_row0" class="row_heading level0 row0" >Array</th>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row0_col0" class="data row0 col0" >12</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row0_col1" class="data row0 col1" >16</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row0_col2" class="data row0 col2" >56</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row0_col3" class="data row0 col3" >15</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row0_col4" class="data row0 col4" >23</td>
</tr>
<tr>
<th id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87level0_row1" class="row_heading level0 row1" >Bag</th>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row1_col0" class="data row1 col0" >7</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row1_col1" class="data row1 col1" >5</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row1_col2" class="data row1 col2" >24</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row1_col3" class="data row1 col3" >7</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row1_col4" class="data row1 col4" >16</td>
</tr>
<tr>
<th id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87level0_row2" class="row_heading level0 row2" >DataFrame</th>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row2_col0" class="data row2 col0" >24</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row2_col1" class="data row2 col1" >21</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row2_col2" class="data row2 col2" >67</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row2_col3" class="data row2 col3" >22</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row2_col4" class="data row2 col4" >41</td>
</tr>
<tr>
<th id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87level0_row3" class="row_heading level0 row3" >Delayed</th>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row3_col0" class="data row3 col0" >15</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row3_col1" class="data row3 col1" >19</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row3_col2" class="data row3 col2" >46</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row3_col3" class="data row3 col3" >17</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row3_col4" class="data row3 col4" >34</td>
</tr>
<tr>
<th id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87level0_row4" class="row_heading level0 row4" >Futures</th>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row4_col0" class="data row4 col0" >9</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row4_col1" class="data row4 col1" >10</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row4_col2" class="data row4 col2" >21</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row4_col3" class="data row4 col3" >13</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row4_col4" class="data row4 col4" >24</td>
</tr>
<tr>
<th id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87level0_row5" class="row_heading level0 row5" >ML</th>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row5_col0" class="data row5 col0" >6</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row5_col1" class="data row5 col1" >4</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row5_col2" class="data row5 col2" >21</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row5_col3" class="data row5 col3" >9</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row5_col4" class="data row5 col4" >12</td>
</tr>
<tr>
<th id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87level0_row6" class="row_heading level0 row6" >Xarray</th>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row6_col0" class="data row6 col0" >3</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row6_col1" class="data row6 col1" >4</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row6_col2" class="data row6 col2" >25</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row6_col3" class="data row6 col3" >9</td>
<td id="T_0a8d3eac_e96b_11ea_9e95_186590cd1c87row6_col4" class="data row6 col4" >13</td>
</tr>
</tbody></table>
</pre></div>
</div>
<p>Examples are again the most important (for all API groups except <code class="docutils literal notranslate"><span class="pre">Futures</span></code>). But “Performance improvements” is now the second-most important improvement (except for <code class="docutils literal notranslate"><span class="pre">Futures</span></code> where it’s most important). How should we interpret this? A charitable interpretation is that Dask’s users are scaling to larger problems and are running into new scaling challenges. A less charitable interpretation is that our user’s workflows are the same but Dask is getting slower!</p>
<aside class="system-message">
<p class="system-message-title">System Message: WARNING/2 (<span class="docutils literal">/opt/build/repo/2020/09/22/user_survey.md</span>, line 422)</p>
<p>Document headings start at H2, not H1 [myst.header]</p>
</aside>
</section>
<section id="what-other-systems-do-you-use">
<h1>What other systems do you use?</h1>
<p>SSH continues to be the most popular “cluster resource mananger”. This was the big surprise last year, so we put in some work to make it nicer. Aside from that, not much has changed.</p>
<img src="/images/2020_survey/2020_25_0.png">
<p>And Dask users are about as happy with its stability as last year.</p>
<img src="/images/2020_survey/2020_27_0.png">
<aside class="system-message">
<p class="system-message-title">System Message: WARNING/2 (<span class="docutils literal">/opt/build/repo/2020/09/22/user_survey.md</span>, line 432)</p>
<p>Document headings start at H2, not H1 [myst.header]</p>
</aside>
</section>
<section id="takeaways">
<h1>Takeaways</h1>
<ol class="arabic simple">
<li><p>Overall, most things are similar to last year.</p></li>
<li><p>Documentation, especially domain-specific examples, continues to be important. That said, our documentation is probably better than it was last year.</p></li>
<li><p>More users are pushing Dask further. Investing in performance is likely to be valuable.</p></li>
</ol>
<p>Thanks again to all the respondents!</p>
</section>
This post presents the results of the 2020 Dask User Survey,
which ran earlier this summer. Thanks to everyone who took the time to fill out the survey!
These results help us better understand the Dask community and will guide future development efforts.2020-09-22T00:00:00+00:00https://blog.dask.org/2020/08/31/helm_daskhub/Announcing the DaskHub Helm Chart2020-08-31T00:00:00+00:00Tom Augspurger<p>Today we’re announcing the release of the
<a class="reference external" href="https://github.com/dask/helm-chart/blob/master/daskhub/README.md"><code class="docutils literal notranslate"><span class="pre">daskhub</span></code></a>
helm chart. This is a <a class="reference external" href="https://helm.sh/">Helm</a> chart to easily install
<a class="reference external" href="https://jupyter.org/hub">JupyterHub</a> and Dask for multiple users on a
Kubernetes Cluster. If you’re managing deployment for many people that needs
interactive, scalable computing (say for a class of students, a data science
team, or a research lab) then <code class="docutils literal notranslate"><span class="pre">dask/daskhub</span></code> might be right for you.</p>
<p>You can install <code class="docutils literal notranslate"><span class="pre">dask/daskhub</span></code> on a Kubernetes cluster today with</p>
<div class="highlight-console notranslate"><div class="highlight"><pre><span></span><span class="go">helm repo add dask https://helm.dask.org/</span>
<span class="go">helm repo update</span>
<span class="go">helm upgrade --install dhub dask/daskhub</span>
</pre></div>
</div>
<aside class="system-message">
<p class="system-message-title">System Message: WARNING/2 (<span class="docutils literal">/opt/build/repo/2020/08/31/helm_daskhub.md</span>, line 26)</p>
<p>Document headings start at H2, not H1 [myst.header]</p>
</aside>
<section id="history">
<p>The <code class="docutils literal notranslate"><span class="pre">dask/daskhub</span></code> helm chart is an evolution of the <a class="reference external" href="http://pangeo.io/">Pangeo</a>
helm chart, which came out of that community’s attempts to do big data
geoscience on the cloud. We’re very grateful to have years of experience using
Dask and JupyterHub together. Pangeo was always aware that there wasn’t anything
geoscience-specific to their Helm chart and so were eager to contribute it to
Dask to share the maintenance burden. In the process of moving it over to Dask’s
chart repository we took the opportunity to clean up a few rough edges.</p>
<p>It’s interesting to read the <a class="reference external" href="https://blog.dask.org/2018/01/22/pangeo-2">original
announcement</a> of Pangeo’s JupyterHub
deployment. A lot has improved, and we hope that this helm chart assists more
groups in deploying JupyterHubs capable of scalable computations with Dask.</p>
<aside class="system-message">
<p class="system-message-title">System Message: WARNING/2 (<span class="docutils literal">/opt/build/repo/2020/08/31/helm_daskhub.md</span>, line 41)</p>
<p>Document headings start at H2, not H1 [myst.header]</p>
</aside>
</section>
<section id="details">
<h1>Details</h1>
<p>Internally, the DaskHub helm chart is relatively simple combination of the
<a class="reference external" href="https://github.com/jupyterhub/zero-to-jupyterhub-k8s">JupyterHub</a> and <a class="reference external" href="https://github.com/dask/dask-gateway/">Dask
Gateway</a> helm charts. The only additional
magic is some configuration to</p>
<ol class="arabic simple">
<li><p>Register Dask Gateway as a <a class="reference external" href="https://jupyterhub.readthedocs.io/en/stable/reference/services.html">JupyterHub
service</a>.</p></li>
<li><p>Set environment variables to make using Dask Gateway easy for your users.</p></li>
</ol>
<p>With the default configuration, your users will be able to create and connect to
Dask Clusters, including their dashboards, with a simple</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">dask_gateway</span><span class="w"> </span><span class="kn">import</span> <span class="n">GatewayCluster</span>
<span class="gp">>>> </span><span class="n">cluster</span> <span class="o">=</span> <span class="n">GatewayCluster</span><span class="p">()</span>
<span class="gp">>>> </span><span class="n">client</span> <span class="o">=</span> <span class="n">cluster</span><span class="o">.</span><span class="n">get_client</span><span class="p">()</span>
</pre></div>
</div>
<p>Check out the
<a class="reference external" href="https://docs.dask.org/en/latest/setup/kubernetes-helm.html">documentation</a> for
details and let us know if you run into any difficulties.</p>
</section>
Today we’re announcing the release of the
daskhub
helm chart. This is a Helm chart to easily install
JupyterHub and Dask for multiple users on a
Kubernetes Cluster. If you’re managing deployment for many people that needs
interactive, scalable computing (say for a class of students, a data science
team, or a research lab) then dask/daskhub might be right for you.2020-08-31T00:00:00+00:00https://blog.dask.org/2019/08/05/user-survey/2019 Dask User Survey2019-08-05T00:00:00+00:00Tom Augspurger<style type="text/css">
table td {
background: none;
}
table tr.even td {
background: none;
}
table {
text-shadow: none;
}
</style>
<aside class="system-message">
<p class="system-message-title">System Message: WARNING/2 (<span class="docutils literal">/opt/build/repo/2019/08/05/user-survey.md</span>, line 25)</p>
<p>Document headings start at H2, not H1 [myst.header]</p>
</aside>
<section id="dask-user-survey-results">
<p>This notebook presents the results of the 2019 Dask User Survey,
which ran earlier this summer. Thanks to everyone who took the time to fill out the survey!
These results help us better understand the Dask community and will guide future development efforts.</p>
<p>The raw data, as well as the start of an analysis, can be found in this binder:</p>
<p><a class="reference external" href="https://mybinder.org/v2/gh/dask/dask-examples/main?urlpath=%2Ftree%2Fsurveys%2F2019.ipynb"><img alt="Binder" src="https://mybinder.org/badge_logo.svg" /></a></p>
<p>Let us know if you find anything in the data.</p>
<aside class="system-message">
<p class="system-message-title">System Message: WARNING/2 (<span class="docutils literal">/opt/build/repo/2019/08/05/user-survey.md</span>, line 37)</p>
<p>Document headings start at H2, not H1 [myst.header]</p>
</aside>
</section>
<section id="highlights">
<h1>Highlights</h1>
<p>We had 259 responses to the survey. Overall, we found that the survey respondents really care about improved documentation, and ease of use (including ease of deployment), and scaling. While Dask brings together many different communities (big arrays versus big dataframes, traditional HPC users versus cloud-native resource managers), there was general agreement in what is most important for Dask.</p>
<p>Now we’ll go through some individual items questions, highlighting particularly interesting results.</p>
<aside class="system-message">
<p class="system-message-title">System Message: WARNING/2 (<span class="docutils literal">/opt/build/repo/2019/08/05/user-survey.md</span>, line 43)</p>
<p>Document headings start at H2, not H1 [myst.header]</p>
</aside>
</section>
<section id="how-do-you-use-dask">
<h1>How do you use Dask?</h1>
<p>For learning resources, almost every respondent uses the documentation.</p>
<p><img alt="svg" src="https://blog.dask.org/_images/analyze_4_0.svg" /></p>
<p>Most respondents use Dask at least occasionally. Fortunately we had a decent number of respondents who are just looking into Dask, yet still spent the time to take the survey.</p>
<p><img alt="svg" src="https://blog.dask.org/_images/analyze_6_0.svg" /></p>
<p>I’m curiuos about how learning resource usage changes as users become more experienced. We might expect those just looking into Dask to start with <code class="docutils literal notranslate"><span class="pre">examples.dask.org</span></code>, where they can try out Dask without installing anything.</p>
<p><img alt="svg" src="https://blog.dask.org/_images/analyze_8_0.svg" /></p>
<p>Overall, documentation is still the leader across user user groups.</p>
<p>The usage of the <a class="reference external" href="https://github.com/dask/dask-tutorial">Dask tutorial</a> and the <a class="reference internal" href="#examples.dask.org"><span class="xref myst">dask examples</span></a> are relatively consistent across groups. The primary difference between regular and new users is that regular users are more likely to engage on GitHub.</p>
<p>From StackOverflow questions and GitHub issues, we have a vague idea about which parts of the library are used.
The survey shows that (for our respondents at least) DataFrame and Delayed are the most commonly used APIs.</p>
<p><img alt="svg" src="https://blog.dask.org/_images/analyze_10_0.svg" /></p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>About 65.49% of our respondests are using Dask on a Cluster.
</pre></div>
</div>
<p>But the majority of respondents <em>also</em> use Dask on their laptop.
This highlights the importance of Dask scaling down, either for
prototyping with a <code class="docutils literal notranslate"><span class="pre">LocalCluster</span></code>, or for out-of-core analysis
using <code class="docutils literal notranslate"><span class="pre">LocalCluster</span></code> or one of the single-machine schedulers.</p>
<p><img alt="svg" src="https://blog.dask.org/_images/analyze_13_0.svg" /></p>
<p>Most respondents use Dask interactively, at least some of the time.</p>
<p><img alt="svg" src="https://blog.dask.org/_images/analyze_15_0.svg" /></p>
<p>Most repondents thought that more documentation and examples would be the most valuable improvements to the project. This is especially pronounced among new users. But even among those using Dask everyday more people thought that “More examples” is more valuable than “New features” or “Performance improvements”.</p>
<style type="text/css" >
#T_820ef326_b488_11e9_ad41_186590cd1c87row0_col0 {
background-color: #3b92c1;
color: #000000;
} #T_820ef326_b488_11e9_ad41_186590cd1c87row0_col1 {
background-color: #b4c4df;
color: #000000;
} #T_820ef326_b488_11e9_ad41_186590cd1c87row0_col2 {
background-color: #dad9ea;
color: #000000;
} #T_820ef326_b488_11e9_ad41_186590cd1c87row0_col3 {
background-color: #023858;
color: #f1f1f1;
} #T_820ef326_b488_11e9_ad41_186590cd1c87row0_col4 {
background-color: #023858;
color: #f1f1f1;
} #T_820ef326_b488_11e9_ad41_186590cd1c87row1_col0 {
background-color: #fff7fb;
color: #000000;
} #T_820ef326_b488_11e9_ad41_186590cd1c87row1_col1 {
background-color: #fff7fb;
color: #000000;
} #T_820ef326_b488_11e9_ad41_186590cd1c87row1_col2 {
background-color: #fff7fb;
color: #000000;
} #T_820ef326_b488_11e9_ad41_186590cd1c87row1_col3 {
background-color: #fff7fb;
color: #000000;
} #T_820ef326_b488_11e9_ad41_186590cd1c87row1_col4 {
background-color: #fff7fb;
color: #000000;
} #T_820ef326_b488_11e9_ad41_186590cd1c87row2_col0 {
background-color: #023858;
color: #f1f1f1;
} #T_820ef326_b488_11e9_ad41_186590cd1c87row2_col1 {
background-color: #023858;
color: #f1f1f1;
} #T_820ef326_b488_11e9_ad41_186590cd1c87row2_col2 {
background-color: #023858;
color: #f1f1f1;
} #T_820ef326_b488_11e9_ad41_186590cd1c87row2_col3 {
background-color: #1b7eb7;
color: #000000;
} #T_820ef326_b488_11e9_ad41_186590cd1c87row2_col4 {
background-color: #589ec8;
color: #000000;
}</style><table id="T_820ef326_b488_11e9_ad41_186590cd1c87" ><caption>Normalized by row. Darker means that a higher proporiton of users with that usage frequency prefer that priority.</caption><thead> <tr> <th class="index_name level0" >Which would help you most right now?</th> <th class="col_heading level0 col0" >Bug fixes</th> <th class="col_heading level0 col1" >More documentation</th> <th class="col_heading level0 col2" >More examples in my field</th> <th class="col_heading level0 col3" >New features</th> <th class="col_heading level0 col4" >Performance improvements</th> </tr> <tr> <th class="index_name level0" >How often do you use Dask?</th> <th class="blank" ></th> <th class="blank" ></th> <th class="blank" ></th> <th class="blank" ></th> <th class="blank" ></th> </tr></thead><tbody>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span> <tr>
<th id="T_820ef326_b488_11e9_ad41_186590cd1c87level0_row0" class="row_heading level0 row0" >Every day</th>
<td id="T_820ef326_b488_11e9_ad41_186590cd1c87row0_col0" class="data row0 col0" >9</td>
<td id="T_820ef326_b488_11e9_ad41_186590cd1c87row0_col1" class="data row0 col1" >11</td>
<td id="T_820ef326_b488_11e9_ad41_186590cd1c87row0_col2" class="data row0 col2" >25</td>
<td id="T_820ef326_b488_11e9_ad41_186590cd1c87row0_col3" class="data row0 col3" >22</td>
<td id="T_820ef326_b488_11e9_ad41_186590cd1c87row0_col4" class="data row0 col4" >23</td>
</tr>
<tr>
<th id="T_820ef326_b488_11e9_ad41_186590cd1c87level0_row1" class="row_heading level0 row1" >Just looking for now</th>
<td id="T_820ef326_b488_11e9_ad41_186590cd1c87row1_col0" class="data row1 col0" >1</td>
<td id="T_820ef326_b488_11e9_ad41_186590cd1c87row1_col1" class="data row1 col1" >3</td>
<td id="T_820ef326_b488_11e9_ad41_186590cd1c87row1_col2" class="data row1 col2" >18</td>
<td id="T_820ef326_b488_11e9_ad41_186590cd1c87row1_col3" class="data row1 col3" >9</td>
<td id="T_820ef326_b488_11e9_ad41_186590cd1c87row1_col4" class="data row1 col4" >5</td>
</tr>
<tr>
<th id="T_820ef326_b488_11e9_ad41_186590cd1c87level0_row2" class="row_heading level0 row2" >Occasionally</th>
<td id="T_820ef326_b488_11e9_ad41_186590cd1c87row2_col0" class="data row2 col0" >14</td>
<td id="T_820ef326_b488_11e9_ad41_186590cd1c87row2_col1" class="data row2 col1" >27</td>
<td id="T_820ef326_b488_11e9_ad41_186590cd1c87row2_col2" class="data row2 col2" >52</td>
<td id="T_820ef326_b488_11e9_ad41_186590cd1c87row2_col3" class="data row2 col3" >18</td>
<td id="T_820ef326_b488_11e9_ad41_186590cd1c87row2_col4" class="data row2 col4" >15</td>
</tr>
</tbody></table>
</pre></div>
</div>
<p>Perhaps users of certain dask APIs feel differenlty from the group as a whole? We perform a similar analysis grouped by API use, rather than frequency of use.</p>
<style type="text/css" >
#T_821479f4_b488_11e9_ad41_186590cd1c87row0_col0 {
background-color: #fff7fb;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row0_col1 {
background-color: #cacee5;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row0_col2 {
background-color: #023858;
color: #f1f1f1;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row0_col3 {
background-color: #f1ebf4;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row0_col4 {
background-color: #c4cbe3;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row1_col0 {
background-color: #fff7fb;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row1_col1 {
background-color: #3b92c1;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row1_col2 {
background-color: #023858;
color: #f1f1f1;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row1_col3 {
background-color: #62a2cb;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row1_col4 {
background-color: #bdc8e1;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row2_col0 {
background-color: #fff7fb;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row2_col1 {
background-color: #c2cbe2;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row2_col2 {
background-color: #023858;
color: #f1f1f1;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row2_col3 {
background-color: #94b6d7;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row2_col4 {
background-color: #e0dded;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row3_col0 {
background-color: #fff7fb;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row3_col1 {
background-color: #e6e2ef;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row3_col2 {
background-color: #023858;
color: #f1f1f1;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row3_col3 {
background-color: #ced0e6;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row3_col4 {
background-color: #c5cce3;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row4_col0 {
background-color: #dedcec;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row4_col1 {
background-color: #fff7fb;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row4_col2 {
background-color: #023858;
color: #f1f1f1;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row4_col3 {
background-color: #1c7fb8;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row4_col4 {
background-color: #73a9cf;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row5_col0 {
background-color: #fff7fb;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row5_col1 {
background-color: #b4c4df;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row5_col2 {
background-color: #023858;
color: #f1f1f1;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row5_col3 {
background-color: #b4c4df;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row5_col4 {
background-color: #eee9f3;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row6_col0 {
background-color: #faf2f8;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row6_col1 {
background-color: #e7e3f0;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row6_col2 {
background-color: #023858;
color: #f1f1f1;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row6_col3 {
background-color: #fff7fb;
color: #000000;
} #T_821479f4_b488_11e9_ad41_186590cd1c87row6_col4 {
background-color: #f4eef6;
color: #000000;
}</style><table id="T_821479f4_b488_11e9_ad41_186590cd1c87" ><caption>Normalized by row. Darker means that a higher proporiton of users of that API prefer that priority.</caption><thead> <tr> <th class="index_name level0" >Which would help you most right now?</th> <th class="col_heading level0 col0" >Bug fixes</th> <th class="col_heading level0 col1" >More documentation</th> <th class="col_heading level0 col2" >More examples in my field</th> <th class="col_heading level0 col3" >New features</th> <th class="col_heading level0 col4" >Performance improvements</th> </tr> <tr> <th class="index_name level0" >Dask APIs</th> <th class="blank" ></th> <th class="blank" ></th> <th class="blank" ></th> <th class="blank" ></th> <th class="blank" ></th> </tr></thead><tbody>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span> <tr>
<th id="T_821479f4_b488_11e9_ad41_186590cd1c87level0_row0" class="row_heading level0 row0" >Array</th>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row0_col0" class="data row0 col0" >10</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row0_col1" class="data row0 col1" >24</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row0_col2" class="data row0 col2" >62</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row0_col3" class="data row0 col3" >15</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row0_col4" class="data row0 col4" >25</td>
</tr>
<tr>
<th id="T_821479f4_b488_11e9_ad41_186590cd1c87level0_row1" class="row_heading level0 row1" >Bag</th>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row1_col0" class="data row1 col0" >3</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row1_col1" class="data row1 col1" >11</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row1_col2" class="data row1 col2" >16</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row1_col3" class="data row1 col3" >10</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row1_col4" class="data row1 col4" >7</td>
</tr>
<tr>
<th id="T_821479f4_b488_11e9_ad41_186590cd1c87level0_row2" class="row_heading level0 row2" >DataFrame</th>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row2_col0" class="data row2 col0" >16</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row2_col1" class="data row2 col1" >32</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row2_col2" class="data row2 col2" >71</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row2_col3" class="data row2 col3" >39</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row2_col4" class="data row2 col4" >26</td>
</tr>
<tr>
<th id="T_821479f4_b488_11e9_ad41_186590cd1c87level0_row3" class="row_heading level0 row3" >Delayed</th>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row3_col0" class="data row3 col0" >16</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row3_col1" class="data row3 col1" >22</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row3_col2" class="data row3 col2" >55</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row3_col3" class="data row3 col3" >26</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row3_col4" class="data row3 col4" >27</td>
</tr>
<tr>
<th id="T_821479f4_b488_11e9_ad41_186590cd1c87level0_row4" class="row_heading level0 row4" >Futures</th>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row4_col0" class="data row4 col0" >12</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row4_col1" class="data row4 col1" >9</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row4_col2" class="data row4 col2" >25</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row4_col3" class="data row4 col3" >20</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row4_col4" class="data row4 col4" >17</td>
</tr>
<tr>
<th id="T_821479f4_b488_11e9_ad41_186590cd1c87level0_row5" class="row_heading level0 row5" >ML</th>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row5_col0" class="data row5 col0" >5</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row5_col1" class="data row5 col1" >11</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row5_col2" class="data row5 col2" >23</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row5_col3" class="data row5 col3" >11</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row5_col4" class="data row5 col4" >7</td>
</tr>
<tr>
<th id="T_821479f4_b488_11e9_ad41_186590cd1c87level0_row6" class="row_heading level0 row6" >Xarray</th>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row6_col0" class="data row6 col0" >8</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row6_col1" class="data row6 col1" >11</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row6_col2" class="data row6 col2" >34</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row6_col3" class="data row6 col3" >7</td>
<td id="T_821479f4_b488_11e9_ad41_186590cd1c87row6_col4" class="data row6 col4" >9</td>
</tr>
</tbody></table>
</pre></div>
</div>
<p>Nothing really stands out. The “futures” users (who we expect to be relatively advanced) may prioritize features and performance over documentation. But everyone agrees that more examples are the highest priority.</p>
<aside class="system-message">
<p class="system-message-title">System Message: WARNING/2 (<span class="docutils literal">/opt/build/repo/2019/08/05/user-survey.md</span>, line 325)</p>
<p>Document headings start at H2, not H1 [myst.header]</p>
</aside>
</section>
<section id="common-feature-requests">
<h1>Common Feature Requests</h1>
<p>For specific features, we made a list of things that we (as developers) thought might be important.</p>
<p><img alt="svg" src="https://blog.dask.org/_images/analyze_22_0.svg" /></p>
<p>The clearest standout is how many people thought “Better NumPy/Pandas support” was “most critical”. In hindsight, it’d be good to have a followup fill-in field to undertand what each respondent meant by that. The parsimonious interpretion is “cover more of the NumPy / pandas API”.</p>
<p>“Ease of deployment” had a high proportion of “critical to me”. Again in hindsight, I notice a bit of ambiguity. Does this mean people want Dask to be easier to deploy? Or does this mean that Dask, which they currently find easy to deploy, is critically important? Regardless, we can prioritize simplicity in deployment.</p>
<p>Relatively few respondents care about things like “Managing many users”, though we expect that this would be relatively popular among system administartors, who are a smaller population.</p>
<p>And of course, we have people pushing Dask to its limits for whom “Improving scaling” is critically important.</p>
<aside class="system-message">
<p class="system-message-title">System Message: WARNING/2 (<span class="docutils literal">/opt/build/repo/2019/08/05/user-survey.md</span>, line 339)</p>
<p>Document headings start at H2, not H1 [myst.header]</p>
</aside>
</section>
<section id="what-other-systems-do-you-use">
<h1>What other systems do you use?</h1>
<p>A relatively high proportion of respondents use Python 3 (97% compared to 84% in the most recent <a class="reference external" href="https://www.jetbrains.com/research/python-developers-survey-2018/">Python Developers Survey</a>).</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>3 97.29%
2 2.71%
Name: Python 2 or 3?, dtype: object
</pre></div>
</div>
<p>We were a bit surprised to see that SSH is the most popular “cluster resource manager”.</p>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span>SSH 98
Kubernetes 73
HPC resource manager (SLURM, PBS, SGE, LSF or similar) 61
My workplace has a custom solution for this 23
I don't know, someone else does this for me 16
Hadoop / Yarn / EMR 14
Name: If you use a cluster, how do you launch Dask? , dtype: int64
</pre></div>
</div>
<p>How does cluster-resource manager compare with API usage?</p>
<style type="text/css" >
#T_8326d0f8_b488_11e9_ad41_186590cd1c87row0_col0 {
background-color: #056faf;
color: #f1f1f1;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row0_col1 {
background-color: #fff7fb;
color: #000000;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row0_col2 {
background-color: #023858;
color: #f1f1f1;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row0_col3 {
background-color: #034e7b;
color: #f1f1f1;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row0_col4 {
background-color: #2685bb;
color: #000000;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row0_col5 {
background-color: #fff7fb;
color: #000000;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row0_col6 {
background-color: #f2ecf5;
color: #000000;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row1_col0 {
background-color: #023858;
color: #f1f1f1;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row1_col1 {
background-color: #f7f0f7;
color: #000000;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row1_col2 {
background-color: #0771b1;
color: #f1f1f1;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row1_col3 {
background-color: #0771b1;
color: #f1f1f1;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row1_col4 {
background-color: #c5cce3;
color: #000000;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row1_col5 {
background-color: #fff7fb;
color: #000000;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row1_col6 {
background-color: #79abd0;
color: #000000;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row2_col0 {
background-color: #8bb2d4;
color: #000000;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row2_col1 {
background-color: #b4c4df;
color: #000000;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row2_col2 {
background-color: #023858;
color: #f1f1f1;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row2_col3 {
background-color: #589ec8;
color: #000000;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row2_col4 {
background-color: #eee9f3;
color: #000000;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row2_col5 {
background-color: #8bb2d4;
color: #000000;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row2_col6 {
background-color: #fff7fb;
color: #000000;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row3_col0 {
background-color: #4c99c5;
color: #000000;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row3_col1 {
background-color: #fff7fb;
color: #000000;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row3_col2 {
background-color: #023858;
color: #f1f1f1;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row3_col3 {
background-color: #056dac;
color: #f1f1f1;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row3_col4 {
background-color: #73a9cf;
color: #000000;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row3_col5 {
background-color: #d9d8ea;
color: #000000;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row3_col6 {
background-color: #f3edf5;
color: #000000;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row4_col0 {
background-color: #056ba9;
color: #f1f1f1;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row4_col1 {
background-color: #fff7fb;
color: #000000;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row4_col2 {
background-color: #023858;
color: #f1f1f1;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row4_col3 {
background-color: #1379b5;
color: #f1f1f1;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row4_col4 {
background-color: #dfddec;
color: #000000;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row4_col5 {
background-color: #e8e4f0;
color: #000000;
} #T_8326d0f8_b488_11e9_ad41_186590cd1c87row4_col6 {
background-color: #f9f2f8;
color: #000000;
}</style><table id="T_8326d0f8_b488_11e9_ad41_186590cd1c87" ><thead> <tr> <th class="index_name level0" >Dask APIs</th> <th class="col_heading level0 col0" >Array</th> <th class="col_heading level0 col1" >Bag</th> <th class="col_heading level0 col2" >DataFrame</th> <th class="col_heading level0 col3" >Delayed</th> <th class="col_heading level0 col4" >Futures</th> <th class="col_heading level0 col5" >ML</th> <th class="col_heading level0 col6" >Xarray</th> </tr> <tr> <th class="index_name level0" >If you use a cluster, how do you launch Dask? </th> <th class="blank" ></th> <th class="blank" ></th> <th class="blank" ></th> <th class="blank" ></th> <th class="blank" ></th> <th class="blank" ></th> <th class="blank" ></th> </tr></thead><tbody>
<div class="highlight-none notranslate"><div class="highlight"><pre><span></span> <tr>
<th id="T_8326d0f8_b488_11e9_ad41_186590cd1c87level0_row0" class="row_heading level0 row0" >Custom</th>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row0_col0" class="data row0 col0" >15</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row0_col1" class="data row0 col1" >6</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row0_col2" class="data row0 col2" >18</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row0_col3" class="data row0 col3" >17</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row0_col4" class="data row0 col4" >14</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row0_col5" class="data row0 col5" >6</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row0_col6" class="data row0 col6" >7</td>
</tr>
<tr>
<th id="T_8326d0f8_b488_11e9_ad41_186590cd1c87level0_row1" class="row_heading level0 row1" >HPC</th>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row1_col0" class="data row1 col0" >50</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row1_col1" class="data row1 col1" >13</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row1_col2" class="data row1 col2" >40</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row1_col3" class="data row1 col3" >40</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row1_col4" class="data row1 col4" >22</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row1_col5" class="data row1 col5" >11</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row1_col6" class="data row1 col6" >30</td>
</tr>
<tr>
<th id="T_8326d0f8_b488_11e9_ad41_186590cd1c87level0_row2" class="row_heading level0 row2" >Hadoop / Yarn / EMR</th>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row2_col0" class="data row2 col0" >7</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row2_col1" class="data row2 col1" >6</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row2_col2" class="data row2 col2" >12</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row2_col3" class="data row2 col3" >8</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row2_col4" class="data row2 col4" >4</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row2_col5" class="data row2 col5" >7</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row2_col6" class="data row2 col6" >3</td>
</tr>
<tr>
<th id="T_8326d0f8_b488_11e9_ad41_186590cd1c87level0_row3" class="row_heading level0 row3" >Kubernetes</th>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row3_col0" class="data row3 col0" >40</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row3_col1" class="data row3 col1" >18</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row3_col2" class="data row3 col2" >56</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row3_col3" class="data row3 col3" >47</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row3_col4" class="data row3 col4" >37</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row3_col5" class="data row3 col5" >26</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row3_col6" class="data row3 col6" >21</td>
</tr>
<tr>
<th id="T_8326d0f8_b488_11e9_ad41_186590cd1c87level0_row4" class="row_heading level0 row4" >SSH</th>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row4_col0" class="data row4 col0" >61</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row4_col1" class="data row4 col1" >23</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row4_col2" class="data row4 col2" >72</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row4_col3" class="data row4 col3" >58</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row4_col4" class="data row4 col4" >32</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row4_col5" class="data row4 col5" >30</td>
<td id="T_8326d0f8_b488_11e9_ad41_186590cd1c87row4_col6" class="data row4 col6" >25</td>
</tr>
</tbody></table>
</pre></div>
</div>
<p>HPC users are relatively heavy users of <code class="docutils literal notranslate"><span class="pre">dask.array</span></code> and xarray.</p>
<p>Somewhat surprisingly, Dask’s heaviest users find dask stable enough. Perhaps they’ve pushed past the bugs and found workarounds (percentages are normalized by row).</p>
<p><img alt="svg" src="https://blog.dask.org/_images/analyze_32_0.svg" /></p>
<aside class="system-message">
<p class="system-message-title">System Message: WARNING/2 (<span class="docutils literal">/opt/build/repo/2019/08/05/user-survey.md</span>, line 525)</p>
<p>Document headings start at H2, not H1 [myst.header]</p>
</aside>
</section>
<section id="takeaways">
<h1>Takeaways</h1>
<ol class="arabic simple">
<li><p>We should prioritize improving and expanding our documentation and examples. This may be
accomplished by Dask maintainers seeking examples from the community. Many of the examples
on <a class="reference external" href="https://examples.dask.org">https://examples.dask.org</a> were developed by domain specialist who use Dask.</p></li>
<li><p>Improved scaling to larger problems is important, but we shouldn’t
sacrifice the single-machine usecase to get there.</p></li>
<li><p>Both interactive and batch workflows are important.</p></li>
<li><p>Dask’s various sub-communities are more similar than they are different.</p></li>
</ol>
<p>Thanks again to all the respondents. We look forward to repeating this process to identify trends over time.</p>
</section>
Document headings start at H2, not H1 [myst.header]2019-08-05T00:00:00+00:00https://blog.dask.org/2019/01/22/dask-extension-arrays/Extension Arrays in Dask DataFrame2019-01-22T00:00:00+00:00Tom Augspurger<p><em>This work is supported by <a class="reference external" href="http://anaconda.com">Anaconda Inc</a></em></p>
<aside class="system-message">
<p class="system-message-title">System Message: WARNING/2 (<span class="docutils literal">/opt/build/repo/2019/01/22/dask-extension-arrays.md</span>, line 11)</p>
<p>Document headings start at H2, not H1 [myst.header]</p>
</aside>
<section id="summary">
<p>Dask DataFrame works well with pandas’ new Extension Array interface, including
third-party extension arrays. This lets Dask</p>
<ol class="arabic simple">
<li><p>easily support pandas’ new extension arrays, like their new <a class="reference external" href="http://pandas.pydata.org/pandas-docs/version/0.24/whatsnew/v0.24.0.html#optional-integer-na-support">nullable integer
array</a></p></li>
<li><p>support third-party extension array arrays, like <a class="reference external" href="https://cyberpandas.readthedocs.io">cyberpandas’s</a>
<code class="docutils literal notranslate"><span class="pre">IPArray</span></code></p></li>
</ol>
<aside class="system-message">
<p class="system-message-title">System Message: WARNING/2 (<span class="docutils literal">/opt/build/repo/2019/01/22/dask-extension-arrays.md</span>, line 21)</p>
<p>Document headings start at H2, not H1 [myst.header]</p>
</aside>
</section>
<section id="background">
<h1>Background</h1>
<p>Pandas 0.23 introduced the <a class="reference external" href="http://pandas.pydata.org/pandas-docs/version/0.24/extending.html#extension-types"><code class="docutils literal notranslate"><span class="pre">ExtensionArray</span></code></a>, a way to store things other
than a simple NumPy array in a DataFrame or Series. Internally pandas uses this
for data types that aren’t handled natively by NumPy like datetimes with
timezones, Categorical, or (the new!) nullable integer arrays.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">s</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">Series</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">date_range</span><span class="p">(</span><span class="s1">'2000'</span><span class="p">,</span> <span class="n">periods</span><span class="o">=</span><span class="mi">4</span><span class="p">,</span> <span class="n">tz</span><span class="o">=</span><span class="s2">"US/Central"</span><span class="p">))</span>
<span class="gp">>>> </span><span class="n">s</span>
<span class="go">0 2000-01-01 00:00:00-06:00</span>
<span class="go">1 2000-01-02 00:00:00-06:00</span>
<span class="go">2 2000-01-03 00:00:00-06:00</span>
<span class="go">3 2000-01-04 00:00:00-06:00</span>
<span class="go">dtype: datetime64[ns, US/Central]</span>
</pre></div>
</div>
<p><code class="docutils literal notranslate"><span class="pre">dask.dataframe</span></code> has always supported the extension types that pandas defines.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">import</span><span class="w"> </span><span class="nn">dask.dataframe</span><span class="w"> </span><span class="k">as</span><span class="w"> </span><span class="nn">dd</span>
<span class="gp">>>> </span><span class="n">dd</span><span class="o">.</span><span class="n">from_pandas</span><span class="p">(</span><span class="n">s</span><span class="p">,</span> <span class="n">npartitions</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="go">Dask Series Structure:</span>
<span class="go">npartitions=2</span>
<span class="go">0 datetime64[ns, US/Central]</span>
<span class="go">2 ...</span>
<span class="go">3 ...</span>
<span class="go">dtype: datetime64[ns, US/Central]</span>
<span class="go">Dask Name: from_pandas, 2 tasks</span>
</pre></div>
</div>
<aside class="system-message">
<p class="system-message-title">System Message: WARNING/2 (<span class="docutils literal">/opt/build/repo/2019/01/22/dask-extension-arrays.md</span>, line 52)</p>
<p>Document headings start at H2, not H1 [myst.header]</p>
</aside>
</section>
<section id="the-challenge">
<h1>The Challenge</h1>
<p>Newer versions of pandas allow third-party libraries to write custom extension
arrays. These arrays can be placed inside a DataFrame or Series, and work
just as well as any extension array defined within pandas itself. However,
third-party extension arrays provide a slight challenge for Dask.</p>
<p>Recall: <code class="docutils literal notranslate"><span class="pre">dask.dataframe</span></code> is lazy. We use a familiar pandas-like API to build up
a task graph, rather than executing immediately. But if Dask DataFrame is lazy,
then how do things like the following work?</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">({</span><span class="s2">"A"</span><span class="p">:</span> <span class="p">[</span><span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">],</span> <span class="s1">'B'</span><span class="p">:</span> <span class="p">[</span><span class="mi">3</span><span class="p">,</span> <span class="mi">4</span><span class="p">]})</span>
<span class="gp">>>> </span><span class="n">ddf</span> <span class="o">=</span> <span class="n">dd</span><span class="o">.</span><span class="n">from_pandas</span><span class="p">(</span><span class="n">df</span><span class="p">,</span> <span class="n">npartitions</span><span class="o">=</span><span class="mi">2</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">ddf</span><span class="p">[[</span><span class="s1">'B'</span><span class="p">]]</span><span class="o">.</span><span class="n">columns</span>
<span class="go">Index(['B'], dtype='object')</span>
</pre></div>
</div>
<p><code class="docutils literal notranslate"><span class="pre">ddf[['B']]</span></code> (lazily) selects the column <code class="docutils literal notranslate"><span class="pre">'B'</span></code> from the dataframe. But accessing
<code class="docutils literal notranslate"><span class="pre">.columns</span></code> <em>immediately</em> returns a pandas Index object with just the selected
columns.</p>
<p>No real computation has happened (you could just as easily swap out the
<code class="docutils literal notranslate"><span class="pre">from_pandas</span></code> for a <code class="docutils literal notranslate"><span class="pre">dd.read_parquet</span></code> on a larger-than-memory dataset, and the
behavior would be the same). Dask is able to do these kinds of “metadata-only”
computations, where the output depends only on the columns and the dtypes,
without executing the task graph. Internally, Dask does this by keeping a pair
of dummy pandas DataFrames on each Dask DataFrame.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">ddf</span><span class="o">.</span><span class="n">_meta</span>
<span class="go">Empty DataFrame</span>
<span class="go">Columns: [A, B]</span>
<span class="go">Index: []</span>
<span class="gp">>>> </span><span class="n">ddf</span><span class="o">.</span><span class="n">_meta_nonempty</span>
<span class="go">ddf._meta_nonempty</span>
<span class="go"> A B</span>
<span class="go">0 1 1</span>
<span class="go">1 1 1</span>
</pre></div>
</div>
<p>We need the <code class="docutils literal notranslate"><span class="pre">_meta_nonempty</span></code>, since some operations in pandas behave differently
on an Empty DataFrame than on a non-empty one (either by design or,
occasionally, a bug in pandas).</p>
<p>The issue with third-party extension arrays is that Dask doesn’t know what
values to put in the <code class="docutils literal notranslate"><span class="pre">_meta_nonempty</span></code>. We’re quite happy to do it for each NumPy
dtype and each of pandas’ own extension dtypes. But any third-party library
could create an ExtensionArray for any type, and Dask would have no way of
knowing what’s a valid value for it.</p>
<aside class="system-message">
<p class="system-message-title">System Message: WARNING/2 (<span class="docutils literal">/opt/build/repo/2019/01/22/dask-extension-arrays.md</span>, line 104)</p>
<p>Document headings start at H2, not H1 [myst.header]</p>
</aside>
</section>
<section id="the-solution">
<h1>The Solution</h1>
<p>Rather than Dask guessing what values to use for the <code class="docutils literal notranslate"><span class="pre">_meta_nonempty</span></code>, extension
array authors (or users) can register their extension dtype with Dask. Once
registered, Dask will be able to generate the <code class="docutils literal notranslate"><span class="pre">_meta_nonempty</span></code>, and things
should work fine from there. For example, we can register the dummy <code class="docutils literal notranslate"><span class="pre">DecimalArray</span></code>
that pandas uses for testing (this isn’t part of pandas’ public API) with Dask.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="kn">from</span><span class="w"> </span><span class="nn">decimal</span><span class="w"> </span><span class="kn">import</span> <span class="n">Decimal</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">pandas.tests.extension.decimal</span><span class="w"> </span><span class="kn">import</span> <span class="n">DecimalArray</span><span class="p">,</span> <span class="n">DecimalDtype</span>
<span class="c1"># The actual registration that would be done in the 3rd-party library</span>
<span class="kn">from</span><span class="w"> </span><span class="nn">dask.dataframe.extensions</span><span class="w"> </span><span class="kn">import</span> <span class="n">make_array_nonempty</span>
<span class="nd">@make_array_nonempty</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">DecimalDtype</span><span class="p">)</span>
<span class="k">def</span><span class="w"> </span><span class="nf">_</span><span class="p">(</span><span class="n">dtype</span><span class="p">):</span>
<span class="k">return</span> <span class="n">DecimalArray</span><span class="o">.</span><span class="n">_from_sequence</span><span class="p">([</span><span class="n">Decimal</span><span class="p">(</span><span class="s1">'0'</span><span class="p">),</span> <span class="n">Decimal</span><span class="p">(</span><span class="s1">'NaN'</span><span class="p">)],</span>
<span class="n">dtype</span><span class="o">=</span><span class="n">dtype</span><span class="p">)</span>
</pre></div>
</div>
<p>Now users of that extension type can place those arrays inside a Dask DataFrame
or Series.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">({</span><span class="s2">"A"</span><span class="p">:</span> <span class="n">DecimalArray</span><span class="p">([</span><span class="n">Decimal</span><span class="p">(</span><span class="s1">'1.0'</span><span class="p">),</span> <span class="n">Decimal</span><span class="p">(</span><span class="s1">'2.0'</span><span class="p">),</span>
<span class="gp">... </span> <span class="n">Decimal</span><span class="p">(</span><span class="s1">'3.0'</span><span class="p">)])})</span>
<span class="gp">>>> </span><span class="n">ddf</span> <span class="o">=</span> <span class="n">dd</span><span class="o">.</span><span class="n">from_pandas</span><span class="p">(</span><span class="n">df</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
<span class="gp">>>> </span><span class="n">ddf</span>
<span class="go">Dask DataFrame Structure:</span>
<span class="go"> A</span>
<span class="go">npartitions=1</span>
<span class="go">0 decimal</span>
<span class="go">2 ...</span>
<span class="go">Dask Name: from_pandas, 1 tasks</span>
<span class="gp">>>> </span><span class="n">ddf</span><span class="o">.</span><span class="n">dtypes</span>
<span class="go">A decimal</span>
<span class="go">dtype: object</span>
</pre></div>
</div>
<p>And from there, the usual operations just as they would in pandas.</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="gp">>>> </span><span class="kn">from</span><span class="w"> </span><span class="nn">random</span><span class="w"> </span><span class="kn">import</span> <span class="n">choices</span>
<span class="gp">>>> </span><span class="n">df</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">DataFrame</span><span class="p">({</span><span class="s2">"A"</span><span class="p">:</span> <span class="n">DecimalArray</span><span class="p">(</span><span class="n">choices</span><span class="p">([</span><span class="n">Decimal</span><span class="p">(</span><span class="s1">'1.0'</span><span class="p">),</span>
<span class="gp">... </span> <span class="n">Decimal</span><span class="p">(</span><span class="s1">'2.0'</span><span class="p">)],</span>
<span class="gp">... </span> <span class="n">k</span><span class="o">=</span><span class="mi">100</span><span class="p">)),</span>
<span class="gp">... </span> <span class="s2">"B"</span><span class="p">:</span> <span class="n">np</span><span class="o">.</span><span class="n">random</span><span class="o">.</span><span class="n">choice</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span><span class="p">],</span> <span class="n">size</span><span class="o">=</span><span class="p">(</span><span class="mi">100</span><span class="p">,))})</span>
<span class="gp">>>> </span><span class="n">ddf</span> <span class="o">=</span> <span class="n">dd</span><span class="o">.</span><span class="n">from_pandas</span><span class="p">(</span><span class="n">df</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
<span class="go">In [35]: ddf.groupby("A").B.mean().compute()</span>
<span class="go">Out[35]:</span>
<span class="go">A</span>
<span class="go">1.0 1.50</span>
<span class="go">2.0 1.48</span>
<span class="go">Name: B, dtype: float64</span>
</pre></div>
</div>
<aside class="system-message">
<p class="system-message-title">System Message: WARNING/2 (<span class="docutils literal">/opt/build/repo/2019/01/22/dask-extension-arrays.md</span>, line 165)</p>
<p>Document headings start at H2, not H1 [myst.header]</p>
</aside>
</section>
<section id="the-real-lesson">
<h1>The Real Lesson</h1>
<p>It’s neat that Dask now supports extension arrays. But to me, the exciting thing
is just how little work this took. The
<a class="reference external" href="https://github.com/dask/dask/pull/4379/files">PR</a> implementing support for
third-party extension arrays is quite short, just defining the object that
third-parties register with, and using it to generate the data when dtype is
detected. Supporting the three new extension arrays in pandas 0.24.0
(<code class="docutils literal notranslate"><span class="pre">IntegerArray</span></code>, <code class="docutils literal notranslate"><span class="pre">PeriodArray</span></code>, and <code class="docutils literal notranslate"><span class="pre">IntervalArray</span></code>), takes a handful of lines
of code</p>
<div class="highlight-python notranslate"><div class="highlight"><pre><span></span><span class="nd">@make_array_nonempty</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">Interval</span><span class="p">):</span>
<span class="k">def</span><span class="w"> </span><span class="nf">_</span><span class="p">(</span><span class="n">dtype</span><span class="p">):</span>
<span class="k">return</span> <span class="n">IntervalArray</span><span class="o">.</span><span class="n">from_breaks</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">],</span> <span class="n">closed</span><span class="o">=</span><span class="n">dtype</span><span class="o">.</span><span class="n">closed</span><span class="p">)</span>
<span class="nd">@make_array_nonempty</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">pd</span><span class="o">.</span><span class="n">Period</span><span class="p">):</span>
<span class="k">def</span><span class="w"> </span><span class="nf">_</span><span class="p">(</span><span class="n">dtype</span><span class="p">):</span>
<span class="k">return</span> <span class="n">period_array</span><span class="p">([</span><span class="mi">2000</span><span class="p">,</span> <span class="mi">2001</span><span class="p">],</span> <span class="n">freq</span><span class="o">=</span><span class="n">dtype</span><span class="o">.</span><span class="n">freq</span><span class="p">)</span>
<span class="nd">@make_array_nonempty</span><span class="o">.</span><span class="n">register</span><span class="p">(</span><span class="n">_IntegerDtype</span><span class="p">):</span>
<span class="k">def</span><span class="w"> </span><span class="nf">_</span><span class="p">(</span><span class="n">dtype</span><span class="p">):</span>
<span class="k">return</span> <span class="n">integer_array</span><span class="p">([</span><span class="mi">0</span><span class="p">,</span> <span class="kc">None</span><span class="p">],</span> <span class="n">dtype</span><span class="o">=</span><span class="n">dtype</span><span class="p">)</span>
</pre></div>
</div>
<p>Dask benefits directly from improvements made to pandas. Dask didn’t have to
build out a new parallel extension array interface, and reimplement all the new
extension arrays using the parallel interface. We just re-used what pandas
already did, and it fits into the existing Dask structure.</p>
<p>For third-party extension array authors, like <a class="reference external" href="https://cyberpandas.readthedocs.io">cyberpandas</a>, the
work is similarly minimal. They don’t need to re-implement everything from the
ground up, just to play well with Dask.</p>
<p>This highlights the importance of one of the Dask project’s core values: working
with the community. If you visit <a class="reference external" href="https://dask.org">dask.org</a>, you’ll see
phrases like</p>
<blockquote>
<div><p>Integrates with existing projects</p>
</div></blockquote>
<p>and</p>
<blockquote>
<div><p>Built with the broader community</p>
</div></blockquote>
<p>At the start of Dask, the developers <em>could</em> have gone off and re-written pandas
or NumPy from scratch to be parallel friendly (though we’d probably still be
working on that part today, since that’s such a massive undertaking). Instead,
the Dask developers worked with the community, occasionally nudging it in
directions that would help out dask. For example, many places in pandas <a class="reference external" href="http://matthewrocklin.com/blog/work/2015/03/10/PyData-GIL">held
the GIL</a>, preventing
thread-based parallelism. Rather than abandoning pandas, the Dask and pandas
developers worked together to release the GIL where possible when it was a
bottleneck for <code class="docutils literal notranslate"><span class="pre">dask.dataframe</span></code>. This benefited Dask and anyone else trying to
do thread-based parallelism with pandas DataFrames.</p>
<p>And now, when pandas introduces new features like nullable integers,
<code class="docutils literal notranslate"><span class="pre">dask.dataframe</span></code> just needs to register it as an extension type and immediately
benefits from it. And third-party extension array authors can do the same for
their extension arrays.</p>
<p>If you’re writing an ExtensionArray, make sure to add it to the <a class="reference external" href="http://pandas.pydata.org/pandas-docs/version/0.24/ecosystem.html#extension-data-types">pandas
ecosystem</a> page, and register it with Dask!</p>
</section>
This work is supported by Anaconda Inc2019-01-22T00:00:00+00:00