Dask Working Notes - Posts tagged Dask Summit

Dask Down Under

2021-06-25T00:00:00+00:00

Dask Down Under was a special event held for the first time last month during the 2021 Dask Summit. It featured talks, tutorials, and events tailored specifically for an Australian (and wider Oceania) audience.

To get involved in the new Pangeo Oceania community group, register your interest here.

What is Dask Down Under?
Who came?
Watch the talks
What’s next? Here’s how to get involved!

What is Dask Down Under?

Dask down under is a chance for everyone in Oceania to forge links and build community here in our backyard. Dask down under we feature talks, tutorials and panel discussions on using Dask to accelerate research. All levels from beginner to expert are encouraged to attend.

Dask Down Under involved two days of events:

5 talks
2 tutorials
1 panel discussion
1 meet and greet networking event

Who came?

There was a strong geoscience theme across Dask Down Under. This reflects the strong scientific community we have in these areas. People came from government organisations, universities, and industry.

We expected most attendees would be based in the Asia-Pacific region, since those were the timezones targeted by these events.

Unexpectedly, we also saw a lot of extra traffic at the talks on day one, likely from US timezones. Publicity from Dask Summit emails and tweets mentioning Dask Down Under resulted in a lot of people stopping by to watch. This more than doubled our live attendance during the first event. It was great to see so much interest coming from other parts of the world, too.

Watch the talks

You can watch the talks and tutorials from the Dask Dwon Under workshop on the Dask youtube channel. The full playlist for the workshop is available here.

Panel discussion

A panel discussion was held, bringing together a diverse group of users from novice to expert, academic to commercial. We hope this discussion will start a conversation about using Dask in Australia, how we build our community, contribute and stay in touch with the rest of the world. You can watch it here:

Moderator: Draga Doncila Pop
Panelists:

Ben Leighton, CSIRO
Tisham Dhar, Geoscience Australia
Genevieve Buckley, Dask life science fellow
Hugo Bowne-Anderson, Coiled

Invited talks

The full playlist for the workshop is available here.

Featured talks include:

Draga Doncila Pop, Interactive visualization and near real-time analysis on out-of-core satellite images
Tisham Dhar: Dask DevOps for Remote Sensing
Kirill Kouzoubov: Patterns for large scale temporal processing of geo-spatial data using Dask
Ben Leighton and Kim Opie: Image Processing Using Dask - Using dask and skimage to identity vegetation morphology across the Australian landscape
Nick Mortimer: Making the most of your schedule: From HPC to Local Cluster

What’s next?

Here’s how you can get involved:

Several people have discussed setting up a new Pangeo Oceania group. You can register your interest here.

> Soon we'll start holding regular Pangeo Oceania meetups for sharing information, support, training, and workflow advocacy across our region.  We look forward to you helping to shape the Pangeo Oceania community. And if you have a friend or colleague that should be here too, please share this sign-up link: http://bit.ly/Pangeo_email_signup

The Python for Atmosphere and Ocean Science (PyAOS) provides information and resources to the user community: https://pyaos.github.io/ To keep the site up-to-date, the first ever PyAOS census is being conducted. It would be great if Python users in the atmosphere and/or ocean science community could take a few minutes to fill out the survey. https://forms.gle/L84W7bsxmP86G3Ji9

Life sciences at the 2021 Dask Summit

2021-05-24T00:00:00+00:00

The Dask life science workshop ran as part of the 2021 Dask Summit. Lightning talks from this workshop are available here, and you can read on for a summary of the event.

What is the Dask life science workshop?

The Dask life science workshop ran as part of the 2021 Dask Summit. Currently many people in life sciences use Dask, but individual groups are relatively isolated from one another. This workshop gave us an opportunity to learn from each other, as well as opportunities to identify common frustrations and areas for improvement.

The workshop involved:

Pre-recorded lightning talks
Interactive discussion times (accessible across timezones in Europe, Oceania, and the Americas)
Asynchronous text chat throughout the Dask Summit

If I missed it, how can I catch up?

If you missed the Dask Summit, you can catch up on YouTube. There is a playlist of all the life science lightning talks available here.

You can also join our #life-science channel on Slack: Click here for an invitation link.

Who came?

We invited attendees at the life science workshop to do a short Q&A about their work with Dask. This is a small subset of the people who joined us, many people came to the conference and did not do a Q&A.

The responses give us an overview of the diversity of work people in the community are doing. In no particular order, here are some of those Q&As:

Name: Tom White
Timezone: EU/UK
What kind of science do you work on? Statistical genetics
Something you’ve tried (or would like to try) with Dask? Run per-row linear regressions at scale.
What do you want to do next with Dask? Collaborative optimization of a public workflow (GWAS).
Lightning talk: click here

Name: Giovanni Palla
Affiliation: Helmholtz Center Munich
Timezone: Europe
What kind of science do you work on? Computational Biology and Spatial transcriptomics
Something you’ve tried (or would like to try) with Dask? dask-image for image processing.
**What do you want to do next with Dask? Further integration with Squidpy.
**Lightning talk:** click here

Name: Isaac Virshup
Affiliation: University of Melbourne. Open source projects Scanpy and AnnData Timezone: AEST
What kind of science do you work on? Single cell omics data.
Something you’ve tried (or would like to try) with Dask?
I’ve used dask for some nested embarrassingly parallel calculations. Having an intelligent scheduler with good monitoring made this task as easy as it should be, especially compared with multiprocessing or joblib.
What do you want to do next with Dask?
I would love to get AnnData, a container for working with single cell assays integrated with dask. Dataset sizes in this field are constantly increasing, and it would be good to be able to work with the coolest new dataset regardless of available RAM.
Since we rely heavily on sparse arrays, a key step towards this will be getting better sparse array support (CSC and CSR especially) inside dask. After all, it’s not great if our strategy for scaling out requires many times the total memory! As a maintainer, I’m interested in hearing people’s experience with distributing tools that integrate well with dask.
Lightning talk: click here

Name: Anna Kreshuk
Affiliation: European Molecular Biology Laboratory
Timezone: CEST (GMT+2)
What kind of science do you work on? Machine learning for microscopy image analysis.
Something you’ve tried (or would like to try) with Dask? We run a lot of image processing workflows and want to see how Dask can be exploited in this context.

Name: Beth Cimini
Affiliation: Broad Institute
Timezone: US-East
What kind of science do you work on? User friendly image analysis tools for microscopy imaging.
Something you’ve tried (or would like to try) with Dask? Making Dask work in CellProfiler, to make it easy to analyze big images in high throughput!
Lightning talk: click here

Name: Volker Hilsenstein
Affiliation: EMBL / Alexandrov lab
Timezone: Central European Summer Time
What kind of science do you work on? Spatial Metabolomics, combining microscopy and mass spectrometry.
Something I would like to try with dask: fusing large mosaics of individual images or image volumes for which affine transformation into a joint coordinate system are available.

Name: Marvin Albert
Affiliation: University of Zurich
Timezone: UTC/GMT +2
What kind of science do you work on? Life sciences / image analysis
Something you’ve tried (or would like to try) with Dask? What do you want to do next with Dask? Parallelise / reduce the memory footprint of image processing tasks and define workflows that can run on different compute environments.
Lightning talk: click here

Name: Jordao Bragantini
Affiliation: CZ Biohub
Timezone: Pacific Daylight Time (UTC -7)
What kind of science do you work on? Light-sheet microscopy
Something you’ve tried (or would like to try) with Dask? Image processing of very large data.
What do you want to do next with Dask? Implement algorithms for cell segmentation.
Lightning talk: click here

Name: Josh Moore
Affiliation: Open Microscopy Environment (OME)
Timezone: CEST
What kind of science do you work on? Bioimaging (infrastructure for RDM)
Something you’ve tried (or would like to try) with Dask? Accessing large image (Zarr) volumes over HTTP, primarily. What do you want to do next with Dask? Improve pre-fetching for typical usage patterns, possibly integrating multiscale data (i.e. google maps zooming)
Lightning talk: click here

Name: Jackson Maxfield Brown
Timezone: PST
What kind of science do you work in? Cell biology, specifically microscopy and computational biology.
Something you’ve tried (or would like to try) with Dask? Built a metadata aware / backed microscopy imaging reading library that uses Dask to read any size image w/ chunking by metadata dimension information. As well as TB-scale image processing pipelines using Dask + Prefect.
What do you want to do next with Dask? Tighter integration with other libraries. I see cuCim from the RAPIDs team and would love to extend work with them to have a more general “bio-image-spec” so we can all play nicely together.
Lightning talk: click here

Name: Gregory R. Lee
Affiliation: Quansight
Timezone: EST (UTC-5)
What kind of science do you work on? Scientific software development (with a background doing research in magnetic resonance imaging).
Something you’ve tried (or would like to try) with Dask?
In past research work, I used Dask primarily in two scenarios, both on a single workstation:

To achieve multi-threading by processing image blocks in parallel on the CPU (e.g. like in dask-image)
Serial blockwise processing of large volumetric data on the GPU (i.e. CuPy arrays of 10-100 GB in size) to reduce peak memory requirements.

What do you want to do next with Dask?
Audit scikit-image functions to determine which can easily be accelerated using block-wise approaches as in dask-image. Ideally a subset of functions would work directly with dask-arrays as inputs rather than requiring users to learn about Dask’s map_overlap, etc. to use this feature.
Lightning talk: click here

What’s next?

Dask is now considering holding “office hours” for the life science community. If we can find enough maintainers able to host one-hour Q&A sessions, then we’ll trial this for a short period of time.

Dask Working Notes - Posts tagged Dask Summit

Dask Down Under

Contents

What is Dask Down Under?

Who came?

Watch the talks

Panel discussion

Invited talks

What’s next?

Life sciences at the 2021 Dask Summit

What is the Dask life science workshop?

If I missed it, how can I catch up?

Who came?

What’s next?