Dask Working Notes - Posts tagged life science

Reflections on one year as the Dask life science fellow

2021-12-15T00:00:00+00:00

Genevieve Buckley was hired as a Dask Life Science Fellow in 2021 funded by CZI. The goal was to improve Dask, with a specific focus on the life science community. This blogpost contains another progress update, and some personal reflections looking back over this year.

Progress update
Personal reflections
What’s next in Dask?

Progress update

A previous progress update for February to September 2021 is available here. Read on for a progress update for the period September to December 2021.

To summarize, between September and December 2021 inclusive, there were:

32 merged pull requests acorss 7 repositories (dask, distributed, dask-image, dask-tutorial, ITK, napari, and napari.github.io)
8 pending pull requests
1 new dask-image release
1 Dask tutorial run, and assisted with a second tutorial.
4 new Dask blogposts published (five, if we count this one)

Read on for a more detailed description of special projects within this time.

Dask stale issues sprint

In two weeks I was able to:

close 117 stale issues, and
identify another 25 potential easy wins for the maintainer team to investigate further.

Lots of other people did work around the same time, following up on old pull requests and other maintanence work. The sprint was very successful overall.

Dask user survey results analysis

In September I analyzed the results from the 2021 Dask user survey. This was a really fun task. Because we asked a lot more questions in 2021 (18 new questions, 43 questions in total) there was was a lot more data to dig into, compared with previous years. You can read the full details about it here.

The biggest benefit from this work is that now we can use this data to prioritize improvements to the documentation and examples. The top two user requests are for more documentation and more examples from their industry. But it wasn’t until this year that we started asking what industries people worked in, so we can target new narrative documentation to the areas that need it most (geoscience, life science, and finance).

ITK compatibility with Dask

I implemented pickle serialization for itk images (ITK PR #2829). This should be one of the last major pieces of the puzzle needed to make ITK images compatible with Dask. It builds on earlier work by Matt McCormick and John Kirkham (you can read a blog post about their earlier work here).

Better cross-compatibility for Dask with other projects was a major goal of mine, so this is an important piece of work. I outline the next steps in the section What’s next in Dask?

Improve rechunking

I implemented PR #8124 fix a bug where reshaping a Dask array can cause an output array with chunks that are much too large to fit in memory. Feedback from the life science user survey indicates that improving Dask’s performance around rechunking is a priority. This work helps to address that.

High level graph work

A major piece of work earlier this year was introducing high level graphs for array slicing and array overlap operations. That is a big effort requiring a lot of ongoing work. PR #8467 tackles one of the next steps for this work.

Find objects function for dask-image

I implemented a find_objects function for dask-image in PR #240. This implementation does not need to know the maximum label number ahead of time, a subtantial improement over the previous attempt. This is a major step forward, because it removes a major blocker to introducing scikit-image like regionprops functionality.

Blogposts

Dask blogposts published between September through to December 2021 include:

Choosing good chunk sizes in Dask
- This blogpost addresses some very common concerns and questions about using Dask. I’m very pleased with this article, due to several thoughtful reviewers the final work is a much stronger and more comprehensive than the twitter thread that inspired it.
- It’s also high impact work. In the Dask survey the most common request is for more documentation, and this content helps to address that. Twitter analytics also show much higher engagement with this content than for other similar tweets, indicating a demand in the community for this type of explanation.
Mosaic Image Fusion (co-authored with Volker Hisenstein and Marvin Albert)
- This blogpost was several months in the making (started in mid-August and published in December). It’s fantastic to have people sharing some of the very cool work they do with Dask on real world problems.
CZI EOSS Update
- This blogpost shares with the community an interim progress update provided to CZI.
2021 Dask user survey results
- Discussed in more detail above, the analysis results from the Dask User Survey were published in September 2021.

Tutorials

I presented a Dask tutorial at the ResBaz Sydney online conference on the 25th of November 2021. Thanks to the ResBaz organisers and to David McFarlane, Svetlana Tkachenko, and Oksana Tkachenko for monitoring the chat for questions on the day.
Naty Clementi ran a Dask tutorial for the Women Who Code DC meetup on the 4th of November 2021. I assisted Naty, mostly by monitoring questions in the chat.

Personal reflections

Reflecting back over the whole year, there were some things that worked well and some things that were less successful.

Highlights from this year

My personal highlights include:

ITK + Dask integration work (discussed in more detail above).
A find objects fucntion for dask-image (discussed in more detail above).
Visualization work, because it’s very high impact. We’re solving issues raised by life science groups, but the improved tools benefit EVERYONE who uses Dask.
This bugfix from dask PR #7391, because this single change fixed problems in four places at once (scikit-image, dask-ml, xgcm/xhistogram, and the cupy dask tests).
Community building, conferences, and engagement. Lots of effort went into events over this year, and it’s certainly paid dividends.

What worked well

Dask stale issues sprint

This was useful for the project, as well as useful for me. Sorting through old issues was an incredibly effective way to get familiar with who the experts are for particular topics. It would have been even better if this happened in the first few months of working on Dask, instead of the last few months.
It’s been suggested that one good way to gain familiarity is spending 6 months full time managing the issue tracker. Maybe that’s true, but the much shorter stale issue sprint was a very efficient way of getting a lot of the same benefits in a short space of time. I’d recommend it for new maintainers or triage team members.

Community building events

We had a very successful year in terms of community building and events. This included tutorials, workshops, conferences, and community outreach. Summary of major events:

Led a Dask tutorial at ResBaz Sydney 2021 in November.
Co-led a half-day tutorial on napari and Dask at the Light Microscopy Australia Meeting in August.
SciPy 2021 presentation Scaling Science: leveraging Dask for life sciences in July.
Organized the Dask Life Science workshop at the Dask Summit in May 2021. The life science workshop included 15 pre-recorded talks, and 3 interactive discussions.
Co-organised the Dask Down Under workshop for the Dask Summit in May 2021. Dask Down Under contained 5 talks, 2 tutorials, 1 panel discussion, and 1 meet and greet networking event. Dask Down Under
Expert panelist at the VIS2021 symposium in February.

Visualization work

This has been very high impact work, and I’m pleased with what we’ve achieved. Improved tools for visualization were requested by users in our survey of the life science community. This was a high priority, because improvements to visuzliation tools benefit EVERYONE who uses Dask.

What didn’t work so well

Technical resources

We never really solved the problem of finding someone I could go to with technical questions. I did have people to ask about some specific projects, but in most cases I didn’t have a good way to direct questions to the right people. This is a challenging problem, especially because most Dask maintainers and contributors have full time jobs doing other things too. In my opinion, this negatively impacted the work and what we were able to achieve.

Being added to the @dask/maintenance team

There’s no point getting notifications if you don’t have GitHub permissions to do anything about them. In future I think we should add only people with at least triage or write permissions to the github teams.

Real time interaction

We tried out “Ask a maintainer” office hours for the life science community, but they were poorly attended, so we cancelled this.
We added some “Dask social chat” events to the calendar, but they were not very well attended outside of the first few. Most often, zero people attended. (There is another social chat for the Americas/Europe time zones, which is at a more convenient time for most people and might be more popular.)

Slack

Slack works well to DM specific people to set up meeting times, etc, but the public channels didn’t end up being very useful for me personally.

Lack of integration with other project teams

You can only get so much done as a solo developer. We had hoped that I would naturally end up working with teams from several different projects, but this didn’t really end up being the case. The napari project is an exception to this, and that relationship was well established before starting work for Dask. Perhaps there’s something more we could have done here to facilitate more interaction.

What’s next for Genevieve?

Genevieve will be starting a new job next year, you can find her on GitHub @GeneviveeBuckley.

What’s next in Dask?

Lots of stuff has happened in Dask, but there is still lots left to do. Here is a summary of the next steps for several projects. We’d love it if new people would like to take up the torch and contribute to any of these projects.

ITK image compatibility with Dask

The next steps for the ITK + Dask project require ITK release candidate 5.3rc3 or above to become available (likely early in 2022).
When the release is available, the next step is to try to re-run the code from the original ITK blogpost.
If there’s still work to be done we’ll need to open issues for the remaining blockers. And if it all works well, we’d like someone to write a second ITK + Dask blogpost to publicize the new functionality.

Improving performance around rechunking

More performance improvements related to rechunking is required (see #7950 and #7980).

High level graph work for arrays and slicing

The high level graph work for slicing and overlapping arrays has a number of next steps. Ian Rose has written an excellent summary here. Briefly, thecull and get_output_keys methods must be implemented, then low level fusion and optimizations can be done.

Relevant links:

Implement cull method for ArrayOverlapLayer #7789
Implement get_output_keys method for ArrayOverlapLayer #7791
Array slicing HighLevelGraph layer #7655

Documentation

Dask needs better documentation for high level graphs. Both user documentation and developer documentation is required.
At some future point, it might be worthwhile integrating blogpost content from Choosing good chunk sizes in Dask into the main Dask documentation, for better discoverability.

Mosaic Image Fusion

2021-12-01T00:00:00+00:00

This blogpost shows a case study where a researcher uses Dask for mosaic image fusion. Mosaic image fusion is when you combine multiple smaller images taken at known locations and stitch them together into a single image with a very large field of view. Full code examples are available on GitHub from the DaskFusion repository: VolkerH/DaskFusion

The problem

Image mosaicing in microscopy

In optical microscopy, a single field of view captured with a 20x objective typically has a diagonal on the order of a few 100 μm (exact dimensions depend on other parts of the optical system, including the size of the camera chip). A typical sample slide has a size of 25mm by 75mm. Therefore, when imaging a whole slide, one has to acquire hundreds of images, typically with some overlap between individual tiles. With increasing magnification, the required number of images increases accordingly.

To obtain an overview one has to fuse this large number of individual image tiles into a large mosaic image. Here, we assume that the information required for positioning and alignment of the individual image tiles is known. In the example presented here, this information is available as metadata recorded by the microscope, namely the microscope stage position and the pixel scale. Alternatively, this information could also be derived from the image data directly, e.g. through a registration step that matches corresponding image features in the areas where tiles overlap.

The solution

The array that can hold the resulting mosaic image will often have a size that is too large to fit in RAM, therefore we will use Dask arrays and the map_blocks function to enable out-of-core processing. The map_blocks function will process smaller blocks (a.k.a chunks) of the output array individually, thus eliminating the need to hold the whole output array in memory. If sufficient resources are available, dask will also distribute the processing of blocks across several workers, thus we also get parallel processing for free, which can help speed up the fusion process.

Typically whenever we want to join Dask arrays, we use Stack, Concatenate, and Block. However, these are not good tools for mosaic image fusion, because:

The image tiles will be be overlapping,
Tiles may not be positioned on an exact grid and will typically also have slight rotations as the alignment of stage and camera is not perfect. In the most general case, for example in panaromic photo mosaics, individual image tiles could be arbitrarily rotated or skewed.

The starting point for this mosaic prototype was some code that reads in the stage metadate for all tiles and calculates an affine transformation for each tile that would place it at the correct location in the output array.

The image below shows preliminary work placing mosaic image tiles into the correct positions using the napari image viewer. Shown here is a small example with 63 image tiles.

And here is an animation of placing the individual tiles.

To leverage processing with Dask we created a fuse function that generates a small block of the final mosaic and is invoked by map_blocks for each chunk of the output array. On each invocation of the fuse function map_blocks passes a dictionary (block_info). From the Dask documentation:

Your block function gets information about where it is in the array by accepting a special block_info or block_id keyword argument.

The basic outline of the fuse function of the mosaic workflow is as follows. For each chunk of the output array:

Determine which source image tiles intersect with the chunk.
Adjust the image tiles’ affine transformations to take the offset of the chunk within the array into account.
Load all intersectiong image tiles and apply their respective adjusted affine transformation to map them into the chunk.
Blend the tiles using a simple maximum projection.
Return the blended chunk.

Using a maximum projection to blend areas with overlapping tiles can lead to artifacts such as ghost images and visible tile seams, so you would typically want to use something more sophisticated in production.

Results

For datasets with many image tiles (~500-1000 tiles), we could speed up the mosaic generation from several hours to tens of minutes using this Dask based method (compared to a previous workflow using ImageJ plugins runnning on the same workstation). Due to Dask’s ability to handle data out-of-core and chunked array storage using zarr it is also possible to run the fusion on hardware with limited RAM.

Finally, we have the final mosaic fusion result.

Code

Code relatiing to this mosaic image fusion project can be found in the DaskFusion GitHub repository here: VolkerH/DaskFusion

There is a self-contained example available in this notebook, which downloads reduced-size example data to demonstrate the process.

What’s next?

Currently, the DaskFusion code is a proof of concept for single-channel 2D images and simple maximum projection for blending the tiles in overlapping areas, it is not production code. However, the same principle can be used for fusing multi-channel image volumes, such as from Light-Sheet data if the tile chunk intersection calculation is extended to higher-dimensional arrays. Such even larger datasets will benefit even more from leveraging dask, as the processing can be distributed across multiple nodes of a HPC cluster using dask jobqueue.

Also see

Marvin’s lightning talk on multi-view image fusion: 15 minute video available here on YouTube

The GitHub repository MVRegFus that Marvin talks about in the video is available here: m-albert/MVRegFus

The napari-lazy-openslide visualization plugin by Trevor Manz: “An experimental plugin to lazily load multiscale whole-slide tiff images with openslide and dask.”

For further information on alternative approaches to image stitching:

ASHLAR: Alignment by Simultaneous Harmonization of Layer / Adjacency Registration
Microscopy Image Stitching Tool (MIST)
The m2stitch python package by Yohsuke T. Fukai: “Provides robust stitching of tiled microscope images on a regular grid” (based on the MIST algorithm)

Acknowledgements

This computational work was done by Volker Hilsenstein, in conjunction with Marvin Albert. Volker Hilsenstein is a scientific software developer at EMBL in Theodore Alexandrov’s lab with a focus on spatial metabolomics and bio-image analysis.

The sample images were prepared and imaged by Mohammed Shahraz from the Alexandrov lab at EMBL Heidelberg.

Genevieve Buckley and Volker Hilsenstein wrote this blogpost.

CZI EOSS Update

2021-10-20T00:00:00+00:00

Dask was awarded funding last year in round 2 of the CZI Essential Open Source Software grant program. That funding was used to hire Genevieve Buckley to work on Dask with a focus on life sciences. Last month Dask submitted an interim progress report to CZI, covering the period from February to September 2021. That progress update is published verbatim below, to share with the wider Dask community.

Brief summary

The scope of work performed by the Dask fellow includes code contributions, conference presentations and tutorials, community engagement, and outreach including blogposts.

The primary deliverable of this proposal is consistency and the success of neighboring software projects

Project work to date includes:

38 pull requests merged (plus 6 draft pull requests) across 5 different repositories.
3 conferences (presentations and organising of specialist workshops)
1 half day workshop (plus another one upcoming)
Student supervision for Dask’s Google Summer of Code project
9 blogposts (plus 2 drafts for upcoming publication)

Code contributions

Code contributions are not limiteed to the main Dask repository, but also neighbouring software projects which use Dask as well (like the napari software project), including: dask, dask-image, dask-examples, napari, & napari.github.io.

To date, across the five repositories named above the Dask fellow has contributed:

38 pull requests
6 draft pull requests
12 closed pull requests (not merged, discarded in favour of another approach)

The Dask fellow is an official maintainer of the dask-image project, and additional milestones achieved for that project include:

The maintainer team has been grown by one (we welcome Marvin Albert to our ranks)
2 new dask-image releases in 2020

Code contribution highlights

Highlights include:

Bugfixes benefitting the broader community
- dask PR #7391: This PR fixed slicing the output from Dask’s bincount function. The impact of this fix was substantial, as it solved issues filed in four separate projects: scikit-image, dask-ml, xgcm/xhistogram and the cupy dask tests.
Expanded GPU support
- dask PR #6680: This PR provided support for different array types in the *_like array creation functions. Now users can create cupy like Dask arrays for GPU processing, or indeed any other array type (eg: sparse).
- dask-image PR #157: This PR provided GPU support for binary morphological functions in the dask-image project.
Visualization tools benefitting all Dask users
- dask PR #7716: This PR automatically displays the high level graph visualization in the jupyter notebook cell output (somthing already done automatically for low level graphs).
- dask PR #7763: This PR introduced a HTML representation for Dask HighLevelGraph objects. This allows users and developers a much easier way to inspect the structure and status of HighLevelGraphs.
- Further developed on during the Dask Google Summer of Code project, full report available here.
High Level Graphs
- dask PR #7595: This PR introduced a high level graph layer for array overlaps. High level graphs are a tool we can use to optimize Dask’s performance.
- dask PR #7655 (ongoing): This PR introduces a high level graph for Dask array slicing operations.
Memory improvements (ongoing)
- dask PR #8124 (ongoing): This PR investigates improved automatic rechunking strategies for memory problems caused by reshaping Dask arrays.
- dask PR #7950 (ongoing): This PR aims to improve memory and performance of the tensordot function with auto-rechunking of Dask arrays.
- dask PR #7980 (ongoing): This PR aims to fix the unbounded memory use problem in tensordot, reported here.

Conferences

Notable conference events in 2021 included the SciPy conference, the Dask Summit, and VIS2021.

SciPy conference

The Dask fellow presented a talk titled “Scaling Science: leveraging Dask for life sciences” at the 2021 SciPy conference. Full recording available here.

Dask Summit

The Dask fellow organised two workshops at the 2021 Dask Summit:

Dask Down Under (co-organised with Nick Mortimer), and
The Dask life science workshop

Dask Down Under

The scope of Dask Down Under was more like a mini-conference for Australian timezones, rather than a typical workshop. Dask Down Under involved two days of events, covering:

5 talks
2 tutorials
1 panel discussion
1 meet and greet networking event

It was very well recieved by the community. A full report on the Dask Down under events is available here. A YouTube playlist of the Dask Down Under events is available here on the Dask YouTube channel.

Dask life science workshop

The Dask life science workshop involved:

15 pre-recorded lightning talks
3 interactive discussion times (accessible across timezones in Europe, Oceania, and the Americas)
Asynchronous text chat throughout the Dask Summit

A full report on the Dask life science workshop is available here. A YouTube playlist of all the Dask life science lightning talks is available here on the Dask YouTube channel.

VIS2021 symposium

The Dask fellow was an invited panellist at the VIS2021 symposium in February 2021. The “Problem Solver” panel discussion covered practical problems in image analysis and how tools like Dask and napari can help solve them.

Tutorials and workshops

The Dask fellow co-presented a half-day workshop (five hours) at the 2021 Light Microscopy Australia Meeting with Juan Nunez-Iglesias. napari is an open source multidimensional image viewer built using Dask for out-of-core image processing. Workshop content is available at this link: jni/lma-2021-bioimage-analysis-python

Upcoming workshop: The Dask fellow has been invited to deliver a workshop on napari and big data using Dask at an upcoming NEUBIAS Academy. Workshop content is available at this link: GenevieveBuckley/napari-big-data-training

Google Summer of Code

The Dask fellow supervised a Google Summer of Code student in 2021. Martin Durant acted as a secondary supervisor. The project ran over a 3 month period, and involved implementing a number of features to improve visualization of Dask graphs and objects. A full report on the Dask GSOC project is available here.

Blogposts

We set a goal of one blogpost per month, and exceeded it. To date, nine blogposts have been published by the Dask fellow, with another two currently in draft status.

Getting to know the life science community
Dask with PyTorch for large scale image analysis (co-authored with Nick Sofreniew)
Skeleton analysis
Life sciences at the 2021 Dask Summit
The 2021 Dask User Survey is out now
Dask Down Under (co-authored with Nick Mortimer)
Ragged output, how to handle awkward shaped results
High Level Graphs update
Google Summer of Code 2021 - Dask Project

Draft status, will be published soon:

Mosaic Image Fusion (co-authored with Volker Hisenstein)
2021 Dask user survey results

Life sciences at the 2021 Dask Summit

2021-05-24T00:00:00+00:00

The Dask life science workshop ran as part of the 2021 Dask Summit. Lightning talks from this workshop are available here, and you can read on for a summary of the event.

What is the Dask life science workshop?

The Dask life science workshop ran as part of the 2021 Dask Summit. Currently many people in life sciences use Dask, but individual groups are relatively isolated from one another. This workshop gave us an opportunity to learn from each other, as well as opportunities to identify common frustrations and areas for improvement.

The workshop involved:

Pre-recorded lightning talks
Interactive discussion times (accessible across timezones in Europe, Oceania, and the Americas)
Asynchronous text chat throughout the Dask Summit

If I missed it, how can I catch up?

If you missed the Dask Summit, you can catch up on YouTube. There is a playlist of all the life science lightning talks available here.

You can also join our #life-science channel on Slack: Click here for an invitation link.

Who came?

We invited attendees at the life science workshop to do a short Q&A about their work with Dask. This is a small subset of the people who joined us, many people came to the conference and did not do a Q&A.

The responses give us an overview of the diversity of work people in the community are doing. In no particular order, here are some of those Q&As:

Name: Tom White
Timezone: EU/UK
What kind of science do you work on? Statistical genetics
Something you’ve tried (or would like to try) with Dask? Run per-row linear regressions at scale.
What do you want to do next with Dask? Collaborative optimization of a public workflow (GWAS).
Lightning talk: click here

Name: Giovanni Palla
Affiliation: Helmholtz Center Munich
Timezone: Europe
What kind of science do you work on? Computational Biology and Spatial transcriptomics
Something you’ve tried (or would like to try) with Dask? dask-image for image processing.
**What do you want to do next with Dask? Further integration with Squidpy.
**Lightning talk:** click here

Name: Isaac Virshup
Affiliation: University of Melbourne. Open source projects Scanpy and AnnData Timezone: AEST
What kind of science do you work on? Single cell omics data.
Something you’ve tried (or would like to try) with Dask?
I’ve used dask for some nested embarrassingly parallel calculations. Having an intelligent scheduler with good monitoring made this task as easy as it should be, especially compared with multiprocessing or joblib.
What do you want to do next with Dask?
I would love to get AnnData, a container for working with single cell assays integrated with dask. Dataset sizes in this field are constantly increasing, and it would be good to be able to work with the coolest new dataset regardless of available RAM.
Since we rely heavily on sparse arrays, a key step towards this will be getting better sparse array support (CSC and CSR especially) inside dask. After all, it’s not great if our strategy for scaling out requires many times the total memory! As a maintainer, I’m interested in hearing people’s experience with distributing tools that integrate well with dask.
Lightning talk: click here

Name: Anna Kreshuk
Affiliation: European Molecular Biology Laboratory
Timezone: CEST (GMT+2)
What kind of science do you work on? Machine learning for microscopy image analysis.
Something you’ve tried (or would like to try) with Dask? We run a lot of image processing workflows and want to see how Dask can be exploited in this context.

Name: Beth Cimini
Affiliation: Broad Institute
Timezone: US-East
What kind of science do you work on? User friendly image analysis tools for microscopy imaging.
Something you’ve tried (or would like to try) with Dask? Making Dask work in CellProfiler, to make it easy to analyze big images in high throughput!
Lightning talk: click here

Name: Volker Hilsenstein
Affiliation: EMBL / Alexandrov lab
Timezone: Central European Summer Time
What kind of science do you work on? Spatial Metabolomics, combining microscopy and mass spectrometry.
Something I would like to try with dask: fusing large mosaics of individual images or image volumes for which affine transformation into a joint coordinate system are available.

Name: Marvin Albert
Affiliation: University of Zurich
Timezone: UTC/GMT +2
What kind of science do you work on? Life sciences / image analysis
Something you’ve tried (or would like to try) with Dask? What do you want to do next with Dask? Parallelise / reduce the memory footprint of image processing tasks and define workflows that can run on different compute environments.
Lightning talk: click here

Name: Jordao Bragantini
Affiliation: CZ Biohub
Timezone: Pacific Daylight Time (UTC -7)
What kind of science do you work on? Light-sheet microscopy
Something you’ve tried (or would like to try) with Dask? Image processing of very large data.
What do you want to do next with Dask? Implement algorithms for cell segmentation.
Lightning talk: click here

Name: Josh Moore
Affiliation: Open Microscopy Environment (OME)
Timezone: CEST
What kind of science do you work on? Bioimaging (infrastructure for RDM)
Something you’ve tried (or would like to try) with Dask? Accessing large image (Zarr) volumes over HTTP, primarily. What do you want to do next with Dask? Improve pre-fetching for typical usage patterns, possibly integrating multiscale data (i.e. google maps zooming)
Lightning talk: click here

Name: Jackson Maxfield Brown
Timezone: PST
What kind of science do you work in? Cell biology, specifically microscopy and computational biology.
Something you’ve tried (or would like to try) with Dask? Built a metadata aware / backed microscopy imaging reading library that uses Dask to read any size image w/ chunking by metadata dimension information. As well as TB-scale image processing pipelines using Dask + Prefect.
What do you want to do next with Dask? Tighter integration with other libraries. I see cuCim from the RAPIDs team and would love to extend work with them to have a more general “bio-image-spec” so we can all play nicely together.
Lightning talk: click here

Name: Gregory R. Lee
Affiliation: Quansight
Timezone: EST (UTC-5)
What kind of science do you work on? Scientific software development (with a background doing research in magnetic resonance imaging).
Something you’ve tried (or would like to try) with Dask?
In past research work, I used Dask primarily in two scenarios, both on a single workstation:

To achieve multi-threading by processing image blocks in parallel on the CPU (e.g. like in dask-image)
Serial blockwise processing of large volumetric data on the GPU (i.e. CuPy arrays of 10-100 GB in size) to reduce peak memory requirements.

What do you want to do next with Dask?
Audit scikit-image functions to determine which can easily be accelerated using block-wise approaches as in dask-image. Ideally a subset of functions would work directly with dask-arrays as inputs rather than requiring users to learn about Dask’s map_overlap, etc. to use this feature.
Lightning talk: click here

What’s next?

Dask is now considering holding “office hours” for the life science community. If we can find enough maintainers able to host one-hour Q&A sessions, then we’ll trial this for a short period of time.

Skeleton analysis

2021-05-07T00:00:00+00:00

In this blogpost, we show how to modify a skeleton network analysis with Dask to work with constrained RAM (eg: on your laptop). This makes it more accessible: it can run on a small laptop, instead of requiring access to a supercomputing cluster. Example code is also provided here.

Skeleton structures are everywhere
The scientific problem
The compute problem
Our approach
Results
Limitations
Problems encountered
How we solved them
What’s next
How you can help

Skeleton structures are everywhere

Lots of biological structures have a skeleton or network-like shape. We see these in all kinds of places, including:

blood vessel branching
the branching of airways
neuron networks in the brain
the root structure of plants
the capillaries in leaves
… and many more

Analysing the structure of these skeletons can give us important information about the biology of that system.

The scientific problem

For this bogpost, we will look at the blood vessels inside of a lung. This data was shared with us by Marcus Kitchen, Andrew Stainsby, and their team of collaborators.

This research group focusses on lung development. We want to compare the blood vessels in a healthy lung, against a lung from a hernia model. In the hernia model the lung is underdeveloped, squashed, and smaller.

The compute problem

These image volumes have a shape of roughtly 1000x1000x1000 pixels. That doesn’t seem huge but given the high RAM consumption involved in processing the analysis, it crashes when running on a laptop.

If you’re running out of RAM, there are two possible appoaches:

Get more RAM. Run things on a bigger computer, or move things to a supercomputing cluster. This has the advantage that you don’t need to rewrite your code, but it does require access to more powerful computer hardware.
Manage the RAM you’ve got. Dask is good for this. If we use Dask, and some reasonable chunking of our arrays, we can manage things so that we never hit the RAM ceiling and crash. This has the advantage that you don’t need to buy more computer hardware, but it will require re-writing some code.

Our approach

We took the second approach, using Dask so we can run our analysis on a small laptop with constrained RAM without crashing. This makes it more accessible, to more people.

All the image pre-processing steps will be done with dask-image, and the skeletonize function of scikit-image.

We use skan as the backbone of our analysis pipeline. skan is a library for skeleton image analysis. Given a skeleton image, it can describe statistics of the branches. To make it fast, the library is accelerated with numba (if you’re curious, you can hear more about that in this talk and its related notebook).

There is an example notebook containing the full details of the skeleton analysis available here. You can read on to hear just the highlights.

Results

The statistics from the blood vessel branches in the healthy and herniated lung shows clear differences between the two.

Most striking is the difference in the number of blood vessel branches. The herniated lung has less than 40% of the number of blood vessel branches in the healthy lung.

There are also quantitative differences in the sizes of the blood vessels. Here is a violin plot showing the distribution of the distances between the start and end points of each blood vessel branch. We can see that overall the blood vessel branches start and end closer together in the herniated lung. This is consistent with what we might expect, since the healthy lung is more well developed than the lung from the hernia model and the hernia has compressed that lung into a smaller overall space.

EDIT: This blogpost previously described the euclidean distance violin plot as measuring the thickness of the blood vessels. This is incorrect, and the mistake was not caught in the review process before publication. This post has been updated to correctly describe the euclidean-distance measuremet as the distance between the start and end of branches, as if you pulled a string taught between those points. An alternative measurement, branch-length describes the total branch length, including any winding twists and turns.

Limitations

We rely on one big assumption: once skeletonized the reduced non-zero pixel data will fit into memory. While this holds true for datasets of this size (the cropped rabbit lung datasets are roughly 1000 x 1000 x 1000 pixels), it may not hold true for much larger data.

Dask computation is also triggered at a few points through our prototype workflow. Ideally all computation would be delayed until the very final stage.

Problems encountered

This project was originally intended to be a quick & easy one. Famous last words!

What I wanted to do was to put the image data in a Dask array, and then use the map_overlap function to do the image filtering, thresholding, skeletonizing, and skeleton analysis. What I soon found was that although the image filtering, thresholding, and skeletonization worked well, the skeleton analysis step had some problems:

Dask’s map_overlap function doesn’t handle ragged or non-uniformly shaped results from different image chunks very well, and…
Internal function in the skan library were written in a way that was incompatible with distributed computation.

How we solved them

Problem 1: The skeletonize function from scikit-image crashes due to lack of RAM

The skeletonize function of scikit-image is very memory intensive, and was crashing on a laptop with 16GB RAM.

We solved this by:

Putting our image data into a Dask array with dask-image imread,
Rechunking the Dask array. We need to change the chunk shapes from 2D slices to small cuboid volumes, so the next step in the computation is efficient. We can choose the overall size of the chunks so that we can stay under the memory threshold needed for skeletonize.
Finally, we run the skeletonize function on the Dask array chunks using the map_overlap function. By limiting the size of the array chunks, we stay under our memory threshold!

Problem 2: Ragged or non-uniform output from Dask array chunks

The skeleton analysis functions will return results with ragged or non-uniform length for each image chunk. This is unsurpising, because different chunks will have different numbers of non-zero pixels in our skeleton shape.

When working with Dask arrays, there are two very commonly used functions: map_blocks and map_overlap. Here’s what happens when we try a function with ragged outputs with map_blocks versus map_overlap.

import dask.array as da
import numpy as np

x = da.ones((100, 10), chunks=(10, 10))

def foo(a):  # our dummy analysis function
    random_length = np.random.randint(1, 7)
    return np.arange(random_length)

With map_blocks, everything works well:

result = da.map_blocks(foo, x, drop_axis=1)
result.compute()  # this works well

But if we need some overlap for function foo to work correctly, then we run into problems:

result = da.map_overlap(foo, x, depth=1, drop_axis=1)
result.compute()  # incorrect results

Here, the first and last element of the results from foo are trimmed off before the results are concatenated, which we don’t want! Setting the keyword argument trim=False would help avoid this problem, except then we get an error:

result = da.map_overlap(foo, x, depth=1, trim=False, drop_axis=1)
result.compute()  # ValueError

Unfortunately for us, it’s really important to have a 1 pixel overlap in our array chunks, so that we can tell if a skeleton branch is ending or continuing on into the next chunk.

There’s some complexity in the way map_overlap results are concatenated back together so rather than diving into that, a more straightforward solution is to use Dask delayed instead. Chris Roat shows a nice example of how we can use Dask delayed in a list comprehension that is then concatenated with Dask (link to original discussion).

import numpy as np
import pandas as pd

import dask
import dask.array as da
import dask.dataframe as dd

x = da.ones((20, 10), chunks=(10, 10))

@dask.delayed
def foo(a):
    size = np.random.randint(1,10)  # Make each dataframe a different size
    return pd.DataFrame({'x': np.arange(size),
                         'y': np.arange(10, 10+size)})

meta = dd.utils.make_meta([('x', np.int64), ('y', np.int64)])
blocks = x.to_delayed().ravel()  # no overlap
results = [dd.from_delayed(foo(b), meta=meta) for b in blocks]
ddf = dd.concat(results)
ddf.compute()

Warning: It’s very important to pass in a meta keyword argument to the function from_delayed. Without it, things will be extremely inefficient!

If the meta keyword argument is not given, Dask will try and work out what it should be. Ordinarily that might be a good thing, but inside a list comprehension that means those tasks are computed slowly and sequentially before the main computation even begins, which is horribly inefficient. Since we know ahead of time what kinds of results we expect from our analysis function (we just don’t know the length of each set of results), we can use the utils.make_meta function to help us here.

Problem 3: Grabbing the image chunks with an overlap

Now that we’re using Dask delayed to piece together our skeleton analysis results, it’s up to us to handle the array chunks overlap ourselves.

We’ll do that by modifying Dask’s dask.array.core.slices_from_chunks function, into something that will be able to handle an overlap. Some special handling is required at the boundaries of the Dask array, so that we don’t try to slice past the edge of the array.

Here’s what that looks like (gist):

from itertools import product
from dask.array.slicing import cached_cumsum

def slices_from_chunks_overlap(chunks, array_shape, depth=1):
    cumdims = [cached_cumsum(bds, initial_zero=True) for bds in chunks]

    slices = []
    for starts, shapes in zip(cumdims, chunks):
        inner_slices = []
        for s, dim, maxshape in zip(starts, shapes, array_shape):
            slice_start = s
            slice_stop = s + dim
            if slice_start > 0:
                slice_start -= depth
            if slice_stop >= maxshape:
                slice_stop += depth
            inner_slices.append(slice(slice_start, slice_stop))
        slices.append(inner_slices)

    return list(product(*slices))

Now that we can slice an image chunk plus an extra pixel of overlap, all we need is a way to do that for all the chunks in an array. Drawing inspiration from this block iteration we make a similar iterator.

block_iter = zip(
    np.ndindex(*image.numblocks),
    map(functools.partial(operator.getitem, image),
        slices_from_chunks_overlap(image.chunks, image.shape, depth=1))
)

meta = dd.utils.make_meta([('row', np.int64), ('col', np.int64), ('data', np.float64)])
intermediate_results = [dd.from_delayed(skeleton_graph_func(block), meta=meta) for _, block in block_iter]
results = dd.concat(intermediate_results)
results = results.drop_duplicates()  # we need to drop duplicates because it counts pixels in the overlapping region twice

With these results, we’re able to create the sparse skeleton graph.

Problem 4: Summary statistics with skan

Skeleton branch statistics can be calculate with the skan summarize function. The problem here is that the function expects a Skeleton object instance, but initializing a Skeleton object calls methods that are not compatible for distributed analysis.

We’ll solve this problem by first initializing a Skeleton object instance with a tiny dummy dataset, then overwriting the attributes of the skeleton object with our real results. This is a hack, but it lets us achieve our goal: summary branch statistics for our large dataset.

First we make a Skeleton object instance with dummy data:

from skan._testdata import skeleton0

skeleton_object = Skeleton(skeleton0)  # initialize with dummy data

Then we overwrite the attributes with the previously calculated results:

skeleton_object.skeleton_image = ...
skeleton_object.graph = ...
skeleton_object.coordinates
skeleton_object.degrees = ...
skeleton_object.distances = ...
...

Then finally we can calculate the summary branch statistics:

from skan import summarize

statistics = summarize(skel_obj)
statistics.head()

	skeleton-id	node-id-src	node-id-dst	branch-distance	branch-type	mean-pixel-value	stdev-pixel-value	image-coord-src-0	image-coord-src-1	image-coord-src-2	image-coord-dst-0	image-coord-dst-1	image-coord-dst-2	coord-src-0	coord-src-1	coord-src-2	coord-dst-0	coord-dst-1	coord-dst-2	euclidean-distance
0	1	1	2	1	2	0.474584	0.00262514	22	400	595	22	400	596	22	400	595	22	400	596	1
1	2	3	9	8.19615	2	0.464662	0.00299629	37	400	622	43	392	590	37	400	622	43	392	590	33.5261
2	3	10	11	1	2	0.483393	0.00771038	49	391	589	50	391	589	49	391	589	50	391	589	1
3	5	13	19	6.82843	2	0.464325	0.0139064	52	389	588	55	385	588	52	389	588	55	385	588	5
4	7	21	23	2	2	0.45862	0.0104024	57	382	587	58	380	586	57	382	587	58	380	586	2.44949

statistics.describe()

	skeleton-id	node-id-src	node-id-dst	branch-distance	branch-type	mean-pixel-value	stdev-pixel-value	image-coord-src-0	image-coord-src-1	image-coord-src-2	image-coord-dst-0	image-coord-dst-1	image-coord-dst-2	coord-src-0	coord-src-1	coord-src-2	coord-dst-0	coord-dst-1	coord-dst-2	euclidean-distance
count	1095	1095	1095	1095	1095	1095	1095	1095	1095	1095	1095	1095	1095	1095	1095	1095	1095	1095	1095	1095
mean	2089.38	11520.1	11608.6	22.9079	2.00091	0.663422	0.0418607	591.939	430.303	377.409	594.325	436.596	373.419	591.939	430.303	377.409	594.325	436.596	373.419	190.13
std	636.377	6057.61	6061.18	24.2646	0.0302199	0.242828	0.0559064	174.04	194.499	97.0219	173.353	188.708	96.8276	174.04	194.499	97.0219	173.353	188.708	96.8276	151.171
min	1	1	2	1	2	0.414659	6.79493e-06	22	39	116	22	39	114	22	39	116	22	39	114	0
25%	1586	6215.5	6429.5	1.73205	2	0.482	0.00710439	468.5	278.5	313	475	299.5	307	468.5	278.5	313	475	299.5	307	72.6946
50%	2431	11977	12010	16.6814	2	0.552626	0.0189069	626	405	388	627	410	381	626	405	388	627	410	381	161.059
75%	2542.5	16526.5	16583	35.0433	2	0.768359	0.0528814	732	579	434	734	590	432	732	579	434	734	590	432	265.948
max	8034	26820	26822	197.147	3	1.29687	0.357193	976	833	622	976	841	606	976	833	622	976	841	606	737.835

Success!

We’ve achieved distributed skeleton analysis with Dask. You can see the example notebook containing the full details of the skeleton analysis here.

What’s next?

A good next step is modifing the skan library code so that it directly supports distributed skeleton analysis.

How you can help

If you’d like to get involved, there are a couple of options:

Try a similar analysis on your own data. The notebook with the full example code is available here. You can share or ask questions in the Dask slack or on twitter.
Help add support for distributed skeleton analysis to skan. Head on over to the skan issues page and leave a comment if you’d like to join in.

Dask Working Notes - Posts tagged life science

Reflections on one year as the Dask life science fellow

Contents

Progress update

Personal reflections

Highlights from this year

What worked well

What didn’t work so well

What’s next for Genevieve?

What’s next in Dask?

Mosaic Image Fusion

The problem

Image mosaicing in microscopy

The solution

Results

Code

What’s next?

Also see

Acknowledgements

CZI EOSS Update

Brief summary

Code contributions

Code contribution highlights

Conferences

SciPy conference

Dask Summit

Dask Down Under

Dask life science workshop

VIS2021 symposium

Tutorials and workshops

Google Summer of Code

Blogposts

Life sciences at the 2021 Dask Summit

What is the Dask life science workshop?

If I missed it, how can I catch up?

Who came?

What’s next?

Skeleton analysis

Contents

Skeleton structures are everywhere

The scientific problem

The compute problem

Our approach

Results

Limitations

Problems encountered

How we solved them

Problem 1: The skeletonize function from scikit-image crashes due to lack of RAM

Problem 2: Ragged or non-uniform output from Dask array chunks

Problem 3: Grabbing the image chunks with an overlap

Problem 4: Summary statistics with skan

What’s next?

How you can help