_Note: this content is pulled from
[stsievert/dask-hyperband-comparison], or makes slight modifications._
[stsievert/dask-hyperband-comparison]: https://github.com/stsievert/dask-hyperband-comparison
Let's build a fully connected neural net with 24 neurons for classification:
```python
>>> from sklearn.neural_network import MLPClassifier
>>> model = MLPClassifier()
```
Building the neural net with PyTorch is also possible[^skorch] (and what I used in development).
[^skorch]: through the Scikit-learn API wrapper [skorch]
This neural net's behavior is dictated by 6 hyperparameters. Only one controls
the model of the optimal architecture (`hidden_layer_sizes`, the number of
neurons in each layer). The rest control finding the best model of that
architecture. Details on the hyperparameters are in the
_[Appendix](#appendix)_.
```python
>>> params = ... # details in appendix
>>> params.keys()
dict_keys(['hidden_layer_sizes', 'alpha', 'batch_size', 'learning_rate'
'learning_rate_init', 'power_t', 'momentum'])
>>> params["hidden_layer_sizes"] # always 24 neurons
[(24, ), (12, 12), (6, 6, 6, 6), (4, 4, 4, 4, 4, 4), (12, 6, 3, 3)]
```
I choose these hyperparameters to have a complex search space that mimics the
searches performed for most neural networks. These searches typically involve
hyperparameters like "dropout", "learning rate", "momentum" and "weight
decay".[^user-facing]
End users don't care hyperparameters like these; they don't change the
model architecture, only finding the best model of a particular architecture.
[^user-facing]: There's less tuning for adaptive step size methods like [Adam] or [Adagrad], but they might under-perform on the test data (see "[The Marginal Value of Adaptive Gradient Methods for Machine Learning][adamarginal]")
How can high performing hyperparameter values be found quickly?
[adamarginal]: https://arxiv.org/abs/1705.08292
[adam]: https://arxiv.org/abs/1412.6980
[adagrad]: http://jmlr.org/papers/v12/duchi11a.html
## Finding the best parameters
First, let's look at the parameters required for Dask-ML's implementation
of Hyperband (which is in the class `HyperbandSearchCV`).
### Hyperband parameters: rule-of-thumb
`HyperbandSearchCV` has two inputs:
1. `max_iter`, which determines how many times to call `partial_fit`
2. the chunk size of the Dask array, which determines how many data each
`partial_fit` call receives.
These fall out pretty naturally once it's known how long to train the best
model and very approximately how many parameters to sample:
```python
n_examples = 50 * len(X_train) # 50 passes through dataset for best model
n_params = 299 # sample about 300 parameters
# inputs to hyperband
max_iter = n_params
chunk_size = n_examples // n_params
```
The inputs to this rule-of-thumb are exactly what the user cares about:
- a measure of how complex the search space is (via `n_params`)
- how long to train the best model (via `n_examples`)
Notably, there's no tradeoff between `n_examples` and `n_params` like with
Scikit-learn's `RandomizedSearchCV` because `n_examples` is only for _some_
models, not for _all_ models. There's more details on this
rule-of-thumb in the "Notes" section of the [`HyperbandSearchCV`
docs][hyperband-docstring].
[hyperband-docstring]: https://ml.dask.org/modules/generated/dask_ml.model_selection.HyperbandSearchCV.html#dask_ml.model_selection.HyperbandSearchCV
With these inputs a `HyperbandSearchCV` object can easily be created.
### Finding the best performing hyperparameters
This model selection algorithm Hyperband is implemented in the class
`HyperbandSearchCV`. Let's create an instance of that class:
```python
>>> from dask_ml.model_selection import HyperbandSearchCV
>>>
>>> search = HyperbandSearchCV(
... est, params, max_iter=max_iter, aggressiveness=4
... )
```
`aggressiveness` defaults to 3. `aggressiveness=4` is chosen because this is an
_initial_ search; I know nothing about how this search space. Then, this search
should be more aggressive in culling off bad models.
Hyperband hides some details from the user (which enables the mathematical
guarantees), specifically the details on the amount of training and
the number of models created. These details are available in the `metadata`
attribute:
```python
>>> search.metadata["n_models"]
378
>>> search.metadata["partial_fit_calls"]
5721
```
Now that we have some idea on how long the computation will take, let's ask it
to find the best set of hyperparameters:
```python
>>> from dask_ml.model_selection import train_test_split
>>> X_train, y_train, X_test, y_test = train_test_split(X, y)
>>>
>>> X_train = X_train.rechunk(chunk_size)
>>> y_train = y_train.rechunk(chunk_size)
>>>
>>> search.fit(X_train, y_train)
```
The dashboard will be active during this time[^dashboard]:
[^dashboard]: But it probably won't be this fast: the video is sped up by a factor of 3.
How well do these hyperparameters perform? ```python >>> search.best_score_ 0.9019221418447483 ``` `HyperbandSearchCV` mirrors Scikit-learn's API for [RandomizedSearchCV], so it has access to all the expected attributes and methods: ```python >>> search.best_params_ {"batch_size": 64, "hidden_layer_sizes": [6, 6, 6, 6], ...} >>> search.score(X_test, y_test) 0.8989070100111217 >>> search.best_model_ MLPClassifier(...) ``` Details on the attributes and methods are in the [HyperbandSearchCV documentation][hyperband-docs]. [hyperband-docs]: https://ml.dask.org/modules/generated/dask_ml.model_selection.HyperbandSearchCV.html ## Performance I ran this 200 times on my personal laptop with 4 cores. Let's look at the distribution of final validation scores:
The "passive" comparison is really `RandomizedSearchCV` configured so it takes
an equal amount of work as `HyperbandSearchCV`. Let's see how this does over
time:
This graph shows the mean score over the 200 runs with the solid line, and the
shaded region represents the [interquartile range]. The dotted green
line indicates the data required to train 4 models to completion.
"Passes through the dataset" is a good proxy
for "time to solution" because there are only 4 workers.
[interquartile range]: https://en.wikipedia.org/wiki/Interquartile_range
This graph shows that `HyperbandSearchCV` will find parameters at least 3 times
quicker than `RandomizedSearchCV`.
### Dask opportunities
What opportunities does combining Hyperband and Dask create?
`HyperbandSearchCV` has a lot of internal parallelism and Dask is an advanced task
scheduler.
The most obvious opportunity involves job prioritization. Hyperband fits many
models in parallel and Dask might not have that
workers available. This means some jobs have to wait for other jobs
to finish. Of course, Dask can prioritize jobs[^prior] and choose which models
to fit first.
[^prior]: See Dask's documentation on [Prioritizing Work](https://distributed.dask.org/en/latest/priority.html)
[dask-prior]: https://distributed.dask.org/en/latest/priority.html
Let's assign the priority for fitting a certain model to be the model's most
recent score. How does this prioritization scheme influence the score? Let's
compare the prioritization schemes in
a single run of the 200 above:
These two lines are the same in every way except for
the prioritization scheme.
This graph compares the "high scores" prioritization scheme and the Dask's
default prioritization scheme ("fifo").
This graph is certainly helped by the fact that is run with only 4 workers.
Job priority does not matter if every job can be run right away (there's
nothing to assign priority too!).
### Amenability to parallelism
How does Hyperband scale with the number of workers?
I ran another separate experiment to measure. This experiment is described more in the [corresponding
paper][scipy19], but the relevant difference is that a [PyTorch] neural network is used
through [skorch] instead of Scikit-learn's MLPClassifier.
I ran the _same_ experiment with a different number of Dask
workers.[^same] Here's how `HyperbandSearchCV` scales:
[skorch]: https://skorch.readthedocs.io/en/stable/
[pytorch]: https://pytorch.org/
Training one model to completion requires 243 seconds (which is marked by the
white line). This is a comparison with `patience`, which stops training models
if their scores aren't increasing enough. Functionally, this is very useful
because the user might accidentally specify `n_examples` to be too large.
It looks like the speedups start to saturate somewhere
between 16 and 24 workers, at least for this example.
Of course, `patience` doesn't work as well for a large number of
workers.[^scale-worker]
[^scale-worker]: There's no time benefit to stopping jobs early if there are infinite workers; there's never a queue of jobs waiting to be run
[^same]: Everything is the same between different runs: the hyperparameters sampled, the model's internal random state, the data passed for fitting. Only the number of workers varies.
## Future work
There's some ongoing pull requests to improve `HyperbandSearchCV`. The most
significant of these involves tweaking some Hyperband internals so `HyperbandSearchCV`
works better with initial or very exploratory searches ([dask/dask-ml #532]).
[dask/dask-ml #532]: https://github.com/dask/dask-ml/pull/532
The biggest improvement I see is treating _dataset size_ as the scarce resource
that needs to be preserved instead of _training time_. This would allow
Hyperband to work with any model, instead of only models that implement
`partial_fit`.
Serialization is an important part of the distributed Hyperband implementation
in `HyperbandSearchCV`. Scikit-learn and PyTorch can easily handle this because
they support the Pickle protocol[^pickle-post], but
Keras/Tensorflow/MXNet present challenges. The use of `HyperbandSearchCV` could
be increased by resolving this issue.
[^pickle-post]: "[Pickle isn't slow, it's a protocol][matt-pickle]" by Matthew Rocklin
[matt-pickle]: http://matthewrocklin.com/blog/work/2018/07/23/protocols-pickle
## Appendix
I choose to tune 7 hyperparameters, which are
- `hidden_layer_sizes`, which controls the activation function used at each
neuron
- `alpha`, which controls the amount of regularization
More hyperparameters control finding the best neural network:
- `batch_size`, which controls the number of examples the `optimizer` uses to
approximate the gradient
- `learning_rate`, `learning_rate_init`, `power_t`, which control some basic
hyperparameters for the SGD optimizer I'll be using
- `momentum`, a more advanced hyperparameter for SGD with Nesterov's momentum.