Galaxy Zoo has been running since 2007. Volunteers look at a survey image and walk a tree of questions about it. Smooth or featured? Is there a spiral? How many arms? By the time the crowd is done with a galaxy, its answers have been collapsed into 37 numbers between 0 and 1, each one a weighted share of what people saw.

I spent six weeks this spring, April 13 through May 27, trying to get a convolutional network to reproduce those 37 numbers from raw pixels on our university’s GPU cluster. The wreckage is all still in the repo: twenty-four training logs, TensorBoard folders numbered v1 through v6 plus a v5.5 wedged in between, one saved checkpoint. Also preserved in there, unfortunately, is the fact that most of April went to training the wrong kind of model.

Every run shared the same pipeline, drawn below. The two ends stayed fixed the whole spring. What changed was the middle, and the loss.

A 424x424x3 galaxy image flowing through convolutional and pooling blocks into a 37-length output vector An input image block on the left, labeled 424 by 424 by 3, passes through a sequence of shrinking convolution and pooling blocks, then into a dense head that emits a vertical strip of 37 numbers. 424x424x3 input image conv + pool conv + pool conv + pool dense regression head 37 values
The whole pipeline on one substrate: a 424x424x3 image is compressed through stacked conv and pool blocks, then a dense head emits a 37-length morphology vector. The depth of the conv stack is what I vary across experiments; the input and the 37-value head stay fixed.

One jpeg in, 37 numbers out

The images come from the Sloan Digital Sky Survey, and the labels from Galaxy Zoo, packaged for the Kaggle Galaxy Challenge as 61,578 training jpegs plus a training_solutions_rev1.csv keyed by GalaxyID. To quote the SDSS site:

The Sloan Digital Sky Survey has created the most detailed three-dimensional maps of the Universe ever made, with deep multi-color images of one third of the sky, and spectra for more than three million astronomical objects.

Each label row is the crowd’s verdict on one galaxy, produced by volunteers walking this decision tree:

Galaxy Zoo decision tree of classification questions
The Galaxy Zoo decision tree, from Galaxy Zoo 2 (Willett et al. 2013). Each volunteer walks these branching questions for an image, and the weighted answers become the 37-value target the network has to predict.
A grid of sample galaxy images from the dataset
Sample galaxies from the dataset, SDSS image cutouts. Each is a single 424x424x3 jpeg that has to come out the other end as 37 numbers.

Every jpeg is 424x424x3. None of my runs consumed it at that size: the loader goes through cv2.resize, and over the six weeks the working resolution wandered from 32x32 sanity checks up to 256x256. The one preprocessing step every run shares is a rescale by 1/255, done inside the model in a Lambda layer, so the files on disk stay untouched and anything that loads the checkpoint gets the scaling for free. Beyond that the array goes in raw.

A galaxy image as a 424 by 424 by 3 numpy array of integer pixel values Three grids are stacked front to back for the red, green, and blue channels. The front grid shows sample integer pixel values from 0 to 255 with ellipses standing in for the full 424 by 424 extent. Brackets label the height, the width, and the depth of three channels. 324158… 276091… 4488150… ………… R G B 424 424 3 channels (R, G, B)
The same galaxy as a numpy array: a 424x424x3 block of uint8 values, three color channels stacked, each pixel from 0 to 255. From here on, every layer is a transform on this block of numbers.

The runs themselves happened on BlueHive, the university cluster, through SLURM: the gpu partition, two GPUs and 55 GB of memory per job, module load tensorflow, and a time limit a shade under five days. Everything is Keras on the TensorFlow backend. The 55 GB is a real requirement, because the scripts load the whole training set into one array before fitting starts, and at 256x256 that int16 array is 24 GB on its own. The second GPU is a decoration. Training sits pinned to one device; multi-GPU Keras in 2017 is a hand-rolled affair, and I never rolled it.

April: the classification dead end

My first models were classifiers, because that is what a stack of image tutorials trains you to reach for. Take the 37-value vector, argmax it into a single winning answer, one-hot the winner, minimize categorical crossentropy. The model.py from that era is still in the repo, ending in a sigmoid 1x1 convolution head.

The logs say exactly how that went. One convnet jumped to 0.61 validation accuracy almost immediately and stayed there for 64 epochs. The next climbed from 0.59 to 0.75 over 33 epochs; another took 57 epochs to reach the same 0.75 and stopped. The longest run of the era recast the answers as separate categories and needed 190 epochs to crawl from under one percent to 0.47. For the three argmax runs the crossentropy column barely breathed, easing from 14.4 to 13.8 while accuracy climbed sixteen points.

Validation accuracy and loss curves for four classification runs, two plateauing near 0.75
The four classification-era runs, replotted from the committed training logs. Two runs hit the same ceiling near 0.75 accuracy and flatline there; a third stalls at 0.61.

When three architectures stall at the same number, the ceiling probably belongs to the question rather than the network. I say probably because the head of that era was its own mess, a sigmoid output trained under categorical crossentropy, and that pairing alone will pin a loss column near its clipping ceiling no matter what accuracy does. Still, a cleaner classifier would only have answered the wrong question more sharply. Argmax is first-past-the-post voting. A galaxy the crowd split 51-49 lands in the same class as one they called 99-1. Volunteers disagree about ambiguous smudges of light, that disagreement is graded and it is informative, and it is the entire reason the label is 37 numbers instead of one. My argmax threw it away before training even started.

The head the data wanted

So the target stays a vector and the head becomes a regression. All 37 outputs stay, the one-hot goes, and the loss is mean squared error against the vote fractions themselves. Instead of calling the election I am now predicting the poll. This also happens to be exactly what the challenge scores, RMSE over all 37 values. It is on the evaluation page, which I evidently did not read carefully enough in April.

There is a trap sitting right next to that fix, though. Reflex says put a softmax on a 37-way output. Softmax renormalizes the vector to sum to 1, and these vectors do not sum to 1, because Galaxy Zoo scales each question’s answers by the share of volunteers who actually reached that question in the tree. The structured thing to do would be a per-question softmax rescaled by the parent weight. I never built it. My heads stay unnormalized, with a relu on the convnet’s output so it cannot predict a negative vote share the way a plain linear unit sometimes will.

The stack between the two ends

The middle of the pipeline is the standard alternation, and the committed convnet spells out one concrete instance of it: seven blocks, filter counts stepping 32, 64, 128, 256, 512, 768, 1024, kernels of 4x4 in the first block and 2x2 after, all same-padded so the grid size survives the convolution. Each block ends in a relu, a max pool that halves the grid (4x4 in the first block), and a Dropout(0.2). After the last block a dense layer of 1024 units feeds the 37-value head.

The convolutional layers are the part I actually care about. In a classic astronomy pipeline someone hand-designs the features: concentration, asymmetry, the Gini coefficient of the light distribution (Conselice 2003; Lotz, Primack and Madau 2004). Here the 4x4 and 2x2 kernels are fit against the vote target, and whatever transforms best predict the crowd’s answers are the features. Nobody tells the network what a spiral arm looks like, and by the end nothing in the weights is guaranteed to correspond to one.

Batch normalization between the blocks

In the convnet script, a BatchNormalization layer opens every one of the seven blocks. The reason comes from Ioffe and Szegedy’s 2015 paper: as earlier layers update, the distribution of activations arriving at each later layer keeps sliding, a moving target they name internal covariate shift, and it is what forces small learning rates. Their fix normalizes each channel’s activations across the current mini-batch, then lets a learned affine transform undo exactly as much of that as the layer finds useful:

\[X_\text{out} = \gamma \cdot \frac{X_\text{in} - \mu_X}{\sigma_X} + \beta\]

Both statistics, $\mu_X$ and $\sigma_X$, are channelwise, computed over the mini-batch. Ioffe and Szegedy report reaching their ImageNet baseline’s accuracy in roughly 14 times fewer steps. Did batch norm buy me anything comparable on galaxies? I never ran the ablation, so I cannot say. The layers went in on the paper’s word and stayed in because the deeper regression runs trained without any of the learning-rate babysitting April had needed.

An offset activation distribution being re-centered to mean zero and unit variance On the left, a lopsided bump sits to one side of a baseline axis. An arrow points right to a symmetric bump centered on the axis, labeled mu equals zero and sigma equals one. Schematic curves with no numeric axis ticks. offset, wide mu = 0, sigma = 1
Batch norm takes the drifting, offset activation distribution on the left and re-centers it to mean zero and unit variance on the right before the learned affine transform restores whatever scale the next layer actually needs.

Dropout, and where mine actually went

Dropout, from Srivastava et al. in 2014, zeroes a random fraction of a layer’s units at every training step, so no unit gets to be indispensable; at test time everything stays on, scaled by the keep-probability p. Going in, my guess was simple. Overfitting bites where the parameters are, the parameters are densest in the 1024-unit layer feeding the head, therefore the dropout belongs there.

The committed script disagrees with my guess. Dropout sits at 0.2 after every conv block and nowhere on the dense layer at all. I do not have a clean record of why it migrated; the configuration that survived is the one that spread small dropout through the stack. Whether the head would have overfit without it is an ablation the record does not contain.

A fully connected network before and after dropout Panel a shows a small fully connected network with an input layer, two hidden layers, and an output, all units active and fully wired. Panel b shows the same network with a random subset of hidden units faded out and their edges removed, illustrating dropout with keep-probability p. (a) standard network (b) with dropout faded units are dropped this step; a different subset is kept with probability p each time
(a) the full network. (b) the same network with a random subset of hidden units zeroed and their edges gone for this training step, so no single unit can become indispensable. Layout after Figure 1 of Srivastava et al. 2014.

Six weeks of regression runs

With the head fixed, May belonged to architecture sweeps. The plain convnet’s best regression run flattened at a validation MSE of 0.031 and would not go lower. The jump came from swapping the backbone for a DenseNet, a checked-in Keras implementation of last year’s densely connected networks, configured with depth 64, growth rate 10, bottleneck layers, and a 0.5 compression at each transition. The DenseNet runs also picked up augmentation the earlier ones lacked, rotation_range=180 with both flips, free here because a galaxy has no right way up. Both changes landed in the same step, so the logs cannot split the 0.031-to-0.015 drop between them; a DenseNet run without augmentation is simply not in the record. Batches stayed small, 25 for the DenseNets against the convnet’s 50, with Adam starting at 1e-5 and early stopping at a patience of 25.

The record of the sweep, replotted straight from the loss CSVs:

Validation MSE curves for six regression runs on a log scale, DenseNet reaching 0.0142
Validation MSE for the regression-era runs, log scale. The plain convnet and two DenseNet configurations flatten near 0.03 to 0.045; densenet 01 descends to 0.015 over 54 epochs, and its fine-tune holds 0.0142. The dashed grey line at the bottom comes from a target-loading bug.

The chart compresses a month of runs, so a few of its stories deserve prose. The run that was both smallest and lowest-resolution, 32x32 on a thousand samples, never left an MSE of 0.37; two variables moved at once there too, but it sent me up the resolution ladder first. The fast DenseNet variant at 64x64 settled at 0.045; 128x128 was the one genuine divergence, ending its eighth epoch with a validation loss above 0.6, almost four times where it began, and I still do not know why. A 256x256 sibling bottomed out at 0.025 and then spiked to 0.53 on its final epoch, which is the kind of exit that makes you grateful for ModelCheckpoint(save_best_only=True). One run’s very first epoch logged a training loss just over 950,000 before settling under 0.07 by epoch seven, a reminder of what an unlucky initialization does to squared error.

The run that mattered, densenet 01, walked down from 0.16 to 0.0152 over 54 epochs. Its continuation picked up from the checkpoint at 0.0142, and the log of that fine-tune is mostly a portrait of diminishing returns: ReduceLROnPlateau cut the learning rate from 1e-6 to 1e-9 in three steps across fourteen epochs, and validation MSE moved from 0.01421 to 0.01420.

One caveat belongs next to those digits rather than buried at the end. The DenseNet script’s validation generator draws from the same arrays its training generator does. The convnet script had done this correctly, holding out a validation_split=0.1; the split never made it across the rewrite. Every DenseNet validation number in this chart is therefore optimistic, closer to a training fit than a held-out error.

And the dashed grey line? That was briefly the most exciting number of the spring: a convnet variant reporting 0.0027 from its first epoch, five times better than anything else ever managed. It is fake. The committed script allocates the target array as int16, so every vote fraction below 1.0 truncated to zero as the array filled, and the model earned its score by learning to predict near-zero against targets that mostly were zero. A first epoch that starts where other runs end is not a breakthrough; it is a dtype.

Where it landed

Six weeks reduce to one checkpoint. densenet_02_best.h5, validation MSE 0.0142, which on the vote fractions is an RMSE of 0.119. For scale I have two anchors. Kaggle’s central-pixel benchmark, basically averaging the votes of training galaxies with similar core colors, scored 0.16194. Sander Dieleman won the 2014 challenge at a private-leaderboard RMSE of 0.07492. My checkpoint sits between those two, knowing more about a galaxy than its central colors and a great deal less than the winner did.

All of it is bounded by the validation leak above, so porting the convnet’s validation_split back over is the first thing on the list. And 0.119 is a validation number in any case; lining it up against 0.07492 honestly would take the test-set submission I have not made. Here is the narrow thing the six weeks actually bought. The same pixels that spent all of April refusing to classify past 0.75 will happily regress a 37-value crowd vote, as soon as the head stops demanding a single winner.

Browse the code and training runs

References

  1. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014). Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 15.
  2. Ioffe, S. and Szegedy, C. (2015). Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. Proceedings of the 32nd International Conference on Machine Learning.

You May Also Enjoy