When I started this project I assumed I would need most of the 561 features the dataset hands you. I was wrong about how few it takes. A phone clipped to your waist can tell walking-upstairs from sitting at 94.37% accuracy, on people it never saw during training, and the final model gets there on 5 of those 561 numbers. The other 556 I threw away, and the accuracy barely moved. I did not expect that going in. So I worked my way down the dependency chain to understand it: what the raw signal actually carries, then which of its features survive a hard cut, then which model reads the survivors best, and finally where the whole thing would stop holding.

The project and code live here:

Companion repo on GitHub

The signal: 30 people, 6 activities, 561 features

Before any of the modeling, I wanted to understand what was actually being measured. The data comes from the UCI Human Activity Recognition project. 30 volunteers, aged between 19-48 years, each wore a waist-mounted smartphone with embedded inertial sensors while going about ordinary daily living. The raw signal is acceleration along the x, y, and z axes, sampled over time. From those raw streams the project derives 561 attributes per record: time- and frequency-domain summaries alongside orientation angles like angle(X,gravityMean) and the tGravityAcc family. Two extra columns name the subject and the activity. The full data and the related papers live at the UCI ML repository page.

There are 6 categories of activity to tell apart:

  1. standing
  2. sitting
  3. laying
  4. walking
  5. walking-downstairs
  6. walking-upstairs

I used the dataset packaged as a single .RData file rather than stitching together the separate raw text files. A subject column identifies the user, the last column names the activity, and every other column holds one of the 561 derived features. The values are already normalized, so feature scales are comparable out of the box.

The first thing I noticed once I started poking at the columns is that 561 features do not span 561 independent dimensions. They are all derived from a handful of raw acceleration streams, so a lot of them move together. To see that structure I built a correlation matrix across all 561 variables, and it came back dense with high-correlation blocks. When a cluster of features carries the same signal, you can keep one of them and drop the rest without losing what it tells you.

Correlation matrix for all 561 sensor features
Correlation matrix across all 561 features. The bright square blocks along the diagonal are the gravity and body-acceleration families correlating tightly within themselves, so a single member stands in for the whole cluster.

A couple of other checks pointed the same way. Some variables barely change across records, and a feature with near-zero variance carries almost nothing for a classifier, so it is a safe drop. There were no missing values anywhere in the columns, which meant I could work with the complete dataset and not worry about imputation. I tried plotting per-category distributions as well, but with this many features that quickly turns into a wall of charts. The one thing that does jump out of those distributions is a split into two broad groups of activity, which felt like a clue worth holding onto.

Representative distribution of a sensor feature across activity categories
A representative feature distribution. Even one feature separates the activities into two broad groups, a hint that a small handful of features might carry the classification.

The features: letting the forest pick them

So the signal was smaller than its dimension count, but I still did not know which features to keep. I did not want to pick them by hand. My hunches about which sensor readings mattered would just be hunches, so I let the selection run on Random Forest importance scores rather than domain intuition.

First I had to split the data, and I wanted to get that right before anything else touched it. I divided it into train and test sets in a 7:3 ratio by random sampling without replacement, so each set stays representative of the whole. Sampling per output class instead made no meaningful difference here. To keep everything reproducible I set RandomState=42 at the top, which fixes the train and test sets and the feature draws on every run.

Then the test set goes in a drawer. It plays no part in feature selection, training, or tuning. For tuning I lean on the out-of-bag (OOB) score instead. A Random Forest trains each tree on a bootstrap sample, and the rows left out of a given tree’s bootstrap act as a built-in held-out set for that tree. Averaged across the forest, the OOB score gives me a validation signal for free during training, without ever touching the real test set. That property is what let me tune freely without worrying I was quietly leaking the test set into my decisions.

OOB score as an internal validation loop The full dataset splits 70 percent train and 30 percent frozen test. Inside the training set, out-of-bag rows act as a validation signal during training, while the test set stays untouched until the final check. All records 561 features Train (70%) bootstrap trees OOB rows = validation Test (30%), frozen tune on OOB touched once, last
How OOB score lets you tune without leaking the test set. The data splits 70/30, the test set stays frozen, and out-of-bag rows inside the training set act as an internal validation loop until the very last accuracy check.

To rank the features I trained a forest on the training set using all 561 variables and read off the importance scores. Most features sit near zero. A short head of them carries the signal, which matched what the correlation blocks had already suggested.

Random forest variable importance scores for all 561 features
Importance scores across all 561 features. The long flat tail is the redundancy from the correlation matrix showing up again: most features add almost nothing once a few are in.

With the features ranked, two knobs were left to turn: how many top features to keep, and how many trees to grow. I swept them both, iterating over 0-150 trees and for 1-25 variables, and watched the OOB score to see where it flattened out.

OOB error against number of variables selected OOB error against number of trees
OOB score against number of features (left) and number of trees (right). Both curves are flat by the low single digits of features and a few dozen trees, which is the whole argument for a tiny model.

The curves plateau fast. By the time you reach a few features and a few dozen trees, adding more buys almost nothing. So I kept 5 features, set the number of trees to 25, and froze the model there, leaving the held-out test accuracy for the very last step.

Funnel from 561 features down to 5 561 engineered features narrow through Random Forest importance ranking and OOB-guided iteration over feature count and tree count down to the 5 features that survive in the final model. 561 features RF importance short head OOB sweep 5 + 25 trees
The core arc of the project. 561 engineered features narrow through Random Forest importance and OOB-guided iteration down to the 5 that survive, with the tree count set to 25.

Here are the 5 that made the final model:

  • angle(X,gravityMean)
  • tGravityAcc-mean()-Y
  • tGravityAcc-min()-X
  • tGravityAcc-max()-X
  • tBodyAcc-mad()-X

Four of the five are gravity-orientation features, and the fifth is body-acceleration spread along X. That lines up with intuition once you say it out loud. Gravity points a fixed way relative to the phone, so the angle between the device and gravity is a clean read on posture. Sitting, standing, and laying are mostly posture. Walking and its stair variants are mostly motion. A few orientation features plus one motion-spread feature is enough to put a record on the right side of that divide.

Variable importance scores for the final 5 selected features
Importance scores for the final 5 features. Gravity-orientation dominates, which fits the posture-versus-motion split.

I want to be careful about what I am claiming here. The forest picked these features, not a biomechanics argument. The posture-versus-motion reading is a story I tell after the fact about why the selection makes sense, and it holds up, but the selection itself was algorithmic.

The model: RF versus SVM

With the 5 features chosen, the next question was which model reads them best. I trained both a Support Vector Machine and a Random Forest on the same setup and compared them on the held-out test set.

  Random-Forest SVM
Train 94.50% (oob) 83.48%
Test 94.37% 82.37%

Random Forest wins by about 12 points on the test set. The part I found more reassuring than the win is the consistency: the Random Forest OOB validation estimate (94.50%) almost exactly matches the held-out test accuracy (94.37%), which is the signal that the 5-feature model is not memorizing the training set.

Random Forest versus SVM, train and test accuracy Four horizontal bars whose lengths are proportional to the four accuracies from the table: Random Forest train 94.50 percent, Random Forest test 94.37 percent, SVM train 83.48 percent, SVM test 82.37 percent. Bar length equals accuracy times 5.6 pixels, so 100 percent maps to the 560-pixel axis. 0% 100% RF train 94.50% RF test 94.37% SVM train 83.48% SVM test 82.37%
Random Forest against SVM, train and test. Bar lengths are proportional to the four exact accuracies from the table above (length equals accuracy times 5.6 pixels). The gap between the two models is wide, the gap between Random Forest train and test is barely there.

To see where the remaining errors actually live, I looked at the confusion matrices. The motion activities are clean to tell from the still ones, since orientation alone separates posture from movement. The confusions that survive appear to sit within a group, among the activities that look alike to the sensors. Sitting versus standing is the pair you would expect to blur, since both are upright and still.

Confusion matrix for Random Forest on the test set Confusion matrix for SVM on the test set
Confusion matrices on the test set, Random Forest (left) and SVM (right). The off-diagonal mass appears to cluster among the similar activities rather than scattering across all six.

The same separability shows up if you go back to raw feature space. Plotting the distribution of tBodyAccJerk-std()-X colored by activity, some categories fall into tidy clusters while others overlap, which is exactly the pattern the confusion matrices show. It was reassuring to find the model’s errors and the raw data telling the same story.

Distribution of tBodyAccJerk-std()-X across all six activity categories
Distribution of `tBodyAccJerk-std()-X` colored by activity. Some categories separate cleanly, others overlap, which mirrors where the confusion matrices put the errors.

What breaks: the bias caveat

The collapse from 561 to 5 is the part of this that I keep coming back to. On a phone it matters in a concrete way. The whole point of activity recognition on a device is that the device has to do it, on its own battery, while doing everything else. A model that reads 5 features and runs 25 small trees is cheap enough to run continuously without anyone noticing the drain. So the dimensionality collapse is not a tidiness win, it is the thing that makes on-device inference practical at all.

The real limit is the one I built in by choosing Random Forest. I leaned on the assumption that random forests do not usually overfit the training set, and the tight 94.50% to 94.37% gap is what that assumption looks like when it is holding. But it breaks down when the training data is heavily biased. This dataset is relatively balanced across activities and subjects, so the assumption holds here. Move to a population that skews one way, an age band these 30 volunteers never covered, a gait the sensors never saw, and the no-overfit comfort is the first thing I would stop trusting.

That is the part I would want to test next, and it is also the part I cannot answer from this dataset alone. The five numbers are enough to tell what these 30 people were doing. Whether they are enough to tell what you are doing is a question for data I do not have yet.

References

Random Forest:

SVM:

OOB Score:

UCI-ML dataset location:

Scikit-Learn:

GitHub Page:

You May Also Enjoy