Physical activity is commonly used in biomedical settings as an outcome or covariate. Researchers have traditionally relied on surveys to quantify subjects' activity levels, but surveys are not objective in nature and can have recall bias. Smartphones provide an opportunity for unobtrusive objective measurement of physical activity in naturalistic settings, but their data tends to be noisy and must be analyzed with care. We explored the potential of smartphone accelerometer and gyroscope data to distinguish between different activities: walking, using stairs, sitting, and standing. We conducted a study in which four subjects followed a study protocol and performed a sequence of activities with one phone in their front pocket and another phone in their back pocket. The subjects were filmed, and the obtained footage was annotated to establish moment-by-moment ground truth activity. We introduce a modified version of the so-called movelet method to classify activity type and to quantify the uncertainty present in that classification. Our results demonstrate the promise of smartphones for activity recognition in naturalistic settings and highlight challenges in this field of research.