New technologies are being developed to improve medication adherence in challenging populations. While smart devices that quantify adherence are an improvement over self-reported adherence, they do have limitations, especially for researchers interested in drug pharmacokinetic properties in a real-world setting. Our experience from a clinical study which randomized to either directly observed therapy (cell phone videos) or wirelessly observed therapy (smart pill box) uncovered issues to consider during study development prior to data collection. Specifically, understanding if dosing is based on calendar day and how this decision impacts calculation of missed doses. In particular, device flexibility around multiple doses in any one day and dosing that may occur around midnight. In addition, administrators may also need the option of inputting dosing information directly and how these data will be stored and merged with electronic device data must be understood. Lastly, in order to examine data quality assurance prior to analysis, paper records or a separate electronic database recording dosing anomalies should be maintained to validate device output.