Online Program Home
  My Program

Abstract Details

Activity Number: 588 - Statistical Learning: Clustering
Type: Contributed
Date/Time: Wednesday, August 2, 2017 : 2:00 PM to 3:50 PM
Sponsor: Section on Statistical Learning and Data Science
Abstract #324412 View Presentation
Title: Multi Level Clustering Technique Leveraging Expert Insight
Author(s): Sudhanshu Singh* and Ritwik Chaudhuri and Manu Kuchhal and Sarthak Ahuja and Gyana Parija
Companies: IBM India Pvt. ltd. and IBM India Pvt. ltd. and IBM India Pvt. ltd. and IBM India Pvt. ltd. and IBM India Pvt. ltd.
Keywords: clustering ; recruitment process benchmarking ; Spark

State of the art clustering algorithms operate well on numeric data but for textual data rely on conversion to numeric representation. This conversion is done by adopting approaches like TFIDF, Word2Vec, etc. and require large amount of contextual data to do the learning. Such contextual data may not be always available for the given domain. 

We propose a novel algorithm that incorporates Subject Matter Experts' (SME) inputs in lieu of contextual data to be able to do effective clustering of a mix of textual and numeric data. We leverage simple semantic rules provided by SMEs to do a multi-level iterative clustering that is executed on the Apache Spark Platform for accelerated outcome. The semantic rules are used to generate large number of small sized clusters which are qualitatively merged using the principles of Graph Colouring.   We present the results from a Recruitment Process Benchmarking case study on data from multiple jobs. We applied the proposed technique to create suitable job categories for establishing benchmarks. This approach provides far more meaningful insights than traditional approach were benchmarks are calculated for all jobs put together.

Authors who are presenting talks have a * after their name.

Back to the full JSM 2017 program

Copyright © American Statistical Association