Our Secret Sauce with Intent Analysis

Interpreting the intent behind spoken or written language is a very complex process that humans learn through experience in different social settings. This is one reason why literature is studied in universities around the world and academics earn doctoral degrees distilling the various nuances of the works of great literary figures.

3Pillar recently teamed up with one of our customers that promises disruption in the way businesses understand competitor strategies and respond to them. The customer has invested years of research and thought leadership in dissecting discourse from a wide variety of industries. The outcome of this research is a model for classifying discourse as a particular strategy that you may be battling or playing. The model further suggests your counter strategies or assistive strategies.

The key challenge in building an automated decision system based on this model is that the computational linguistics involved with intent analysis are not well developed or understood today. The customer asked 3Pillar to build a beta version of the product to understand the possibilities and the challenges. When the 3Pillar team analyzed the model, it realized that many components of the model could be realized with Natural Language Processing (NLP). NLP in and of itself is not the solution to intent analysis, but starting from an ontology or a model of human interaction, NLP should be the next stop.

Screenshot 2016-02-09 23.36.09

We buzzed around armed with our Python REPLs (this is not a word yet, but it should be!) and promptly imported the venerable NLTK library and its corpora. The most useful components of the library were as follows:

  • Tokenization – Tokenizing text found on the web is a notoriously difficult activity. There is an entire ecosystem busy with scraping web pages with automated and manual processes. NLTK made it simple to tokenize content in two steps – first to sentences and next to words, and we did not need to bother ourselves with various sentence and word delimiters.
  • Pronunciation – The CMU Pronouncing Dictionary contains about 150,000 words from North American English and their pronunciations. Phonetic analysis of words in a sentence allowed us to find patterns with alliteration and rhyming words.
  • Part of Speech Tagging – We used the Average Perceptron tagger with the UPENN tagset to understand the structure of sentences. Understanding if a sentence began with a verb, if it had cardinal numbers or if it had superlatives were essential for the decision system.

Our choice of Python ultimately enabled us to deliver the decision system prototype complete with an API built with the Flask microframework and an automated installation procedure.

One more successful product and a happy customer. Interested in finding out more? Contact us to hear the rest of the story!

Sayantam Dey

Sayantam Dey

Senior Director Engineering

Sayantam Dey is the Senior Director of Engineering at 3Pillar Global, working out of our office in Noida, India. He has been with 3Pillar for ten years, delivering enterprise products and building frameworks for accelerated software development and testing in various technologies. His current areas of interest are data analytics, messaging systems and cloud services. He has authored the ‘Spring Integration AWS’ open source project and contributes to other open source projects such as SocialAuth and SocialAuth Android.

Leave a Reply

Related Posts

Take 3, Scene 23: The Influence and Evolution of Blockchain Michael Lisse and Derek Tiffany join us in the studio for this episode of Take 3 to talk about the influence and evolution of blockchain in the financ...
The Innovation Engine from TechCrunch Disrupt New York For a very special episode of The Innovation Engine, we're bringing you a number of interviews from the floor of last week's TechCrunch Disrupt New Yo...
Take 3, Scene 22: Minimizing Time to Value On this episode of Take 3, we dive into the concept of Minimizing Time to Value with Andy Zipfel and Paul Doman, two members of 3Pillar's Client Servi...
Inside the Refraction Space, with Rachael Stott On this episode of The Innovation Engine, we bring the podcast inside the Refraction headquarters to talk again with Rachael Stott on how place and sp...
Health & Wellness is a Reactive Industry, but Technolog... Skyrocketing healthcare spending is having far-reaching consequences on the American economy. It contributes to keeping wages stagnant, and crowds out...


Sign up today to receive our monthly product development tips newsletter.