Our Secret Sauce with Intent Analysis

Interpreting the intent behind spoken or written language is a very complex process that humans learn through experience in different social settings. This is one reason why literature is studied in universities around the world and academics earn doctoral degrees distilling the various nuances of the works of great literary figures.

3Pillar recently teamed up with one of our customers that promises disruption in the way businesses understand competitor strategies and respond to them. The customer has invested years of research and thought leadership in dissecting discourse from a wide variety of industries. The outcome of this research is a model for classifying discourse as a particular strategy that you may be battling or playing. The model further suggests your counter strategies or assistive strategies.

The key challenge in building an automated decision system based on this model is that the computational linguistics involved with intent analysis are not well developed or understood today. The customer asked 3Pillar to build a beta version of the product to understand the possibilities and the challenges. When the 3Pillar team analyzed the model, it realized that many components of the model could be realized with Natural Language Processing (NLP). NLP in and of itself is not the solution to intent analysis, but starting from an ontology or a model of human interaction, NLP should be the next stop.

Screenshot 2016-02-09 23.36.09

We buzzed around armed with our Python REPLs (this is not a word yet, but it should be!) and promptly imported the venerable NLTK library and its corpora. The most useful components of the library were as follows:

  • Tokenization – Tokenizing text found on the web is a notoriously difficult activity. There is an entire ecosystem busy with scraping web pages with automated and manual processes. NLTK made it simple to tokenize content in two steps – first to sentences and next to words, and we did not need to bother ourselves with various sentence and word delimiters.
  • Pronunciation – The CMU Pronouncing Dictionary contains about 150,000 words from North American English and their pronunciations. Phonetic analysis of words in a sentence allowed us to find patterns with alliteration and rhyming words.
  • Part of Speech Tagging – We used the Average Perceptron tagger with the UPENN tagset to understand the structure of sentences. Understanding if a sentence began with a verb, if it had cardinal numbers or if it had superlatives were essential for the decision system.

Our choice of Python ultimately enabled us to deliver the decision system prototype complete with an API built with the Flask microframework and an automated installation procedure.

One more successful product and a happy customer. Interested in finding out more? Contact us to hear the rest of the story!

Sayantam Dey

Sayantam Dey

Senior Director Engineering

Sayantam Dey is the Senior Director of Engineering at 3Pillar Global, working out of our office in Noida, India. He has been with 3Pillar for ten years, delivering enterprise products and building frameworks for accelerated software development and testing in various technologies. His current areas of interest are data analytics, messaging systems and cloud services. He has authored the ‘Spring Integration AWS’ open source project and contributes to other open source projects such as SocialAuth and SocialAuth Android.

Leave a Reply

Related Posts

Tarun Kumar at the Component Driven Development Using React.... On June 29th, Tarun Kumar spoke at Component Driven Development Using React.js at 3Pillar Global's office in Noida, India. Component Driven Developmen...
Interviews from Industry Summit 2017 – with Guest Host... On this special double episode of The Innovation Engine, the 3Pillar team interviews many of the speakers who took the stage at Industry Summit 2017. ...
Ravish Tiwari at the Component Driven Development Using Reac... On June 29th, Ravish Tiwari spoke at Component Driven Development Using React.js at 3Pillar Global's office in Noida, India. Component Driven Developm...
Raj Kumar Bharti at the Component Driven Development Using R... On June 29th, Raj Kumar Bharti spoke at Component Driven Development Using React.js at 3Pillar Global's office in Noida, India. Component Driven Dev...
Selecting The Minimum Viable Toolset for Product Managers This summer I was attending a machine learning conference and during a break, found myself deep in conversation with fellow product managers. As typic...

SUBSCRIBE TODAY


Sign up today to receive our monthly product development tips newsletter.