Posts Tagged ‘unstructured text’

Rumsfeld Conundrum- Finding the Unknown Unknown

Tuesday, January 27th, 2015

Since we began the process of building applications using our AI engine, we have been focused on working with ideas or concepts. With BrainDocs we built intelligent agents to find and score similarity for ideas in paragraphs, but still fell short of the vision we have for our solution. Missing was an intuitive and visual UI to explore content interactively using multiple concepts and  metadata (like dates, locations, etc). We want to give our users the power to create a rich and personal context to power through their research. What do I call this?

Some Google research led me to a great visualization and blog by David McCandless on the Taxonomy of Ideas. While the words in his viz are attributes of ideas, not the ideas themselves, it got me thinking in different ways about the problem.

Taxonomy of Ideas

If you substitute an idea (product or problem) in David’s matrix and add the dimension of time, you create a useful framework. If the idea above was “car”, then the top right might be Tesla and bottom left a Yugo (remember those?). Narrow the definition to “electric car” or generalize to “eco-friendly personal transportation” and the matrix changes. But insert an unsolved problem and now you have trouble applying the attributes. You also arrive at an innovator’s dilemma (not the seminal book by Clayton Christensen), the challenge of researching something that hasn’t been labeled and categorized yet.

Ideas begin in someone’s head. With research, debate, and engineering, they become products. Products have labels and categories that facilitate communication, search and commerce. The challenge for idea search on future problems is that the opposite occurs: products are not yet ideas and the problems they solve may not have been defined yet. If I may, Donald Rumsfeld nailed the problem with this famous quote:

“There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we know we don’t know. But there are also unknown unknowns. There are things we don’t know we don’t know.”

And if it’s an unknown unknown, it certainly hasn’t been labeled yet so how do you search for it? Our CEO Walt Diggelmann used to say it this way, “ai-one gives you an answer to a question, you did not know that you have to ask….!

Innovators work in this whitespace.

If you could build and combine different intelligent (idea) agents for problems as easily as you test different combinations of words in a search box, you could drive an interactive and spontaneous exploration of ideas. In some ways this is the gift of our intelligence. New ideas and innovation are in great part combinatorial, collaborative and stimulated by bringing together seemingly unrelated knowledge to find new solutions.

Instead of pumping everything into your brain (or an AI) and hoping the ideas pop out, we want to give you the ability to mix combinations of brains, add goals and constraints and see what you can create. Matt Ridley termed this “ideas having sex”. This is our goal for Topic-Mapper (not the sex part).

So what better place to apply this approach than to the exploration of space? NASA already created a “taxonomy of ideas” for the missions of the next few decades. In my next blog I’ll describe the demo we’re working on for the grandest of the grand challenges, human space exploration.

Tom

Gartner Names ai-one Cool Vendor 2012 for Content Analytics

Tuesday, May 15th, 2012

Gartner Cool Vendor in Content Analytics, 2012

 

*GARTNER named ai-one in Cool Vendors in Content Analytics, 2012. The report reviews five vendors from around the world that offer potentially disruptive innovations for analyzing data to find actionable insights. Unlike traditional business intelligence solutions, these vendors provide technologies that can understand multiple types of information — including both structured and unstructured data.

The core value of ai-one’s technology is to make it easy for programmers to build intelligence into any application. Our APIs provide a way to mimic the way people detect patterns. “This is why we call it biologically inspired intelligence,” says founder and CEO

Answering the Most Important Questions, Mr. Walter Diggelmann, “because it works just like the human brain.”

 These companies have received tremendous publicity. Both are funded by traditional Silicon Valley venture capital firms. No surprise that they strive to provide comprehensive machine learning solutions rather than a tool for the general programming public.

“We do something completely different! We provide a general purpose tool that you can combine with other technologies to solve a specific problem. We do not try to do everything. Rather we just do one thing: We find the answer to the question you didn’t know to ask.” says Diggelmann

The advantage of ai-one’s approach to developers is that using the API is easy. The tool finds the inherent meaning of any data by detecting patterns. For example, feed it text and it will find every keyword and determine the association words that give each keyword context. Together, keywords and associations provide a complete and accurate summary of a document. The API gives precise results almost instantly and does not require any specialized training to use. Moreover, it is autonomic — as it works without any human intervention.

ai-one follows a technology licensing model — much like Qualcomm. The company makes money when licensees embed the API into commercial applications. ai-one works closely with its OEM partners to ensure that their products are successful.

ai-one’s technology enables programmers to build hybrid analytics solutions that integrate content from almost any digital source, in any language, regardless of its structure (or lack of structure). This capability has the potential to transform the way we think about business intelligence. “90% of the world’s data is unstructured,” says Diggelmann, “but 100% of the major business intelligence systems can’t read or understand it.  We provide a tool to bridge the gap.”

*Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings.  Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact.  Garner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

 

Building Machine Learning Tools to Mine Unstructured Text

Friday, February 17th, 2012

This presentation describes how to build tools to find the meaning of unstructured text using machine generated knowledge representation graphs using NLP and ai-one’s Topic-Mapper API.
The prototype solution, called ai-Browser, is a generalized approach that can solve the following types of use cases:
  • Sentiment analysis of social media feeds
  • Evaluating electronic medical records for clinical decision support systems
  • Comparing news feeds
  • Electronic discovery for legal purposes
  • Automatically tagging documents
  • Building intelligent search agents
The source code for ai-Browser is available to developers to customize to meet specific requirements. For example:
  • Healthcare providers can use ai-Browser to analyze medical records by using ontologies and medical lexicons.
  • Social media marketing agencies can use ai-Browser to create personal profiles of customers by reading social media feeds.
  • Researchers can use ai-Browser to mine PubMed and other repositories.
Our goal is to get the source code and the API into the hands of commercial companies who want to tailor the application to solve specific problems.
Click here to download the presentation from SlideShare:
View more presentations from ai-one

Mining Unstructured Text: A new machine learning approach

Monday, February 13th, 2012

We believe we have found a new approach to apply a new general purpose machine learning technology to solve domain-specific problems by mining unstructured text. The solution addresses fundamental problems in knowledge management:

ai-browser is a tool for mining unstructured textHow to find information that is difficult to describe?

For example, you want to find a match between two people to fill an empty job position. What attributes do you use to represent a complex subject (like a person) to find the best fit?

What if the single best answer is hidden within a vast amount of unstructured text?

Let’s say you want to repurpose a drug – such as using the side-effect of a chemical to treat a disease using a newly discovered metabolic pathway. How would you search through the 21+ million research articles in PubMed to find the best match from more than 2,000+ known drug compounds?

What if the textual information is constantly changing?

What if you want to provide personalized marketing to a person based on what they are saying on Facebook, Twitter or LinkedIn?  To do this, you must understand the meaning of what they are saying. The most accurate approach is to have people read and interpret the conversations because we are fantastic at understanding the complexity of language. But to do this with a computer requires a different approach: Machines must learn like humans. They must understand how meaning evolves in a conversation, how to disambiguate, how to detect the single most important concepts, etc.

Big Data Means Big Opportunity

These are classic “Big Data” problems – and they are rampant. Finding a solution would change everything; from how we discover new drugs to what social media would tell us about ourselves.

There have been many attempts to find ways for machines to learn like a human. Artificial intelligence has made bold promises that have been consistently broken for more than 50 years. Yet, we still don’t have a universal approach for machines to learn and understand language like a human.

Growth of Websites

Now, more than ever, we need to find a new approach to mine unstructured text. As of February 2012, it is estimated that the Internet has more than 614 million websites. More than 1.8 zettabytes of information was created in 2011 – more than much of it unstructured text from our comments on websites, news articles, social media feeds… just about anything where people are communicating with language rather than numbers.

Unstructured text can’t be processed like structured data. Rather it requires an approach that enables knowledge representation in a form that can be processed by machines.

Knowledge representation is a rich field and there has been tremendous effort and innovation – too many to describe here. However, we still live in a world where the overwhelming majority of people (including almost every CIO, developer and consumer) CANNOT find the information they seek with a simple query. Rather, the domain of data mining text analytics is dominated by specialists who use tools that are very difficult to learn and very expensive to deploy (because they require highly skilled programmers).

We set out to create a new toolset that would be easy to use for almost any programmer to build data mining tools for unstructured text.

ai-browser: A prototype for human-machine collaboration

For the past several months, we have been working on a new approach for text analytics and data mining. The idea is to create a tool that enables human-machine collaboration to quickly mine unstructured data to find the single best answer.

We now have a working prototype, called ai-browser, that solves knowledge management and data mining problems involving unstructured text. It combines natural language processing (NLP) and pattern recognition technologies to generate a precise knowledge representation graph.  Our team selected OpenNLP because it is open-source, easy to use and customize. We used the Topic-Mapper API to detect patterns within the text after it was pre-processed to isolate parts of speech. The system also allows users to use ontologies and/or reference documents to sharpen the results. The output is a graph that can be used in a number of ways with 3rd party products, such as:

  • Submission to search appliances like Google, Bing, Lucene, etc.
  • Analysis with modelling tools like Cytoscape, MATlab, SAS, etc.
  • Enterprise systems for reporting, knowledge management and/or decision support

This graph makes it easy to ask questions like, “Find me something like _______!” and get a very tightly clustered group of results – rather than millions of hits.

Even more impressive, ai-browser’s graph is a powerful tool that can be applied to a wide range of applications, such as:

  • Healthcare – clinical decision support systems to enable physicians to make better decisions by understanding all the relevant information held in electronic medical records (EMRs) – including emerging trends and relationships within the patient population.
  • Social media – detecting and tracking sentiments in conversations over time (such as Twitter) to understand how brands are perceived by customers.
  • Innovation management – discovering the relationships of information across disciplines to foster more productive collaboration and interdisciplinary discoveries.
  • Information comparison and confirmation – determine the similarities and differences between two different sources of content.
  • Human resources – sourcing and placement of the best candidate for a job based on previous work experience.

The intent of the ai-browser design is to provide a starting point for developers to build solutions to meet the specific needs of enterprise customers. For example, modifying the system enables solutions to the following use cases:

  • Help a physician determine if additional tests are necessary to confirm a diagnosis.
  • Determine how perceptions about a brand are change through conversations on Twitter.
  • Find new uses for a drug by reviewing clinical studies published on PubMed and determining if there are relevant patent filings.
  • Identify stock market trading opportunities by comparing news feeds and SEC filings on a particular company or industry.
  • Finding the best person for a job by searching the internet for someone that is “just like person who has this job last year.”

Enterprise Data Mining: A far easier, lower cost approach.

Unlike other data mining approaches, ai-browser learns the meaning of documents by generating a lightweight ontology – a dynamic file that describes every relationship between every data element. It detects keywords and their association words which provide context. The combination of a keyword and all the association words can be thought of as a coordinate (x,y0->T) where x is the keyword and y0->T is the series of association words for that specific keyword. The collection of these coordinates creates a topology for the document: G(V,E) where G is graph and V is the set of vertices (or nodes) represented by each keyword and E is the edge represented by the associations to the keyword.

ai-fingerprint of Fox News Article

We call this graph the “ai-fingerprint.” It is a lossless knowledge representation model. It captures the meaning of the document by showing the context of words and the clustering of concepts. It is lossless because it captures every relationship in a directed graph – thereby revealing the significance of a word that may only appear once yet is central to the meaning of a large, complex textual data set.

ai-browser expresses ai-fingerprints uses the XGMML format in REST. This enables it to accommodate dynamic data, so it can change as the underlying text changes (such as in text from social media feeds).

Contact Olin Hyde to schedule a demo of ai-Browser. The source code is available to programmers to license and modify to solve specific problems.