A Case Study in Evaluating a Knowledge Representation (part 1)

I had to give a little ad hoc talk on “OWL vs. Rules”. This comes up a lot, and not just within rules communities.

When answering the question “Why would we use OWL over rules?” or “What’s the difference between OWL and rules?” or any of these variants (which often are, “Someone convinced me to use a rules engine that I don’t fully understand…does that get me out of having to use Yet Another Thing?”), I prefer not to evangelise. I like various rule languages. Datalog is pretty sweet. Production rules are nifty. I’ve written my share of XSLT and Prolog. And for many things, they are the right thing. So, of course, the first problem is “Which rule flavour do you mean?”

What I like to do is talk about evaluation techniques and criteria. Most people who are asking the question don’t know how to do this. They don’t know how to tell whether a KR formalism is fit for purpose. It’s not a trivial problem!

I use something like the following framework:

  1. Check the adequacies:
    1. Representational
      • The language can express required knowledge more or less directly, naturally, effectively
    2. Cognitive
      • The language is usable
    3. Computational
      • The language is reasonably implementable with appropriate performance
  2. What are the services (both possible and actually implemented)?
  3. Do we get surprising reuse?

Obviously, the ground issue is whether a particular representation does the application job at all! But I’m presuming we believe that all candidates can get somewhere, esp. if we add a lot of code to fill in the gaps. Minimising the hacks is a key goal!

My current example use scenario is the following: We’re trying to implement the CHA2DS2-VASc scoring system against an Electronic Health Record (EHR) system. So we have some databases we need to query.

CHA2DS2-VASc has a number of risk factors (or variables) that we need to determine, e.g., whether the patient has hypertension (HTN).  Each of these corresponds to a clinical concept (for hypertension, call it…HTN). These concepts can have various definitions for various purposes, most obviously, they may have different operationalisations. So, I might say that a patient has HTN if they have a typical blood pressure of over 140/80, or if in the last three months they had a blood pressure reading of >140/80, or if they are on 2 classes of anti-HTN drugs. These definitions are operational: They are in terms of metrics, not underlying concepts. But when trying to translate research into practice, the (various) operational definitions are key!

We can, of course, capture these in a variety of ways in a variety of formalisms.

HTN Example

Here’s a simple example to work through:

We want to capture the concept of a patient with hypertension  with the (one of many) operational definition of “on at least one member of each of two distinct classes of anti-HTN drug” (derived from a given paper on afib risk assessment).

Ultimately, we want to capture this is such a way that we can use it to help guide the formulation of queries against a medical record, or building a capture form, or similar clinical informatics task. We want it to be elaboration tolerant in the sense that we can fit in other operationalisations usefully.

RDFS Attempt

We can capture some of this in as simple a formalism as RDFS (using a slight variant of Manchester Syntax):

Class: P1-PatientWithHTN SubClassOf: PatientWithHTN
Class: Drug1 SubClassOf: ClassA
Class: Drug2 SubClassOf: ClassA
Class: Drug3 SubClassOf: ClassB
Class: Drug4 SubClassOf: ClassB
Class: ClassA SubClassOf: HTNDrug
Class: ClassB SubClassOf: HTNDrug
Class: OnClassA
Class: OnClassB

ObjectProperty: on Range: HTNDrug

(The range is a bit gnarly as we don’t want it to be restricted to anti HTN drugs only, but that’s not the limitations I’m focused on.)

As is clear, we get stuck fairly quickly. We can’t, in RDFS, express that the class of drug (or the drugs themselves!) are disjoint, which is important! It’s easy to imagine a case where the classes of drugs represented functionality,  and, in some cases, the same drug could be on both lists (e.g., both a cough suppressant and a decongestant). In that case, sometimes being on a single drug would be enough. Not so in this case!

We also can’t represent the fact that the sufficient condition is being on two drugs. If we say:

Class: OnClassA SubClassOf: P1-PatientWithHTN

We’ve missed the OnClassB requirement. We can’t just add the corresponding ClassB requirement:

Class: OnClassB SubClassOf: P1-PatientWithHTN

You could try to hack it with a named bridge class:

Class: OnClassAandB SubClassOf: OnClassA, OnClassB, P1-PatientWithHTN

This is a bit tempting because if you are a member of OnClassAandB you will be a member of OnClassA, OnClassB, and P1-PatientWithHTN, which is also true if being both OnClassA and OnClassB implies being a P1-PatientWithHTN. But this is compatible with someone being in each of OnClassA and OnClassB but in neither of OnClassAandB or P1-PaitentWithHTN. We either have to have the discipline to always enter people in OnClassAandB (even if we discover that they are OnClassA and OnClassB in very different contexts) or we need to add a rule, OWL axiom, or code to do the recognition.

Because that just adds in people who are only on ClassB drugs. We have no way to express a conjunctive condition. And, of course, we can’t express that being “with HTN” is a function of being “on certain drugs”. The relational aspects don’t show up in any of our definitions, mostly because we have no definitions!

Of course, we can express the concrete, extensional, relational situations that we want. We can assert that some patient is OnClassA, or is in an on relation to some specific class A drug. What we can’t do is make such a relational structure trigger recognition in a class. And, of course, we would need out of band information to say, “Hey! It’s this relational structure that form the recognition conditions for Patients with HTN!!!”

So, we are forced into another formalism or ad hoc application code. We can conclude that for this task in this domain, RDFS is representationally inadequate. I would also claim that it is cognitively inadequate, because it’s not too hard to trick yourself into thinking you can squeeze out more than you can. Worse, it seems easy to “fix” using a bit of code or some hacky rules, and thus not recognise that you are producing a code smell.

In the next instalment, we’ll look at using a richer language to capture the missing bits.

Note: Oy, this is where I got stuck. It’s taking so long to write it all out. I’m sorta transliterating a talk, and I kinda wish I would just put up the slides! I seem committed now though. These were in Powerpoint so….