Things, links, and facts
Graph engineering means turning documents into a map of things and the links between them, so that a search can follow those links instead of looking for text that sounds like the question.
A document is the wrong unit
Most search systems store documents and return documents. But a document is just whatever someone wrote in one sitting.
- The fact you need is usually one sentence inside it.
- The fact that completes the answer is often a sentence in a different document, written by someone else.
Graph engineering works with single facts instead. Each fact is called a triple.
Triple: one fact with three parts, a subject, a link and an object. “Release v4.2.0 includes change CHG-7043” is a triple. A forty-sentence document might hold eight of them. The rest is text about those facts.INCLUDES is one of eight link types this lab allows. Step 4 explains why the list is fixed.- The sentence comes from one of the lab's documents.
- Its subject and object lift out as two nodes.
- The link between them becomes an arrow with a type:
INCLUDES.
A graph has two parts
| part | what it is | what it carries |
|---|---|---|
| node | one thing, named once: a test, a release, a person | a label saying what kind of thing it is |
| edge | one link between two nodes | its direction, the document it came from, and how sure the extractor was |
That's all a graph is. The rest of this volume is about how nodes and edges get made, kept or refused.
What the links give you
- Paths that no document states. One document says a test covers the cart code. Another says a change modified the cart code. The graph now has a two-step path from the test to the change, even though neither document mentions the other.
- Questions about distance. “What is three steps from here?” is a question you can ask a graph, and can't ask a pile of documents.
- Each document gives up a few facts.
- Facts that mention the same thing share a node.
- Those shared nodes join everything into one graph instead of five separate pieces.
What it costs
A graph isn't free, and this volume doesn't assume you should build one. You pay for three things:
- a fixed list of the kinds of things and links you allow, which someone has to keep up to date;
- a pass over every document to pull the facts out;
- a rule for deciding when two names mean the same thing.
Step 11 shows when that's worth paying, and when it isn't.
python3 lab.py test-triage
Part 2 pulls the facts out of the lab's five documents one by one, and shows the two it refuses.
What to remember
- Work with single facts, not whole documents.
- A graph is just nodes and edges.
- You gain paths that no single document states. You pay with a fixed vocabulary, a pass over every document, and a rule for matching names.