Graph Engineering Concepts › Things, links, and facts
01 · part 1, conceptsconcept

Things, links, and facts

Graph engineering means turning documents into a map of things and the links between them, so that a search can follow those links instead of looking for text that sounds like the question.

A document is the wrong unit

Most search systems store documents and return documents. But a document is just whatever someone wrote in one sitting.

  • The fact you need is usually one sentence inside it.
  • The fact that completes the answer is often a sentence in a different document, written by someone else.

Graph engineering works with single facts instead. Each fact is called a triple.

Triple: one fact with three parts, a subject, a link and an object. “Release v4.2.0 includes change CHG-7043” is a triple. A forty-sentence document might hold eight of them. The rest is text about those facts.
One sentence becomes one triple doc2 · Release notes v4.2.0 Components rebuilt in this release: cart from change CHG-7043 v4.2.0 subject :Release CHG-7043 object :Change INCLUDES one of eight relation types the schema allows The other words in that sentence are prose about the fact. The triple is the fact. Five documents give 11 facts kept in all.
A sentence from the lab, and the fact inside it. Follow it left to right.
The link is not free text. INCLUDES is one of eight link types this lab allows. Step 4 explains why the list is fixed.
  • The sentence comes from one of the lab's documents.
  • Its subject and object lift out as two nodes.
  • The link between them becomes an arrow with a type: INCLUDES.

A graph has two parts

partwhat it iswhat it carries
nodeone thing, named once: a test, a release, a persona label saying what kind of thing it is
edgeone link between two nodesits direction, the document it came from, and how sure the extractor was

That's all a graph is. The rest of this volume is about how nodes and edges get made, kept or refused.

What the links give you

  • Paths that no document states. One document says a test covers the cart code. Another says a change modified the cart code. The graph now has a two-step path from the test to the change, even though neither document mentions the other.
  • Questions about distance. “What is three steps from here?” is a question you can ask a graph, and can't ask a pile of documents.
Five documents, one graph no single document contains the answer CI run 88213 Release notes Change CHG-7043 BUG-18402 Release note 10 nodes · 11 edges · every edge carries the document it came from
The five documents of the test triage lab, and the graph inside them. Watch the documents give up their facts.
10 nodes and 11 edges from five short documents, 101 words in all.
  • Each document gives up a few facts.
  • Facts that mention the same thing share a node.
  • Those shared nodes join everything into one graph instead of five separate pieces.

What it costs

A graph isn't free, and this volume doesn't assume you should build one. You pay for three things:

  • a fixed list of the kinds of things and links you allow, which someone has to keep up to date;
  • a pass over every document to pull the facts out;
  • a rule for deciding when two names mean the same thing.

Step 11 shows when that's worth paying, and when it isn't.

The labs throw facts away as well as keep them. In the test triage lab, 11 facts are kept and 2 are refused. The refused list is useful output, not a failure. Step 4 explains why.
Try it. In the labs folder, run:
python3 lab.py test-triage

Part 2 pulls the facts out of the lab's five documents one by one, and shows the two it refuses.

What to remember

  • Work with single facts, not whole documents.
  • A graph is just nodes and edges.
  • You gain paths that no single document states. You pay with a fixed vocabulary, a pass over every document, and a rule for matching names.