Transaction Searching: Unifying Field Names posted by David Carraso.
From the post:
Problem
You need to build transactions from multiple data sources that use different field names for the same identifier.
Solution
Typically, you can join transactions with common fields like:
... | transaction username
But when the username identifier is called different names (login, name, user, owner, and so on) in different data sources, you need to normalize the field names.
If sourcetype A only contains field_A and sourcetype B only contains field_B, create a new field called field_Z which is either field_A or field_B, depending on which is present in an event. You can then build the transaction based on the value of field_Z.
sourcetype=A OR sourcetype=B
| eval field_Z = coalesce(field_A, field_B)
| transaction field_Z
Looks a lot like a topic map merging operation doesn’t it?
But “looks a lot like” doesn’t mean it is “the same as” a topic map merging operation.
How would you say it is different?
While the outcome may be the same as a merging operation (which are legend defined), I would say that I don’t know how we got from A or B to Z?
That is next month or six months from now, or even two years down the road, I have C and I want to modify this transaction.
Question: Can I safely modify this transaction to add C?
I suspect the answer is:
“We don’t know. Have to go back to confirm what A and B (as well as C) mean and get back to you on that question.”
For a toy example that seems like overkill, but what if you have thousands of columns spread over hundreds of instances of active data systems.
Still feel confident about offering an answer without researching it?
Topic map based merging could give you that confidence.
Even if like Scotty, you say two weeks and have the answer later that day, to age a bit before delivering it ahead of schedule. 😉