Personally I think security is far ahead here compared to normal observability tools. I decided to work on a side-project to try and add SIEM like functionality to a clickhouse backed otel platform; by the end of it the thing I came to believe the tooling blue teams use should be used in observability generally, not just security. Incident/event management is a very powerful concept, and provides a clean framework to hang all of this information on. Then your solution for mechanically finding the neighborhood in k8s is one way to add observations to an event. Some SIEMs have started having agents recommend stuff to be added to an event, which is a nice middle-ground of having agents help but not completely control the discovery and diagnostic effort (as well as providing a clean feedback loop for training data synthesis).
That's all to say, have you considered that framing, and if so, have any opinions on why more general observability tools haven't gone that direction?
Are you saying that if everything became an event, its easier to correlate because the event is more free form and it's easier to add custom metadata?
In theory, I agree with you. But you can't eliminate 'joins' all together, IMO, because it's not practical to know beforehand what context will be useful.
configmap that was updated recently and incorrectly: how would you know the cm update is the cause? How would you sort that out if you made 5/10 updates?
say a node is low on disk: what about setting up monitoring alerts for those king of things.
In general, alerts are of great help. An alert fired after an update is a big smell about that update causing the issue.
So if there is a spike on say errors, you do a group by pod, click on the pod in the legend, see the k8s neighborhood which means discover the pod --> config map association. You see the config map and all the recent updates to it, and can even diff them like code.
An alert on the other hand, say on a node that is low on disk, doesn't automatically tell you all the affected workloads. Not saying the alert doesn't help, but this experience helps identify the affected pods and apps much faster.
Personally I think security is far ahead here compared to normal observability tools. I decided to work on a side-project to try and add SIEM like functionality to a clickhouse backed otel platform; by the end of it the thing I came to believe the tooling blue teams use should be used in observability generally, not just security. Incident/event management is a very powerful concept, and provides a clean framework to hang all of this information on. Then your solution for mechanically finding the neighborhood in k8s is one way to add observations to an event. Some SIEMs have started having agents recommend stuff to be added to an event, which is a nice middle-ground of having agents help but not completely control the discovery and diagnostic effort (as well as providing a clean feedback loop for training data synthesis).
That's all to say, have you considered that framing, and if so, have any opinions on why more general observability tools haven't gone that direction?
Are you saying that if everything became an event, its easier to correlate because the event is more free form and it's easier to add custom metadata?
In theory, I agree with you. But you can't eliminate 'joins' all together, IMO, because it's not practical to know beforehand what context will be useful.
configmap that was updated recently and incorrectly: how would you know the cm update is the cause? How would you sort that out if you made 5/10 updates?
say a node is low on disk: what about setting up monitoring alerts for those king of things.
In general, alerts are of great help. An alert fired after an update is a big smell about that update causing the issue.
So if there is a spike on say errors, you do a group by pod, click on the pod in the legend, see the k8s neighborhood which means discover the pod --> config map association. You see the config map and all the recent updates to it, and can even diff them like code.
An alert on the other hand, say on a node that is low on disk, doesn't automatically tell you all the affected workloads. Not saying the alert doesn't help, but this experience helps identify the affected pods and apps much faster.
[flagged]
[dead]