1. posterid:556948
  2. Recommendations for Open Source Analytic (OLAP) system / API to mine Thanos/Prometheus data.

Re: Recommendations for Open Source Analytic (OLAP) system / API to mine Thanos/Prometheus data.

Richard Hartmann
 

Hijacking top email to reply across the board.

As many of you will know, I have been nagging Prometheus-team about this for years, so yes, I think we should cover this.

At PromCon 2017's dev summit hallway track, we talked about connectors to existing data analysis, e.g. an R interface to natively access data stored in Prometheus format. Thanos' block storage would solve a lot of pain points and Promtheus' remote read/write API is another obvious immediate attach point. Also at around the same time, I started a discussion about extending PromQL in this direction, a discussion which never went anywhere, but which I can see being revived.

I disagree that the topic should be death-by-committee'd day 1 by splitting it across several SIGs. Concerted effort and input from subject-matter experts is good, though. But get something off the ground first before making it more cumbersome.

Overall, I think it's something which we should at least take a look at in the context of this SIG. Deeper analysis of data definitely falls under o11y.


Best,
Richard


On Fri, May 29, 2020 at 6:00 PM Bartłomiej Płotka <bwplotka@...> wrote:
Hi SIG Observability! 👋

I recently noticed that many of CNCF's Prometheus and Thanos users often desire to use their metric data collected by Prometheus for more advanced Analytics cases. Something more suitable for Business Intelligence / OLAP use cases. 

As the Prometheus maintainers, we designed Prometheus Query API and PomQL for realtime monitoring, or at most for simple analytics. It's far from being efficient for Data Mining or Data Exploration.

I feel there are two things we are missing in the CNCF space: 

1. Please tell me if I am wrong here, but I don't see any particular BI/OLAP open source project in the CNCF space. If not, I think as CNCF SIG Observability there is some possibility for us to encourage some project for this to either join or at least be closer integrated with the community. Do you think as the CNCF SIG Observability should we be doing this? 🤔

2. Metric data from, especially if you have years of it thanks to Thanos or Cortex, is an amazing source of information. In the Thanos community, we are actively looking for a project that will fit most of the requirements stated here. Are you currently a user of some Open Source OLAP system worth recommending? If yes, which one? Would you like to have good integration of such a system with metrics? 

We are looking for your feedback, preferably on this GitHub issue: https://github.com/thanos-io/thanos/issues/2682, I plan to also put this topic for the next SIG agenda if we will have time for it. 🤗 

Kind Regards and have a good weekend!
Bartek
Join {cncf-tag-observability@lists.cncf.io to automatically receive all group messages.