[Q] Production ready and related to Graphite Project?

[nook24]

Hi,

At the moment, I'm running this Graphite setup:

• Carbon-C-Relay as load balancer
• 2 x Carbon-Cache instances
• Graphite-Web
  The Graphite-Web is only running to provide the HTTP API, so the interface itself is not used by a User.

I stumbled across this project and I think this is a really cool project and good idea to replace python as underlying programming language for such a task.

It looks like this project is around since 2015, so I assume it is production ready (#35) by now. Is this correct?

If my understanding is correct, i should be able to replace my current setup with go-carbon and carbonapi.
Are there still any cache instances required to handle heavy load, or is this now all done by the cache setting of go-carbon?
In case there is still some load balancing required, I would probably replace the Carbon-C-Relay in my setup with https://github.com/grafana/carbon-relay-ng to have a 100% Go based setup.

Is the WHISPER_FALLOCATE_CREATE still a thing? This option had to be disabled, when using NFS shares.

I also come across a potential bug within the traditional Graphite setup.
When a metric is split across two storage schemas, for example 15s:1d, 60s5d I currently facing the issue that i can not query a graph across the whole time range. It is possible to query both storage schemas separately. Does the carbonapi also have this limitation?

Thanks for your hard work!

[Civil]

so I assume it is production ready (#35) by now. Is this correct?

go-carbon initially was created by @lomik to replace carbon-cache at the place where he worked back then and is also used at large scale at booking.com (and basically maintained by people who mostly work there).

As far as I know it still doesn't support necessary bits to make tags working and doesn't support pluggable storages (it is essentially whisper only). Other than that it should work as drop-in replacement and even provide extra features (like whisper compression).

Instead of carbon-c-relay you can use either carbon-relay-ng from Grafana - and it should be 100% compatible with carbon-relay. Alternatively you can use carbon-c-relay (either original or fork that is stored here - fork wasn't updated for quiet some time as original one had a period when it was not very stable and testing each release took some time and effort, and since I don't work at booking I don't have setup to extensively test relay). People at booking developed https://github.com/bookingcom/nanotube - it doesn't support tags, but claims to be faster than carbon-c-relay (which was faster than carbon-relay-ng, and that usually means less requirements for CPU) and support some other instrumentation features which others doesn't support. It is also written in Go.

About carbonapi - there is go-graphite/carbonapi that should be production ready and somewhat aims to be compatible with graphite-web (actually people behind grafana contributed really a lot of bugfixes and improvements for functions that are supported there, however there still might be behavioral difference between graphite-web and carbonapi) and it supports some other features, like you can add different backends and even use prometheus or victoriametrics as backend (though that is experimental) - and in case backend support tags - carbonapi will support tags as well.

There is bookingcom's fork of carbonapi (bookingcom/carbonapi) which forked from before tag support was implemented upstream and since then diverged quiet a bit (as well with aim to add more instrumentation support, etc, etc). It is used in production by booking.com as far as I know, so also can be considered as production ready.

I hope that helps. And for other questions, maybe @deniszh or some one else would be able to answer.

[Civil]

And about tagging support - after I finished my reply I've remembered that it is not entierly correct. go-carbon supports tags on write-path so you can feed it tagged metrics and it will store them correctly, it also supports tags if you'll use carbonlink (for fetching fresh data), but it doesn't have support on clusterserver part of the protocol as it was way more complex to properly implement and as well there were some concerns in terms of performacne with vanilla implementation of tags, those it was never finished (there were at least 3 attempts to do that - during internal hackathon in 2017 or 2018, then that work was picked up by @bom-d-van and he reimplemented backend part of it and I tried to implement APIs in ~2019, but haven't finished that job as I don't have a use case for it myself).

Maybe later there were other attempts, but I haven't heard about them.

[nook24]

Hi @Civil thanks for your detailed and fast reply.

go-carbon initially was created by @lomik to replace carbon-cache

That's perfect! My basic idea is to only have one instance of go-carbon by default, but to have the option to fire up more instances when required. At the moment most of our systems are running with 2 or 4 carbon-cache instances, but from what the readme claims about performance I guess go-carbon should handle this quite easily.

As far as I know it still doesn't support necessary bits to make tags working and doesn't support pluggable storages (it is essentially whisper only)

At the moment we do not use Tags at all. I'm developer of an open source monitoring suite and we use Graphite as our primary time series database.

A typical series will be stored as: <host-uuid>.<service-uuid>.<metric>, e.g: 03cb5ace-6dc1-455e-b311-18b37e8d1c87.d0e28652-36b8-4c80-a43b-a9f7f44090e0.rta

Instead of carbon-c-relay you can use either carbon-relay-ng from Grafana

That would be probably my go-to candidate

About carbonapi - there is go-graphite/carbonapi that should be production ready and somewhat aims to be compatible with graphite-web (actually people behind grafana contributed really a lot of bugfixes and improvements for functions that are supported there

This sounds good to me. We do not have very complex queries so probably everything will be working out of the box, especially due to most of the users are using Grafana anyways.

I will definitely give this approach a shot. It would be really cool if I could replace all Python components as they tend to break quite frequently. (new Python version, new pip --something options required etc)

Thanks again for your time, I appreciate this!