Prometheus Metrics: Process Engine Plugin (contribution)


#1

Hey

Wanted to share the above.

It was a small plugin i put together that enables Prometheus metrics reporting, but the extra “flavour” is you can use the metrics system to generate “business centric” metrics that are generated through the bpmn’s dmn’s, and cmmn’s usage.

few examples:

import io.digitalstate.camunda.prometheus.collectors.SimpleGaugeMetric;

def openCases = new SimpleGaugeMetric('open_cases', 'Number of Open Cases, labeled by Case Type', ['type'])

openCases.increment(['standard'])
import io.digitalstate.camunda.prometheus.collectors.SimpleHistogramMetric

def httpRequest = new SimpleHistogramMetric('legacy_system_123_request', 'Connection duration time, labeled by HTTP Method', null, ['method'])

httpRequest.startTimer(['POST'])

sleep(Math.abs(new Random().nextInt() % 5000) + 650) // Simulates a delay

httpRequest.observeDuration()

import io.digitalstate.camunda.prometheus.collectors.SimpleGaugeMetric

def money = new SimpleGaugeMetric('money_collected', 'dollar values collected, labeled by form of payment', ['payment_form'])

def amount = Math.abs(new Random().nextDouble() % 284.03) + 23.54 // Random dollar value
money.increment(amount, ['credit-card'])
import io.digitalstate.camunda.prometheus.collectors.SimpleGaugeMetric

def openCases = new SimpleGaugeMetric('open_cases')

openCases.decrement(['standard'])

def closedCases = new SimpleGaugeMetric('closed_cases', 'Number of Open Cases, labeled by Case Type', ['type'])
closedCases.increment(['standard'])

You can also setup internal recurring API queries that generate the metrics so certain queries that are run continually can be done with the java api rather than using black-box exporter.

All the data easily ends up in Prometheus and can be visualized with grafana!

Enjoy


Prometheus metrics exporter?
JSON History Provider (usecase: ElasticSearch indexing)
JSON History Provider (usecase: ElasticSearch indexing)
#2

I have updated the metrics plugin with support for all of the internal Camunda Metrics:


#3

another useful metric and shown use case of job flooding occurring


#4

another update


#5

Hello @StephenOTT

Thanks for the updates. I’m very eager to play around with this.

How would I install this plug-in on the Tomcat / Webapps bundled CE of camunda v7.9?

I’ve read the docs on how to register an engine plugin (1) but fail to understand where I need to copy your configuration example (2) to. Should I copy it directly into …/conf/bpm-platform.xml?

1 : https://docs.camunda.org/manual/latest/user-guide/process-engine/process-engine-plugins/#configure-process-engine-plugins
2 : https://github.com/StephenOTT/camunda-prometheus-process-metrics-process-engine-plugin#plugin-configuration

Thanks so much


#6

@eugene you can manually build the package with dependencies and place it in your lib folder and then update the xml config as described in the readme. Its still a work in progress so things are to change.

I will be updating soon with Groovy script query support so you can configure all of the metrics through YML and scripts that can be placed in a volume. This will allow you to run your metrics without having to recompile anything.


#7

Few more updates


#8

@eugene I have made some updates
You should be good to go now.

You can install similar to the docker setup in the repo or you can bring the plugin through maven dep

Notice the new features as well:

  1. Custom metrics are executed with compiled groovy script
  2. Yaml file configs for each metric file

#9

Hi @StephenOTT

we played with your plugin yesterday in combination with SpringBoot and it looks very promising and easy to use. You can check our example app at https://github.com/holunda-io/camunda-example-prometheus.

What we found a bit annoying is that the prometheus-metrics.yml file cannot be read from classpath resources, so we could not just put it into src/main/resources but had to reference it from the file system and, in case of Docker, copy it manually into our container.

It should not be too hard to fix, but I wanted to check if there was any specific reason for that.

Also it would be nice to include your custom metrics directly in your plugin, just like the system metrics, instead of providing Groovy scripts that, again, have to be referenced from the file system. Or maybe we missed something?

But overall, I think it is a very useful and easy to use plugin (especially if you have Prometheus running anyway)!

Cheers,
Stefan


#10

@stefanzilske thanks for the great feedback!

What we found a bit annoying is that the prometheus-metrics.yml file cannot be read from classpath resources, so we could not just put it into src/main/resources but had to reference it from the file system and, in case of Docker, copy it manually into our container.

I have not tested or considered the changes needed for springboot style builds yet (but its one of the next things on my list to do). Most of the design was about adding the jar into a existing shared engine.

It should not be too hard to fix, but I wanted to check if there was any specific reason for that.
Agreed, not a big change. The current design was under the thinking of:

  1. The metrics you want to collect change over time, but you are not necessary rebuilding/packaging your app, you are just adding the new script and restarting the server, and everything kicks in.
  2. Scripts would be stored as in a container volume in most cases, and scripts would/could be shared across engines as most scripts would likely have large reuse options.

Also it would be nice to include your custom metrics directly in your plugin, just like the system metrics, instead of providing Groovy scripts that, again, have to be referenced from the file system. Or maybe we missed something?

What are your thoughts on the custom metrics I provided? Originally i built them as built in java classes and then moved them to scripts outside of the jar. I had built them more as a proof of concept and examples of usage. Do you see value in using them?

edit: @stefanzilske additionally, how would you want to indicate that the resource comes from the jar/classpath vs arbitrary filesystem path? Any preference of formats?


#11

@stefanzilske want to give the latest a try.

I have added support for classpath custom scripts, and I have added the custom scripts that I built directly into the package.

See the YAML example in the readme for updates. TLDR: just add classpath: in-front of your path.


#12

I have added a collector script for the User Operation Log

https://docs.camunda.org/javadoc/camunda-bpm-platform/7.9/org/camunda/bpm/engine/history/UserOperationLogEntry.html#OPERATION_TYPE_CREATE

provides counts of each type of operation type:

Not that this can add up in number of queries being performed, so pay attention to performance factors and how often you need to update this info.


#13

External Task collectors have been added!


#14

Some More updates

have add Grafana Annotation support and better Activity Duration tracking: