Prometheus Metrics: Process Engine Plugin (contribution)

Hey

Wanted to share the above.

It was a small plugin i put together that enables Prometheus metrics reporting, but the extra “flavour” is you can use the metrics system to generate “business centric” metrics that are generated through the bpmn’s dmn’s, and cmmn’s usage.

few examples:

import io.digitalstate.camunda.prometheus.collectors.SimpleGaugeMetric;

def openCases = new SimpleGaugeMetric('open_cases', 'Number of Open Cases, labeled by Case Type', ['type'])

openCases.increment(['standard'])
import io.digitalstate.camunda.prometheus.collectors.SimpleHistogramMetric

def httpRequest = new SimpleHistogramMetric('legacy_system_123_request', 'Connection duration time, labeled by HTTP Method', null, ['method'])

httpRequest.startTimer(['POST'])

sleep(Math.abs(new Random().nextInt() % 5000) + 650) // Simulates a delay

httpRequest.observeDuration()

import io.digitalstate.camunda.prometheus.collectors.SimpleGaugeMetric

def money = new SimpleGaugeMetric('money_collected', 'dollar values collected, labeled by form of payment', ['payment_form'])

def amount = Math.abs(new Random().nextDouble() % 284.03) + 23.54 // Random dollar value
money.increment(amount, ['credit-card'])
import io.digitalstate.camunda.prometheus.collectors.SimpleGaugeMetric

def openCases = new SimpleGaugeMetric('open_cases')

openCases.decrement(['standard'])

def closedCases = new SimpleGaugeMetric('closed_cases', 'Number of Open Cases, labeled by Case Type', ['type'])
closedCases.increment(['standard'])

You can also setup internal recurring API queries that generate the metrics so certain queries that are run continually can be done with the java api rather than using black-box exporter.

All the data easily ends up in Prometheus and can be visualized with grafana!

Enjoy

4 Likes

I have updated the metrics plugin with support for all of the internal Camunda Metrics:

1 Like

another useful metric and shown use case of job flooding occurring

1 Like

another update

1 Like

Hello @StephenOTT

Thanks for the updates. I’m very eager to play around with this.

How would I install this plug-in on the Tomcat / Webapps bundled CE of camunda v7.9?

I’ve read the docs on how to register an engine plugin (1) but fail to understand where I need to copy your configuration example (2) to. Should I copy it directly into …/conf/bpm-platform.xml?

1 : https://docs.camunda.org/manual/latest/user-guide/process-engine/process-engine-plugins/#configure-process-engine-plugins
2 : https://github.com/StephenOTT/camunda-prometheus-process-metrics-process-engine-plugin#plugin-configuration

Thanks so much

@eugene you can manually build the package with dependencies and place it in your lib folder and then update the xml config as described in the readme. Its still a work in progress so things are to change.

I will be updating soon with Groovy script query support so you can configure all of the metrics through YML and scripts that can be placed in a volume. This will allow you to run your metrics without having to recompile anything.

Few more updates

@eugene I have made some updates
You should be good to go now.

You can install similar to the docker setup in the repo or you can bring the plugin through maven dep

Notice the new features as well:

  1. Custom metrics are executed with compiled groovy script
  2. Yaml file configs for each metric file

Hi @StephenOTT

we played with your plugin yesterday in combination with SpringBoot and it looks very promising and easy to use. You can check our example app at https://github.com/holunda-io/camunda-example-prometheus.

What we found a bit annoying is that the prometheus-metrics.yml file cannot be read from classpath resources, so we could not just put it into src/main/resources but had to reference it from the file system and, in case of Docker, copy it manually into our container.

It should not be too hard to fix, but I wanted to check if there was any specific reason for that.

Also it would be nice to include your custom metrics directly in your plugin, just like the system metrics, instead of providing Groovy scripts that, again, have to be referenced from the file system. Or maybe we missed something?

But overall, I think it is a very useful and easy to use plugin (especially if you have Prometheus running anyway)!

Cheers,
Stefan

2 Likes

@stefanzilske thanks for the great feedback!

What we found a bit annoying is that the prometheus-metrics.yml file cannot be read from classpath resources, so we could not just put it into src/main/resources but had to reference it from the file system and, in case of Docker, copy it manually into our container.

I have not tested or considered the changes needed for springboot style builds yet (but its one of the next things on my list to do). Most of the design was about adding the jar into a existing shared engine.

It should not be too hard to fix, but I wanted to check if there was any specific reason for that.
Agreed, not a big change. The current design was under the thinking of:

  1. The metrics you want to collect change over time, but you are not necessary rebuilding/packaging your app, you are just adding the new script and restarting the server, and everything kicks in.
  2. Scripts would be stored as in a container volume in most cases, and scripts would/could be shared across engines as most scripts would likely have large reuse options.

Also it would be nice to include your custom metrics directly in your plugin, just like the system metrics, instead of providing Groovy scripts that, again, have to be referenced from the file system. Or maybe we missed something?

What are your thoughts on the custom metrics I provided? Originally i built them as built in java classes and then moved them to scripts outside of the jar. I had built them more as a proof of concept and examples of usage. Do you see value in using them?

edit: @stefanzilske additionally, how would you want to indicate that the resource comes from the jar/classpath vs arbitrary filesystem path? Any preference of formats?

@stefanzilske want to give the latest a try.

I have added support for classpath custom scripts, and I have added the custom scripts that I built directly into the package.

See the YAML example in the readme for updates. TLDR: just add classpath: in-front of your path.

I have added a collector script for the User Operation Log

https://docs.camunda.org/javadoc/camunda-bpm-platform/7.9/org/camunda/bpm/engine/history/UserOperationLogEntry.html#OPERATION_TYPE_CREATE

provides counts of each type of operation type:

Not that this can add up in number of queries being performed, so pay attention to performance factors and how often you need to update this info.

1 Like

External Task collectors have been added!

2 Likes

Some More updates

have add Grafana Annotation support and better Activity Duration tracking:

1 Like

@StephenOTT, Wonderful work there on metrics plug in.

I am trying to configure the plugin in my local instance (docker based) and followed the instructions to build the docker, however, encounter the following error, any idea how to move past this and actualise these wonderful visualizations ?

Thanks, Have got the metrics dashboard up and running, however, my question is how do we get the process variable based dashboards - Number of approvals, rejections etc in a process.

@Kameswaran_Subbaram1 use the custom metrics directly in scripts. So when you generate variables and you want to report their status, you can submit your values into your counter/gauge.

See the readme for examples of generating metric data directly in scripts of the Bpmn

Hello all!

Recently we have been building exporters that access the db directly to you can perform Group By SQL queries to create much more powerful queries and joins. Such as “count totals of instances for script tasks by tenant and by process dev key”. Or the same but for timers. “How many times are set to be due before 10am, 3pm, this week, this month, etc”.

These queries work the same as the current groovy scripts but you can access the the connection pool and run SQL using groovy’s SQL classes and dsl.

Do others have interest in this capability?

2 Likes

Hi @StephenOTT.
I’m very interested in implementing Prometheus into my Spring Boot app using Gradle, Camunda BPM 7.13, MySQL as DB.
So, as a base config, I’m trying to use your implementation.
However, when I build my SB app I get this error:

Errors occurred while build effective model from /home/diego.scheinin/.gradle/caches/modules-2/files-2.1/com.github.jknack/handlebars/4.0.7/d1b7cce1d9ea939a9f9082b8b42eedb17f5ae56b/handlebars-4.0.7.pom:
    Invalid packaging for parent POM com.github.jknack:handlebars.java:4.0.7, must be "pom" but is "jar" in com.github.jknack:handlebars.java:4.0.7
    'dependencies.dependency.version' for org.apache.commons:commons-lang3:jar is missing. in com.github.jknack:handlebars:4.0.7
    'dependencies.dependency.version' for org.mozilla:rhino:jar is missing. in com.github.jknack:handlebars:4.0.7
    'dependencies.dependency.version' for org.slf4j:slf4j-api:jar is missing. in com.github.jknack:handlebars:4.0.7
    'dependencies.dependency.version' for javax.servlet:servlet-api:jar is missing. in com.github.jknack:handlebars:4.0.7
    'dependencies.dependency.version' for ch.qos.logback:logback-classic:jar is missing. in com.github.jknack:handlebars:4.0.7
    'dependencies.dependency.version' for junit:junit:jar is missing. in com.github.jknack:handlebars:4.0.7
    'dependencies.dependency.version' for org.yaml:snakeyaml:jar is missing. in com.github.jknack:handlebars:4.0.7
    'dependencies.dependency.version' for org.easymock:easymock:jar is missing. in com.github.jknack:handlebars:4.0.7
    'dependencies.dependency.version' for org.powermock:powermock-api-easymock:jar is missing. in com.github.jknack:handlebars:4.0.7
    'dependencies.dependency.version' for org.powermock:powermock-module-junit4:jar is missing. in com.github.jknack:handlebars:4.0.7
Errors occurred while build effective model from /home/diego.scheinin/.gradle/caches/modules-2/files-2.1/com.github.jknack/handlebars-helpers/4.0.7/7977fc38cca72fd2037f4556edfca00a3850b110/handlebars-helpers-4.0.7.pom:
    Invalid packaging for parent POM com.github.jknack:handlebars.java:4.0.7, must be "pom" but is "jar" in com.github.jknack:handlebars-helpers:4.0.7
    'dependencies.dependency.version' for javax.servlet:servlet-api:jar is missing. in com.github.jknack:handlebars-helpers:4.0.7
    'dependencies.dependency.version' for ch.qos.logback:logback-classic:jar is missing. in com.github.jknack:handlebars-helpers:4.0.7
    'dependencies.dependency.version' for junit:junit:jar is missing. in com.github.jknack:handlebars-helpers:4.0.7
    'dependencies.dependency.version' for org.easymock:easymock:jar is missing. in com.github.jknack:handlebars-helpers:4.0.7
    'dependencies.dependency.version' for org.yaml:snakeyaml:jar is missing. in com.github.jknack:handlebars-helpers:4.0.7

I’m trying to figure out what’s going on, but any help will be appreciated :slight_smile:

Cheers,
Diego

Can you provide more information about your configuration and setup?