Thanks a lot @Ingo_Richtsmeier for the quick response and clarification.
We are using Camunda for handling complex business processes which are spanning across multiple microservices. The orders volumes are very high and high amount data is being generated and we need to keep this data as it is required for the investigation purpose.
We understand that have to perform the history-cleanup to cleanup the data regularly. But looking at the volumes we would need to clean-up the history db and keep this data some kind of archive db so that it can be referred to even after TTL/removalTime.
It would be good if the history db can be separated from the runtime db(not sure how this can be done) so that there would be no impact on the performance of the running orders.
I understand from the documentation if we go by the Camunda out-of-the-box history cleanup options there are two strategies available
- Removal-Time-based Strategy
- End-Time-based Strategy
For our case there are long running orders(running for 3 to 6 months or even more) and there will be multiple process microservices participating in for fulfilling the order. There will be a “main orchestrator microservice” which will generate events to invoke other microservices (which also have embedded camunda bpm processes).
It might happen that for a given order “microservice 1” has finished execution and is now eligible for the cleanup but the “microservice 2” for the same order/buisinessKey is still running. So now when the clean job runs all the data corresponding to “microservice 1” gets purged which we do not want. The reason is if someone searches by that order number/businessKey in the cockpit ,then he/she cannot get a view on the part of business process handled by “microservice 1”.
Need your suggestion on what strategy we should use here - The removal end-time startegy would work well for the “main orchestration microservice” as it would always be the last one to finish for the given order but this might not work well for other microservices (as I explained in the above paragraph)
We have Optimize but it does not give a detailed view similar to the Cockpit. So what we are looking for is the separate history db(i.e separate from runtime db) to get the maximum performance benefits and at the same time use Cockpit for archive db/separate history db (so that we do not need to develop Custom UI) to view the processes information.