We are trying to convert an excel file with about 70000 rows to dmn decision table. Based on this article, Converting Excel Worksheets to DMN | Camunda we picked up convertor-dmn-xlsx-cli binary and tried to use it. When the number of rows in excel file are small, things work fine. Once it goes past about 1000 rows, things take hours to complete and also we are sometimes unable to complete the process. In addition, decision table with large number of rows takes a very long time to load in DMN embedded engine.
Thanks @Webcyberrob and @Niall for your responses. @Webcyberrob, when you mention lookup table, is it a table in some sort of SQL store or hash table in memory? Or is there some sort of construct in DMN to realize this please?
@Niall I will try the new hack project. But since there are too many rules, runtime will be a challenge even if I am able to convert the large file correct?
Yes by lookup table I mean an implementation based on a database lookup or a hashtable etc. You can implement a lookup table in DMN, but its unlikely to be an efficient or elegant implementation.
Perhaps you could elaborate a little more on the nature of these 70k rows - are they actually business rules (eg if x<y then z) or are they more like a lookup table?
In terms of performance, the DMN engine is quite fast and I see benchmarks such as 200,000 evaluations per second. So if your rules are simple, DMN performance may not be such an issue, but maintaining and understanding a DMN table with that many rules could be the real problem you have to face…
We have an excel spread sheet with columns A through R as inputs and S as output. When the data comes in, all we want to do is based on which row matches A - R columns, pick column S as output in the row that matched. Currently this sheet sits in a database table and we left join it with other tables to get the result. However, we are changing our paradigm to stream processing and we thought of using embedded camunda engine with the excel sheet transformed to decision table in camunda so that as the data is flowing in the stream, camunda embedded engine can run the decision table on data in the stream and provide results. This is primarily our use case to do near real time processing of data as it flows in a stream. @Webcyberrob does this answer your question?
Yes sounds like a lookup table…rather than DMN, you could have it as a static file eg csv and load it into an internal hashmap or even an in memory DB. Of course this assumes its quite static and your service is designed to work this way…