Incidents is not inserting to the DB

Hi,

I created a custom IncidentHandler which extends DefaultIncidentHandler and registered it into my process engine.

When an exception is occured anywhere within my JavaDelegates, my IncidentHandler#handleIncident method is working successfully but the issue is the incident is not inserted to DB. When I check act_hi_incident and act_ru_incident I can’t see any record created.

Additionally, after a while, the IncidentHandler#handleIncident method works again for same error. I think somehow the engine tries to retry
that incident

This is my incident handler:

@Component
@Slf4j
public class IncidentHandler extends DefaultIncidentHandler {

    private final BpmStateMessageSender messageSender;
    private final ApplicationContext applicationContext;

    public IncidentHandler(BpmStateMessageSender messageSender, ApplicationContext applicationContext) {
        super(Incident.FAILED_JOB_HANDLER_TYPE);
        this.messageSender = messageSender;
        this.applicationContext = applicationContext;
    }

    @Override
    public Incident handleIncident(IncidentContext context, String message) {
        ProcessEngine processEngine = applicationContext.getBean(ProcessEngine.class);
        log.debug("INCIDENT HAPPENED : {}", message);

        try {
            log.debug("Fetching active execution");
            ProcessInstance processInstance = (ProcessInstance) processEngine
                    .getRuntimeService()
                    .createExecutionQuery()
                    .active()
                    .executionId(context.getExecutionId())
                    .singleResult();

            log.debug("Found execution: {}", processInstance);
            log.debug("Building websocket message..");
            WebSocketMessage streamMessage = WebSocketMessage.builder()
                    .status(ProcessStatus.INCIDENT_HAPPENED)
                    .user(ProcessUtil.getUserAssigneeId())
                    .businessKey(processInstance.getBusinessKey())
                    .build();
            log.debug("Sending websocket message..");
            messageSender.send(streamMessage);
        } catch (Exception e) {
            log.error("Failed on sending incident message through websocket");
            log.error(e.getMessage(), e);
        }

        return super.handleIncident(context, message);
    }
}

Hey @Fariz_Latifov,

The default behavior of the Camunda Engine is to retry a service task that fails 3 times. If the third time fails an incident is created. That one should be then available in the DB and visible in Cockpit. If you want to change that behavior you can select the Service Task in the BPMN model and define your own retry rule in the property panel. I hope that helps.

Cheers
Nele

Hey @Nele,

By exploring logs I found something about this issue: HistoricIncidentEntity.insertHistoricIncidentEvent is unable to insert incident info to act_hi_incident table because of missing history_configuration_ column.

org.camunda.bpm.engine.ProcessEngineException: ENGINE-03004 Exception while executing Database Operation 'INSERT HistoricIncidentEventEntity[46168a1e-28c6-11eb-aa29-7a0dfed44cfc]' with message '
### Error flushing statements.  Cause: org.apache.ibatis.executor.BatchExecutorException: org.camunda.bpm.engine.impl.persistence.entity.HistoricIncidentEntity.insertHistoricIncidentEvent (batch index #2) failed. 1 prior sub executor(s) completed successfully, but will be rolled back. Cause: java.sql.BatchUpdateException: Batch entry 0 insert into ACT_HI_INCIDENT (
        ID_,
        PROC_DEF_KEY_,
        PROC_DEF_ID_,
        ROOT_PROC_INST_ID_,
        PROC_INST_ID_,
        EXECUTION_ID_,
        CREATE_TIME_,
        END_TIME_,
        INCIDENT_MSG_,
        INCIDENT_TYPE_,
        ACTIVITY_ID_,
        CAUSE_INCIDENT_ID_,
        ROOT_CAUSE_INCIDENT_ID_,
        CONFIGURATION_,
        HISTORY_CONFIGURATION_,
        INCIDENT_STATE_,
        TENANT_ID_,
        JOB_DEF_ID_,
        REMOVAL_TIME_
      ) values (
        '46168a1e-28c6-11eb-aa29-7a0dfed44cfc',
        'customer_info',
        'customer_info:12:373c9f46-2895-11eb-8941-7a0dfed44cfc',
        '33dbcae5-28c6-11eb-aa29-7a0dfed44cfc',
        '3590800d-28c6-11eb-aa29-7a0dfed44cfc',
        '3590800d-28c6-11eb-aa29-7a0dfed44cfc',
        '2020-11-17 15:16:00.945+04',
        NULL,
        'ms-customer executing GET http://ms-customer/customer-profile?cif=1049640&pin=1LCE168',
        'failedJob',
        'Activity_get_customer_data',
        '46168a1e-28c6-11eb-aa29-7a0dfed44cfc',
        '46168a1e-28c6-11eb-aa29-7a0dfed44cfc',
        '3593dbb2-28c6-11eb-aa29-7a0dfed44cfc',
        '45d90bec-28c6-11eb-aa29-7a0dfed44cfc',
        0,
        NULL,
        '373cc65a-2895-11eb-8941-7a0dfed44cfc',
        NULL
      ) was aborted: ERROR: column "history_configuration_" of relation "act_hi_incident" does not exist

I want to mention that 1 week before I upgraded ‘camunda-bpm-spring-boot-starter’ version from 3.3.1 to 3.4.0. I think there is Entity and Table incompability causes this error.

Did you upgrade Camunda recently? (if so, you may have missed a script) Looking at the SQL upgrade scripts, HISTORY_CONFIGURATION_ was added in $DATABASENAME_engine_7.11_to_7.12.sql.

1 Like