Google Spreadsheets, Oauth2 and async continuation

[eaglecros]

Dear camunders,

I have been trying to access data contained in a Google Spreadsheet from within a Camunda process for some time now. Finally, I decided the best option would be to create a process (P1) that launches the Oauth2 flow. Such process contains a single service task (T1) that “does everything”, but the part that the user needs to do manually: authorize access, including starting a listener that waits for the Oauth2 callback. Since (afaik) the service task cannot prompt the user to open a new window (and hence grant the Oauth2 access), I decided to start a process (P2) that again contains a single user task (T2) that prompts the user to open the corresponding URL. P1 then should receive the callback, and the Oauth2 flow should be complete. However, I stumbled into one problem:

• T1 needs to be specified as asyncBefore (or asyncAfter, not too sure) so that task T1 does not block the engine until the listener in T1 finishes; T1 will never finish because for T1 to finish, T2 needs to be completed first (and afaik the engine will not start T2 until T1 is completed).
• In specifying T1 as async, I get two problems:

1. that the field injections in the modeller seem not to work. I can bypass this easy by using process variables and execution.getVariable() - is this supposed to be so?
2. and more importantly, task T2 is nowhere to be found in the cockpit (and hence the user is never prompted to complete the task). I suspect that these two problems are related, namely, that DelegateExecution execution as provided to the execute method is not able to provide the appropriate data for
   somevariable.getValue(execution).toString();
   (which fails with null pointer exception), and
   rts = execution.getProcessEngineServices().getRuntimeService("P2"); rts.startProcessInstanceByKey();
   which might be starting P2 in some place which I don’t have control over.

So then my questions:

• what am I failing to understand and do correctly when calling startProcessInstanceByKey?
• is there some other simpler way to do this?
• is there some working oauth2 implementation out there? I wouldn’t think that I’m the first one to stumble into this.

Thanks in advance!

Cheers,

Germán Sanchis-Trilles

[aakhmerov]

Hi @eaglecros,

I am not sure I understand your question, could you provide a unit test simulating what you are trying to achieve? You can see all examples that we have here https://github.com/camunda/camunda-bpm-examples. There is no OAuth implementation sample there though, but there are examples of startProcessInstanceByKey

Alternative to starting process from delegate would be call activity, you can read more about it here https://docs.camunda.org/manual/7.6/reference/bpmn20/subprocesses/call-activity/

Does that help you?
Askar

[eaglecros]

Hi @aakhmerov.

Thanks for your reply.

I’m not all too sure I would be able to provide a (minimal) unit test, since I’m quite new to Java as well. However, I’ll do my best. I am trying to:

Here a quick sketch of what I’m trying to achieve Oauth:

1. Start a process (P1) that manages Oauth authentication
2. Within P1, have a task (T1) that launches the (local) Oauth listener. T1 is marked as asyncBefore and asyncAfter.
3. Within T1, start a process (P2) that replaces the typical Oauth pop-up asking the user to login (since I understand this is not feasible inside Camunda) with a call to a P2 process
4. Launch the P2 process from within T1.
5. P2 has a single user task, that asks the user to browse to the Oauth authentication URL, that then redirects to the callback URL
6. capture the callback within the listener, and finish the process.

Here are the two diagrams I’m using (P1 first, then P2):

In this case, I would think “call activity” would not solve my problem, since (afaik) a call activity would happen synchronously, but in order to launch the oauth listener and keep it listening while the user opens the authentication window I would need it to happen async.

So, code-wise, I have a manageOauth class which (mainly) contains (credit: some of the code has been extracted from one of the Google Oauth libraries; HTTP_TRANSPORT, JSON_FACTORY, CLIENTID, SECRET, SCOPES, DATA_STORE_FACTORY have been properly initialized):

public static void doBrowse(String url) {
    Map<String,Object> processVariables = new HashMap<String,Object>();
    processVariables.put("oauthUser","user");
    processVariables.put("oauthUrl",url);

    ProcessInstance oauthProcess =
        runtimeService.startProcessInstanceByKey("doOauth",processVariables);
}

public static void authenticateUser() throws IOException {
    GoogleAuthorizationCodeFlow flow = 
        new GoogleAuthorizationCodeFlow.Builder(
            HTTP_TRANSPORT, JSON_FACTORY, CLIENTID, SECRET, SCOPES)
        .setDataStoreFactory(DATA_STORE_FACTORY)
        .setAccessType("offline")
        .build();
    PublicLocalServerReceiver plsr = new PublicLocalServerReceiver.Builder()
        .setPort(28080)
        .build();

    try {
        Credential credential = flow.loadCredential(userId);
        if (!(credential != null
            && (credential.getRefreshToken() != null || 
                credential.getExpiresInSeconds() > 60))) {
            String redirectUri = plsr.getRedirectUri();
            AuthorizationCodeRequestUrl authorizationUrl = 
                flow.newAuthorizationUrl().setRedirectUri(redirectUri);
            doBrowse(authorizationUrl.build());
            String code = plsr.waitForCode();
            TokenResponse response = 
                flow.newTokenRequest(code).setRedirectUri(redirectUri).execute();
            credential = flow.createAndStoreCredential(response, userId);
        }
    } finally {
        plsr.stop();
    }
}

public void execute (DelegateExecution execution) throws Exception {
    currentUser = (String) execution.getVariable("oauthUser");
    runtimeService = execution.getProcessEngineServices().getRuntimeService();
    authenticateUser();

However, my problem is that, when starting the process, the doOauth process is nowhere to be seen in the cockpit.

Browsing through the internet, I found this post, which describe a similar situation to what I’m experiencing:
https://community.alfresco.com/thread/219808-when-startprocessinstancebykey-gives-the-instance-back
Reading this code, and trying to grasp the code in camunda-bpm-examples/servicetask/service-invocation-asynchronous I seem to glimpse that the execute method is not the one I should be implementing here… is this the case? Could you point me in the appropriate direction for reading on about this if it is?

Thanks again, and sorry if the question is derived from my lack of experience.

Cheers,

Germán Sanchis-Trilles

[aakhmerov]

Hi @eaglecros,

No worries about experience, forum is existing exactly for cases where people need guidance. Let me split this in questions and answers that I see.

• Q: Is execute right method to implement in JavaDelegate for service task?
  A: Yes, to understand topic a bit better you can read following docs: https://docs.camunda.org/manual/7.6/reference/bpmn20/tasks/service-task/ and https://docs.camunda.org/manual/7.6/user-guide/process-engine/delegation-code/#java-delegate

• Q: when I start second process (P2) I cannot see it anywhere in cockpit, why?
  A: In your P2 you have a user task, which will show up in Task List assuming that assignee is set properly, you can read more about it here https://docs.camunda.org/manual/7.6/reference/bpmn20/tasks/user-task/ and I think you should take a look at https://github.com/camunda/camunda-bpm-examples/tree/master/usertask/task-form-embedded. It would help me if you could attach your .bpmn file here, so that I can see how you declare your user task. In cockpit you should see token in wait state on user task, more on wait states here https://docs.camunda.org/manual/7.6/user-guide/process-engine/transactions-in-processes/. To check if your process is in proper state you can take a look in cockpit that there is process instance running and when you open it you see token on T2. In your specific case I would assume that execution token stays hanging in T1.

• I think that your T1 should not be marked as asynchronous. Since if it is, your process will stop there and never reach user task. Your use case is probably a good fit for process with multiple tokens, one waiting for credentials and another one at user task itself, which is achieved through parallel gateways usually.

Does that help you?
Askar

[eaglecros]

Hi @aakhmerov,

• Thanks for the documentation. I’m reading through it to settle down some concepts which I do have quite unstable.

• With respect to Q2, the problem is exactly what you describe: the token is left hanging in T1, and no token goes through to P2 (and T2 is not reached), even though the call to startProcessInstanceByKey does return control to the calling method. I understand that having the user set properly is not really a problem, since the task would show up in the task list without an assigned user, but the process would show up as started. Since you were expecting this to happen ("[…] the execution token stays hanging in T1"), I assume this is an expected behaviour, but why? Why does the startProcessInstanceByKey method seem to finish properly, but no P2 process is actually started?

• I did not implement this as a parallel gateway because of two reasons:

1. My initial intention was to follow the Oauth logic, in which case the “start process P2” would be a method provided to the Oauth class, which would start the process instead of opening up a browser window (which is the typical behaviour).
2. In case a parallel gateway is implemented, I fear it could lead to unexpected behaviours: the oauth callback listener needs to be started before the user opens the authorization URL; how would I ensure that the user does not fulfill the user task before the service task launches the oauth listener?

Thanks for your help!!

Here are the two BPMNs I’m currently using:

launchOauth.bpmn (P1; 3.8 KB)

Cheers,

Germán Sanchis-Trilles

[eaglecros]

(and here the second attachment because of forum rules)

doOauth.bpmn (P2; 3.5 KB)

[aakhmerov]

Hi @eaglecros,

before we dive into explanations, are you using embedded or shared engine?

Cheers,
Askar

[eaglecros]

Mmmmm… I would guess embedded? Not sure:

We have an own Camunda server, cloned from hub.docker.com, running inside a docker in one of our machines.

Cheers,

Germán Sanchis-Trilles

[aakhmerov]

@eaglecros,

is it running inside of some application server? Which docker container is it exactly? could you send me a link?

Cheers,
Askar

[eaglecros]

Yes, it is running inside Tomcat. Here are two links:
The docker container:
https://hub.docker.com/r/camunda/camunda-bpm-platform/

[eaglecros]

(second link in second post because of forum rules)

The Github from which our installation is cloned:

camunda/docker-camunda-bpm-platform

Thanks,

Germán Sanchis-Trilles

[aakhmerov]

ok, so this is a tomcat distro with shared engine.

[eaglecros]

Ok… so what does that imply? Is there any difference in the behaviour I should expect in this case?

[aakhmerov]

It means that the job executor is turned on by default. So what happens is that token reaches T1 and hangs there, as you are opening a listener there, which is actually pretty bad, as you are binding on port 28080, and you are able to do that only once. T1 hangs there due to the internal implementation of listener, which is blocking current Thread I assume and therefore your code which starts second process is not really executed. To verify that, could you may be add logging right before starting process?

So the “proper” way to do that would be to start your listener in some external service and once this service got everything needed for authentication (OAuth Token) send that in your process and do whatever you need with it.

Does that help you?
Askar

[eaglecros]

It does [help me], although I still don’t understand why P2 doesn’t appear in the cockpit (note that process P2 is summoned before the listener is started)…

How would the “proper” way work? Ok, I start the listener in some other external service (which gets triggered by a Camunda service task, I assume), and then have a service task wait for some sort of signal?

As for the logs, this is what I get when Task T1 (the service task) is marked as asyncBefore

18-Jan-2017 16:34:58.427 INFO [pool-2-thread-1] org.mortbay.log.Slf4jLog.info Logging to org.slf4j.impl.JDK14LoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
18-Jan-2017 16:34:58.428 INFO [pool-2-thread-1] org.mortbay.log.Slf4jLog.info jetty-6.1.26
18-Jan-2017 16:34:58.439 INFO [pool-2-thread-1] org.mortbay.log.Slf4jLog.info Started SocketConnector@0.0.0.0:28080

And this is what I get when task T1 is not marked as asyncBefore

18-Jan-2017 16:37:32.386 INFO [http-nio-8080-exec-5] org.mortbay.log.Slf4jLog.info Logging to org.slf4j.impl.JDK14LoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
18-Jan-2017 16:37:32.387 INFO [http-nio-8080-exec-5] org.mortbay.log.Slf4jLog.info jetty-6.1.26
18-Jan-2017 16:37:32.398 INFO [http-nio-8080-exec-5] org.mortbay.log.Slf4jLog.info Started SocketConnector@0.0.0.0:28080

(they look very similar to me, but just in case the bracketed part is relevant…)

In the interface, however, the first case (asyncBefore) returns control to the user immediately, whereas in the second case (not asyncBefore) the interface is stuck in the “Start process” screen.

For the record, and regarding

@aakhmerov

[…] as you are binding on port 28080, and you are able to do that only once

binding on port 28080 (and not on a random port, which I could do several times) is because I need to know the port beforehand to have the port appropriately forwarded in the docker config.

Thank you very much for your help!

[aakhmerov]

Hi @eaglecros,

I think you are misinterpreting asyncBefore flag, which you don’t need in your case at all. AsyncBefore only means that state of process will be persisted in the database, you can read more about the concept here https://docs.camunda.org/manual/7.6/user-guide/process-engine/transactions-in-processes/.

Your process is hanging not due to camunda engine or your process, but rather to blocking nature of the socket listener that you open. You should read more about server sockets, somewhere like here http://www.oracle.com/technetwork/java/socket-140484.html

Waiting for signal is usually implemented using message events https://docs.camunda.org/manual/7.6/reference/bpmn20/events/message-events/

Cheers,
Askar.

[eaglecros]

Hi @aakhmerov,

I think I’m going to implement this as a form, in which case it should work pretty straight-forward in the same way regular pages implement it; giving it a second thought, I think it’s the correct way, given that Oauth is thought to be performed with a web interface.

I must say, though, that even though I’ve read through that page you are pointing at (the one about async) several times, I still don’t seem to understand it correctly.

Just one more question, however: I understand that the socket is blocking the process (which it must, since it’s waiting for the user); what I don’t understand is why the process engine does not launch P2 before being blocked by the listener. This is the behaviour I was expecting, since I understand that the socket blocks the process, but I was expecting it to get blocked in the point I start the listener, not before (and it seems to be blocked before, since P2 is never started).

Cheers,

Germán Sanchis-Trilles

[aakhmerov]

Hi @eaglecros,

I think this happens due to doBrowse, which is supposed to start process, being invoked after the server socket is getting instantiated.

Cheers,
Askar

[eaglecros]

Actually, no: doBrowse is invoked before waitForCode(), which is the method that instantiates the socket

            String redirectUri = plsr.getRedirectUri();
            AuthorizationCodeRequestUrl authorizationUrl = 
                flow.newAuthorizationUrl().setRedirectUri(redirectUri);
            doBrowse(authorizationUrl.build());
            String code = plsr.waitForCode();
            TokenResponse response = 
                flow.newTokenRequest(code).setRedirectUri(redirectUri).execute();
            credential = flow.createAndStoreCredential(response, userId);

Before writing here, I read somewhere something like that the process engine basically does stuff in the process until it reaches a wait/user task, and then looks at other tasks/process which require attention. If this is so, this would explain it, but I thought async was the way around it.

Cheers,

Germán