Let me elaborate a little on how the timer works behind the scenes which may help you with your question.
To start with, the BPMN intent of a timer should be interpreted as the timer transition will not occur before the timer due date. Often the timer construct is interpreted more like a realtime operating system, ie the timer will interrupt at precisely the due date. This is not the case.
Hence behind the scenes, the engine creates a job in the job table with a due date set to the duedate of the timer. Thus the job will remain dormant in the job table. When the system time reaches the duedate, then the job then becomes a candidate for acquisition by the job executor. Hence only when the job executor acquires the job and begins the transition will the timer construct be effected. In the interim, if the process context removes the timer, eg its a task boundary timer however the task is complete, the timer job will be removed from the job table and thus will never be run.
Hence for your use case, consider calaculating the due date taking into consideration weekends, holidays etc. Then set the timer to this due date. The engine will then take care of the rest.