Processing a huge amount of workflows

Sometimes you have a huge workload to process with workflows e.g. LoudnessAnalyze an existing DigaSystem table of 1.000.000 entries.

Using a WorkflowTableWatcher approach would result in creating 1.000.000 workflows in your workflow queue. This has the following disadvantages:

• Very high load on DPE Services when creating 1.000.000 workflows 
• Querying the workflow queue will become slow
• Workflow queue is flooded and you cannot see important things in all scheduled workflows
• Daily work may have to wait until 1.000.000 workflows have executed
• It is hard to cancel 1.000.000 workflows (when you detect a problem)

We have to avoid flooding of the workflow queue.

WorkflowBlockCreator is the tool that helps. It allows to

• Break a huge workload into small blocks
• Execute the blocks independently
• Throttle the amount of concurrently scheduled workflows

LoudnessAnalyze Example

Create blocks of todos using WorkflowBlockCreator

1. Start WorkflowBlockCreator.exe, which you can inside your DPE_WorkflowServer software package
2. Enter the DPE Server Address
3. a DigaSystem User and Password
4. Specify an Output folder for block files to be created
5. and the Block Size (=amount of entries per block file)
6. DigaSystem Tables to be processed (you have to click Refresh after entering server address and credentials to get available tables)
   

   

   
   
   
7. After pressing Create Blocks you will get files describing processing blocks in the output folder:
   

   

   
   
   
8. Each file contains (up to) 500 entry ids (according your chosen Block Size).

Configure WorkflowScheduler to schedule workflows in a throttled way

A workflow that is scheduling workflows for each entry in a block file is needed, in this example LoudnessAnalyzeNextBlock.

Execution of this workflow is throttled by observing the amount of non-finished LoudnessAnalyzeEntry workflows, in this example: Watermark  is set 450 (as the block size is 500).

You should also set the scheduling interval for LoudnessAnalyzeBlock  to a value that makes sense for the size of one block.

Example: if 500 entries in a block in your environment take 10 minutes in average you could set the scheduling interval to 5 minutes.

Using more complex schedules you could for example also run the workflows at night, not conflicting with daily work:

How does a NextBlock workflow look like?

The following generic code can be reused for workflows that do not have additional arguments.

If you have to forward additional arguments to the created workflow you have to adapt the CreateWorkflow step.

Arguments

Variables

Code

Download a GenericNextBlock.wft