杀死 Azure 数据工厂管道/触发器

我创建了几个并发设置为 1 的管道,并使用计划触发器触发了它们。但是,如果管道没有在预期时间内完成执行,则下一个运行中的触发器将排队。在这种情况下,我怎样才能杀死触发器或管道?即使我手动取消了执行,触发器/管道仍然需要额外的时间才能取消。 ADF中是否有任何方法可以直接杀死进程?

问候,桑迪普

stack overflow KILL Azure Data Factory pipeline/trigger
原文答案

答案:

作者头像

I offer two solutions both require the concurrency limit to be > 1 (cannot queue the same pipeline or will not work):

  1. Figure out if the pipeline being triggered is already running. If it is, then allow the already running pipeline to continue and kill the new one.

  2. Figure out if the pipeline triggered is already running. If it is, then kill the old one and allow the new one to continue. (the reverse of the first option)

HERE IS OPTION 1:

To figure out if the pipeline being triggered is currently running you can use the queryPipelineRuns ADF Rest API Post command. Here is an example of this below. The output comes back in the Json format under the "value" object. You can then filter out the current run, so that if anything comes back in a Queued or In-Progress state, you know that you shouldn't run this particular trigger.

1st, create the web request

Get Pipelines Running or Queued

URL: https://management.azure.com/subscriptions/@{pipeline().parameters.SubscriptionId}/resourceGroups/@{pipeline().parameters.ResourceGroupName}/providers/Microsoft.DataFactory/factories/@{pipeline().DataFactory}/queryPipelineRuns?api-version=2018-06-01

Body:

{
  "lastUpdatedAfter": "2018-06-16T00:36:44.3345758Z",
  "lastUpdatedBefore": "@{utcnow()}",
  "filters": [

{
    "operand":"PipelineName",
    "operator":"Equals",
    "values":[
        "@{pipeline().parameters.PipelineName}"
        ]
    },
{
    "operand":"Status",
    "operator":"In",
    "values":[
        "InProgress",
"Queued"
        ]
    }
  ]
}

2nd, Filter the Output of the Web Request to anything that is not your current Pipeline running using the runId of the pipeline

Web Request Output Filter

3rd, Use an IF Activity to determine if the FilteredItemCount is greater than zero, if it is then force fail the pipeline

FYI, you can use a stored procedure to create a custom error message and force an error, but in my example I just divided by zero when setting a placeholder variable value.

If Activity to Force Error if Pipeline running already

4th, after the If Activity this is really where you put whatever your pipeline actually needs to do, in my case I just used a wait activity

Where your Pipeline really starts

I tested this and it works.

HERE IS OPTION 2:

In order to cancel the pipeline that is already in progress. You can start with the 3rd step from above and instead of forcing the failure of the current pipeline, instead use the RunID from the other pipelines that are already in progress to Cancel those pipelines. After that command is completed you can continue on with your pipeline running. Remember, this method will still need concurrent runs to be at 2 or more.