DataFactoryTestingFrameworkExpressionsEvaluator adds status property even if None is set.
Assessment
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Newbie friendliness
- 68/100
- Issue type
- Bug
- Clarity
- Clearly specified
- Activity status
- Stale
- Tech stack
- azure, python
- Domain
- testing-qa
Research direction
Open src/data_factory_testing_framework/_pythonnet/data_factory_testing_framework_expressions_evaluator.py around line 40 and inspect how activity result properties are added. Reproduce the expression-evaluation case described in the issue, then verify that properties set to None are omitted while non-None properties remain available.
Written by the indexing model from the issue text.
Description
We have scenarios where certain Activity outputs (like status as described in #162 ) are not present on the Activity Outputs.
Currently neither functional tests nor activity tests can capture these scenarios because we add the status always to the activity results (see https://github.com/microsoft/data-factory-testing-framework/blob/main/src/data_factory_testing_framework/_pythonnet/data_factory_testing_framework_expressions_evaluator.py#L40).
The expression evaluator returns a error we can handle to mimic the behaviour:
> raise Exception(f"Unknown error: {e}") from e
E Exception: Unknown error: The template language expression 'contains(
E createArray(
E activity('Fail').status,
E activity('Notebook').status
E ),
E 'Failed'
E )' cannot be evaluated because property 'status' doesn't exist, available properties are 'output, error'. Please see https://aka.ms/logicexpressions for usage details.
E at Microsoft.Azure.Workflows.Templates.Expressions.TemplateFunctionTextExpression.SelectObjectProperty(JObject root, JToken property, Boolean safeDereference)
E at Microsoft.Azure.Workflows.Templates.Expressions.TemplateFunctionTextExpression.SelectProperty(JToken root, TemplateDereferenceExpression[] dereferences, TemplateExpressionEvaluationContext context, Boolean failOnError)
E at Microsoft.Azure.Workflows.Templates.Expressions.TemplateFunctionTextExpression.EvaluateRootExpression(TemplateExpressionEvaluationContext context, Boolean failOnError, ISkippableEnumerator`1 parameterValues, Int32 parametersCount)
E at Microsoft.Azure.Workflows.Templates.Expressions.TemplateFunctionExpression.EvaluateExpression(TemplateExpressionEvaluationContext context, Boolean failOnError)
E at Microsoft.Azure.Workflows.Templates.Expressions.TemplateFunctionExpression.LazyParameterEnumerator.MoveNext(Int32 skip)
E at Microsoft.Azure.Workflows.Templates.ISkippableEnumeratorExtensions.ToEnumerable[T](ISkippableEnumerator`1 enumerator)+MoveNext()
E at System.Collections.Generic.LargeArrayBuilder`1.AddRange(IEnumerable`1 items)
E at System.Collections.Generic.EnumerableHelpers.ToArray[T](IEnumerable`1 source)
E at Microsoft.Azure.Workflows.Templates.Expressions.TemplateExpressionBuiltInFunctions.EvaluateFunction(String functionName, ISkippableEnumerator`1 parameters, Int32 parametersCount, Boolean preserveAnnotations)
E at Microsoft.Azure.Workflows.Templates.Expressions.TemplateExpressionEvaluationHelper.EvaluateFunction(String functionName, ISkippableEnumerator`1 parameters, Int32 parametersCount)
E at Microsoft.Azure.Workflows.Templates.Expressions.TemplateFunctionTextExpression.EvaluateRootExpression(TemplateExpressionEvaluationContext context, Boolean failOnError, ISkippableEnumerator`1 parameterValues, Int32 parametersCount)
E at Microsoft.Azure.Workflows.Templates.Expressions.TemplateFunctionExpression.EvaluateExpression(TemplateExpressionEvaluationContext context, Boolean failOnError)
E at Microsoft.Azure.Workflows.Templates.Expressions.TemplateFunctionExpression.LazyParameterEnumerator.MoveNext(Int32 skip)
E at Microsoft.Azure.Workflows.Templates.ISkippableEnumeratorExtensions.ToEnumerable[T](ISkippableEnumerator`1 enumerator)+MoveNext()
E at System.Collections.Generic.LargeArrayBuilder`1.AddRange(IEnumerable`1 items)
E at System.Collections.Generic.EnumerableHelpers.ToArray[T](IEnumerable`1 source)
E at Microsoft.Azure.Workflows.Templates.Expressions.TemplateExpressionBuiltInFunctions.EvaluateFunction(String functionName, ISkippableEnumerator`1 parameters, Int32 parametersCount, Boolean preserveAnnotations)
E at Microsoft.Azure.Workflows.Templates.Expressions.TemplateExpressionEvaluationHelper.EvaluateFunction(String functionName, ISkippableEnumerator`1 parameters, Int32 parametersCount)
E at Microsoft.Azure.Workflows.Templates.Expressions.TemplateFunctionTextExpression.EvaluateRootExpression(TemplateExpressionEvaluationContext context, Boolean failOnError, ISkippableEnumerator`1 parameterValues, Int32 parametersCount)
E at Microsoft.Azure.Workflows.Templates.Expressions.TemplateFunctionExpression.EvaluateExpression(TemplateExpressionEvaluationContext context, Boolean failOnError)
E at Microsoft.Azure.Workflows.Templates.Helpers.TemplateExpressionsHelper.EvaluateTemplateLanguageExpression(String expression, TemplateExpressionEvaluationContext evaluationContext)
E at Microsoft.Azure.DataFactoryTestingFramework.Expressions.Evaluator.EvaluateExpression(String expression, String parametersJson, String variablesJson, String itemValueJson, String activityValuesJson)
E at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
E at System.Reflection.MethodBaseInvoker.InvokeWithManyArgs(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
Proposed solution:
Only add the add properties if they are set to anything other than None.
- Dominant language
- Python
- Stars
- 135
- Forks
- 43
- Avg merge
- 1h 14m
- Merged PRs (30d)
- 1
Getting set up
Starts the project's dev container in your browser, under your own GitHub account.
- No Dockerfile or Docker Compose file
- No pull request template
- No contributing guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from microsoft/data-factory-testing-framework
-
A new ExpressionRuntime (Lark grammar compile) per evaluated expression makes evaluation ~5x slowerOpen
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
IfCondition evaluates the expressions of the branch not taken (case-sensitive "activities" check)Open
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
-
Difficulty 3/5 1-2 days Newbie friendliness 35/100
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
All issues in microsoft/data-factory-testing-framework
Similar issues
-
needs-human needs-triage
Difficulty 2/5 1-3 hours Newbie friendliness 76/100
gke-labs/kube-agents#2400 · 1 comment ·
Maintainers usually reply within 1 day
-
Device Details tables: FS/SF columns contradict each other (nfet_01v8 Vt row, pfet_01v8 Idsat row)Open
Difficulty 2/5 1-3 hours Newbie friendliness 75/100
google/skywater-pdk#450 ·
-
Drained trajectory arrays are overwritten when the sequence buffer is reusedPossibly taken @sylvesterkaczmarek claimed this today. Open
Difficulty 2/5 1-3 hours Newbie friendliness 78/100
google-deepmind/bsuite#56 ·
-
Difficulty 2/5 1-3 hours Newbie friendliness 82/100
LearningCircuit/local-deep-research#7206 ·
Maintainers usually reply within 1 day
-
Difficulty 2/5 1-3 hours Newbie friendliness 68/100
chingu-voyages/V62-tier3-team-33#285 ·
Maintainers usually reply within 1 day