321460_en-US

liliconcom

Airflow Cloud Build Operator not executing python function

I can’t get the second arg statement to execute:

<SPAN class="hljs-string">"python"</SPAN>, <SPAN class="hljs-string">"-c"</SPAN>, <SPAN class="hljs-string">"from main import foo;print(foo('hello'))"</SPAN>

I’m triggering a Cloud Build using the Airflow operator  python function. Both the triggering DAG and Cloud Build return successful.

 

The first arg to install the requirements.txt file appears to work based on the Cloud Build returned log message:

Requirement already satisfied: setuptools>=<SPAN class="hljs-number">40.3</SPAN><SPAN class="hljs-number">.0</SPAN> <SPAN class="hljs-keyword">in</SPAN> /usr/local/lib/python3<SPAN class="hljs-number">.7</SPAN>/site-packages (<SPAN class="hljs-keyword">from</SPAN> google-auth>=<SPAN class="hljs-number">1.2</SPAN>->gcsfs>=<SPAN class="hljs-number">0.6</SPAN><SPAN class="hljs-number">.2</SPAN>

My function is using the gcsfs package. The function I have used in the question is just for demonstration.

Before using Airflow I had this working in a yaml file. Tried a number of different ways of stating the argument, such as:

<SPAN class="hljs-string">"args"</SPAN>:[<SPAN class="hljs-string">"-c"</SPAN>,<SPAN class="hljs-string">"pip install -V -r requirements.txt; python -c from main import foo;print(foo('hello'))"</SPAN>]

But now going round in circles.

I would greatly appreciate any help to get that args statement working.

This is the CloudBuildCreateBuildOperator from the DAG:

 create_build_from_storage_body = {
  <SPAN class="hljs-string">"source"</SPAN>: {
    <SPAN class="hljs-string">"repoSource"</SPAN>: {
      <SPAN class="hljs-string">"projectId"</SPAN>: <SPAN class="hljs-string">"dev-6767"</SPAN>,
      <SPAN class="hljs-string">"repoName"</SPAN>: <SPAN class="hljs-string">"bq-pipeline"</SPAN>,
      <SPAN class="hljs-string">"branchName"</SPAN>: <SPAN class="hljs-string">"master"</SPAN>,
    }
  },
  <SPAN class="hljs-string">"steps"</SPAN>: [{
    <SPAN class="hljs-string">"name"</SPAN>: <SPAN class="hljs-string">"docker.io/library/python:3.7"</SPAN>,
    <SPAN class="hljs-string">"entrypoint"</SPAN>: <SPAN class="hljs-string">"/bin/bash"</SPAN>,
    <SPAN class="hljs-string">"args"</SPAN>: [<SPAN class="hljs-string">"-c"</SPAN>, <SPAN class="hljs-string">"pip install -V -r requirements.txt"</SPAN>,
      <SPAN class="hljs-string">"python"</SPAN>, <SPAN class="hljs-string">"-c"</SPAN>, <SPAN class="hljs-string">"from main import foo;print(foo('hello'))"</SPAN>
    ],
  }],
}

 

AnalyticsData TableRe: Airflow Cloud Build Operator not executing python function

It may be that someone here can offer help, but I suspect you may have more success in a forum more related to Airflow.