Cloudsql-proxy not executing in time for cloudbuild

[jbmarsha]

Problem you have encountered:
I am currently unable to connect to an instance of cloud_sql_proxy from within a cloud build, one that was setup in an earlier executed step. I am using the gce-proxy docker image. When running the cloud_build_local command line tool I was able to successfully run through the entirety of the steps defined in my cloudbuild.json file. But when I try to run this through cloud builds directly, the step doesn’t seem to finish until after the cloud build fails, no matter what order I put things in or even if I put a generous sleep step in there.

What you expected to happen:
I expected to be able to connect to the localhost of the cloud build after a previous step was run, namely trying to connect to port 5432 after a cloud_sql_proxy step was run and accepting connections on port 5432.

Steps to reproduce:
I’m not sure what I can supply here without handing over sensitive information, but here is an output of one of the builds, and each one fails in the same way:

Starting Step #4 - "proxy"
Step #0: Already have image (with digest): gcr.io/cloud-builders/gsutil
Step #4 - "proxy": Pulling image: gcr.io/cloudsql-docker/gce-proxy
Step #4 - "proxy": Using default tag: latest
Step #4 - "proxy": latest: Pulling from cloudsql-docker/gce-proxy
Step #0: / [0 files][ 0.0 B/ 157.0 B] 
/ [1 files][ 157.0 B/ 157.0 B] 
Step #0: Operation completed over 1 objects/157.0 B. 
Step #4 - "proxy": 10f5d9575767: Pull complete
Step #4 - "proxy": Digest: sha256:96689ad665bffc521fc9ac3cbcaa90f7d543a3fc6f1c84f81e4148a22ffa66e0
Step #4 - "proxy": Status: Downloaded newer image for gcr.io/cloudsql-docker/gce-proxy:latest
Finished Step #0
Starting Step #1
Step #1: Pulling image: ubuntu
Step #4 - "proxy": 2019/04/30 17:55:45 current FDs rlimit set to 1048576, wanted limit is 8500. Nothing to do here.
Step #4 - "proxy": 2019/04/30 17:55:45 Listening on 127.0.0.1:5432
Step #4 - "proxy": 2019/04/30 17:55:45 Ready for new connections
Step #1: Using default tag: latest
Step #1: latest: Pulling from library/ubuntu
Step #1: Digest: sha256:ab6cb8de3ad7bb33e2534677f865008535427390b117d7939193f8d1a6613e34
Step #1: Status: Downloaded newer image for ubuntu:latest
Finished Step #1
Starting Step #2
Step #2: Already have image (with digest): gcr.io/cloud-builders/npm
Finished Step #2
Starting Step #3 - "install"
Step #3 - "install": Already have image (with digest): gcr.io/cloud-builders/npm
Finished Step #3 - "install"
Starting Step #5
Step #5: Already have image (with digest): gcr.io/cloud-builders/npm
Step #5: Loaded configuration file "src/configs/database.js".
Step #5: ERROR: getaddrinfo ENOTFOUND host.docker.internal host.docker.internal:5432
Step #5: npm ERR! Exit status 1
Step #5: npm ERR! A complete log of this run can be found in:
Step #5: npm ERR! /builder/home/.npm/_logs/2019-04-30T17_57_54_048Z-debug.log
Finished Step #5
Finished Step #4 - "proxy"
ERROR
ERROR: build step 5 "gcr.io/cloud-builders/npm" failed: exit status 1

That line "Finished Step 4 - "proxy"" at the ends needs to happen sooner than step 5, but I can’t get it to finish any earlier than after the build has failed.

Other information (workarounds you have tried, documentation consulted, etc):
I have combed over the github issues page and the issues in the issue tracker and have found nothing that remedied the problem, I’ve found some information in stack overflow and those github issues that helped me get the local cloud build working, but nothing for the cloud build itself. We decided to give Google Cloud a try after being a primarily AWS focused company, and so far have been very pleased with the process. This however has been giving us quite the headache. I have played with changing the steps, making sure that the stepped is properly waited for, even adding some sleeps in there, and still nothing has worked.

[Carrotman42]

First thought: the proxy is configured to only accept connections from the loopback device (i.e. it says it is listening on 127.0.0.1). In the desktop world that would mean that you can't connect to it, even from the same host, unless you explicitly are connecting to that same loopback address (even the hostname wouldn't work, unless the dns resolver resolves the hostname to 127.0.0.1). So when the log later says it can't connect using "host.docker.internal host.docker.internal" as the hostname, I would expect it to fail (again, unless that hostname is literally defined as the loopback device for the precise container which is hosting the Proxy). My guess is that there's some complexity here related to exactly how the cloud build is executing things remotely: since it works locally I am guessing that either the hostnames work out somehow or that the local version isn't using containers in the exact same way the remote version is (which makes sense to me as a possible, given the security model is different). Unfortunately I don't have any good idea here other than maybe trying to mess with hostnames to get the right device to be used. Unless there are further firewall restrictions for the cloud build, I would hesitate to suggest asking the Proxy to listen on a wider interface than 127.0.0.1. Have you tried not using TCP, and using Unix sockets instead? There's some trickiness with these sockets and containers, but IIRC it works fine as long as the Proxy initializes before depending containers start up.

…

On Thu, May 9, 2019, 10:25 AM Jack Marshall ***@***.***> wrote: Problem you have encountered: I am currently unable to connect to an instance of cloud_sql_proxy from within a cloud build, one that was setup in an earlier executed step. I am using the gce-proxy docker image. When running the cloud_build_local command line tool I was able to successfully run through the entirety of the steps defined in my cloudbuild.json file. But when I try to run this through cloud builds directly, the step doesn’t seem to finish until after the cloud build fails, no matter what order I put things in or even if I put a generous sleep step in there. What you expected to happen: I expected to be able to connect to the localhost of the cloud build after a previous step was run, namely trying to connect to port 5432 after a cloud_sql_proxy step was run and accepting connections on port 5432. Steps to reproduce: I’m not sure what I can supply here without handing over sensitive information, but here is an output of one of the builds, and each one fails in the same way: Starting Step #4 <#4> - "proxy" Step #0: Already have image (with digest): gcr.io/cloud-builders/gsutil Step #4 <#4> - "proxy": Pulling image: gcr.io/cloudsql-docker/gce-proxy Step #4 <#4> - "proxy": Using default tag: latest Step #4 <#4> - "proxy": latest: Pulling from cloudsql-docker/gce-proxy Step #0: / [0 files][ 0.0 B/ 157.0 B] / [1 files][ 157.0 B/ 157.0 B] Step #0: Operation completed over 1 objects/157.0 B. Step #4 <#4> - "proxy": 10f5d9575767: Pull complete Step #4 <#4> - "proxy": Digest: sha256:96689ad665bffc521fc9ac3cbcaa90f7d543a3fc6f1c84f81e4148a22ffa66e0 Step #4 <#4> - "proxy": Status: Downloaded newer image for gcr.io/cloudsql-docker/gce-proxy:latest Finished Step #0 Starting Step #1 <#1> Step #1 <#1>: Pulling image: ubuntu Step #4 <#4> - "proxy": 2019/04/30 17:55:45 current FDs rlimit set to 1048576, wanted limit is 8500. Nothing to do here. Step #4 <#4> - "proxy": 2019/04/30 17:55:45 Listening on 127.0.0.1:5432 Step #4 <#4> - "proxy": 2019/04/30 17:55:45 Ready for new connections Step #1 <#1>: Using default tag: latest Step #1 <#1>: latest: Pulling from library/ubuntu Step #1 <#1>: Digest: sha256:ab6cb8de3ad7bb33e2534677f865008535427390b117d7939193f8d1a6613e34 Step #1 <#1>: Status: Downloaded newer image for ubuntu:latest Finished Step #1 <#1> Starting Step #2 <#2> Step #2 <#2>: Already have image (with digest): gcr.io/cloud-builders/npm Finished Step #2 <#2> Starting Step #3 <#3> - "install" Step #3 <#3> - "install": Already have image (with digest): gcr.io/cloud-builders/npm Finished Step #3 <#3> - "install" Starting Step #5 <#5> Step #5 <#5>: Already have image (with digest): gcr.io/cloud-builders/npm Step #5 <#5>: Loaded configuration file "src/configs/database.js". Step #5 <#5>: ERROR: getaddrinfo ENOTFOUND host.docker.internal host.docker.internal:5432 Step #5 <#5>: npm ERR! Exit status 1 Step #5 <#5>: npm ERR! A complete log of this run can be found in: Step #5 <#5>: npm ERR! /builder/home/.npm/_logs/2019-04-30T17_57_54_048Z-debug.log Finished Step #5 <#5> Finished Step #4 <#4> - "proxy" ERROR ERROR: build step 5 "gcr.io/cloud-builders/npm" failed: exit status 1 That line "Finished Step #4 <#4> - "proxy"" at the ends needs to happen sooner than step 5, but I can’t get it to finish any earlier than after the build has failed. Other information (workarounds you have tried, documentation consulted, etc): I have combed over the github issues page and the issues in the issue tracker and have found nothing that remedied the problem, I’ve found some information in stack overflow and those github issues that helped me get the local cloud build working, but nothing for the cloud build itself. We decided to give Google Cloud a try after being a primarily AWS focused company, and so far have been very pleased with the process. This however has been giving us quite the headache. I have played with changing the steps, making sure that the stepped is properly waited for, even adding some sleeps in there, and still nothing has worked. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#276>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAELF3645KBJQZ3OP42HR7LPURM7JANCNFSM4HL4PCRQ> .

[Carrotman42]

Also, I didn't quite understand what you meant by wanting Step 4 to finish earlier. I do not have a lot of experience with Cloud Build, so please don't just trust me on what I'm about to say: I am expecting that when a Step finishes, the associated container/processes have exited. That means I would expect that we want the step 4 to stay up until step 5 finishes (since step 5 wants to use stuff from step 4). I am interpretting the fact that step4 and step5 finish after step5 fails as meaning that when step5 fails, the build is forced to a stop and step 4 is forcefully terminated. Am I off base here?

[kurtisvg]

@jbmarsha - @Carrotman42's assessment is correct. Cloud Builds are isolated by design - you don't want something from step 4 to effect step 5. Step 4 will also timeout every time unless you stop the proxy.

If looks like you are trying to run some javascript that connects to a Cloud SQL instance. You could wrap that script with a bash script that starts the proxy in the background, runs the javascript, then stop the proxy once the javascript is completed. However, running multiple processes in a container is usually considered an anti-pattern, so you may want to use a SSL certificate instead. (I don't know which module you are using to connect, but here are the instructions for node-postgres).