GoogleContainerTools/skaffold

Mysterious Warn logs in dev loop when container is terminated and watching for changes

开放

#5,565 创建于 2021年3月17日

 (0 条评论) (0 个反应) (0 位负责人)Go (1,416 个派生)batch import
area/status-checkbugbash/q4-2022help wantedkind/frictionpriority/p2

仓库指标

星标
 (12,822 个星标)
PR 合并指标
 (平均合并 3天 6小时) (30 天内合并 16 个 PR)

描述

I am testing a scenario where users have a deployed container fails before status check phase is complete.

As expected Skaffold

  1. reports the container is terminated with exit code.
  2. Tries fetching logs which arent there.
  3. quits the status check loop and watches for changes

Unexpected:

  1. I see the following warn messages in the watching phase WARN[0046] exit status 1
  2. WARN[0046] Skipping deploy due to error: 1/2 deployment(s) failed is incorrect because the deploy already happened and we don't clean up.
Waiting for deployments to stabilize...
 - deployment/leeroy-web is ready. [1/2 deployment(s) still pending]
WARN[0045] exit status 1                                
 - deployment/leeroy-app: container leeroy-app terminated with exit code 2
    - pod/leeroy-app-794c7c586d-2qrh2: container leeroy-app terminated with exit code 2
      > Error retrieving logs for pod leeroy-app-794c7c586d-2qrh2. Try `kubectl logs leeroy-app-794c7c586d-2qrh2 -n default -c leeroy-app`
 - deployment/leeroy-app failed. Error: container leeroy-app terminated with exit code 2.
WARN[0046] Skipping deploy due to error: 1/2 deployment(s) failed 
Watching for changes...
WARN[0046] exit status 1                                
WARN[0060] exit status 1                                
WARN[0090] exit status 1  

How to reproduce

  1. Add a lag in status check so the container has time to crash before status check ends.
+++ b/pkg/skaffold/deploy/status/status_check.go
@@ -97,6 +97,7 @@ func NewStatusChecker(cfg Config, labeller *label.DefaultLabeller) Checker {
 // Run runs the status checks on deployments and pods deployed in current skaffold dev iteration.
 func (s statusChecker) Check(ctx context.Context, out io.Writer) error {
        event.StatusCheckEventStarted()
+       time.Sleep(1*time.Second)
        errCode, err := s.statusCheck(ctx, out)
        event.StatusCheckEventEnded(errCode, err)
        return err
  1. LOCAL=true make

  2. run skaffold dev

  3. After the first deploy, add a NPE to mimic container crash

--- a/examples/microservices/leeroy-app/app.go
+++ b/examples/microservices/leeroy-app/app.go
@@ -12,6 +12,7 @@ func handler(w http.ResponseWriter, r *http.Request) {
 
 func main() {
        log.Print("leeroy app server ready")
+       fmt.Fprintf(nil, "leeroooooy app!!\n")
        http.HandleFunc("/", handler)
        http.ListenAndServe(":50051", nil)
 }

贡献者指南