GoogleContainerTools/skaffold

Mysterious Warn logs in dev loop when container is terminated and watching for changes

開放

#5,565 建立於 2021年3月17日

 (0 則留言) (0 個反應) (0 位負責人)Go (1,416 個分叉)batch import
area/status-checkbugbash/q4-2022help wantedkind/frictionpriority/p2

倉庫指標

星標
 (12,822 顆星)
PR 合併指標
 (平均合併 3天 6小時) (30 天內合併 16 個 PR)

描述

I am testing a scenario where users have a deployed container fails before status check phase is complete.

As expected Skaffold

  1. reports the container is terminated with exit code.
  2. Tries fetching logs which arent there.
  3. quits the status check loop and watches for changes

Unexpected:

  1. I see the following warn messages in the watching phase WARN[0046] exit status 1
  2. WARN[0046] Skipping deploy due to error: 1/2 deployment(s) failed is incorrect because the deploy already happened and we don't clean up.
Waiting for deployments to stabilize...
 - deployment/leeroy-web is ready. [1/2 deployment(s) still pending]
WARN[0045] exit status 1                                
 - deployment/leeroy-app: container leeroy-app terminated with exit code 2
    - pod/leeroy-app-794c7c586d-2qrh2: container leeroy-app terminated with exit code 2
      > Error retrieving logs for pod leeroy-app-794c7c586d-2qrh2. Try `kubectl logs leeroy-app-794c7c586d-2qrh2 -n default -c leeroy-app`
 - deployment/leeroy-app failed. Error: container leeroy-app terminated with exit code 2.
WARN[0046] Skipping deploy due to error: 1/2 deployment(s) failed 
Watching for changes...
WARN[0046] exit status 1                                
WARN[0060] exit status 1                                
WARN[0090] exit status 1  

How to reproduce

  1. Add a lag in status check so the container has time to crash before status check ends.
+++ b/pkg/skaffold/deploy/status/status_check.go
@@ -97,6 +97,7 @@ func NewStatusChecker(cfg Config, labeller *label.DefaultLabeller) Checker {
 // Run runs the status checks on deployments and pods deployed in current skaffold dev iteration.
 func (s statusChecker) Check(ctx context.Context, out io.Writer) error {
        event.StatusCheckEventStarted()
+       time.Sleep(1*time.Second)
        errCode, err := s.statusCheck(ctx, out)
        event.StatusCheckEventEnded(errCode, err)
        return err
  1. LOCAL=true make

  2. run skaffold dev

  3. After the first deploy, add a NPE to mimic container crash

--- a/examples/microservices/leeroy-app/app.go
+++ b/examples/microservices/leeroy-app/app.go
@@ -12,6 +12,7 @@ func handler(w http.ResponseWriter, r *http.Request) {
 
 func main() {
        log.Print("leeroy app server ready")
+       fmt.Fprintf(nil, "leeroooooy app!!\n")
        http.HandleFunc("/", handler)
        http.ListenAndServe(":50051", nil)
 }

貢獻者指南