Unexpected behavior trying to set job.cwd to another value in the same function

Start with the reproduction in johb_test.lua and the plenary.job entry point created by j:new(), focusing on how sync() uses the mutable cwd field between runs. Run nvim --headless -c 'PlenaryBustedFile johb_test.lua' from the reproduced project tree. Done means the relevant tests pass when the same job changes cwd between buffer directories, or the expected limitation is documented.

Tree structure of the project:

johb
├── johb.lua

├ ── johb_1
│   └── johb_1.lua
└── johb_test.lua

This test tries to:

1. Open two buffers
2. Buffer 1 = johb.lua
3. Buffer 2 = johb_1.lua
4. Call get_p:sync() with cwd set on vim.fn.expand('%:p:h')
5. Change focus from buffer 1 to buffer 2 via vim.api.nvim_set_current_buf
6. Reset the local variable cwd to vim.fn.expand('%:p:h')
7. Reset get_p.cwd with the local variable cwd
8. Call get_p:sync() with get_p.cwd set as the local variable cwd

-- johb_test.lua
--
-- [[
-- function get_p
--    Call the command 'git rev-parse --show-toplevel'
--
-- function last_item
--    Return the last item of given path.
--    foo/bar/baz => baz
-- ]]

local j = require("plenary.job")

describe("plenary.job testing", function()
  -- Set the path to reflect your local project path
  local path_to_johb = '/Users/lorenzo/.config/nvim/scratch/johb/'
  local path_to_johb_1 = '/Users/lorenzo/.config/nvim/scratch/johb/johb_1'
  local buf_count = 0
  local buf_list = {}
  vim.cmd.edit('johb.lua')
  vim.cmd.edit('johb_1/johb_1.lua')

  local last_item = function(p)
    local sp = vim.split(p, '/')
    return sp[#sp]
  end

  buf_list = vim.api.nvim_list_bufs()

  buf_count = #buf_list

  local get_p = j:new({
    'git',
    'rev-parse',
    '--show-toplevel',
  })

  before_each(function() vim.api.nvim_set_current_buf(1) end)
  it("(pre) Should open two buffer", function()
    assert.equals(2, buf_count)
  end)

  it("(pre) Buffer 1 should be 'johb.lua'", function()
    local buf_01 = vim.api.nvim_buf_get_name(1)
    assert.equals(last_item(buf_01), 'johb.lua')
  end)

  it("(pre) Buffer 2 should be 'johb_1.lua'", function()
    local buf_02 = vim.api.nvim_buf_get_name(2)

    local buf = vim.api.nvim_buf_get_name(0)
    assert.equals(last_item(buf_02), 'johb_1.lua')
  end)

  it("(pre) Should move focus to Buffer 1", function()
    local cur_buf = vim.api.nvim_buf_get_name(0)
    assert.equals(last_item(cur_buf), 'johb.lua')
  end)

  it("1. Outside of job it returns the correct path", function()
    local cwd = vim.fn.expand('%:p:h')
    assert.equals(last_item(cwd), 'johb')

    -- Move focus on johb_1.lua buffer
    vim.api.nvim_set_current_buf(2)

    -- recalculate path after moving focus to buffer 2
    cwd = vim.fn.expand('%:p:h')
    assert.equals(last_item(cwd), 'johb_1')
  end)

  it("2. Should return correct path if job.cwd is set manually", function()
    get_p.cwd = path_to_johb
    local ok, _ = get_p:sync()
    assert.equals(last_item(ok[1]), 'johb')

    -- Move focus on johb_1.lua buffer
    vim.api.nvim_set_current_buf(2)

    get_p.cwd = path_to_johb_1
    ok, _ = get_p:sync()
    assert.equals(last_item(ok[1]), 'johb_1', 'It fails when you try to set to a new cwd the job..')
  end)

  it("3. Should return  correct path after re-set get_p.cwd", function()
    local cwd = vim.fn.expand('%:p:h')[1]

    get_p.cwd = cwd
    local ok, _ = get_p:sync()
    assert.equals(last_item(ok[1]), 'johb')

    -- Move focus on johb_1.lua buffer
    vim.api.nvim_set_current_buf(2)

    -- fix: (maybe?)
    -- to print cwd you should pass
    --    cwd = vim.fn.expand('%:p:h')
    -- to pass cwd as argument to job:new().cmd you should pass
    --    cwd = vim.fn.expand('%:p:h')[1]
    --
    -- print(cwd) returns the correct path, but it pass the old one to get_p.cwd
    cwd = vim.fn.expand('%:p:h')[1]
    get_p.cwd = cwd
    ok, _ = get_p:sync()
    assert.equals(last_item(ok[1]), 'johb_1')
  end)

  it("4. Should return correct path after run a new job with new cwd", function()
    local cwd = vim.fn.expand('%:p:h')[1]

    get_p.cwd = cwd
    local ok, _ = get_p:sync()
    assert.equals(last_item(ok[1]), 'johb')

    -- Move focus on johb_1.lua buffer
    vim.api.nvim_set_current_buf(2)

    -- print(cwd) returns the correct path, but it pass the old one to get_p.cwd
    cwd = vim.fn.expand('%:p:h')[1]

    local get_p2 = j:new({
      'git',
      'rev-parse',
      '--show-toplevel',
      cwd = cwd
    })
    ok, _ = get_p2:sync()
    assert.equals(last_item(ok[1]), 'johb_1')
  end)
end)

Take a look at test 1:
I've set local cwd = vim.fn.expand('%:p:h')
Move focus to buffer 2 and reset that with the same expand
The test passed.

In tests 2, 3 I've done the same thing and set the job.cwd property to the local cwd but the test failed.
This is something strange to me because in tests 2, 3, 4 only the second part of the test fails when I reset the cwd variable

To be sure to reproduce exactly how I do:

1. Recreate the tree structure as mentioned in this issue. ( contents of johb.lua and johb_1.lua it's not important, I've leaved them both empty )
2. Replace local path_to_johb and local path_to_johb_1 with the correct path, based on your local machine
3. Run this command from the root folder:

nvim --headless -c 'PlenaryBustedFile johb_test.lua'

Running the command from the terminal returns this result:

❯ echo $PWD && git -C $PWD rev-parse --show-toplevel
/Users/lorenzo/.config/nvim/scratch/johb/johb_1 # this is the same path passed as path_to_johb
/Users/lorenzo/.config/nvim/scratch/johb/johb_1

❯ echo $PWD && git -C $PWD rev-parse --show-toplevel
/Users/lorenzo/.config/nvim/scratch/johb # this is the same path passed as path_to_johb_1
/Users/lorenzo/.config/nvim/scratch/johb

It's possible that I've missed or misunderstood something...