`sls deploy --package` deletes .serverless directory

For some reason, the deploy --package option causes serverless to delete the .serverless directory, even when nothing has changed or gets deployed. This behavior is unexpected, because sls deploy does not nuke the directory.

In particular, this is bad when a specially built .serverless directory is built, intending to be deployed by serverless, and the command is:

sls deploy --package .serverless

In that case, the specially built zipfiles are also deleted along with the directory!

The more common case, comparing sls deploy and sls deploy --package temp (with a copy of .serverless dir:

$ ls .serverless/
cloudformation-template-update-stack.json	dumblogger.zip					serverless-state.json				signer.zip					wrapper.zip
$ sls deploy --verbose
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Excluding development dependencies...
Serverless: Service files not changed. Skipping deployment...
Service Information
service: signer
stage: dev
region: us-east-1
stack: signer-dev
api keys:
  None
endpoints:
  None
functions:
  dumblogger: signer-dev-dumblogger
  wrapper: signer-dev-wrapper
  signer: signer-dev-signer
layers:
  None

Stack Outputs...
ServerlessDeploymentBucketName: ...

$ ls .serverless/
cloudformation-template-update-stack.json	dumblogger.zip					serverless-state.json				signer.zip					wrapper.zip
$ cp -R .serverless temp
$ sls deploy --verbose --package temp
Serverless: Service files not changed. Skipping deployment...
Service Information
service: signer
stage: dev
region: us-east-1
stack: signer-dev
api keys:
  None
endpoints:
  None
functions:
  dumblogger: signer-dev-dumblogger
  wrapper: signer-dev-wrapper
  signer: signer-dev-signer
layers:
  None

Stack Outputs...
ServerlessDeploymentBucketName: ...

$ ls .serverless/
ls: .serverless/: No such file or directory

Expected behavior: deployment never deletes the .serverless directory, regardless of options used.

Solution proposal:

(Added by maintainers):

Issue is driven by internal processing flow, in which:

• When packaging, deployment artifacts are always packaged to service local .serverless directory, and then eventually moved to target output directory (as indicated by --package)
• When deploying, deployment artifacts are always moved to local .serverless directory and deployment always happens from there.

This design doesn't seem great, it definitely introduces side effects. I believe it can be fixed (but that won't be straightforward). Ideally if:

• Framework by default builds to some real temporary directory (in scope of system temp dir), and then moves generated build to target package dir (by default .serverless)
• Framework deploys from package directory directly. If custom is provided, files are not moved to .serverless dir but deployed directly from there.

I believe both problems can be addressed independently, spec on how to tackle this:

A. Build to system temporary directory

1.1 Preliminary refactor steps (should be addressed, before tackling next step. Ideally first PR should handle just those)

Done with #8533

Tests

If any test breaks and fix is not straightforward, ideally we should refactor given test files to runServerless variant. That again should be addressed with different PR and spec for refactor written upfront (like one proposed here: https://github.com/serverless/serverless/issues/8523) - if there's a need let me know, and I'll provide a needed spec.

1.2 Implementation

1. In Serverless class configure on Serverless.prototype lazy packageBuildTempDirDeferred property, which should simply return a temp dir as resolved by process-utils/tmpdir/provision (we need to also move this dependency from devDependencies to dependencies then)

2. In all internal logic where we put to be packaged files into .serverless, ensure they're put into await packageBuildTempDirDeferred:

   1. In custom resource builder, ensure zipFilePath puts archive in newly designated temp dir
   2. Ensure that core CF template is generated to newly designated temp dir
   3. Ensure that compiled CF template is generated to newly designated temp dir
   4. Ensure that service state is stored in newly designated temp dir
   5. Ensure artifacts are zipped into newly designated temp dir

3. In moveArtifactsToPackage() ensure that content of await packageBuildTempDirDeferred is moved to desired package directory (by default .serverless) which is emptied upfront

4. Remove cleanupTempDir() module, and deprecate associated hooks (it was dedicated to cleaning up .serverless dir for temporary build)

5. Update test fixture plugin to respect newly designed temp dir

Tests

This should not require any new tests to be written, but due to nature of our tests, it's likely that some will break We should not attempt to fix broken tests (in the form they are written), but instead they have to be refactored to runServerless variant, and that again should be addressed with different PR with spec for refactor written upfront (like one proposed here: https://github.com/serverless/serverless/issues/8523). I'll provide a needed spec, but let's first clarify which test files may need it

Additionally confirm that nothing got broken by running npm run integration-test-run-package and all our integration tests

B. Ensure deployment is made directly from package folder

1. Remove moveArtifactsToTemp() and deprecate associated hooks
2. In checkIfDeploymentIsNecessary() refer to resolved package path (not hardcoded .serverless path)
3. In extendedValidate() refer to resolved package path (not hardcoded .serverless path)
4. in uploadCustomResources() refer to resolved package path (not hardcoded .serverless path)

Tests

This should not require any new tests to be written, but due to nature of our tests, it's likely that some will break We should not attempt to fix broken tests (in the form they are written), but instead they have to be refactored to runServerless variant, and that again should be addressed with different PR with spec for refactor written upfront (like one proposed here: https://github.com/serverless/serverless/issues/8523). I'll provide a needed spec, but let's first clarify which test files may need it

Additionally confirm that nothing got broken by running npm run integration-test-run-package and all our integration tests