SwaggerResponseAttribute.IsNullable is only respected when defaultResponseReferenceTypeNullHandling is set to Null

I'm using latest (NSwag v13.8.2.0 (NJsonSchema v10.2.1.0 (Newtonsoft.Json v12.0.0.0))) version of NSwag. My controller method definition is this:

        [HttpGet("model/{sessionId}/default")]
        [SwaggerResponse(200, typeof(ModelDefinitionDto), IsNullable = false)]
        [SwaggerResponse(204, typeof(ModelDefinitionDto), IsNullable = true)]
        [SwaggerResponse(500, typeof(Exception))]
        public async Task<IActionResult> GetDefaultModel([FromRoute] Guid sessionId, [FromQuery] Guid? revisionId)
        { .... }

by default, my .nswag file has "defaultResponseReferenceTypeNullHandling": "NotNull",

so no matter what I put for SwaggerResponseAttribute.IsNullable, the swagger spec has x-nullable: false

        "responses": {
          "200": {
            "x-nullable": false,
            "description": "",
            "schema": {
              "$ref": "#/definitions/ModelDefinitionDto"
            }
          },
          "204": {
            "x-nullable": false,
            "description": "",
            "schema": {
              "$ref": "#/definitions/ModelDefinitionDto"
            }
          },
          "500": {
            "x-nullable": false,
            "description": "",
            "schema": {
              "$ref": "#/definitions/Exception"
            }
          }
        }

It only respects SwaggerResponseAttribute.IsNullable if I set "defaultResponseReferenceTypeNullHandling": "Null",

but then all reference responses becomes nullable, so I have to add SwaggerResponseAttribute to all controllers methods.