[Feature Request] Production ready NextJS integration that scales

Comienza con el ejemplo existente nextjs-ecommerce-oneclick y, después, revisa el layout propuesto de workflows/, workflows/tsconfig.json, pages/api/workflows/[[...slug]].ts y los puntos de entrada del worker y del workflow descritos en el issue. Ejecuta los comandos docker-compose y npm indicados para verificar el ejemplo localmente; se considera terminado cuando una integración escalable de Next.js puede añadir colas y workflows y operar mediante las rutas de API documentadas.

Is your feature request related to a problem? Please describe.

First, to clarify. All examples in this repo are perfect for what they were created to demonstrate, that is most minimalistic implementation of specific feature and is very high value for us all. I am just proposing that we add one more advanced example that will make easier for developers to jump into Temporal fully configured and just start using it. Many developers just give up because setting up stuff for NextJS/Temporal is not within reach of all devs out there.

Figuring out state machines is hard. Figuring out that Temporal workflow instance is a "living statemachine" inside Temporal server is hard. Figuring out Temporal is hard. Figuring out how to integrate Temporal into NextJS properly is hard. Then figuring out how to compile TS to JS so that independent worker can use it is another step to do. Your brain will melt during your voyage thru the matter.

Describe the solution you'd like

Add Temporal integration based on existing "One Click Buy" but one that allows user to easily add more workflows and queues without brain damage during the process.

Additional context

Steps to create tight integration with NextJS that scales:

1. use this example as starting point https://github.com/temporalio/samples-typescript/tree/main/nextjs-ecommerce-oneclick (then just rewire 2 api calls to use new folder structure proposed below, and all should worK)

2. look at screenshot to get better understanding of folder structure that is being proposed. all files (almost) live in ./workflows directory only inside nextjs app

3. read explanation why this folder structure is choosen, then get source code for each file below

• ./workflows/ - all files for Temporal are only here (except single dynamic API endpoint that maps to actions)
  • ./tsconfig.json - typescript compiler config that will be used to convert .ts to .js as source code for worker instance
  • ./__worker_source__ - here we compile .ts to .js and this is folder that will worker directly use to start itselft, pure .js files
  • ./queue1/ is sample naming for first queue, inside it we have 3 files that will be directly consumed by worker instance
    • ./activities - exports all activities for worker to import
    • ./workflows - exports all workflows for worker to import
    • ./worker - actual code that represents worker instance (the file that will be run as worker)
    • ./oneClickBuy/ - folders are only used for defining workflows that contain this file structure
      • ./_workflow-1.ts - first version of workflow definition
      • ./_workflow-2.ts - second revision of workflow definition
      • ./_workflow-current.ts - current revision of workflow definition (we are creating revisions of the same workflow during time, we need to keep all previous versions of workflow definitions that still have at least 1 instance running inside temporal server, so that we can compare old and new workflow definition and make change in latest current so that older workflow instances can work. if no instances of previous revisions exists, they are deleted or after implementation of new)
      • ./CANCEL_act.ts - code that will worker execute against external internet resources if needed
      • ./CANCEL_api.ts - processes request that we got from /api/workflows/queue1/oneClickBuy/CANCEL request
      • ./GETSTATE_api.ts - processes request that we got from /api/workflows/queue1/oneClickBuy/GETSTATE request
      • ./START_act.ts - code that will worker execute against external internet resources if needed
      • ./START_api.ts - processes request that we got from /api/workflows/queue1/oneClickBuy/START request

4. find out about reasons behind choosing names for files

Anyone familiar with state machines (ie. Xstate) and used it in production systems both front and backend or has used Redux as concept to simplify state management, will be able to recognize pattern here.

With UPPERCASE letters we write "events" that interact with our workflow instance. The workflow instance template is in _workflow-current.ts file and START_api.ts will be responsible to create instance of that workflow in temporal server.

Now reasoning about the code is much much easier. We define workflow and we define events that interact with the instance of workflow that lives in Temporal server, very easy. Each interaction with any queue / workflow / event is as easy as triggering /api/workflows/queue1/oneClickBuy/START and that is it, it can be consumed from nextjs app or from external system.

5. look at the list of commands to execute after all is set then move to next steps

• docker-compose -f ./docker-compose-temporal.yml up -d to start temporal server on local machine
• npm run temporaldev - to start typescript compiler that will read ./workflows and create .js
• npm run temporalworkerqueue1 - to start instance of worker
• npm run dev - to run nextjs dev server
• then goto localhost:3000 and goto page to click on products to order
• when you click to buy item, then goto http://localhost:8088 Temporal web interface and check if all works

6. copy paste code into nextjs app

./package.json in scripts segment

    "temporaldev": "tsc --build --watch ./workflows/tsconfig.json",
    "temporalbuild": "tsc --build ./workflows/tsconfig.json",
    "temporalworkerqueue1": "node ./workflows/__worker_source__/queue1/worker.js",

/pages/api/workflows/[[...slug]].ts

// file with this contents, which will be universal endpoint for all actions that we will define in /workflows directory later.
import { NextApiRequest, NextApiResponse } from 'next';

// This dynamic api endpoint will use path provided to hit specific queue/workflow/activity file
// ie. https://SITENAME.com/api/workflows/queue1/oneClickBuy/CANCEL

export default async (req: NextApiRequest, res: NextApiResponse): Promise<any> => {
  const { slug } = req.query || {};
  const [queue, workflow, activity] = slug ? (Array.isArray(slug) ? slug : [slug]) : [];
  const DynamicAPILoader = await import(`@/workflows/${queue}/${workflow}/${activity}_api.ts`);
  if (DynamicAPILoader.default) {
    const [ERRreturnJSON, returnJSON] = await DynamicAPILoader.default({ req, res })
      .then((r: any) => [null, r])
      .catch((e: any) => [e]);
    if (ERRreturnJSON) {
      return res.status(500).json({ errors: [{ error: ERRreturnJSON }] });
    }
    return res.status(200).json(returnJSON);
  }
  return res.status(500).json({ error: 'No API endpoint!' });
};

./workflows/tsconfig.json

{
  "extends": "@tsconfig/node16/tsconfig.json",
  "version": "4.4.2",
  "compilerOptions": {
    "declaration": false,
    "declarationMap": false,
    "sourceMap": true,
    "rootDir": "./",
    "outDir": "./__worker_source__",
    "noImplicitAny": false
  },
  "include": ["**/*.ts"]
}

./workflows/workflows.ts

import { oneClickBuy } from './oneClickBuy/_workflow-current';

export { oneClickBuy };

./workflows/activities.ts

import { oneClickBuy_CANCEL_act } from './oneClickBuy/CANCEL_act';
import { oneClickBuy_START_act } from './oneClickBuy/START_act';

export {
  oneClickBuy_CANCEL_act,
  oneClickBuy_START_act,
};

./workflows/worker.ts

import { Worker } from '@temporalio/worker'; //eslint-disable-line
import * as workerActivities from './activities';

async function run() {
  const worker = await Worker.create({
    taskQueue: `queue1`,
    workflowsPath: require.resolve('./workflows'),
    activities: workerActivities,
  });
  await worker.run();
}
run().catch((err) => {});

./workflows/queue1/_workflow-current.ts

// DEFAULT WORKFLOW DISABLE ESLINT RULES
/* eslint-disable no-void */
/* eslint-disable no-return-assign */
/* eslint-disable no-return-await */
// eslint-disable-next-line
import * as wf from '@temporalio/workflow';

// ACTIVITIES
import type * as activities from '../activities';
// activity configuratino and exporting variables
const { oneClickBuy_CANCEL_act, oneClickBuy_START_act } = wf.proxyActivities<typeof activities>({
  startToCloseTimeout: '1 minute',
});

// STATES
type PurchaseState = 'PURCHASE_PENDING' | 'PURCHASE_CONFIRMED' | 'PURCHASE_CANCELED';

// SIGNALS
export const cancelPurchase = wf.defineSignal('cancelPurchase');
export const purchaseStateQuery = wf.defineQuery<PurchaseState>('purchaseState');

// WORKFLOW
export async function oneClickBuy(props) {
  const { itemId } = props || {};
  const itemToBuy = itemId;
  let purchaseState: PurchaseState = 'PURCHASE_PENDING';

  // ADD HANDLERS < called with: await workflow.signal('cancelPurchase');
  wf.setHandler(cancelPurchase, () => void (purchaseState = 'PURCHASE_CANCELED'));
  wf.setHandler(purchaseStateQuery, () => purchaseState);

  // WORKFLOW CODE
  if (await wf.condition(() => purchaseState === 'PURCHASE_CANCELED', '5s')) {
    return await oneClickBuy_CANCEL_act(itemToBuy);
  }
  purchaseState = 'PURCHASE_CONFIRMED';
  return await oneClickBuy_START_act(itemToBuy);
}

./workflows/queue1/CANCEL_act.ts

export async function oneClickBuy_CANCEL_act(itemId: string): Promise<string> {
  return `canceled purchase ${itemId}!`;
}

./workflows/queue1/CANCEL_api.ts

// eslint-disable-next-line
import { Connection, WorkflowClient } from '@temporalio/client';

export default async function oneClickBuy_CANCEL_api({ req, res }) {
  const { id } = req?.query || {};
  if (!id) {
    res.status(405).send({ message: 'must send workflow id to cancel' });
    return;
  }

  try {
    const connection = new Connection({ address: process?.env?.TEMPORAL_SERVER || 'http://localhost:7233' });
    const client = new WorkflowClient(connection.service, { namespace: 'default' });
    const workflow = client.getHandle(id);
    await workflow.signal('cancelPurchase');

    res.status(200).json({ cancelled: id });
  } catch (e: any) {
    res.status(500).send({ message: e?.details, errorCode: e?.code });
  }
}

./workflows/queue1/GETSTATE_api.ts

// eslint-disable-next-line
import { Connection, WorkflowClient } from '@temporalio/client';

export default async function oneClickBuy_GETSTATE_api({ req, res }) {
  const { id } = req?.query || {};
  // console.log({ id });
  if (!id) {
    res.status(405).send({ message: 'must send workflow id to query' });
    return;
  }

  try {
    const connection = new Connection({ address: process?.env?.TEMPORAL_SERVER || 'http://localhost:7233' });
    const client = new WorkflowClient(connection.service, { namespace: 'default' });
    const workflow = client.getHandle(id);
    const purchaseState = await workflow.query('purchaseState');

    res.status(200).json({ purchaseState });
  } catch (e: any) {
    res.status(500).send({ message: e?.details, errorCode: e?.code });
  }
}

./workflows/queue1/CANCEL_act.ts

export async function oneClickBuy_START_act(itemId: string): Promise<string> {
  return `checking out ${itemId}!`;
}

./workflows/queue1/CANCEL_act.ts

// eslint-disable-next-line
import { Connection, WorkflowClient } from '@temporalio/client';
import { oneClickBuy as wtf } from './_workflow-current';

export default async function oneClickBuy_START_api({ req, res }) {
  if (req.method !== 'POST') {
    res.status(405).send({ message: 'Only POST requests allowed' });
    return;
  }
  const { itemId, transactionId } = req.body;
  if (!itemId) {
    res.status(405).send({ message: 'must send itemId to buy' });
    return;
  }

  const connection = new Connection({ address: 'localhost:7233' });
  const client = new WorkflowClient(connection.service, {
    namespace: 'default',
  });
  await client
    .start(wtf, {
      taskQueue: 'queue1',
      workflowId: transactionId,
      args: [{ itemId }], // only add params inside object, first arr element
    })
    .then((r) => {
      // console.log({ r });
    })
    .catch((e) => {
      // REPORT TO SENTRY HERE
      // console.log({ e });
    });

  res.status(200).json({ ok: true });
}

ADDITIONAL FILES NEEDED

./docker-compose-temporal.yml

#
#
#    docker-compose -f ./docker-compose-temporal.yml up -d
#    docker-compose -f ./docker-compose-temporal.yml down
#
#
version: "3.5"
services:
  postgresql:
    container_name: temporal-postgresql
    environment:
      POSTGRES_PASSWORD: temporal
      POSTGRES_USER: temporal
    image: postgres:13
    networks:
      - temporal-network
    ports:
      - 5432:5432
  temporal:
    container_name: temporal
    depends_on:
      - postgresql
      - elasticsearch
    environment:
      - DB=postgresql
      - DB_PORT=5432
      - POSTGRES_USER=temporal
      - POSTGRES_PWD=temporal
      - POSTGRES_SEEDS=postgresql
      - DYNAMIC_CONFIG_FILE_PATH=config/.docker-compose/temporal-dev-es.yaml
      - ENABLE_ES=true
      - ES_SEEDS=elasticsearch
      - ES_VERSION=v7
    image: temporalio/auto-setup:1.14.0
    networks:
      - temporal-network
    ports:
      - 7233:7233
    volumes:
      - ./.docker-compose:/etc/temporal/config/.docker-compose
  temporal-admin-tools:
    container_name: temporal-admin-tools
    depends_on:
      - temporal
    environment:
      - TEMPORAL_CLI_ADDRESS=temporal:7233
    image: temporalio/admin-tools:1.14.0
    networks:
      - temporal-network
    stdin_open: true
    tty: true
  temporal-web:
    container_name: temporal-web
    depends_on:
      - temporal
    environment:
      - TEMPORAL_GRPC_ENDPOINT=temporal:7233
      - TEMPORAL_PERMIT_WRITE_API=true
    image: temporalio/web:1.13.0
    networks:
      - temporal-network
    ports:
      - 8088:8088
  elasticsearch:
    container_name: temporal-elasticsearch
    environment:
      - cluster.routing.allocation.disk.threshold_enabled=true
      - cluster.routing.allocation.disk.watermark.low=512mb
      - cluster.routing.allocation.disk.watermark.high=256mb
      - cluster.routing.allocation.disk.watermark.flood_stage=128mb
      - discovery.type=single-node
      - ES_JAVA_OPTS=-Xms100m -Xmx100m
    image: elasticsearch:7.10.1
    networks:
      - temporal-network
    ports:
      - 9200:9200
networks:
  temporal-network:
    driver: bridge
    name: temporal-network

./.docker-compose/temporal-dev-es.yaml

frontend.enableClientVersionCheck:
- value: true
  constraints: {}
history.persistenceMaxQPS:
- value: 3000
  constraints: {}
frontend.persistenceMaxQPS:
- value: 3000
  constraints: {}
frontend.historyMgrNumConns:
- value: 10
  constraints: {}
frontend.throttledLogRPS:
- value: 20
  constraints: {}
history.historyMgrNumConns:
- value: 50
  constraints: {}
history.defaultActivityRetryPolicy:
- value:
    InitialIntervalInSeconds: 1
    MaximumIntervalCoefficient: 100.0
    BackoffCoefficient: 2.0
    MaximumAttempts: 0
history.defaultWorkflowRetryPolicy:
- value:
    InitialIntervalInSeconds: 1
    MaximumIntervalCoefficient: 100.0
    BackoffCoefficient: 2.0
    MaximumAttempts: 0
system.advancedVisibilityWritingMode:
  - value: "on"
    constraints: {}
system.enableReadVisibilityFromES:
  - value: true
    constraints: {}

./docker-compose/temporal-dev.yaml

frontend.enableClientVersionCheck:
- value: true
  constraints: {}
history.persistenceMaxQPS:
- value: 3000
  constraints: {}
frontend.persistenceMaxQPS:
- value: 3000
  constraints: {}
frontend.historyMgrNumConns:
- value: 10
  constraints: {}
frontend.throttledLogRPS:
- value: 20
  constraints: {}
history.historyMgrNumConns:
- value: 50
  constraints: {}
history.defaultActivityRetryPolicy:
- value:
    InitialIntervalInSeconds: 1
    MaximumIntervalCoefficient: 100.0
    BackoffCoefficient: 2.0
    MaximumAttempts: 0
history.defaultWorkflowRetryPolicy:
- value:
    InitialIntervalInSeconds: 1
    MaximumIntervalCoefficient: 100.0
    BackoffCoefficient: 2.0
    MaximumAttempts: 0
system.advancedVisibilityWritingMode:
  - value: "off"
    constraints: {}