Workers - overview

info

tl;dr: Genearlly, web requests should fetch data from the database and serialize that data in the response. Hard work should be sent to a background job—either by backgrounding it explicitly or setting up a scheduled job. Backgrounding work provides:

1. faster web responses
2. automatic retry of work that fails
3. rate-limiting certain types of work (when using named workstreams—requires a BullMQ Pro license)

A web application is like an ecosystem, with many different players interacting and responding to eachother's behavior. Much of this is done in the form of api requests being triggered from clients to your backend application servers, but some things are not. For example, you may find yourself needing a piece of code that runs every hour of every day, or something that just runs on the first monday of each 3rd month.

Additionally, it is important to keep your web servers fluid. You don't want to expose any endpoints that respond slowly, because, while node concurrency is helpful, you don't want to lean on it to save you from slow endpoint responses, since this makes you vulnerable to DDoS.

to alleviate the pressure on your web application server, you can offload costly or process-intensive services to background processes, so that your web server can continue to be reliable and performant. Additionally, you can use cron-like features within Psychic to schedule your jobs to run at specific times, such as the third tuesday of the month at noon UTC.

Logging in Background Jobs

Background methods can optionally receive a Job parameter as their last argument to access logging functionality:

import { Job } from 'bullmq'

class DataProcessingService extends ApplicationBackgroundedService {
  public static async processLargeDataset(datasetId: string) {
    await this.background('_processLargeDataset', datasetId)
  }

  public static async _processLargeDataset(datasetId: string, job: Job) {
    const dataset = await Dataset.findOrFail(datasetId)

    await job.log(`Starting processing of dataset ${datasetId}`)

    let processedRows = 0
    let invalidRows = 0

    for (const row of dataset.rows) {
      try {
        await this.processRow(row)
        processedRows++

        if (processedRows % 100 === 0) {
          await job.log(`processedRows: ${processedRows}, invalidRows: ${invalidRows}`)
        }
      } catch (error) {
        invalidRows++
        await job.log(`Invalid row ${row.id}: ${error.message}`)
      }
    }

    await job.log(`Completed: processedRows: ${processedRows}, invalidRows: ${invalidRows}`)
  }
}

The Job parameter is optional and always comes last in the method signature. Job logs are accessible through the BullMQ dashboard and can be retrieved programmatically.

tip

• To learn how to install Psychic Workers, see our installation guide
• To learn how to configure Psychic Workers, see our config guide
• To learn how to use background services, see our services guide
• To learn how to use backgrounding functions within models, see our models guide
• To learn how to schedule your services to run at specific times, see our scheduled guide