Use the browser

browser.run(fn, data) starts a new browser, opens one page, calls fn({ page, data }), closes the browser, and returns what fn returned.

• fn is sent to the browser as text. It cannot see variables, helpers, or packages from your node. Put everything it needs into data.
• data and the return value travel as JSON. Return plain objects, arrays, strings, numbers, and booleans. A Date arrives as a string.
• page is a Puppeteer page (Puppeteer 24). page.browser() is the browser it belongs to.
• Every call starts empty: no cookies, no localStorage, no login from an earlier call. Carry them over yourself, as shown below.
• The page is 800×600 pixels. Change it with page.setViewport. The user agent says HeadlessChrome. Set another one with page.setUserAgent before goto.
• One call may run for 10 minutes. The whole node also stops after 10 minutes. goto, waitForSelector, and locator give up after 30 seconds unless you pass timeout.
• Each call costs 1 credit, plus 1 credit per started 5 seconds. A call that fails costs the same. The node's own time is counted on top.

async ({ browser }) => {
  return await browser.run(
    async ({ page, data }) => {
      await page.goto(data.url, { waitUntil: "domcontentloaded" })
      return {
        url: page.url(),
        title: await page.title(),
        heading: await page.$eval("h1", (h1) => h1.textContent.trim()),
      }
    },
    { url: "https://quotes.toscrape.com" },
  )
}

page.$eval(selector, callback) runs callback inside the website on the first match and returns its result. That callback cannot see data either. Pass a value as an extra argument: page.$eval("h1", (el, max) => el.textContent.slice(0, max), data.max).

async ({ browser }) => {
  return await browser.run(
    async ({ page, data }) => {
      await page.goto(data.url)
      return await page.$$eval(".quote", (quotes) =>
        quotes.map((quote) => ({
          text: quote.querySelector(".text").textContent,
          author: quote.querySelector(".author").textContent,
          authorUrl: quote.querySelector("a[href^='/author/']").href,
          tags: [...quote.querySelectorAll(".tag")].map(
            (tag) => tag.textContent,
          ),
        })),
      )
    },
    { url: "https://quotes.toscrape.com" },
  )
}

page.$$eval gets every match at once. Build the objects inside the callback, then return them. href on a link is already the full URL.

async ({ browser }) => {
  return await browser.run(
    async ({ page, data }) => {
      await page.goto(data.url)
      return await page.$eval(data.table, (table) => {
        const headers = [...table.querySelectorAll("thead th")].map((th) =>
          th.textContent.trim(),
        )
        return [...table.querySelectorAll("tbody tr")].map((tr) => {
          const cells = [...tr.querySelectorAll("td")].map((td) =>
            td.textContent.trim(),
          )
          return Object.fromEntries(
            headers.map((header, i) => [header, cells[i]]),
          )
        })
      })
    },
    { url: "https://the-internet.herokuapp.com/tables", table: "#table1" },
  )
}

The header cells become the keys. Each row comes back as one object, for example { "Last Name": "Smith", "Email": "jsmith@gmail.com" }.

async ({ browser }) => {
  return await browser.run(
    async ({ page, data }) => {
      const books = []
      let url = data.url
      let pages = 0
      while (url && pages < data.maxPages) {
        await page.goto(url)
        const rows = await page.$$eval("article.product_pod", (articles) =>
          articles.map((article) => ({
            title: article.querySelector("h3 a").title,
            url: article.querySelector("h3 a").href,
            price: Number(
              article
                .querySelector(".price_color")
                .textContent.replace(/[^0-9.]/g, ""),
            ),
            rating: article.querySelector(".star-rating").classList[1],
            inStock: article
              .querySelector(".availability")
              .textContent.includes("In stock"),
          })),
        )
        books.push(...rows)
        pages++
        url = await page.$eval("li.next a", (a) => a.href).catch(() => null)
      }
      return { pages, count: books.length, books }
    },
    { url: "https://books.toscrape.com", maxPages: 3 },
  )
}

The loop reads a page, then follows the next link. It ends when there is no next link or when it reached maxPages. .catch(() => null) turns a missing next link into the end of the loop. Read many pages in one call. That is cheaper than one call per page.

async ({ browser }) => {
  return await browser.run(
    async ({ page, data }) => {
      await page.goto(data.url)
      await page.waitForSelector(".quote", { timeout: 20000 })
      return await page.$$eval(".quote .text", (texts) =>
        texts.map((text) => text.textContent),
      )
    },
    { url: "https://quotes.toscrape.com/js-delayed/" },
  )
}

goto returns when the page has loaded, not when its scripts have filled it. Wait for the element you want to read. This page fills itself after about 10 seconds, so the timeout is 20 seconds. Without the wait, the list would be empty.

1. Open Settings, then Credentials.
2. Choose New credential and name it site-username.
3. Open its menu, choose Edit, and paste the user name.
4. Do the same for site-password.

The node reads both from orbi.project.credentials and passes them to the browser in data. The password is never written in the node. If a credential is missing, the node stops with Add the credential site-username under Credentials.

The login examples use public practice sites, so you can run them as they are. the-internet.herokuapp.com accepts tomsmith with SuperSecretPassword!. quotes.toscrape.com accepts any user name and password.

async ({ browser, orbi }) => {
  const credential = (name) => {
    const value = orbi.project.credentials.find((c) => c.name === name)?.value
    if (!value) throw new Error(`Add the credential ${name} under Credentials`)
    return value
  }

  const result = await browser.run(
    async ({ page, data }) => {
      try {
        await page.goto(data.loginUrl)
        await page.locator("#username").fill(data.username)
        await page.locator("#password").fill(data.password)
        await Promise.all([
          page.waitForNavigation(),
          page.locator("button[type='submit']").click(),
        ])
        const message = await page.$eval("#flash", (el) =>
          el.textContent.replace("×", "").trim(),
        )
        if (!page.url().endsWith("/secure")) return { error: message }
        return { loggedIn: true, message, url: page.url() }
      } catch (error) {
        return { error: error.message, url: page.url() }
      }
    },
    {
      loginUrl: "https://the-internet.herokuapp.com/login",
      username: credential("site-username"),
      password: credential("site-password"),
    },
  )

  if (!result) throw new Error("The browser run returned nothing")
  if (result.error) throw new Error(`Login failed: ${result.error}`)
  return result
}

• locator(...).fill waits for the field, clears it, and types the value.
• Start waitForNavigation and the click together in Promise.all. If you click first and wait afterwards, the navigation can already be over.
• Check where you landed. Use the URL, or an element that only exists when you are logged in. A wrong password stops the node with Login failed: Your password is invalid!
• Hidden fields, such as a CSRF token, are sent along because you submit the site's real form.
• A two-factor code, a CAPTCHA, or a "Was this you?" check stops the login. Use the site's API or a connected account for those sites.

async ({ browser, orbi }) => {
  const credential = (name) => {
    const value = orbi.project.credentials.find((c) => c.name === name)?.value
    if (!value) throw new Error(`Add the credential ${name} under Credentials`)
    return value
  }

  const result = await browser.run(
    async ({ page, data }) => {
      let logins = 0

      const openLoggedIn = async (url) => {
        await page.goto(url)
        if (await page.$(data.loggedInMarker)) return
        if (logins >= data.maxLogins)
          throw new Error("The session keeps ending")
        await page.goto(data.origin + "/login")
        await page.locator("#username").fill(data.username)
        await page.locator("#password").fill(data.password)
        await Promise.all([
          page.waitForNavigation(),
          page.locator("input[type='submit']").click(),
        ])
        logins++
        await page.goto(url)
        if (!(await page.$(data.loggedInMarker)))
          throw new Error("Login failed")
      }

      try {
        const quotes = []
        let url = data.origin + "/"
        while (url) {
          await openLoggedIn(url)
          const rows = await page.$$eval(".quote", (els) =>
            els.map((el) => ({
              text: el.querySelector(".text").textContent,
              author: el.querySelector(".author").textContent,
              goodreads: el.querySelector("a[href*='goodreads']")?.href ?? null,
            })),
          )
          quotes.push(...rows)
          url = await page.$eval("li.next a", (a) => a.href).catch(() => null)
        }
        return { logins, count: quotes.length, quotes }
      } catch (error) {
        return { error: error.message, url: page.url(), logins }
      }
    },
    {
      origin: "https://quotes.toscrape.com",
      loggedInMarker: "a[href='/logout']",
      maxLogins: 3,
      username: credential("site-username"),
      password: credential("site-password"),
    },
  )

  if (!result) throw new Error("The browser run returned nothing")
  if (result.error) throw new Error(result.error)
  return result
}

The session lives as long as the browser. Log in and read everything inside the same call. openLoggedIn opens a page and logs in only when the logged-in marker is missing: the first time, and again if the site ends the session halfway. maxLogins stops the run when the login never sticks. The result has 100 quotes, each with a link that only logged-in visitors see, and logins: 1.

async ({ browser, orbi, input }) => {
  const credential = (name) => {
    const value = orbi.project.credentials.find((c) => c.name === name)?.value
    if (!value) throw new Error(`Add the credential ${name} under Credentials`)
    return value
  }

  const step = async ({ page, data }) => {
    let logins = 0

    const openLoggedIn = async (url) => {
      await page.goto(url)
      if (await page.$(data.loggedInMarker)) return
      if (logins >= data.maxLogins) throw new Error("The session keeps ending")
      await page.goto(data.origin + "/login")
      await page.locator("#username").fill(data.username)
      await page.locator("#password").fill(data.password)
      await Promise.all([
        page.waitForNavigation(),
        page.locator("input[type='submit']").click(),
      ])
      logins++
      await page.goto(url)
      if (!(await page.$(data.loggedInMarker))) throw new Error("Login failed")
    }

    try {
      if (data.cookies.length) await page.browser().setCookie(...data.cookies)
      const quotes = []
      let url = data.url
      for (let i = 0; url && i < data.pagesPerRun; i++) {
        await openLoggedIn(url)
        const rows = await page.$$eval(".quote", (els) =>
          els.map((el) => ({
            text: el.querySelector(".text").textContent,
            author: el.querySelector(".author").textContent,
          })),
        )
        quotes.push(...rows)
        url = await page.$eval("li.next a", (a) => a.href).catch(() => null)
      }
      return {
        quotes,
        next: url,
        logins,
        cookies: await page.browser().cookies(),
      }
    } catch (error) {
      return { error: error.message, url: page.url() }
    }
  }

  const origin = "https://quotes.toscrape.com"
  let cookies = input?.cookies ?? []
  let url = origin + "/"
  let runs = 0
  let logins = 0
  const quotes = []

  while (url) {
    const result = await browser.run(step, {
      origin,
      url,
      cookies,
      pagesPerRun: 3,
      loggedInMarker: "a[href='/logout']",
      maxLogins: 2,
      username: credential("site-username"),
      password: credential("site-password"),
    })
    if (!result) throw new Error("The browser run returned nothing")
    if (result.error) throw new Error(`${result.error} (${result.url})`)
    quotes.push(...result.quotes)
    cookies = result.cookies
    url = result.next
    logins += result.logins
    runs++
  }

  return { runs, logins, count: quotes.length, quotes, cookies }
}

Use this when the work is spread over several calls or nodes. Each call ends with page.browser().cookies(). The next call puts them back with page.browser().setCookie(...) before its first goto. The site sees the same session every time.

• The example reads 10 pages in 4 calls and logs in once: { runs: 4, logins: 1, count: 100 }.
• When the cookies expired, openLoggedIn logs in again and the loop goes on.
• The node returns cookies. A following node reads input.cookies and continues without a login. Use this for jobs longer than the 10 minutes one node may run.
• The cookies are the login. Anyone who can read the workflow run can use them. Leave cookies out of the return value when no later node needs them.
• Each call costs at least 1 credit and takes a moment to start. Split only when you need to.

async ({ browser }) => {
  const origin = "https://quotes.toscrape.com"

  const storage = await browser.run(
    async ({ page, data }) => {
      await page.goto(data.origin)
      await page.evaluate(() => localStorage.setItem("token", "demo-token"))
      return await page.evaluate(() => ({ ...localStorage }))
    },
    { origin },
  )

  return await browser.run(
    async ({ page, data }) => {
      await page.evaluateOnNewDocument(
        (origin, storage) => {
          if (location.origin !== origin) return
          for (const [key, value] of Object.entries(storage))
            localStorage.setItem(key, value)
        },
        data.origin,
        data.storage,
      )
      await page.goto(data.origin)
      return { token: await page.evaluate(() => localStorage.getItem("token")) }
    },
    { origin, storage },
  )
}

Some web apps keep their token in localStorage instead of a cookie. Save { ...localStorage } at the end of one call. In the next call, page.evaluateOnNewDocument writes it back before the app's own scripts run. The origin check keeps the values on that one site. The example writes a demo value, so you can see it come back.

async ({ orbi, input }) => {
  const credential = (name) => {
    const value = orbi.project.credentials.find((c) => c.name === name)?.value
    if (!value) throw new Error(`Add the credential ${name} under Credentials`)
    return value
  }

  const api = "https://dummyjson.com"
  let token = input?.token ?? null
  let logins = 0

  const login = async () => {
    const response = await fetch(api + "/auth/login", {
      method: "POST",
      headers: { "content-type": "application/json" },
      body: JSON.stringify({
        username: credential("api-username"),
        password: credential("api-password"),
        expiresInMins: 30,
      }),
    })
    if (!response.ok) throw new Error(`Login failed: ${response.status}`)
    token = (await response.json()).accessToken
    logins++
  }

  const get = async (path) => {
    if (!token) await login()
    let response = await fetch(api + path, {
      headers: { authorization: `Bearer ${token}` },
    })
    if (response.status === 401) {
      await login()
      response = await fetch(api + path, {
        headers: { authorization: `Bearer ${token}` },
      })
    }
    if (!response.ok) throw new Error(`${path} failed: ${response.status}`)
    return await response.json()
  }

  const products = []
  for (let skip = 0; ; skip += 50) {
    const page = await get(
      `/auth/products?limit=50&skip=${skip}&select=title,price`,
    )
    products.push(...page.products)
    if (skip + 50 >= page.total) break
  }

  return { logins, count: products.length, products, token }
}

A JSON API does not need the browser. Log in with fetch, keep the token, and send it with each request. A 401 means the token expired: log in again and repeat that request once. The node returns token, so the next node can pass input.token and skip the login. The example uses the public practice API dummyjson.com. Save api-username as emilys and api-password as emilyspass to try it.

When the function throws or a call hits its time limit, browser.run returns undefined. It does not throw, and the message is lost. The failed call still costs its credits, so a loop that keeps failing keeps spending. Catch errors inside the function and return { error }. Then decide in the node: throw to stop the workflow, or return the details while you fix the selector.

async ({ browser }) => {
  const result = await browser.run(
    async ({ page, data }) => {
      try {
        await page.goto(data.url)
        await page.waitForSelector(data.selector, { timeout: 5000 })
        return { text: await page.$eval(data.selector, (el) => el.textContent) }
      } catch (error) {
        return {
          error: error.message,
          url: page.url(),
          title: await page.title(),
          screenshot: await page.screenshot({
            encoding: "base64",
            type: "jpeg",
            quality: 50,
          }),
        }
      }
    },
    { url: "https://quotes.toscrape.com", selector: ".price" },
  )

  if (!result) throw new Error("The browser run returned nothing")
  return result
}

The result has the message, the URL, the page title, and a JPEG screenshot as base64. Open data:image/jpeg;base64, followed by that string in your own browser to see the page.

• failed to find element matching selector: the element is not there. Check the selector, or wait for it first.
• Waiting for selector ... failed: the element did not appear within the timeout.
• Navigation timeout of 30000 ms exceeded: the site is slow, or it holds back headless browsers. Pass a larger timeout to goto, or set a user agent.
• ... is not defined: the function used a variable from the node. Pass it in data.