What are some strategies for improving search efficiency in arrays containing over 50,000 elements?

Question

What are some strategies for improving search efficiency in arrays containing over 50,000 elements?

I am working with a large array of strings containing about 50,000 elements.

export const companies = [
  "000014",
  "000016",
  "000017",
  "000019",
  "000020",
  "000021",
  "000023",
  "000025",
  ...
]

These strings represent company names for which specific pages are displayed. I have implemented a middleware function that iterates through this array using a loop.

import { NextResponse, type NextRequest } from "next/server";
import { companies } from "./assets/companies";

export async function middleware(req: NextRequest) {
  const { pathname } = req.nextUrl;

  // Loop to compare current URL path with company names
  await for (let i = 0; i < companies.length; i++) {
    if (pathname.startsWith(`/${companies[i]}`))
        return NextResponse.redirect(new URL("/", req.url)); // Redirect to main page if company name matches current path
  }
}

This process takes some time and I'm exploring ways to optimize it. One idea was to divide the array into chunks, but that may not be the most efficient solution.

javascript next.js

Answer 1

Answer №1

If the pathname follows a format like /000014/abc/xyz, you can eliminate the need for array iteration by using the following approach:

import { NextResponse, type NextRequest } from "next/server";
import { companies } from "./assets/companies";

const companiesSet = new Set(companies);

export async function middleware(req: NextRequest) {
  const { pathname } = req.nextUrl;
  
  // It would be better to retrieve this value from req.params.companyId, but its existence depends on the routing code, which is not visible here.
  const companyId = req.pathname.match(/\/([0-9]+)\//)?.[1];

  if (companiesSet.has(companyId)) {
    return NextResponse.redirect(new URL("/", req.url)); // redirect to main page if companies match with current pathname
  }
}

In any case, having 50,000 elements isn't considered too large and the code should not have been slow. The unnecessary await and string building within the loop could have contributed to any performance issues.

Answer 2

If the pathname follows a format like /000014/abc/xyz, you can eliminate the need for array iteration by using the following approach:

import { NextResponse, type NextRequest } from "next/server";
import { companies } from "./assets/companies";

const companiesSet = new Set(companies);

export async function middleware(req: NextRequest) {
  const { pathname } = req.nextUrl;
  
  // It would be better to retrieve this value from req.params.companyId, but its existence depends on the routing code, which is not visible here.
  const companyId = req.pathname.match(/\/([0-9]+)\//)?.[1];

  if (companiesSet.has(companyId)) {
    return NextResponse.redirect(new URL("/", req.url)); // redirect to main page if companies match with current pathname
  }
}

In any case, having 50,000 elements isn't considered too large and the code should not have been slow. The unnecessary await and string building within the loop could have contributed to any performance issues.

Answer 3

Answer №2

If you find yourself unable to rely solely on map or object lookups, such as @Aurast suggested (for instance, if you require free text searching for finding the closest matching name), then one straightforward solution is to group companies in order to break down the list into smaller segments and search within a single segment at a time. One way to do this is by dividing the companies based on their initial letter:

export const companies = {
    "a": ["aardvarks, inc", "AppendeciteCorp", ],
    "b": ["boulevard bricklayers", "bandits and bakers", "brompton fan club"],
}

By organizing the data in this manner, you can search by the first letter alone, significantly reducing the size of the array that needs to be processed. This approach is simple to implement.

Another option is to extend this concept further by recursively grouping companies based on their first letter in a data structure known as a trie or prefix tree. With this method, you can quickly search for information even when an exact match is not required.

https://i.sstatic.net/3pByO.png

Answer 4

If you find yourself unable to rely solely on map or object lookups, such as @Aurast suggested (for instance, if you require free text searching for finding the closest matching name), then one straightforward solution is to group companies in order to break down the list into smaller segments and search within a single segment at a time. One way to do this is by dividing the companies based on their initial letter:

export const companies = {
    "a": ["aardvarks, inc", "AppendeciteCorp", ],
    "b": ["boulevard bricklayers", "bandits and bakers", "brompton fan club"],
}

By organizing the data in this manner, you can search by the first letter alone, significantly reducing the size of the array that needs to be processed. This approach is simple to implement.

Another option is to extend this concept further by recursively grouping companies based on their first letter in a data structure known as a trie or prefix tree. With this method, you can quickly search for information even when an exact match is not required.

https://i.sstatic.net/3pByO.png

What are some strategies for improving search efficiency in arrays containing over 50,000 elements?

Answer №1

Answer №2

Similar questions

What is the most effective method for displaying two external web pages next to each other?

"Apple TV now features an iOS app that allows users to watch YouTube videos

Do not open iframe links in a new window

Mastering the Art of Concise Writing: Tips to

I recently created a "dist" folder through Babel. Is it possible to integrate the files within this folder directly into a fresh React project?

Is Sass only compatible with Ubuntu for monitoring once?

The correlation between frames per second (FPS) and the milliseconds required to render a frame in the stats plugin is known as the frame

What is the best way to determine the number of subchildren within a parent div using JavaScript or jQuery?

Launching a centered pop-up window to display a submitted form message

Having trouble getting Firebase phone number authentication to work with Vue.js

What will be the quickest and most reliable trigger - matchMedia, window.resize, orientationchange, or perhaps both combined?

Non-negotiable, non-deletable component of a text field

Confusion with Javascript callbacks - seeking clarity

Converting data from a JSON-like file format into valid JSON using JavaScript

Exploring the depths of deep populating in Mongo and Node.js

Utilize Next JS pages api to generate dynamic routes based on unique ids

What could be causing the show button to change its style every time I click it?

The extjs datecolumn is displaying dates with a one-day discrepancy

Incorporating an NPM module with dependencies within the Meteor framework

Automatically trigger a popup box to appear following an AJAX request