What are some strategies for improving search efficiency in arrays containing over 50,000 elements?

Question

What are some strategies for improving search efficiency in arrays containing over 50,000 elements?

I am working with a large array of strings containing about 50,000 elements.

export const companies = [
  "000014",
  "000016",
  "000017",
  "000019",
  "000020",
  "000021",
  "000023",
  "000025",
  ...
]

These strings represent company names for which specific pages are displayed. I have implemented a middleware function that iterates through this array using a loop.

import { NextResponse, type NextRequest } from "next/server";
import { companies } from "./assets/companies";

export async function middleware(req: NextRequest) {
  const { pathname } = req.nextUrl;

  // Loop to compare current URL path with company names
  await for (let i = 0; i < companies.length; i++) {
    if (pathname.startsWith(`/${companies[i]}`))
        return NextResponse.redirect(new URL("/", req.url)); // Redirect to main page if company name matches current path
  }
}

This process takes some time and I'm exploring ways to optimize it. One idea was to divide the array into chunks, but that may not be the most efficient solution.

javascript next.js

Answer 1

Answer №1

If the pathname follows a format like /000014/abc/xyz, you can eliminate the need for array iteration by using the following approach:

import { NextResponse, type NextRequest } from "next/server";
import { companies } from "./assets/companies";

const companiesSet = new Set(companies);

export async function middleware(req: NextRequest) {
  const { pathname } = req.nextUrl;
  
  // It would be better to retrieve this value from req.params.companyId, but its existence depends on the routing code, which is not visible here.
  const companyId = req.pathname.match(/\/([0-9]+)\//)?.[1];

  if (companiesSet.has(companyId)) {
    return NextResponse.redirect(new URL("/", req.url)); // redirect to main page if companies match with current pathname
  }
}

In any case, having 50,000 elements isn't considered too large and the code should not have been slow. The unnecessary await and string building within the loop could have contributed to any performance issues.

Answer 2

If the pathname follows a format like /000014/abc/xyz, you can eliminate the need for array iteration by using the following approach:

import { NextResponse, type NextRequest } from "next/server";
import { companies } from "./assets/companies";

const companiesSet = new Set(companies);

export async function middleware(req: NextRequest) {
  const { pathname } = req.nextUrl;
  
  // It would be better to retrieve this value from req.params.companyId, but its existence depends on the routing code, which is not visible here.
  const companyId = req.pathname.match(/\/([0-9]+)\//)?.[1];

  if (companiesSet.has(companyId)) {
    return NextResponse.redirect(new URL("/", req.url)); // redirect to main page if companies match with current pathname
  }
}

In any case, having 50,000 elements isn't considered too large and the code should not have been slow. The unnecessary await and string building within the loop could have contributed to any performance issues.

Answer 3

Answer №2

If you find yourself unable to rely solely on map or object lookups, such as @Aurast suggested (for instance, if you require free text searching for finding the closest matching name), then one straightforward solution is to group companies in order to break down the list into smaller segments and search within a single segment at a time. One way to do this is by dividing the companies based on their initial letter:

export const companies = {
    "a": ["aardvarks, inc", "AppendeciteCorp", ],
    "b": ["boulevard bricklayers", "bandits and bakers", "brompton fan club"],
}

By organizing the data in this manner, you can search by the first letter alone, significantly reducing the size of the array that needs to be processed. This approach is simple to implement.

Another option is to extend this concept further by recursively grouping companies based on their first letter in a data structure known as a trie or prefix tree. With this method, you can quickly search for information even when an exact match is not required.

Answer 4

If you find yourself unable to rely solely on map or object lookups, such as @Aurast suggested (for instance, if you require free text searching for finding the closest matching name), then one straightforward solution is to group companies in order to break down the list into smaller segments and search within a single segment at a time. One way to do this is by dividing the companies based on their initial letter:

export const companies = {
    "a": ["aardvarks, inc", "AppendeciteCorp", ],
    "b": ["boulevard bricklayers", "bandits and bakers", "brompton fan club"],
}

By organizing the data in this manner, you can search by the first letter alone, significantly reducing the size of the array that needs to be processed. This approach is simple to implement.

Another option is to extend this concept further by recursively grouping companies based on their first letter in a data structure known as a trie or prefix tree. With this method, you can quickly search for information even when an exact match is not required.

What are some strategies for improving search efficiency in arrays containing over 50,000 elements?

Answer №1

Answer №2

Similar questions

Implementing a keypress function that can handle duplicate names

Leverage AJAX data to dynamically generate an input field within a Laravel application

Discover a technique to display every individual "echo" statement from PHP in sequence, rather than waiting for the entire script to finish executing

Guide on incorporating mouse movement while the mouse button is held down in JavaScript

Is there a method to make this package compatible with Angular version 16?

Creating a dynamic dropdown menu using JQuery that does not automatically submit the form when a value is

Using VueJs's createElement() to dynamically insert HTML content

Issue with the positioning of bootstrap popover after content is added

Experience seamless integration of Mantine UI with NextJS when utilizing useMantineColorScheme. Uncover the power of color customization,

Finding the offsetWidth (or similar measurement) for a list item (LI) element

Error: The requested resource, youtube#videoListResponse, is currently unavailable

Creating dynamic dropdowns with Ajax and HTML on the code side

AngularJS- issue with button visibility on 'toolbar' widget

Display conceal class following successful ajax response

How can I arrange selected options at the top in MUI autocomplete?

Using Regular Expressions to eliminate any characters that are not directly adjacent to numbers (beside specific characters)

Will it function properly if it is (NULL)?

Show or hide a component based on a mouse click in Vue JS

Incorporate javascript into your XML transformations with XSLT

Unable to resubmit form via ajax more than once