Efficient method for sending a high volume of rows to AWS Lambda from datatables (approximately 10,000 rows)

Currently, I am working on an extension for Tableau that captures user decisions and saves them as rows in datatables using JavaScript. These modified rows are then sent to AWS Lambda in JSON format. In AWS Lambda, we process the data from these rows by generating update SQL queries and executing them one by one on a Redshift database using a for loop.

However, this process is proving to be time-consuming. Can anyone recommend a more efficient approach?

  1. Is there a better method for sending data to AWS Lambda, such as packing or zipping the data before transmission?
  2. Are there alternative approaches to mass updating the database instead of using a for loop?

Note: Each modified row may contain different values.

javascriptaws-lambdadatatablesamazon-redshifttableau-api

Answer №1

Are you aware of where your time is being consumed? I have a couple of ideas on what might be happening, but having data to support it would be beneficial.

Redshift functions as a columnar database specifically designed for bulk data uploads rather than individual row updates. It is tailored for handling large volumes of data (100GB and above). However, based on the limited information provided, it is unclear if this aligns with your current needs. Additionally, Redshift is not optimized for transferring data from SQL to compute nodes efficiently when loading data using SQL literals.

If dealing with substantial amounts of data, it is advisable to aggregate enough records (100MB+) before uploading to Redshift through S3. This method may result in a brief delay before the data becomes accessible within Redshift. Should this approach not meet your requirements, alternative services should be considered. In this scenario, Lambda can consolidate records into sizable S3 files, which can then be COPY'ed into Redshift at set intervals, mirroring Kinesis Firehose functionality - an AWS service worth exploring.

Redshift thrives on numerous reads and infrequent, sizeable writes. If an equal number of small writes and reads are necessitated, opting for a transactional database might be more appropriate. To maintain a PostgreSQL-style interface, utilizing Postgresql in RDS remains a viable choice.

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

Design a div in the shape of a trapezium with clipped overflow

Is there a way to create a trapezium using CSS/Html/Canvas? I have attempted to do so, but my current method is messy and not practical. div { width:0; margin-left:-1000px; height:100px; border-right:1000px solid lightblue; border-top:60px solid tra ...

javascriptjqueryhtmlcss

Steps for retrieving the group names of properties in Autodesk Forge

Can someone please help me figure out how to obtain the property group names in Autodesk Forge? I have attempted to use getProperties() and getBulkProperties(), but have had no success in retrieving the group names. Any guidance on how to accomplish this w ...

javascriptautodesk-forge

The email sending feature of the ajax jquery form has unexpectedly ceased functioning

I'm encountering an issue with a form that was previously able to send emails from the server, but suddenly stopped working. I'm having trouble identifying the cause of the problem. Interestingly, when I use the form without ajax, the email funct ...

javascriptphpjqueryajax

The JQuery .replaceWith method exclusively offers unprocessed HTML content

I'm experiencing a minor issue with my ajax response where it returns raw html and does not execute any scripts, resulting in the jquery ui scripts not being executed. Here are some details: I am currently working with ASP.NET MVC and I need to comm ...

javascriptjqueryhtmlajax

AWS Lambda function with dual handlers

Currently, I am working on a lambda function in node.js to retrieve information from my AWS account. However, I am encountering an issue where the required information is not being passed correctly in the callback response. While I can see the information ...

node.jsamazon-web-servicesaws-lambda

What methods can I use to test a Django view that includes a form?

As I embark on developing an application using Django, I find myself faced with a particular challenge. I have created a view that receives a form from the HTML code and then searches the database for any instances of a model based on the values specified ...

javascriptjquerydjangoformsconfirm

Trigger price update in jquery based on radio or checkbox selection and specific conditions

https://i.sstatic.net/OIvgF.png In my product form, I have implemented a feature where selecting the "product type" and "product size" will update the price. However, if the user changes the product type after selecting the size, the price does not update ...

javascriptjquery

Once you address a block using jQuery

$(function(){ $(window).scroll(function () { var scrollVal = $(this).scrollTop(); var adscrtop =$(".header").offset().top // 在 RWD 767以下不作動 if(window.screen.width>767){ if(sc ...

javascriptjquerycss

Error: The expression `xmlDoc.load` returns a value of undefined, which is not an object

My current challenge involves loading an XML file using Javascript in IE, Firefox, and Safari. I have yet to find a function that works well across all three browsers. The load function I am currently using is similar to the one found in w3schools tutorial ...

javascriptxmlajax

What is the best method to create random coordinates that fall outside of the circle located within a rectangular area?

Suppose the following scenario: A rectangular area with dimensions length l and breadth b A circle with a radius r The circle is positioned inside the rectangular area as depicted in the image below Check out the image here - Red areas show expected even ...

javascriptmathgeometry2dpolar-coordinates

The nested div within the ui-view element is failing to expand to the entire width of the window

I am facing a challenge in making a child div stretch to the entire width of the window, rather than just the width of its parent. I am incorporating AngularJS with UI-Router, and I'm not sure if that is causing the issue. Attached is a screenshot. ...

javascripthtmlcssangularjs

Button functions properly after the second click

import { Input, Box, Text, Divider, Button } from '@chakra-ui/react'; import { useState } from 'react'; export default function GithubSearchApp() { const [username, setUsername] = useState(''); const [data, setData] = use ...

javascriptnext.jsgithub-api

Tap, swipe the inner contents of a <div> element without being inside it

(Make sure to check out the image below) I'm facing an issue with a <div> on my website. It's not taking up the full width of the page, and the scrolling functionality within the <body> is not working as expected. I just want the cont ...

javascriptjqueryhtml

My program is throwing an error due to invalid JSON format

Snippet of code: var data = $.parseJSON(data); Error message: Encountered Invalid JSON: {times: [{'9:30am','10:00am','10:30am','11:00am','11:30am','12:00pm','12:30pm','1:00pm&apo ...

javascriptajaxjson

Solving Issues with Google Maps Javascript API v3

My code doesn't seem to be functioning properly. Instead of displaying an error message, it's just showing a blank screen. Could you please take a look at my code to identify the issue? Thank you. <!DOCTYPE HTML> <html> <h ...

javascriptgoogle-maps

Experiencing issues with obtaining req.params.id undefined while initiating a put request

Encountering an issue while making a PUT request using Postman, as an error occurs in the VSCode terminal with the message: let product = await Product.findById(req.params.id); ^ TypeError: Cannot read property 'id' of undefined. The request ...

javascriptnode.jsmongodbexpressmongoose

Using jQuery to retrieve the text content of child elements

Struggling to extract the text of child elements using jQuery. I've been at this for a couple of days and can't seem to make it work. If anyone could spare a moment to review, I would greatly appreciate it! SCRIPT: function generateRemoveSect ...

javascriptjquery

Surprising results from implementing the useLocation() hook in a React application

Currently, I am embarking on a personal endeavor to create an online auction platform. One of the key functionalities I am working on is rendering detailed product information when a user clicks on a specific product for purchase. Initially, everything wor ...

javascriptnode.jsreactjsexpress

An array of size 10x10 where each new array replaces the previous one

I'm currently working on a function that generates a 10 by 10 array filled with random D's and E's. Below is the code snippet: function generateTenByTenArray(){ var outerArray = []; var innerArray = []; for(var i = 0; i < 10 ...

javascriptarrays

The regular expression should consist of just letters and a single dot

Currently, I am working on creating a regular expression that allows only letters and one dot. However, there is a specific rule to follow: the dot cannot be located at the beginning or end of the string. For example: abc.difference This is what I attem ...

javascriptregex