Storing data in a JSON format

Question

Storing data in a JSON format

Our team requires a service that can periodically generate up-to-date JSON data every 15 minutes for our AJAX services to consume. This JSON will be used for metric analysis and charts among other things.

The reason behind the 15-minute interval is due to the time-consuming nature of the JSON generation process, with queries taking 45-60 seconds to complete. As our application grows, this duration is expected to increase further.

EDIT: All our data is sourced from a SQL database, which is already established as a large relational database setup. Swiftly delivering this data to our web users is essential, hence real-time querying from the database is not feasible due to slow response times.

EDIT 2 The JSON data needs to be completely refreshed every 15 minutes and made readily available. While it's acceptable if the background service takes time to run, ensuring that the JSON is generated on schedule is our priority.

The challenge lies in determining the optimal method to store this growing number of JSON objects - around 30-40 need to be created and serialized for transmission. Additionally, we have predefined time periods for JSON generation: daily, weekly, monthly, quarterly, semiannually, annually, and biennially.

I've contemplated using a flat database table, although I'm hesitant about having tables with numerous columns and only one row. Another option is generating a "data.json" file on the server to pull data from, but are there any potential drawbacks or limitations to this approach? Can it be cached without compromising its purpose?

We're open to exploring alternative storage methods - could an expert provide insights on the best way to handle this scenario?

javascript json ajax database

Answer 1

Answer №1

Instead of generating data dynamically, storing it as text files is the way to go. It's quick, straightforward, and can handle growth with ease. You have the option to create an archive of past information or overwrite old files if they're no longer needed.

To prevent json file caching, you have a couple of options.

$.ajax({ cache: false })

You could also manage caching in the server headers when serving the file. Setting a 15-minute cache time can help reduce strain on your server. The method to implement this depends on the technology you're using.

Answer 2

Instead of generating data dynamically, storing it as text files is the way to go. It's quick, straightforward, and can handle growth with ease. You have the option to create an archive of past information or overwrite old files if they're no longer needed.

To prevent json file caching, you have a couple of options.

$.ajax({ cache: false })

You could also manage caching in the server headers when serving the file. Setting a 15-minute cache time can help reduce strain on your server. The method to implement this depends on the technology you're using.

Answer 3

Answer №2

It seems like you might be relying too heavily on this one example as your primary source of data, possibly overlooking the potential of utilizing APIs from other major companies like Tumblr or Facebook.

However, I want to emphasize that using a database to store your data would offer greater efficiency compared to storing it in JSON format.

If you opt to store data in a file, you'll need to develop your own plugins to manage and validate the data within the file, ensuring its accuracy and relevance. This could be quite time-consuming in terms of coding. On the contrary, databases allow you to effortlessly select specific rows and columns, convert that data into JSON format, and then deliver it back to the client as a JSON object - which is likely your desired outcome.

By utilizing a database, you can create a table for each distinct JSON object you plan to use, similar to the following:

ObjId | ObjName | ObjProperty
1     | User    | Employee
2     | User    | Client

This serves as an illustrative table layout. After successfully storing the data, retrieving it simply involves executing a SELECT statement like so:

SELECT * FROM table_name WHERE ObjId = 1

Upon execution, the first row will be returned. You can then stringify it and send it as a string to the client for further processing.

Visit this link for additional information on converting objects to strings using JSON.stringify

Answer 4

It seems like you might be relying too heavily on this one example as your primary source of data, possibly overlooking the potential of utilizing APIs from other major companies like Tumblr or Facebook.

However, I want to emphasize that using a database to store your data would offer greater efficiency compared to storing it in JSON format.

If you opt to store data in a file, you'll need to develop your own plugins to manage and validate the data within the file, ensuring its accuracy and relevance. This could be quite time-consuming in terms of coding. On the contrary, databases allow you to effortlessly select specific rows and columns, convert that data into JSON format, and then deliver it back to the client as a JSON object - which is likely your desired outcome.

By utilizing a database, you can create a table for each distinct JSON object you plan to use, similar to the following:

ObjId | ObjName | ObjProperty
1     | User    | Employee
2     | User    | Client

This serves as an illustrative table layout. After successfully storing the data, retrieving it simply involves executing a SELECT statement like so:

SELECT * FROM table_name WHERE ObjId = 1

Upon execution, the first row will be returned. You can then stringify it and send it as a string to the client for further processing.

Visit this link for additional information on converting objects to strings using JSON.stringify

Storing data in a JSON format

Answer №1

Answer №2

Similar questions

CORS (Cross-Origin Resource Sharing) Request Failed

JQuery is displaying the word "undefined" instead of the expected JSON array values

After the form is submitted, attach a div containing the saved record

Guide to sending both JSON data and form data in a single request using Laravel 9

The next.js Head component that is imported from next/head is not functioning correctly when used on share pages

What is causing the scripts to fail when I attempt to press a button?

Tips for interpreting JSON information and showcasing it with the assistance of Ajax in JQuery?

Storybook encounters an undefined error with the Vuex store

The embedded iframe on YouTube is displaying the incorrect video

What could be the reason that data-bs-placement="right" is not functioning as expected?

Exploring the differences between UTC and non-UTC date formats in Javascript

Saving changes made in HTML is not supported when a checkbox is selected and the page is navigated in a React application using Bootstrap

What is the best way to save my output in a JSON format file?

What is the best way to create a taskbar using HTML or Javascript?

Activating events with Jquery

Another option for authorizing a document

Issue with Chrome not triggering onMouseEnter event when an element blocking the cursor disappears in React

Error message encountered when submitting a form after receiving an AJAX response: VerifyCsrfToken Exception

Why is the Axios instance/function not being passed the argument?

"Using jQuery's Ajax feature to efficiently delete a record