Having trouble accessing the loadTokenizer function in Tensorflow JS

Question

Having trouble accessing the loadTokenizer function in Tensorflow JS

As a beginner with Tensorflow.js concepts, I recently attempted to tokenize a sentence using the Universal Sentence Encoder in Javascript. You can explore more about it on Github Reference

$ npm install @tensorflow/tfjs @tensorflow-models/universal-sentence-encoder

After running this command, a package-lock.json file was generated which I placed alongside my index.html file within the same directory structure shown below.

/*
  Folder
    |_index.html
    |_package-lock.json
    |_index.js
    |_index.css
*/

Within index.html:

<head>
  <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs"></script>
  <script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/universal-sentence-encoder"></script>   
  <script src="index.js" defer></script> 
</head>

Contents of index.js:

function tokenizePad(text){
    text = use.loadTokenizer().then(tokenizer => {
        tokenizer.encode(text); 
    });
    return text;
}

text = "I enjoy my holiday very much."
var tokenized = tokenizePad(text); //error

The console displayed an error message as follows:

Uncaught TypeError: use.loadTokenizer is not a function

Is there a solution to this issue? Are there alternative methods to achieve the desired outcome of converting the string into an array of encoded values like [341, 4125, 8, 140, 31, 19, 54, ......] mentioned in the Github Reference link?

javascript npm package tokenize tensorflow.js

Answer 1

Answer №1

I faced a similar challenge and came up with the solution below:

import use from 'module';

use.load().then(useObj => {
    model = useObj.model;
    tokenizer = useObj.tokenizer;

    text = "I absolutely love going on vacation."
    var tokenized = tokenizer.encode(text); 

    console.log(tokenized); //[7933, 2222, 0, 109, 7933, 2222, 0, 154, 2174, 48, 7933, 2222, 0, 1272, 7933, 2222, 0, 645, 336, 944, 7933, 2222, 0, 5568, 7933, 2222, 0, 47, 1788, 6]
});

The approach above focuses on character-level encoding. If you discover a method for word-based encoding, please share that with me.

Answer 2

I faced a similar challenge and came up with the solution below:

import use from 'module';

use.load().then(useObj => {
    model = useObj.model;
    tokenizer = useObj.tokenizer;

    text = "I absolutely love going on vacation."
    var tokenized = tokenizer.encode(text); 

    console.log(tokenized); //[7933, 2222, 0, 109, 7933, 2222, 0, 154, 2174, 48, 7933, 2222, 0, 1272, 7933, 2222, 0, 645, 336, 944, 7933, 2222, 0, 5568, 7933, 2222, 0, 47, 1788, 6]
});

The approach above focuses on character-level encoding. If you discover a method for word-based encoding, please share that with me.

Answer 3

Answer №2

This code snippet showcases a more sophisticated and refined approach. Begin by importing it in the following manner:

import * as USE from '@tensorflow-models/universal-sentence-encoder';

Then proceed to utilize it with USE:

// Load the model.
USE.load().then(model => {
  // Embed an array of sentences.
  const sentences = [
    'Greetings.',
    'How do you do?'
  ];
  model.embed(sentences).then(embeddings => {
    // The variable `embeddings` is a 2D tensor containing 512-dimensional embeddings for each sentence.
    // Therefore, in this scenario, `embeddings` has dimensions [2, 512].
    embeddings.print(true /* verbose */);
  });
});

Answer 4

This code snippet showcases a more sophisticated and refined approach. Begin by importing it in the following manner:

import * as USE from '@tensorflow-models/universal-sentence-encoder';

Then proceed to utilize it with USE:

// Load the model.
USE.load().then(model => {
  // Embed an array of sentences.
  const sentences = [
    'Greetings.',
    'How do you do?'
  ];
  model.embed(sentences).then(embeddings => {
    // The variable `embeddings` is a 2D tensor containing 512-dimensional embeddings for each sentence.
    // Therefore, in this scenario, `embeddings` has dimensions [2, 512].
    embeddings.print(true /* verbose */);
  });
});

Having trouble accessing the loadTokenizer function in Tensorflow JS

Answer №1

Answer №2

Similar questions

What should we do to resolve the uncommon use of the 'this' es-lint error? Which rule should we disable?

What is the best way to keep calling an AJAX function until it receives a response from a previous AJAX call

The IIS URL rewrite is causing issues with the rewriting of CSS and JS files

The integration between Javascript, PHP, and SQL Server does not display the retrieved data

JavaScript For Each loops are really starting to frustrate me

How can I utilize the "remark" tool to handle Markdown files with URLs that don't adhere to Markdown formatting? Are there any supplemental plugins available for this

A JSON request is being processed within a while loop

Interactive JQuery calendar

Encountering a SyntaxError while implementing lightweight-charts in NextJS

Having difficulty integrating plugins like Lighthouse into my Cypress project

I have successfully implemented an onChange function with its corresponding set of parameters. However, I now desire to extend its functionality by incorporating

It appears that the Facebook share feature is not picking up any meta OG tags

The AngularJS ngModel directive encounters issues when used within a ui-bootstrap tabset

Accessing loop variables in Render and passing them into componentDidMount() in ReactJS to include as a query parameter in an API call

Guide on implementing jQuery Validation plugin with server-side validation at the form level

Maxlength and Minlength attributes are not considered when using input type=“number”

Is it considered best practice to call the same function within itself in a function?

What is the best way to link a newly created contact to the current user in Mongoose?

When I open Firefox, all I see is a blank page with only the h2 heading displayed

Sweetalert seems to have hit a roadblock and is not functioning properly. An error has been detected in its TS file