My ETL process transfers data from Sybase to MongoDB, involving matching a legacy id to populate the correct mongo ObjectId in the model. Initially, this method works smoothly for the first few thousand records. However, an issue arises when encountering a record without a legacy id, leading to an error due to the model expecting an objectId value. Simply implementing a null check within the ETL file is not sufficient to resolve this problem. I need a solution that addresses both the absence of a legacy id and the model's requirement for a valid objectId. How should I approach this?
The section of my ETL code relevant to this issue is as follows:
let agency = await Agency.findOne({ agencyId: record.agent_house }).exec();
This snippet is then integrated like so:
agency: {
id: agency._id ? agency._id : null,
// Other properties,
// Other properties
}
The data processed by this code is sent to the model structured like this:
agency: {
id: { type: mongoose.Schema.Types.ObjectId, ref: 'agencies' },
// Other properties,
// Other properties
}
To handle scenarios where there is no value present (resulting in an assignment failure for the objectId), how can I modify my process effectively despite already having a null check in place within the ETL file?