Download as PDF, PPTX
![module.exports = setup((patch) => {
patch.version('2.4.3');
patch.update('settings', search, (settings, callback) => {
return callback(null, modifier);
});
patch.after(whiteList({
'fieldThatAllowedToBeChanged': ['added|removed|modified'],
}));
});](https://image.slidesharecdn.com/datamigrationinnode-161204130612/85/Data-migration-in-node-js-rest-api-and-mongo-db-31-320.jpg)
![module.exports = setup((patch) => {
patch.version('2.4.3');
patch.update('settings', search, (settings, callback) => {
return callback(null, modifier);
});
patch.after(whiteList({
'fieldThatAllowedToBeChanged': ['added|removed|modified'],
}));
});](https://image.slidesharecdn.com/datamigrationinnode-161204130612/75/Data-migration-in-node-js-rest-api-and-mongo-db-31-2048.jpg)
Uploaded byFwdays
Андрей Шумада "Data migration in node.js rest api and mongo db "
The document discusses data migration processes in a Node.js REST API using MongoDB, focusing on transforming document structures between different versions (v1, v2, v3). It outlines how to handle API requests for retrieving, saving, and transforming documents while maintaining compatibility across versions. Various methods, including streaming and transformation functions, are described for managing data versions in a secure and efficient way.
More Related Content
Similar to Андрей Шумада "Data migration in node.js rest api and mongo db "
Андрей Шумада "Data migration in node.js rest api and mongo db "
- 1. Data migration inNode.js REST API and MongoDB Andrii Shumada @eagleeye_s Sr. Software Developer at
- 3. frontend REST API MongoDB database JSON JSON 3rd partyclients && Mobile apps JSON
- 4. Description ID Line1: amount, taxRate Line2:amount, taxRate ID Line1: amount, taxRate description Line2: amount, taxRate description Yesterday Document v1 Tomorrow Document v2
- 5. REST API v1 V2 or v1 Youcan’t release new version of mobile app as fast your frontend app v1 Your partners can’t switch to new version instantly
- 9. Because documents usuallyare independent! /document/v1 /document/v2 /document/v3
- 10. GET /document/:id/v2 Find document in DB Returnthis document GET /document/:id/v1 Assuming for now that document has v2 structure in DB/business logic Find document in DB Transform v2 -> v1 Return this document
- 11. transformObject(document, 'v2', 'v2|v1',options, (err, documentVx) => { return res.json(documentVx); }); Return ‘downgraded’ version of document
- 12. POST/PUT document/:id/v1 Save document in DBv2 Transform v1 -> v2 Return this document in v1 Validate document v2 Transform v2 -> v1
- 13. transformObject(documentVX, 'v1|2', 'v2',options, (err, documentV2) => { //business logic with documentV2 //maybe save in db with V2 transformObject(document, 'v2', 'v2|v1', options, (err, documentVx) => { res.json(documentVx) // same version as we receive }); }); First “Upgrade”, then “Downgrade”
- 14. GET /documents/v2 Find documents in DB Returnall documents v2 (via transform stream) (Stream all documents to res) (get readable stream) Transform all documents v2 -> v1
- 15.
- 16. const through2 =require('through2'); let stream = through2.obj((obj, encoding, callback) => { //async transform single object here through.push(); callback(); }); Transform stream
- 17. function getTransformStream(options) { returnthrough2.obj(function(obj, encoding, callback){ var through = this; if(!preparedDataSets) {// Do all async stuff once prepareTransform(options, (err, dataSets) => { through.push(syncTransformReadyData(obj, dataSets, options)); callback(); }); } else { through.push(syncTransformReadyData(obj, dataSets)); callback(); } });
- 18. POST and PUT/api/entity/v1 /api/entity/v2 Transformv1 to v2 data format /api/entity/v3 Transform v2 to v3 data format Business logic (only accepts v3 data format in) API version layer GET and result of POST and PUT /api/entity/v1 /api/entity/v2 Transform v1 to v2 data format /api/entity/v3 Transform v2 to v3 data format Business logic (only returns v3 data format out) API version layer v2, v3Handle different versions in DB v2, v3 Handle different versions in DB v3 v3
- 19.
- 21. var transformConfig ={ v2: { V1toV2: { }, V2toV1: { }, }, v3: { V2toV3: { transform: function (data, preparedData) { //sync code return data; }, prepareTransform: function(options, callback) { //async code return callback(error, asyncData); } }, V3toV2: { transform: function (data) { //sync code return data; }, prepareTransform: function(options, callback) { //async code return callback(error, asyncData); } } } };
- 22. Possible categories of solutions: Withdowntime Without downtime
- 23.
- 24. API request (POST, PUT,GETdocument) Patch user data Document is “patched”? Proceed with business logic No documents.findOne({id, schema: ‘v2’}) ? Yes documets.update({...,id, schema: ‘v2’})
- 25. Any API REQUEST Patchuser data User data is “patched”? Proceed with business logic No settings.findOne({id, patched: true}) Yes invoices.update({..., patched: true}) transactions.update({..., patched: true}) expenses.update({..., patched: true})
- 26. But there couldbe something better and more secure
- 27.
- 30.
- 31. module.exports = setup((patch)=> { patch.version('2.4.3'); patch.update('settings', search, (settings, callback) => { return callback(null, modifier); }); patch.after(whiteList({ 'fieldThatAllowedToBeChanged': ['added|removed|modified'], })); });
- 32.