How to Create Paginated Records in MongoDB and Node Js and Node Rest API Server Side

In a real-world scenario, most of the time we have to deal with a huge number of data. So imagine a scenario where you have 1 million records in the database and your requirement is to present it on a web page using an REST API. you just can’t fetch all records and present it, it’s just not possible and feasible. To handle such cases, we use pagination it's presented record page by page, just like the book.

There are two ways you can implement pagination.

• Client Side Pagination
• Server Side Pagination

Client Side Pagination

Client-side pagination makes one query to the server to fetch all data. The client-side keeps all data in the memory and displays a portion of them by page index. Client-side pagination always has the total data count.it is fast and easy but not reliable to hold large volume (because of browser crash you know).

Server Side Pagination

For a large volume of data, we use server side pagination.today, we are going to learn server side pagination and we are storing data in mongodb and paginating it using api’s built-in node js.

Prerequisites

Make sure you have Nodejs version v16+ and MongoDB latest installed. You can refer Node and MongoDB official site for the installation instruction.And, we need some large amount of data. If you don’t have the data, let us generate some mock one.

Where do you have to generate online data Click here ? 

After download the file. Now, let’s dump those records in our MongoDB database.

Step : 1 Dumping data in MongoDB

MongoDB comes with the command line program called Mongoimport to import a huge amount of data in the MongoDB collection. We are going to use the same command line tool to import our mock data.

Copy C:\Program Files\MongoDB\Server\5.0\bin> location and open this in CMD

This is my computer bin location you can find your bin directory where install MongoDB Change it to your need.

Syntax

mongoimport -d database -c collection --type csv mockdata.csv --headerline
//Or
mongoimport -d database -c collection --type json mockdata.json --headerline

Example

mongoimport -d JSNInfratechLlp -c salebillmas --type csv salebillmas.json --headerline

The JSNInfratechLlp is the database name and salebillmas is collection name. Change it to your need.

Once the data is imported, you can check that in Mongo shell using the following command.

use JSNInfratechLlp;
db.salebillmas.find({});

Step : 2 We are Writing Server code for pagination

Before we jump into the code, let’s understand the basic math behind the pagination.

There are two parameters involved in pagination.

• Size – How many records per page
• Page No – the number of the page

Suppose you have 10000 documents and you want 10 documents per page i.e size of the page is 10. So a number of pages are 10000/10 = 1000.

Step : 3 Start a new Node project.

npm init --y

Install the dependencies.

npm i --S express mongoose

We will work to make the model and controller and route all separate so that the solution is easy

App structure

app
-config
--database.config.js
-controller
--sale.controller.js
-model
--sale.model.js
-routes
--api.js
index.js

index.js

const express = require('express');
// set up our express app
const app = express();
app.use(express.json());

// connect to mongodb
const dbConfig = require('./app/config/database.config.js');
const mongoose = require('mongoose');
const { type } = require('express/lib/response');
mongoose.connect(dbConfig.url, { useNewUrlParser: true, useUnifiedTopology: true })
    .then(() => {
        console.log('database connection stable !');
    })
    .catch(error => {
        console.log('database connection failed ' + error);
    });

mongoose.Promise = global.Promise;
// initialize routes
app.use('/api/v1', require('./app/routes/api'));
// error handling middleware
app.use(function (error, request, response, next) {
    response.status(422).send({ status: false, message: request })
    next()
});
// listen for requests
app.listen(process.env.PORT || 5000, function () {
    console.log('Server running On this URL ! {' + process.env.MONGO_HOST + ':' + process.env.PORT + '}');
});

database.config.js

require('dotenv').config();
//live connection
module.exports = {
   url: 'mongodb+srv://astrologersapp:astrologersapp@astrologersapp.1ubet.mongodb.net/astrologersapp?retryWrites=true&w=majority'
}

sale.model.js

const mongoose = require('mongoose');
var aggregatePaginate = require("mongoose-aggregate-paginate-v2");
const sale = mongoose.Schema(
    {
        blno: {
            type: Number,
            required: true,
            unique: true
        },
        bldate: Date,
        bltime: String,
        acccode: String
    }, {
    timestamps: true
});
sale.plugin(aggregatePaginate);
module.exports = mongoose.model('salebillmas', sale);

sale.controller.js

const salebillmas = require('../models/sale.model.js');
// Sale Summry Reports
exports.saleSummry = (request, response) => {
    var date = new Date();

    priceQtyConversionStage = {
        $addFields: {
            convertedRate: { $toDecimal: "$salerate" },
            convertedQty: { $toInt: "$totqty" },
            convertedVatamt: { $toDecimal: "$vatamt" },
            convertedTotal: { $toDecimal: "$total" },
        }
    };
    // Define stage to calculate total price by multiplying convertedPrice and convertedQty fields
    totalPriceCalculationStage = {
        $project: { ward_no: 1, ward: 1, shopcode: 1, blno: 1, totalPrice: { $multiply: ["$convertedRate", "$convertedQty"] }, totalQty: { $sum: "$convertedQty" }, vatTotal: { $sum: "$convertedVatamt" }, saleTotal: { $sum: "$convertedTotal" }, }
    };
    // Define stage to filter data by shopcode
    if (request.body.shopcode.length > 0) {
        WhereAndShop = {
            $match: {
                $and: [{ "shopcode": { $in: request.body.shopcode }, }],
            }
        }
    } else {
        WhereAndShop = { $project: { shopcode: 0 } }
    }
    // Define stage to filter data by shopcode

    if (request.body.ward.length > 0) {
        WhereAndWard = {
            $match: {
                $and: [{ "ward_no": { $in: request.body.ward }, }],
            }
        }
    } else {
        WhereAndWard = { $project: { ward_no: 0, ward: 0 } }
    }

    if (request.body.from_date.length > 0 && request.body.to_date.length > 0) {
        WheredateRange = {
            $match: {
                "bldate":
                {
                    $gte: new Date(request.body.from_date),
                    $lt: new Date(request.body.to_date)
                }
            }
        };
    } else {
        WheredateRange = {
            $match: {
                "bldate":
                {
                    $gte: new Date(date.setMonth(date.getMonth() - 1)),
                    $lt: date
                }
            }
        };
    }

    var aggregateQuery = salebillmas.aggregate([
        WhereAndWard,
        WhereAndShop,
        WheredateRange,
        priceQtyConversionStage,
        totalPriceCalculationStage,
        { "$project": { "_id": 1, "blno": 1, "zone": 1, "totalPrice": 1, "totalQty": 1, "vatTotal": 1, "saleTotal": 1, "shopcode": 1, "ward_no": 1, "ward": 1, } },
    ])

    salebillmas.aggregatePaginate(aggregateQuery, {
        limit: request.body.limit ? parseInt(request.body.limit) : 10,
        page: request.body.page ? parseInt(request.body.page) + 1 : 1
    }).then(returns => {
        response.status(200).send({
            status: true,
            message: "Sale Summry Data found !",
            data: returns
        });
    }).catch(errors => {
        response.status(500).send({
            message: errors.message || "Some error occurred while retrieving student."
        });
    });
};