Laravel-datatables: Server Side Export 200000 records

You need to implement your own excel file builder to handle this bulk request. On your ServiceDataTable class, you can override protected function buildExcelFile() method and maybe do some chunk, etc. See laravel-excel docs for more info.

    /**
     * Build excel file and prepare for export.
     *
     * @return \Maatwebsite\Excel\Writers\LaravelExcelWriter
     */
    protected function buildExcelFile()
    {
        /** @var \Maatwebsite\Excel\Excel $excel */
        $excel = app('excel');

        return $excel->create($this->getFilename(), function (LaravelExcelWriter $excel) {
            $excel->sheet('exported-data', function (LaravelExcelWorksheet $sheet) {
                $sheet->fromArray($this->getDataForExport());
            });
        });
    }

I think Following code creating exception
$this->getDataForExport();
Gets exception when server-side editColumn() defined or used
there are 200000 records.
Is there any other way to get data without processing or not allocating more memory.
I have following link but yajra yajra/laravel-datatables-buttons : ^3.1 yet not support Laravel/Excel : 3.0
https://laravel-excel.maatwebsite.nl/docs/3.0/export/queued

There should a way which directly export from query or eloquent way.

I use Postman to mimic that operation but

Allowed memory size of 134217728 bytes exhausted

exception thrown. I have no of rows which used to generate html, I use rawColumns() to escape html, My query takes 0.420 ms in phpmyadmin and At first 1.4 sec.

If we are able to put this onto queue, we should be also able to save it to a folder on server and then provide the link to user to download it.

Thanking you for response.

I am using it in JQuery Datatable or yajra datatable and want to export from server side.

I think it's not feasible to save into folder on server, there are many issues that will generate problem.

I need streaming of data.

Amount of data is of size 20MB in response or larger.

My requirement is similar, but I would like to save the file on to server (or S3) and give a link to download it either in my application dashboard or email. I am seeing the possibility of doing this, if I find something I will update here.

I have issue with serverside proccessing which consumes large amount of ram and default script memory allocation size is 60MB.
Look at memory_limit in php.ini
Execution time limit max_execution_time is 60.
After increasing value you were able to make request using serverside export.

I modified the excel function to do store the file instead of download, it is storing the file in storage/exports/ folder.

public function excel()
{
    $this->buildExcelFile()->store('xls');
}

currently I am not having large amount of data in my development env, can you make above change in you setup and see if it is able to create the excel in that folder without hitting the memory cap?

Another thing to note is I am using Laravel 5.1 thus maatwebsite/excel version 2.1

What I really want to do is put this in queue instead and return to page with a message that report creation is in progress and link to download the file will be email to your registered id"

in 2.1 version not sure how to queue, but I found this where they use "Queued Chunk" , but I am not sure how to implement this.

So any updates about this?

So any updates about this?

If anyone uses Laravel Excel 3.0+ the upper solution won't work. so you should override buildExcelFile() by using "Queued Export".

Or I did something else by using chunking also but by creating my trait and use CSV to do my work and this an example

any updates ?

Starting with Laravel 6 you can use LazyCollections to load only one item at a time in memory. This fixes the issue of running out of memory.

Only need to extend DataTable class and write an adapted DataTablesExportHandler.

Check out my gist

Enjoy!

@singhofmarco Can you make example?

@singhofmarco I agree, have started a v10 project last year that will utilize LazyCollections but can't continue it atm.

I went a different route. I've overridden the csv function, calling DataTables with one page of data each time, effectively chunking. Using fputcsv on the result rows until the file is complete, then just return the file content in a Laravel Response.

Not only is the memory limit never reached anymore (until the CSV itself grows beyond the PHP memory limit, then a streaming response might help further), but it's also more than twice as fast......

@ameenross if possible can you share your code please.

@karmendra I'd rather not, as it's for an older version of Yajra Datatables. But basically what I'm doing is I'm merging some page&length values in the request, call the code that normally returns a page of data for AJAX ($this->ajax()->getData(true)['data'] in my case). I have a base class for datatables which extends the Yajra Datatable class, with a function getCsv (decided to not override the CSV function after all, but that's not important). This goes in a loop until there are no more pages left.

@ameenross Thanks that is helpful. I will give this a try.

Laravel Datatable:: v1.5.0
Here is my solution based on @TheGeekyM comment, i leave it in case it helps someone else.
I created a ChunkedDatatableExportHandler class that implements FromQuery and then override the buildExcelFile method in my ServiceDataTables to use it.

<?php

namespace App\DataTables;

use Yajra\DataTables\Services\DataTable;

class ServiceDataTable extends DataTable
{
  protected $exportClass = ChunkedDatatableExportHandler::class;

  protected function buildExcelFile()
  {
      $query = app()->call([$this, 'query']);
      $query = $this->applyScopes($query);

      $dataTable = app()->call([$this, 'dataTable'], compact('query'));
      $dataTable->skipPaging();
      $data_query = $dataTable->getFilteredQuery();

      return new $this->exportClass($data_query);
  }
}

<?php
namespace App\DataTables;

use Illuminate\Database\Query\Builder;
use Maatwebsite\Excel\Concerns\Exportable;
use Maatwebsite\Excel\Concerns\FromQuery;
use Maatwebsite\Excel\Concerns\WithHeadings;

class ChunkedDatatableExportHandler implements FromQuery, WithHeadings
{
    use Exportable;

    /**
     * @var Builder 
     */
    protected $query;

    /**
     * ChunkDatatablesExportHandler constructor.
     * @param Builder $query
     */
    public function __construct(Builder $query)
    {
        $this->query = $query;
    }

    /**
     * @return array
     */
    public function headings(): array
    {
        $first = $this->query()->first();

        if ($first) {
            return array_keys((array)$first);
        }

        return [];
    }

    /**
     * @return Builder
     */
    public function query()
    {
        return $this->query;
    }
}