How to Implement CSV Exports (Millions of Rows) in Laravel with a Distributed Approach Using Queues and Jobs

I was working remotely and needed to implement an export feature for a list of members on a system. The resulting file had to be a CSV. There were millions of members in the database, and the Laravel stream download wasn't working:

return response()->streamDownload($callback, 'download.csv', $headers);

Due to the sheer amount of data, the download would break midway and the full dataset never came back.

I also tried running the export inside a job, but the job kept running out of memory — even after unsetting variables I no longer needed. The problem: PHP's garbage collector runs on its own schedule, so there was no way to force a collection mid-loop.

Increasing the memory limit for the job was a no-go for me. It may work for some cases, but I wanted an approach that requires less memory, not more.

The approach: a job that re-dispatches itself

I created a job that processes one page of results at a time, appends rows to a file on disk, then dispatches itself again for the next page — until there are no more pages. The user downloads the pre-generated file afterwards, at which point the stream downloader works fine because the file already exists.

Here's a stripped-down version of the job:

<?php

namespace App\Jobs;

use App\Models\GeneralExport;
use Storage;

class CreateGeneralExportFileJob implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    public $timeout = 1200;

    public function __construct(
        private GeneralExport $export,
        private string $exportFileName,
        private int $page = 1,
    ) {}

    public function handle()
    {
        $members = $this->getMembers();

        $columns = [
            'Member ID',
            'Full Name',
            'Phone Number',
            'Gender',
            'Date of Birth',
            'Email',
        ];

        $filesystemAdapter = Storage::disk('public');

        if ($this->export->status === 'pending') {
            $fileName = 'general_exports/' . Carbon::now()->timestamp
                . '-' . $this->exportFileName
                . '-' . $this->export->user_id . '.csv';

            // Add headers only on the first run; subsequent runs append data only
            $filesystemAdapter->append($fileName, implode(',', $columns) . PHP_EOL);
        } else {
            $fileName = $this->exportFileName;
        }

        if ($this->export->status !== 'processing') {
            $this->export->update([
                'status' => 'processing',
                'status_message' => "Job {$this->page} in export processing started",
            ]);
        } else {
            $this->export->update([
                'status_message' => "Job {$this->page} in export processing started",
            ]);
        }

        $fileResource = fopen($filesystemAdapter->path($fileName), 'a+');

        foreach ($members as $member) {
            fwrite($fileResource, implode(',', [
                $member->id,
                $member->user->first_name . ' ' . $member->user->last_name,
                $member->user->phone,
                $member->gender,
                $member->dob,
                $member->user->email,
            ]) . PHP_EOL);
        }

        fclose($fileResource);

        $nextPageUrl = $members->nextPageUrl();
        $nextPage = null;
        if (!is_null($nextPageUrl)) {
            $nextPage = explode('=', $nextPageUrl, 2)[1];
        }

        if (is_null($nextPage)) {
            // All pages processed — mark as done
            $this->export->update([
                'status' => 'processed',
                'status_message' => 'Export file processed successfully and ready for download',
                'file' => $fileName,
            ]);
            return;
        }

        // Refresh to get current state before dispatching the next job
        $this->export->refresh();

        dispatch(new static($this->export, $fileName, $nextPage));
    }

    public function getMembers()
    {
        return Member::paginate(10000, ['*'], 'page', $this->page);
    }
}

To kick it off from your controller:

dispatch(new CreateGeneralExportFileJob($export, 'download.csv', 1));

Why this works

Low memory — each job only holds one page of 10,000 rows in memory at a time. When the job finishes, PHP cleans up that memory before the next job starts.

No overlapping — the next job is only queued once the current one finishes writing its page. No need for Laravel's WithoutOverlapping middleware or any coordination logic.

Resumable — if a job fails partway through, you can inspect the GeneralExport model's status and status_message to see exactly which page it was on.

Hope this helps someone facing the same issue. Cheers 🍻