Meilisearch-laravel-scout: I can't index my documents (they are ok at Algolia)

Hello @elfeffe,

This is hard to help you without more information.

if you log your meilisearch instance did you get:

"POST /indexes/local_product/documents?primaryKey=id HTTP/1.1" 202 14 

If you have use of toSearchableArray() in your Product model, can you show me the content, so I might be able to reproduce the same error.

I’m using MongoDb. I think the key is _id
Can be that?
I use toSearchableArray to filter some fields, but they index on Algolia. And I dump the final array and it’s ok.
I don’t have the computer now, what do you want me to try?

I can't understand what you mean with

if you log your meilisearch instance did you get:

Let me know what do you want me to try

If I try

curl \-X POST 'http://localhost:7700/indexes/local_product/documents' \--data '[{"id": 287947,"title": "Shazam","poster": "https://image.tmdb.org/t/p/w1280/xnopI5Xtky18MPhK40cZAGAOVeV.jpg","overview": "A boy is given the ability to become an adult superhero in times of need with a single magic word.","release_date": "2019-03-23"}]'

I get
{"updateId":31}

But then I do
curl -X GET 'http://localhost:7700/indexes/local_product/documents'
And I get

[]
Again

Hello @elfeffe

You can always check an update status, most of the times MeiliSearch will tell you what the error was, in case the update failed.

You can check: Get an update status

And for the last example you can try

curl -X GET 'http://localhost:7700/indexes/local_product/updates/31'

which should give you more information on what is happening.

I’m using MongoDb. I think the key is _id
Can be that?

When you have created your index local_product did you specify the key parameter ?
php artisan scout:index local_product --key _id

I use toSearchableArray to filter some fields, but they index on Algolia. And I dump the final array and it’s ok.

Can you also try by comment the toSearchableArray(), I don't think this is the problem here, but it cost nothing to try.

I have removed toSearchableArray()
Then I have tried doing:
php artisan scout:index local_product --key _id
Doesn't work.

Then doing
php artisan scout:index local_product --key id
Together with

/**
     * Get the value used to index the model.
     *
     * @return mixed
     */
    public function getScoutKey()
    {
        return $this->id;
    }

    /**
     * Get the key name used to index the model.
     *
     * @return mixed
     */
    public function getScoutKeyName()
    {
        return 'id';
    }

Nothing, when I do
curl -X GET 'http://localhost:7700/indexes/local_product/documents'

I just get one document, the one indexed using

curl \-X POST 'http://localhost:7700/indexes/local_product/documents' \--data '[{"id": 287947,"title": "Shazam","poster": "https://image.tmdb.org/t/p/w1280/xnopI5Xtky18MPhK40cZAGAOVeV.jpg","overview": "A boy is given the ability to become an adult superhero in times of need with a single magic word.","release_date": "2019-03-23"}]'

Is there any log that I can check?

When you import your data, did you get something like this:

shok@pop-os:~/web/blog$ php artisan scout:import "App\Models\Post"
Imported [App\Models\Post] models up to ID: 100
All [App\Models\Post] records have been imported.

logs are located storage/logs/laravel.log

Did you set queue => true in scout.php or SCOUT_QUEUE=true in your .env ?

I have tried with both true and false for queues. Always I get

Imported [AppModelsProduct] models up to ID: 5eea4b314c80095faf5be593
Imported [AppModelsProduct] models up to ID: 5eea4bd94c80095faf5be787
Imported [AppModelsProduct] models up to ID: 5eea4cc34c80095faf5be97b
Imported [AppModelsProduct] models up to ID: 5eea4dfa4c80095faf5beb6f
Imported [AppModelsProduct] models up to ID: 5eea4f6b4c80095faf5bed63
Imported [AppModelsProduct] models up to ID: 5eea51164c80095faf5bef57
Imported [AppModelsProduct] models up to ID: 5eea524e4c80095faf5bf099

But there are no ducuments.
I know where laravel's logs are located, but I have no errors there, what I want to see are the MS logs.
Is there any way to see what is happening?

With all the same settings, Algolia is indexing my documents.

Ok, Meilisearch laravel scout is only an engine like a copy paste of the Algolia one. So it shouldn't be a problem but also I have never tried to run it with mongodb...

@eskombro @curquiza can answer to were are the MS logs.

This is related to MongoDB, because Users model is MySQL and works.
I mean, this is related to MongoDB / Meilisearch, because Algolia also work with MongoDB.

It will be great if i can ee some logs from MS, it is sad if Algolia supports it but not Meilisearch.

I can try to dig the problem, but I need your help on how I can install mongoDB with laravel

As for now, MeiliSearch logs are directly printed to stdout, so it depends on where you are running your MeiliSearch instance. Where is it running?

I opened an issue for discussing persistent logs in MeiliSearch here: https://github.com/meilisearch/MeiliSearch/issues/898

@elfeffe did you check the update status as I suggested in previous comments? If the documents were successfully sent to MeilISearch and you got a status ID, you will very probably find inside that update status the information about what is the problem when MeilISearch tries to index. Try this: https://github.com/meilisearch/meilisearch-laravel-scout/issues/36#issuecomment-668787383

Look at this

curl \-X POST 'http://localhost:7700/indexes/local_product/documents' \--data '[{"id": 287947,"title": "Shazam","poster": "https://image.tmdb.org/t/p/w1280/xnopI5Xtky18MPhK40cZAGAOVeV.jpg","overview": "A boy is given the ability to become an adult superhero in times of need with a single magic word.","release_date": "2019-03-23"}]'

That works, but

curl \-X POST 'http://localhost:7700/indexes/local_product/documents' \--data '[{"id": 5eea4bd94c80095faf5be787,"title": "Shazam 2","poster": "https://image.tmdb.org/t/p/w1280/xnopI5Xtky18MPhK40cZAGAOVeV.jpg","overview": "A boy is given the ability to become an adult superhero in times of need with a single magic word.","release_date": "2019-03-23"}]'

"5eea4bd94c80095faf5be787" is a MongoDB _id
That returns this beauty

{"message":"Invalid JSON: invalid number at line 1 column 11","errorCode":"bad_request","errorType":"invalid_request_error","errorLink":"https://docs.meilisearch.com/errors#bad_request"}

I think that JSON is valid, there is some problem with MS and custom Key.
I don't know if it's the length, the letters, or what.

After some tries I think the letters are the problem, MS doesn't accept letters on the id. But Laravel let you put letters there.
That is the problem I think

Yes, you just need to send that as a string, surrounded by double quotes

It will work with "id": "5eea4bd94c80095faf5be787" instead of "id": 5eea4bd94c80095faf5be787 because 5eea4bd94c80095faf5be787 is not a valid number 🙂

But I don't control how your Scout extension send it 🙂

Even doing this

/**
     * Get the value used to index the model.
     *
     * @return mixed
     */
    public function getScoutKey()
    {
        return (string)$this->_id;
    }

It doesn't work, I think this engine must handle string ids

do the same thing in toSearchableArray()

return [ 'id' => (string) $this->_id]
or
return [ '_id' => (string) $this->_id]

I don't know what you have set as key

 public function toSearchableArray()
    {
        $array = $this->toArray();

        $array['id'] = (string)$array['_id'];

        return $array;
    }

That works

One last thing.
Algolia handle's that automatically, together with the index creation.
I mean, Algolia creates the index if doesn't exist, and I can index IDs with letters without any conversion.
I think MS should include both features.

I mean, Algolia creates the index if doesn't exist

Indeed, the MeiliSearch API does not handle this feature yet.
As an alternative solution, most of our SDKs provide a get_or_create_index method (as the PHP SDK does) 🙂

and I can index IDs with letters without any conversion.

The problem you had comes from the JSON validity. Even Algolia API does not accept invalid JSON: sending a number with letters without quoting it (") is considered as an invalid JSON and would fail.

See the Algolia dashboard:

Maybe the Algolia's Laravel SDK transforms your number with letters into a string and that's why you didn't have any failure 🙂

I think MS should include both features.

So maybe meilisearch-laravel-scout could include the "quoting feature", but MeiliSearch could not since it expects a valid JSON 😉

Yes, in both cases I'm speaking about this Scout Engine, we are in the Scout integration here.
The SDK can provide a method to create the index, but Scout fails if the index doesn't exist.

The same with Id with letters, I understand that JSOn is malformed, but the Scout engine shpuld control that before it fails.
As the Algolia one does, because that one doesn't fail with letters.

Ok ok wasn't sure to understand 😉

They could be nice features to implement then! 🙂

Edit: we should think if we implement that in the meilisearch-php or in the meilisearch-laravel-scout package.

@curquiza
It depends, both are good.

If we implement it in scout I guess the symfony(and others framework implementation) should also implement it (may be it's already the case). So it might be better to have it in the SDK instead ?

I'm not sure scout implement it for Algolia, I think this is handle by the Algolia PHP SDK, I will check this when I have a bit of free time.

@shokme, you're right, it might be smarter to do it in the PHP SDK rather in the laravel-scout integration.
Let us know about your investigation, I will move the issue (or recreate one) accordingly! 🙂

So, after investigating the whole thing.
I have installed mongodb in laravel using jenssegers/laravel-mongodb.

I have started by creating my index.
php artisan scout:index cars -k _id

create a model

<?php

namespace App;

use Jenssegers\Mongodb\Eloquent\Model;
use Laravel\Scout\Searchable;

class Car extends Model
{
    use Searchable;

    public function getScoutKey() // not required if this is already your primary key
    {
        return $this->_id;
    }

    public function getScoutKeyName() // not required if this is already your primary key
    {
        return '_id';
    }
}

creating a new car

$car = new App\Car();
$car->name = 'mustang';
$car->constructor = 'ford';
$car->type = 'muscle';
$car->horses = 280;
$car->save();

The model was send to meilisearch without any problem !

{
  "status": "processed",
  "updateId": 1,
  "type": {
    "name": "DocumentsAddition",
    "number": 1
  },
  "duration": 0.005275826,
  "enqueuedAt": "2020-08-29T17:43:55.104917708Z",
  "processedAt": "2020-08-29T17:43:55.121602037Z"
}

When doing a search I get the correct result

App\Car::search('mustang')->get();

```json
[
{
"_id": "5f4a93dbd9668370086c7752",
"name": "mustang",
"constructor": "ford",
"type": "muscle",
"horses": 280,
"updated_at": "2020-08-29T17:43:55.102000Z",
"created_at": "2020-08-29T17:43:55.102000Z"
}
]

The only problem I have encounter while trying to reproduce the bug was by forgot to set `_id` has primaryKey
and the update status was returning:
```json
{
  "status": "failed",
  "updateId": 3,
  "type": {
    "name": "DocumentsAddition",
    "number": 1
  },
  "error": "serializer error; Primary key is missing.",
  "duration": 9.817e-6,
  "enqueuedAt": "2020-08-29T17:21:00.098580201Z",
  "processedAt": "2020-08-29T17:21:00.100495474Z"
}

So, to be honest I don't understand why you don't get a string back on $this->_id @elfeffe

@shokme I don’t know. I will resume my work with search in a few days and I will let you know if I find something new.
The SDK should create the index automatically as Algolia does, with the correct I’d, to avoid mistakes.

The SDK should create the index automatically as Algolia does, with the correct I’d, to avoid mistakes.

It is planned in the MeiliSearch core-engine! 🙂
https://github.com/meilisearch/MeiliSearch/issues/918