Server-tools: [RFC] base_elasticsearch

Hello @lasley at Akretion we probably started what you need:
https://github.com/akretion/connector-nosql
also see the solr branch https://github.com/akretion/connector-nosql/tree/9.0-solr
cc @sebastienbeau
please let's work on this together. We have a great experience with integrating Odoo with Solr and Algolia NoSQL data stores already.

Nice, thanks @rvalyi - it does indeed look like you have started what I need.

Unfortunately one of the requirements of my project is that it has to be LGPL - I should have mentioned that in my RFC in hindsight (edited to include it). The LGPL requirement rules out the use of connector, and also rules out inclusion of this logic into base_external_dbsource. This requirement is due to my planned use of this module as part of my core SaaS billing infrastructure.

I see a lot of export logic here, but nothing for reading the data. Am I correct in that evaluation?

How did you circumvent the need to store data in Odoo?

@lasley I'm the author for the logic in the base_external_dbsource. I looked at the history and the only other original contribution is adding the Firebird DB connections. If it's important, we could consider relicensing that module to LGPL.
In fact, since they are just technical tools, I wouldn't see a problem for the server-tools repo to be as much LGPL as possible.

@dreispt - I would love to add the logic to the existing module if you're good with the relicense. I'm going to be using this base for metrics ingestion and subsequent billing of my SaaS offerings, so not allowing the AGPL to permeate into that is an absolute blocker.

At the beginning a simple Elastic connection is really all we need too, so base_external_dbsource fits. I would also likely refactor it a bit to get rid of the if chains inside of conn_open and execute that will become exponentially larger as more sources start being added.

NoSQL is a bit different in terms of querying, but I can still fit into the execute interface with a bit of creative adaptation.

@lasley OK. To respect all other contributions, such as porting from versions, I'll open an Issue to propose and discuss the license change.
And we should make the module "pluggable", where the support to additional databases, like Firebird, can be plugged in by an additional module.

Thanks @dreispt - much appreciated!

Regarding the pluggable - What did you have in mind? My planned refactoring was really just to break out the two giant if chains, but I'm open to other improvements while it's on my dev plate.

@lasley I mean making it easy to be extensible by another module. Additional db support could be added by an additional extension module.

@dreispt Ahhh yes gotcha. Definitely!

Any thoughts to me stripping out all of the independent database sources into their own modules? I feel like this try/except thing at the top is a bit unwieldy, do you agree? This is the current solution, but I think it could still be improved.

Is this something that we have to wait to v11 on because stable release? Or is releasing alongside with a replacement module(s) acceptable?

I also think it would maybe be nice to add a basic CRUD type interface as well so that maybe we could think of moving away from direct SQL.

IMO it would be cleaner for the extension module to just do the usual think: inherit the base.external.dbsource model to modify features and models.

It's reasonable to extract the db providers for 10.0, since it's a fresh version, but it probably won't be reasonable for 9.0 - and update would break existing installations.

10.0 is already released though & looks to be a 1:1 migration. I think we would have to wait to 11 to adhere to stable release policy, right?

Strictly speaking yes.
That means that we shouldn't remove features from the current module.

What if we made the new modules auto install in the lower versions? Would allow the isolation while I'm here, with an easy off switch for later.

The external facing API will be the same, so I don't think we'd cause any issues?

@lasley Yes, that's a good idea.

Hi there,

@lasley May be you want to look at https://github.com/camptocamp/odoo-elasticsearch-kibana

Regards,

Joël

Following up with what @jgrandguillaume shows above, we just discuss with him that what would be great would be to have a tool like https://github.com/OCA/reporting-engine/tree/8.0/bi_view_editor to define the reporting model in Odoo, and then to be able to export the data for this model into ElasticSearch, so that it can be combined with other data sources and then report on it using Kibana.

Now, I'm not exactly sure if this the approach that @lasley is following on this RFC?¿?

Interesting on both counts, thanks @jgrandguillaume & @jbeficent.

Is bi_view_editor that @jbeficent linked the upgraded form of the sql_view that is linked as a module dependency for odoo-elasticsearch-kibana?

I have kind of a multi-purpose intent with this RFC. Main intent is to push statistics out of Odoo, similar to what is being done in both linked libraries.

Slightly further down the road, I'm going to need to reverse some of the data paths. Kibana is a great visualizer for me, but I will need to build some of the dashboards out in Odoo for direct reporting via client dashboards.

@lasley See bi_view_editor here: https://github.com/OCA/reporting-engine/tree/8.0/bi_view_editor. It is not currently a dependency for odoo-elasticsearch-kibana. It is a tool that allows a normal user to design Odoo models for reporting purposes. Similar to doing an sql_view, but the power is on the hand of the user.

We also want to push statistics out of Odoo, and the aim would be that the user has the capability to design what statistics he'd like to export from Odoo.

Reverse path would also be great.

@jbeficent - Well that's pretty cool indeed. Creating something like this for NoSQL would likely be much easier too due to lack of relations.

I probably need to add some methods for indices into my #645 in order to support something like this, which would need a concept of index analysis that I didn't develop.

Ping to anyone interested - I put a prelim data design in https://github.com/OCA/server-tools/pull/660#issuecomment-273583948 if anyone wants to take a look. This would still be lacking a view framework, but I think maybe puts us on the right track.