+1 for nested object aggregation.

+1

+1

+1

+1

+1

+1

+1

+1

+1

+100000

+1

to be clear, there is no way to do a nested filter/query/agg in kibana 4 right now, is there?

+1

+11111

+1

+1

+1

+1

+1

+1

+1

+2

+1

+1 Because denormalizing nested objects is not always an option as this may lead to a mapping explosion.

Mapping:

{ 
 "timestamp":{ "type":"date"},
 "cluster_id": { "type":"string"},
 "pools":{
    "type":"nested",
    "properties":{
      "size":{
        "type":"long"
      },
      "name":{
        "type":"string",
        "index":"not_analyzed"
      }
    }
  }
}

First of all I would like to the line chart to be able to show the average size over time for each pool name. Assuming that there are that many names so that denormalizing is not a good idea, this might lead to many graphs in the chart. In order to work with such cases it would also be beneficial to be able to use the filter agregation inside a nested aggregation. Being able to filter on nested fields in the top search would also be great.

To make things even more interresting it would be really great to be able to visualize an aggregation like this:

"aggs": {
        "poolagg": {
            "nested": {
                "path": "pools"
            },
            "aggs": {
                "old": {
                    "filter": {
                        "term": {
                            "name": "some pool name"
                        }
                    },
                    "aggs": {
                        "avg_size": {
                            "avg": {
                                "field": "size"
                            }
                        },
                        "distribution": {
                            "histogram": {
                                "field": "size",
                                "interval": 5
                            },
                            "aggs": {
                                "pool_to_cluster": {
                                    "reverse_nested": {},
                                    "aggs": {
                                        "clusters": {
                                            "cardinality": {
                                                "field": "cluster_id"
                                            }
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }

+1

+2

+1

+1

+1

+1

+10 !

It would be powerfull with this !

So can someone clarify this; In this post (https://www.elastic.co/blog/kibana-4-beta-1-released) for Kibana4beta1 it states that "Kibana 4 brings the power of Elasticsearch's nested aggregations to the click of a mouse. " However I'm unable to create any visualizations on documents with nested objects. I've also made sure the nested objects in my index template are marked as "nested". So is Kibana's support for nested aggregations not the same as ES's support for nested objects? What am I missing? Thanks.

@cslinuxboy - I think they're using "nested" here to refer to grouping via multiple fields, eg "aggregate by time then geo" (not "nested" as in its use in the platform for "nested objects")

@Alex-Ikanow - Thanks for the reply. Too bad this is not possible at this time. I got my hopes up when reading the misleading description on their beta-1 post.

+1 for support of nested objects within visualization aggregation.

+1

I'm currently using parent/child relationships as a workaround which seems to be working fine.

@calvdee Have you got has_parent or has_child queries to work in the Kibana search bar? This isn't working for us and it's a huge problem, I would be ETERNALLY grateful if you have this working and can let me know...thanks!!!

No, for our use case all parents have children and all children necessarily have parents since we are indexing invoice data so regular queries just work (see image).

@calvdee Thanks so much for the reply! We have a similar data model but want to be able to find parents by their children in Kibana, it's not working :0(

No worries, good luck!

On Thu, Mar 26, 2015, 11:36 ajrasch [email protected] wrote:

@calvdee https://github.com/calvdee Thanks so much for the reply! We
have a similar data model but want to be able to find parents by their
children in Kibana, it's not working :0(

—
Reply to this email directly or view it on GitHub
https://github.com/elastic/kibana/issues/1084#issuecomment-86575796.

+1

+1

You read the _mapping field, so you should know when a particular field is nested, so can it not automatically apply the correct nested facet/query when such a field is selected in queries or facets?

Yes, :+1: on this one. Although I managed to write the correct nested aggregation on CLI, and can pull data with curl, I haven't found a way to get it working in Kibana's "Visualise" tab, not even using the JSON edit box. Admittedly, I never use that feature (the box), but it seems that it is possible only to "add" stuff to an existing aggregation, not to use it for creating a new aggregation from scratch... (will appreciate to be corrected if I am wrong on this!).

Yes, nested type aggregations are key and becoming more widely used because they solve a specific problem with flat data.

If Kibana4 is the visualization product for ES, it should support all ES aggregations.

It would be nice to at least see this on the roadmap for Kibana4.

+1

+1

+1

+1

+1

From #3729 :-)

I would like to see an aggregate option "child documents"
(after date histogram, histogram, etc.) that opens up the
parameters for a query DSL for children aggregate, like
"child type", "fields", etc.

This will allow one to create nested aggregates on child
documents.

I don't seem to be able to create such a query using
the advanced editing feature. May be someone can
enlighten me.

Thanks

+1

+1

+1

+1 really need this ability. I could strip out all parents with logstash as a workaound, but that would require a huge config file as I have hundreds of fields.

+1 anyone know if this is planned?

+1

Right now, I'm considering changing some of my mappings to stop using "nested" type objects, since I can't create Kibana visualizations on any of those fields. If I knew the issue was at least on the roadmap, it would be a really big help in making that decision.

@benjismith - I also bit the bullet and switched the processing of my data from nested to parent/child docs. So far it's working nicely, but I agree with you; it would be nice to know if there is any chance of this ever becoming a feature in Kibana so we all can either wait for it or move on.

Best of luck.

+1

+1

+1 for nested type aggregation support

+1

+1

+1

+1

+1

+1

For those that are looking to do simple aggregations across specific fields within the nested documents (e.g. a "terms" aggregation on a field to find the top values), you may consider adding the "include_in_parent" setting to your Elasticsearch mapping. This creates a flattened version of a field in the nested document at the parent level.

https://www.elastic.co/guide/en/elasticsearch/reference/1.7/mapping-nested-type.html

These fields will not show up in the Discover field list on the left side, though you'll see them in the document detail view as an array field. However, you'll have access to the individual fields in the metrics list in Visualize, and you'll be able to run aggregations on them.

This is what we're doing to visualize Watch History, which is based on a nested document structure (see "results.actions"):

{
   ".watch_history-2015.06.12": {
      "mappings": {
         "watch_record": {
            "dynamic": "strict",
               "result": {
                  "dynamic": "true",
                  "properties": {
                     "actions": {
                        "type": "nested",
                        "include_in_parent": true,
                         …
}

+1

+1

+1

+1

+1

Instead of another +1... Maybe a summary of where Nested Type Aggregations stands would help.

I'm 5 days into learning how to make pretty graphs so forgive me if any of this is obvious.

What's the problem?

Is it an ElasticSearch or Lucene query problem still?

It isn't solved by aggregations, but it is solved by the fact that we allow you to type elasticsearch JSON in the input box. It isn't ideal, but unless elasticsearch extends lucene query string syntax to specify nested fields, its the best we can do. -- rashidkpc's year old comment

Kibana Fix Possible?

If ES/Lucene, could Kibana provide an intermediate hack/solution in the meantime? Think ES6 shims and vendor prefix CSS.

For nested mapping : opportunity to choose nested in the editor (and configure a path...) for Kibana's panels. bobmercer

Or:

You read the _mapping field, so you should know when a particular field is nested, so can it not automatically apply the correct nested facet/query when such a field is selected in queries or facets?
Alex-Ikanow, OP

Is anyone hacking on a solution? Anyone with an idea/direction of where to hack?

Work arounds?

Has anyone had any success with nested/include_in_parent? Does it require the dynamic: static, dynamic: true. My attempts failed with 0 results. tbragin

Anyone have examples for the JSON input box alluded to above by rashidkpc?

Parent/Child

This is my next step. I'm sure there's plenty of reference material online for this, but it wouldn't hurt to reference examples/tutorials for this alternative.

Lacking knowledge of kibana internals, I'm considering writing a REST proxy sitting in between elastic and kibana. When querying from Kibana for a specific type given some term-based search criteria, this proxy first queries the parent type to find the set of parents fulfilling the search criteria (they fit memory). Then it finds all children for these parents and returns these to kibana denormalised with all the parent fields added. This will allow us to have a parent-child model in Elastic Search which avoids the storage explosion from denormalising everything to billions of child objects, and at the same time graph the data based on parent fields.

Ideally this was part of Kibana. The world is not flat!

+1

+1. We used to split function stack or url_args into different fields. But this came with too large cluster state and too many mapping refresh actions. So we change this into a nested object. Now, we need aggs in K4....

+1

+1 need parent -child aggregation visualization.

+1

Please do not believe that all Elastic users who wish to use Kibana are using it for log analysis. We have extensive data, with nested objects, populated in our cluster that we would love to be able to perform analytics against without having to pull and transform the data into yet another system.

Being able to create nested aggregations, nested queries, and even reverse_nested aggregations are a must have. The longer Kibana goes without this feature equates to the sooner we must find an alternative without using Elastic/Kibana. If providing an easy to use UI for this kind of feature is the difficult part, start with providing the ability for your users to provide the complete json query for elastic that returns the required data.

+1

Agreed with @ppadovani . We were evaluating BI tools and would have loved to use Kibana, but it didn't support the relationships in our business entities we need to report on, and wasn't user friendly enough for a non-tech user to explore. We ended up going with Looker (a GUI for SQL essentially). Looking at looker might offer some ideas of how kibana could be extended to serve more diverse use cases in the future.

I decided to look at the kibana 4.1 code .. (I can't use master as it only works with 2.x elastic) .. Please correct me if I'm wrong but it would seem an easy first step to do something along these lines:

1) Under the 'advanced' collapsible area of an aggregation, add a text field called something like 'nested path'.
2) If the user places a string in that field, then the aggregation this is set on, will have a nested aggregation prepended to it like this:
"aggs":{
"2":{
"nested":{ "path":"foo" },
"aggs":{
"3":{
"date_histogram":{

To handle the case of multiple levels of nested objects, you could add a + that allows the user to add additional nested paths. Additionally, to handle the reverse nested, just add a checkbox labeled 'reverse'.

This would at least provide limited 'nested' aggregation support.

For nested query support, the only solution I can think of short term, would be to allow the user to enter hand coded elastic search json.

+1

+1 on all this and allowing user to enter hard coded elastic search json

+1

+1

+1. I was very excited about using Kibana for our environment visualization needs, but without support for nested object visualization it's pretty painful to use Kibana for our purposes there.

+1, just ran into this.

+1

+1

+1

+1

+1

At least 82 people get your "+1" and this is NOT helpful.

I disagree strongly. More data is never bad.

My team and I are working on this and hope to maybe have something to show by the end of the week.

EDIT: Please see my comments in: https://github.com/elastic/kibana/pull/4645#issuecomment-132908544

+1

+1

+1

Done and a pull request created here:
https://github.com/elastic/kibana/pull/4806

Quick update for those watching... waiting on Elasticsearch co. to ack the CLA, otherwise I think it is good to go.

+1

+1 as well

+1

+1

+1

+1

+1

+1

+10

+1

+1

+1

+1

Yes please.

+1

+1

+1

+1

The pull request against 4.1 is closed in favor of this one against master.

https://github.com/elastic/kibana/pull/5411

If there is demand for a 4.1 version of this, I can reopen the pull request with the correct code.

+1

+1

+1

+1

I don't understanding what is blocking on this issue.
There is a pull request waiting https://github.com/elastic/kibana/pull/5411

There is people here ready to contribute. What need to be done for merging this PR ?

Definitely more +1 are needed

@Filirom1 A few of us have been on the road, so we haven't had a chance to review the PR yet :( We'll try to get to it this week. It's definitely a much-needed enhancement and we are very excited about the community contribution here!

Great :-)
Thanks !

I took a look at #5411 and unfortunately it isn't the direction that we wish to go with this right now. #5411 implemented a stop gap, but caused important features, like filtering and searching to break. It also caused the agg builder it self to appear broken to those that do not understand the underlying implementation of nested aggregations. After we sat down and look deeper we realized the amount of work to support nested documents in a cohesive way was extensive.

Its not that we don't want to do this, its that if we do it, we want to do it properly, not just work around it.

We recognize that many are interested in visualizing relational data, however given other priorities and concerns we do not have resources to dedicate to this project at this time and are unable to commit to adding it to the roadmap for the foreseeable future.

FYI we were hoping to use kibana with nested aggregations for internal analytics but ended up going with a commercial product, 'Looker', that sits directly on our rdbms. I would highly recommend that the Elastic developers take a look at what they've done to make exploring a relational model simple, as many non-tech people are now able to browse our live db for their day to day questions about the product. I evaluated a number of products and Looker came out on top, and I'd love to see similar functionality in Kibana one day.

Given Rashid's latest comment and the extensive waiting for this feature, I recommend that this issue be closed. If this issue stays open then it just give users false hope that there will be any possibility of this being implemented in the near future. Let's close it and bench the idea until the developers have resources to figure out how to implement it.

Copying my post from the pull request:

There IS a solution here, but it would require a significant amount of work and I just don't have the time. We implemented this in JAVA so I know this is possible.

1) Each index mapping needs to pull and understand nested fields.
2) Build a custom AST that provides a simplified query language instead of attempting to just use Elasticsearch's.
3) Build a query adapter that understands the AST and can validate and convert the query to the proper JSON.
4) Update the aggregations in Kibana to properly handle nested fields based on the internal understanding of nested fields.

This is not impossible to do, just requires significant work. The benefits for implementing the above include query validation and a simplified syntax. For example our AST allows us to create a query like this:

(owner.user = "/users/00a0/18066271-29f0-40af-83ad-e5a0c8fc5944") AND (druid = "/druids/0060/77dd14b1-b7f0-4851-9ef8-74daa18d9d4d") AND ((owner.lastMessageReceived.inserted >= 0) OR (conversationLifecycleState = "RESERVATION_REQUEST")) AND (owner.conversationArchived = false) AND ((units.site IS NULL) OR (units.site IN {"HOMEAWAY_DE"})) AND ((inquiry.inserted >= 0) OR NOT (reservation.availabilityStatus = "DELETE")) AND NOT (owner.markedSpam = true) AND (lastMessage.inserted >= 0)

How exactly could I make this query with the existing language?

IMHO - waiting for nirvana from Elasticsearch before acting will lead to a drop in adoption and loss of users of Kibana.

Hate to sound like i am doing marketing but this is relevant. (seen people mention other products..)

We found a lot of ppl that were using heavily nesting wanted to actually search relational data and ended up nesting records that instead should have been simply "joined" late. (Padovani, this might be your case, i see "users" "messages" etc.. these would be very nicely kept as separate records)
This is the reason we created the SIREn Join elasticsearch plugin and the Kibi our -friendly - Kibana fork that offers filters and functionalities for this.

We're now working to make as much of Kibi as possible into Kibana 4.4 (aka 5.0) compatible plugins for the benefits of all.

The Join plugin has been released yesterday and it's open source also.

In the meanwhile, the version available http://siren.solutions/kibi frankly works like a charm and had many of our customers not need nested data any longer.

jccq: Never knew of Kibi or the join plugin. Thanks for the info!

+1

+1
this is a must have issue...

@jccq Our use case is not about joining only for queries, we use the 'joined' entity or 'view' as we call it as true entity data with its own lifecycle. This allows clients to pull the joined data without having to make multiple GET calls to our API.

In terms of nested support in Kibana, we are evolving our approach and may have something to share with the community sometime in Q2 or earlier depending upon resources and time. This new approach would support nested seamlessly in both the aggregations AND the queries. I will not say more, as it is still in very early stages of the implementation.

+1

@ppadovani you are the one !

+1

+1

+1
It is really important for us.
We are waiting for almost a year for this feature...

+1

Update: I don't see my new branch being ready for preliminary feedback or testing for at least a month due to my work load. However I wanted to put forward what I'm doing to gather any feedback from the community as I move forward.

The basic design revolves around two things:

1) A new query parser for the Kibana free form query field. This parser uses the standard Bison syntax definition (see the Jison project for the javascript version that I am using). The BNF I am using is based on the existing BNF we use at Homeaway for our custom query language against Elasticsearch. See my comment above for an example. I chose this approach to allow future enhancements by the community as needed.

I have the query parser working in Kibana, but still have work to do to allow the user to switch between the existing query style used by Kibana and this new style.:

2) Change the getFieldMapping call in mapper.js to getMapping and process the results differently such that the nested path on fields is captured and added to the field information that Kibana stores.

When a query is typed in the parser will validate that not only have fields been correctly named, but that the values provided are valid for the field types. I.e. a date field gets a date, or a boolean was given a boolean. Additionally, nested fields will be handled automatically by the parser and the proper Elasticsearch query json will be generated instead of the simple query language that is used now.

Finally for aggregations, since the fields now contain the clues about nested paths, it will be trivial to work against my previous branch to automatically inject the nested aggregations as needed based on the fields selected.

Milestones:
1 - Get the parser functional, and generating queries
2 - Update mapper.js, and implement nested query support
3 - Implement nested aggregation support
4 - Testing/cleanup

Any feedback on this approach would be greatly appreciated. Thanks!

Update on the above:

• Parser complete
• Reverse parser complete (takes an elasticsearch json and converts it back to the custom query language)
• Kibana now figures out and saves nestedPaths on fields
• Parsers now have access to the field information

Still todo:

• nested path support in both parsers
• UI construct to allow user to switch which query style should be used, and allows for legacy queries to be saved/used.
• finish error handling in parsers, and how the UI displays parsing issues/mistakes
• UI construct to provide type-ahead support for field names in the query
• nested support in buckets/metrics
• unit tests, unit tests, unit tests.
• ... ?

+1

Update:

• Parser automatically handles/injects nested information
• Aggregations now automatically handle/inject nested information

TODO:

• UI changes for query style selection, and saving style selection with query in kibana index.
• error handling for parser errors, and how UI displays parsing issues/mistakes
• UI type ahead support for query building/writing
• unit tests, unit tests, unit tests...

Plan:
I want to get at least some progress on the UI work, not my strong suit, then we will push a branch/fork out to the HomeAway github repo to allow for feedback/contributions. I'll post here again when this is done so that any of you that want to pull the fork and play with it are more than welcome to. Once it is polished enough I'll create the official pull request.

One final note: This work is being done against the Kibana 4.3.1 branch.

As a follow-on to my previous comment about using "include_in_parent" and "include_in_root" to copy selected fields from nested documents to the top level for the purposes of running aggregations on them, in ES 2.0 "copy_to" functionality was introduced that provides another option for this sort of thing: https://www.elastic.co/guide/en/elasticsearch/reference/current/copy-to.html
There is some talk of deprecating "include_in_parent" and "include_in_root" in favor of "copy_to" in a future ES version: https://github.com/elastic/elasticsearch/issues/12461 If you have experience with both, feel free to weigh in.

+1

ppadovani, appreciate what you are trying to do. This feature is very important to us
A few questions:

1. I understand this will take time? Is there an estimated time this feature will be available?
2. Has anyone tried any alternative? Like changing the log format from nested json(arrays) to something else? If so what should the intended format be, to work with ELK ?
3. Is there any other product in market which can help achieve this functionality? I am all for ELK because it is open source, but until it is not, we want something cheaper than Splunk. We explored many options, like Loggly, sumologic, logentries, logscape,graylog(either as expensive as splunk or they do not have this functionality)

Thanks much!

1. Has anyone tried any alternative? Like changing the log format from nested json(arrays) to something else? If so what should the intended format be, to work with ELK ?

You can flatten the schema or use ES mapping options "include_in_parent" or "copy_to" to copy some of the fields from nested documents to parent documents. Doesn't work for all use cases, but in some cases this will allow you to use Kibana out of the box. We use the "include_in_parent" approach internally at Elastic.

1. I have a branch that 'works', but needs more TLC in the form of UI work. This isn't my primary job so I am only able to work on it as I have time.
2. As @tbragin indicates, you can flatten the data. However this can lead to invalid query results.
3. I am not aware of any alternatives at this point.

To be more clear as to what the query language looks like, since I've been asked on my old pull request, here is a BNF summary:

Comparison: field [=,<,>,<=,>=,~=] value
Note the ~= .. this indicates LIKE which in turn will cause a wildcard query
IN: field IN {value,value,...} Set operation
field IN [value,value] Range operation using [ ] or ( ) depending upon inclusive/exclusive
IS: field IS NULL
expression: IS | IN | Comparison
NOT: NOT expression
AND | OR: expression AND | OR : scope the expression
EXISTS: EXISTS expression
Exists is the way scoping of nested happens. Typically without using EXISTS all expressions that are next to each other and have the same nested path will be combined into the same nested query. However, you can break up the nested query block by using EXISTS to scope particular nested queries from each other.

As stated earlier the language uses JISON a javascript BISON equivalent, and will allow us to extend the language as needed with very little effort.

UPDATE:

I believe I am close to being able to share a branch with everyone to test and provide feedback. I have the parser(s) working and at least the syntax feedback working as well as unit tests against the parser. Some screenshots:

I'm hopeful that I will have the branch ready later this week. When I do have it ready I'll link to a blog post here that will link to the branch and have a full rundown on syntax, usage etc. My plan is to gather feedback on the branch, fix issues, improve as requested and then get a pull request submitted.

I'd like to test it (here is an example of our usage of nested aggregations in K3 https://discuss.elastic.co/t/nested-aggregation-charts/41523, won't migrate without it).

@Robitx I don't think that will be an issue... we have documents that have at least two levels of nested objects... for example:

A->B->C

Where a single A document might have one or more B, and each of those contains a list of C that are on the B. Each nested document has multiple fields of varying types. I have tested this code such that I can build a histogram or multi-layer pie chart against the innermost nested data.

To be clear, our mappings are auto-generated from our pojos and can become very complex.

+1

UPDATE:

I wanted to get this out for people to start playing with rather than wait for an official blog post on my part to appear.

Fork/Branch can be found here:

https://github.com/homeaway/kibana/tree/fullNestedSupport

README:

https://github.com/homeaway/kibana/blob/fullNestedSupport/NESTED_README.md

The contents of the README are essentially the contents of the blog post that will appear at some point.

Feel free to open issues/pull requests against the fullNestedSupport branch if you so choose. I'll try and stay on top of any issues that people find.

+10000

+100500

+100

Hi ppadovani ,

Could you please advice, what should I do

This field is present in your elasticsearch mapping but not in any documents in the search results. You may still be able to visualize or search on it.

Thanks much!

Hi ppadovani,

We have a field as nested arrays.
"abc":[["3815222235847451","131712121218083052"]]
OR
"abc":[["3815222235847451","131712121218083052","131712121217783052"]]
OR
"abc":[["3815222235847451"]]

The values can be any 1-10

While for other nested fields I see warning that the field is not indexed(which I suppose I need to use mappings?) for this field and others like these, each set is treated as a separate value? Additionally the field type is showing as "string" instead of number, which is not an issue in itself, but what it means is I cannot search for any individual value of abc..?

Thanks much!

Found few minutes to start testing :) :|

Error: [illegal_argument_exception] Invalid format: "1457354016603" is malformed at "6603"
    at respond (http://elastic.dev:5601/bundles/kibana.bundle.js:76155:16)
    at checkRespForFailure (http://elastic.dev:5601/bundles/kibana.bundle.js:76118:8)
    at http://elastic.dev:5601/bundles/kibana.bundle.js:74736:8
    at processQueue (http://elastic.dev:5601/bundles/commons.bundle.js:42333:29)
    at http://elastic.dev:5601/bundles/commons.bundle.js:42349:28
    at Scope.$eval (http://elastic.dev:5601/bundles/commons.bundle.js:43577:29)
    at Scope.$digest (http://elastic.dev:5601/bundles/commons.bundle.js:43388:32)
    at Scope.$apply (http://elastic.dev:5601/bundles/commons.bundle.js:43685:25)
    at done (http://elastic.dev:5601/bundles/commons.bundle.js:38134:48)
    at completeRequest (http://elastic.dev:5601/bundles/commons.bundle.js:38332:8)

@BigDataEngineer
1) - I don't believe that is a message generated by my changes, and may be something that existed in Kibana prior.
2) - So yes.. it looks like the values are stored as a string, but these are possibly not nested.. I would have to see what the mapping on the index looks like to understand what/if issue exists. Please paste the mapping here.

@Robitx
I assume this was a date field... the epoch time has too many digits should be 10 not 13.. can you update/paste the query you issued?

@ppadovani
I've just choosen the default index pattern in settings and did go back to discover tab

We use

      "timestamp": {
        "format": "dateOptionalTime",
        "type": "date"
      }

K 4.4.1+ES 2.2 works fine, it might be K 4.3 specific (I haven't tried this version before)

@Robitx
I'm looking for the query used... or are you saying that the query passed was just the standard date window from the UI and you didn't specify a query? I'll rebase my changes against a later Kibana release and repost when the branch is updated.

@ppadovani yep just * and some time range

+1 for nested objects in visualization section

@Robitx
That operation that you ran never hit any of my parser code... so I doubt that was due to something I did... My setup is K 4.3.1 + ES 2.1.1 - I'll upgrade my ES to 2.2 and see if I get the same behavior, then I'll rebase the branch to K 4.4.1

Just upgraded to ES 2.2.1 w/ K 4.3.1 + my code... couldn't reproduce:

I will still rebase to 4.4.1 - the current release, will update this post when the branch is ready.

UPDATE:

Rebased to 4.4.1 on a new branch: https://github.com/homeaway/kibana/tree/nestedSupport-4.4.1

Tested on ES 2.2.0 and K 4.4.1

Hi ppadovani ,

Regarding my previous questions, I will forego those. I already have an elastic search instance in AWS (along with mappings) and I am trying to hook this up to that. However the kibana server status on the UI says:

plugin:elasticsearch This version of Kibana requires Elasticsearch ^2.1.0 on all nodes. I found the following incompatible nodes in your cluster: Elasticsearch v1.5.2 @ undefined (undefined)

I am still using https://github.com/homeaway/kibana/tree/fullNestedSupport and not the latest one provided by you. Is it possible for you to make it compatible for 1.5.2?
Kindly advice.

Thanks much!!

@ppadovani
I can understand if this may not be possible, since we are going backwards, however, Amazon Elasticsearch Service is not very keen on upgrading to newer versions, which is understandable. So, I have to work with whatever we have. We have invested a lot of efforts in setting up AWS instance(along with log forwarding from multiple nodes, streaming events, and other minute details) and re-inventing everything on a separate platform from scratch is not an option for us. It would be good to be able to hook this as an additional Frontend. I am not even sure if there will be another roadblock down the line?

Thanks!!

@BigDataEngineer
After looking at the Kibana code in previous versions, the first version I could try and apply the changes to is 4.1.6. However, there was a significant rewrite/refactor/reorganization of code and I cannot simply patch that branch. It would require a significant amount of work to try and get my code in and working.

I honestly don't know why the Kibana teams ups the required elastic version like they do with each release.. the REST interface just doesn't change that often. I speculate that they do it to force users to upgrade their elastic clusters.

One thought, you might try changing the version in src/plugins/elasticsearch/index.js around line 27

@ppadovani
it worked. Thanks.

+1

@ppadovani Hi, thanks for upgrading to the 4.4.1 version, sorry for not responding sooner (I've missed your update in one of the previous comments).

It's working now, but first thing I've noticed are performance problems and occasionally kibana freezes completely (I've been unable to test more complicated queries).

There are few things that might contribute to this issue, one of them is a number of fields in our daily index (there are hundreds of them http://pastebin.com/fktN0dR5).

@Robitx Do you have these same issues with the base 4.4.1 K code without my changes? Or is it just with my changes? I do know that very large index patterns cause significant performance issues for K. If it is just with my code, then I think I know what it might be. I'll take a look at it when I come up for air in a day or so.

@ppadovani the base K 4.4.1 doesn't have this problem

A year since this issue isn't fixed...

Damn, elasticsearch has pretty needed feature "Nested Objects", and Kibana from the same developers still don't have support of this feature.

You have fork that already implement this feature and still don't merged in main source code, with proper support.

And we still can not use in our project stock version of KIbana with "Nested Objects" support.

F*cking amazing!!!

@ppadovani big thanks for your work on fork=)

@Robitx Can you tell me when Kibana freezes up for you? IndexPattern definition? Or when you are starting a new query? There are possible areas where this might be, and I want to narrow it down.

I fixed one issue that might have been contributing when bringing up the discovery/visualize tabs... I've pushed a fix, please re-test.

@rashidkpc I am ready to generate a pull request based on this work. Can you tell me which branch I should rebase my work against? I currently have it against 4.3.1, 4.4.1, and 4.x. (4.x is close, but I am having issues getting the unit tests to run. The test cluster fails to start...)

Hey gang (cc @ppadovani)

As I mentioned in https://github.com/elastic/kibana/pull/5411 there are a number of limitations in Elasticsearch itself, particularly that nested aggs/filters aren’t automatic and lucene query syntax doesn’t support nested search. While the approach taken here would chart a different path to solving the problem, it isn't the direction we want to go. This is a solution to a narrow problem, but we want Kibana to solve a wide swath of challenges. In this case that means sacrificing smaller wins for bigger wins down the road.

While we are considering possibilities for a language for Kibana we haven’t decided exactly what we want the feature set to be and don’t want to do it half way, or with a single goal in mind. We’re considering some tactics and feature sets, both in Elasticsearch and in Kibana, but we’re still in the formative stages. Over time we’d like this to include search, transformation and visualization, like you see in something like timelion, and we’ll certainly keep the idea of querying nested documents in mind as we do that

I noted that this stores the nested path, however we’re removing the cached mapping https://github.com/elastic/kibana/pull/6648 and replacing it with new APIs in Elasticsearch: https://github.com/elastic/elasticsearch/issues/15728. Please weigh in on that issue, it would be great if Kibana didn’t need to parse out the nested path. This is especially important to our goal of making very large documents accessible in Kibana

For now we recommend taking @tbragin’s approach using include_in_parent or copy_to. For 90% of aggregations this approach will work perfectly.

I’m glad this solution is working for those that can not use include_in_parent or copy_to, super impressed with what @ppadovani has accomplished. I’d love to see something like this implemented as a plugin. Right now the query input essentially takes 2 kinds of queries anyway, we could probably find a way to make that pluggable.

hey, chiming in. I chatted to Rashid about this, and while I feel the pain of users wanting to use Kibana for nested mappings, supporting it in a more general manner (that might imply additional features in Elasticsearch level itself) is the path forward that maintains the flexibility we need to have in Kibana. While getting this suggested change in might solve the short term problem of not supporting nested, it will prove to be problematic down the road.

I head and feel the need here for Kibana to support nested, but this is one of those cases where if it is not obvious how it needs to be solved, it is better to leave it unsolved until we have a solution that feels natural. We definitely need to continue and explore that, one of those, which we chatted in different places, is automatically supporting nested (wrapping and so on) in ES itself.

I'm bristling with excitement to see this elegantly solved.

+1

+1

IMHO I must respectfully disagree with the stance of the Kibana maintainers. However, if we can find a way to make something along these lines plug in to Kibana I would be all for it.

In the meantime, I will continue to maintain and fix bugs on my fork(s)/branches for those that wish to continue to use the version I have created. I know that we will be using this extensively for live BI analytic dashboards in the future.

Cattle. Not pet.

+1 :)
It could be great to use Nested objects in Kibana !! (Someone has a plugin for that or not ?...)

+1

+1

+1

@tbragin The approach you mentioned doesn't work on nested types. It will aggregate all the data irrespective of type.

+1

+1

+1

+1

This feature is such a no brainer I was actually shocked to find that it's not only not supported, but that the developers apparently have no plans to ever support it. Hell, talk to your project managers and hire @ppadovani to make a plugin if you guys won't do it yourself.

+1 as the lack of nested objects hinders my project substantially

For anyone looking for discussion regarding denormalization as a way to circumvent this issue: Getting around Kibana's missing support for Nested Objects & Parent/Child

Elastic it would be great if there was a disclaimer on the site regarding this issue to avoid users from wasting there time trying to implement a unsupported feature. Why? Kibana product page says "Seamless Integration with ElasticSearch" which is not true here :)

FYI - The branch of my code referenced in the above discussion is old.. the current branch is:

https://github.com/homeaway/kibana/tree/nestedSupport-4.x

We are actively using it internally, and continue to update our internal version. I can/will update the github version should there be interest.

Pierre

+1

+1

+1

+1

+1

+1

I remember +1'ing this over a year ago, and since then the Kibana development team has done nothing but dig their heals-in and mostly ignored users and finally when their feet are put to the fire they respond with a more-or-less "NO", stating that "it doesn't feel natural".

I also see this pattern on a numerous of other requested features such as:

• Support for calling OS-side groovy scripts.
• Support for being able to use ES Scripted Metric Aggregations (especially useful for calculating weighted averages).
• And so on...

This all goes against the stance on the whole Elastic Stack 5 vision which was stated that they (Elastic) would be supporting more underlying features from Elasticsearch in Kibana. But I have seen very little to back up these claims.

As a result I see Kibana loosing ground to forks such as Siren's Kibi, which decided to take the torch on such items as this topic and come up with a solution.

I am grateful for the developers of Kibana for giving us a great visualization tool. But Elastic needs to decide going forward if Kibana is to stay a simplistic visualization tool or listen to the community and expand it's usefulness. If the decision is the former then expect users to leave when others decide to take advantage of these shortcomings.

+1

@cslinuxboy

Support for calling OS-side groovy scripts.

Most of the use cases covered by this will be solved by https://github.com/elastic/kibana/pull/7700.

Support for being able to use ES Scripted Metric Aggregations

I don't think anyone is against this (at least, don't see any opposition here https://github.com/elastic/kibana/issues/2646), in fact now is the time to add it since Elasticsearch has added the painless scripting language. It's really just a matter of someone finding the time.

+1

+1

+1

I am starting to work on my fork again. I wish to apologize to the community, I had no idea until a month or so ago that I had not turned on issues for the fork, so no one would have been able to indicate that there were bugs. That has been rectified.

Current branch: https://github.com/homeaway/kibana/tree/nestedSupport-4.5.4

The updates in the order I intend to implement them:

• Add support for date offsets to the query parser
• Add support to discovery for nested fields when looking at a result - DONE
• Add support for parent/child queries and aggregations
• Add support for geo shape types for queries (i.e. point, box etc) This is a port of a recent enhancement to our internal language.
• Add type-ahead for field names in the query field
• I might try and build a parser for the existing Elasticsearch simple query language in order to fix the annoyance of an invalid query either causing a stack trace to be returned from Elasticsearch, or no results being returned with no indication why. If I do work on this it will be after I have completed the above. If I do this I will look at adding nested and parent/child support to the language on the Kibana side.

I wish to add my voice to those that have indicated that Kibana team is not listening to the community. The Kibana team CANNOT simply rely on Elasticsearch to provide the needed functionality if there is a gap in order to keep Kibana "pure". Marketshare is not gained that way, it is lost.

+1

+1
@Bargs: Any progress on this issue? When this will be get addressed/prioritized?
Very long thread ....This is not good for product like kibana..
We appreciate your efforts

This sucks :-(

homeway has nested aggs & query suport build on Kibana 4.3.1 check it out.. hope this help..

https://github.com/homeaway/kibana/tree/nestedSupport-4.x

@ankitchheda I know about it, but a fork maintained by few people that goes against the philosophy of the main project (which is under heavy development) is not a solution, pretending it is won't help anyone..

I'd like to do something about it, but I don't have time, so for now I at least try to apply pressure and hope devs will stop ignoring this issue :|

+1

FYI - The fork I maintain for nested support, now has support for the following versions:

4.5.X
4.6
4.7
5.X

It may not follow the 'philosophy' of the main devs, but it does work and work well.

I've just run up against this. It's a bit of a drawback and it would be really useful to have a workable solution.

@tbragin, @rashidkpc - the proposed workaround misses the point - you get
something out with nested objects - but you get the wrong results! Nested
aggregations give different results (I'll do a little worked example and post it here later on).

I'll test the fork of @ppadovani.

+1

:turtle::dash:

@Bargs Hey is changes from fork harms to the performance of base functionality?

+1

+1

+1

+10086

+1

+1

+1

include_in_parent still works on ES and Kibana 5.2? I tried to use as a alternative without success.

@gustavomr I believe it will work, but only for certain use cases. It will not work for all possible query use cases that nested queries/aggregations can provide.

NOTE: My 5.1 fork of Kibana now uses a toggle next to the search icon to switch between the native elastic simple query language and the nested query language that I've included. I've also fixed a variety of issues around metrics for nested fields.
https://github.com/homeaway/kibana/tree/nestedSupport-5.1

@ppadovani thanks so much for doing this. Is it possible to separate your work enabling nested objects in kibana from your work creating a new query language? If these are can be separate branches than we can just use the former and merge it with newer kibana releases as they come out instead of having to merge both features.

Also do you already have a docker created for this fork?

@ppadovani +1 for docker container for your fork.

@gkozyryatskyy - Please open an issue on the fork, and I will look at it.

@imranq2 - I could do this, however please note that the simple elastic query language doesn't support nested queries. If you have nested data and want to query it, you will have to hand build the query as an elasticsearch json blob, and paste that into the query box.

+1

My fork now supports 5.2 on the nestedSupport-5.2 branch.

@ppadovani That's great! Let me know if you need help with making a docker container for this.

Haven't gotten the docker container done, but had some time to play with the discovery tab and how it displays nested data... looking for feedback from those that are monitoring the nested support issue. The table is recursive and the filters for nested fields seem to work well.. but the toggle field in column doesn't work yet... I still need to think through how/if that should work.

FYI - This work is basically complete and all supported forks/branches have the changes starting with the nestedSupport-4.5.4 through nestedSupport-5.2

@Bargs - I know none of the nested work I've done is something you guys are looking to pull in, but I am curious if the work around displaying nested data in the discovery results is something you might want. It isn't dependent on any of my previous work. See this issue for updated screenshots:
https://github.com/homeaway/kibana/issues/12

If this is something you guys want, let me know and I'll open an issue and include a patch.

Looks interesting @ppadovani ! If you'd like to open a PR I'd definitely be interested in checking it out. We've talked a bit about adding better support for nested fields to Discover so it would be great to have something concrete to discuss.

@ppadovani I'm creating a docker container for this right now so I can use it. Do you already have tar ball created like http://staging.elastic.co/$(VERSION_TAG)/downloads/kibana/kibana-${ELASTIC_VERSION}-linux-x86_64.tar.gz? If so I can just pull that into my docker container. Otherwise I'll need to build your branch and create the tarball.

@imranq2 I do not provide distributions of my fork, so you will need to build it yourself.

FYI - I've created a pull request for supporting nested data in the discovery tab for those that are interested:
https://github.com/elastic/kibana/pull/10814

+1

+1 for me
+10 for my colleagues

+1

+1

+1

+1

+1

+1

+1. Is this supported in ELK5?

+1

this is really a requirement as not supporting nested object is really an impediment to the broad adoption of kibana in my company as you have to create an index for kibana and another one with nested documents for more clever searches through the plain API

+1
In my mapping it would prevent possible "explosion" of index properties.

+1

+1

+1

Please, stop posting +1 comments which are only spamming all people subscribed to this thread. Instead, click on the big yellow "thumbs up" button at the top of this issue to upvote it.

+1

Developers of Kibana, please do something, guys really. Usage of Kibana without nested objects in the many cases are useless. You preparing version 6.x without nested objects...

Our system scans text for houses and then save results into ES. So ES contains main documents with array of houses. House is the nested objects.

I can't use visualisation to create dashboard with analyze of houses that we found in text.
Do something please. The houses is on fire.

Desperately need to use nested objects in kibana. Feels bad that this is not available in-built.

███████╗████████╗██╗██╗     ██╗         ██╗    ██╗ █████╗ ██╗████████╗
██╔════╝╚══██╔══╝██║██║     ██║         ██║    ██║██╔══██╗██║╚══██╔══╝
███████╗   ██║   ██║██║     ██║         ██║ █╗ ██║███████║██║   ██║   
╚════██║   ██║   ██║██║     ██║         ██║███╗██║██╔══██║██║   ██║   
███████║   ██║   ██║███████╗███████╗    ╚███╔███╔╝██║  ██║██║   ██║   
╚══════╝   ╚═╝   ╚═╝╚══════╝╚══════╝     ╚══╝╚══╝ ╚═╝  ╚═╝╚═╝   ╚═╝   

+1

+1

+1

+1

+1

+1

+1

+1

+1

+1

another one here!

For those that may have used my fork of Kibana with nested support, I am discontinuing support for that fork past Kibana release 5.4.x. Instead I will be moving most if not all of the functionality into a Kibana plugin. I hope to have the plugin ready for the latest 5.x release by the end of the year. You can follow the progress here: https://github.com/ppadovani/KibanaNestedSupportPlugin

I've just started the work, so don't expect anything significant to show up for a few weeks.

+1

+1

+1

+1

+1

I've got screenshots and status updates posted for the work I'm doing on the nested support plugin. Watch the issue to get status updates.

Screenshots and updates

I've gone ahead and released a pre-release of 1.0.0 with support for Kibana 5.6.5.

See this issue for details on what is in the initial pre-release: Screenshots and updates

V1.0.0-beta1

My plugin is feature complete for version 5.6.5 of Kibana. I have a few cleanup tasks, then I will cut the 5.6.5 version and begin forward porting to 6.1.X.

Features:

• Nested query support
• Nested aggregation support
• Discover results nested support
• Discover summary field display priority (this is actually something new)

See the README for details.

My plugin is now released! Support for 5.5.3, 5.6.5, and 5.6.6. I'll be porting to 6.0.X this weekend.

I will likely not be updating status here on this issue any longer. Please visit my GitHub page to view releases, issues etc.

Thanks!

@ppadovani can you release a support version for 5.4.0 thanks .

@ppadovani I'll be watching for the 6.0.x port!

@SolomonShorser-OICR
I've released a 6.0.1 Beta 1 build.

The only known limitation is that the filter bar is non-operational due to being hard coded to only work with the lucene query language. I'm working out a way around that, but might not have a solution in place until next weekend.

I have now release a plugin build for 6.0.1, 6.1.x is next.

@ppadovani
Thanks for your work and 6.0.1 release!

based on kibana 6.0.1, after installed this plugin, some kibana features doesnt work well.

when click timelion, it shows an error message:

if x-pack installed, some x-pack feature "discover" has another error message in "Foreach"

Hey Kibana team, why you not hired @ppadovani yet?

@sccds - Please move this bug report to my GitHub repo:

https://github.com/ppadovani/KibanaNestedSupportPlugin/issues/27

If someone has issues with my plugin, please open issues on my repo and don't start a conversation here. This issue has far too many subscribers.

@Hronom - I appreciate the thought, but my strengths are not in Javascript.... although building this plugin and hacking Kibana has certainly helped my abilities in that area.

FYI - My plugin is now up to date with Kibana releases. I've released 6.1.2 support.

Thank you @ppadovani, keep it coming!

+1

+1

+1 Hello, Tomitribe is very interested in this feature. Do know when this feature will be implemented?

@ppadovani where can I ask about functionality? I am struggling with nested aggregation in data table.

@bumerankkk Go here and open an issue: https://github.com/ppadovani/KibanaNestedSupportPlugin

Alternatively, if you go to one of the documentation pages that aligns with your question, you can add a comment to the bottom of the page.

https://ppadovani.github.io/knql_plugin/overview/

Is this being actively worked on? Do we have timelines on when such a feature would experience a production release?

Happy Birthday 🎂 'Nested type aggregations', 🎁

Now you're already four years old. Not so long ago you were so little and cute.
Hope you will grow up and we'll have some common good years in the future.

best Malu

+1

+1

+1

+1

+1

+1

+1

Why isnt this implemented yet

cuse reasons, with this feature whey wouldn't have dis thread, no fun and nobody will go to github of the kibana, this some kind of beacon

The latest elk version supports nested data correctly for me

El El mié, 6 jun 2018 a las 22:08, Eugene notifications@github.com
escribió:

cuse reasons, with this feature whey wouldn't have dis thread, no fun and
nobody will go to github of the kibana, this some kind of beacon

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/elastic/kibana/issues/1084#issuecomment-395214259,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMK55lokgriJHhPF5EuBN6yREr5-dT4-ks5t6ETAgaJpZM4Bru7J
.

@javixeneize I'm sitting here on 6.2.4 version, can't find dat nested objects support, correct me if I wrong

I have that version and I can access to a.b, where my structure is {Id:xx,
a: {b:xx}}

El El mié, 6 jun 2018 a las 22:18, Eugene notifications@github.com
escribió:

@javixeneize https://github.com/javixeneize I'm sitting here on 6.2.4
version, can't find dat nested objects support, correct me if I wrong.

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/elastic/kibana/issues/1084#issuecomment-395216828,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMK55tPrh5Qi8m7PyHbQatkRAw8qj4RGks5t6EcKgaJpZM4Bru7J
.

@javixeneize do you have next mapping with type: nested?
You can get mapping by GET /index-name

{
  "document": {
    "properties": {
      "locations": {
        "type": "nested",
        "properties": {
          "name": {
            "type": "keyword"
          },
          "popularity": {
            "type": "long"
          }
        }
      }
    }
  }
}

I will check tomorrow but I have the default config

El El mié, 6 jun 2018 a las 22:24, Eugene notifications@github.com
escribió:

@javixeneize https://github.com/javixeneize do you have next mapping
with type: nested?
You can get mapping by GET /index-name

{
"document": {
"properties": {
"locations": {
"type": "nested",
"properties": {
"begin": {
"type": "long"
},
"end": {
"type": "long"
},
"normalized": {
"type": "keyword"
},
"original": {
"type": "keyword"
}
}
}
}
}
}
}

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/elastic/kibana/issues/1084#issuecomment-395218644,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMK55npNY8uTVUPgVIbjXfAXSB5tPwDwks5t6EikgaJpZM4Bru7J
.

@javixeneize thanks in advance!
You probably have sub json object that was flattened to main document, but this not a nested object.

Yeah be careful @javixeneize your data wont correlate as expected unless you specifically set the mapping in ES for that field to nested

https://www.elastic.co/guide/en/elasticsearch/reference/current/nested.html

Main issue each field in the nested object just becomes an array and you lose correlation between the properties.

I just flattened my data before dumping into es for now.

If you cannot wait for Elastic to implement this, you can use my Kibana plugin:

Overview:
https://ppadovani.github.io/knql_plugin/overview/
Installation:
https://ppadovani.github.io/knql_plugin/installation/
Query language (SQL like):
https://ppadovani.github.io/knql_plugin/knql/

Support for 5.5.3 through 6.2.4, if a version is missing, after 5.5 please open an issue:
https://github.com/ppadovani/KibanaNestedSupportPlugin

Contributions, feature requests and bug reports welcome.

Ok, I have to wait then...

El El jue, 7 jun 2018 a las 0:38, Pierre Padovani notifications@github.com
escribió:

If you cannot wait for Elastic to implement this, you can use my Kibana
plugin:

Overview:
https://ppadovani.github.io/knql_plugin/overview/
Installation:
https://ppadovani.github.io/knql_plugin/installation/
Query language (SQL like):
https://ppadovani.github.io/knql_plugin/knql/

Support for 5.5.3 through 6.2.4, if a version is missing, after 5.5 please
open an issue:
https://github.com/ppadovani/KibanaNestedSupportPlugin

Contributions, feature requests and bug reports welcome.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/elastic/kibana/issues/1084#issuecomment-395246977,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMK55q36DWICC4QoNald5fO7rCBnLdTgks5t6GgJgaJpZM4Bru7J
.

+1

+1

+1

+1

+1

+1

very usefull feature... +1

Kibana is absolutely useless for my use cases without this support. I shouldn't have to flatten or restructure my data. I should just be able to put data in and get visualizations or aggregations out. Four years later and still no nested object support?

Any estimation when this feature is available?

Could someone from elastic please let us know whether this feature gets added to Kibana? This ticket is open for almost 5 years now.

Also, what is the point of releasing such a feature if it cannot be used in Kibana?

Don't get me wrong but it feels a bit weird.

+1

+1

@dayotoro @berkoaviv @yechanpark @mackermann2 @mahnejo @bphenriques

Please don't add comments like +1 comments. Especially for issues you truly care about! Let me explain, anyone with notifications turned on will easily get annoyed and unsubscribe from the thread. That means the contributors won't see this as an important issue purely based on how many participants involved. When people unsubscribe, this number goes down.

Instead, what YOU should do is subscribe to the thread via the box in the top right corner and give the initial Github issue a thumbs up to show your support.

@bugs181 this not works here, this issue it's anomaly and absorbs all incoming waves, for five years now!

I think scientists should explore this black hole!

The devs of Kibana seems have abnormal power!!!

@bugs181 this not works here, this issue it's anomaly and absorbs all incoming waves, for five years now!

Have you ever thought they didn't want to read through the entire issue with tons of "+1 me too" comments fishing for the information that could be valuable to them?

@ppadovani has an open source plugin which could be used (look at his comment above)

@mika76
im using this.

very useful in simple case

@mika76 just be aware, that due to time constraints it's not currently actively maintained.

ppadovani commented 28 days ago
Hey folks... I have found my time severely limited due to work and life. Additionally even though I was able to get things partially up, the changes that the Kibana team made in moving to React will require mostly rewriting large chunks of what I've done.

@mika76 yes this plugin is the only way to get nested objects works right now.
But what is strange to me, that Elasticsearch has official support for nested objects, but Kibana don't.

As mentioned @bugs181 it's not currently actively maintained and this means that you can't upgrade to the latest version of ELK stack.

So official support means proper maintaining too. But because dev's is spitting on this issue we don't have official support for this.

I must second @bugs181 here. '+1'-ing this issue and otherwise spamming it, doesn't help much raising awareness for this, but will only make people mute it or bring us closer the point where I'll need to lock it down, which I really don't want, because I would like to keep every issue open for discussion for everyone, since I believe in open communication.

So far I also always recommended using that plugin. I wasn't aware that this is no longer actively maintained and am sad to hear that. We also know that this issue has been open for 5 years, and we're running analysis on issues (in Kibana obviously - see screenshot attached) and know that this is the most reacted issues:

But as you are pretty much aware there are (literally) thousands of other issues open, so we always need to balance between features, bug fixing, etc. to find a good prioritization. Also (but not only) because that feature always had a pretty solid working community plugin, it hasn't had gotten enough priority (to beat out other issues) so far. Also as so often there are often technical relation between different things, and for example for nested field supports we're currently first want to finish our overhaul of the whole visualization rendering pipeline (#19813), before we start this, since it's highly linked together. Nevertheless we're currently having this issue on our roadmap for 7.x and hopefully won't be blocked by other technical changes, so we can soon move that feature into the Kibana core to make it available without a community plugin.

Does the request to support nested object visualizations include support for parent-child relationships? I have a customer inquiring about parent-child visualizations.

@MorrieAtElastic no, parent-child would be a separate issue.

https://github.com/elastic/kibana/issues/3730
https://github.com/elastic/kibana/issues/20255

It is really big issue as we are unable to show 1:M relations properly in Kibana as ElasticSearch support Nested Object so we can load data but can't view properly. This must be fixed soon.

Thanks,
Rakesh

Started working on nested field support in KQL today. Created a separate issue to track since full nested field support in Kibana involves many changes beyond just KQL.

https://github.com/elastic/kibana/issues/44554

OMG is this real? After 5 years...

What about the new datatype: flattened? Will there be support for visualizations for that new type in the future? Many service clients are being steered to this new type and they are asking if/when visualizations might come into play.

@Barrybigbuddy I'm not sure what the plans are for flattened, could you create a separate ticket for it?

I would like to have the parent and child relationship represented in Kibana. So request you to have this feature prioritized. Thanks. M'Jay

+1

Awesome! +1 here! Is there a specific timeframe for this feature?

Let me be the party pooper here and give a short reminder: There are a lot of people subscribed to this issue (228 community people + a couple of Elastic teams), so it would be nice if we keep +1 to a minimum. Thanks to GitHubs nice reaction feature you can also add your approval and love for @Bargs to his comments without triggering a notification to all subscribers (or your disapproval, then he can stop working on nested field support again :wink:). Also especially since this is such a large ticket, please don't use it for unrelated questions about other field types.

I'll leave a couple of references here for other field types:

• Support for flattened field type - currently not on our near-term roadmap
• Parent/child support - currently not on our near-term roadmap

When will this feature arrive? We're actively working on this and there will be different phases to this (KQL, filtering support, visualizations, etc.) which could arrive in different versions, and it will depend on how development process will go on that. We'll post pull requests related to this feature as a comment here, so you can see in the PR which version that specific phase/part of support will land in.

I am not sure if this is the same issue, but I have an index with list of objects, but not using the "nested" datatype. However in Kibana under "Available fields", I do not see fields inside the objects which are in the list. Is this a known limitation?

@ppadovani is the plugin is available for kibana-7.3.1?

Hi everyone, I think I found some kind of solution for this problem. Reading the source code of the mappings I found that there is an include_in_parent option for the nested mapping type. When using this option, I am able to visualize an array of objects in Kibana without any problems!!! For some reason this option DOES NOT appear in the ES documentation. Maybe I am getting something wrong but apparently everything works. I use the fields option so I can search both as keyword and as full-text.

Mappings

PUT /test_index
```json
{
"mappings": {
"dynamic": "strict",
"properties": {
"state": {
"type": "keyword"
},
"created_by": {
"type": "nested",
"include_in_parent": true,
"properties": {
"first_name": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"last_name": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
}
},
"dynamic": "strict"
},
"people": {
"type": "nested",
"include_in_parent": true,
"properties": {
"first_name": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"last_name": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
}
},
"dynamic": "strict"
}
}
}
}

### Documents
POST test_index/_doc
```json
{
  "state": "done",
  "created_by": {
    "first_name": "Patricio",
    "last_name": "de Villa"
  },
  "people": [
    {
      "first_name": "Patricio",
      "last_name": "de Villa"
    },
    {
      "first_name": "Test",
      "last_name": "Test"
    }
  ]
}

@patodevilla

Using include_in_parent or copy_to as a workaround is not supported and may stop functioning in future releases.

https://www.elastic.co/guide/en/kibana/7.x/nested-objects.html

Nested Field support Phase 1 released in 7.6.0

A small update to this issue: We just released 7.6.0 of Kibana, which has the initial support for nested fields in it. You now are able to use nested fields in the following places in Kibana:

• Index Patterns will detect Nested Fields correctly
• You will be able to watch Nested Fields in Discover
• Filtering on Nested Fields via the filter bar works
• KQL allows searching for nested fields (see the KQL documentation for an explanation of the syntax on querying nested fields)

We're currently working on enabling nested fields in visualizations and we'll continue updating this issue with relevant information.

Hi! We are waiting for the NESTED functional. When can we see it? This is the only moment that does not completely switch to Kibana (Elastic is the best). The whole world is watching you.

Nested field querying is still bugged with KQL.
Example:
Consider index mapping defined as
"first": { "type": "nested", "properties": { "second": { "type": "nested", "properties": { "field": { "type": "text" } } } } }
I have created Index pattern based on this index and want to query first.second.field : "test"
This query in the inspect tab will generate
"filter": [ { "bool": { "should": [ { "match": { "first.second.field": "test" } } ], "minimum_should_match": 1 } } ],
which is incorrect.
Correct version should also include nested syntax "nested": {"path": "first.second",...}

Pinging @elastic/kibana-app-arch (Team:AppArch)

@IlyaHalsky Please check the KQL documentation on nested fields. Nested fields require a specific syntax to query, since you have multiple ways to query on them (in your case you most likely just want to do: first.second:{ field: "test" }).

You should also see a toast notification when you try using a nested field for the first time in an KQL query that will link you to that explanation.

I ask if there is a new kibana release that support nested field in visualize:
my data example:
{
"fieldX": "x",
"fiedY": "Y",
"anomalies": [
{
"category": "system",
"name": "cpu",
"date": "2020-03-11T13:33:40.000Z"
},
{
"category": "reboot",
"name": "reset",
"date": "2020-03-11T13:33:40.000Z"
},
{
"category": "system",
"name": "memory",
"date": "2020-03-11T13:33:40.000Z"
}
]
}

I want to visualize a Pie chart in kibana where:
slice size count = Total count of anomalies array objects (documents count x objects count by anomalies array)
first bucket = anomalies.category
second bucket = anomalies.name

in other words i want to visualize the distribution of anomaly name by anomaly category ?

+1

Any news on this? The 7.6 release is already a couple of months old, 7.7 and 7.8 did not have any mentions of this in the Release notes and the docs for 7.9, 7.x and master also contain no new infos about this functionality.

Just chiming in to also express our high hopes of getting nested aggregations support in Visualizations. Would be amazing!