Lorawan-stack: Duplicate correlation IDs

@htdvisser the problem seems that the correlation IDs from the event payload are appended to those of the context, i.e. if you have this;

ctx := events.ContextWithCorrelationID(ctx, fmt.Sprintf("gs:uplink:%s", events.NewCorrelationID()))
msg.CorrelationIDs = append(msg.CorrelationIDs, events.CorrelationIDsFromContext(ctx)...)
registerReceiveUplink(ctx, conn.Gateway(), msg)

The correlation IDs are duplicate in the event as they are both in ctx and in msg (through GetCorrelationIDs()).

Should we check uniqueness here?

https://github.com/TheThingsNetwork/lorawan-stack/blob/master/pkg/events/events.go#L148

Yes, we can move the merge algorithm in pkg/events/correlation_context.go into its own func (mergeCorrelationIDs) and use that in pkg/events/events.go:

    if cids := data.GetCorrelationIDs(); len(cids) > 0 {
        cids = append(cids[:0:0], cids...)
        sort.Strings(cids)
        evt.innerEvent.CorrelationIDs = mergeCorrelationIDs(evt.innerEvent.CorrelationIDs, cids)
    }

We should probably add a note that evt.innerEvent.CorrelationIDs must be sorted (which it is, because correlation IDs in the context are sorted).

What is the status @pgalic96 ?

got busy with stuff, I made the check for uniqueness, something like this:

if data, ok := data.(interface{ GetCorrelationIDs() []string }); ok {
        mapCorrelationIds := make(map[string]struct{})
        for _, correlationID := range evt.innerEvent.CorrelationIDs {
            mapCorrelationIds[correlationID] = struct{}{}
        }
        for _, correlationID := range data.GetCorrelationIDs() {
            if _, ok := mapCorrelationIds[correlationID]; !ok {
                evt.innerEvent.CorrelationIDs = append(evt.innerEvent.CorrelationIDs, correlationID)
            }
        }
    }

haven't written a test yet, but it would be creating an event with TestCorrelationID and then supplying the same TestCorrelationID in the payload, and then checking that the length of the CorrelationIDs of the message is indeed 1 and not 2.

Apart from the variable name it generally looks good, although the evt.innerEvent.CorrelationIDs don't get deduplicated this way

I would prefer using @rvolosatovs's merge algorithm (see my comment). We already spent the time on making an efficient implementation for merging correlation IDs, so it would be a shame if we'd merge/dedup with @pgalic96's proposed map implementation that is a lot less efficient and also returns unsorted IDs.

@pgalic96 can you give a status update?

I haven't been working on it this week yet, coming to office tomorrow after exam. will try to finalize it tomorrow.

I will change the code according to @htdvisser suggestion.