Lorawan-stack: Support multiple resulting messages from one format

@adriansmares I am currently working on this issue. After changing from []byte to [][]byte for the *ttnpb.ApplicationUp function in the formatter.go file, I also changed it in the json.go and protobuf.go files. But now, it gives the following error:

Kindly provide some suggestion. Thank you.

@mihirr93 Since you are now modifying the call sites to return a slice of results (each result being a slice of bytes), you should wrap places where only one result is returned with [][]byte{ slice-containing-result }.

Hello Mr. @adriansmares . I am bit confused regarding our approach. We are tying to slice the front-end message (raw payload) into multiple messages. But in this case, the point at which the message will be sliced depends on its encoding (Cayenne or any custom encoding) which is unknown to us. So, to do this we need to use the decoded payload. Once the user enters the decoder in the payload format tab of the TTN, the application actually shows the decoded values in JSON (as seen in the picture). Shouldn't we direclty split this decoded payload and push it further using http?

Could you please throw some light on this confusion?
And, do we just modify the pkg files and the .proto files will be automatically updated?
Thank you so much for your kind support and time.

_Keep in mind that this repository contains the V3 version of our LoRaWAN stack, and the picture provided is from TTN, which runs the V2 version._

Please keep in mind that this issue deals only with the support for producing multiple payloads from one message, and the new format issue that you are probably interested in is https://github.com/TheThingsNetwork/lorawan-stack/issues/1158. Please check if this issue is actually the one you are looking for.

Nonetheless, I'd suggest in this case that you introduce a new format, based on the JSON one, which produces multiple payloads based on the decoded information found inside the uplink.
You'll probably want to handle the ApplicationUp_UplinkMessage uplink message specifically, since it contains the fields you are interested in as part of the DecodedPayload field.

You shouldn't have to modify the proto files for this use case.

I am still trying to understand the internal working and code of the stack. Currently, I am focusing on this issue only, and not #1158. As you suggested to modify the ApplicationUp_Uplinkmessage, I am trying to figure that out. I have following queries at this moment:
1) The struct ApplicationUplink and the func ApplicationUp_UplinkMessage is defined in the messages.pb.go file in the ttnpb directory. However the header of the file mentions "DO NOT EDIT". Should I ignore it? or is there any alternate approach to modify it?
2) Can you please describe this in more detail "introduce a new format, based on the JSON one"? I thought that we just have to modify the existing format instead of creating a new one.
3) You also mentioned the wrapping of results where only one result is returned. What exactly will that do? Will it re-combine the multiple message into one and return just one result? And do we need to do that for Uplink message only?

[Thank you @adriansmares for your continuous guidance. Although, I took the golang tutorial, this is my first interaction with the GO language, pardon for bothering you with so many doubts]

• You do not need to edit that file, nor any *.pb.* files. These are generated from the *.proto files automatically when you run mage proto:all. Nonetheless, for the purpose of this issue no such modifications have to be done.
• Please follow that up in the other issue.
• The suggestion for wrapping allows you to keep the existing code intact (still returning one payload to be posted from one uplink message).

Let me try to explain the whole pipeline better, maybe this will shed some light on how this works

1. A device sends 1 uplink.
2. The 1 uplink is received by the gateway and sent to the Gateway Server which forwards it to the Network Server, which then forwards it to the Application Server. The message that reaches the AS is of type *ttnpb.ApplicationUplink.
3. The AS takes this 1 uplink and attempts to decode its binary payload into the decoded fields (turn the bytes in the FRMPayload field into the decoded structure of DecodedPayload) using the device payload formatter.
4. It then wraps it up in a *ttnpb.ApplicationUp and sends it to the frontends (MQTT, webhooks, PubSub, application packages).
5. The frontends receive this 1 *ttnpb.ApplicationUp and based on their settings (the format) choose which formatter to use (the formatter interface is defined in pkg/applicationserver/io/formatters).
6. The frontend calls formatter.FromUp with the 1 *ttnph.ApplicationUp and receives a slice of byte ([]byte), representing the body of 1 message.

Now, concerning this issue, I hope it is visible that only step 6 needs changes:

• Changing the return types of both the interface and the implementations such that FromUp can return multiple payloads (a slice of slices of byte).
• The existing Protobuf and JSON formats should not change their behavior, and as such should continue to return only 1 payload (that is why the wrap is required).
• After changing these 2 (the interface and the formatters) to return [][]byte you will start seeing errors (since most of the places that call FromUp expect only one payload to be received - you have to update these call sites and make them loop through the list of payloads and send each of them.

Only after this issue, which does not change the behavior at all, has been fixed, you can go forward with implementing your own formatter, which would return a slice of multiple payloads, one for each measurement (but keep in mind that this new format is out of scope for this issue !).

Thank you so much @adriansmares for such a detailed explaination and guide. I think I have managed to solve this issue now. I also tested it with ./mage go:test js:test jsSDK:test and it gives no error. The major changes made in the code are highlighted below:
formatter.go

type Formatter interface {
    FromUp(*ttnpb.ApplicationUp) ([][]byte, error)
    ToDownlinks([]byte) (*ttnpb.ApplicationDownlinks, error)
    ToDownlinkQueueRequest([]byte) (*ttnpb.DownlinkQueueRequest, error)
}

json.go

func (json) FromUp(msg *ttnpb.ApplicationUp) ([][]byte, error) {
    m, e := jsonpb.TTN().Marshal(msg)
    return [][]byte{m}, e
}

protobuf.go
Same as above

mqtt.go

buf, err := c.format.FromUp(up.ApplicationUp)
                if err != nil {
                    logger.WithError(err).Warn("Failed to marshal upstream message")
                    continue
                }
                logger.Debug("Publish upstream message")
                for _, v := range buf {
                    c.session.Publish(&packet.PublishPacket{
                        TopicName:  topic.Join(topicParts),
                        TopicParts: topicParts,
                        QoS:        qosUpstream,
                        Message:    v,
                    })
                }

pubsub.go
Same as above

webhooks.go

unc (w *webhooks) newRequest(ctx context.Context, msg *ttnpb.ApplicationUp, hook *ttnpb.ApplicationWebhook) ([]*http.Request, error) {
    var cfg *ttnpb.ApplicationWebhook_Message
    switch msg.Up.(type) {
    case *ttnpb.ApplicationUp_UplinkMessage:
        cfg = hook.UplinkMessage
    case *ttnpb.ApplicationUp_JoinAccept:
        cfg = hook.JoinAccept
    case *ttnpb.ApplicationUp_DownlinkAck:
        cfg = hook.DownlinkAck
    case *ttnpb.ApplicationUp_DownlinkNack:
        cfg = hook.DownlinkNack
    case *ttnpb.ApplicationUp_DownlinkSent:
        cfg = hook.DownlinkSent
    case *ttnpb.ApplicationUp_DownlinkFailed:
        cfg = hook.DownlinkFailed
    case *ttnpb.ApplicationUp_DownlinkQueued:
        cfg = hook.DownlinkQueued
    case *ttnpb.ApplicationUp_LocationSolved:
        cfg = hook.LocationSolved
    }
    if cfg == nil {
        return nil, nil
    }
    url, err := url.Parse(hook.BaseURL)
    if err != nil {
        return nil, err
    }
    url.Path = path.Join(url.Path, cfg.Path)
    expandVariables(url, msg)
    if err != nil {
        return nil, err
    }
    format, ok := formats[hook.Format]
    if !ok {
        return nil, errFormatNotFound.WithAttributes("format", hook.Format)
    }
    buf, err := format.FromUp(msg)
    if err != nil {
        return nil, err
    }
    var requests []*http.Request
    for i, v := range buf {
        req, err := http.NewRequest(http.MethodPost, url.String(), bytes.NewReader(v))
        requests[i] = req
        if err != nil {
            return nil, err
        }
        for key, value := range hook.Headers {
            req.Header.Set(key, value)
        }
        if hook.DownlinkAPIKey != "" {
            req.Header.Set(downlinkKeyHeader, hook.DownlinkAPIKey)
            req.Header.Set(downlinkPushHeader, w.createDownlinkURL(ctx, hook.ApplicationWebhookIdentifiers, msg.EndDeviceIdentifiers, "push"))
            req.Header.Set(downlinkReplaceHeader, w.createDownlinkURL(ctx, hook.ApplicationWebhookIdentifiers, msg.EndDeviceIdentifiers, "replace"))
        }
        req.Header.Set("Content-Type", format.ContentType)
        req.Header.Set("User-Agent", userAgent)
    }
    return requests, nil
}

Kindly provide your feedback and comments. I will make the corrections (Var naming, indentation,etc) as per the code style guidelines once the task is completely finished. And if this looks correct, could you please guide me further for issue #1158 . Thank you once again.

@mihirr93 looks good ! Please commit your changes and submit a pull request for this issue. After merging it we can move forward to the new formatter mentioned in #1158.

@adriansmares Sure, I will do that. Should that pull request be based at the mentioned environment or at the latest version of the stack?

I'd recommend that you rebase your changes on top of the latest master in order to avoid conflicts later.

This issue has been inactive for quite a while, so let's move it back to triage to see if there's still demand for this, or that we should just drop it.

This is no longer needed, as it scales poorly.