This appears to only happen when the pad is included through a plugin with requireSession set to true. Otherwise import works as expected.

@JohnMcLear, can you have a look?

@Rillke, some context for you:

The message is part of a security tightening introduced with Etherpad 1.8.3. The main commit introducing it is 24ee37a38ff8. That change, though, should only apply to imports done via the Web UI. I _think_ you are using the API? If so, it is a bug.

Could you please give John a quick way to replicate the problem (i.e.: the relevant settings & preconditions, a command to trigger the problem)?

I cannot look at it in the next days because I am busy with finishing the 1.8.4 release.

Thanks.

Will do. Thanks! I probably don't need a way to replicate but if you have I will take it. My ETA for fix is 48 hours. That said

This appears to only happen when the pad is included through a plugin with requireSession set to true. Otherwise import works as expected.

I think I just have to set requireSession to true to make this situation become realized :) It's the language of "included" that confuses me!

I guess the question is does from authorManager

let author = await db.get("globalAuthor:" + authorID);

it could be that globalAuthor doesn't set for group pads..

Oh wait, there is a PR waiting for this use case .....

https://github.com/ether/etherpad-lite/pull/3965

note that if this is a requireSession is true the user must be authed no?

Anyway my steps will be.

1. Create script to simulate a session.
2. Use that session to visit a pad
3. Try import

@Rillke does https://github.com/ether/etherpad-lite/pull/3965 solve the problem for you?

If so, would it make sense for https://github.com/ether/etherpad-lite/pull/3965 to always be active if requireSession is true?

Could you please give John a quick way to replicate the problem (i.e.: the relevant settings &
preconditions, a command to trigger the problem)?

Steps to reproduce:

1. Create a new Etherpad using this plugin
2. Visit the Etherpad (it's embedded in an iFrame)
   The following requests by the managing system are logged by our proxy

"GET /api/1/createGroupIfNotExistsFor?groupMapper=185390&apikey=XXXX HTTP/1.1" 200 65 "-" "-"
"GET /api/1/createAuthorIfNotExistsFor?authorMapper=185388&name=some+user&apikey=XXXX HTTP/1.1" 200 66 "-" "-"
"GET /api/1/listPads?groupID=g.ggggggggg&apikey=XXXX HTTP/1.1" 200 87 "-" "-"
"GET /api/1/listSessionsOfGroup?groupID=g.ggggggggg&apikey=XXXX HTTP/1.1" 200 538 "-" "-"
"GET /api/1/deleteSession?sessionID=s.eeeeeeeeeeeee&apikey=XXXX HTTP/1.1" 200 37 "-" "-"
"GET /api/1/createSession?groupID=g.ggggggggg&authorID=a.AAAAAAA&validUntil=1589580000&apikey=XXXX HTTP/1.1" 200 83 "-" "-"

afterwards, the client gets a Cookie by the managing system and connects to the Etherpad.

1. Attempt to import through UI
   
2. A request is sent to the server that immediately fails (405) and the error is logged but the UI keeps saying "importing"
3. Wait.
4. After some time, the UI says the import failed.

settings.json

{
    "title": "EPad (MLU)",
    "favicon": "favicon.ico",
    "skinName": "no-skin",
    "showSettingsInAdminPage": true,
    "ip": "0.0.0.0",
    "port": "9001",
    "soffice": "/usr/bin/soffice",
    "dbType": "mysql",
    "dbSettings": {
        "user": "ep",
        "host": "mariadb",
        "port": "3306",
        "password": "XXXXXXXXXXXXXXX",
        "database": "XXXXXXXXXXXXXXX",
        "charset": "utf8mb4"
    },
    "loglevel": "WARN",
    "logconfig": {
        "appenders": [
            {
                "type": "file",
                "filename": "/var/log/etherpad/pad.log",
                "maxLogSize": 20480,
                "backups": 10
            }
        ]
    },
    "defaultPadText": "XXXXXXXXXXXXXXXXXXXX",
    "users": {
        "uuuuuuu": {
            "password": "XXXXXXXXXXXXXX",
            "is_admin": true
        }
    },
    "requireSession": true
}

Great thanks this will get confirmed today and either fixed today or tomorrow.

that plugin needs work 🖌️

1. https://github.com/ether/etherpad-lite/pull/4012 script made to generate session.

Confirmed

[WARN] console - Unable to import file into "g.8CeZzlrugSkgVtAG$undefined". Author "a.wO8h57aAvrfm9X4M" exists but he never contributed to this pad

Will need to disect the author logic to find out why this author doesn't exist in the group of authors that have existed.

I did notice something weird about the script btw, it creates padIds likethis http://192.168.1.48:9001/p/g.8CeZzlrugSkgVtAG$undefined

...

New ETA for fix is 24 hours. I will try land it today but I got stuff to do but this is on my radar :)

@Rillke does #3965 solve the problem for you?

Yes, it does.

that plugin needs work :paintbrush:

Would an issue at the project's issue tracker make sense, saying it needs to update ... ?

I would say that it needs a rewrite so it's not forcing a rewrite the core Etherpad files.... It must have been a nightmare to maintain as we updated!

That's a separate issue though, for now I will focus on the bug.

• [ ] Make group PadID show up in authorManager.listPadsOfAuthor(author)
• [ ] Make the msg return to the client properly

Well the value is being written to the DB but isn't available in listPadsOfAuthor! :D Will continue investigating..

{
  "key": "globalAuthor:a.aIY0kmB4Zgp1kpI9",
  "val": {
    "colorId": 13,
    "timestamp": 1589552855152,
    "padIDs": {
      "g.svUaskIyPncnCVDD$undefined": 1
    }
  }
}

WTF the authorIDs don't match!

[2020-05-15 14:34:46.694] [WARN] console - Unable to import file into "g.svUaskIyPncnCVDD$undefined". Author a.Vb0hC3C2ve5aEK7v exists but he never contributed to this pad <-- requesting import.
[2020-05-15 14:34:58.688] [WARN] console - adding g.svUaskIyPncnCVDD$undefined a.aIY0kmB4Zgp1kpI9 <-- adding content to pad.

No wonder!

userId from clientVars is a.aIY0kmB4Zgp1kpI9

So where is a.Vb0hC3C2ve5aEK7v coming from?

We run the script agian

jose@server:~/develop$ node bin/createUserSession.js
groupID g.iQEz4Zv7Jcv6Z012
authorID a.83QdnDCKFMALhwXN
validUntil 1589613414
Test Pad ID ====>  g.iQEz4Zv7Jcv6Z012$undefined
Session made: ====> create a cookie named sessionID and set it's value to  s.df5791c8f2e75078eead74ef0a6d7a9a
jose@server:~/develop$

This time we delete all cookies and only set sessionID..
We visit /p/g.iQEz4Zv7Jcv6Z012$undefined
We type into the pad
Our authorID should be a.izLoj1kz285kznqZ we check clientVars and discover it is correct "a.izLoj1kz285kznqZ" .
We type some characters: "hello world" is fine.
We check the data base to see if the values are right.

{"key":"pad:g.tBJKtRMG5ePWgtxO$undefined","val":{"atext":{"text":"hello world...":{"0":["author","a.izLoj1kz285kznqZ"]},"nextNum":1},"head":3,"chatHead":-1,"publicStatus":false,"passwordHash":null,"savedRevisions":[]}}

{"key":"globalAuthor:a.izLoj1kz285kznqZ","val":{"colorId":4,"timestamp":1589553559400,"padIDs":{"g.tBJKtRMG5ePWgtxO$undefined":1}}}

We note that author is correct in both pad and globalAuthor items.

For some reason this guy then exists in the database

{"key":"globalAuthor:a.MUEkJofnDYt8FTaz","val":{"colorId":5,"name":null,"timestamp":1589553753138}}
{"key":"globalAuthor:a.MUEkJofnDYt8FTaz","val":{"colorId":5,"name":null,"timestamp":1589553753139}}

I think the offending line is:

let author = await authorManager.getAuthor4Token(req.cookies.token);   // author is of the form: "a.g2droBYw1prY7HW9"

In my database I see

{"key":"token2author:t.n5EWBniH5hguO3b97qcw","val":"a.Vb0hC3C2ve5aEK7v"}
{"key":"token2author:t.osEaKXq1hTMRkBmxMLFV","val":"a.ZeKg3KzPyiMffYCb"}
{"key":"token2author:t.weaih14n2ysGMrmugJy9","val":"a.FqS9HNqmeMBJ7ArG"}
{"key":"token2author:t.YLeHFwF9wiQvb3MV5jC7","val":"a.MUEkJofnDYt8FTaz"}

Note the lack of token for a.izLoj1kz285kznqZ

Surely there should be one?

So yeah, this boils down to us checking token2author but for session based activity this doesn't work.

1. Is token2author required to be populated for sessions?
2. Can we just check that the session is valid in the group then skip token2author (for group / session users).
3. Sessions expire, would you want to be able to import once the session has expired? One thinks not.. So perhaps group users should check for a valid session before importing, other non group pads can use the token2author check (which only checks if the user has contributed (not if their session has expired))...

I think to keep this issue in scope.

1. Will do #3 which will fix this problem and this problem only. This gives time to look at why a token is not being made for a group Session.

The whole token thing is weird, I'm not sure why we have it. I should be able to land a session lookup patch tonight and it will probably mean that if the session exists and is valid it wont need an edit IE the user will be able to use the pad from the get go.

Until the patch lands please use the setting in settings.json to override this requirement.

FTR (don't want to add noise here) but it may also affect bin/rebuildPad.js (?)

I found that using my createSession script which does create an author and seems to function properly when visiting a pad provides the wrong authorId to the session.. So wtf! Something is weird...

Client generates token.
Client calls authorID the userId
userId is being passed correctly to the client and being properly stored in clientVars.userId

So the server and the client knows the session
The server and the client know the userId.

So surely we just need the client to tell the Server the token value to use for it's authorId value?

I feel this is what's really broken.. When the client says "hey here is my token" it's missing something like the authorId value or so. That or the server is saying, "Oh that's nice you have a token" but because I can't find it here is a new authorId 🗡️

Okay wow that wa sa lot of digging and I had to do a rather hammer-esque-fix.

https://github.com/ether/etherpad-lite/pull/4012/files#diff-5d2aa4336c4964b828a123e20e09b73cR921

I think it needs refinement but it highlights the problem....

Gotta go nappy shopping now. That's all I got time for today!

@Rillke can you please test 4012.

Okay wow that wa sa lot of digging and I had to do a rather hammer-esque-fix.

Thanks

@Rillke can you please test 4012.

On my TODO List.

@Rillke can you please test 4012.

Imports without allowAnyoneToImport and "requireSession": true using the cited plugin work with 4012.

Thanks.

Waiting on @muxator to review/merge https://github.com/ether/etherpad-lite/pull/4012

4012 merged.