Etherpad-lite: Toolbar Icons not showing after update (1.6.2 -> 1.8.3) when exposing in a subdir via a reverse proxy and minify = true

Please provide details.

Are you running behind reverse proxy?
Are you running in development or production mode.
Minified?
Secured copy or your settings.json
Etc.

https://mclear.co.uk/2015/01/15/how-to-write-a-bug-report/

It's running in development mode, although the systemd service has "Environment=NODE_ENV=production" in it, no reverse proxy, the css is minified, settings.json is barely touched (only mysql and admin pw is changed).

It looks like etherpad can't load all fonts because it looks at /static/font/*.ttf instead of /etherpad/src/static/font/

Try turn minify to false and maxage to 0 in settings.json

I did, nothing changed

At my laptop now, might be able to help more. Most useful thing is to get your settings.json (but remove any passwords). Can you provide that please?

FWIW I tested 1.6.* > 1.8.3 and didn't experience this but it could be related to your settings.json having some legacy cruft in :!

If you don't want to provide settings one thing to do is:

cp settings.json settings.backup.json
cp settings.json.template settings.json
Then copy your database host/user/pass from settings.backup.json into settings.json

Restart and see if it's fixed.

/*
  This file must be valid JSON. But comments are allowed

  Please edit settings.json, not settings.json.template

  To still commit settings without credentials you can
  store any credential settings in credentials.json
*/
{
  // Name your instance!
  "title": "HelpMe-Etherpad",

  // favicon default name
  // alternatively, set up a fully specified Url to your own favicon
  "favicon": "favicon.ico",

  //IP and port which etherpad should bind at
  "ip": "0.0.0.0",
  "port" : 9001,

  // Option to hide/show the settings.json in admin page, default option is set to true
  "showSettingsInAdminPage" : true,

  /*
  // Node native SSL support
  // this is disabled by default
  //
  // make sure to have the minimum and correct file access permissions set
  // so that the Etherpad server can access them

  "ssl" : {
            "key"  : "/path-to-your/epl-server.key",
            "cert" : "/path-to-your/epl-server.crt",
            "ca": ["/path-to-your/epl-intermediate-cert1.crt", "/path-to-your/epl-intermediate-cert2.crt"]
          },

  */

  //The Type of the database. You can choose between dirty, postgres, sqlite and mysql
  //You shouldn't use "dirty" for for anything else than testing or development
  /*"dbType" : "dirty",
  //the database specific settings
  "dbSettings" : {
                   "filename" : "var/dirty.db"
                 },

   An Example of MySQL Configuration*/
   "dbType" : "mysql",
   "dbSettings" : {
                    "user"    : "etherpad",
                    "host"    : "localhost",
                    "password": "",
                    "database": "etherpad",
                    "charset" : "utf8mb4"
                  },
   "users": {
    "admin": {
        "password": "",
        "is_admin": true
        }
            },

  //the default text of a pad
  "defaultPadText" : "Willkommen zum CN-Etherpad!\n\nDieses Dokument wird beim Tippen laufend aktualisiert, sodass jeder immer denselben Text sieht. So kann man nahtlos zusammen an Dokumenten arbeiten.\n",

  /* Default Pad behavior, users can override by changing */
  "padOptions": {
    "noColors": false,
    "showControls": true,
    "showChat": true,
    "showLineNumbers": true,
    "useMonospaceFont": false,
    "userName": false,
    "userColor": false,
    "rtl": false,
    "alwaysShowChat": false,
    "chatAndUsers": false,
    "lang": "de-de"
  },

  /* Should we suppress errors from being visible in the default Pad Text? */
  "suppressErrorsInPadText" : false,

  /* Users must have a session to access pads. This effectively allows only group pads to be accessed. */
  "requireSession" : false,

  /* Users may edit pads but not create new ones. Pad creation is only via the API. This applies both to group pads and regular pads. */
  "editOnly" : false,

  /* Users, who have a valid session, automatically get granted access to password protected pads */
  "sessionNoPassword" : false,

  /* if true, all css & js will be minified before sending to the client. This will improve the loading performance massivly,
     but makes it impossible to debug the javascript/css */
  "minify" : false,

  /* How long may clients use served javascript code (in seconds)? Without versioning this
     may cause problems during deployment. Set to 0 to disable caching */
  "maxAge" : 21600, // 60 * 60 * 6 = 6 hours

  /* This is the absolute path to the Abiword executable. Setting it to null, disables abiword.
     Abiword is needed to advanced import/export features of pads*/
  "abiword" : "/usr/local/bin/abiword",

  /* This is the absolute path to the soffice executable. Setting it to null, disables LibreOffice exporting.
     LibreOffice can be used in lieu of Abiword to export pads */
  "soffice" : null,

  /* This is the path to the Tidy executable. Setting it to null, disables Tidy.
     Tidy is used to improve the quality of exported pads*/
  "tidyHtml" : "/usr/bin/tidy",

  /* Allow import of file types other than the supported types: txt, doc, docx, rtf, odt, html & htm */
  "allowUnknownFileEnds" : true,

  /* This setting is used if you require authentication of all users.
     Note: /admin always requires authentication. */
  "requireAuthentication" : false,

  /* Require authorization by a module, or a user with is_admin set, see below. */
  "requireAuthorization" : false,

  /*when you use NginX or another proxy/ load-balancer set this to true*/
  "trustProxy" : false,

  /* Privacy: disable IP logging */
  "disableIPlogging" : false,

  /* Users for basic authentication. is_admin = true gives access to /admin.
     If you do not uncomment this, /admin will not be available! */


  // restrict socket.io transport methods
  "socketTransportProtocols" : ["xhr-polling", "jsonp-polling", "htmlfile"],

  // Allow Load Testing tools to hit the Etherpad Instance.  Warning this will disable security on the instance.
  "loadTest": false,

  // Disable indentation on new line when previous line ends with some special chars (':', '[', '(', '{')
  /*
  "indentationOnNewLine": false,
  */

  /* The toolbar buttons configuration.
  "toolbar": {
    "left": [
      ["bold", "italic", "underline", "strikethrough"],
      ["orderedlist", "unorderedlist", "indent", "outdent"],
      ["undo", "redo"],
      ["clearauthorship"]
    ],
    "right": [
      ["importexport", "timeslider", "savedrevision"],
      ["settings", "embed"],
      ["showusers"]
    ],
    "timeslider": [
      ["timeslider_export", "timeslider_returnToPad"]
    ]
  },
*/
  /* The log level we are using, can be: DEBUG, INFO, WARN, ERROR */
  "loglevel": "INFO",

  //Logging configuration. See log4js documentation for further information
  // https://github.com/nomiddlename/log4js-node
  // You can add as many appenders as you want here:
  "logconfig" :
    { "appenders": [
        { "type": "console"
        //, "category": "access"// only logs pad access
        }
    /*
      , { "type": "file"
      , "filename": "your-log-file-here.log"
      , "maxLogSize": 1024
      , "backups": 3 // how many log files there're gonna be at max
      //, "category": "test" // only log a specific category
        }*/
    /*
      , { "type": "logLevelFilter"
        , "level": "warn" // filters out all log messages that have a lower level than "error"
        , "appender":
          {  Use whatever appender you want here  }
        }*/
    /*
      , { "type": "logLevelFilter"
        , "level": "error" // filters out all log messages that have a lower level than "error"
        , "appender":
          { "type": "smtp"
          , "subject": "An error occurred in your EPL instance!"
          , "recipients": "[email protected], [email protected]"
          , "sendInterval": 300 // 60 * 5 = 5 minutes -- will buffer log messages; set to 0 to send a mail for every message
          , "transport": "SMTP", "SMTP": { // see https://github.com/andris9/Nodemailer#possible-transport-methods
              "host": "smtp.example.com", "port": 465,
              "secureConnection": true,
              "auth": {
                  "user": "[email protected]",
                  "pass": "bar_foo"
              }
            }
          }
        }*/
      ]
    }
}

Okay cool, gimme 2 mins.

Nope, that didn't replicate, toolbar buttons exist for me..

Is this instance public that I can access? feel free to email me URL [email protected]

Stupid question.. Did you

• [x] Clear browser cache?
• [x] Attempt to run using bin/run.sh

Thanks!

Sadly it's not a public instance.
I replaced the settings.json with the new template, also changed skin to no-skin. stilll no success..
And yes, I've run run.sh, restarted the service and cleared the cash :)

This is the systemd service btw:
[Unit]
Description=Etherpad
After=syslog.target network.target

[Service]
Type=simple
User=etherpad
Group=etherpad
WorkingDirectory=/opt/etherpad
Environment=NODE_ENV=production
ExecStart=/opt/etherpad/bin/run.sh
Restart=always

[Install]
WantedBy=multi-user.target

How did you install? Did you git clone or did you grab the released .zip and unpack?

Request URL: https://dev.com:9001/static/font/RobotoMono-Regular.ttf

jose@server:~/develop$ locate RobotoMono | grep develop | grep Reg
/home/jose/develop/src/static/font/RobotoMono-Regular.ttf

All looks fine to me.

I installed it via git clone, or rather updated it. I believe it was installed a few years ago (by another admin) through zip unpack. I used rsync to update everything

Ah, I wonder if the last admin put some hacks in? if you git status do you see any modified files?

What happens if you git clone a fresh repo then copy your settings.json over? That will work but obviously isn't fixing the problem that something is weird..

I'm going to do that now. yes, it's very weird.. as if the root folder for the fonts is not the same as etherpad

The code is right btw. src/ is the root folder..

I think the previous admin hacked something, I can't say what without seeing a git diff. Can we close this now?

previous admin didn't use git (was not installed), still doesn't work after clean clone..

@skramsa lulwut. Completely clean clone and run and it doesn't work? :! I have a feeling something is weird with your setup.. Any errors or warnings in log?

What if you do a completely clean clone and don't change any settings? Does it still fail then?

Any chance you can make it public temporarily for me to debug?

no it's on a secured intranet.. could the upgrade from node 6 to 12 have something to do with it?

Not sure.. I'm not really sure other debug steps here..

The only clue I have is..
https://github.com/ether/etherpad-lite/blob/develop/src/node/utils/Minify.js#L429

But you said you cleaned all cache and do a clean run..

You disabled minify, did rm -Rf src/node_modules && rm src/package-lock.json then ran bin/run.sh yeah? I mean you did a clean clone so ;\

What's your git sha, npm --version and nodejs version values please?

npm is 6.14.4 and node -v is 12.16.2. stupid question, but what exactly do you mean with "my" git sha?

I rerolled to 1.6.2 and set minify to false and suddenly it works again (before that not even the old version's icons did function). That's a relief.. I will try to make another clean setup next week with this settings, maybe this time it will work :D Thank you so much for your help!!

In your etherpad folder type git rev-parse HEAD

I wanna see which commit you are on.

Hello
I have exactly the same problem except I'm behind a proxy.
I made a test by saving my installation and cloning the repository. Then I simply copied my configuration file and restarted the service.
I still have the same problem despite the fresh installation.

But :
On the other hand, interesting thing, when I access the service without using the proxy and without https, no problem, the icons are displayed correctly !!

The apache proxy configuration is coming from the documentation, nothing exotic on my side.

Had the exact same problem.

I use a reverse proxy and the Etherpad instance is in a subdirectory (/etherpad). I managed to make it right adding the following to my apache configuration.

<Location /static/font>
    ProxyPass http://localhost:9001/static/font
    ProxyPassReverse http://localhost:9001/static/font
</Location>

I did not change anything in settings.json.

@skramsa hasn't mentioned any reverse proxy so I don't think it's related? Skramsa please confirm there is no reverse proxy in your setup.

Had the exact same problem.

I use a reverse proxy and the Etherpad instance is in a subdirectory (/etherpad). I managed to make it right adding the following to my apache configuration.

<Location /static/font>
    ProxyPass http://localhost:9001/static/font retry=0 timeout=30
    ProxyPassReverse http://localhost:9001/static/font
</Location>

I did not change anything in settings.json.

Why the retry and timeout rule?

@skramsa hasn't mentioned any reverse proxy so I don't think it's related? Skramsa please confirm there is no reverse proxy in your setup.

Yes but @julpec is behind a proxy, so I though it could be helpful ;)

Had the exact same problem.
I use a reverse proxy and the Etherpad instance is in a subdirectory (/etherpad). I managed to make it right adding the following to my apache configuration.

<Location /static/font>
    ProxyPass http://localhost:9001/static/font retry=0 timeout=30
    ProxyPassReverse http://localhost:9001/static/font
</Location>

I did not change anything in settings.json.

Why the retry and timeout rule?

You're right, I don't think it's needed, I just did this in a hurry yesterday after the update ;)

@skramsa hasn't mentioned any reverse proxy so I don't think it's related? Skramsa please confirm there is no reverse proxy in your setup.

Yes but @julpec is behind a proxy, so I though it could be helpful ;)

It is thanks

As a reminder :

On the other hand, interesting thing, when I access the service without using the proxy and without https, no problem, the icons are displayed correctly !!

In my case, it's obvious it's proxy related.

In addition, I am in a migration situation of my Etherpad instance and I migrated my database and after that I updated Etherpad. I have this coming out in the logs now:

[2020-04-29 10:43:29.440] [ERROR] console - table is not configured with charset utf8mb4 -- This may lead to crashes when certain characters are pasted in pads

Don't know if it can cause errors…

As a reminder :

On the other hand, interesting thing, when I access the service without using the proxy and without https, no problem, the icons are displayed correctly !!

In my case, it's obvious it's proxy related.

In addition, I am in a migration situation of my Etherpad instance and I migrated my database and after that I updated Etherpad. I have this coming out in the logs now:

[2020-04-29 10:43:29.440] [ERROR] console - table is not configured with charset utf8mb4 -- This may lead to crashes when certain characters are pasted in pads
Don't know if it can cause errors…

Create a separate issue for this please. It needs addressing.

I have the same problem.
I'm behind a reverse proxy (Apache) and I'm using the (currently) latest docker container (pulled the image one minute ago):

$ docker image ls
REPOSITORY           TAG            IMAGE ID            CREATED             SIZE
etherpad/etherpad    latest         878d71df1b8e        44 hours ago        273MB

The toolbar looks exactly as in the original report. Etherpad is proxied under /pad/, but the fonts are loaded from /static/, which results in 404 errors.

I'd rather not use the workaround by @lucaguindani, because /static/ is a very common path which is also used by other applications.

Okay, will sort ASAP. Should be today, might not be merged this week though so you guys might need to apply a patch.

After release week we're all trying to spend time with family etc.

I can't replicate this with my apache config. Can someone paste an offending apache config?

Tested nginx too w/ the wiki config and can't replicate there ;/

Makes me think you guys are all experience a weird Apache mis-config / quirk. Where did you all get your apache subdir config from? We don't provide one on the wiki afaik so I think maybe you guys went rogue? :P

PS I'm testing against develop, which is pretty much release atm and the nginx config is:

server {
        listen       80;
        server_name  pad.example.com;

        access_log  /var/log/nginx/eplite.access.log;
        error_log   /var/log/nginx/eplite.error.log;

    location /pad {
        rewrite                /pad/(.*) /$1 break;
        rewrite                ^/pad$ /pad/ permanent;
        proxy_pass             http://localhost:9001/;
        proxy_pass_header Server;
        proxy_redirect         / /pad/;
        proxy_set_header       Host $host;
        proxy_buffering off;
    }

    location /pad/socket.io {
        rewrite /pad/socket.io/(.*) /socket.io/$1 break;
        proxy_pass http://localhost:9001/;
        proxy_redirect         / /pad/;
        proxy_set_header Host $host;
        proxy_buffering off;
        proxy_set_header X-Real-IP $remote_addr;  # http://wiki.nginx.org/HttpProxyModule
        proxy_set_header X-Forwarded-For $remote_addr; # EP logs to show the actual remote IP
        proxy_set_header X-Forwarded-Proto $scheme; # for EP to set secure cookie flag when https is used
        proxy_set_header Host $host;  # pass the host header
        proxy_http_version 1.1;  # recommended with keepalive connections
        # WebSocket proxying - from http://nginx.org/en/docs/http/websocket.html
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }

    location /static {
        rewrite /static/(.*) /static/$1 break;
        proxy_pass http://localhost:9001/;
        proxy_set_header Host $host;
        proxy_buffering off;
    }

}

# we're in the http context here
map $http_upgrade $connection_upgrade {
  default upgrade;
  ''      close;
}

I did notice tests/frontend/index.html fails which seems weird

I also tried on /pad/foo/p/blah just to confirm there is no bug Etherpad side and things look fine...

I can't replicate this with my apache config. Can someone paste an offending apache config?

Here is mine:

<Location /pad/>
    Order allow,deny
    Allow from all
    Options All +MultiViews -ExecCGI
    AuthType Basic
    AuthName "authentification"
    AuthBasicProvider       ldap
    AuthLDAPURL "ldap://127.0.0.1:389/ou=****,dc=****,dc=****,dc=org"
    AuthLDAPGroupAttribute memberUid
    AuthLDAPGroupAttributeIsDN off
    require ldap-group cn=****,ou=***,dc=*****,dc=****,dc=org

    ProxyPass http://localhost:9002/ 
    ProxyPassReverse http://localhost:9002/
    RewriteEngine On
    RewriteCond %{LA-U:REMOTE_USER,NS} (.+)
    RewriteRule . - [E=RU:%1]
    RequestHeader add X-Forwarded-User %{RU}e
</Location>

Awesome thanks gimme an hour will try tonight.

Makes me think you guys are all experience a weird Apache mis-config / quirk. Where did you all get your apache subdir config from? We don't provide one on the wiki afaik so I think maybe you guys went rogue? :P

Here

Makes me think you guys are all experience a weird Apache mis-config / quirk. Where did you all get your apache subdir config from? We don't provide one on the wiki afaik so I think maybe you guys went rogue? :P

Here

I looked there and the examples don't cover putting Etherpad in a subdir for Apache...

Using a slightly modified version of your snip

<VirtualHost *:81>
<Location /pad/>
    Order allow,deny
    Allow from all
    Options All +MultiViews -ExecCGI
    ProxyPass http://localhost:9001/
    ProxyPassReverse http://localhost:9001/
    RewriteEngine On
    RewriteCond %{LA-U:REMOTE_USER,NS} (.+)
    RewriteRule . - [E=RU:%1]
</Location>
</VirtualHost>

jose@server:~/develop$ apache2 -v
Server version: Apache/2.4.29 (Ubuntu)
Server built:   2020-03-13T12:26:16

I looked there and the examples don't cover putting Etherpad in a subdir for Apache...

it doesn't make sense. You can put whatever you want in the location tag. But I will try with you config and without "subdirectory" right now.
Thank for your investigations btw

it doesn't make sense. You can put whatever you want in the location tag. But I will try with you config and without "subdirectory" right now.

it doesn't make sense... in my brain. But in the real world that's what's causing the problem!
Good point :+1:
However, I don't understand why everything worked so well before, it's been running for years behind a reverse proxy with this as a configuration.
I really need a subdirectory since I have different services running behind an apache "single sign-on".
Would it be possible to use something other than "/" as a location?

Thanks for you work on this Issue

Copy/paste my config from above, I use a different location...

I don't see the difference with my own, except for the authentication settings.
I'm gonna try right now

Copy/paste my config from above, I use a different location...

still the same problem: no display of icons

Apache version?

Server version: Apache/2.4.38 (Debian)
Server built: 2019-10-15T19:53:42

I investigated with the element inspector and this is what he gives me as URL for the fonts :

Request URL: https://example.org/static/font/RobotoMono-Regular.ttf

It should be:

Request URL: https://example.org/pad/static/font/RobotoMono-Regular.ttf

I can't understand why I don't have the "pad" subdirectory.

So, I started from scratch on a new host.

docker run --name etherpad -d \
    -e NODE_ENV=production -p 9001:9001 \
    --mount type=bind,src=/etc/localtime,dst=/etc/localtime \
    --restart=unless-stopped \
    etherpad/etherpad

Nothing changed in the original configuration.

Apache config:

<VirtualHost _default_:443>
    ServerName host
    <Proxy *>
        Require all granted
    </Proxy>
    ProxyRequests Off
    ProxyPreserveHost On
    ProxyPass             /pad/     http://localhost:9001/
    ProxyPassReverse      /pad/     http://localhost:9001/
    SSLCertificateFile  /etc/apache2/ssl/host.example.com_cert_with_chain.pem
    SSLCertificateKeyFile /etc/apache2/ssl/host.example.com_sec_key.pem
    SSLCertificateChainFile /etc/apache2/ssl/chain.pem
</VirtualHost>

Apache version is 2.4.29.

Cleared the browser cache, and the result:

Weird. I will try use the docker image locally to see if that changes things

@JohnMcLear it's really unbelievable. I tried with exactly the same configuration as you, on port 81, in a separate vhost, etc... and the problem still occurs.
I made sure to empty the cache every time.

Here are my software versions:

node: v10.20.1
npm: 6.14.4
apache2: Apache/2.4.38 (Debian)
etherpad: 1.8.3 (6210114)

I have exactly the same issue. The icons are missing when run over the proxy (nginx) and minify is set to true. As soon as I set minify to false and start it with clearRun.sh and clear the browser cache, the icons are working. In the older version we used for quite some time, it was working with minify: true and this Nginx config:

server {
        listen          80;
        listen          443 ssl;
        include snippets/mortis2-ssl.conf;
        server_name     mortis2;
        location /pad/
        {
                #proxy_pass        http://localhost:9001/;
                proxy_set_header  Host $host;
                proxy_buffering   off;
        }
}

I found out, when I use the nginx config from @JohnMcLear and use rewrite, it will work. I guess that problem only pops up, when we use a different location (like /pad).

Nginx config that is working:

server {
        listen          80;
        listen          443 ssl;
        include snippets/mortis2-ssl.conf;
        server_name     mortis2;
        location /pad/
        {
                rewrite                /pad/(.*) /$1 break;
                rewrite                ^/pad$ /pad/ permanent;
                proxy_pass        http://localhost:9001/;
                proxy_pass_header Server;
                proxy_set_header  Host $host;
                proxy_buffering   off;
        }

        location /pad/socket.io {
                rewrite /pad/socket.io/(.*) /socket.io/$1 break;
                proxy_pass http://localhost:9001/;
                proxy_redirect         / /pad/;
                proxy_set_header Host $host;
                proxy_buffering off;
                proxy_set_header X-Real-IP $remote_addr;  # http://wiki.nginx.org/HttpProxyModule
                proxy_set_header X-Forwarded-For $remote_addr; # EP logs to show the actual remote IP
                proxy_set_header X-Forwarded-Proto $scheme; # for EP to set secure cookie flag when https is used
                proxy_set_header Host $host;  # pass the host header
                proxy_http_version 1.1;  # recommended with keepalive connections
                # WebSocket proxying - from http://nginx.org/en/docs/http/websocket.html
                proxy_set_header Upgrade $http_upgrade;
                #proxy_set_header Connection $connection_upgrade;
        }

        location /static {
                rewrite /static/(.*) /static/$1 break;
                proxy_pass http://localhost:9001/;
                proxy_set_header Host $host;
                proxy_buffering off;
        }
}

Op said minify true/false doesn't change outcome. Op also didn't mention any rev proxy.... I will check with minify true when I get home.

Confirmed it's caused by minify.

https://github.com/ether/etherpad-lite/issues/3956#issuecomment-620476150 claimed minify false made no difference and I should have checked myself if that was the case..

*facepalm.

Will fix ASAP.

FWIW Nginx also suffers from this failure and the failure is present on all pages (pad, index, timeslider etc).

The problem is with our implementation of clean-css. Css 3 to 4 removed the relativeTo option and this removal has caused this bug. I switched to rebase and rebaseTo in my example

https://github.com/JohnMcLear/etherpad-lite/commit/e684a3ba4129d1bae54e7b65a316a7d690217241 and is a semi-fix and https://github.com/JohnMcLear/etherpad-lite/commit/a8c66b3b513b058e89f70b8e8d5ab622b33f167d is defo a hack. Icons will appear but the font is wrong for some pad internals and probably other items..

Any suggestions @jakubpawlowicz @seballot ?

@JohnMcLear one way to make it work for you may be through arbitrary transformations, see https://github.com/jakubpawlowicz/clean-css#how-to-apply-arbitrary-transformations-to-css-properties

The other idea is to pass input options to clean-css via a hash, see https://github.com/jakubpawlowicz/clean-css/blob/master/test/module-test.js#L620, e.g.

new CleanCSS().minify({"path/to/stylesheet/a.css": { styles: "/* source of a.css goes here */" }})

and use rebaseTo option, but without further insight I cannot tell if it would work or not.

Good luck!

Closed ? really ?

OP closed... It should not be closed. OP FWIW you can click unfollow or unsubscribe or something if you don't want noise.

This is still not resolved, if it's not had a patch by EOW I will try to patch.

I'd really like to see a community patch for this though if someone can please find time.

@JohnMcLear I would really like to be able to help and take the time on this but I have no developer skills and even less in nodejs. I hope others will be able to do it. I can only support you with my thoughts…

While I appreciate it, your thoughts don't change nappies ;)

bump @seballot

Hi, do not have a lot od idea on this one sorry... Is there a way to reproduce it locally?

1. git clone develop
2. turn minify to true
3. Run Etherpad and visit a pad (ensuring things are working)
4. install nginx
5. Copy the below to /etc/nginx/sites-enabled/default

server {
        listen       80;
        server_name  pad.example.com;

        access_log  /var/log/nginx/eplite.access.log;
        error_log   /var/log/nginx/eplite.error.log;

    location /pad/foo {
        rewrite                /pad/foo/(.*) /$1 break;
        rewrite                ^/pad/foo$ /pad/foo permanent;
        proxy_pass             http://localhost:9001/;
        proxy_pass_header Server;
        proxy_redirect         / /pad/foo;
        proxy_set_header       Host $host;
        proxy_buffering off;
    }

    location /pad/foo/socket.io {
        rewrite /pad/foo/socket.io/(.*) /socket.io/$1 break;
        proxy_pass http://localhost:9001/;
        proxy_redirect         / /pad/foo/;
        proxy_set_header Host $host;
        proxy_buffering off;
        proxy_set_header X-Real-IP $remote_addr;  # http://wiki.nginx.org/HttpProxyModule
        proxy_set_header X-Forwarded-For $remote_addr; # EP logs to show the actual remote IP
        proxy_set_header X-Forwarded-Proto $scheme; # for EP to set secure cookie flag when https is used
        proxy_set_header Host $host;  # pass the host header
        proxy_http_version 1.1;  # recommended with keepalive connections
        # WebSocket proxying - from http://nginx.org/en/docs/http/websocket.html
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }

    location /static {
        rewrite /static/(.*) /static/$1 break;
        proxy_pass http://localhost:9001/;
        proxy_set_header Host $host;
        proxy_buffering off;
    }

}

# we're in the http context here
map $http_upgrade $connection_upgrade {
  default upgrade;
  ''      close;
}

1. Restart nginx
   http://127.0.0.1:9001/p/blah becomes http://127.0.0.1/pad/foo/p/blah
2. Open http://127.0.0.1/pad/foo/p/blah in browser
3. Click network tab in browser & notice that the iframe inner fonts 404. This is due to the @import passing the right path but cleanCSS rewriting it (rightly so) to ../../../ -- my understanding is that because both the pad and editor use the same @font.css and they both have ?different basePaths? then one works and one doesn't upon minification. I can't think of an elegant fix but I think you might be able to?

@seballot any progress buddy?

No sorry I cannot find a fix...

Me neither, the cleanCSS is doing some magic that I just don't understand (I'm not a CSS guy) and none of my instances run in subdirs so it's kinda not interesting for me to learn either

I don't want to but I might need to replace cleanCSS with csso because I can get csso to to work but I think that ultimately I'm not solving the problem properly and csso is no where near as performant as cleanCSS for our use case.

I bumped cleanCSS dev to see if we can collaborate to fix :)

nice ! if it works nice with csso, why not go with it?

csso minifies but doesn't reduce # of page loads. Also I prefer cleanCSS as a project/docs etc. and don't mind throwing some resources at a good project it means I can focus on Ether-specific-ventures

I am facing the same issue as in this thread.

• Installed etherpad from git repository and when I don ./bin/run.sh, and open port 9001 on localhost, the installation is good and icons properly shown.
• When i run this on nginx server, (which is also running jitsi), I do not see the toolbar icons. Everything else seems to work great.

Can someone guide me how I can workaround this problem. There are no major changes in settings.json file.

Workaround is to set minify to false in settings.json

I'll try to have a look this night.

I was able to replicate the issue using the nginx config given by John in https://github.com/ether/etherpad-lite/issues/3956#issuecomment-625163151 in this same issue.

I think there is a problem both in the posted Nginx configuration and in Etherpad's code.

More details later.

I just spent 2 seconds looking at the nginx code and it shouldn't have config for /static as it is in the copy/paste I did but the root of the problem is deep within cleanCSS...

We have various issues going on here.

1. the test Nginx config file had the wrong redirect for /static. The whole location /static section has to be deleted from here. In the email storm received during these days, I think to remember that a user complained - indeed - that /static was going to clash with a lot of other web applications. Probably he was misled by an example similar to this one;

   This has to be fixed going through the documentation (the wiki? Somehwere else on the internet?) and improving the quality of the examples.

2. 4177b3f9434e moved the font-face declarations from src/static/css/pad.css to two imported files (src/static/css/pad/fonts.css and src/static/css/pad/toolbar.css) in a different directory, and URLs in src are relative to the stylesheet path;

3. on top of that, after merging 95fd5ce2a45e for the forthcoming 1.8.4, we would have a cache staleness problem that also needs to be addressed.

I'm doing some tests and preparing a PR.

Opened #4002 to fix this issue. Many thanks to @JohnMcLear for the preliminary work.

@skramsa, @julpec, @lucaguindani, @ashishhari (and John, obviously): is someone of you in the position of testing that PR to see if it works for them? I would like an independent validation, if possible.

Please also note that the reverse proxy configuration needs no special rule for "/static".

Thank your for acknowledging. Setting minify to false did not solve the
problem for me.
I am using this on a jitsi installation on an AWS EC2 instance, and there
is a nginx server being used to host jitsi and also to start the node
server for etherpad service on same machine on port 9001.

Does this mean I may need to tweak some other nginx setting? (I dont
understand reverse proxy properly but could this be that case?), Or do I
just need to wait for the fix to be available.

On Thu, May 14, 2020 at 6:07 AM muxator notifications@github.com wrote:

We have various issues going on here.

1.

the test Nginx config file had the wrong redirect for /static. The
whole location /static section has to be deleted from here
https://github.com/ether/etherpad-lite/issues/3956#issuecomment-625163151.
In the email storm received during these days, I think to remember that a
user complained - indeed - that /static was going to clash with a lot of
other web applications. Probably he was misled by an example similar
to this one;

This has to be fixed going through the documentation (the wiki?
Somehwere else on the internet?) and improving the quality of the
examples.
2.

4177b3f
https://github.com/ether/etherpad-lite/commit/4177b3f9434e8516f8fe4f8e02b2a75ce5d64173
moved the font-face declarations from src/static/css/pad.css to two
imported files (src/static/css/pad/fonts.css and
src/static/css/pad/toolbar.css) in a different directory, and URLs in
src are relative to the stylesheet path
https://developer.mozilla.org/en-US/docs/Web/CSS/@font-face/src;
3.

on top of that, after merging 95fd5ce
https://github.com/ether/etherpad-lite/commit/95fd5ce2a45e31980ad8c2562e3fe6f77c52d4ff
for the forthcoming 1.8.4, we would have a cache staleness problem
that also needs to be addressed.

I'm doing some tests and preparing a PR.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ether/etherpad-lite/issues/3956#issuecomment-628317976,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AH7VXAS65ZUAR33CPKSI6OLRRM4KZANCNFSM4MSSRTRQ
.

@skramsa, @julpec, @lucaguindani, @ashishhari (and John, obviously): is someone of you in the position of testing that PR to see if it works for them? I would like an independent validation, if possible.

Ok for me to test. I don't have special configuration for static.

Thank your for acknowledging. Setting minify to false did not solve the problem for me.

But it works for me. Under apache, I switched to false and everything started working fine.

I can test it.

On Thu, May 14, 2020 at 1:30 PM julpec notifications@github.com wrote:

@skramsa https://github.com/skramsa, @julpec https://github.com/julpec,
@lucaguindani https://github.com/lucaguindani, @ashishhari
https://github.com/ashishhari (and John, obviously): is someone of you
in the position of testing that PR to see if it works for them? I would
like an independent validation, if possible.

Ok for me to test. I don't have special configuration for static.

Thank your for acknowledging. Setting minify to false did not solve the
problem for me.

But it works for me. Under apache, I switched to false and everything
started working fine.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ether/etherpad-lite/issues/3956#issuecomment-628464590,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AH7VXAXKTJKZCAZVRXZIJX3RROQLPANCNFSM4MSSRTRQ
.

Also confirming -> setting minify to false did work correctly on my system.
(i probably did not restart the service correctly last time).

On Thu, May 14, 2020 at 1:30 PM julpec notifications@github.com wrote:

@skramsa https://github.com/skramsa, @julpec https://github.com/julpec,
@lucaguindani https://github.com/lucaguindani, @ashishhari
https://github.com/ashishhari (and John, obviously): is someone of you
in the position of testing that PR to see if it works for them? I would
like an independent validation, if possible.

Ok for me to test. I don't have special configuration for static.

Thank your for acknowledging. Setting minify to false did not solve the
problem for me.

But it works for me. Under apache, I switched to false and everything
started working fine.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ether/etherpad-lite/issues/3956#issuecomment-628464590,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AH7VXAXKTJKZCAZVRXZIJX3RROQLPANCNFSM4MSSRTRQ
.

But it works for me. Under apache, I switched to false and everything started working fine.
Also confirming -> setting minify to false did work correctly on my system.

@julpec, @ashishhari: setting minify = false is a known workaround.
The hope now is to see if #4002 fixes the bug when minify = true.

The procedure would be the following, do this only if all the steps are clear for you:

Start replicating the bug:

• ensure minify = true in settings.json
• restart Etherapd
• visit Etherpad page and verify that the Toolbar icons are not shown

Try the fix:

• keep minify = true in settings.json
• apply #4002 (do it only if you are comfortable with it)
• restart Etherpad
• visit Etherpad page and verify if the Toolbar icons are now correctly shown

I would like but I don't know the procedure to apply the PR :(

Thanks @muxator I will test within the hour.

Oh wow I would have never come up with that fix! That's bonkers.

git remote add mux https://github.com/muxator/etherpad-lite/
git fetch mux
git checkout mux/fix/fonts-behind-reverse-proxy

For those also wanting to test..

after testing remember to git checkout develop else you will in muxs' stale branch

thanks, I will test this afternoon

Hrm I think something is not quite performant @muxator.. I'm seeing ~54 requests, vs ~49 (on current develop) -- Note that I have plugins enabled giving extra requests. So this PR would be a page load performance regression to satisfy an undocumented/unsupported use case which is not acceptable IMHO.

Note the multiple requests for the same file... :( It's defo progress but multiple requests for the same file is not the solution we're looking for.

the multiple request come from the different iframe I think. Both main page and editor iframe need the font, it has always been like this no?

afaik cleanCSS was used to fix that issue @seballot

it has always been like this no?

Yes, it was always this way.

I think cleanCss was more to avoid loading each css partial (pad/popups.css, pad/editbar.css, pad/chat.css ...), and instead load only one pad.css

But using iframe, the browser is obliged to request twice each file share with the iframe. But the second call should use local cache ("loaded from cache")

I applied the pull request and it works without any problem (with apache)

Thanks @julpec.

This evening I'll try to address the # of requests concern @JohnMcLear brought up. If there is nothing going on, I'll merge the PR.

Awesome tnx mux

Is there a way to upload HAR files somewhere in Github? I have tested #4002 and I am willing to share the recordings, but I do not know where to upload them.

Common elements:

• visited http://localhost:9001/p/ciao
• no reverse proxy
• production mode: false
• start with empty browser cache

Variables:

• version: 4816785aef45 vs #4002
• cache: cold (first load) vs hot (>= 2nd load)
• minify: true vs false
• browser: chromium 81 vs firefox 76

These are four binary variables -> 2^4 = 16 possibilities.
This took a lot of time!

Here's the more relevant cases:
|Description|current develop (4816785aef45)|PR #4002|
|----|----|----|
|Firefox, minified, first load|34 requests
1,61 MB
1.61 MB transferred|36 requests
1,61 MB
1.53 MB transferred|
|Firefox, minified, hot cache|29 requests
1,12 MB
216 KB transferred|31 requests
1,18 MB
216 KB transferred|
|Chromium, minified, first load|44 requests
2,0 MB
1,5 MB transferred|109 richieste
3,0 MB
1,1 MB transferred|
|Chromium, minified, hot cache|45 requests
2,2 MB
5,7 KB transferred|129 requests
4,7 MB
4,7 KB transferred|

The bottom line is:

• Chrome and Firefox have a completely different way of measuring their traffic
• PR #4002 introduces more requests to the server
• The cache hit ratio improves: the transferred column decreases!
• Load times stay the same, within limits of measurement noise

These were my findings too, I also took the time to research page load speed performance profiling tools (beyond developer tools) and decided that if developer tools weren't giving me significant deviation from a mean then the results were suitable enough for our application.

Total transfer size decreasing is clearly a good thing, but http request overhead can be expensive due to latency. We can merge this but be mindful that when we rewrite the front end (which feels inevitable due to all of the inline JS that will ultimately need to be shifted around) then this is an area we should strive to improve.

All things said, Etherpad is great compared to most others in htis field, the exception being Google docs which is the gold standard for absolutely shockingly fast "time to edit" which in this world is king, not end of page 'load'.

So TLDR; Merge away @muxator - this thread exists as a resource for those looking to optimize for Etherpad instances NOT in subfolders.

great work !

+1 tnx!

:+1: