Dietpi: Problem with cron and auto backup (from v6.10-6.11)?

@eljotx
Thank you for your report.

G_USER_INPUT=0 should be actually auto detected on cron executions, thus not necessary to assign. But if you want it to have affect, you need to export it, which is of course worth to try:
export G_USER_INPUT=0

On non-interactive executions, the script would exit, if (close to) insufficient free space is detected on target drive: https://github.com/Fourdee/DietPi/blob/master/dietpi/dietpi-backup#L456
But you say, manual execution works fine? In case of insufficient free space you would see whiptail prompt then that gives you the choice to ignore or exit.

About the log entries, some cryptic colour codes + checking for elevation root access is expected. Try:
cat /mnt/rpi/backup.log
to have the colour codes active and thus formatted log. Also please provide this output here, so we can check at which stage the issue occurred.

Script working well when I run it manually.

Strange.

Try running the cron job and paste log results /mnt/rpi/backup.log

NB: dietpi-backup automatically outputs rsync to logfile /var/log/dietpi-backup.log, you could use that aswell.

So, i try running the cron job without G_USER_INPUT=0 (I removed this from script) and...

My backup log file from /mnt/rpi/backup.log
[......] Checking for (elevated) root access

dietpi-backup.log is empty

I did not see any active process after time when cron started.

So, I added to testing another script, to see output in file after cron start:

#!/bin/sh

echo 'Cron works!' > /mnt/rpi/cronworks.log

but this script not started - there is no file "cronworks.log" at /mnt/rpi/
it's strange because there is output file (backup.log) from my script with backup (from 1st my post), but no from this simple

@eljotx
Okay there seems to be at least as well an error with cron. Note that cron starts the scripts via run-parts /etc/cron.X/script and there are some rules, e.g. the scripts must be owned by root (AFAIK) and more sure the scripts are not allowed to have file endings. That means /etc/cron.daily/script.sh will be skipped, the name needs to be /etc/cron.daily/script instead.

Also it should not be needed, but try to use #!/bin/bash as shebang when using with DietPi scripts. Those themselves have #!/bin/bash, but to be fail safe try to use it as well within the cron script.

Script always had a name "backup" (without .sh), I change "#!/bin/sh" to "#!/bin/bash" in script too, and this script has owned by root. My "test" script from last post working now too.

I tried everything, but still is not working. I change script to print to log file after finish one job and I saw that script starts, but backup isn't start or finished (dietpi-backup.log is empty), because can't go to next job like tar, rclone etc. And also after running cron I saw process /usr/sbin/cron -f working and -bash - it's normal?

Script still working if I run it manually - but when backup finish, I must click OK and Cancel (whiptail screens), and after that - next jobs starts.

I think that problem is with using dietpi-backup to auto backup started by cron. It worked on 6.9 version, but no working after rewriting dietpi-backup.

@eljotx
So again, just to be sure:

• If you execute manually /DietPi/dietpi/dietpi-backup 1 is successfully runs through?
• If you run G_USER_INPUTS=0 /DietPi/dietpi/dietpi-backup 1 is works without any error as well, showing something like:
  

root@VM-Jessie:~# G_USER_INPUTS=0 /DietPi/dietpi/dietpi-backup 1
[  OK  ] Root access verified.
[  OK  ] DietPi-Drive_Manager | RootFS R/W access verified.

[  OK  ] DietPi-Backup | Checking for pre-req APT packages: rsync
[ INFO ] DietPi-Backup | Flagged for installation: rsync
[  OK  ] DietPi-Backup | APT installation for: rsync, please wait...
Selecting previously unselected package rsync.
(Reading database ... 36227 files and directories currently installed.)
Preparing to unpack .../rsync_3.1.1-3+deb8u1_amd64.deb ...
Unpacking rsync (3.1.1-3+deb8u1) ...
Processing triggers for systemd (215-17+deb8u7) ...
Setting up rsync (3.1.1-3+deb8u1) ...
Processing triggers for systemd (215-17+deb8u7) ...

[  OK  ] DietPi-Backup | G_AGI: rsync

 DietPi-Backup
─────────────────────────────────────────────────────
 Mode: Backup

[  OK  ] DietPi-Backup | DietPi-Userdata validation: /mnt/dietpi_userdata
[ SUB1 ] DietPi-Services > stop
[  OK  ] DietPi-Services | occ maintenance:mode --on
[  OK  ] DietPi-Services | stop : cron
[  OK  ] DietPi-Services | stop : sonarr
[  OK  ] DietPi-Services | stop : lighttpd
[  OK  ] DietPi-Services | stop : php5-fpm
[  OK  ] DietPi-Services | stop : mysql
[ SUB1 ] DietPi-Services > stop
[  OK  ] DietPi-Services | stop : cron
[  OK  ] DietPi-Services | stop : sonarr
[  OK  ] DietPi-Services | stop : lighttpd
[  OK  ] DietPi-Services | stop : php5-fpm
[  OK  ] DietPi-Services | stop : mysql
[ INFO ] DietPi-Backup | Backing up to: /mnt/dietpi-backup
[  OK  ] DietPi-Backup | Free space check: path=/mnt/dietpi-backup/data | available=5695 MB | required=1861 MB
[  OK  ] DietPi-Backup | rsync -aH --delete --delete-excluded --exclude-from=/tmp/.dietpi-backup_filter_inc_exc -v --log-file=/var/log/dietpi-backup.log / /mnt/dietpi-backup/data/
[ INFO ] DietPi-Backup | Backup Completed:

Backup was saved to:
- /mnt/dietpi-backup
- Log file: /var/log/dietpi-backup.log
[ SUB1 ] DietPi-Services > start
[  OK  ] DietPi-Services | start : mysql
[  OK  ] DietPi-Services | start : php5-fpm
[  OK  ] DietPi-Services | start : lighttpd
[  OK  ] DietPi-Services | start : sonarr
[  OK  ] DietPi-Services | start : cron
[ SUB2 ] DietPi-Process_tool > Apply
[  OK  ] DietPi-Process_tool | Completed
[  OK  ] DietPi-Services | occ maintenance:mode --off

• Running the exact command row from the script

/DietPi/dietpi/dietpi-backup 1 > /mnt/rpi/backup.log && tar zcfv /mnt/rpi/backup.tar.gz /mnt/backup/dietpi-backup/ >> /mnt/rpi/backup.log && rclone copy /mnt/rpi/backup.tar.gz dysk: -L >> /mnt/rpi/backup.log && rm -r /mnt/rpi/backup.tar.gz >> /mnt/rpi/backup.log && reboot

works as well, showing all got executed and finished in /mnt/rpi/backup.log?

• run-parts --test /etc/cron.daily lists your script (if it's in daily, otherwise adjust)
• /usr/sbin/cron -f + -bash is expected yes, as it starts a bash environment due to #!/bin/bash.

If all the above is true, then could you try to manually start run-parts and see if it executes as expected:

• Move/copy script into separate folder, e.g. ~/testdir/backup
• Then run-parts ~/testdir

I tested as you show and on every your question: yes, everything working when I start it manually.
I tested it with run-parts command, but it is still working - when I do it manually.

My script is listed after use run-parts --test /etc/cron.daily

If I add G_USER_INPUTS=0 and start script manually, should I see the screens (whiptail) with information about log etc.? Because I see it always.

@eljotx
G_USER_INPUTS=0 just disables whiptail menus to show. Information is just shown as terminal messages and questions are answered as "no" (cancel in case of insufficient free space check, no log view).

To force it for the script, you need to either export it before executing script:

#!/bin/bash
export G_USER_INPUTS=0
/DietPi/dietpi/dietpi-backup 1

or easier just directly give it over: G_USER_INPUTS=0 /DietPi/dietpi/dietpi-backup 1

I just recognized that you used G_USER_INPUT=0 above without S, it needs to be G_USER_INPUTS=0 with S.

But if not the variable is exported, scripts will determine, if user inputs can be made or not. Within cron and e.g. systemd unit executions, all DietPi scripts should automatically determine and assign G_USER_INPUTS=0.
You could verify that our method works, by adding to start of your cron job (or a test job):
[[ -t 0 ]] && echo 'This environments allows user inputs' > /mnt/rpi/inputs.log || echo 'This environment does not allow user inputs' > /mnt/rpi/inputs.log
When running the script manually, it should say that user inputs are allowed (whiptail menus show up), if it is started via cron, then it should not, and no whiptail menus should make the script hang.

First I test G_USER_INPUTS=0 and if I start script manually, log:
This environments allows user inputs
else if start by cron:
This environment does not allow user inputs

Next I try to change my script to this:

#!/bin/bash
export G_USER_INPUTS=0
date > /mnt/rpi/backup.log && /DietPi/dietpi/dietpi-backup 1 && (echo "Backup: DONE. " >> /mnt/rpi/backup.log && tar zcfv /mnt/rpi/backup.tar.gz /mnt/backup/dietpi-backup/ && echo "Tar backup: DONE. " >> /mnt/rpi/backup.log && rclone copy /mnt/rpi/backup.tar.gz dysk: && echo "Copy to GDrive: DONE. " >> /mnt/rpi/backup.log && rm -r /mnt/rpi/backup.tar.gz && echo "Remove tar: DONE." >> /mnt/rpi/backup.log && reboot) || echo "Backup failed!" >> /mnt/rpi/backup.log

and dietpi-backup.log is still empty, but backup.log:

when cron started and automatically start this script:

pon, 9 lip 2018, 18:54:01 CEST
Backup failed!

when I started script manually:

pon, 9 lip 2018, 18:58:40 CEST
Backup: DONE.
Tar backup: DONE.
Copy to GDrive: DONE.
Remove tar: DONE.

It's very annoying... next, my script only has:

#!/bin/bash
export G_USER_INPUTS=0
/DietPi/dietpi/dietpi-backup 1

and still not working if this script is started by cron.

Problem is still with dietpi-backup started by cron from v6.10. I think that is problem with changes did you made in v6.10 in dietpi-backup script.

dietpi-backup.log is still empty

@eljotx
Thanks for your testing effort.
I will go through our changed. Have no clue so far. At least according to your tests, the script is actually not hanging (whiptail/G_USER_INPUTS is not the issue), but failing.

Made some own testing:

[......] Checking for (elevated) root access

After commenting root user check, within dietpi-backup, executing it via cron leads to:
[FAILED] RootFS is currently Read Only, unable to continue.
But:

root@VM-Jessie:~# G_CHECK_ROOTFS_RW
[  OK  ] Root access verified.
[  OK  ] DietPi-Drive_Manager | RootFS R/W access verified.

@eljotx

tput is the issue:
tput: No value for $TERM and no -T specified

• Interesting, as we called that with v6.9 as well 🤔.

Fix: https://github.com/Fourdee/DietPi/commit/82ac7b32d32dca9db4fdb824c7ead80174844090

This patch also fixes another problem: "run_ntpd 1" run by cron daily or hourly does not sync the time (the timestamp of /var/lib/systemd/clock does not change).

@kmakisara
Jep you are right, this makes sense.
Just to avoid manual script edit to apply fix, all with the same issue should do: wget https://raw.githubusercontent.com/Fourdee/DietPi/82ac7b32d32dca9db4fdb824c7ead80174844090/dietpi/func/dietpi-globals -O /DietPi/dietpi/func/dietpi-globals

I think some of our opened issues are related to this. If run_ntpd within cron.hourly fails, dietpi-logclear will not be called afterwards as well, leading to /var/log filling up: https://github.com/Fourdee/DietPi/issues/1920

I am just wondering, why this issue showed up first with v6.10+, as we used tput already before within our cron jobs.
Maybe someone can find the change, that caused the issues, so we know how to take care such better in the future:

• Calling animated process notification v6.9: https://github.com/Fourdee/DietPi/blob/7b682286e72ef339518d110892a54e7ce405b648/dietpi/func/dietpi-globals#L301-L315
• Changes made with v6.10: https://github.com/Fourdee/DietPi/pull/1867/files
• tput cuu and tput cols is called in both cases, as well as tput ed on every other DietPi notification when clearing the processing animation.
• Maybe it is was not the changes on the notifications that caused the issue, but cron handling errors (or tput) differently after certain update.

Further testing on this:

root@VM-Stretch:/tmp# cat /etc/cron.minutely/test
#!/bin/bash

echo "$TERM" &> /tmp/cron.test
tput ed &>> /tmp/cron.test
echo 'finish' &>> /tmp/cron.test
root@VM-Stretch:/tmp# cat cron.test
dumb
tput: No value for $TERM and no -T specified

• $TERM on cron is set to dumb
• tput claims there is no value for the variable, even it is, although invalid.

root@VM-Stretch:/tmp# TERM='dumb' tput cols
237
root@VM-Stretch:/tmp# export TERM='dumb' tput cols
root@VM-Stretch:/tmp# export TERM='dumb' tput cols && echo continue
continue

• As invalid $TERM does not lead to any error (just leads to no tput output), cron tput error seems to be related to missing -T, [[ -t 0 ]] results in false. Maybe also this behaved differently before. Last ncurses-bin APT update on 2018-02-15: http://ftp.de.debian.org/debian/pool/main/n/ncurses/

• We had issues with non-interactive/invalid terminals already, thus now exit dietpi-* scripts, if empty $TERM or 'unknown' is found: https://github.com/Fourdee/DietPi/blob/testing/dietpi/func/dietpi-globals#L30-L35

• A bid inconsistent, since we might want to allow non-interactive executions (also without any terminal attached), but need to take care about calling tput respectively other commands that need a valid terminal.

Test on v6.9:

root@DietPi:/tmp# cat cron.test
dumb
tput: No value for $TERM and no -T specified
finish

• Error shows up, but script resumes.

APT update contains no cron and/or ncurses (reducing by tested unrelated packages):

• Even after apt-get dist-upgrade and reboot, cron does not stop execution after tput error:

root@DietPi:/tmp# cat /etc/cron.minutely/test
#!/bin/bash

echo "$TERM" &> /tmp/cron.test
[[ -t 0 ]] && echo 'interactive' &>> /tmp/cron.test
tput ed &>> /tmp/cron.test
echo 'finish' &>> /tmp/cron.test
root@DietPi:/tmp# cat cron.test
dumb
tput: No value for $TERM and no -T specified
finish

• dietpi-update to v6.11 master: Veeery strange, still no issue...

root@DietPi:/tmp# cat cron.test
dumb
tput: No value for $TERM and no -T specified
finish

• Adding dietpi-backup execution:

root@DietPi:/tmp# cat /etc/cron.minutely/test
#!/bin/bash

echo "$TERM" &> /tmp/cron.test
[[ -t 0 ]] && echo 'interactive' &>> /tmp/cron.test
tput ed &>> /tmp/cron.test
/DietPi/dietpi/dietpi-backup 1 &>> /tmp/cron.test
echo 'finish' &>> /tmp/cron.test
root@DietPi:/tmp# cat cron.test
dumb
tput: No value for $TERM and no -T specified
[......] Checking for (elevated) root accesstput: No value for $TERM and no -T specified
/DietPi/dietpi/func/dietpi-globals: line 266: ( 38 + 6 ) /  : syntax error: operand expected (error token is "/  ")
finish

• Script fails but cron job goes on. Test with dietpi-globals v6.9:

root@DietPi:/tmp# cat cron.test
dumb
tput: No value for $TERM and no -T specified
         Checking for (elevated) root accesstput: No value for $TERM and no -T specified
/DietPi/dietpi/func/dietpi-globals: line 308: ((: lines=(38+6)/ : syntax error: operand expected (error token is "/ ")
tput: No value for $TERM and no -T specified
[  OK  ] Root access verified.
         DietPi-Run_ntpd | systemctl restart systemd-timesyncdtput: No value for $TERM and no -T specified
/DietPi/dietpi/func/dietpi-globals: line 308: ((: lines=(38+6)/ : syntax error: operand expected (error token is "/ ")
[..    ] tput: No value for $TERM and no -T specified
[  OK  ] DietPi-Run_ntpd | systemctl restart systemd-timesyncd
[ INFO ] DietPi-Run_ntpd | NTPD: Waiting for completion of systemd-timesyncd (1/60)
[  OK  ] DietPi-Run_ntpd | NTPD: systemd-timesyncd synced
[  OK  ] NTPD: time sync | Completed
finish

- Okay, it IS related to globals changes, but since tput did and does fail, why do our scripts exit now and didn't exit on v6.9?

Related change identified: https://github.com/Fourdee/DietPi/blob/master/dietpi/func/dietpi-globals#L266
local lines=$(( (${#input_string}+6)/$(tput cols) )) leads to whole script exit, while (previous)

local lines
(( lines=(${#input_string}+6)/$(tput cols) ))

showed the error, but allowed script to go on (see above).

Fix https://github.com/Fourdee/DietPi/commit/82ac7b32d32dca9db4fdb824c7ead80174844090 works and is also the cleanest solution, as tput will be completely skipped within non-interactive shells. But to be failure safe, we should revert to previous method to declare variable first and then assign value afterwards within arithmetic environment: Commit done https://github.com/Fourdee/DietPi/commit/0f18aa4dc0af8ab910a0173dce8849d5b53c30b0

@eljotx @kmakisara
To clean up a bid I opened a new issue, please try to apply the fix mentioned there and report back, if it solves all issues: https://github.com/Fourdee/DietPi/issues/1923

I will close this issue in favour of the new one.

I could successfully ssh into my v6.13 but it would hang right after.

The "tput cols" command was outputting tput: unknown terminal "rxvt-256color", resulting in an empty value on line 328 of dietpi-login, making it hang forever after having successfully read the database upon initial boot.

Exporting TERM=xterm, an ugly hack, seemed to have fixed the login script, and can now begin install.

If this is the wrong section please be patient, for i don't really know how to use github issues; I have found this thread after hours of useless googling for an error, if you move it maybe leave a remainder about rxvt-256color errors, in case someone has my same problem.

Best regards!

@wuhei

Hi,

Thanks for the report and solution 👍

Which SSH client are you running?

@Fourdee
I was opening a separate issue for this. Identified the issue: https://github.com/Fourdee/DietPi/issues/2034

@wuhei
After logging into SSH please try: export TERM='xterm-256color' €: Ah this is basically what you already found out 😄.
If this works, you could add a script to /etc/profile.d/ that contains something like:
[[ $SSH_TTY ]] && [[ $TERM =~ 256 ]] && export TERM='xterm-256color'

• In words: If this is an SSH connection, and the SSH client passed a 256 bit color terminal, then adjust it to xterm-256color, which is supported by DietPi.

An alternative would be to install a package that contains more terminal definitions: G_AGI ncurses-term

@Fourdee OpenSSH_7.7p1, OpenSSL 1.1.0h on Arch Linux

@MichaIng thanks glad I could help by chance!

Please keep in mind I am a bad sysadmin and by following this article by Tom Ryder i have found out I had a pretty misconfigured terminal "declaration" config.

In the end what fixed the problem for me was

• exporting the correct $TERM in /root/.bashrc
• copying the correct terminfo entry from my arch install in the pi's /root/.terminfo folder

I couldn't scp the terminfo file to my pi, because scp is not installed (maybe consider installing it together with the dropbear server?), so i uploaded it to a server I own and just wget'ted it from the pi.

Glad i could help, cheers and sorry again for the mess - rock on!

@wuhei
Jep, setting $TERM within /root/.bashrc of course works as well, adding it to /etc/profile[.d/] solves it system wide for all users, which is a solution we'd go for with DietPi.

The article you linked is pretty nice, explaining everything and possibilities very well 👍. So you basically installed the required terminal type manually. The article also mentions the ncurses-term package as an option to install a wide range of terminal definitions. But to keep DietPi slim, I am not too keen to pre-install this package for such rare cases. Instead better, go with the bashrc/profile solution; It's possible as well to simply check terminal support before applying the fix on SSH login.

For testing with PuTTY client: PuTTY > Connection > Data > Terminal-type string

About installing SCP: There are several other file transfer protocols, FTP, SFTP or network drives NFS, SMB, to transfer files. I would no pre-install on, but leave choice to user how to get files from client to server. USB stick, external drive of course is also possible or simply copy&paste into SSH terminal in case. But anyway as a general solution I would go with one of the two ways mentioned above.