View on GitHub

Settings_Linux

Useful bash scripts and settings for managing a Linux server

Upgrade of fishfish

4/4/2018~4/8/2018

By Adam Lu


  1. Backed up fishfish boot drive
  sudo dd if=/dev/sde1 | bzip2 > /media/barrettlabX/fishfish_sde1_backup_20180404.bz2
  1. Backed up fishfish root directory (didn’t finish)
  sudo dd if=/dev/sde2 | bzip2 > /media/barrettlabX/fishfish_sde2_backup_20180405.bz2
+ Used `df -Th` to probe the size of `/dev/sde2`:  
Filesystem Type Size Used Avail Use% Mounted On
/dev/sde2 ext4 2.7T 661G 1.9T 26% /
+ Used `sudo kill -USR1 $(pidof dd)` to monitor progress:

  >  **338 GB** copied in **62545.6 s**, **5.4 MB/s**

+ This would have taken at least 18 more hours (if not more) to complete

+ Based on the discussion in [this thread](https://unix.stackexchange.com/questions/140991/is-it-safe-to-take-the-drive-image-of-the-current-working-drive), decided to terminate `dd` prematurely. Renamed `fishfish_sde2_backup_20180405.bz2` as `fishfish_sde2_backup_20180405_incomplete.bz2`
  1. Dealt with custom bash scripts:

    • Transferred all *.sh files under /bin to /usr/bin
    • In Webmin, Removed all instances of /bin/ from the cron job commands.
  2. In Webmin, backed up configuration files:

    • Modules to backup: Selected all
    • Backup destination: /media/adamX/Settings_Linux/backup_configuration_fishfish.tar.gz
    • Included in backup: Clicked on Server configuration files (When I clicked on Webmin module configuration files, it was automatically unchecked after backup)
    • Also did this for chalkboard with backup destination: /home/adam/Settings_Linux/backup_configuration_chalkboard.tar.gz
  3. Investigated the disk usage in the /home directory

    • Command: sudo du -sh /home/* > /media/adamX/Settings_Linux/home_disk_usage_20180405.log

    • Also tried using ncdu (NCurses Disk Usage):

    • This lists the directories taking up most space in descending order

    • Screenshot:

      home_disk_usage_ncdu_20180405.png

  4. Dealt with the various versions of barrettlab

    • Renamed /media/barrettlabX/barrettlab/barrettlab/media/barrettlabX/barrettlab/barrettlab_file_that_may_be_nonsense

    • Renamed /media/barrettlabX/barrettlab/media/barrettlabX/barrettlab_fishfish

    • Renamed /usr/bin/barretlabBackup.sh/usr/bin/barrettlabBackup.sh and updated the corresponding cron job in Webmin

    • Modified /usr/bin/barrettlabBackup.sh:

    Changed rsync -r -t -v --progress --update mark@128.143.16.133:/home/barrettlab/ /home/barrettlab/. to rsync -avhu --progress mark@128.143.16.133:/home/barrettlab/ /media/barrettlabX/barrettlab_chalkboard/.

    • On chalkboard:

      • Modified /bin/comprehensiveBackup3.sh:

        Changed rsync -r -t -v --progress --update mark@128.143.16.164:/media/barrettlabX /media/barrettlab_liseBU/barrettlab/. to rsync -avhu --progress mark@128.143.16.164:/media/barrettlabX/barrettlab_fishfish /media/barrettlab_liseBU

      • Removed with subdirectories /media/barrettlab_liseBU/barrettlab/barrettlabX

      • Moved /media/barrettlab_liseBU/barrettlab/barrettlab to /media/barrettlab_liseBU/barrettlab_fishfish and removed /media/barrettlab_liseBU/barrettlab

      • Moved files with the command sudo mv /home/barrettlab/* /media/barrettlabX/barrettlab_chalkboard/

  5. Cleaned up other home directories on fishfish with the following commands in a root terminal (sudo bash):

    • Moved /home/user/public_html/media/driveX/public_html_user
    • Moved data to /media/driveX/:
      • /home/becca/*/media/shareX/former_members/Becca
      • /home/ashley/*/media/shareX/Ashley/from_home_ashley
      • /home/courtney/*/media/shareX/Courtney
      • /home/jessica/*/media/shareX/former_members/Jessica
      • /home/katie/*/media/katieX/from_home_katie
      • /home/lise/*/media/liseX/from_home_lise
      • /home/lise/Desktop/ImageJ-linux64/media/shareX/Software
      • /home/lise2/*/media/liseX/from_home_lise2
      • /home/mark/*/media/markX/from_home_mark
    • Identified large files in /home/Matlab/:
      • Command:

        find /home/Matlab -type f -size +10M
        
      • The largest file is /home/Matlab/Marks_Functions/scott_eeg/eeg_data/01_10_2014/video/2014-01-10_09h_13_12.66.mpg

    • Note: This file is replicated 8 times in /media/shareX/:

      • Command:

        find /media/shareX/ -type f -name "2014-01-10_09h_13_12.66.mpg"
        
      • Output:

        /media/shareX/malcom_archive/home/share/Katie/Kir6.2_Project/DBA_Experiments/EEG/01_10_2014/video/2014-01-10_09h_13_12.66.mpg
        /media/shareX/malcom_archive/home/share/Katie/Kir6.2_Project/paper/Figures/Figure_1/EEG/01_10_2014/video/2014-01-10_09h_13_12.66.mpg
        /media/shareX/malcom_archive/home/share/EEG/EEG/01_10_2014/video/2014-01-10_09h_13_12.66.mpg
        /media/shareX/Lab_Papers/Glucose_Paper/Data_and_Figures/DBA_Fast_and_Seizures/data/01_10_2014/video/2014-01-10_09h_13_12.66.mpg
        /media/shareX/Lab_Papers/Glucose_Paper/Old_figures_and_code/Figure_1/EEG/01_10_2014/video/2014-01-10_09h_13_12.66.mpg
        /media/shareX/share/Katie/Glucose_Paper/Data/DBA_Fast_and_Seizures/data/01_10_2014/video/2014-01-10_09h_13_12.66.mpg
        /media/shareX/share/Katie/Glucose_Paper/Old_figures_and_code/Figure_1/EEG/01_10_2014/video/2014-01-10_09h_13_12.66.mpg
        /media/shareX/temp_backup_before_new_server_is_built_01142015/Matlab/Matlab/Marks_Functions/scott_eeg/eeg_data/01_10_2014/video/2014-01-10_09h_13_12.66.mpg

    • Moved data:
      • /home/Matlab/Marks_Functions/scott_eeg//media/markX/from_home_Matlab_Marks_Functions
      • ``/home/Matlab/Marks_Functions/Nik_Analysis/*.mat → /media/markX/from_home_Matlab_Marks_Functions/Nik_Analysis_matfiles`
      • /home/Matlab/Marks_Functions/Nik_Analysis/pclamp_data/media/markX/from_home_Matlab_Marks_Functions/Nik_Analysis_pclamp_data
      • /home/Matlab/Marks_Functions/oscRUNCH*.tar.gz and /home/Matlab/Marks_Functions/--*/media/markX/from_home_Matlab_Marks_Functions/
      • /home/Matlab/FromSami/*.png and /home/Matlab/FromSami/*.fig/media/shareX/Sami/
      • /home/Matlab/Downloaded_Functions/*.mat/media/shareX/Matlab/Downloaded_Functions_matfiles
      • /home/Matlab/Downloaded_Functions/*.zip/media/shareX/Matlab/Downloaded_Functions_zipfiles
    • Renamed folders:
      • /home/Matlab/FromSami//home/Matlab/Samis_Functions/
      • /home/Matlab/Downloaded_Functions/eeglab13_5_4b/home/Matlab/eeglab13_5_4b/
  6. Made sure data were backed up on chalkboard correctly

    • Cleaned up BU drives (except /media/shareXBU/) with the following commands under /media/driveXBU in a root terminal (sudo bash):

      1. Moved all files and non-duplicate directories up one level:

        mv -f driveX/* .
        mv -f driveX/.* .
        
      2. Removed duplicate directories in the top level:

        ls driveX | xargs -I @ rm -R @
        
      3. Moved all previously duplicate directories up one level

        mv -f driveX/* .
        
      4. Removed driveX

        rmdir driveX
        
    • Under /media/shareXBU, moved Matlab to home/Matlab. The entire /home/ directory on fishfish will be backed up instead.

    • Modified /bin/comprehensiveBackup3.sh:

      Changed rsync -r -t -v --progress --update mark@128.143.16.164:/media/markX /media/markXBU/. to rsync -avhu --progress mark@128.143.16.164:/media/markX/ /media/markXBU

      Changed rsync -r -t -v --progress --update mark@128.143.16.164:/media/katieX /media/katieXBU/. to rsync -avhu --progress mark@128.143.16.164:/media/katieX/ /media/katieXBU

      Changed rsync -r -t -v --progress --update mark@128.143.16.164:/media/liseX /media/barrettlab_liseBU/lise . to rsync -avhu --progress mark@128.143.16.164:/media/liseX/ /media/barrettlab_liseBU/lise

      Changed rsync -r -t -v --progress --update mark@128.143.16.164:/media/adamX /media/adamXBU/. to rsync -avhu --progress mark@128.143.16.164:/media/adamX/ /media/adamXBU

      Changed rsync -r -t -v --progress --update mark@128.143.16.164:/media/shareX /media/shareXBU/. to rsync -avhu --progress mark@128.143.16.164:/media/shareX/ /media/shareXBU/shareX

      Changed rsync -r -t -v --progress --update mark@128.143.16.164:/media/EEG_2016- /media/EEG_2016-BU/. to rsync -avhu --progress mark@128.143.16.164:/media/EEG_2016-/ /media/EEG_2016-BU

      Changed rsync -r -t -v --progress --update mark@128.143.16.164:/media/barrettlabX /media/barrettlab_liseBU/barrettlab/. to rsync -avhu --progress mark@128.143.16.164:/media/barrettlabX/barrettlab_fishfish /media/barrettlab_liseBU

      Changed rsync -r -t -v --progress --update mark@128.143.16.164:/home/Matlab /media/shareXBU/. to rsync -avhu --progress --exclude ".*/" --exclude ".*" mark@128.143.16.164:/home/ /media/shareXBU/home

      Added rsync -avhu --progress mark@128.143.16.164:/media/peterX/ /media/peterXBU (/media/peterXBU/ has only one directory chalkboard_private/ – the rest of /media/peterX/ was never backed up!)

    • Under root privileges (sudo bash), performed each of the lines in /bin/comprehensiveBackup3.sh but with adam instead of mark

    • Changed permissions (the cron jobs are all executed with the user mark)

      • Under each /media/driveXBU/ in chalkboard:

        chown -R mark .
        chgrp -R LabFolks .
        
      • Changed permissions for the directories /home/, /media/adamX/, /media/shareX/ in fishfish with the command:

        chmod -R 775 .
        
    • Observations/Questions:

      • /media/katieXBU/ contains many files from /media/markX/ – clean up?
      • /media/shareXBU/ also has a barrettlab/ and a baylisslab/ – what versions are these?
  7. Saved ufw firewall allowed IP addresses:
    • fishfish:

      sudo ufw status numbered > /media/adamX/Settings_Linux/ufw_IPs_fishfish_20180405.log
      
    • chalkboard:

      sudo ufw status numbered > /home/adam/Settings_Linux/ufw_IPs_chalkboard_20180405.log 
      
  8. Upgraded fishfish from Ubuntu 12.04 LTS to Ubuntu 14.04 LTS:
  1. Upgraded from Ubuntu 14.04 LTS to Ubuntu 16.04 LTS:

    • With a shell logged in as root (sudo bash), tried these steps:

      • sudo apt-get update had output:

        Fetched 5,903 kB in 5s (1,140 kB/s)

      • sudo apt-get upgradehad output:

        The following package was automatically installed and is no longer required:

        libpostproc52

        Use ‘apt-get autoremove’ to remove it.

        0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

      • sudo apt-get dist-upgrade had output:

        The following package was automatically installed and is no longer required:

        libpostproc52

        Use ‘apt-get autoremove’ to remove it.

        0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

      • sudo apt-get install update-manager-core had output:

        update-manager-core is already the newest version.

        update-manager-core set to manually installed.

    • Performed upgrade (do-release-upgrade) with initial message:

      16 installed packages are no longer supported by Canonical. You can still get support from the community.

      70 packages are going to be removed. 680 new packages are going to be installed. 2053 packages are going to be upgraded.

      You have to download a total of 2,409 M. This download will take about 5 hours with a 1Mbit DSL connection and about 3 days 21 hours with a 56k modem.

      Fetching and installing the upgrade can take several hours. Once the download has finished, the process cannot be canceled.

    • Prompted to choose default display manager (gdm or gmd3 or lightdm):

      As you might be aware the Ubuntu desktop has decided to switch to using GNOME and this means we will be using GDM instead of LightDM Unfortunately this means there is likely to be a reduction in development effort from Canonical.

      Ultimately we think that LightDM is the right cross-desktop solution for display management, it’s just not something we can make use of in Ubuntu desktop at this time. Who knows how the future will pan out :)

    • During configuration of libssl1.0.0:amd64

      • Prompted with this message :

      There are services installed on your system which need to be restarted when certain libraries, such as libpam, libc, and libssl, are upgraded. Since these restarts may cause interruptions of service for the system, you will normally be prompted on each upgrade for the list of services you wish to restart. You can choose this option to avoid being prompted; instead, all necessary restarts will be done for you automatically so you can avoid being asked questions on each library upgrade.

      Restart services during package upgrades without asking?

      • Decided Yes
    • There was a conflict for /etc/cups/cups-browsed.conf:

      • The old version seems to have been installed with the last upgrade (2018-04-06 14:45:20)
      • Chose to install the package maintainer’s version.
    • Process was halted with the following messages:

      Processing was halted because there were too many errors.

      Upgrade complete

      The upgrade has completed but there were errors during the upgrade process.

    • Verified that the Ubuntu version is now 16.04.4 LTS with the command: lsb_release -a

    • Tried these commands:

      • sudo apt-get update had output:

        Fetched 306 kB in 0s (462 kB/s)

      • sudo apt-get upgradehad initial output:

        848 not fully installed or removed.

      • Continued installation

    • There was a conflict for /etc/apt/apt.conf.d/50unattended-upgrades.ucftmp:

      • The new version had one added line
      • Chose to install the package maintainer’s version.
    • Final message before stopping:

      Rebuilding /usr/share/applications/bamf-2.index…

    • Tried these commands again:

      • sudo apt-get update had output:

        Fetched 13.8 MB in 2s (6,850 kB/s)

      • sudo apt-get upgrade had output:

        The following packages were automatically installed and are no longer required:

        Use ‘sudo apt autoremove’ to remove them.

        The following packages have been kept back:

        libparse-debianchangelog-perl

        0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.

      • sudo apt autoremove

        • Initial output:

        The following packages will be REMOVED:

        0 upgraded, 0 newly installed, 333 to remove and 1 not upgraded.

        After this operation, 653 MB disk space will be freed.

        • Finished without error
      • sudo apt-get update had output:

        Reading package lists… Done

      • sudo apt-get upgrade had output:

        The following packages have been kept back:

        libparse-debianchangelog-perl

        0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.

      • sudo apt-get dist-upgrade:

        • Initial output:

        The following NEW packages will be installed:

        libcgi-fast-perl libcgi-pm-perl libfcgi-perl

        The following packages will be upgraded:

        libparse-debianchangelog-perl

        1 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.

        Need to get 278 kB of archives.

        After this operation, 604 kB of additional disk space will be used.

        • Completed without error
    • sudo do-release-upgrade had output:

      Checking for a new Ubuntu release

      No new release found.

    • sudo reboot stopped in emergency mode with this error:

      [ 2.133899] ERST: Can not request [mem 0xcfe9e000-0xcfe9ffff] for ERST.

      /dev/sde2: clean, 4790759/178954240 files, 42625134/715793664 blocks

    • Opened the system log:

      • Command:

        journalctl -xb
        
      • Showed these errors in red:

        ERST: Can not request [mem 0xcfe9e000-0xcfe9ffff] for ERST.

        Failed to mount /media/adamX

        fishfish systemd[1]: Failed to start Raise network interfaces.

        Failed to mount /media/adamX

        fishfish systemd[1]: Failed to start Raise network interfaces.

      • According to this thread, typed the command:

        ln -s /dev/null /etc/udev/rules.d/80-net-setup-link.rules
        
    • sudo reboot still stopped in emergency mode with the same errors

      • The system log still showed the same errors

      • Command:

        systemctl status networking.service
        
      • Output:

        systemd[1]: Starting Raise network interfaces…

        ifup[1387]: RTNETLINK answers: File exists

        ifup[1387]: Failed to bring up eth0.

        networking.service: Main process exited, code=exited, status=1/FAILURE

    • Performed a file system check:

      • Command:

        fsck.ext4 /dev/sde2
        
      • Failed because the drive was mounted

    • Removed /tmp_old/ recursively

    • Commented out mounting /media/adamX/ in /etc/fstab

    • systemctl reboot still stopped in emergency mode with these errors:

      [ 2.134955] ERST: Can not request [mem 0xcfe9e000-0xcfe9ffff] for ERST.

      /dev/sda2: clean, 4141648/178954240 files, 41977662/715793664 blocks

    • The system log now shows these errors:

      ERST: Can not request [mem 0xcfe9e000-0xcfe9ffff] for ERST.

      Failed to mount /media/EEG_2012-15

      EXT4-fs (sdh): VFS: Can’t find ext4 filesystem

      fishfish systemd[1]: Failed to start Raise network interfaces.

      Timed out wating for device dev-sdi1.device

      Timed out waiting for device dev-sdj.device

    • Commented out mounting all drives in /etc/fstab

    • systemctl reboot successfully started Ubuntu, but upon login, created a flickering desktop with no toolbar

    • Changed the default display manager

      sudo dpkg-reconfigure lightdm
      
      • Checked with cat /etc/X11/default-display-manager:

      /usr/sbin/lightdm

      • Output:

      insserv: warning: script ‘vncserver’ missing LSB tags and overrides initctl: Unable to connect to Upstart: Failed to connect to socket /com/ubuntu/upstart: Connection refused insserv: warning: script ‘binfmt-support’ missing LSB tags and overrides insserv: Default-Start undefined, assuming empty start runlevel(s) for script binfmt-support' insserv: Default-Stop undefined, assuming empty stop runlevel(s) for script binfmt-support’

      • After systemctl reboot, same flickering screen as before …
    • Tried installing nvidia-current:

      • Command:

        sudo apt-get install --yes nvidia-current
        sudo nvidia-xconfig
        
      • The installation was successful, but a warning occurred for the configuration:

        WARNING: Unable to locate/open X configuration file.

        Package xorg-server was not found in the pkg-config search path. Perhaps you should add the directory containing `xorg-server.pc’ to the PKG_CONFIG_PATH environment variable No package ‘xorg-server’ found New X configuration file written to ‘/etc/X11/xorg.conf’

      • After systemctl reboot, same flickering screen as before …

    • Modified /etc/fstab (uncommented the other drives):

      • Before:

        /dev/sdb      /media/markX    ext4    defaults        0       0
        /dev/sdc      /media/katieX   ext4    defaults        0       0
        /dev/sdd      /media/peterX   ext4    defaults        0       0
        /dev/sde      /media/adamX    ext4    defaults        0       0
        
      • After:

        /dev/sda        /media/markX    ext4    defaults        0       0
        /dev/sdb        /media/katieX   ext4    defaults        0       0
        /dev/sdc        /media/peterX   ext4    defaults        0       0
        /dev/sdd        /media/adamX    ext4    defaults        0       0
        
      • After systemctl reboot, same flickering screen as before

      • The flickering screen went away by choosing GNOME 3 or XFCE instead of Ubuntu (Default) when logging in. However, GNOME 3 was very laggy, so decided on XFCE.

  2. Other things set up after upgrade:

    • Webmin was upgraded to version 1.881

    • Cron jobs:

      • Resumed barrettlabBackup.sh on fishfish and comprehensiveBackup3.sh on chalkboard
    • Samba Windows File Sharing:

      • Re-established the shared locations shareX, MatlabFishFish, etc.
      • Changed file settings so that:
      1. All the shared folders should be readable to anyone in the LabFolks group
      2. The locations shareX, barrettlabX, MatlabFishFish are readable/writable to anyone in LabFolks.
      3. Grad_Books is writable to grad student accounts only
      4. Only mark and the user can write in the user-specific driveX.
    • MATLAB R2017b:

      • Error upon starting:

        /usr/local/MATLAB/R2017b/bin/glnxa64/../../sys/os/glnxa64/libstdc++.so.6: version `GLIBCXX_3.4.21’ not found (required by /usr/lib/x86_64-linux-gnu/libproxy.so.1) Failed to load module: /usr/lib/x86_64-linux-gnu/gio/modules/libgiolibproxy.so

      • Based on this thread, applied these commands to remove the “libstdc++.so.6” library file shipped with MATLAB R2017b, so that MATLAB will use the system version of that library:

        cd  /usr/local/MATLAB/R2017b/sys/os/glnxa64
        mkdir exclude
        mv libstdc++.so.6* exclude/
        
    • X2Go:

      • Tried to install X2Go Server again (consulted this thread) with these commands:

        sudo apt-add-repository ppa:x2go/stable
        sudo apt-get update
        sudo apt-get install x2goserver x2goserver-xsession
        sudo apt-get install xfce4
        
      • Result: Was able to login (no more authentication error), but the screen was all black. The log file had these warnings:

        Warning: Unrecognized session type ‘unix-kde-depth_32’. Assuming agent session.

        Warning: Failed to read data from the X auth command.

        Warning: Generated a fake cookie for X authentication.

      • In the X2Go Client, changed the session type to XFCE, then it worked! This means that the GNOME desktop environment might just need to be updated

      • Installed the GNOME desktop environment with these commands:

        sudo apt install ubuntu-gnome-desktop
        
      • The GNOME desktop still doesn’t load. Based on this thread, perhaps gdm3 is a better choice for the display manager if we were to use GNOME, so I applied the command:

        sudo dpkg-reconfigure gdm3
        
      • However, the classic GNOME desktop is no longer continued in GNOME 3, but is instead being actively developed in MATE, therefore, installed the Ubuntu MATE desktop with these commands:

        # Make sure package list is up-to-date
        sudo apt-get update
        # Install the base packages required for a minimal MATE desktop
        sudo apt-get install mate-desktop-environment-core
        # Install Ubuntu MATE
        sudo apt-get install ubuntu-mate-core ubuntu-mate-desktop
        
      • If we are not using GNOME anymore, might as well switch back to lightdm:

        sudo dpkg-reconfigure lightdm
        
      • After reading all possible options, decided to also install LXDE (Lightweight X11 Desktop Environment), which needs an extra step for X2Go:

        sudo apt-get install lxde
        sudo apt-get install x2golxdebindings
        
      • After reboot, only XFCE and MATE seems to be working well through X2Go. Caveat: upon resizing, screen started flickering again…

    • Attempted to fix VNC server:

      • Removed these lines from /etc/environment:

        QT_XKB_CONFIG_ROOT="/usr/share/X11/xkb"
        
      • Same problems as before

  3. To do soon? Upgrade to Ubuntu 18.04 LTS when it’s released.

To place in Common_Unix_Commands.txt:

cat /etc/X11/default-display-manager    # Check which display manager is used
lsb_release -a                          # Check Ubuntu version
fsck.ext4 /dev/sde2                     # File system check