Monday, December 16, 2019

Quick Tips: How To Keep Ownership And File Permissions Intact When Copying Files Or Directories

How To Keep Ownership And File Permissions Intact When Copying Files Or Directories

The other day I planned to backup some data from my Ubuntu desktop system to an external USB drive. After I transferred all data to the external drive, I noticed that the owner and permissions of the files and directories on source and destination are different. But, I wanted to keep existing file attributes (such as owner, group and timestamp etc.) intact on both location. After a quick Google search and going through man pages of “cp” command, I found that we can keep ownership and file permissions intact when copying files and directories on Linux. If you ever been in a situation like this, here is a quick workaround to do it.

Keep Ownership And File Permissions Intact When Copying Files Or Directories On Linux

The cp command has an option to copy files and directories but preserves modification times, access times, and modes from the original file.
$ cp -rp ~/data /media/sk/sk_seagate/data/
Here, I am copying the contents of a folder named “data” to my external drive.
From the cp command’s man pages:
-p     same as --preserve=mode,ownership,timestamps

       --preserve[=ATTR_LIST]
              preserve the specified attributes (default: mode,ownership,timestamps), if possible
              additional attributes: context, links, xattr, all
So, if you use -p flag, it will preserve all existing file attributes such as mode, ownership and timestamps from original file.
And the “-r” flag is used here to copy directories recursively. Meaning – it will copy directories and its sub-directories and files.
Alternatively, you can use -a flag. It includes the -r flag and preserves everything, such as links, xattr, SELinux attributes etc.
$ cp -a ~/data /media/sk/sk_seagate/data/
From cp command’s man pages:
-a, --archive
              same as -dR --preserve=all
To verify if the file permissions and ownership are intact, use getfacl command on both locations i.e. source and destination.
$ getfacl ~/data
$ getfacl /media/sk/sk_seagate/data/
If you want to copy files between from the local system to a remote system in the same network, you can use “scp” command to transfer files from one system to another like below.
$ scp -rp ~/data senthil@192.168.225.22:/home/senthil/
The above command will copy the contents folder named “data” from my local system to a remote system. Here, 192168.225.22 is the IP address of my remote system and “senthil” is the user name of remote system.
Check the file permissions and ownership of the “data” directory on both systems using “getfacl” command.
First let check the file attributes of “data” directory on the local system:
$ getfacl data/
Sample output:
# file: data/
# owner: sk
# group: sk
user::rwx
group::rwx
other::r-x
Next, check check the file attributes of “data” directory on the remote system. You can directly log in to the remote system and check the fie attributes of a remote directory via ssh command like below.
$ ssh senthil@192.168.225.22 getfacl data
Sample output:
senthil@192.168.225.22's password:
# file: data
# owner: senthil
# group: senthil
user::rwx
group::rwx
other::r-x
As you may noticed in the above outputs, the owner and group information (i.e. senthil) are different in my remote system. Because, the -p flag of scp command doesn’t preserve all permissions.
From the man pages of scp command:
-p      Preserves modification times, access times, and modes from the original file.
It is clear that the -p flag will only preserve modification times, access times, and modes from the original file, but not the ownership. In such cases, you can create a common user name on both systems and try the above command to carry the same ownership and file permissions on different systems. Or simply use the “chown” command on the destination system to change the ownership.
For more details, refer man pages of cp and scp commands.
$ man cp
$ man scp

Suggested read:

Hope this helps.

Thursday, August 1, 2019

Ubuntu : "CLIPGRAB" Dedicated software for Video Downloader for Ubuntu

Dedicated software for Video Downloader for Ubuntu



Clipgrab

Clipgrab is yet another useful tool that helps you download videos from Youtube and other sites such as Vimeo and Daily Motion. With just a few clicks, you simply copy the Video Url and paste it in Clipgrab, as you define the resolution then finally hit Grab this Clip button.
To install Clipgrab, use the official PPA by the Clipgrab team.
# add-apt-repository ppa:clipgrab-team/ppa
Update the System
# apt update
Install Clipgrab
# apt install clipgrab
To launch Clipgrab run
# clipgrab
Download YouTube videos in Ubuntu

Thursday, July 4, 2019

XPEnology : DS disappears - "Recoverable" loop fix

Xpenology/ Synology : 
DS disappears - "Recoverable" loop fix

I had a problem where my baremetal (6.1.3-8) became undiscoverable on the network following a reboot and would not pick up an ip address (according to the router).  Putting in a newly created bootloader USB made the device discoverable through Synology Assistant and find.synology.com, but was listed as "recoverable".  Clicking "recover" (the only option) started an immediate reboot which caused the diskstation to become undiscoverable again.  
 
Symptoms:
1.  Diskstation disappears from network and is undiscoverable (no IP address) - possibly associated with an update.  
2.  New USB bootloader makes DS reappear but is marked as "RECOVERABLE"
3.  Hitting recover causes an immediate reboot where DS returns to symptom #1 (disappears from the network)
4.  Going to an older bootloader version shows the DS as "migratable" but refuses to install the associated .PAT due to the older version
5.  Going to a newer bootloader version shows as "recoverable", returns to symptom #1

Causes
- Not sure but likely a corrupted or mismatch in the DSM software on the volume.

Fix (following the synology data migrate process for 5.x)
1.  Remove ALL HDDs associated with the DS
2.  Prepare new USB bootloader using the established processes.  Must be the same version of your old volumes or later
3.  Plug in blank temporary HDD by itself (I used an 80GB laptop HDD so anything will do)
4.  Go through normal installation process.  Setup as per old DSM settings (can use config backup files).  Test with a restart or two
5.  Shutdown DS, unplug temp HDD, reconnect HDDs
6.  Power-on.  Find using find.synology.com or Synology Assistant.  Should be shown as "Migratable"
7.  Chose "Fresh Install, Keep my data".  Go through normal process.
8.  Setup however you like.  Success!!
 
I'm not advanced enough of a user to know if there are other methods of repairing this problem; this is the one that worked for me.


Wednesday, May 8, 2019

Ubuntu 16.04 Developer Tools Installation

Ubuntu 16.04 Developer Tools Installation

First things first !
sudo apt update
sudo apt upgrade

Standard Developer Tools

sudo apt-get install build-essential git

Virtualbox Guest Utlities for Shared Drive and Clipboard (optional)

sudo apt install dkms build-essential linux-headers-`uname -r`
sudo apt install virtualbox-guest-utils virtualbox-guest-dkms
# To access shared drive on nautilus folder
sudo addgroup <YOUR USERNAME> vboxsf

Python Development

sudo apt-get install python-dev python-pip python-virtualenv python-numpy python-matplotlib

Virtualenv Wrapper Installation

sudo pip install virtualenvwrapper
echo "source /usr/local/bin/virtualenvwrapper.sh" >> ~/.bashrc
# Create Virtualenv (env location is ~/.virtualenvs)
mkvirtualenv <env_name>
# select virtualenv
workon <env_name>

pip packages

For Machine Learning
# Create Virtualenv
mkvirtualenv ML
# Activate it
workon ML
# Install packages (if behinde proxy use pip install --proxy=<proxy_address>)
pip install scipy scikit-learn matplotlib jupyter pandas numpy tensorflow keras seaborn
For Web Development
# Create Virtualenv
mkvirtualenv web
# Activate it
workon web
# Install packages (if behinde proxy use pip install --proxy=<proxy_address>)
pip install django beautifulsoup4 requests ipython

Compiler Development

sudo apt install flex bison

Networking tools

sudo apt install libpcap-dev libnet1-dev rpcbind openssh-server nmap

Computer Vision or Image Processing with OpenCV

sudo apt install python-numpy
cd ~
# Clone latest codebase for opencv (do git checkout for specific version)
git clone https://github.com/Itseez/opencv.git
# Clone opencv contrib plugins
git clone https://github.com/Itseez/opencv_contrib.git
# install prerequisites
sudo apt-get install cmake pkg-config libjpeg8-dev libtiff-dev libjasper-dev libpng12-dev libgtk2.0-dev libavcodec-dev libavformat-dev libswscale-dev libv4l-dev libatlas-base-dev gfortran libopencv-dev build-essential checkinstall cmake pkg-config yasm libtiff5-dev libjpeg-dev libjasper-dev libavcodec-dev libavformat-dev libswscale-dev libdc1394-22-dev libxine2-dev libgstreamer0.10-dev libgstreamer-plugins-base0.10-dev libv4l-dev python-dev python-numpy libtbb-dev libqt4-dev libgtk2.0-dev libfaac-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev x264 v4l-utils ffmpeg libgphoto2-dev
# Start Build Process
cd opencv
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D INSTALL_C_EXAMPLES=ON -D INSTALL_PYTHON_EXAMPLES=ON -D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib/modules -D BUILD_EXAMPLES=ON -D WITH_QT=ON -D WITH_OPENGL=ON ..
make -j4
sudo make install
sudo ldconfig

Parallel Processing with OpenMPI

#install pre-requisites
sudo apt-get install libibnetdisc-dev
# download codebase
wget https://www.open-mpi.org/software/ompi/v1.10/downloads/openmpi-1.10.2.tar.gz
# extract
tar -xzvf openmpi-1.10.2.tar.gz
# configure and build
./configure --prefix="~/.openmpi"
make -j4
sudo make install
echo "export PATH=$PATH:$HOME/.openmpi/bin" >> ~/.bashrc
echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/.openmpi/lib/" >> ~/.bashrc

Android tools

sudo apt install android-tools-adb android-tools-fastboot

Media tools

sudo apt install flashplugin-installer vlc ffmpeg

Editor

install vim
sudo apt install vim
Install Atom Editor download
sudo dpkg -i atom-amd64.dev
Install Visual Studio Code download
sudo dpkg -i code_<version>.deb

Ubuntu Customization (Theme and Icons)

Install Gnome Tweak Tool
sudo apt install gnome-tweak-tool

# Create Theme and Icon directory
mkdir ~/.icons ~/.themes
Download Themes and Icons, Place it under respective directory and open tweak tool select your themes
Suggested Themes and
  1. Flatabulous
  2. Paper-Theme
  3. Paper-Icon-Theme

Install ZSH

sudo apt install zsh
curl -L https://github.com/robbyrussell/oh-my-zsh/raw/master/tools/install.sh | sh
chsh -s `which zsh`

Web Development in PHP with Apache2 Web Server

#Install apache2 and php
sudo apt install apache2 apache2-utils libapache2-mod-php
sudo apt install php php-dev php-mcrypt php-mysql php-mbstring php-dom
# Enable mod_rewrite for apache2
sudo a2enmod rewrite
Configure index.php for priorities add index.php sudo nano /etc/apache2/mods-available/dir.conf
<IfModule mod_dir.c>
 DirectoryIndex index.php index.html index.cgi index.pl index.xhtml index.htm
</IfModule>
Configure Mod Rewrite sudo nano /etc/apache2/sites-available/000-default.conf
<VirtualHost *:80>
 ServerAdmin webmaster@localhost
 DocumentRoot /var/www/html

    <Directory /var/www/html>
        Options FollowSymLinks MultiViews Indexes
        AllowOverride All
        Order allow,deny
        allow from all
    </Directory>

 ErrorLog ${APACHE_LOG_DIR}/error.log
 CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>
Reload Apache sudo service apache2 restart

Java Development

# install jdk
sudo apt install openjdk-8-jdk
# install maven
sudo apt install maven
Install eclipse dowload

Ruby Development

# install pre requisites
sudo apt-get install git-core curl zlib1g-dev build-essential libssl-dev libreadline-dev libyaml-dev libsqlite3-dev sqlite3 libxml2-dev libxslt1-dev libcurl4-openssl-dev python-software-properties libffi-dev

# clone rbenv
git clone https://github.com/rbenv/rbenv.git ~/.rbenv
git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build
git clone https://github.com/rbenv/rbenv-gem-rehash.git ~/.rbenv/plugins/rbenv-gem-rehash

# add configuration to bashrc
echo "export PATH=$PATH:$HOME/.rbenv/bin:$HOME/.rbenv/shims" >> ~/.bashrc
echo "eval $(rbenv init -)" >> ~/.bashrc
echo "export PATH=$HOME/.rbenv/plugins/ruby-build/bin:$PATH" >> ~/.bashrc
# install and set specific ruby version
rbenv install 2.3.0
rbenv global 2.3.0
# install some gems
gem install bundler jekyll mysql2 rails jekyll-pagedown

Database

# install nosql database
sudo apt install mongodb-server
# install sqlite3 for smaller db
sudo apt install libsqlite3-dev sqlite3
# mysql community version installation
sudo apt install mysql-server libmysqlclient-dev

NodeJs Development

curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo npm install -g grunt-cli yarn @angular/cli

ASP.NET Development

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
echo "deb http://download.mono-project.com/repo/debian wheezy main" | sudo tee /etc/apt/sources.list.d/mono-xamarin.list
sudo apt update
# install mono libraries
sudo apt install mono-devel mono-complete referenceassemblies-pcl ca-certificates-mono mono-xsp4
# install mono ide
sudo apt install monodevelop-nunit monodevelop-versioncontrol monodevelop-database

CLoud Storage (Dropbox)

cd ~ && wget -O - "https://www.dropbox.com/download?plat=lnx.x86_64" | tar xzf -
# to start dropbox
sh ~/.dropbox-dist/dropboxd