Tuesday, November 24, 2015

[Useful Commands]: Useful commands for Linux normal users and administrators

Useful commands for Linux normal users and administrators


  •  For Linux/Unix administrators:

1- Compare a remote dir with a local dir
 $ diff -y <(ssh user@host find /boot|sort) <(find /boot|sort)
2- Trace and view network traffic
tcpdump -s 0 -w - port <port> | tcpdump -r - -A
3- Check if a machine is online
ping -c 1 -q MACHINE_IP_OR_NAME >/dev/null 2>&1 && echo ONLINE || echo OFFLINE
PING
parameters
c 1 limits to 1 pinging attempt
q makes the command quiet (or silent mode)
/dev/null 2>&1 is to remove the display
&& echo ONLINE is executed if previous command is successful (return value 0)
|| echo OFFLINE is executed otherwise (return value of 1 if unreachable or 2 if you’re offline yourself).
I personally use this command as an alias with a predefined machine name but there are at least 2 improvements that may be done.
Asking for the machine name or IP
Escaping the output so that it displays ONLINE in green and OFFLINE in red (for instance).
4-  Reverse SSH
ssh -f -N -R 8888:localhost:22 user@somedomain.org
Reverse SSH
this command from the source server and this follow in the destination server:
 ssh user@localhost -p 8888
5- Show mounted volumes in a table (Thanks to Shawn):
mount | column -t
6- The fastest way to kill tcp or udp port is to run the following command (Thanks to Sriram):
fuser -k PORT-NUMBER-HERE/tcp
fuser -k PORT-NUMBER-HERE/udp
7- edit a file on a remote server using vim from your Linux / Unix / OS X desktop:
vim scp://sriram@server1/etc/http/httpd.conf
  • Commands for Both Normal users and Linux admins

8- Grep your command history (Thanks to Shawn)
history | grep /pattern/
9- Extract audio from Flash video (*.flv) as mp3 file
 ffmpeg -i video.flv -vn -ar 44100 -ac 2 -ab 192k -f mp3 audio.mp3
10- copy hybrid iso images to USB key for booting from it, progress bar and remaining time are displayed while copying
time (pv file.iso | dd bs=1M oflag=sync of=/dev/sdX 2>/dev/null)
11- what model of computer I’m using?
sudo dmidecode | grep Product
In this post we will see 10 commands that can be useful for any Linux user.
1 Find corrupted JPG image files
This command line allow you to find all corrupted JPG files in current directory and its subdirectories and displays any error or warning found:
$ find . -name "*jpg" -exec jpeginfo -c {} \; | grep -E "WARNING|ERROR"
2 Get all possible problems from any log files
Using the grep command, retrieve all lines from any log files in /var/log/ that have one of the problem states:
$ grep -2 -iIr "err\|warn\|fail\|crit" /var/log/*
3 Combine mkdir & cd into single command
This command allow you to create and cd the new directory in a single command:
mkdir-cd
 4 Measure, explain and minimize a computer’s electrical power consumption
Run this command as root to get enough stats. It works on AMD and Intel machines including desktops. If you run on a laptop it’ll give you suggestions on extending your battery life.
You’ll need to install PowerTOP if you don’t have, via ‘apt-get install powertop‘ etc.
To grep the output use:
$ sudo powertop -d | grep ...
Powertop
5 Convert lowercase letters to upper case and vice versa
The following command is used to convert the letters from lower case to upper case and vice versa. It will be used mainly when creating scripts:
$ echo unixmen | tr '[a-z]' '[A-Z]'
UNIXMEN
$ echo UNIXMEN | tr '[A-Z]' '[a-z]'
unixmen
6 Record your desktop using ffmpeg command
Using the following command we can record our desktop and save as a video file:
$ ffmpeg -f alsa -ac 2 -i hw:0,0 -f x11grab -s $(xwininfo -root | grep 'geometry' | awk '{print $2;}') -r 25 -i :0.0 -vcodec mpeg2video -ar 44100 -s wvga -y -sameq desktop.mpg
The above command will record all your desktop activities and store it in your current working folder.
7 Execute a command at a particular time
It’s similar to cronjob command. It will execute the users command at a given time:
$ echo "cat > sk.txt" | at 15.04
warning: commands will be executed using /bin/sh
job 1 at Wed May 29 15:04:00 2013
The above command will create a new file called sk.txt at 3:04pm.
8 Display top ten running processes sorted by memory usage
$ ps aux | sort -nk +4 | tail
The above command will display the top ten processes which are sorted by their memory usage.
9 Check your system type whether it is 32 or 64bit
The below command will display if the system is 32 or 64 bit. Run the following command in your terminal and verify your system version:
$ getconf LONG_BIT
64
10 Display current Internet usage of applications
The below command will show you which applications are using the internet at present:
$ lsof -P -i -n

Friday, November 13, 2015

[FreeNAS]: Hardware Compatibility and naming Conventions

Hardware Compatibility and naming Conventions

 What is the hardware naming convention under FreeNAS(FreeBSD)?


ad0 is the first hard-drive, the Operatin System drive. ad1,ad2,ad3 .... ad999 are the second et .......storage drives.
ad0s1 is the first partition on the first drive, and ad5s4 is the fourth partition of the sixth drive. 
fd0 is the first flopy drive.
acd0 is the first ATAPI (IDE) CDROM drive.
cd0 is the first SCSI CDROM drive.
da0 is the first SCSI direct access disk-drive(can be USB drive).
da0s1 is the first partition on the first SCSI direct access disk-drive, and da5s4 is the fourth partition of the sixth SCSI direct access disk-drive. 

Warning: Drive names may change as drives are added or removed.

---------------------------------------------------------------------------------------------------------------------------------------

What is the best hardware solution for your own FreeNAS server?


Before you do the math:
Think about your data storage requirements
► How important it is to you (personal or business data)
► Can it be easily replaced or is it replicated elsewhere?
► How often you access/change the information
► What is the expected growth rate of data (will I max out the storage media and how do I expand it?)
The Performance
  • How many users access the information
  • What your network infrastructure is (the old 10/100/1000 debate)
  • Cables,
  • Switches (and hubs),
  • Routers
  • (some have multiport routers with wireless and are talking about GigE (why? wirless does not do GigE and the switch ports will more than likely be 10/100)
  • Accessability and Security
  • Will you be accessing the data locally, or via a WAN?
            and finally
The budget
Do I have any/some/no existing hardware that can be used to build my FreeNAS platform?
How much money have I got to spend?
 
Note: by draggaj


Hard Drive
Replacement: Hard drives are the most important parts of data storage devices.If the drive fails there are no replacements 
Performance: Size, drive speed, transfer speed, buffer memory, brand and model are just some characteristics on witch depend your system performance.
Failure: If your system breakdown you can replace the failed parts, reinstall FreeNAS or move the hard drive to a different system to recover your data. If your hard drive die, the information on the drive is lost forever.
Size: You can use hard drives up to 2TB under FreeNAS(FreeBSD ), but you must keep in mind that you need a hard drive supported buy BIOS to boot FreeNAS from hard drive.
Motherboard
Replacement: Motherboards can be replaced or upgraded without any data lost. 
Performance: 
    P2 motherboards
    P3 motherboards are the good choice, old but not too old.
    P4 motherboards

Processor
Replacement: Processors can be replaced or upgraded without any data lost. L1 and L2 cache size of the processor can make a difference.
Performance:
    P2
  • P3 it’s a good choice, for the server reliability and SoftwareRAID.
    P4
    Multi-Processors
RAM
Replacement: memory can be replaced or upgraded without any data lost.
    128MB:
  • 256MB: can be a good decision for small system < 1 TB.
    512+MB :
Network Cards
Replacement: Network cards can be replaced or upgraded without any data lost. 
    10Mb
  • 100Mb NIC’s, cheap cables and switches are the best solution for the money.
    1Gig
                What is the point of having one single gigabit connection to one single computer on your LAN?
                Full upgrade to gigabit include industrial grade cables (home made cables are too noisy), gigabit switches, gigabit routers, gigabit NIC’s, in one word the whole network.
                Once you upgrade everything, you wrongfully believe that your network will work at Gigabit speed? Wrong!
                If your workstation access the internet you must setup the maximum MTU to 1500, like that you can’t take advantage of higher “jumbo frames” MTU 9000. If your server use MTU 9000 and your workstations use MTU 1500,  packet fragmentation will slow down your network.
                What about the hard drives transfer speed? 

If you don't have a need for more speed than 100BaseT gives you, there's no reason to spend bigger bucks for gigabit upgrade.
Here is the math calculation:

1 x 1 x 1 x 0 x 1 x 1 x 1 x 1 = 0
For any zero in your formula the network upgrade results will be zero.


[FreeNAS acces freeNAS console remotely]: Can I acces freeNAS console remotely over the network?

Can I access freeNAS console remotely over the network?


You can access the console remotely using Putty or .. ssh client on your *Nix system.

1. On FreeNAS   WebGUI :Services: SSH enable SSHd service
  1. Enable SSH service
  2. Set the port , default port is 22 however you should consider using a different port between 10000 and 65535, see http://www.iana.org/assignments/port-numbers.
  3. For the beginning allow 'root' login if you are behind a router/firewall but consider to disable 'root' login in the future.
  4. Enable keyboard-interactive authentications, make sure that you have a complex admin/root password .
  5. Not cover by this howto.
  6. Enable compression. For internet transfer only.
  7. Server Private key. Not cover by this howto.
  8. When you finish click on "Save and Restart" to apply changes.
  9. If everything is OK you should receive the message that "changes have been applied successfully"


2. Download PuTTy (ssh client)  from : http://the.earth.li/~sgtatham/putty/latest/x86/putty.exe


3. Run PuTTy and set the connection parameters and click Open.
  1. Enter FreeNAS IP or FreeNAS host name, on this example we use the default IP 192.168.1.250
  2. FreeNAS SSHd port, on this example we use the default port 22
  3. Select SSH protocol
  4. You can save the session now or later for future use. I chose to save the session under "freenas" name.
  5. If you chose the save the session click "Save"
  6. Now to open a remote connection  click  "Open"
  7. Latter you can get back and set advanced connection option, please read the PuTTy  manual. Not cover by this howto.



4.  First time when you connect to a remote server you will see a similar pop-up box. If you accept to connect to the server click Yes. The server key will be cached for future use.




5. On this example we use the user 'root' (the super user or Administrator).




6. The password is the FreeNAS admin password ( the WebGUI password).




7. If everything is OK you should get the shell prompt.


[FeeeNAS]: Can I mount a FreeNAS server(slave) to a second FreeNAS server(master)?

Can I mount a FreeNAS server(slave) to a second FreeNAS server(master)?



Yes.
On the slave FreeNAS (10.10.10.2) you have to:
a. enable the NFS server.
b. map all user to root: NO
c. authorised network: 10.10.10.1 & netmask /1. 

On your master FreeNAS(10.10.10.1) box from WebGUI -> Diagnostics -> Commands:
a. mkdir /mnt/MyShare/slave
b. mount -t nfs 10.10.10.2:/mnt/YourShare /mnt/MyShare/slave
c. ls /mnt/MyShare/slave

Warning: FreeNAS startup script don't include the support of mounting NFS share at boot time.

[FreeNAS Disable "Opportunistic Locking".]: Disable "Opportunistic Locking".

Disable "Opportunistic Locking".


smbd[xxx]: read_data: read failure for 4 bytes to client XXX.XXX.XXX.XXX Error = Connection reset by peer

Did you see this error message in your log's?


What Operating Systems are Affected?

All server computers running Windows operating systems in the NT family, Windows NT, Windows 2000, Windows XP and Windows 2003 Server.

All workstation computers running Windows operating systems in the 9X family , this includes Windows 98 and Windows ME.

Link : 
Solutions:

1. Server side(FreeNAS )
The safe way will be the server side(FreeNAS), however not confirmed for the moment.
On WebGUI : Services: CIFS /SMB : Settings > “Auxiliary parameters” you can add:
Dave suggested on this thread https://sourceforge.net/forum/message.php?msg_id=4849132  , that setting only  oplocks = no fix the problem for him (clients Win XP and 2003).

level2 oplocks = no 
kernel oplocks = no

Note:
  • Changing the configuration on the server side will affect all clients connected to that specific server(CIFS/SMB only).
  • No restart required.
  • Safer that editing window registry.

2. Client side(Windows only).

Important This article contains information about how to modify the registry. Make sure that you back up the registry before you modify it. Make sure that you know how to restore the registry if a problem occurs. For more information about how to back up, restore, and modify the registry see: http://support.microsoft.com/kb/256986/

Disabling Read Caching on Windows Workstations (All Windows operating systems in the 9X family) 

The Windows registry entry that controls read caching on Windows network clients is:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VxD\VREDIR
DiscardCacheOnOpen REG_BINARY 0 or 1
Default: 0 (not disabled)
To disable read caching, the value of DiscardCacheOnOpen must be set to 1.

You can verify (or change or add, if necessary) this Registry value using the Windows Registry Editor (regedit.exe).
If you do change this Registry value, you will have to reboot the PC to ensure that the new setting goes into effect.

Please read the Microsoft disclaimer regarding editing of the Windows registry here.

STEPS:

1. Start > Run > Regedit.exe
2. Click on the + (plus sign) next to HKey_Local_Machine
3. Click on the + (plus sign) next to System
4. Click on the + (plus sign) next to CurrentControlSet
5. Click on the + (plus sign) next to Services
6. Click on the + (plus sign) next to VxD
7. Click on the VREDIR entry on the left-hand side of Registry Editor

8. If the DiscardCacheOnOpen registry value already exists (on the right-hand side of Registry Editor), ensure that its value is 1

9. If the DiscardCacheOnOpen value already exists but its value is not 1, double-click on DiscardCacheOnOpen to change its value to 1

10. If the DiscardCacheOnOpen entry does not exist, right-click in the white space of the right-hand side of Registry Editor
11. Select New > Binary value
12. Rename the value to DiscardCacheOnOpen
13. Double-click on DiscardCacheOnOpen to change its value to 1

Disabling Opportunistic Locking on Windows Servers ( Windows NT, Windows 2000, Windows XP and Windows 2003 Server).

There are 2 Windows registry entries that control opportunistic locking (oplocks) on Windows network servers:

1. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters EnableOpLockForceClose
2. HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters EnableOplocks

1. EnableOpLockForceClose

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters
EnableOpLockForceClose REG_DWORD 0 or 1
Default: 0 (not disabled)

To disable oplocks, the value of EnableOpLockForceClose must be set to 1.

You can verify (or change or add, if necessary) this Registry value using the Windows Registry Editor (regedit.exe).
If you do change this Registry value, you will have to reboot the PC to ensure that the new setting goes into effect.

Please read the Microsoft disclaimer regarding editing of the Windows registry here.

STEPS:

1. Start > Run > Regedit.exe
2. Click on the + (plus sign) next to HKey_Local_Machine
3. Click on the + (plus sign) next to System
4. Click on the + (plus sign) next to CurrentControlSet
5. Click on the + (plus sign) next to Services
6. Click on the + (plus sign) next to LanManServer
7. Click on the Parameters entry on the left-hand side of Registry Editor

8. If the EnableOpLockForceClose registry value already exists (on the right-hand side of Registry Editor), ensure that its value is 1

9. If the EnableOpLockForceClose value already exists but its value is not 1, double-click on EnableOpLockForceClose to change its value to 1

10. If the EnableOpLockForceClose entry does not exist, right-click in the white space of the right-hand side of Registry Editor
11. Select New > DWORD value
12. Rename the value to EnableOpLockForceClose
13. Double-click on EnableOpLockForceClose to change its value to 1

[FreeNAS Slow Network]: Slow Network

Slow Network FreeNAS

502. Slow network.

 
Views: 15800
Votes: 8
Posted on Tuesday, December 19, 2006 by Tim Fehlman
http://www.dailycupoftech.com/category/freenas /

I had deployed a new network with a server and a number of workstations. Everything worked fine and the network connectivity was there but everything was painfully slow. All the network applications took forever to come up and file transfers were ridiculous. After about two weeks, I finally discovered the problem.

When I created the images for the workstations, I built the image with the network card set to a speed of 100 Mbits and full duplex. At the last minute, we switched from a managed switch to an unmanaged switch. Most unmanaged switches have their speed duplexing set to autonegotiation. It was at this point that I made a bad assumption.

And I assumed that since the network cards on each workstations were hard coded to 100/full, the switch would automatically recognize this and set each port to 100/full as well. In reality, I had created a duplex mismatch. Essentially, the network cards were looking for a 100/full link on the other end of their patch cable while the switch was expecting an autonegotiation session. Neither got what they were looking for and trouble ensued!

The long and the short of it is that you have to be sure that what is on one end of a patch cable is the same on the other end of the patch cable. That also goes for autonegotiation. Both devices need to be set to autonegotiation. If only one is set that way, then you will experience all types of network slowdowns! So remember:

10 to 10
100 to 100
1000 to 1000
half to half
full to full
auto to auto
Make sure that all of your speed and duplex match on all of your devices.
Quick Summary

  • Auto-negotiation is not 100% reliable, but it does generally work.
  • Links with a duplex mismatch will operate, but will generate large numbers of errors, and can slow down busy networks.
  • For most interfaces, both speed and duplex need to be set to auto for full auto-negotiation to work.
  • Forcing a Catalyst switch port to a specific speed disables auto-negotiation for the duplex setting.
  • Full-duplex mode can be achieved only if both sides of the link are either set to auto-negotiate or manually configured to use full-duplex.
  • Full-duplex will work only if a host is connected directly to a switch or other device, with no repeaters or hubs in-between.
  • If auto-negotiation is enabled on only one side of the link, it will always default to half-duplex, regardless of what the other side of the link is forced to.
  • If one side of a link is forced to full duplex and the other is set to auto-negotiation, a duplex mismatch will occur.
  • You can force a new auto-negotiation by simply unplugging a host cable for 10 seconds.
  • Most 10 Mb interfaces can run only in 10Mb half-duplex mode.
  • Most 10/100 Mb interfaces can do auto-negotiation. Most 10/100 Mb interfaces with RJ-45 twisted pair jacks can run in full-duplex mode.
  • Any network connected via an AUI port (with an external transceiver, for example) can run only in 10Mb half-duplex mode.

[FreeNAS Boot Failure]: If the boot device, HHD or USB drive, fails, how do you get FreeNAS up?

If the boot device, HHD or USB drive, fails, how do you get FreeNAS up?



► Power Off the system.

►
 Disconnect your boot drive and all data hard-drives from the system.

► Connect only your new boot drive, setup the drive on BIOS and reinstall FreeNAS on the new drive (hard drive, compact flash, USB ).

► Reboot the system and check if the FreeNAS installation is OK.

► Power Off the system, connect you data hard-drives back on the system.

► Start FreeNAS.
► If you have a back-up configuration file, you can restore it but you must remove the old mount point for your first drive/second partition and remove the old first drive.
► If you don’t have a back-up configuration file, just add all your drives, YOU DON’T NEED TO FORMAT ANY DRIVES and create the mount points for each drive.
Software RAID : If previously you use Software RAID, just add your drives on disks management, DON’T FORMAT THE DRIVES, built your Software RAID in the same way as before (DON’T FORMAT THE RAID ARRAY) and create the mount point for the RAID array.
► Setup the groups, users, services and you are back online.


[FreeNAS]: Easy Automated Snapshot-Style Backups with FreeNAS and Rsync

 Easy Automated Snapshot-Style Backups with FreeNAS and Rsync


For extra information about this script and before asking me any questions pleas read Mike Rubel page.
http://www.mikerubel.org/computers/rsync_snapshots/

► You can run this script as cron job or manually from the shell.

► This script was made to run on FreeNAS , I tested myself, and works as expected.





Download rsync_snap_d2d.sh