Skip to content

ddrescue clone failing hard drive

  • Clone hard drive with ddrescue

    Testing a hard drive for failure via smartmontools

    Install smartmontools if not already present: (debian)

    apt-get install smartmontools

    Locate specific hard drives currently installed:

    lsblk


    In this case I choose device ‘sda’

    smartctl --all /dev/sda -q errorsonly

    Example of failing hard drive: smart-failure.png


    Preparing a used hard drive for cloning

    Clearing existing partitions from destination hard drive:

    In this case the destination drive is defined as sdc:

    I’ll use fdisk to clear all existing partitions:

    fdisk /dev/sdc Then print partition existing on sdc by typing the letter p

    p

    fdisk.png

    The above image shows some detail on the destination drive, including on FreeBSD partition named /dev/sdc4

    Since there is only one partition on the destination drive, we simply need to type the letter d at the fdisk command prompt:

    d

    fdisk-delete.png

    If there are more existing partitions on the destination drive, we are presented with them, and numbers to go along, we select which partition via numbers what we delete (in this case everything)

    fdisk-multiple-partitions.png

    Once we’ve removed the partitions, we type the letter w to write the changes, (actually delete the partitions)

    I type the letter p once again to make sure no partitions are still on the disk and then the letter w to write the changes:

    fdisk-write.png


    Using ddrescue to clone the hard drive:

    !Insure your drive letters are correct!

    The command I use to simplify the procedure including reading the damaged drive and putting as little pressure on it as we can manage, while getting as much data as we can, are as follows:

    Insure existing, or damaged hard drive first, and destination hard drive second (In other terms input file first and output file second)

    ddrescue -f -n /dev/sda /dev/sdc /root/recovery.log

    From https://linux.die.net/man/1/ddrescue

    -f, --force overwrite output device or partition

    -n, --no-split do not try to split or retry failed blocks

    We set a log file, which ddrescue can read later if for any reason the proceedure fails in the middle. Or also if we do not get as close to a 100% success rate when finished, the log file will set ddrescue to only attempt to get failed blocks on additional runs.

    ddrescue.png

    In this case 100% of the data was recovered, and transferred to the new hard drive:

    ddrescue-complete.png

    I run lsblk once again to check partitions have been transferred: ddrescue-new-partitions.png


    I then restart the machine:

    shutdown -r now

    After a restart, the journal will be recovered as well as other routine scripts run. I remove other old, damaged hard drives during the restart just before the BIOS kicks up.

    Again after a restart I run :

    lsblk

    ddrescue-new-harddrive.png

    Fin!


    Note This local server is a test environment running ProxMox. Everything running as it should after the clone.

    • Locate HDD

      Linux Systems Guides
      1
      0 Votes
      1 Posts
      449 Views
      No one has replied
    • Basic file permissions

      Linux Systems Guides
      1
      0 Votes
      1 Posts
      963 Views
      No one has replied
    • Install locate *nix

      Linux Systems Guides
      1
      0 Votes
      1 Posts
      1k Views
      No one has replied
    • Set script to executable

      Linux Systems Guides
      1
      +0
      0 Votes
      1 Posts
      1k Views
      No one has replied
    • 0 Votes
      3 Posts
      3k Views
      rickR

      New script due to dependency changes:

      #! /bin/sh set -e

      if [ “$1” = “install” ] || [ “$1” = “upgrade” ]; then

      ACTION=$1

      elif [ “$1” = “” ]; then ACTION=install else echo “usage: $(basename $0) action [source]” echo “action is either install (default) or upgrade” echo “source is either tar (default) or git” exit 1 fi

      if [ “$2” = “git” ] || [ “$2” = “tar” ]; then SOURCE=$2 elif [ “$2” = “” ]; then SOURCE=tar else echo “Unknown source $2!” exit 1 fi

      if [ “$BPC_UID” ] && [ $(getent passwd “$BPC_UID” | cut -d ‘:’ -f 1) != “backuppc” ]; then echo “The uid = $BPC_UID is already in use!” exit 1 fi

      #Install dependencies

      apt-get -q update export DEBIAN_FRONTEND=noninteractive apt-get install -q -y apache2 apache2-utils libapache2-mod-perl2 par2 perl smbclient rsync tar gcc zlib1g zlib1g-dev rrdtool git make perl-doc libarchive-zip-perl libfile-listing-perl libxml-rss-perl libcgi-session-perl libacl1-dev wget iputils-ping pwgen

      #Set up backuppc user and directory

      if ! id backuppc >/dev/null 2>&1; then if [ “$BPC_UID” ]; then adduser --system --home /var/lib/backuppc --group --disabled-password --shell /bin/false --uid=“$BPC_UID” backuppc else adduser --system --home /var/lib/backuppc --group --disabled-password --shell /bin/false backuppc fi fi mkdir -p /var/lib/backuppc/.ssh chmod 700 /var/lib/backuppc/.ssh echo -e “BatchMode yes\nStrictHostKeyChecking no” > /var/lib/backuppc/.ssh/config if [ ! -e /var/lib/backuppc/.ssh/id_rsa ]; then ssh-keygen -q -t rsa -b 4096 -N ‘’ -C “BackupPC key” -f /var/lib/backuppc/.ssh/id_rsa fi chmod 600 /var/lib/backuppc/.ssh/id_rsa chmod 644 /var/lib/backuppc/.ssh/id_rsa.pub chown -R backuppc:backuppc /var/lib/backuppc/.ssh

      #Set password or read password file

      if [ “$BPC_PASS” ]; then PASSWORD=“$BPC_PASS” elif [ -e /root/password ]; then PASSWORD=$(cat /root/password) chmod 600 /root/password else PASSWORD=$(pwgen -s -1 32) echo “$PASSWORD” > /root/password chmod 600 /root/password fi echo “backuppc:$PASSWORD” | chpasswd backuppc

      #Get BackupPC release versions

      get_latest_release() { wget -q -O - “https://api.github.com/repos/$1/releases/latest” | grep ‘“tag_name”:’ | sed -E ‘s/.“([^”]+)"./\1/’
      } bpcver=$(get_latest_release “backuppc/backuppc”) bpcxsver=$(get_latest_release “backuppc/backuppc-xs”) rsyncbpcver=$(get_latest_release “backuppc/rsync-bpc”)

      mkdir -p /tmp/bpc cd /tmp/bpc

      if [ $SOURCE = “tar” ]; then

      ##Fetch and install latest stable releases

      wget https://github.com/backuppc/backuppc-xs/releases/download/$bpcxsver/BackupPC-XS-$bpcxsver.tar.gz wget https://github.com/backuppc/rsync-bpc/releases/download/$rsyncbpcver/rsync-bpc-$rsyncbpcver.tar.gz wget https://github.com/backuppc/backuppc/releases/download/$bpcver/BackupPC-$bpcver.tar.gz tar -zxf BackupPC-XS-$bpcxsver.tar.gz tar -zxf rsync-bpc-$rsyncbpcver.tar.gz tar -zxf BackupPC-$bpcver.tar.gz cd BackupPC-XS-$bpcxsver perl Makefile.PL make make test make install cd …/rsync-bpc-$rsyncbpcver ./configure make make install cd …/BackupPC-$bpcver fi

      if [ $SOURCE = “git” ]; then

      #Fetch and install the latest development code instead

      git clone https://github.com/backuppc/backuppc.git git clone https://github.com/backuppc/backuppc-xs.git git clone https://github.com/backuppc/rsync-bpc.git cd backuppc-xs perl Makefile.PL make make test make install cd …/rsync-bpc ./configure make make install cd …/backuppc ./makeDist --nosyntaxCheck --releasedate “date -u "+%d %b %Y"” --version ${bpcver}git tar -zxf dist/BackupPC-${bpcver}git.tar.gz cd BackupPC-${bpcver}git fi

      if [ $ACTION = “install” ]; then ./configure.pl --batch --cgi-dir /var/www/cgi-bin/BackupPC --data-dir /var/lib/backuppc --hostname backuppc --html-dir /var/www/html/BackupPC --html-dir-url /BackupPC --install-dir /usr/local/BackupPC fi

      if [ $ACTION = “upgrade” ]; then ./configure.pl --batch --config-path /etc/BackupPC/config.pl fi

      #Set up web server #Note that changing the apache user and group (/etc/apache2/envvars) could cause other services #provided by apache to fail. There are alternatives if you don’t want to change the apache #user: use SCGI or a setuid BackupPC_Admin script - see the docs. cp httpd/BackupPC.conf /etc/apache2/conf-available/backuppc.conf sed -i “/Require local/d” /etc/apache2/conf-available/backuppc.conf sed -i “s/export APACHE_RUN_USER=www-data/export APACHE_RUN_USER=backuppc/” /etc/apache2/envvars sed -i “s/export APACHE_RUN_GROUP=www-data/export APACHE_RUN_GROUP=backuppc/” /etc/apache2/envvars echo ‘’ > /var/www/html/index.html a2enconf backuppc a2enmod cgid service apache2 restart

      #Set up backuppc service

      cp systemd/init.d/debian-backuppc /etc/init.d/backuppc

      chmod 755 /etc/init.d/backuppc

      update-rc.d backuppc defaults

      chmod u-s /var/www/cgi-bin/BackupPC/BackupPC_Admin

      touch /etc/BackupPC/BackupPC.users

      sed -i “s/$Conf{CgiAdminUserGroup}.*/$Conf{CgiAdminUserGroup} = ‘backuppc’;/” /etc/BackupPC/config.pl

      sed -i “s/$Conf{CgiAdminUsers}.*/$Conf{CgiAdminUsers} = ‘backuppc’;/” /etc/BackupPC/config.pl

      chown -R backuppc:backuppc /etc/BackupPC

      echo $PASSWORD | htpasswd -i /etc/BackupPC/BackupPC.users backuppc

      service backuppc start

      #Clean up

      cd rm -rf /tmp/bpc echo “All done!” exit 0

    • Add tun device to LXC Proxmox

      Linux Systems Guides
      1
      +0
      0 Votes
      1 Posts
      804 Views
      No one has replied
    • Letsencrypt Debian

      Linux Systems Guides
      1
      +0
      0 Votes
      1 Posts
      858 Views
      No one has replied
    • Basic Linux Terminal Commands

      Pinned Linux Systems Guides
      8
      1 Votes
      8 Posts
      7k Views
      rickR

      To list all files in current directory including dot files (hidden files or directories), as well as print permissions :

      ls -la
  • Again running smartctl after all is said and done:

    smartctl --all /dev/sda

    ddrescue-smartctl-after-rescue.png ddrescue-smartctl-2.png

    Yet an old drive in itself, I run the wheels off of them, and monitor regularly as anyone should.

FreeBSD Notes
  • Unable to negotiate with 10.10.1.35 port 22: no matching host key type found. Their offer: ssh-rsa,ssh-dss

    While attempting ssh this error is generally due to mismatched versions of ssh, where an up to date version is attempting to access an older version

    Add the following to your command :

    The proper way:

    ssh -o KexAlgorithms=diffie-hellman-group14-sha1 -oHostKeyAlgorithms=+ssh-dss 10.10.1.35

    The cheap way:

    Example :

    ssh -oHostKeyAlgorithms=+ssh-dss 10.10.1.35

    or ssh -oHostKeyAlgorithms=+ssh-dss user@10.10.1.35

    This can be added to the ~/.ssh/config file

    Host my-server HostName 10.10.1.35 HostKeyAlgorithms=+ssh-dss
    read more

  • Locate hard drive and get information

    ls -l /sys/block | grep sd.

    Output:

    lrwxrwxrwx 1 root root 0 Jun 22 06:28 sda -> ../devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0/block/sda lrwxrwxrwx 1 root root 0 Jun 22 06:28 sdb -> ../devices/pci0000:00/0000:00:1f.2/ata2/host1/target1:0:0/1:0:0:0/block/sdb lrwxrwxrwx 1 root root 0 Jun 22 06:28 sdc -> ../devices/pci0000:00/0000:00:1f.2/ata3/host2/target2:0:0/2:0:0:0/block/sdc lrwxrwxrwx 1 root root 0 Jun 22 06:28 sdd -> ../devices/pci0000:00/0000:00:1f.2/ata4/host3/target3:0:0/3:0:0:0/block/sdd

    Or for a more detailed view

    strace -e trace=open lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda 8:0 0 3.6T 0 disk ├─data--2-data--2_tmeta 252:3 0 15.9G 0 lvm │ └─data--2-data--2-tpool 252:9 0 3.6T 0 lvm │ ├─data--2-data--2 252:10 0 3.6T 1 lvm │ ├─data--2-vm--101--disk--0 252:11 0 8G 0 lvm │ ├─data--2-vm--103--disk--0 252:12 0 32G 0 lvm │ ├─data--2-vm--107--disk--0 252:13 0 500G 0 lvm │ └─data--2-vm--108--disk--0 252:20 0 100G 0 lvm └─data--2-data--2_tdata 252:6 0 3.6T 0 lvm └─data--2-data--2-tpool 252:9 0 3.6T 0 lvm ├─data--2-data--2 252:10 0 3.6T 1 lvm ├─data--2-vm--101--disk--0 252:11 0 8G 0 lvm ├─data--2-vm--103--disk--0 252:12 0 32G 0 lvm ├─data--2-vm--107--disk--0 252:13 0 500G 0 lvm └─data--2-vm--108--disk--0 252:20 0 100G 0 lvm sdb 8:16 0 698.6G 0 disk └─sdb1 8:17 0 698.6G 0 part /mnt/pve/backups sdc 8:32 0 3.6T 0 disk ├─vm--data-vm--data_tmeta 252:4 0 15.9G 0 lvm │ └─vm--data-vm--data-tpool 252:14 0 3.6T 0 lvm │ ├─vm--data-vm--data 252:15 0 3.6T 1 lvm │ ├─vm--data-vm--100--disk--0 252:16 0 270G 0 lvm │ ├─vm--data-vm--102--disk--0 252:17 0 100G 0 lvm │ ├─vm--data-vm--104--disk--0 252:18 0 25G 0 lvm │ └─vm--data-vm--106--disk--0 252:19 0 32G 0 lvm └─vm--data-vm--data_tdata 252:7 0 3.6T 0 lvm └─vm--data-vm--data-tpool 252:14 0 3.6T 0 lvm ├─vm--data-vm--data 252:15 0 3.6T 1 lvm ├─vm--data-vm--100--disk--0 252:16 0 270G 0 lvm ├─vm--data-vm--102--disk--0 252:17 0 100G 0 lvm ├─vm--data-vm--104--disk--0 252:18 0 25G 0 lvm └─vm--data-vm--106--disk--0 252:19 0 32G 0 lvm sdd 8:48 0 931.5G 0 disk ├─sdd1 8:49 0 1007K 0 part ├─sdd2 8:50 0 1G 0 part └─sdd3 8:51 0 930.5G 0 part ├─pve-swap 252:0 0 8G 0 lvm [SWAP] ├─pve-root 252:1 0 96G 0 lvm / ├─pve-data_tmeta 252:2 0 8.1G 0 lvm │ └─pve-data 252:8 0 794.3G 0 lvm └─pve-data_tdata 252:5 0 794.3G 0 lvm └─pve-data 252:8 0 794.3G 0 lvm +++ exited with 0 +++ cat /proc/partitions major minor #blocks name 8 0 3907018584 sda 8 16 732574584 sdb 8 17 732572672 sdb1 8 32 3907018584 sdc 8 48 976762584 sdd 8 49 1007 sdd1 8 50 1048576 sdd2 8 51 975712967 sdd3 252 0 8388608 dm-0 252 1 100663296 dm-1 252 2 8495104 dm-2 252 3 16650240 dm-3 252 4 16650240 dm-4 252 6 3873329152 dm-6 252 5 832888832 dm-5 252 7 3873329152 dm-7 252 8 832888832 dm-8 252 9 3873329152 dm-9 252 10 3873329152 dm-10 252 11 8388608 dm-11 252 12 33554432 dm-12 252 13 524288000 dm-13 252 14 3873329152 dm-14 252 15 3873329152 dm-15 252 16 283115520 dm-16 252 17 104857600 dm-17 252 18 26214400 dm-18 252 19 33554432 dm-19 252 20 104857600 dm-20

    Locate drive by serial and model information

    hdparm -i /dev/sda /dev/sda: Model=WDC WD4000FYYZ-05UL1B0, FwRev=00.0NS05, SerialNo=WD-WCC132262513 Config={ HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq } RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=0 BuffType=unknown, BuffSize=unknown, MaxMultSect=16, MultSect=off CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=7814037168 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120} PIO modes: pio0 pio3 pio4 DMA modes: mdma0 mdma1 mdma2 UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6 AdvancedPM=yes: unknown setting WriteCache=disabled Drive conforms to: Unspecified: ATA/ATAPI-1,2,3,4,5,6,7 * signifies the current active mode
    read more

  • Screen recording can use webm as their format, it can be more simple to use a gif to embed into a website or forum, than adding scripts to host different video format.

    In this case I grabbed a screen record of the progress for writing zeros to a hard drive with dd

    Use ffmpeg to convert webm to gif:

    First create a pallet:

    Move into the directory which the webm is located, or type in the path

    Where ‘dd.webm’ is the screen recording

    ffmpeg -y -i dd.webm -vf palettegen palette.png

    Output:

    dd-ffmpeg.png

    Then convert the webm to gif:

    ffmpeg -y -i dd.webm -i palette.png -filter_complex paletteuse -r 10 dd.gif

    dd-webm-2.png

    This is what I ended up with, looks like any image of any alien on the interwebz, as if shot through a potato, some tweaking of the command is in my future. None the less.

    dd.gif

    read more

  • Write zeros to all sectors

    Use the command ‘lsblk’ to find the drive you wish to erase

    I’m using ‘dd’ to erase things

    In this case I’m torching sdb

    Only use status=progress if you care it’s not necessary

    dd if=/dev/zero of=/dev/sdb bs=12M status=progress

    dd.gif

    read more

  • Remove old kernel images that are cluttering the system

    zstd-no-space-error.png

    Most of these errors are due to low or not enough space left in the root partition.

    If you are using a Debian flavor:

    sudo apt-get autoremove --purge

    zstd-error.png

    Inform grub whenever an old kernel is removed:

    update-grub

    update-grub.png

    Remove the un-used kernel config files:

    These will be the files pre cursed as ‘rc’ where installed kernels use ‘ii’

    … As well as files no longer used or required due to dependencies

    This command will detect, print, and remove left over cruft from previously installed packages or scripts, that have been removed or updated.

    sudo dpkg --purge $(dpkg -l | awk '/^rc/{print $2}')

    purge.png

    You can re-run the following to view the installed kernel(s):

    dpkg --list | grep linux-image

    installed-kernels.png

    read more