B3 Failing to boot

[jeff]

Hello forum

(FYI this issue has been ongoing, I suspect that due to all the activity going on that support has taken a back seat. I am reposting this message as it appears to have been lost during the forum migration. I would really like some help as soon as possible as functioning without a server is incredibly difficult.)

I purchased a B3 back in October of 2012 (Order number 10066598). About 2 months ago I was unable to log into the web interface. I decided to reboot the B3 by holding down the on/off button but it never came back up.

When I power on the hardware, the LED is purple and never changes. I have tried to boot using the rescue/install USB disk I purchased with the server (which I have previously used to reset my password, so I know it works) however the led remains pink/purple.

I have even tried re-installing using the instructions in the link below but the led remains green for hours and doesn't appear to be doing anything.
http://wiki.excito.org/wiki/index.php/R ... ubba2_/_B3

When I try to SSH to the B3 using putty I receive the following error:
"Server's host key did not match signature applied."
I do have an SSH enabled user configured but I never set up the private/public keys. I always manually logged in when required.

I sent a support request to support@excito.com on the 28th of May 2015 and received a ticket number 199. After a week I responded to the email as I had no update.

On the 17 June 2015, I tried to reply again but received a permanent failure for the email address support@excito.com. The email address seems to be working again although only one of my emails has appeared to get through to Excito Support. All the other emails get sent to Rodeus Technologies. Guillaume at Rodeus has tried to forward my request to Excito support but I have not received any contact.

As you can imagine I'm getting quite desperate to get some help. Unless something has changed, my B3 is still under warranty.

Can anyone help?

Thank you
Jeff

[Gordon]

If you start from the rescue stick, the front led color should be green, not purple. So something is wrong with whatever you tried at that point. Do note that the only difference between install and rescue is an ini file setting on the USB stick and that this ini file must have unix/mac line endings (i.e. you'll corrupt the stick if you edit that ini file with a Windows tool).

Yes it is normal for an SSH client to complain that the cipher is different if you boot the B3 from the rescue/install stick and the same DNS name or IP address is used. Simply delete the stored key in the client and it will connect.

Shamefully there's not much information in the led color staying purple. It may mean that the kernel panics or that some service fails to start. The most likely problem you would encounter though is an issue with mounting one of your disks/partitions/volumes. If your disk is damaged that would also explain why reinstalling would stall on that very same issue.

I suggest that you reattempt to boot the rescue system and ssh into it. Verify that you can enable swap (`swapon /dev/sda3`) and mount the boot partition /dev/sda1. Edit etc/fstab in the mount location and disable the entry for /home. Optionally also delete the service start entry for mdadm (I think the system boots to runlevel 2). Then try if the system will boot to blue light. If it does, you may then attempt to fix the issue with the disk.

[jeff]

Thanks for your response Gordon.

Just to clear things up, when I boot from the rescue USB the LED is green.

To the issue at hand, I am using putty to attempt a connection with my b3 internally within my home network (this is the application I have used previously). I have cleared the registry of all reference to putty by running "putty -cleanup" however I still can't seem to connect to the b3.

I specify the host ip (10.1.1.*) and the ssh port 22 and when I try to connect I still get the following message:
"Server's host key did not match the signature supplied"
I have also tried configuring the connection with the "root" username however I get the same error message.

What am I doing wrong?

Cheers
Jeff

[Gordon]

Hard to say. You do have the B3's WAN port connected to the router that hands out the 10.1.1.x address? As an alternative you could try Sakaki's gentoo live USB instead.

[jeff]

Hi Gordon

Yes, I have the WAN port connected to my router and thats where I get the 10.1.1.* address.

I've had some time today to investigate further. It appears I can boot and connect using both the original B3 rescue USB and the gentoo live USB I created. Turns out I had to use a newer version of putty.

With the excito B3 rescue boot, I ssh to the server as root but when I run the command you mentioned in your previous post I get the following:

Code: Select all

login as: root
root@10.1.1.*'s password:
# swapon /dev/sda3
swapon: can't stat '/dev/sda3': No such file or directory

I also tried running the following without success:

Code: Select all

# swapon /dev/sda1
swapon: can't stat '/dev/sda1': No such file or directory

Am I following your instructions correctly?

Cheers
Jeff

[Gordon]

Not exactly.

To mount a partition you should have used the `mount` command. However the response to activating the swap on partition /dev/sda3 says enough. Your disk is either completely empty or dead, deceased, pushing up the daisies, etcetera.

To verify whether you need a new dist you could try and see if you can access the disk with `gdisk` (GPT-able version of fdisk) or parted and if applicable create partitions and file systems on it. If you can get that far you may even perform all the installation steps manually to see at which point it stalls. There's no real mystery there; just find the install.sh file (shamefully MouettE deleted it from github, otherwise I would have directed you there) and copy its contents to your desktop. You can then copy-paste the appropriate sections to be executed on the B3.

[jeff]

Hi Gordon

Just want to thank you for your help and persistence on this.

Is there a way I can install gdisk onto the rescue USB?
When I attempt to run it I'm getting the following:

Code: Select all

# gdisk
-sh: gdisk: not found

I've also tried this from the gentoo live USB but can't find gdisk there either.

Alternatively, if the drive is dead whats the chance of success if I buy a replacement drive and reinstall the system? I assume if its just the drive then this should be achievable? If so what brand would you recommend? (I have/had a standard B3 3TB drive, I think it might be WD which seems to have only lasted a couple of years)

Cheers
Jeff

[Gordon]

Chances are probably slim of finding one that will work with the tiny OS that is the rescue system. But as said, you could also use parted, which is definitely there because the installer uses it. Another alternative is to use Sakaki's Gentoo live USB which holds more recent tools to inspect your current disk.

The B3's were originally shipped with WD Green drives, which have a nasty habit of killing themselves in NAS systems like the B3. You can search the forum for more information, but the short story is that the drive shuts itself down after a few seconds idle time, only to be revived even less seconds later, causing a lot of wear on the mechanics. There is a fix that involves changing the idle3 timer value on the drive, effectively causing the drive never to shut down any more. An alternative is to use the WD Red drive. Other drives may certify as well, but it has to be a low rev drive that generates a low amount of heat (e.g. a WD Black will probably blow itself up in less than a month within the B3 enclosure).

[sakaki]

Hi,

The version of fdisk on the current Gentoo live USB is GPT-capable. So if you like, you could boot from the live-USB again, and then try:

Code: Select all

b3 ~ # lsblk

to see a list of your drives - hopefully you will have sda (the HDD) and sdb (the live-USB) (plus their partitions) visible.
if so then try:

Code: Select all

b3 ~ # fdisk /dev/sda

and press p (and <Enter>), to display a partition table. q (and <Enter>) will then quit without changing anything.

What output do you get?

Best, sakaki

[jeff]

Thanks for the info about the drives Gordon, if I had of know that earlier I may not have been in this pickle.

I followed your instructions sakaki, the output is below:

Code: Select all

b3 ~ # lsblk
NAME      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda         8:0    0  2.7T  0 disk
sdb         8:16   1  7.5G  0 disk
ââsdb1      8:17   1   64M  0 part
ââsdb2      8:18   1    1G  0 part [SWAP]
ââsdb3      8:19   1    6G  0 part /
mtdblock0  31:0    0  768K  0 disk
mtdblock1  31:1    0  128K  0 disk
mtdblock2  31:2    0  1.1M  0 disk
b3 ~ # fdisk /dev/sda

Welcome to fdisk (util-linux 2.25.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

The size of this disk is 2.7 TiB (3000591900160 bytes). DOS partition table format can not be used on drives for volumes larger than 4294966784 bytes for 512-byte sectors. Use GUID partition table format (GPT).

Command (m for help): p

Disk /dev/sda: 2.7 TiB, 3000591900160 bytes, 5860531055 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device     Boot Start        End    Sectors Size Id Type
/dev/sda1           1 4294967295 4294967295   2T ee GPT

Partition 2 does not start on physical sector boundary.

Does this mean I might be able to access the drive afterall?

[Gordon]

Well at least it is accessible. However the disk label type should not be dos, so there is definitely something wrong here. What you'll want to do is create a new gpt label and next create partitions. The layout for the regular Bubba OS is a 10G linux parition, a linux LVM partition that spans all of the rest of the disk except for the last 1G, and a swap partition of 1G - in that order. You may however opt to make the LVM partition a regular partition, making the rest of the procedure a lot easier because you don't need to create the home volume inside the LVM.

You may want to verify the disk health. To do this you'll need smartctl which I don't think you is installed on the Gentoo live USB. The nice thing is that you can install it:

Code: Select all

USE="minimal" emerge -av smartmontools

And to be able to change the idle3 timer value install idle3-tools:

Code: Select all

ACCEPT_KEYWORDS="**" emerge -av idle3-tools

It will probably take ~10 minutes to install these two packages on the USB. Run `smartctl -a /dev/sda` and mark the value for Load Cycle Count. You'll probably find it to be several millions, which is bad.

Assuming you opted not to create the LVM, create file systems ext3 on both /dev/sda1 and /dev/sda2. Create a swap file system on /dev/sda3. Edit the ini file on the Bubba install USB and change the values for (I think) DO_PARTITION and DO_LVM to "0" (zero that is). The installer should then be able to complete.

Or stick to using Gentoo

[jeff]

Hi Gordon

It's been a few days but I have some news.

I followed the steps you mentioned (see below) but the re-install was unsuccessful again. I did notice however that some files were actually being written to the server hard drive. So I cleared the disk again, reset the b3 rescue config for a standard install and tried again. To my surprise about 20 minutes later for the first time in over 2 months the led was blue!!

I've done a few smoke tests, upgraded to the latest version and am almost ready to start reloading my data. Before I do that I wanted to ask a couple of questions:

1. How can I install idle3-tools and configure the drive not to shutdown so often?
2. I'm in two minds about whether or not to continue with the drive or buy a new one. What drive diagnostic tools do you recommend I install/run to confirm the drive is fit for purpose and doesn't have any damaged sectors?

Thanks again Gordon for all your help and for your input sakaki

I've included my attempt at manually configuring the drive in case anyone needs it for future reference. I suspect however I may have done something not quite right as the install didn't work.

Code: Select all

b3 ~ # fdisk /dev/sda

Welcome to fdisk (util-linux 2.25.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): n
Partition number (1-128, default 1):
First sector (34-5860531021, default 2048):
Last sector, +sectors or +size{K,M,G,T,P} (2048-5860531021, default 5860531021): +10G

Created a new partition 1 of type 'Linux filesystem' and of size 10 GiB.

Command (m for help): n
Partition number (2-128, default 2):
First sector (34-5860531021, default 20973568):
Last sector, +sectors or +size{K,M,G,T,P} (20973568-5860531021, default 5860531021): 5858577881

Created a new partition 2 of type 'Linux filesystem' and of size 2.7 TiB.

Command (m for help): n
Partition number (3-128, default 3):
First sector (34-5860531021, default 5858578432):
Last sector, +sectors or +size{K,M,G,T,P} (5858578432-5860531021, default 5860531021):

Created a new partition 3 of type 'Linux filesystem' and of size 953.4 MiB.

Command (m for help): t
Partition number (1-3, default 3):
Partition type (type L to list all types): 14

Changed type of partition 'Linux filesystem' to 'Linux swap'.

Command (m for help): p
Disk /dev/sda: 2.7 TiB, 3000591900160 bytes, 5860531055 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 1634AD81-E347-40C8-9C68-56C70CF9B843

Device          Start        End    Sectors   Size Type
/dev/sda1        2048   20973567   20971520    10G Linux filesystem
/dev/sda2    20973568 5858577881 5837604314   2.7T Linux filesystem
/dev/sda3  5858578432 5860531021    1952590 953.4M Linux swap


Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.



USE="minimal" emerge -av smartmontools
....
wget: unable to resolve host address âjaist.dl.sourceforge.netâ
!!! Couldn't download 'smartmontools-6.3.tar.gz'. Aborting.
 * Fetch failed for 'sys-apps/smartmontools-6.3', Log file:
 *  '/var/tmp/portage/sys-apps/smartmontools-6.3/temp/build.log'

 * Messages for package sys-apps/smartmontools-6.3:

 * Fetch failed for 'sys-apps/smartmontools-6.3', Log file:
 *  '/var/tmp/portage/sys-apps/smartmontools-6.3/temp/build.log'

ACCEPT_KEYWORDS="**" emerge -av idle3-tools
....
Resolving nchc.dl.sourceforge.net... failed: Temporary failure in name resolution.
wget: unable to resolve host address ânchc.dl.sourceforge.netâ
!!! Couldn't download 'idle3-tools-0.9.1.tgz'. Aborting.
 * Fetch failed for 'sys-apps/idle3-tools-0.9.1', Log file:
 *  '/var/tmp/portage/sys-apps/idle3-tools-0.9.1/temp/build.log'

 * Messages for package sys-apps/idle3-tools-0.9.1:

 * Fetch failed for 'sys-apps/idle3-tools-0.9.1', Log file:
 *  '/var/tmp/portage/sys-apps/idle3-tools-0.9.1/temp/build.log'



b3 ~ # lsblk
NAME      MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda         8:0    0   2.7T  0 disk
ââsda1      8:1    0    10G  0 part
ââsda2      8:2    0   2.7T  0 part
ââsda3      8:3    0 953.4M  0 part
sdb         8:16   1   7.5G  0 disk
ââsdb1      8:17   1    64M  0 part
ââsdb2      8:18   1     1G  0 part [SWAP]
ââsdb3      8:19   1     6G  0 part /
mtdblock0  31:0    0   768K  0 disk
mtdblock1  31:1    0   128K  0 disk
mtdblock2  31:2    0   1.1M  0 disk

b3 ~ # /sbin/mkfs -t ext3 /dev/sda1
mke2fs 1.42.12 (29-Aug-2014)
Creating filesystem with 2621440 4k blocks and 655360 inodes
Filesystem UUID: 23549624-ba93-4400-baa8-1ff43f233f0a
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

b3 ~ # /sbin/mkfs -t ext3 /dev/sda2
mke2fs 1.42.12 (29-Aug-2014)
/dev/sda2 contains a LVM2_member file system
Proceed anyway? (y,n) y
Creating filesystem with 729700539 4k blocks and 182427648 inodes
Filesystem UUID: 62c5c203-b1ec-4905-be2c-f7d1c2f9ef3d
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848, 512000000, 550731776, 644972544

Allocating group tables: done
Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

b3 ~ # /sbin/mkswap /dev/sda3
Setting up swapspace version 1, size = 976288 KiB
no label, UUID=8e342710-6269-4247-90ba-01b44dd19741

Cheers
Jeff

[Gordon]

Seems like you didn't have an internet connection while running the Gentoo live USB, so the packages couldn't be fetched. Incorrect values for default router and/or resolv.conf?


Good to hear that you managed to get the install to complete now.
Smartmontools is available as a package in Debian. You can install it with apt-get.

idle3-tools is on sourceforge and needs to be compiled, but bubba-backend comes with a precompiled version of the "official" wdidle3 tool. It's in /usr/share/bubba-backend and all you need to do is unpack the tarball and run the binary inside it.

[jeff]

Hi Gordon

Here is my LCC, pretty terrible as you suspected:

Code: Select all

# smartctl -A /dev/sda | grep "^193"
193 Load_Cycle_Count        0x0032   001   001   000    Old_age   Always       -       3383226

I've executed the following script I found in the tar file to increase the idle to 5 minutes:

Code: Select all

# ./wdSetIdle3To5Min.sh
wdSetIdle3To5Min Version 1.0.2
Set Idle3 Timer to 5 Minutes.
Copyright (C) 2005-2009 Western Digital Corporation


Model:    WDC WD30EZRX-00DC0B0
Serial:   WD-WMC1T0129736
Idle3 Timer is currently 8.000 seconds.
Idle3 Timer is currently 5 minutes, 0 seconds.
Idle3 Timer update Summary
  Successful updates =   1
  Failed updates   =   0

My only question now... Is it too late for the drive, should I just get a new one?

Cheers
Jeff

[Gordon]

Hard to say. Tolerances are a bitch. WD lists the maximum load cycle count at 300.000 and you're well over ten times as much. The drive will probably be safe to operate as long as you keep it powered on, but with every power cycle it may stall indefinitely. Do note that you must do a power cycle (i.e. not just reboot but actually unplug the B3 power cord) to complete the update of the idle3 timer - did you do that?

The thing is: only you can decide whether the B3 hard drive failing will be a complete disaster or just an inconvenience that can be fixed.