B3 RAID recovery progress

[theWebalyst]

The RAID recovery progress is, like so many other progress indicators, no more than an indicator that it hasn't finished yet. Mine has been running for a few hours and shows:

Code: Select all

Recovering RAID array '/dev/md0'.
Current recover progress is 100% and is estimated to finish in 2360 hours 46 minutes.

It has shown 100% the whole time.

Refresh:

Code: Select all

Current recover progress is 100% and is estimated to finish in 1152 hours 38 minutes. 

Refresh:

Code: Select all

Current recover progress is 100% and is estimated to finish in 1517 hours 24 minutes. 

When it first started it was about 300 hours to complete, then quickly down to 60 hours then up again to 00's of hours.

Perhaps this could be made more accurate, or at least made less misleading.

Mark

[nobody]

If you have access to the SSH shell please show the content of /proc/mdstat

[theWebalyst]

If you have access to the SSH shell please show the content of /proc/mdstat

I'm afraid its all gone horribly wrong. Next morning I had red light on B3 and very slow/broken web i/f. SSH still worked with periods of delayed response to commands. Lots of network errors in syslog (e.g. "error on subcontainer 'ia_addr' ").

I tried a soft restart but B3 didn't come up. I tried various power off/ons (with/without eSATA, with different PSU etc.) but b3 won't boot: red LED (or is it purple?), no network, no way to SSH.

I guess next step is to re-image?

This is yet another case of b2/b3 RAID recovery failing. Maybe /dev/sd1b was still not ok, but why should that cause the system to be destroyed? My confidence in using b3 for its main purpose--reliable long term storage--is being shot full of holes!How can I recommend this to clients now?

Is there any alternative to re-imaging? Seems very odd that /dev/sdb1 was showing a fault, /dev/sda1 was fine, had never shown any faults, yet RAID recovery has destroyed the system.

Not happy

Mark

[nobody]

Assuming your first priority is to get access to data stored on the B3, you can access the B3 using a recovery USB disk. Personally I'd be really worried what the quality of that internal disk is as the RAID function should not even come close to the system drive. Potentially it's just the home partition that's fried and the b3 won't boot without it. In that case you do not need to re-image but merely to fix the home partition.

<rant>
As for reliable long time storage, I'd say a RAID1 setup is not a great method to use anyway. Yes, it's useful to keep your system from dying if the boot or root partition fails, or if you really cannot afford to miss todays data. But for the B3, the boot and system disks are not on the RAID, and the targeted audience (SOHO mostly) would be OK with yesterdays backup. So there is NO benefit of RAID1!

Instead, you get a much more resilient and flexible setup by using the eSata disk as it is: an external disk. Use rsnapshot or just rsync with cron to make regular copies from your B3 main disk to the external, and use hard links on that external to keep backups.

RAID1 over eSata is just a really really bad idea and I wish Excito stopped promoting it.
</rant>

[theWebalyst]

Assuming your first priority is to get access to data stored on the B3, you can access the B3 using a recovery USB disk.

Thanks. /home isn't that high a priority - I have alternatives. In fact, the most critical data on it is backup of b3 system config files, because that will help me restore a working b3.

Personally I'd be really worried what the quality of that internal disk is as the RAID function should not even come close to the system drive.

Agreed, but my questioning of b3's RAID recovery needs to be addressed. It is rather suspicious to me that smartctl/mdadm have identified issues with only the external disk, yet within hours of starting RAID recovery my system partition appears fried.

I have no evidence of a problem with the internal disk, other than the belief that RAID recovery shouldn't touch the system partition. Well, that's pretty weak giving the circumstantial evidence pointing the other way, both now and from past poor experiences with RAID recovery.

BTW I am already backing up using rsnapshot, so if I recover /home I'll have my system config files, but I would like to keep this thread on topic: i.e. restoring my b3 & recovering data. It won't help me if people start responding with comments on backup setups.

Thanks for your helpful comments, I appreciate it.

Mark

[Ubi]

BTW I am already backing up using rsnapshot, so if I recover /home I'll have my system config files, but I would like to keep this thread on topic: i.e. restoring my b3 & recovering data. It won't help me if people start responding with comments on backup setups.

Well that is because your original post stated this:

My confidence in using b3 for its main purpose--reliable long term storage--is being shot full of holes!How can I recommend this to clients now?

which makes it on-topic.

The reason why the B3+esSata secondary is a good idea for your clients is that you do not use the secondary drive for RAID, but for semi-remote backups.

[theWebalyst]

The topic is the topic, and not changed by things I mention in relation to it.

You are free to create a new thread about that, and to mention it in this thread. You are free to continue to make off-topic comments.

All I can do is ask you to please keep your comments to helping with the issues I have raised about RAID recovery, and my attempts to recover from a major system failure while using it.

Mark

[nobody]

OK sure, so you ask a combined question on a forum but you then ask people not to respond to the parts of the questions if these does not comply with the stated headline topic in the post? Apologies for not understanding that previously

[theWebalyst]

nobody - you are misrepresent me in your last post.

I'm grateful for your help, but you dwarfed that part of your response when you introduced a different thread which you characterised as a "rant".

I didn't want others to come in responding to that, when I need help with what is this important problem, and want to highlight possible issues with b3's RAID recovery.

I don't understand why you think it is ok to use my request for help to have a rant, and now seem offended, after I acknowledged your help and politely asked everyone to keep to the topic.

Mark

[nobody]

Thats what the <rant> brackets were for. Anyway your strategy backfired as we are niw having a meta discussing about misinterpretations. I' also not offended, but more amused.

The much more interesting issue is whether you got around to booting the b3 from a rescue disk yet? Alternatively you can take the disk out and plug it into a regular pc, but i suppose that is a lot more work... I'm really curious to hear if the mdraid indeed killed the root partition, or hwat turned out to be the reason why the boot sequence failed.

[theWebalyst]

Not looking good. I made the USB (edited bubba.cfg to not install) and b3 booted to green light, but has been sitting link this for forty minutes. Even an install is supposed to last only 15 minutes.

So does boot from USB go from green to blue when booted?

... and how long should it take?

And what might "stuck at green" mean?

Any ideas?

Thanks

Mark

[nobody]

Not sure, i think the lights are a little different in rescue mode. Much more important is whether you can ssh into one of the nics right?

[theWebalyst]

Not sure, i think the lights are a little different in rescue mode. Much more important is whether you can ssh into one of the nics right?

Well I did try the network but no sign of b3 connecting to my router, no resolution of b3, no SSH.

My guess is that on boot it should change LED colour, but I've never done this before so don't know and the b3 manual is not explicit. It says it turns blue after an install, so it looks almost certain to be the case, but it is not explicit.

I'll take a look in case I messed up bubba.cfg - I used Notepad++ to edit it. I'll try notepad2 this time!

Mark