[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: LVM & RAID10
From: |
Wojciech Pyczak |
Subject: |
Re: LVM & RAID10 |
Date: |
Fri, 05 Dec 2008 22:11:50 +0100 |
User-agent: |
Mozilla/5.0 (X11; U; Linux i686 (x86_64); pl; rv:1.9.1b1pre) Gecko/20081006 Shredder/3.0a3 |
Hi,
Perhaps you can describe the steps used to create the LVM volume,
along with version number of tools. Let me see if the error can be
recreated in qemu.
OK. I've made some less chaotic tests:
Hardware (probably insignificant):
- AMD Phenom X3
- Gigabyte mobo (780G/SB700 AMD chipset)
- 2x WD SE-16 6400AAKS (before I started testing I zeroed with dd a
few GB of each drive - just in case..)
Software:
- System RescueCD (used to manage partitions, RAID/LVM volumes)
- Linux sysresccd 2.6.26.07-std111-amd64 #1 SMP Sun Oct 26 11:28:49
UTC 2008 x86_64 AMD Phenom(tm) 8450 Triple-Core Processor AuthenticAMD
GNU/Linux
- LVM version: 2.02.39 (2008-06-27), Library version: 1.02.27
(2008-06-25), Driver version: 4.13.0
- mdadm - v2.6.4 - 19th October 2007
- Gentoo custom ("stable amd64") stage3 -
http://www.funtoo.org/linux/amd64/funtoo-amd64-2008.12.05/ (used for
chrooting)
- Grub2 rev 1933 (latest), build on my current PC:
- ./configure --prefix=/usr --enable-grub-emu --enable-grub-fstest
LDFLAGS=-static
- gcc version 4.1.2 (Gentoo 4.1.2 p1.1)
- GNU Make 3.81
- bison (GNU Bison) 2.3
- sys-devel/binutils-2.18-r3
* IMO also insignificant, since previously I build grub on my new
system which used many "experimental" packages, such as GCC 4.3.2, etc. *
* Small note - I renamed grub binaries (in bin/sbin) to *.org and
made a wrapper script, so I would know arguments used by
grub-install/setup: *
#!/bin/sh
echo \#${0} address@hidden 1>&2
exec ${0}.org address@hidden
I used cfdisk to partition disks. Commands used to create RAID arrays
and LVM volumes:
mdadm --create /dev/md0 (md1) --raid-devices=3 --level=10
--layout=f2 --chunk=256 /dev/sd{a,b}1 (sd{a,b}2) missing
pvcreate /dev/md0
vgcreate test /dev/md0
lvcreate test -ntest -l1788
mdadm --assemble /dev/md0 (md1) /dev/sd{a,b}1 (sd{a,b}2)
pvscan
vgchange -ay
Nothing fancy...
Test procedure looked something like that:
- booting system with RescueCD
- partitioning, creating RAID/LVM, formatting, reboot
- after reboot:
- assembling RAID/LVM volumes
- mounting corresponding device
- unpacking stage3
- unpacking package containing grub
- creating missing directories (/boot/grub) and files (device.map,
simple grub.cfg)
- mouting proc, sys, dev
- grub-install/setup, whatever
===== Single partition, RAID10, reiserfs
Disk /dev/sda: 77825 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
Device Boot Start End #cyls #blocks Id System
/dev/sda1 * 0+ 607 608- 4883728+ da Non-FS data
/dev/sda2 0 - 0 0 0 Empty
/dev/sda3 0 - 0 0 0 Empty
/dev/sda4 0 - 0 0 0 Empty
Disk /dev/sdb: 77825 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
Device Boot Start End #cyls #blocks Id System
/dev/sdb1 * 0+ 607 608- 4883728+ da Non-FS data
/dev/sdb2 0 - 0 0 0 Empty
/dev/sdb3 0 - 0 0 0 Empty
/dev/sdb4 0 - 0 0 0 Empty
--------------------------------------------------------------------------------------------------------------------------
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid10 sda1[0] sdb1[1]
7325184 blocks 256K chunks 2 far-copies [3/2] [UU_]
unused devices: <none>
--------------------------------------------------------------------------------------------------------------------------
sysresccd grub # grub-install "(md0)"
#/usr/sbin/grub-install (md0)
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=device
/boot/grub
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=fs
--device /dev/md0
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map
--target=abstraction --device /dev/md0
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=drive
--device /dev/md0
#/usr/bin/grub-mkimage --output=/boot/grub/core.img
--prefix=(md0)/boot/grub biosdisk reiserfs pc pc raid mdraid
#/usr/sbin/grub-setup --directory=/boot/grub
--device-map=/boot/grub/device.map
(md0)
Installation finished. No error reported.
This is the contents of the device map /boot/grub/device.map.
Check if this is correct or not. If any of the lines is incorrect,
fix it and re-run the script `grub-install'.
(hd0) /dev/sda
(hd1) /dev/sdb
(hd2) /dev/sdc <= USB stick
--------------------------------------------------------------------------------------------------------------------------
Everything works (I copied memtest86 from RescueCD to /boot and managed
to boot it).
===== Single partion, LVM @ RAID10, reiserfs
Partitions/RAID like above
--------------------------------------------------------------------------------------------------------------------------
address@hidden /root % pvdisplay
--- Physical volume ---
PV Name /dev/md0
VG Name test
PV Size 6.99 GB / not usable 1.50 MB
Allocatable yes (but full)
PE Size (KByte) 4096
Total PE 1788
Free PE 0
Allocated PE 1788
PV UUID 6i6MZH-7Lv7-ZcXb-yr2W-ShFz-2c13-srXd07
address@hidden /root % vgdisplay
--- Volume group ---
VG Name test
System ID
Format lvm2
Metadata Areas 2
Metadata Sequence No 2
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 1
Open LV 1
Max PV 0
Cur PV 1
Act PV 1
VG Size 6.98 GB
PE Size 4.00 MB
Total PE 1788
Alloc PE / Size 1788 / 6.98 GB
Free PE / Size 0 / 0
VG UUID aCTAWb-xbpL-3oSp-1V2X-Oy3T-E2D7-gJoCZG
address@hidden /root % lvdisplay
--- Logical volume ---
LV Name /dev/test/test
VG Name test
LV UUID IzYAb2-9y4u-HqB2-27bD-ihvH-uCmn-2a1xlH
LV Write Access read/write
LV Status available
# open 2
LV Size 6.98 GB
Current LE 1788
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:0
--------------------------------------------------------------------------------------------------------------------------
sysresccd / # grub-install '(md0)'
#/usr/sbin/grub-install (md0)
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=device
/boot/grub
error: We don't support multiple metadata areas
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=fs
--device /dev/mapper/test-test
error: We don't support multiple metadata areas
grub-probe: error: no mapping exists for `test-test'
Auto-detection of a filesystem module failed.
Please specify the module with the option `--modules' explicitly.
--------------------------------------------------------------------------------------------------------------------------
sysresccd / # grub-install.org '(md0)' --modules='biosdisk pc raid
mdraid lvm reiserfs'
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=device
/boot/grub
error: We don't support multiple metadata areas
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=fs
--device /dev/mapper/test-test
error: We don't support multiple metadata areas
grub-probe: error: no mapping exists for `test-test'
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map
--target=abstraction --device /dev/mapper/test-test
error: We don't support multiple metadata areas
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=drive
--device /dev/mapper/test-test
error: We don't support multiple metadata areas
#/usr/bin/grub-mkimage --output=/boot/grub/core.img
--prefix=(test-test)/boot/grub biosdisk pc raid mdraid lvm reiserfs
biosdisk lvm
#/usr/sbin/grub-setup --directory=/boot/grub
--device-map=/boot/grub/device.map
(md0)
error: We don't support multiple metadata areas
grub-setup: error: no mapping exists for `test-test'
--------------------------------------------------------------------------------------------------------------------------
Grub works (after reboot) but I can't access filesystem, LVM module is
not present
====== Added second partition
Disk /dev/sda: 77825 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
Device Boot Start End #cyls #blocks Id System
/dev/sda1 * 0+ 607 608- 4883728+ da Non-FS data
/dev/sda2 608 1215 608 4883760 da Non-FS data
/dev/sda3 0 - 0 0 0 Empty
/dev/sda4 0 - 0 0 0 Empty
Disk /dev/sdb: 77825 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
Device Boot Start End #cyls #blocks Id System
/dev/sdb1 * 0+ 607 608- 4883728+ da Non-FS data
/dev/sdb2 608 1215 608 4883760 da Non-FS data
/dev/sdb3 0 - 0 0 0 Empty
/dev/sdb4 0 - 0 0 0 Empty
--------------------------------------------------------------------------------------------------------------------------
RAID/LVM like above
--------------------------------------------------------------------------------------------------------------------------
#/usr/sbin/grub-install (md0)
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=device
/boot/grub
error: no such partition
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=fs
--device /dev/mapper/test-test
error: no such partition
grub-probe: error: no mapping exists for `test-test'
Auto-detection of a filesystem module failed.
Please specify the module with the option `--modules' explicitly.
--------------------------------------------------------------------------------------------------------------------------
sysresccd / # grub-install.org '(md0)' --modules="biosdisk pc raid
mdraid lvm reiserfs"
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=device
/boot/grub
error: no such partition
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=fs
--device /dev/mapper/test-test
error: no such partition
grub-probe: error: no mapping exists for `test-test'
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map
--target=abstraction --device /dev/mapper/test-test
error: no such partition
#/usr/sbin/grub-probe --device-map=/boot/grub/device.map --target=drive
--device /dev/mapper/test-test
error: no such partition
#/usr/bin/grub-mkimage --output=/boot/grub/core.img
--prefix=(test-test)/boot/grub biosdisk pc raid mdraid lvm reiserfs
biosdisk lvm
#/usr/sbin/grub-setup --directory=/boot/grub
--device-map=/boot/grub/device.map
(md0)
grub-setup: error: no mapping exists for `test-test'
--------------------------------------------------------------------------------------------------------------------------
Second partition is unsed, however grub stoped complaining about
"multiple metadata". Desipte that I still can't boot system.
===== Added RAID10 using second partition pair, I'll try to boot system
from it
sysresccd / # grub-mkimage -o /boot/grub/core.img biosdisk pc raid
mdraid lvm reiserfs
#/usr/bin/grub-mkimage -o /boot/grub/core.img biosdisk pc raid mdraid
lvm reiserfs
sysresccd / # grub-setup '(md1)'
#/usr/sbin/grub-setup (md1)
grub-setup: error: Core image is too big for embedding, but this is
required when
the root device is on a RAID array or LVM volume.
--------------------------------------------------------------------------------------------------------------------------
After seeing that error message I created core.img without lvm module
and rebooted computer (I was able to boot system),
then I loaded lvm module but I couldn't see any lvm volumes (despite
fact that I saw md0 containing it). I tried to rmmod/inmod
again but it didn't help.
OK so here are my thoughts - most ppl use ext2 for /boot, I used
reiserfs which has 4k larger module so LVM just don't fit info core.img.
Using grub-install/setup on address@hidden always ends up with some errors I
just couldn't see it before (those errors were discarted
somewhere along the way ?) but it doesn't explain why I couldn't see lvm
volumes desipe the fact that loaded lvm module from disk.
Perhaps I should try to create LVM on md1 and boot from md0 to see what
will happen ?
Anyway there is something wrong with LVM+RAID on single partition ;)
It's caused by size limitation of mbr, which is normally 63 sectors.
When you use the following modules:
biosdisk pc reiserfs lvm mdraid raid6 raid6rec
The size of core.img would exceed 63 sectors, so that it can't be
installed to mbr.
Is it possible to use more sectors by leaving some unused space before
fisrt partion ?