Re: OpenMPIC controller emulation in qemu ?

[BALATON Zoltan]

On Sat, 18 May 2024, Andrew Randrianasulu wrote:
> On Sat, May 18, 2024 at 10:24 PM BALATON Zoltan <balaton@eik.bme.hu> wrote:
>
> > On Sat, 18 May 2024, Andrew Randrianasulu wrote:
> > > Using attached patch I get this  new dmesg:
> > >
> > > Quiescing Open Firmware ...
> > > > > of_client_interface: quiesce
> > > > > of_client_interface return:
> > > Booting Linux via __start() @ 0x01000000 ...
> > > Hello World !
> > > [    0.000000] Total memory = 512MB; using 1024kB for hash table
> > > [    0.000000] Activating Kernel Userspace Execution Prevention
> > > [    0.000000] Activating Kernel Userspace Access Protection
> > > [    0.000000] Linux version 5.13.12_1 (voidlinux@voidlinux) (gcc (GCC)
> > 10.2.1 20201203, GNU ld (GNU Binutils) 2.35.1) #1 SMP Thu Aug 19 14:12:26
> > UTC 2021
> > > [    0.000000] ioremap() called early from pmac_feature_init+0xd4/0xad4.
> > Use early_ioremap() instead
> > > [    0.000000] Found UniNorth memory controller & host bridge @
> > 0xf8000000 revision: 0x07
> > > [    0.000000] Mapped at 0xf73c0000
> > > [    0.000000] ioremap() called early from probe_one_macio+0x17c/0x2b4.
> > Use early_ioremap() instead
> > > [    0.000000] Found a Keylargo mac-io controller, rev: 0, mapped at
> > 0x(ptrval)
> > > [    0.000000] PowerMac motherboard: PowerMac G4 AGP Graphics
> > > [    0.000000] ioremap() called early from udbg_scc_init+0x1e4/0x3f8.
> > Use early_ioremap() instead
> > > [    0.000000] boot stdout isn't a display !
> > > [    0.000000] ioremap() called early from find_via_cuda+0xb4/0x404. Use
> > early_ioremap() instead
> > > [    0.000000] Using PowerMac machine description
> > > [    0.000000] printk: bootconsole [udbg0] enabled
> > > [    0.000000] CPU maps initialized for 1 thread per core
> > > [    0.000000] -----------------------------------------------------
> > > [    0.000000] phys_mem_size     = 0x20000000
> > > [    0.000000] dcache_bsize      = 0x20
> > > [    0.000000] icache_bsize      = 0x20
> > > [    0.000000] cpu_features      = 0x000000000501a00a
> > > [    0.000000]   possible        = 0x00000000277de14a
> > > [    0.000000]   always          = 0x0000000001000000
> > > [    0.000000] cpu_user_features = 0x9c000001 0x00000000
> > > [    0.000000] mmu_features      = 0x00000001
> > > [    0.000000] Hash_size         = 0x100000
> > > [    0.000000] Hash_mask         = 0x3fff
> > > [    0.000000] -----------------------------------------------------
> > > [    0.000000] ioremap() called early from pmac_setup_arch+0x118/0x290.
> > Use early_ioremap() instead
> > > [    0.000000] ioremap() called early from pmac_nvram_init+0x150/0x53c.
> > Use early_ioremap() instead
> > > [    0.000000] nvram: Checking bank 0...
> > > [    0.000000] Invalid signature
> > > [    0.000000] Invalid checksum
> > > [    0.000000] nvram: gen0=0, gen1=0
> > > [    0.000000] nvram: Active bank is: 0
> > > [    0.000000] nvram: OF partition at 0xffffffff
> > > [    0.000000] nvram: XP partition at 0xffffffff
> > > [    0.000000] nvram: NR partition at 0xffffffff
> > > [    0.000000] Zone ranges:
> > > [    0.000000]   DMA      [mem 0x0000000000000000-0x000000001fffffff]
> > > [    0.000000]   Normal   empty
> > > [    0.000000]   HighMem  empty
> > > [    0.000000] Movable zone start for each node
> > > [    0.000000] Early memory node ranges
> > > [    0.000000]   node   0: [mem 0x0000000000000000-0x000000001fffffff]
> > > [    0.000000] Initmem setup node 0 [mem
> > 0x0000000000000000-0x000000001fffffff]
> > > [    0.000000] percpu: Embedded 22 pages/cpu s59884 r8192 d22036 u90112
> > > [    0.000000] Built 1 zonelists, mobility grouping on.  Total pages:
> > 129920
> > > [    0.000000] Kernel command line: console=ttyPZ0
> > > [    0.000000] Dentry cache hash table entries: 65536 (order: 6, 262144
> > bytes, linear)
> > > [    0.000000] Inode-cache hash table entries: 32768 (order: 5, 131072
> > bytes, linear)
> > > [    0.000000] mem auto-init: stack:off, heap alloc:on, heap free:off
> > > [    0.000000] Kernel virtual memory layout:
> > > [    0.000000]   * 0xf7bdf000..0xfffff000  : fixmap
> > > [    0.000000]   * 0xf7400000..0xf7800000  : highmem PTEs
> > > [    0.000000]   * 0xf7338000..0xf7400000  : early ioremap
> > > [    0.000000]   * 0xe1000000..0xf7338000  : vmalloc & ioremap
> > > [    0.000000] Memory: 500988K/524288K available (10912K kernel code,
> > 1444K rwdata, 2512K rodata, 1420K init, 516K bss, 23300K reserved, 0K
> > cma-reserved, 0K highmem)
> > > [    0.000000] random: get_random_u32 called from
> > __kmem_cache_create+0x38/0x57c with crng_init=0
> > > [    0.000000] SLUB: HWalign=32, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
> > > [    0.000000] ftrace: allocating 29527 entries in 87 pages
> > > [    0.000000] ftrace: allocated 87 pages with 5 groups
> > > [    0.000000] rcu: Hierarchical RCU implementation.
> > > [    0.000000] rcu:     RCU restricting CPUs from NR_CPUS=2048 to
> > nr_cpu_ids=4.
> > > [    0.000000]  Trampoline variant of Tasks RCU enabled.
> > > [    0.000000]  Rude variant of Tasks RCU enabled.
> > > [    0.000000]  Tracing variant of Tasks RCU enabled.
> > > [    0.000000] rcu: RCU calculated value of scheduler-enlistment delay
> > is 25 jiffies.
> > > [    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16,
> > nr_cpu_ids=4
> > > [    0.000000] NR_IRQS: 512, nr_irqs: 512, preallocated irqs: 16
> > > [    0.000000] mpic: Setting up MPIC " MPIC 1   " version 1.2 at
> > 80040000, max 4 CPUs
> > > [    0.000000] mpic: ISU size: 64, shift: 6, mask: 3f
> > > [    0.000000] mpic: Initializing for 64 sources
> > > [    0.000000] GMT Delta read from XPRAM: 0 minutes, DST: on
> > > [    0.001114] clocksource: timebase: mask: 0xffffffffffffffff
> > max_cycles: 0x5c40939b5, max_idle_ns: 440795202646 ns
> > > [    0.002661] clocksource: timebase mult[28000000] shift[24] registered
> > > [    0.022081] Console: colour dummy device 80x25
> > > [    0.059474] printk: console [ttyPZ0] enabled
> > > [    0.059474] printk: console [ttyPZ0] enabled
> > > [    0.060950] printk: bootconsole [udbg0] disabled
> > > [    0.060950] printk: bootconsole [udbg0] disabled
> > > [    0.063583] pid_max: default: 32768 minimum: 301
> > > [    0.066030] LSM: Security Framework initializing
> > > [    0.068483] Yama: becoming mindful.
> > > [    0.075446] AppArmor: AppArmor initialized
> > > [    0.076811] Mount-cache hash table entries: 1024 (order: 0, 4096
> > bytes, linear)
> > > [    0.077566] Mountpoint-cache hash table entries: 1024 (order: 0, 4096
> > bytes, linear)
> > > smp_core99_probe
> > > [    0.140858] PowerMac SMP probe found 4 cpus
> >
> > You're still using too many CPUs. Try to get close to real hardware and
> > only try 2 CPUs at first so we don't have to care about things that don't
> > happen on real machine.
>
> yeah, but linux define all four :}
>
> core99_reset_cpu(struct device_node *node, long param, long value)
>
> const int dflt_reset_lines[] = { KL_GPIO_RESET_CPU0,
> KL_GPIO_RESET_CPU1,
> KL_GPIO_RESET_CPU2,
> KL_GPIO_RESET_CPU3 };
>
> and those defined in keylargo.h as
>
> #define KEYLARGO_GPIO_EXTINT_0 0x58
>
> #define KL_GPIO_RESET_CPU0 (KEYLARGO_GPIO_EXTINT_0+0x03)
> #define KL_GPIO_RESET_CPU1 (KEYLARGO_GPIO_EXTINT_0+0x04)
> #define KL_GPIO_RESET_CPU2 (KEYLARGO_GPIO_EXTINT_0+0x0f)
> #define KL_GPIO_RESET_CPU3 (KEYLARGO_GPIO_EXTINT_0+0x10)

Still it may be simpler to deal with only 1 additional CPU first just to 
reduce complexity. So CPU reset seems to be done by core99_reset_cpu() 
which will:

1. Look for something in open firmware device tree - is that there with 
OpenBIOS, is it needed? Could be found by comparing to device tree from 
real machine but you'd need to find a PowerMac3,3 dual CPU dump as 
PowerMac3,1 only was single CPU. Otherwise it defaults to the above 
defines.

2. Poke mac-io to lower the reset line as far as I understand. Is this 
emulated in QEMU and connected to reset the CPUs? The mac_newworld.c seems 
to connect CPU reset to openpic not macio/keylargo or is that the same 
device? Are these KEYLARGO_GPIO_EXTINT_0 registers emulated correctly?

> > > [    0.142883] Processor timebase sync using GPIO 0x73
> > > [    0.143474] mpic: requesting IPIs...
> > > [    0.148795] CPU0: L2CR is 0
> > > [    0.164657] rcu: Hierarchical SRCU implementation.
> > > [    0.186502] smp: Bringing up secondary CPUs ...
> > > smp_core99_kick_cpu
> > > smp_core99_kick_cpu done
> > > [    5.227920] Processor 1 is stuck.
> > > smp_core99_kick_cpu
> > > smp_core99_kick_cpu done
> > > [   10.278140] Processor 2 is stuck.
> > > smp_core99_kick_cpu
> > > smp_core99_kick_cpu done
> > > [   15.309326] Processor 3 is stuck.
> >
> > I guess we need to implement the thing that resets the additional CPUs or
> > if that already happens then something else is wrong. I have no idea how
> > that works on real machine or what's in Linux or QEMU but the way to find
> > out is ro check the Linux code to see what does it poke to reset the other
> > CPU then check if that's handled in QEMU correctly and if not fix that.
> > (After that it will likely still hang because of IRQ routing but maybe
> > should see the CPUs at least.)
> >
> > > [   15.310457] smp: Brought up 1 node, 1 CPU
> > > [   15.336741] devtmpfs: initialized
> > > [   15.358958] Found UniNorth PCI host bridge at 0x00000000f2000000.
> > Firmware bus number: 0->0
> > > [   15.359907] PCI host bridge /pci@f2000000 (primary) ranges:
> > > [   15.362134]   IO 0x00000000f2000000..0x00000000f27fffff ->
> > 0x0000000000000000
> > > [   15.363327]  MEM 0x0000000080000000..0x000000008fffffff ->
> > 0x0000000080000000
> > > [   15.366636] clocksource: jiffies: mask: 0xffffffff max_cycles:
> > 0xffffffff, max_idle_ns: 7645041785100000 ns
> > > [   15.367915] futex hash table entries: 1024 (order: 3, 32768 bytes,
> > linear)
> > > [   15.407396] NET: Registered protocol family 16
> > > [   15.416447] audit: initializing netlink subsys (disabled)
> > > [   15.423487] audit: type=2000 audit(1716054549.076:1):
> > state=initialized audit_enabled=0 res=1
> > > [   15.433124] thermal_sys: Registered thermal governor 'fair_share'
> > >
> > > [   15.433296] thermal_sys: Registered thermal governor 'step_wise'
> > > [   15.444053] PCI: Probing PCI hardware
> > > [   15.454648] PCI host bridge to bus 0000:00
> > > [   15.455908] pci_bus 0000:00: root bus resource [io  0x0000-0x7fffff]
> > > [   15.457308] pci_bus 0000:00: root bus resource [mem
> > 0x80000000-0x8fffffff]
> > > [   15.458521] pci_bus 0000:00: root bus resource [bus 00-ff]
> > > [   15.459767] pci_bus 0000:00: busn_res: [bus 00-ff] end is updated to
> > ff
> > > [   15.464530] pci 0000:00:0b.0: [106b:001f] type 00 class 0x060000
> > > [   15.473861] pci 0000:00:0c.0: [106b:0022] type 00 class 0xff0000
> > > [   15.475845] pci 0000:00:0c.0: reg 0x10: [mem 0x80000000-0x8007ffff]
> > > [   15.481484] pci 0000:00:0d.0: [106b:003f] type 00 class 0x0c0310
> > > [   15.482883] pci 0000:00:0d.0: reg 0x10: [mem 0x80080000-0x800800ff]
> > > [   15.488606] pci 0000:00:0e.0: [1234:1111] type 00 class 0x030000
> > > [   15.490521] pci 0000:00:0e.0: reg 0x10: [mem 0x81000000-0x81ffffff
> > pref]
> > > [   15.494643] pci 0000:00:0e.0: reg 0x18: [mem 0x82000000-0x82000fff]
> > > [   15.501794] pci 0000:00:0e.0: reg 0x30: [mem 0x82010000-0x8201ffff
> > pref]
> > > [   15.503465] pci 0000:00:0f.0: [106b:0021] type 00 class 0x020000
> > > [   15.504560] pci 0000:00:0f.0: reg 0x10: [mem 0x82200000-0x823fffff]
> > > [   15.511499] pci_bus 0000:00: busn_res: [bus 00-ff] end is updated to
> > 00
> > > [   15.520044] pci_bus 0000:00: resource 4 [io  0x0000-0x7fffff]
> > > [   15.520695] pci_bus 0000:00: resource 5 [mem 0x80000000-0x8fffffff]
> > > [   15.541800] Kprobes globally optimized
> > > [   20.205864] iommu: Default domain type: Translated
> > > [   20.214125] pci 0000:00:0e.0: vgaarb: setting as boot VGA device
> > > [   20.215070] pci 0000:00:0e.0: vgaarb: VGA device added:
> > decodes=io+mem,owns=io+mem,locks=none
> > > [   20.216002] pci 0000:00:0e.0: vgaarb: bridge control possible
> > > [   20.216606] vgaarb: loaded
> > > [   20.220636] SCSI subsystem initialized
> > > [   20.253578] clocksource: Switched to clocksource timebase
> > > [   20.452951] VFS: Disk quotas dquot_6.6.0
> > > [   20.453727] VFS: Dquot-cache hash table entries: 1024 (order 0, 4096
> > bytes)
> > > [   20.460869] AppArmor: AppArmor Filesystem Enabled
> > > [   20.495964] NET: Registered protocol family 2
> > > [   20.498712] IP idents hash table entries: 8192 (order: 4, 65536
> > bytes, linear)
> > > [   20.521739] tcp_listen_portaddr_hash hash table entries: 512 (order:
> > 0, 6144 bytes, linear)
> > > [   20.522799] TCP established hash table entries: 4096 (order: 2, 16384
> > bytes, linear)
> > > [   20.523935] TCP bind hash table entries: 4096 (order: 3, 32768 bytes,
> > linear)
> > > [   20.524986] TCP: Hash tables configured (established 4096 bind 4096)
> > > [   20.529295] MPTCP token hash table entries: 512 (order: 1, 8192
> > bytes, linear)
> > > [   20.530582] UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
> > > [   20.531628] UDP-Lite hash table entries: 256 (order: 1, 8192 bytes,
> > linear)
> > > [   20.536784] NET: Registered protocol family 1
> > > [   20.537953] NET: Registered protocol family 44
> > > [   20.542641] PCI: CLS 0 bytes, default 32
> > > [   20.550784] Thermal assist unit using workqueue, shrink_timer: 2000 ms
> > > [   20.567668] Initialise system trusted keyrings
> > > [   20.572356] workingset: timestamp_bits=14 max_order=17 bucket_order=3
> > > [   20.594107] zbud: loaded
> > > [   20.601453] Key type asymmetric registered
> > > [   20.602101] Asymmetric key parser 'x509' registered
> > > [   20.602938] Block layer SCSI generic (bsg) driver version 0.4 loaded
> > (major 250)
> > > [   20.604717] io scheduler mq-deadline registered
> > > [   20.605353] io scheduler kyber registered
> > > [   20.606241] io scheduler bfq registered
> > > [   20.616132] Using unsupported 800x600 (null) at 81000000, depth=32,
> > pitch=3200
> > > [   20.643222] Console: switching to colour frame buffer device 100x37
> > > [   20.652163] fb0: Open Firmware frame buffer device on /pci@f2000000
> > /QEMU,VGA@e
> > > [   20.655049] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
> > > [   20.665430] pmac_zilog: 0.6 (Benjamin Herrenschmidt <
> > benh@kernel.crashing.org>)
> > > [   20.666517] Serial: MPC52xx PSC UART driver
> > > [   20.667710] Non-volatile memory driver v1.3
> > > [   20.668168] Linux agpgart interface v0.103
> > > [   20.672974] MacIO PCI driver attached to Keylargo chipset
> > > [   20.681690] 0.00013020:ch-a: ttyPZ0 at MMIO 0x80013020 (irq = 37,
> > base_baud = 230400) is a Z85c30 ESCC - Serial port
> > > [   20.685655] 0.00013000:ch-b: ttyPZ1 at MMIO 0x80013000 (irq = 36,
> > base_baud = 230400) is a Z85c30 ESCC - Serial port
> > > [   20.691584] Macintosh Cuda and Egret driver.
> >
> > This reminds me that mac99 is not actually PowerMac3.1 that it claims to
> > be so make sure to always use -M mac99,via=pmu instead which is what
> > actually like a PowerMac. Also I don't know if PMU plays a role in SMP or
> > not but let's try to match real hardware here.
> >
> > Here's the patch inline so it's easier to reply:
> >
> > --- /home/guest/botva/src/src/qemu/hw/ppc/mac_newworld.c        2024-03-27
> > 06:44:12.068828324 +0300
> > +++ hw/ppc/mac_newworld.c       2024-05-18 20:48:32.709184875 +0300
> > @@ -128,6 +128,15 @@
> >       cpu->env.nip = PROM_BASE + 0x100;
> >   }
> >
> > +
> > +static void ppc_core99_reset_sec(void *opaque)
> > +{
> > +    PowerPCCPU *cpu = opaque;
> > +    CPUState *cs = CPU(cpu);
> > +    cpu_reset(cs);
> > +    cs->exception_index = EXCP_HLT;
> > +}
> > +
> >   /* PowerPC Mac99 hardware initialisation */
> >   static void ppc_core99_init(MachineState *machine)
> >   {
> > @@ -160,10 +169,27 @@
> >       for (i = 0; i < machine->smp.cpus; i++) {
> >           cpu = POWERPC_CPU(cpu_create(machine->cpu_type));
> >           env = &cpu->env;
> > +       CPUState *cs;
> > +       cs = CPU(cpu);
> >
> > ! I think ou can cast cpu to OBJECT(cpu) then no cs is needed.
> >
> > +
> > +       /*
> > +       * Secondary CPU starts in halted state for now. Needs to change
> > +        * when implementing non-kernel boot.
> > +         */
> > +         object_property_set_bool(OBJECT(cs), "start-powered-off", i != 0,
> > +                                 &error_abort);
> >
> > ! Now I'm not sure the ppc_core99_reset_sec() is really needed or this
> > ! property is enough. e500 seems to do both and some others too but these
> > ! seem to do tha same thing and halt CPU on reset. Whay do we have two
> > ! ways for this and is both needed?
> >
> > +        //qdev_realize_and_unref(DEVICE(cs), NULL, &error_fatal);
> >
> > ! If you remove raliziation of CPU object how does it even work? I think
> > ! this line is still needed,
>
>
> with it machine not even start and assert somewhere ...

Where and why? What's the assert? Without realizing the object might not 
completely initialised so I'm surprised it works at all. Instead of 
removing realize we should find out why it asserts as that might mean 
there's some problem that needs to be fixed.

Regards,
BALATON Zoltan