[RFC] shuffle, or juggle, or ...

[Jose E. Marchesi]

Hi peeps!

So I was writing a pickle for poking at images containing CADR lisp
machine micro-code, when I realized these images use a peculiar
byte-encoding, or endianness: the bytes ABCD are stored like BADC.

I think this is php endian.  This triggered a very useful brainstorming
in IRC with Egeyar, that resulted in a first design of a super-cool and
super-flexible way to support generalized endianness in the IOS.
Something like:

  .set word-size N (defaults to 1 byte)
  .set group-size N (defaults to 0 with means unlimited)
  .set endian {big,ittle} (sets the word endianness)
  .set group-endian {big,little} (sets the group endianness)
  [User with normal needs will just use .set endian {big,little}]

This should cover "normal" endianness and also weird ones like php and
honeywell.

But anyway, today we just support regular garden variety big and little
endian, so I was kind of stuck.  Then I thought about shuffling the
bytes myself so I could test my micro-code pickle. 

But why not making it useful for everyone?  So I wrote the Poke command
below:

fun shuffle = (off64 from = 0#B,
               off64 size = 0#B,
               string from_pattern = "",
               string to_pattern = "",
               off64 ent_size = 1#B,
               int from_ios = get_ios,
               int to_ios = from_ios) void:
{
  type Entity = struct { char code; bit[ent_size] value; };
  
  var to = from + size;
  var end = from;

  while (from < to)
    {
      var collected = Entity[]();

      fun lookup_entity = (char code) Entity:
      {
        for (ent in collected where ent.code == code)
          return ent;
        return Entity {};
      }

      for (var i = 0; i < from_pattern'length; i++, from += ent_size)
        collected += [Entity { code = from_pattern[i],
                               value = unmap bit[ent_size] @ from_ios : from}];

      for (var i = 0; i < to_pattern'length; i++, end += ent_size)
        {
          var ent = lookup_entity (to_pattern[i]);
          if (ent.code != '\0')
            bit[ent_size] @ to_ios : end = ent.value;
        }
   }
}

This works like this: the user specifies a range of memory in some IO
spaces (defaults to the current IO space) a destination IO space (which
defaults to the origin IO space) and entity size (which defaults to 1#B)
and a transformation pattern that defines how elements of that size are
to be shuffled/juggled around.  Examples:

Convert a file from mcr endianness to regular big-endian:

  shuffle :size iosize :from_pattern "abcd" :to_pattern "badc"

Convert a file from 8-bit to 7-bit ASCII:

  shuffle :size iosize :ent_size 1#b :from_pattern "12345678" :to_pattern 
"1234567"

You can use any character in the patterns.

Applied to my specific problem:

(poke) .file ucadr.mcr
(poke) load mcr
(poke) MCR_Section @ 0#B
unhandled constraint violation exception
(poke) dump
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 0000 0100 0000 0000 0000 c528 0000 0008  ...........(....
00000010: 0000 a704 0000 0000 0000 0090 0000 0008  ................
00000020: 0100 a720 0000 0000 0000 0000 0000 0000  ... ............
00000030: 0000 0000 0000 0000 0000 0000 0000 0208  ................
00000040: 009c 6360 0000 0300 009c cb10 0000 0008  ..c`............
00000050: 7c02 2731 0000 6001 8000 2810 0000 0008  |.'1..`...(.....
00000060: 80aa dbd0 0000 0008 7c02 2731 0000 6101  ........|.'1..a.
00000070: 8000 2810 0000 0218 c0a8 a010 0000 6209  ..(...........b.
(poke) shuffle :size iosize :from_pattern "1234" :to_pattern "2143"
(poke) dump
76543210  0011 2233 4455 6677 8899 aabb ccdd eeff  0123456789ABCDEF
00000000: 0000 0001 0000 0000 0000 28c5 0000 0800  ..........(.....
00000010: 0000 04a7 0000 0000 0000 9000 0000 0800  ................
00000020: 0001 20a7 0000 0000 0000 0000 0000 0000  .. .............
00000030: 0000 0000 0000 0000 0000 0000 0000 0802  ................
00000040: 9c00 6063 0000 0003 9c00 10cb 0000 0800  ..`c............
00000050: 027c 3127 0000 0160 0080 1028 0000 0800  .|1'...`...(....
00000060: aa80 d0db 0000 0800 027c 3127 0000 0161  .........|1'...a
00000070: 0080 1028 0000 1802 a8c0 10a0 0000 0962  ...(...........b
(poke) MCR_Section @ 0#B
MCR_Section {
  code=0x1,
  start=0x0,
  size=0x28c5
}

But, before adding this command to poke, there are two problems, a small
one and a big one.

The small problem is that the implementation above is slow, since it is
very general and peeks/pokes arrays of bits.  It is trivial to support
more specialized versions for often-used entities (like bytes) where the
stuff is peeked/poked in units of these entities.  So this is just a
little nuisance which is easily fixed.

The BIG problem and blocking, however, is that certain people
(i.e. Mohammad) don't think this command should be called `shuffle'
because, they say, it denotes randomness.

I like `shuffle', I guess I could live with `juggle', but I would like
to hear your opinion and more suggestions are welcomed.

Ideas? :)