[Fenfire-dev] Re: [vos-d] a 3D interface vision

[Peter Amstutz]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Warning: this is a long email written over the course of a few hours, I
don't feel like checking it over for grammer/spelling so read at your own
risk!  :-)

On Wed, 22 Oct 2003, Toni Alatalo wrote:

> Oh, a general (philosophical?) note about the beginning sentence: you
> may say that we live in a 3d world, but don't we also live in social
> and mental (conceptual) worlds that have quite different structures
> than being points in a xyz coordinate system (although they may map to
> such in the 'wetware implementation', i.e. the brain, but that's
> another discussion i guess :o :) . And as a concrete manifestation of
> the physical 3d world already exists, isn't it perhaps more interesting
> to explore the abstract/conceptual worlds with other kinds of relations
> etc? And of course that's what much of use of computing (e.g. e-mailing
> :) is about.

A very good point, to which I have several responses

1) I think it will be more intuitive to mimic the real world, and then
make obvious exceptions (that being the point of VR) than to build a space
from scratch with its own rules.  The latter may be more powerful in
certain contexts, though.  It depends on what your task is.

2) I've seen VRML worlds which have a few interesting areas, and a very
large amount of empty space (with maybe a few cheesy-looking trees) in
between, with the idea that this is "realistic".  Mostly this is stupid.
I agree that a 1:1 mapping of the real world into the virtual world is
often not desirable or necessary, and for VOS I am much more interested in
a sort of hyperspace of rooms interlinked with portals, where you can see
through from one room to another but don't share the same Euclidian space.

3) The VOS project originally approached 3D interfaces from the direction
of multiuser chat, where the point is to be in a realistic space.  Lately
I have been decided that chat is something of a gimmick and only
interesting for a few minutes (not to mention there are dozens of 3D chat
programs out there) and a much more interesting area is 3D UIs --- how
could one use this to actually get work done?

> i (and 'we' referring to people sharing the Fenfire vision, or anyone
> with common sense perhaps :) agree that something being near another
> thing is very relevant. but, unlike in the physical 3d, nearness need
> not be the distance of two objects in a 3d space, but can be defined
> and dealt with by using other kind of relations, too. i try to approach
> the whole point by discussing this in the following:

A small digression in response to this...

For my "day job" (research assistant) I work for the Center for
Intellegent Information Retrieval (CIIR).  The field of information
retrieval (IR) deals with statistical analysis of text for the purposes of
answering queries about that text, such as finding documents that match
some keywords.

One IR technique for managing large numbers of documents is known as
"clustering".  This involves taking a collection of documents and
automatically partitioning those documents into clusters, where a cluster
is a set of documents which are similar to one another based on some
metric.  This is useful for applications such as "topic detection and
tracking" (TDT) where you are monitoring a stream of news stories and you
want stories about the same event or topic to be filed together; generally
the contents of a cluster are all the stories about a certain topic.

So my job is to write a (conventional) GUI for this system, so that the
user can run queries, browse clusters and read documents.  The application
works much like a two-pane file browser, with a tree control on the left
side and content (icons, document text, etc) on the right side.

So a complex relation between items (clusters of documents) is represented
using a familiar idiom (folders containing files).  The user interface
doesn't have to expose the full complexity of the underlying relation, but
it is there.

Similarly, when I talk about a 3D interface, I am concerned here about
what the user actually perceives.  My argument for "nearness" holds,
because I am talking about things being presented as having 3D positions
which are near to other things.  The deeper relations between things may
be implicit (the user has manually sorted them) or represented at another
level, in which case it makes perfect sense to be able to reorder your
virtual world along various axes of relations between things.

Finally, the VOS object model allows you to attach arbitrary information
to vobjects, so several views or descriptions of the same logical "thing"
can be incorporated into a single vobject.

> is an n-dimensional space, where in each dimension an object may have
> exactly one other object connected it. this makes it locally euclidean,
> i.e. similar to physical 3d and the simulations of it in the sense that
> for every object there may be one object next to it in one direction.

Why not a doubly linked list?  It seems me that in general you want to be
able to traverse an axis in either direction.  In 3D you have 3 axes, but
by you description it sound like "left" and "right" and "up" and "down"
are all seperate axes, making for a total of 6 dimensions rather than 3.

I'd also like to mention that the vobject model (without getting into it
here, lots of info on the interreality.org web page) is perfectly capable
of directly representing zigzag/RDF as you describe it :-)

> for example, in FenPDF which is about reading research articles and
> making notes about them, there are no directories with named files nor
> directories, nor icons representing anything etc. -- just the articles
> and paper notes (canvases) floating around, structured by the
> connections the user makes. there are menus for actions though (both at
> the top and contextual popups) (the data is of course in the end stored
> in the filesystem, but the user normally never sees the files)

One problem is that filesystems lack good metadata.  To know anything
about a file beyond a fixed set of generic information requires being able
to parse out the file format.  What I would like to see would be a breakup
of the monolithic notion of a "file" into pieces representing the
component data.  For an image, this might be height, width, bit depth; for
a text document, it may be title, number of pages, default font; for a 3D
object located in a world it would include position, orientation and so
forth.  Now a file would declare that it supports various sets of metadata
something like "author" would be fairly universal but "number of pages" is
much more specific.

Now a utility which knew about "images" could read at least a little bit
of information about *all* images, even if a certain image used a weird
encoding it didn't understand (fractal wavelets or something :-)
Naturally you could extend the metadata as much as necessary for specific
tasks, but the key is to have broad, generic types that could allow one to
work on a large class of things without knowing the specifics.

It's a familiar idea in object-oriented programming (code operates on
abstract superclasses, the real work gets done in the subclasses, but the
code doesn't always need to know about that).  Also this will sound very
familiar to those of you familiar with VOS :-)

> (i seem to be quoting in length anyhow..)
>
> hm, i don't quite see how the shelves would work - would it be like
> hiding windows/applications that are not worked on right now, to reduce
> clutter? (i often do that by pressing apple-h on macs)

No, the idea is that so long as the shelves are still with in your reach,
you have effectively expanded your storage space without affecting the
usability of the desk itself.  You can still grab something off a shelf
with little effort.  It's kind of a silly analogy (arguably buying a
bigger monitor with a higher resolution would acomplish the same thing ;-)
and the point was really to introduce the idea that 3D is superior for
spatial metaphors.

> the note about the clutter is interesting also because it is related to
> the counter-arguments against 3d interfaces that i should be coming at,
> of which one is about placement on screen. but before that, we should
> note that (like you indirectly partly refer in the following, when
> mention the "Hackers" scene etc.) some things are certainly known to be
> applicable, like having remote things fade/shrink away.
> that some sort of pseudo-3d was prototyped also in gzigzag with
> VanishingViews, see
> http://an.org/gzz/screenshot-gzigzag-stretchvanishing-10.png for a
> small screenshot demonstrating that (that cell there that is being
> edited actually sets the depth used, 10 is impractical (produces
> clutter) but seemed good for showing the effect)

Also keep in mind of course that existing GUIs already have a rough notion
of a Z axis, in the sense of Z-sorting overlapping windows to determine
precedence.

> in FenPDF, if there are a lot of links to/from one place, it may result
> in clutter but that is very fluently solved by zooming in closer to the
> current focus, which makes more remote connections go out of sight.

Right.  I do need to download FenPDF and try it out, it sound like a lot
of fun :-)

> didn't they use an actual 3d file browser in one of those films, that
> had been made (perhaps used to some extent, even?) on SGI (IRIX)? there
> is this quote from it when some of the characters said, seeing the 3d
> infoscape: 'I know this, this is UNIX!" and started working on it :)

That was "Jurrasic Park" and the file browser was an actual IRIX programs
called FSN (File System Navigator).

> btw there is also a 3d Finder (the mac file manager) 'replacement'
> called 3D OS X, see http://www.acm.uiuc.edu/macwarriors/projects/3dosx/

Are you familiar with the site Nooface.org?  They have a sidebar with
links to a bunch of 3D file browsers.  I'm sure they would be interested
in Fenfire, it's certainly relevant.

> http://www.apple.com/macosx/features/fastuserswitching/ (see the
> bottom, each desktop is a texture on a cube that rotates when switching
> the active desktop (default one/user i think but can surely be hacked))

Honestly that's a pretty frivioulous use of 3D, and actually there
something similar for Linux/X already for switching virtual desktops.

> all that is fine, but are there benefits in having the office as a 3d
> space? i mean, for the user interface to resemble a physical space? us
> being vague about the definition of 3d is probably a problem in this
> discussion, hopefully the points get across anyhow.

So one impression I am getting is that we may have two slightly different
user models.  In my model, the user is immersed in the world, the view you
have is a first-person view; navigation is similar to that used by first
person games.  There are certainly other ways of viewing 3D, a common
alternative used by 3D modeling programs in the "orbit" view, where your
view is fixed on a certain point ahead of you and you change the
position/orientation of the "world" rather than the "camera".

So an office-like space makes sense in the context of an immersive view.
I agree that in a different view mode it may be less useful.

> > office: if it is too small, it can be easily made larger without
> > inconveniencing the next door neighbor.  The user can use whatever ad
> (...)
> > proximity ("I know I left that file near the potted plant...").
>
> here are again two central things that have been discussed: growing
> larger in space and being near. the point about zigzag was made that
> there, with n dimensions, there is in a way infinite amount of space
> everywhere, and this is (even more) so in the case of rdf.  this means
> that even though there are already many things in a place, or near some
> object, there need not be basically any limit to the amount of
> additional things that can be there, too. and there also does not need
> to be limits to the amount of places a single thing can be in
> simultaneously, etc.
>
> structure and limits, filtering etc. are of course needed in the user
> interface (e.g. visualisations, potential actions etc.).

Well, first, when I say the office is too small, I mean that the extent of
the "walls" could be easily resized.  More of a visual issue than
anything, as the office would exist in it's own sector (private coordinate
system) and so would exist independent of any other second, except through
portals to other sectors.

> in physical-like 3d, objects are easily hidden behind others in vain
> etc. of course you can take that as a challenge, but i don't see it
> working fluently (e.g. there is the important object by the flower pot,
> and you happen to stand somewhere else in a room looking at something
> related - what if the pot happens to block the line of sight .. would
> you make the program move the pot away, semitransparent or even
> invisible, the object that's behind it move in sight (what if someone
> else was looking at it from another angle and it then becomes hidden
> for her, would you then clone it?) or what?) - departing from
> simulating the physical world and having more abstract structures on
> screen seems to be a way to avoid these issues.

So this is a good example of how some of the goals of "virtual reality"
are in fact in direct conflict with some of the ideas you are putting
forth about 3D UIs.  In the real world, of course something behind
something else is going to be obscured; for VR we want to mimic that.
This is where we're coming from with VOS.  I also think that spatial
reasoning is stronger when you have an absolute sense of where something
is, rather than it's position constantly shifting in response to other
actions.

> fun to see that you come at this, too, 'cause it's been one of the
> fields where also Fenfire has been working and actually the area where
> i was participating last spring. hopefully you'll find our paper about
> it relevant (the long version is outdated but a new one is coming..):

This looks neat.  I'll have to take a look.  One problem though is that it
only works for static data, that is, once published it becomes very hard
to change a document (maybe you can supercede it with a new version?)
Freenet has a similar problem.  VOS is explicitly dynamic, so the host is
coded in the resource name like HTTP URIs.

> this area is where VOS seems strong to me and is also something that
> Fenfire is not addressing (currently), real-time interactions over the
> net like in multiplayer games i mean.

I (obviously) think that the VOS model is a very strong way of
describing/representing distributed dynamic data structures, the likes of
which are needed to describe multiuser 3D spaces.  We've come quite far
and found it useful in a wide variety of problem domains, including many
completely outside of 3D (VOS itself is actually content-neutral,
describing 3D data is an example of an extension upon the base system).

That said, the current implementation is on it's last legs and is going to
be thrown out and rewritten soon to use threads :-)  The design is sound,
the implementation not quite so much so.

> > In summary:
> > * Humans have strong spacial and visual thinking
>
> agreed. one innovation related to visual thinking in Fenfire BTW is
> generating unique textures based on content-id's (for the articles and
> note-papers to be better identifyable, I will always remember the
> unpublished VOS-paper as this greenish thing,
> http://an.org/FirstAndLast/vos.jpg :) you can try the current
> implementation of the unique textures in
> http://himalia.it.jyu.fi/paperbot/paper

Wow, that is a really clever idea.  I'm impressed :-)  It reminds me a bit
of the xchat IRC client which assigns colors to the various denizens of an
irc channel, probably based on some sort of hash of their nick.  Using
color to quickly distinguish things is good strategy.

> it might also be clumsy. oh the point about automation reminded of that
> making macros / scripting like activities should be supported

It's difficult to establish what is/is not clumsy without actually having
tried it out.  One of the points I made is simply that there arn't any
good frameworks for experimenting with 3D UIs, so just testing out an idea
takes forever as you have to code half a 3D engine from scratch to do it.
We're trying to position VOS as something that could be used as such an
enabling technology, but we're not quite there yet.

> > * Architecture should erase as much as possible the difference between
> > local and network resources
>
> "but not more" (or how did the saying go), yes, and in some cases
> location independent identifiers might be a good solution for that
> (well, Fenfire uses them for basically everything :o) - perhaps not so
> in real-time interaction though were vop-URLs may be fine.

Hmm, if you replaced the web with a semantic network that made search
engines like google redundant, would they equip the googlebot with lasers
and heat seeking missles and send it to your doorstep? :-)

> applications is that they tend to lock their data in so that no other
> application can refer to it. for example, there are e-mails discussing
> articles and 'word processing' applications for writing articles, but
> often they don't work together, e.g. there can't be interconnections
> between parts of the data. as said, Fenfire uses RDF for this.

I think this ties a little bit with what I said above about metadata and
file formats.  Also, using VOS can "open up" an application's state to the
world, by representing it in a way that can be accessed over the network.
I think what you have in mind is a little different, though?

> that sounds like something to keep in mind..
>
> ok, hopefully this mess brought something up and didn't cause too bad
> misunderstandings -- looking forward to revisions on all sides and,
> well, funding for the different projects and potential collaboration
> between them..

Well I've spent something like two hours writing this email, so I find it
all very interesting!  Certainly there is much potential points of
collaboration, in ideas if not code (I'd like to borrow your PDF rendering
code, you want to borrow my mozilla code... sounds like a good trade :-)

[   Peter Amstutz   ][ address@hidden ][ address@hidden  ]
[Lead Programmer][Interreality Project][Virtual Reality for the Internet]
[ VOS: Next Generation Internet Communication][ http://interreality.org ]
[ http://interreality.org/~tetron ][ pgpkey:  pgpkeys.mit.edu  18C21DF7 ]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)

iD8DBQE/l4GAaeHUyhjCHfcRAtSjAJsFHU6Qb47A+t1UOwinOeAxZmAbuwCgpMLr
8hWENPmsFRM1NZ6yyCMy8vo=
=LtwS
-----END PGP SIGNATURE-----