[xougen] Dealing with images

[Gian Filippo Pinzari]

On Sunday 07 December 2003 05:05, Herbert Snorrason wrote:
> Now the question is: Who won't want it?
> PNG is in nearly every way *the* lossless raster format. Nearly everyone
> with X is going to have libpng available. It would, frankly, make sense
> to promote it to a default format for any and all bitmap graphics.

I'm personally advocating addition of PNG and JPEG support to the 
X protocol since long. Translation of X bitmaps to PNG and JPEG is in 
already NX. Images are decompressed back to X bitmaps by the X 
server side proxy. Only 'NX aware' X clients can use it as it's not a 
standard X extension and you need to run the NX proxy system, so 
it's clearly not a general solution. In NX we need our own Xlib to make 
this available to all X clients and this is even more clumsy. 

When I'm saying that advocating PNG and JPEG addition, I'm not talking 
about XIE. I'm talking about a way to:

- Save bandwidth sending images in a more compact format.

- Allow clients to 'stream' images on the link and be notified
  when the image has been completely recomposed at the
  X server side.

Additional functionalities (all implemented in NX) that could greatly
improve network efficiency are:

- Separate alpha channel from the PutImage X bitmap so
  alpha (highly compressible) can be cached/compressed as 
  a separate message. This would allow to add alpha to JPEG
  or other image formats that don't natively support it.

- Store 'colortables' in the X server to be used to decompress 
  the X bitmaps, again as a separate (highly cacheable)
  protocol message. I'm not talking about the existing X 
  colormaps here, but the mapping of values to pixels of the
  TrueColor visual, to be used to decompress the image.

- Store images at X server side and let clients verify if the
  image needs to be transferred over the link. X server could
  store PNG an JPEG images in memory or disk and decompress 
  them on the fly, when needed. This is different in respect of
  pixmaps. Clients couldn't manipulate the image, just copy
  (that is uncompress) the image to the drawable. 

- Support other pluggable image formats, such as MS RDP 
  bitmaps or RFB screen updates in different encodings.

Looking even further, I think that X needs to deal with the problem
of transferring huge amounts of bitmap data over the network in a
smarter and definitive way. I'm sure that many people will say that 
X clients should simply forget X bitmaps and use SVG or other vector 
graphics. How these people are going to run a X video-conferencing 
application or a media player over the network? They don't say. 
They just wonder why you would ever want to do that.

What I propose is to make the PutImage request obsolete (note, only
the request, not the idea that clients need bitmaps) and let clients 
use arbitrary image streaming codecs. The X client should be able to 
query which codecs are available, create a 'stream' object and add 
'frames' to it. The X server should decompress the frames to a virtual 
frame-buffer and provide a way to CopyArea from the frame buffer to 
the drawables. Then clients should be able to use all the usual X 
protocol requests to manipulate the drawable and display their 
output.

I started to talk about this with Leon Shiman of MAS but we didn't 
have a chance yet to discuss the technical details. Clearly this overlaps 
in many ways with what MAS is doing. There is a big difference, anyway.
MAS is intended to deal with streaming and displaying of real time MM 
contents while our scope is session persistency, compression and 
network efficiency. MAS is the solution for MM (but, as I said to Leon, 
the project needs to leave the lab and go in the wild) but we still need 
a way to manage images in average X clients (web browsers, office 
automation applications) that don't have time requirements but can't 
afford to loose data due to the stateful nature of X protocol.

Regards,

/Gian Filippo Pinzari.