From: Steffen Seeger <s.seeger@physik.tu-chemnitz.de>
  To  : ggi-develop@eskimo.com
  Date: Tue, 3 Aug 1999 14:24:35 +0200 (CEST)

Re: Matrox GGI accellerator

James Simmons wrote:
 
> On Mon, 2 Aug 1999, Rodolphe Ortalo wrote:
> 
> > Jos Hulzink wrote:
> > 
> > > Writing the ViRGE accelleration code, I took a look at the Matrox driver.
> > > The first thing I saw was a mmap of the accelleration registers in the ggi
> > > driver ! (In other words: Every user has access the accelleration
> > > registers (including DMA command registers) and mess with the system.
> > > I thought
> > > this was one of the reasons why Linus has bad thoughts about ggi....
> > 
> > Not exactly. In fact, one may even say that it's the opposite...
> > It's because we think that full and blind userspace reg. mapping is
> > bad that some people think we want to put all the graphic
> > management code _inside_ the kernel. (We do not WANT that. But
> > safety and security often IMPOSES to put much of the regs. access in the kernel
> > IF you want good security properties. The matrox userspace lib. is
> > explicitly for those WITHOUT security needs.)
> 
> They know that accel handling, not accel registers belongs in the kerenel.

>From my understanding, this statement is a bit confusing. Accelerator 
registers are a hardware property that is just there and cannot be moved.

The problem is that you have to clearly define what registers are handled by
what entity (driver/xserver, application etc.) and how to make sure all
parties stick to the rules. The easiest way to ensure this is by using
protection mechanisms and a central management instance. At least this
is what has been the most successful concept virtualizing 'normal' CPUs.
I don't see any reason it shouldn't work in principle for graphics 
processors. All right, context switches are far more expensive here, 
so you have to minimize them.

To establish this single entity, we started to eliminate all possible
'bypasses' which lead to a redesigned console subsystem. I wouldn't have
messed with the console code if the benefit of it wouldn't compensate for
the effort. And if you are about to rewrite it anyway, you should care
about support of useful features like (true) multihead... That's why
KGI contains input drivers as well. I didn't like to mess with that,
but a consequent solution seems to require that. Given that the 
kernel development heads _now_ (years later) to the same direction...

> That is the distiction that has to get across. Their has been some
> discussion on this. In fact I had a talk with one of the kernel developers
> today about this. The BIGGEST thing is the need for proper virtualization
> of the accel engine. 

Which is -- telling from my experience -- pretty hard to achive if you have 
no central instance that coordinates who gets access to what at what time.
KGI-0.9 is at the end of a long process of reworking and rethinking the
underlying concepts and I wish I had the time I need to finish it.

> Security naturally comes out of this.

I can second this. I have been very impressed by the somewhat
consistent KGI (from the ggi-0.0.9 package) and how easy and secure
graphics hardware hardware access can be.

> I have been
> working on the design for /dev/gfx to deal with such issues. I'm modeling
> /dev/gfx after the way it works on SGI workstations. 

Luckily you seem to have access to either SGI documentation or an
SGI to experiment with. We hadn't. However, from the studies of articles
about SGI hardware and comparing this with what PC hardware offers I would
strongly recommend not just to copy this but to rethink what can be
applied to PC hardware and what not.

> RIght now if I where
> to start 3 OpenGL programs in X windows and if all of them bang on the
> registers at the same time it would surely hose the machine. A per process
> access to the accel engine needs to be set up. 

Which is exactly what the KGI-0.9 /dev/graphic mapper in connection with
the loaded KGI driver does...

> With /dev/gfx I plane to
> have so you allocate a window in the framebuffer for accel work. This
> enables you to use /dev/gfx with or without X windows with the same
> perforamce since in X windows you are opening /dev/gfx to work with a
> window instead of going threw the X server just like its done on SGI
> stations. Plus with the window idea you can mmap the zbuffer the size 
> of the window. Even if the card only allows a z buffer the size of the
> mode /dev/gfx will only mmap the window area of the zbuffer. This way
> each window area can belong only to a specific process. The problem I'm
> have is with window overlapping. 

This is one difference between SGI hardware and PC hardware. SGI hardware
stores a window ID with each pixel, and drawing operations modify other
attributes only if the window ID matches...

> As for mmap the accel registers
> themselves. Well thats tricky. One we have to make sure the accel engine
> and the framebuffer are not hit at the same time. This is the hardest
> part since I'm no guru with the mmap system. 

We have thought of this as well and discussed it exensively,
but dropped it for three reasons: a) it's slow, b) you have to do complex
state management to keep track what may (not) be mapped if resource xyz
is accessed, and c) current hardware manufacturing standards require 
PC hardware to cope with concurrent framebuffer accesses through the
accelerator and the main CPU(s). This is why I favour the ping-pong buffers
or kgi_accelerator resources to do accelerator access and have the
framebuffer 'statically' mapped.

> As for mmaping dangerous
> registers well I was thinking of having a ioctl call that returns a map of
> the registers and need mask on these registers. So for example you have
> 
> struct gfx_registers {
> 	line_x 0x34;
> 	line_x_mask 0xff12;
> 	line_y 0x35
> 	line_y_mask 0xff13;
> 	...
>         ...
> }
> 
> Where line_x represents the offset from the beginning of the accel MMIO
> region. Feedback if this is a good idea. What I trying to do is provide
> the programs where to program and not to good into areas he/she shouldn't.

This doesn't solve the problem. If you mmap a register range, you have
to make sure that the hardware can handle a 

while (1) {
	gfx_registers[random()]= random();
}

without locking the system. This is at least ok for frame buffers and
AGP memory. However, if you can hang the system, only trusted code should
be allowed access to such registers. This is why KGI-0.9 introduces an
access control scheme for each resource. Either registers are dangerous
and cannot be exported or they are not and can be exported to applications.

> > I really think that pingpong buffers are the best way to go, given the graphic
> > card hardware problems that prevent us to mmap fully to userspace without
> > compromising safety on the one hand, and the latency of ioctl on the other
> > hand (this is not to neglect even if I agree that ioctl are not so bad
> > in practice - especially for 2d ops.).
> 
> Hum. Can you expalin in detail the ping pong method. I sort of have a
> idea on how it works but not quite. Thanks.

You just expose a hardware command buffer to the application which 
'compiles' commands into that buffer. Once the buffer is full, you
deny the application access to it, and give it to the driver to 
'execute it'. This means either to start a DMA transfer of the buffer,
do programmed IO to send the commands to the hardware. This allows all
the API specific translation of rendering requests to be translated into
a hardware specific command stream by userspace code. This is why we have
to consider a modular library like libGGI as a part of the driver.

The tricky thing about this is just how to detect if the buffer is full
and when to switch buffers, do context switches etc.. This is worked 
out at code level and already implemented in KGI-0.9. The parts are 
tested independently, but the whole thing is not due to lack of drivers
and time to finish those. It does (in a quite simple but extendible manner)
what you are trying to implement: virtual graphics processors.

Regards,
			Steffen

----------------- e-mail: seeger@physik.tu-chemnitz.de -----------------