Soon it will be a decade since we started the RadeonHD driver, where we pushed ATI to a point of no return, got a proper C coded graphics driver and freely accessible documentation out. We all know just what happened to this in the end, and i will make a rather complete write-up spanning multiple blog entries over the following months. But while i was digging out backed up home directories for information, i came across this...
It is a copy of the content of an email i sent to an executive manager at SuSE/Novell, a textfile called bridgmans_games.txt. This was written in July 2008, after the RadeonHD project gained what was called "Executive oversight". After the executive had first hand seen several of John Bridgmans games, he asked me to provide him with a timeline of some of the games he played with us. The below explanation covers only things that i knew that this executive was not aware of, and it covers a year of RadeonHD development^Wstruggle, from July 2007 until april 2007. It took me quite a while to compile this, and it was a pretty tough task mentally, as it finally showed, unequivocally, just how we had been played all along. After this email was read by this executive, he and my department lead took me to lunch and told me to let this die out slowly, and to not make a fuss. Apparently, I should've counted myself lucky to see these sort of games being played this early on in my career.
This is the raw copy, and only the names of some people, who are not in the public eye have been redacted out (you know who you are), some other names were clarified. All changes are marked with [].
These are:
SuSE Executive manager: [EXEC]
SuSE technical project manager: [TPM]
AMD manager (from a different department than ATI): [AMDMAN]
AMD liaison: [SANTA] (as he was the one who handed me and Egbert Eich a 1 cubic meter box full of graphics cards ;))
[EXEC], i made a rather extensive write-up about the goings-on throughout
the whole project. I hope this will not intrude too much on your time,
please have a quick read through it, mark what you think is significant and
where you need actual facts (like emails that were sent around...) Of
course, this is all very subjective, but for most of this emails can be
provided to back things up (some things were said on the weekly technical
call only).
First some things that happened beforehand...
Before we came around with the radeonhd driver, there were two projects
going
already...
One was Dave Airlie who claimed he had a working driver for ATI Radeon R500
hardware. This was something he wrote during 2006, from information under
NDA, and this nda didn't allow publication. He spent a lot of time bashing
ATI for it, but the code was never seen, even when the NDA was partially
lifted later on.
The other was a project started in March 2007 by Nokia's Daniel Stone and
Ubuntu's [canonical's] Matthew Garreth. The avivo driver. This driver was a
driver separate from the -ati or -radeon drivers, under the GPL, and written
from scratch by tracing the BIOS and tracking register changes [(only the
latter was true)]. Matthew and Daniel were only active contributors for the
first few weeks and quickly lost interest. The bulk of the development was
taken over by Jerome Glisse after that.
Now... Here is where we encounter John [Bridgman]...
Halfway july, when we first got into contact with Bridgman, he suddenly
started bringing up AtomBIOS. When then also we got the first lowlevel
hardware spec, but when we asked for hardware details, how the different
blocks fitted together, we never got an answer as that information was said
to not be available. We also got our hands on some of the atombios
interpreter code, and a rudimentary piece of documentation explaining how to
handle the atombios bytecode. So Matthias [Hopf] created an atombios
disassembler to help us write up all the code for the different bits. And
while we got some further register information when we asked for it,
bridgman kept pushing atombios relentlessly.
This whining went on and on, until we, late august, decided to write up a
report of what problems we were seeing with atombios, both from an interface
as from an actual implementation point of view. We never got even a single
reply to this report, and we billed AMD 10 or so mandays, and [with]
bridgman apparently berated, as he never brought the issue up again for the
next few months... But the story of course didn't end there of course.
At the same time, we wanted to implement one function from atomBIOS at the
time: ASICInit. This function brings the hardware into a state where
modesetting can happen. It replaces a full int10 POST, which is the
standard, antique way of bringing up VGA on IBM hardware, and ASICInit
seemed like a big step forward. John organised a call with the BIOS/fglrx
people to explain to us what ASICInit offered and how we could use it. This
is the only time that Bridgman put us in touch with any ATI people directly.
The proceedings of the call were kind of amusing... After 10 minutes or so,
one of the ATI people said "for fglrx, we don't bother with ASICInit, we
just call the int10 POST". John then quickly stated that they might have to
end the call because apparently other people were wanting to use this
conference room. The call went on for at least half an hour longer, and john
from time to time stated this again. So whether there was truth to this or
not, we don't know, it was a rather amusing coincidence though, and we of
course never gained anything from the call.
Late august and early september, we were working relentlessly towards
getting our driver into a shape that could be exposed to the public.
[AMDMAN] had warned us that there was a big marketing thing planned for the
kernel summit in september, but we never were told any of this through our
partner directly. So we were working like maniacs on getting the driver in a
state so that we could present it at the X Developers Summit in Cambridge.
Early september, we were stuck on several issues because we didn't get any
relevant information from Bridgman. In a mail sent on september 3, [TPM]
made the following statement: "I'm not willing to fall down to Mr. B.'s mode
of working nor let us hold up by him. They might (or not) be trying to stop
the train - they will fail." It was very very clear then already that things
were not being played straight.
Bridgman at this point was begging to see the code, so we created a copy of
a (for us) working version, and handed this over on september the 3rd as
well. We got an extensive review of our driver on a very limited subset of
the hardware, and we were mainly bashed for producing broken modes (monitor
synced off). This had of course nothing to do with us using direct
programming, as this was some hardware limits [PLL] that we eventually had
to go and find ourselves. Bridgman never was able to help provide us with
suitable information on what the issue could be or what the right limits
were, but the fact that the issue wasn't to be found in atombios versus
direct programming didn't make him very happy.
So on the 5th of september, ATI went ahead with its marketing campaign, the
one which we weren't told about at all. They never really mentioned SUSE's
role in this, and Dave Airlie even made a blog entry stating how he and Alex
were the main actors on the X side. The SUSE role in all of this was
severely downplayed everywhere, to the point where it became borderline
insulting. It was also one of the first times where we really felt that
Bridgman maybe was working more closely with Dave and Alex than with us.
We then had to rush off to the X Developers Summit in Cambridge. While we
met ATIs Matthew Tippett and Bridgman beforehand, it was clear that our role
in this whole thing was being actively downplayed, and the attitude towards
egbert and me was predominantly hostile. The talk bridgman and Tippett gave
at XDS once again fully downplayed our effort with the driver. During our
introduction of the radeonHD driver, John Bridgman handed Dave Airlie a CD
with the register specs that went around the world, further reducing our
relevance there. John stated, quite noisily, in the middle of our
presentation, that he wanted Dave to be the first to receive this
documentation. Plus, it became clear that Bridgman was very close to Alex
Deucher.
The fact that we didn't catch all supper/dinner events ourselves worked
against us as well, as we were usually still hacking away feverishly at our
driver. We also had to leave early on the night of the big dinner, as we had
to come back to nürnberg so that we could catch the last two days of the
Labs conference the day after. This apparently was a mistake... We later
found a picture (http://flickr.com/photos/cysglyd/1371922101/) that has the
caption "avivo de-magicifying party". This shows Bridgman, Tippett, Deucher,
all the redhat and intel people in the same room, playing with the (then
competing) avivo driver. This picture was taken while we were on the plane
back to nürnberg. Another signal that, maybe, Mr Bridgman isn't every
committed to the radeonhd driver. Remarkably though, the main developer
behind the avivo driver, Jerome Glisse, is not included here. He was found
to be quite fond of the radeonhd driver and immediately moved on to
something else as his goal was being achieved by radeonhd now.
On monday the 17th, right before the release, ATI was throwing up hurdles...
We suddenly had to put AMD copyright statements on all our code, a
requirement never made for any AMD64 work which is as far as i understood
things also invalid under german copyright law. This was a tough nut for me
to swallow, as a lot of the code in radeonhd modesetting started life 5years
earlier in my unichrome driver. Then there was the fact that the atombios
header didn't come with an acceptable license and copyright statement, and a
similar situation for the atombios interpreter. The latter two gave Egbert a
few extra hours of work that night.
I contacted the phoronix.com owner (Michael Larabel) to talk to him about an
imminent driver release, and this is what he stated: "I am well aware of the
radeonHD driver release today through AMD and already have some articles in
the work for posting later today." The main open source graphics site had
already been given a story by some AMD people, and we were apparently not
supposed to be involved here. We managed to turn this article around there
and then, so that it painted a more correct picture of the situation. And
since then, we have been working very closely with Larabel to make sure that
our message gets out correctly.
The next few months seemed somewhat uneventful again. We worked hard on
making our users happy, on bringing our driver up from an initial
implementation to a full featured modesetting driver. We had to
painstakingly try to drag more information out of John, but mostly find
things out for ourselves. At this time the fact that Dave Airlie was under
NDA still was mentioned quite often in calls, and John explained that this
NDA situation was about to change. Bridgman also brought up that he had
started the procedure to hire somebody to help him with documentation work
so that the flow of information could be streamlined. We kind of assumed
that, from what we saw at XDS, this would be Alex Deucher, but we were never
told anything.
Matthias got in touch with AMDs GPGPU people through [AMDMAN], and we were
eventually (see later) able to get TCore code (which contains a lot of very
valuable information for us) out of this. Towards the end of oktober, i
found an article online about AMDs upcoming Rv670, and i pasted the link in
our radeonhd irc channel. John immediately fired off an email explaining
about the chipset, apologising for not telling us earlier. He repeated some
of his earlier promises of getting hardware sent out to us quicker. When the
hardware release happened, John was the person making statements to the open
source websites (messing up some things in the process), but [SANTA] was the
person who eventually sent us a board.
Suddenly, around the 20th of november, things started moving in the radeon
driver. Apparently Alex Deucher and Dave Airlie had been working together
since November 3rd to add support for r500 and r600 hardware to the radeon
driver. I eventually dug out a statement on fedora devel, where redhat guys,
on the last day of oktober, were actively refusing our driver from being
accepted. Dave Airlie stated the following: "Red Hat and Fedora are
contemplating the future wrt to AMD r5xx cards, all options are going to be
looked at, there may even be options that you don't know about yet.."
They used chunks of code from the avivo driver, chunks of code from the
radeonhd driver (and the bits where we spent ages finding out what to do),
and they implemented some parts of modesetting using AtomBIOS, just like
Bridgman always wanted it. On the same day (20th) Alex posted a blog entry
about being hired by ATI as an open source developer. Apparently, Bridgman
mentioned that Dave had found something when doing AtomBIOS on the weekly
phonecall beforehand. So Bridgman had been in close communication with Dave
for quite a while already.
Our relative silence and safe working ground was shattered. Suddenly it was
completely clear that Bridgman was playing a double game. As some diversion
mechanisms, John now suddenly provided us with the TCore code, and suddenly
gave us access to the AMD NDA website, and also told us on the phone that
Alex is not doing any Radeon work in his worktime and only does this in his
spare time. Surely we could not be against this, as surely we too would like
to see a stable and working radeon driver for r100-r400. He also suddenly
provided those bits of information Dave had been using in the radeon driver,
some of it we had been asking for before and never got an answer to then.
We quickly ramped up the development ourselves and got a 1.0.0 driver
release out about a week later. John in the meantime was expanding his
marketing horizon, he did an interview with the Beyond3D website (there is
of course no mention about us), and he started doing some online forums
(phoronix), replying to user posts there, providing his own views on
"certain" situations (he has since massively increased his time on that
forum).
One interesting result of the competing project is that suddenly we were
getting answers to questions we had been asking for a long time. An example
of such an issue is the card internal view of where the cards own memory
resides. We spent weeks asking around, not getting anywhere, and suddenly
the registers were used in the competing driver, and within hours, we got
the relevant information in emails from Mr Bridgman (November 17 and 20,
"Odds and Ends"). Bridgman later explained that Dave Airlie knew these
registers from his "previous r500 work", and that Dave asked Bridgman for
clearance for release, which he got, after which we also got informed about
these registers as well. The relevant commit message in the radeon driver
predates the email we received with the related information by many hours.
[AMDMAN] had put us in touch with the GPGPU people from AMD, and matthias
and one of the GPGPU people spent a lot of time emailing back and forth. But
suddenly, around the timeframe where everything else happened (competing
driver, alex getting hired), John suddenly conjured up the code that the
GPGPU people had all along: TCore. This signalled to us that John had caught
our plan to bypass him, and that he now took full control of the situation.
It took about a month before John made big online promises about how this
code could provide the basis for a DRM driver, and that it would become
available to all soon. We managed to confirm, in a direct call with both the
GPGPU people and Mr Bridgman that the GPGPU people had no idea about that
John intended to make this code fully public straigth away. The GPGPU people
assumed that Johns questions were fully aimed at handing us the code without
us getting tainted, not that John intended to hand this out publically
immediately. To this day, TCore has not surfaced publically, but we know
that several people inside the Xorg community have this code, and i
seriously doubt that all of those people are under NDA with AMD.
Bridgman also ramped up the marketing campaign. He did an interview with
Beyond3D.com on the 30th where he broadly smeared out the fact that a
community member was just hired to work with the community. John of course
completely overlooked the SUSE involvement in everything. An attempt to
rectify this with an interview of our own to match never materialised due to
the heavy time constraints we are under.
On the 4th of december, a user came to us asking what he should do with a
certain RS600 chipset he had. We had heard from John that this chip was not
relevant to us, as it was not supposed to be handled by our driver (the way
we remember the situation), but when we reported this to John, he claimed
that he thought that this hardware never shipped and that it therefor was
not relevant. The hardware of course did ship, to three different vendors,
and Egbert had to urgently add support for it in the I2C and memory
controller subsystems when users started complaining.
One very notable story of this time is how the bring-up of new hardware was
handled. I mentioned the Rv670 before, we still didn't get this hardware at
this point, as [SANTA] was still trying to get his hands on this. What we
did receive from [SANTA] on the 11th of december was the next generation
hardware, which needed a lot of bring-up work: rv620/rv635. This made us
hopeful that, for once, we could have at least basic support in our driver
on the same day the hardware got announced. But a month and a half later,
when this hardware was launched, John still hadn't provided any information.
I had quite a revealing email exchange with [SANTA] about this too, where he
wondered why John was stalling this. The first bit of information that was
useful to us was given to us on February the 9th, and we had to drag a lot
of the register level information out of John ourselves. Given the massive
changes to the hardware, and the induced complications, it of course took us
quite some time to get this work finished. And this fact was greedily abused
by John during the bring-up time and afterwards. But what he always
overlooks is that it took him close to two months to get us documentation to
use even atombios successfully.
The week of december the 11th is also where Alex was fully assimilated into
ATI. He of course didn't do anything much in his first week at AMD, but in
his second week he started working on the radeon driver again. In the weekly
call then John re-assured us that Alex was doing this work purely in his
spare time, that his task was helping us get the answers we needed. In all
fairness, Alex did provide us with register descriptions more directly than
John, but there was no way he was doing all this radeon driver work in his
spare time. But it would take John another month to admit it, at which time
he took Alex working on the radeon driver as an acquired right.
Bridgman now really started to spend a lot of time on phoronix.com. He
posted what i personally see as rather biased comparisons of the different
drivers out there, and he of course beat the drums heavily on atombios. This
is also the time where he promised the TCore drop.
Then games continued as usual for the next month or so, most of which are
already encompassed in previous paragraphs.
One interesting sidenote is that both Alex and John were heavy on rebranding
AtomBIOS. They actively used the word scripts for the call tables inside
atombios, and were actively labelling C modesetting code as legacy. Both
quite squarely go in against the nature of AtomBIOS versus C code.
Halfway february, discussions started to happen again about the RS690
hardware. Users were having problems. After a while, it became clear that
the RS690, all along, had a display block called DDIA capable of driving a
DVI digital signal. RS690 was considered to be long and fully supported, and
suddenly a major display block popped into life upon us. Bridgmans excuse
was the same as with the RS600; he thought this never would be used in the
first place and that therefor we weren't really supposed to know about this.
On the 23rd and 24th of February, we did FOSDEM. As usual, we had an X.org
Developers Room there, for which i did most of the running around. We worked
with Michael Larabel from phoronix to get the talks taped and online.
Bridgman gave us two surprises at FOSDEM... For one, he released 3D
documentation to the public, we learned about this just a few days before,
we even saw another phoronix statement about TCore being released there and
then, but this never surfaced.
We also learned, on friday evening, that the radeon driver was about to gain
Xvideo support for R5xx and up, through textured video code that alex had
been working on. Not only did they succeed in stealing the sunshine away
from the actual hard work (organising fosdem, finding out and implementing
bits that were never supposed to be used in the first place, etc), they gave
the users something quick and bling and showy, something we today still
think was provided by others inside AMD or generated by a shader compiler.
And this to the competing project.
At FOSDEM itself Bridgman of course was full of stories. One of the most
remarkable ones, which i overheard when on my knees clearing up the
developers room, was bridgman talking to the Phoronix guy and some community
members, stating that people at ATI actually had bets running on how long it
would take Dave Airlie to implement R5xx 3D support for the radeon driver.
He said this loudly, openly and with a lot of panache. But when this support
eventually took more than a month, i took this up with him on the phonecall,
leading of course to his usual nervous laughing and making up a bit of a
coverstory again.
Right before FOSDEM, Egbert had a one-on-one phonecall with Bridgman, hoping
to clear up some things. This is where we first learned that bridgman had
sold atombios big time to redhat, but i think you now know this even better
than i do, as i am pretty hazy on details there. Egbert, if i remember
correctly, on the very same day, had a phonecall with redhat's Kevin Martin
as well. But since nothing of this was put on mail and i was completely
swamped with organising FOSDEM, i have no real recollection of what came out
of that.
Alex then continued with his work on radeon, while being our only real
point of contact for any information. There were several instances where our
requests for information resulted in immediate commits to the radeon driver,
where issues were fixed or bits of functionality were added. Alex also ended
up adding RENDER acceleration to the radeon driver, and when Bridgman was in
Nuernberg, we clearly saw how Bridgman sold this to his management: alex was
working on bits that were meant to be ported directly to RadeonHD.
In march, we spent our time getting rv620/635 working, dragging information,
almost register per register out of our friends. We learned about the
release of RS780 at CEBIT, meaning that we had missed yet another
opportunity for same day support. John had clearly stated, at the time of
the rv620/635 launch that "integrated parts are the only place where we are
'officially' trying to have support in place at or very shortly after
launch".
And with that, we pretty much hit the time frame when you, [EXEC], got
involved...
One interesting kind of new development is Kgrids. Kgrids is some code
written by some ATI people that has some useful information for driver
development, especially useful for the DRM. John Bridgman told the phoronix
author 2-3 weeks ago already that he had this, and that he was planning to
release this immediately. The phoronix author immediately informed me, but
the release of this code never happened so far. Last thursday, Bridgman
brought up Kgrids in the weekly technical phonecall already... When asked
how long he knew about this, he admitted that they knew about this for
several weeks already, which makes me wonder why we weren't informed
earlier, and why we suddenly were informed then...
As said, the above, combined with what already was known to this executive, and the games he saw being played first hand from april 2008 throughout june 2008, marked the start of the end of the RadeonHD project.
I too more and more disconnected myself from development on this, with Egbert taking the brunt of the work. I instead spent more time on the next SuSE enterprise release, having fun doing something productive and with a future. Then, after FOSDEM 2009, when 24 people at the SuSE Nuernberg office were laid off, i was almost relieved to be amongst them.
Time flies. |
tired
drained
happy