?

Log in

The beginning of the end of the RadeonHD driver. - LIBV Intentionally Breaks Videodrivers [entries|archive|friends|userinfo]
Luc Verhaegen

[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

The beginning of the end of the RadeonHD driver. [Feb. 13th, 2017|11:08 am]
Luc Verhaegen
[Tags|, , , , , , , , ]
[Current Location |Rumpelkammer]
[mood |Tired]
[music |Toto - Africa]

Soon it will be a decade since we started the RadeonHD driver, where we pushed ATI to a point of no return, got a proper C coded graphics driver and freely accessible documentation out. We all know just what happened to this in the end, and i will make a rather complete write-up spanning multiple blog entries over the following months. But while i was digging out backed up home directories for information, i came across this...

It is a copy of the content of an email i sent to an executive manager at SuSE/Novell, a textfile called bridgmans_games.txt. This was written in July 2008, after the RadeonHD project gained what was called "Executive oversight". After the executive had first hand seen several of John Bridgmans games, he asked me to provide him with a timeline of some of the games he played with us. The below explanation covers only things that i knew that this executive was not aware of, and it covers a year of RadeonHD development^Wstruggle, from July 2007 until april 2007. It took me quite a while to compile this, and it was a pretty tough task mentally, as it finally showed, unequivocally, just how we had been played all along. After this email was read by this executive, he and my department lead took me to lunch and told me to let this die out slowly, and to not make a fuss. Apparently, I should've counted myself lucky to see these sort of games being played this early on in my career.

This is the raw copy, and only the names of some people, who are not in the public eye have been redacted out (you know who you are), some other names were clarified. All changes are marked with [].

These are:
SuSE Executive manager: [EXEC]
SuSE technical project manager: [TPM]
AMD manager (from a different department than ATI): [AMDMAN]
AMD liaison: [SANTA] (as he was the one who handed me and Egbert Eich a 1 cubic meter box full of graphics cards ;))

[EXEC], i made a rather extensive write-up about the goings-on throughout 
the whole project. I hope this will not intrude too much on your time, 
please have a quick read through it, mark what you think is significant and 
where you need actual facts (like emails that were sent around...) Of 
course, this is all very subjective, but for most of this emails can be 
provided to back things up (some things were said on the weekly technical 
call only).

First some things that happened beforehand...

Before we came around with the radeonhd driver, there were two projects 
going
already...

One was Dave Airlie who claimed he had a working driver for ATI Radeon R500
hardware. This was something he wrote during 2006, from information under 
NDA, and this nda didn't allow publication. He spent a lot of time bashing 
ATI for it, but the code was never seen, even when the NDA was partially 
lifted later on.

The other was a project started in March 2007 by Nokia's Daniel Stone and
Ubuntu's [canonical's] Matthew Garreth. The avivo driver. This driver was a
driver separate from the -ati or -radeon drivers, under the GPL, and written
from scratch by tracing the BIOS and tracking register changes [(only the
latter was true)]. Matthew and Daniel were only active contributors for the
first few weeks and quickly lost interest. The bulk of the development was
taken over by Jerome Glisse after that.

Now... Here is where we encounter John [Bridgman]...

Halfway july, when we first got into contact with Bridgman, he suddenly 
started bringing up AtomBIOS. When then also we got the first lowlevel 
hardware spec, but when we asked for hardware details, how the different 
blocks fitted together, we never got an answer as that information was said 
to not be available. We also got our hands on some of the atombios 
interpreter code, and a rudimentary piece of documentation explaining how to 
handle the atombios bytecode. So Matthias [Hopf] created an atombios 
disassembler to help us write up all the code for the different bits. And 
while we got some further register information when we asked for it, 
bridgman kept pushing atombios relentlessly.

This whining went on and on, until we, late august, decided to write up a
report of what problems we were seeing with atombios, both from an interface 
as from an actual implementation point of view. We never got even a single 
reply to this report, and we billed AMD 10 or so mandays, and [with] 
bridgman apparently berated, as he never brought the issue up again for the 
next few months... But the story of course didn't end there of course.

At the same time, we wanted to implement one function from atomBIOS at the
time: ASICInit. This function brings the hardware into a state where
modesetting can happen. It replaces a full int10 POST, which is the 
standard, antique way of bringing up VGA on IBM hardware, and ASICInit 
seemed like a big step forward. John organised a call with the BIOS/fglrx 
people to explain to us what ASICInit offered and how we could use it. This 
is the only time that Bridgman put us in touch with any ATI people directly. 
The proceedings of the call were kind of amusing... After 10 minutes or so, 
one of the ATI people said "for fglrx, we don't bother with ASICInit, we 
just call the int10 POST". John then quickly stated that they might have to 
end the call because apparently other people were wanting to use this 
conference room. The call went on for at least half an hour longer, and john 
from time to time stated this again. So whether there was truth to this or 
not, we don't know, it was a rather amusing coincidence though, and we of 
course never gained anything from the call.

Late august and early september, we were working relentlessly towards 
getting our driver into a shape that could be exposed to the public. 
[AMDMAN] had warned us that there was a big marketing thing planned for the 
kernel summit in september, but we never were told any of this through our 
partner directly. So we were working like maniacs on getting the driver in a 
state so that we could present it at the X Developers Summit in Cambridge.

Early september, we were stuck on several issues because we didn't get any
relevant information from Bridgman. In a mail sent on september 3, [TPM] 
made the following statement: "I'm not willing to fall down to Mr. B.'s mode 
of working nor let us hold up by him. They might (or not) be trying to stop 
the train - they will fail." It was very very clear then already that things 
were not being played straight.

Bridgman at this point was begging to see the code, so we created a copy of 
a (for us) working version, and handed this over on september the 3rd as 
well. We got an extensive review of our driver on a very limited subset of 
the hardware, and we were mainly bashed for producing broken modes (monitor 
synced off). This had of course nothing to do with us using direct 
programming, as this was some hardware limits [PLL] that we eventually had 
to go and find ourselves. Bridgman never was able to help provide us with 
suitable information on what the issue could be or what the right limits 
were, but the fact that the issue wasn't to be found in atombios versus 
direct programming didn't make him very happy.

So on the 5th of september, ATI went ahead with its marketing campaign, the 
one which we weren't told about at all. They never really mentioned SUSE's 
role in this, and Dave Airlie even made a blog entry stating how he and Alex 
were the main actors on the X side. The SUSE role in all of this was 
severely downplayed everywhere, to the point where it became borderline 
insulting. It was also one of the first times where we really felt that 
Bridgman maybe was working more closely with Dave and Alex than with us.

We then had to rush off to the X Developers Summit in Cambridge. While we 
met ATIs Matthew Tippett and Bridgman beforehand, it was clear that our role 
in this whole thing was being actively downplayed, and the attitude towards 
egbert and me was predominantly hostile. The talk bridgman and Tippett gave 
at XDS once again fully downplayed our effort with the driver. During our 
introduction of the radeonHD driver, John Bridgman handed Dave Airlie a CD 
with the register specs that went around the world, further reducing our 
relevance there. John stated, quite noisily, in the middle of our 
presentation, that he wanted Dave to be the first to receive this 
documentation. Plus, it became clear that Bridgman was very close to Alex 
Deucher.

The fact that we didn't catch all supper/dinner events ourselves worked 
against us as well, as we were usually still hacking away feverishly at our 
driver. We also had to leave early on the night of the big dinner, as we had 
to come back to nürnberg so that we could catch the last two days of the 
Labs conference the day after. This apparently was a mistake... We later 
found a picture (http://flickr.com/photos/cysglyd/1371922101/) that has the 
caption "avivo de-magicifying party". This shows Bridgman, Tippett, Deucher, 
all the redhat and intel people in the same room, playing with the (then 
competing) avivo driver. This picture was taken while we were on the plane 
back to nürnberg. Another signal that, maybe, Mr Bridgman isn't every 
committed to the radeonhd driver. Remarkably though, the main developer 
behind the avivo driver, Jerome Glisse, is not included here. He was found 
to be quite fond of the radeonhd driver and immediately moved on to 
something else as his goal was being achieved by radeonhd now.

On monday the 17th, right before the release, ATI was throwing up hurdles... 
We suddenly had to put AMD copyright statements on all our code, a 
requirement never made for any AMD64 work which is as far as i understood 
things also invalid under german copyright law. This was a tough nut for me 
to swallow, as a lot of the code in radeonhd modesetting started life 5years 
earlier in my unichrome driver. Then there was the fact that the atombios 
header didn't come with an acceptable license and copyright statement, and a 
similar situation for the atombios interpreter. The latter two gave Egbert a 
few extra hours of work that night.

I contacted the phoronix.com owner (Michael Larabel) to talk to him about an 
imminent driver release, and this is what he stated: "I am well aware of the 
radeonHD driver release today through AMD and already have some articles in 
the work for posting later today." The main open source graphics site had 
already been given a story by some AMD people, and we were apparently not 
supposed to be involved here. We managed to turn this article around there 
and then, so that it painted a more correct picture of the situation. And 
since then, we have been working very closely with Larabel to make sure that 
our message gets out correctly.

The next few months seemed somewhat uneventful again. We worked hard on 
making our users happy, on bringing our driver up from an initial 
implementation to a full featured modesetting driver. We had to 
painstakingly try to drag more information out of John, but mostly find 
things out for ourselves. At this time the fact that Dave Airlie was under 
NDA still was mentioned quite often in calls, and John explained that this 
NDA situation was about to change. Bridgman also brought up that he had 
started the procedure to hire somebody to help him with documentation work 
so that the flow of information could be streamlined. We kind of assumed 
that, from what we saw at XDS, this would be Alex Deucher, but we were never 
told anything.

Matthias got in touch with AMDs GPGPU people through [AMDMAN], and we were 
eventually (see later) able to get TCore code (which contains a lot of very
valuable information for us) out of this. Towards the end of oktober, i 
found an article online about AMDs upcoming Rv670, and i pasted the link in 
our radeonhd irc channel. John immediately fired off an email explaining 
about the chipset, apologising for not telling us earlier. He repeated some 
of his earlier promises of getting hardware sent out to us quicker. When the 
hardware release happened, John was the person making statements to the open 
source websites (messing up some things in the process), but [SANTA] was the 
person who eventually sent us a board.

Suddenly, around the 20th of november, things started moving in the radeon
driver. Apparently Alex Deucher and Dave Airlie had been working together 
since November 3rd to add support for r500 and r600 hardware to the radeon 
driver. I eventually dug out a statement on fedora devel, where redhat guys, 
on the last day of oktober, were actively refusing our driver from being 
accepted. Dave Airlie stated the following: "Red Hat and Fedora are 
contemplating the future wrt to AMD r5xx cards, all options are going to be 
looked at, there may even be options that you don't know about yet.."

They used chunks of code from the avivo driver, chunks of code from the
radeonhd driver (and the bits where we spent ages finding out what to do), 
and they implemented some parts of modesetting using AtomBIOS, just like 
Bridgman always wanted it. On the same day (20th) Alex posted a blog entry 
about being hired by ATI as an open source developer. Apparently, Bridgman 
mentioned that Dave had found something when doing AtomBIOS on the weekly 
phonecall beforehand. So Bridgman had been in close communication with Dave 
for quite a while already.

Our relative silence and safe working ground was shattered. Suddenly it was
completely clear that Bridgman was playing a double game. As some diversion
mechanisms, John now suddenly provided us with the TCore code, and suddenly
gave us access to the AMD NDA website, and also told us on the phone that 
Alex is not doing any Radeon work in his worktime and only does this in his 
spare time. Surely we could not be against this, as surely we too would like 
to see a stable and working radeon driver for r100-r400. He also suddenly 
provided those bits of information Dave had been using in the radeon driver, 
some of it we had been asking for before and never got an answer to then.

We quickly ramped up the development ourselves and got a 1.0.0 driver 
release out about a week later. John in the meantime was expanding his 
marketing horizon, he did an interview with the Beyond3D website (there is 
of course no mention about us), and he started doing some online forums 
(phoronix), replying to user posts there, providing his own views on 
"certain" situations (he has since massively increased his time on that 
forum).

One interesting result of the competing project is that suddenly we were
getting answers to questions we had been asking for a long time. An example 
of such an issue is the card internal view of where the cards own memory 
resides. We spent weeks asking around, not getting anywhere, and suddenly 
the registers were used in the competing driver, and within hours, we got 
the relevant information in emails from Mr Bridgman (November 17 and 20, 
"Odds and Ends"). Bridgman later explained that Dave Airlie knew these 
registers from his "previous r500 work", and that Dave asked Bridgman for 
clearance for release, which he got, after which we also got informed about 
these registers as well. The relevant commit message in the radeon driver 
predates the email we received with the related information by many hours.

[AMDMAN] had put us in touch with the GPGPU people from AMD, and matthias 
and one of the GPGPU people spent a lot of time emailing back and forth. But
suddenly, around the timeframe where everything else happened (competing
driver, alex getting hired), John suddenly conjured up the code that the 
GPGPU people had all along: TCore. This signalled to us that John had caught 
our plan to bypass him, and that he now took full control of the situation. 
It took about a month before John made big online promises about how this 
code could provide the basis for a DRM driver, and that it would become 
available to all soon. We managed to confirm, in a direct call with both the 
GPGPU people and Mr Bridgman that the GPGPU people had no idea about that 
John intended to make this code fully public straigth away. The GPGPU people 
assumed that Johns questions were fully aimed at handing us the code without 
us getting tainted, not that John intended to hand this out publically 
immediately. To this day, TCore has not surfaced publically, but we know 
that several people inside the Xorg community have this code, and i 
seriously doubt that all of those people are under NDA with AMD.

Bridgman also ramped up the marketing campaign. He did an interview with
Beyond3D.com on the 30th where he broadly smeared out the fact that a 
community member was just hired to work with the community. John of course 
completely overlooked the SUSE involvement in everything. An attempt to 
rectify this with an interview of our own to match never materialised due to 
the heavy time constraints we are under.

On the 4th of december, a user came to us asking what he should do with a
certain RS600 chipset he had. We had heard from John that this chip was not
relevant to us, as it was not supposed to be handled by our driver (the way 
we remember the situation), but when we reported this to John, he claimed 
that he thought that this hardware never shipped and that it therefor was 
not relevant. The hardware of course did ship, to three different vendors, 
and Egbert had to urgently add support for it in the I2C and memory 
controller subsystems when users started complaining.

One very notable story of this time is how the bring-up of new hardware was
handled. I mentioned the Rv670 before, we still didn't get this hardware at
this point, as [SANTA] was still trying to get his hands on this. What we 
did receive from [SANTA] on the 11th of december was the next generation 
hardware, which needed a lot of bring-up work: rv620/rv635. This made us 
hopeful that, for once, we could have at least basic support in our driver 
on the same day the hardware got announced. But a month and a half later, 
when this hardware was launched, John still hadn't provided any information. 
I had quite a revealing email exchange with [SANTA] about this too, where he 
wondered why John was stalling this. The first bit of information that was 
useful to us was given to us on February the 9th, and we had to drag a lot 
of the register level information out of John ourselves. Given the massive 
changes to the hardware, and the induced complications, it of course took us 
quite some time to get this work finished. And this fact was greedily abused 
by John during the bring-up time and afterwards. But what he always 
overlooks is that it took him close to two months to get us documentation to 
use even atombios successfully.

The week of december the 11th is also where Alex was fully assimilated into
ATI. He of course didn't do anything much in his first week at AMD, but in 
his second week he started working on the radeon driver again. In the weekly 
call then John re-assured us that Alex was doing this work purely in his 
spare time, that his task was helping us get the answers we needed. In all 
fairness, Alex did provide us with register descriptions more directly than 
John, but there was no way he was doing all this radeon driver work in his 
spare time. But it would take John another month to admit it, at which time 
he took Alex working on the radeon driver as an acquired right.

Bridgman now really started to spend a lot of time on phoronix.com. He 
posted what i personally see as rather biased comparisons of the different 
drivers out there, and he of course beat the drums heavily on atombios. This 
is also the time where he promised the TCore drop.

Then games continued as usual for the next month or so, most of which are
already encompassed in previous paragraphs.

One interesting sidenote is that both Alex and John were heavy on rebranding
AtomBIOS. They actively used the word scripts for the call tables inside
atombios, and were actively labelling C modesetting code as legacy. Both 
quite squarely go in against the nature of AtomBIOS versus C code.

Halfway february, discussions started to happen again about the RS690 
hardware. Users were having problems. After a while, it became clear that 
the RS690, all along, had a display block called DDIA capable of driving a 
DVI digital signal. RS690 was considered to be long and fully supported, and 
suddenly a major display block popped into life upon us. Bridgmans excuse 
was the same as with the RS600; he thought this never would be used in the 
first place and that therefor we weren't really supposed to know about this.

On the 23rd and 24th of February, we did FOSDEM. As usual, we had an X.org 
Developers Room there, for which i did most of the running around. We worked
with Michael Larabel from phoronix to get the talks taped and online. 
Bridgman gave us two surprises at FOSDEM... For one, he released 3D 
documentation to the public, we learned about this just a few days before, 
we even saw another phoronix statement about TCore being released there and 
then, but this never surfaced.

We also learned, on friday evening, that the radeon driver was about to gain
Xvideo support for R5xx and up, through textured video code that alex had 
been working on. Not only did they succeed in stealing the sunshine away 
from the actual hard work (organising fosdem, finding out and implementing 
bits that were never supposed to be used in the first place, etc), they gave 
the users something quick and bling and showy, something we today still 
think was provided by others inside AMD or generated by a shader compiler. 
And this to the competing project.

At FOSDEM itself Bridgman of course was full of stories. One of the most
remarkable ones, which i overheard when on my knees clearing up the 
developers room, was bridgman talking to the Phoronix guy and some community 
members, stating that people at ATI actually had bets running on how long it 
would take Dave Airlie to implement R5xx 3D support for the radeon driver. 
He said this loudly, openly and with a lot of panache. But when this support 
eventually took more than a month, i took this up with him on the phonecall, 
leading of course to his usual nervous laughing and making up a bit of a 
coverstory again.

Right before FOSDEM, Egbert had a one-on-one phonecall with Bridgman, hoping 
to clear up some things. This is where we first learned that bridgman had 
sold atombios big time to redhat, but i think you now know this even better 
than i do, as i am pretty hazy on details there. Egbert, if i remember 
correctly, on the very same day, had a phonecall with redhat's Kevin Martin 
as well. But since nothing of this was put on mail and i was completely 
swamped with organising FOSDEM, i have no real recollection of what came out 
of that.

Alex then continued with his work on radeon, while being our only real 
point of contact for any information. There were several instances where our 
requests for information resulted in immediate commits to the radeon driver, 
where issues were fixed or bits of functionality were added. Alex also ended 
up adding RENDER acceleration to the radeon driver, and when Bridgman was in
Nuernberg, we clearly saw how Bridgman sold this to his management: alex was
working on bits that were meant to be ported directly to RadeonHD.

In march, we spent our time getting rv620/635 working, dragging information,
almost register per register out of our friends. We learned about the 
release of RS780 at CEBIT, meaning that we had missed yet another 
opportunity for same day support. John had clearly stated, at the time of 
the rv620/635 launch that "integrated parts are the only place where we are 
'officially' trying to have support in place at or very shortly after 
launch".

And with that, we pretty much hit the time frame when you, [EXEC], got
involved...

One interesting kind of new development is Kgrids. Kgrids is some code 
written by some ATI people that has some useful information for driver 
development, especially useful for the DRM. John Bridgman told the phoronix 
author 2-3 weeks ago already that he had this, and that he was planning to 
release this immediately. The phoronix author immediately informed me, but 
the release of this code never happened so far. Last thursday, Bridgman 
brought up Kgrids in the weekly technical phonecall already... When asked 
how long he knew about this, he admitted that they knew about this for 
several weeks already, which makes me wonder why we weren't informed 
earlier, and why we suddenly were informed then...


As said, the above, combined with what already was known to this executive, and the games he saw being played first hand from april 2008 throughout june 2008, marked the start of the end of the RadeonHD project.

I too more and more disconnected myself from development on this, with Egbert taking the brunt of the work. I instead spent more time on the next SuSE enterprise release, having fun doing something productive and with a future. Then, after FOSDEM 2009, when 24 people at the SuSE Nuernberg office were laid off, i was almost relieved to be amongst them.

Time flies.
linkReply