You are viewing libv

LIBV Intentionally Breaks Videodrivers - The Raspberry Pi, en route to becoming a proper open platform. [entries|archive|friends|userinfo]
Luc Verhaegen

[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

The Raspberry Pi, en route to becoming a proper open platform. [Mar. 5th, 2014|05:32 pm]
Previous Entry Share Next Entry
[Tags|, , , , , , ]
[Current Location |Rumpelkammer]
[mood |happyhappy]
[music |Fatboy Slim - We've come a long way baby - Praise you]

Like most, I was taken by surprise last weekend, I never expected this to happen, at all.

Given that I was the one to poke a rather big hole in the Raspberry Pi image a year and a half ago, and since i have been playing a bit of a role in freeing ARM GPUs these last few years, I thought I'd say a few words on how I see things, and why I said what I said back then, and why I didn't say other things.

It has become a bit of a long read, but this it's a eventful story that spans about two years.

Getting acquainted.

I talked about the lima driver at linuxtag in may 2012. As my contract work wasn't going any where real, I spent two weeks of codethink time (well, it's not as if I ever did spent just 8h per day working on lima) on getting the next step working with lima, so I had something nice to demo at Linuxtag. During that bringup, I got asked to form a plan of attack for freeing the Raspberry Pi GPU, for a big potential customer. Just after my lima talk at the conference, I downloaded some code and images through the linuxtag wifi, and set about finding the shader compiler on the train on the way back.

I am not one for big micro-managed teams where constant communication is required. I am very big on structure and on finding natural boundaries (which is why you have the term modesetting and the structured approach to developing display drivers today). I tend to split up a massive task along those natural boundaries, and then have clued folks take full control and responsibility of those big tough subtasks. This is why I split the massive task of REing a GPU between the command stream side and the shader compiler. This is the plan of attack that I had formed in the first half of 2011 for Mali (which we are still sticking to today), and I was of course looking to repeat that for the RPi.

On the train, I disassembled glCompileShader from the VideoCore userspace binary, and found the code to be doing surprisingly little. It quickly went down to some routines which were used by pretty much all gl and egl calls. And those lower routines were basically taking in some ids and some data, and then passing it to the kernel. A look at the kernel code then told me that there was nothing but passing data through happening at that side...

There was nothing real going on, as all the nifty bits were running somewhere else.

In the next few days, I talked to some advanced RPi users on IRC, and we figured out that even the bootloader runs on the Videocore and that that brings up the ARM core. A look at the 3 different binary blobs, for 3 different memory divisions between the ARM and the VideoCore, revealed that they were raw blobs, running directly on the videocore. Strings were very clearly visible, and what was immediately clear is that it was mostly differences in addresses between the 3 blobs. Apart from the addresses, nothing was immediately apparent, apart from the fact that even display code was done by the Videocore. I confirmed with Ben Brewer, who was then working on the shaders for Lima, and who has a processor design background.

The news therefor wasn't good. The Raspberry Pi is a completely closed system with a massive black box pulling all the strings. But at least it is consistently closed, and there is only limited scope for making userspace and 'The Blob' getting out of sync, unlike the PowerVR, where userspace, kernel and microcode are almost impossible to keep in sync.

So I wrote up the proposal for the potential customer (you know who you are), basically a few manmonths of busywork (my optimistic estimate doubled, and then doubled again, as I never do manage to keep my own optimistic timelines) to free the userspace shim, and a potential month for Ben for poking at the videocore blobs. Of course, I never heard back from codethink sales, which was expected with news that was this bad.

I decided to stay silent about the Raspberry Pi being such a closed system, at least for a few weeks. For once, this customer was trying to do The Right Thing by talking to the right people, instead of just making big statements, no matter what damage they cause or what morals they breach. I ended up being quite busy in the next few months and I kind of forgot about making some noise. This came back to haunt me later on.

The noise.

So on oktober 24th of 2012, the big announcement came. Here is the excerpt that I find striking:
"[...] the Raspberry Pi is the first ARM-based multimedia SoC with
fully-functional, vendor-provided (as opposed to partial, reverse engineered)
fully open-source drivers, and that Broadcom is the first vendor to open their
mobile GPU drivers up in this way. We at the Raspberry Pi Foundation hope to
see others follow."
While the text of the announcement judiciously talks about "userland" and "VideoCore driver code which runs on the ARM", it all goes massively downhill with the above excerpt. Together wit the massive cheering coming from the seemingly very religious RPi community, the fact that the RPis SoC was a massively closed system was completely drowned out.

Sure, this release was a very good step in the right direction. It allowed people to port and properly maintain the RPi userspace drivers, and took away a lot of the overhead for integration. But the above statement very deliberately brushed the truth about the BCM2835 SoC under the carpet, and then went on to brashly call for other vendors to follow suit, even though those others had chosen very different paths in their system design, and did not depend on a massive all-controlling black box.

I was appalled. Why was this message so deliberately crafted in what should be a very open project? How could people be so shortsighted and not see the truth, even with the sourcecode available?

What was even worse was the secondary message here. To me, it stated "we are perfectly happy with the big binary blob pulling the strings in the background". And with their call to other vendors, they pretty much told them: "keep the real work closed, but create an artificial shim to aid porting!". Not good.

So I went and talked to the other guys on the #lima channel on irc. The feeling amongst the folks in our channel was pretty uniform one of disgust. At first glance, this post looked as a clear and loud message to support open drivers, it only looked like that on the absolute surface. Anyone who would dig slightly deeper would soon see the truth, and the message would turn into "Keep the blob guys!".

It was clear that this message from the RPi foundation was not helping the cause of freeing ARM GPUs at all. So I decided to react to this announcement.

The shouting

So I decided to go post in the broadcom forums, and this is what I wrote:
Erm… Isn’t all the magic in the videocore blob? Isn’t the userland code you
just made available the simple shim that passes messages and data back and forth
to the huge blob running on the highly proprietary videocore dsp?

– the developer of the lima driver.
Given the specific way in which the original RPI foundation announcement was written, i had expected all from the RPI foundation itself to know quite clearly that this was indeed just the shim driver. The reminder of me being the lima driver developer, should've taken away any doubt about me actually knowing what i was talking about. But sadly, these assumptions were proven quite wrong when Liz Upton replied:
No.

There’s some microcode in the Videocore – not something you should confuse with an
ARM-side blob, which could actually prevent you from understanding or modifying
anything that your computer does. That microcode is effectively firmware.
She really had no idea to what extent the videocore was running the show, and then happily went and argued with someone whom most people would label an industry expert. Quite astounding.

Things of course went further downhill from there, with several RPI users showing themselves from their best side. Although I did like the comment about ATI and ATOMBIOS, some people at least hadn't forgotten that nasty story.

All-in-all, this was truly astounding stuff, and it didn't help my feeling that the RPI foundation was not really intending to do The Right Thing, but instead was more about making noise, under the guise of marketing. None of this was helping ARM GPUs or ARM userspace blobs in any way.

More shouting

What happened next was also bad. Instead of sticking to the facts, Dave Airlie went and posted this blog entry, reducing himself to the level of Miss Upton.

A few excerpts:
"Why is this not like other firmware (AMD/NVIDIA etc)?
The firmware we ship on AMD and nvidia via nouveau isn't directly controlling the GPU
shader cores. It mostly does ancillary tasks like power management and CPU offloading.
There are some firmwares for video decoding that would start to fall into the same
category as this stuff. So if you want to add a new 3D feature to the AMD gpu driver
you can just write code to do it, not so with the rPI driver stack."
"Will this mean the broadcom kernel driver will get merged?
No.

This is like Ethernet cards with TCP offload, where the full TCP/IP stack is run on the
Ethernet card firmware. These cards seem like a good plan until you find bugs in their
firmware stack or find out their TCP/IP implementation isn't actually any good. The same
problem will occur with this. I would take bets the GLES implementation sucks, because
they all do, but the whole point of open sourcing something is to allow other to improve
it something that can't be done in this case."
Amazing stuff.

First of all, with the videocore being responsible for booting the bcm2835, and the bcm2835 already being a supported platform in the linux kernel (as of version 3.7), all the damage had been done already. Refusing yet another tiny message passing driver, which has a massive user-base and which will see a lot of testing, then is rather short-sighted. Especially since this would have made it much easier for the massive RPI userbase to use the upstream kernel.

Then there is the fact that this supposed kernel driver would not be a drm driver. It's simply not up to the self-appointed drm maintainer to either accept or refuse this.

And to finish off, a few years ago, Dave forked RadeonHD driver code to create a driver that did things the ATI way (read: much less free, much more dependent on bios bytecode, and with many other corners being implemented the broken ATI way), wasted loads of AMD money, stopped register documentation from coming out, and generally made sure that fglrx still rules supreme today. And all of this for just "marketing reasons", namely the ability to make trump SuSE with noise. With a history like that, you just cannot go and claim the moral high ground on anything anymore.

Like I've stated before, open source software is about making noise, and Dave his blog entry was nothing more and nothing less. Noise breeds more noise in our parts. This is why I decided to not provide a full and proper analysis of the situation back then, as we had lowered ourselves to the level that the RPI foundation had chosen, and nothing I could have said would've improved on that.

Redemption

I received an email from Eben Upton last friday, where he pointed out the big news. I replied that I would have to take a thorough look first, but then also stated that I didn't expect Eben to email me if this hadn't been the real thing. I then congratulated him extensively.

And congratulations really are in order here.

The Raspberry PI is a global sales phenomenon. It created a whole new market, a space where devices like the Cubieboard and Olimex boards also are in now. It made ARM development boards accessible and affordable, and put a linux on ARM in the hands of 2.5 million people. It is so big that the original goal of the Raspberry Pi, namely that of education, is kind of drowned out. But just the fact that so many of these devices were sold already, will have a bigger educational impact than any direct action towards that goal. This gives the RPI foundation a massive amount of power, so much so, that even the most closed SoC around is being opened now.

But in Oktober of 2012, my understanding was that the RPI foundation wasn't interested in wielding that power to reach those goals which I and many others hold dear. And I was really surprised when it now was revealed that they did, as I had personally given up on this ever happening, and I had assumed that the RPI foundation was not intending to go all the way.

What's there?

After the email from Eben, I joined the #raspberrypi-internals channel on freenode to ask the guys who are REing the VideoCore, to get an expert opinion from those actually working on the hardware.

So what has appeared:
* A user manual for the 3D core.
* Some sample code for a different SoC, but code which which runs on the ARM core and not on the videocore.
* This sample code includes some interesting headers which helps the guys who are REing the Videocore.
* There's a big promise of releasing more about the Videocore and at least providing a properly free bootloader.

These GPU docs and code not only means that Broadcom has beaten ARM, Qualcomm, Nvidia, Vivante and Imagination technologies to the post, it also makes broadcom second only to Intel. (yes, in front of AMD, or should I say ATI, as ATI has run most of the show since 2008).

Then there is the extra info on the videocore and the promise of a fully free bootloader. Yes, there still is the massive dependance on the VideoCore for doing all sorts of important SoC things, but releasing this information takes time, especially since this was not part of the original SoC release process and it has to be made up for afterwards. The future is looking bright for this SoC though, and we really might get to a point where the heavily VideoCore based BCM2835 SoC becomes one of the most open SoCs around.

Today though, the BCM2835 still doesn't hold a candle to the likes of the Allwinner A10 with regards to openness. But the mid to long term difference is that most of Allwinners openness so far was accidental (they are just now starting to directly cooperate with the linux-sunxi community), and Allwinner doesn't own all the IP used in their SoC (ARM owns the GPU for instance, and it is being a big old embedded dinosaur about it too). The RPI SoC not only has very active backing from the SoC vendor, that SoC vendor also happens to be the full owner of the IP at the same time.

The RPI future looks really bright!

Whining.

I do not like the 10k bounty on getting Q3A going with the freshly released code. I do not think this is the best way to get a solid gpu driver out, as it mostly encourages a single person to make a quick and dirty port of the freshly released code. I would've preferred to have seen someone from the RPI foundation create a kanban with clearly defined goals, and then let the community work off the different work items, after which those work-items are graded and rewarded. But then, it's the RPI foundations money, and this bounty is a positive action overall.

I am more worried about something else. With the bounty and with the promise of a free bootloader, the crazy guys who are REing the VideoCore get demotivated twice. This 10k bounty rules them out, as it calls for someone to do a quick and dirty hack, and not for long hard labour towards The Right Thing. The promise of a free bootloader might make their current work superfluous, and has the potential to halt REing work altogether. These guys are working on something which is or was at least an order of magnitude harder than what the normal GPU REing projects have or had to deal with. I truly hope that they get some direct encouragement and support from the RPI foundation, and that both parties work together on getting the rest of the VideoCore free.

All in all, this Broadcom release is the best news I have heard since I started on the open ARM GPU path, it actually is the best news I've had since AMD accepted our proposal to free the ATI Radeon (back in june 2007). This release fully makes up for the bad communication of Oktober 2012, and has opened the door to making the BCM2835 the most free SoC out there.

So to Eben and the rest of the guys from the RPI Foundation: Well done! You made us all very happy and proud.
linkReply

Comments:
[User Picture]From: shnatsel
2014-03-05 05:58 pm (UTC)

(Link)

Does the drop include source code anything running on the VideoCore?

Also, what kind of possibilities does the newly-released 3D documentation open? I've skimmed the register documentation but I don't understand what having this documentation will allow on such a weird platform.
From: (Anonymous)
2014-03-05 06:55 pm (UTC)

(Link)

Broadcom now please give us an open source ADSL driver for BCM63xx. Netgear was kind enough to leak it with their DG834GBv4_V5.01.01_src.zip but it's proprietary source code. For some reason Netgear didn't included a binary blob and instead they gave us the full source. Broadcom we want full OpenWrt support for our gear.
From: (Anonymous)
2014-03-06 12:10 am (UTC)

(Link)

My feeling on the userspace shim code release was that it was a somewhat disappointing step in the right direction, however Broadcom and the foundation were milking it for all it's worth which soured that.

I must point out that Broadcom has traditionally been open-source hostile and has only recently started working on an official upstream wireless driver. That said, they still refuses to acknowledge any wireless card that's older than the ones supported by those drivers. (By "acknowledge" I mean their developers, who are excellent members of the community, seem to be unable to provide documentation, firmware, code or support.)

My thoughts on the shim layer were that it might be useful if only as a "map" of how the GPU blob worked. I didn't realise at the time that the blob was so big and so controlling.

The release of the RPi Wayland compositor using the hardware scaling and compositing engine was a bit of a shock, not because of the fact that it was released, but because someone (maybe Eben) commented that the ARM core was just a tiny part of the chip and it was, overall, significantly underutilised by Linux.

My feeling on this latest release, particularly now that you've listed exactly what it is, is disappointment again. We still don't have any real documentation for the SoC's internals, it's various ISAs, (I believe there's at least two different ones in the "GPU" part alone) or any directly opened code.

I'm sure that the RE guys + whoever steps up for this bounty will produce something "useful" out of this, but I don't see this as being anything more than another step towards a 100% "open" chip.
From: (Anonymous)
2014-03-15 11:13 am (UTC)

(Link)

Luc, please keep up the work on the limadriver for Mali GPUs, it is needed badly. Don't let you get sidetracked too much by legacy stuff.