You are viewing libv

LIBV Intentionally Breaks Videodrivers - Q3A with open source generated shaders! [entries|archive|friends|userinfo]
Luc Verhaegen

[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

Q3A with open source generated shaders! [Mar. 18th, 2013|08:28 pm]
Previous Entry Share Next Entry
[Tags|, , , , , , , , , ]
[Current Location |Rumpelkammer]
[mood |Happy]
[music |Cornershop - Brimful of asha (norman cook remix)]

The combination of limare and open-gpu-tools can now run Quake 3 Arena timedemo without depending on the binary driver for the shader compiler!

Connor Abbott has been being his amazing (16y old!) self again in the weeks after his talk at FOSDEM, and he pushed his compiler work in his open-gpu-tools tree to be able to handle basic vertex shaders. Remember that our vertex shader is a rather insane one, where the compiler has to work real hard on getting scheduling absolutely right. This is why an assembler for our vertex shader was not too useful and the most part of a compiler had to be written for it to generate useful results. A mammoth task, and Connor his vertex shader code is now larger than the code I have in my limare library.

So it was high time that we brought limare and OGT together to see what they were capable of with some basic shaders. Luckily, the Q3A GLES1 emulation has basic shaders, what a nice coincidence :)

So Connor turned my simple vertex shader essl into the high level language used by the OGT vertex shader compiler, and through steps described at this wiki page, turned them into MBS files (Mali Binary Shader - the file type output by the standalone compiler, and also by newer binary driver integrated compilers). Limare can then load and parse those MBS files, and run the shaders. No need to involve the ARM binary anymore when we have OGT generated MBS files :)

The result was quite impressive. We had a few issues where the limare driver (which has mostly taken its cues from the output of the binary driver) and OGT disagreed over symbol layout, but apart from that, bringing up the shaders connor produced was pretty painless. Amazingly effortless, for such a big step.

Connor then spent another day playing with the fragment shader assembler, fixed some bugs, and produced 3 fragment shaders for us. One for the clear shader used by limare directly, and 2 for Q3A. After some more symbol layout issues, these also just worked! We even seem to be error-margin faster with the MBS files (due to texture coordinate varyings being laid out differently).

So this is a really big milestone for the lima driver project. Even with our insane pre-optimized architecture, we now are able to run Quake 3 Arena without any external dependencies, and we are beating the ARM binary while doing so.

For generating your own shader MBS files, check out Connors OGT, and then you can head straight to Connors wiki page. My Q3A tree now has the MBS code included directly. And i pushed a dirty version of my FOSDEM limare code.

As for this new limare code, this fosdem_2013_pile branch will vanish soon, as i need to properly pry things apart still. This is run-for-the-price code, and often includes many unrelated fixes in the same commit. It's better to do archeology on it now, than 3y from now, so this needs to be split. But in the meantime, you all can go and give Q3A on a fully free driver stack on Mali hw a go :)

I will not post a video, as there really is nothing new to see. It is the exact same timedemo, running some promille faster. Build things, and then run it yourself on your sunxi hardware (i am still working on porting it to the new kernel of a more powerful platform). That's the best proof there is!

For building limare, check out the fosdem2013_pile branch and then just run make/make install.

For building Q3A all you need to do is run:
make ARCH=arm USE_LIMARE=1
And, when you have the full quake installed in ~ioquake3/baseq3, you can create a file called ~ioquake3/baseq3/demofour.cfg with the following content:
cg_drawfps 1
timedemo 1
set demodone  "quit"
set demoloop1 "demo four; set nextdemo vstr demodone"
vstr demoloop1
You can then run the ioquake3 binary with "+exec demofour.cfg" added to the command line, and you will have the demo running on top of fully free software!

Now we really have covered all the basics, time to find out how Mesa will play with our plans :)
linkReply

Comments:
From: (Anonymous)
2013-03-19 03:44 pm (UTC)

(Link)

..... nice but we still cant use/test anything, shaders or not..... is a modification to make these drivers a decent substitute of mali-drivers planned? For when?
[User Picture]From: libv
2013-03-19 03:47 pm (UTC)

(Link)

Did you read the full blog entry?

This is a rather hardcore reverse engineering project, done in our spare time, things do not happen overnight, and you should not expect this to be otherwise.
From: (Anonymous)
2013-03-22 01:45 pm (UTC)

(Link)

Hehe, this happens everytime.... but when will it be ready. Luc the only thing I do hope is that this will one day be ready. It's a pity to see so many open source projects start and then end without producing any visible end-user results.
[User Picture]From: libv
2013-03-22 02:03 pm (UTC)

(Link)

Really?

Amazing.

Thanks for your contribution.
[User Picture]From: Marius Cirsta
2013-03-24 07:54 pm (UTC)

(Link)

You're wellcome :).
I do realize you and the other devs are committed to your work so don't take it personally.
And I do contribute to open source projects (I'm a developer for a Linux distro) so I know how it is when end users expect this and that.
From: (Anonymous)
2013-05-01 07:33 pm (UTC)

(Link)

Links to (substantial) contributions, or I call bullsh1t. No one who has said experience would utter such clueless, unappreciative words.
From: (Anonymous)
2013-09-15 01:13 pm (UTC)

(Link)

is the binary mali driver that bad?
From: (Anonymous)
2013-04-11 09:39 am (UTC)

(Link)

Stop for a moment and consider that he's doing this work pro bono. Developer time is expensive and industry expert time even more so.

Think yourself lucky that you live in a world where such people care about giving their time to working on such causes. He probably has to listen to pundits like you constantly -- and he *still* keeps on working on the project anyway.

P.S. instead of hoping try *helping*.
From: (Anonymous)
2013-04-15 12:56 pm (UTC)

(Link)

Uhm... yes, you can test something. He provides all the build steps to be able to run quake3 without proprietary drivers on a Mali GPU. That's it. If people would help testing and developing instead of bitching and waiting, open source GPU drivers would be running circles around the closed drivers by now. There is no magic involved.
From: (Anonymous)
2013-03-22 03:39 pm (UTC)

(Link)

Hello,
It is really impressive the progress you guys are making even without
any hints/help from the companies that sell and make money out of this hardware
(which I guess is really annoying/frustrating).

So thank you for the time and effort you put into this project.
From: (Anonymous)
2013-03-23 09:14 pm (UTC)

(Link)

awesome, i love yours contributes... May the force be with yours

greatings from germany ;)
From: (Anonymous)
2013-04-08 04:28 pm (UTC)

(Link)

hey, you're an awesome team!

i never thought something this cool could happen to free software and 3d graphics! :)

i also look forward to coreboot on exynos 5:
http://www.phoronix.com/scan.php?page=news_item&px=MTI3ODA

i wonder if it will be a fully free boot stack, or if fwbl1/bl1 will be unreplaced...
From: (Anonymous)
2013-04-27 03:53 am (UTC)

(Link)

Wow i dont even know trolls gtfo and give these guys some credit, i dont understand all (almost none) of the technical stuff but when i see an FOSS GPU driver beat an binary counterpart, that should say alot.
i recently read a blog post about the state of SoC FOSS drivers, posted on Phoronix (http://blog.emmanueldeloget.com/index.php?post/2013/03/08/The-SoC-GPU-driver-interview), and it was a very interesting read and i urge everyone to at least take the time read it before posting a bunch of retarded comments, it reminds me of the console scene and the constant nagging about iso loaders...
anyway must congratulate you Luc and Connor aswell Great job and good luck on the t6xx.
From: (Anonymous)
2013-04-27 03:56 am (UTC)

(Link)

Wow i dont even know trolls gtfo and give these guys some credit, i dont understand all (almost none) of the technical stuff but when i see an FOSS GPU driver beat an binary counterpart, that should say alot.
i recently read a blog post about the state of SoC FOSS drivers, posted on Phoronix: http://blog.emmanueldeloget.com/index.php?post/2013/03/08/The-SoC-GPU-driver-interview (http://blog.emmanueldeloget.com/index.php?post/2013/03/08/The-SoC-GPU-driver-interview), and it was a very interesting read and i urge everyone to at least take the time read it before posting a bunch of retarded comments, it reminds me of the console scene and the constant nagging about iso loaders...
anyway must congratulate you Luc and Connor aswell Great job and good luck on the t6xx.
From: (Anonymous)
2013-07-11 07:10 am (UTC)

(Link)

How long will this open source driver take it's been so long months now and when will it be merged included into the linux arm kernel tree when will it be released and done ?
Will it be done when this mali hardware becomes obsolete!??

I just want to run Linux with this open source hardware acceleration on my tablet.

This is really frustrating dont these hacker guys have any sense why can't you just take steal it or take a peek at the original mali source and then just do some little modifications to show it's different and make open source and then it will be easier and faster.
From: (Anonymous)
2013-09-02 03:06 am (UTC)

(Link)

Primarily because it would be stealing and it would not qualify for GPL, they have to reserve engineer it or ARM release the programming specifications.
From: (Anonymous)
2014-01-30 07:28 pm (UTC)

(Link)

True, and while reverse engineering is much slower (and harder!!), it can also have a positive effect: you don't rely on the possibly inefficient coding of the binary driver. You write an opensource driver based on the behaviour of the binary driver, not on the code of the binary driver. And you get rid of bugs that exist in the binary driver (because the human perception of a driver's behaviour will filter out bugs), however, for firmware, it's a different story.
This is something I learnt of the reverse engineering of the whole Logitech force feedback stack. It will be implemented in a proper way for Linux in some time. Of course, that's much easier than reverse engineering a graphics driver!

Amazing job, Luc! Much respect!
From: (Anonymous)
2014-01-30 03:12 pm (UTC)

(Link)

Luc, did you give up on this now its been 10 months since your last entry

i ask as iv only just found this linaro gpgpu list
http://lists.linaro.org/pipermail/linaro-gpgpu/2013-September/000006.html

and it seems all their professional devs would be glad to get your driver if it were somewhat complete for them to help you expand it
[User Picture]From: libv
2014-01-30 09:36 pm (UTC)

(Link)

No, i did not give up. Check out the FOSDEM graphics devroom schedule.

As for "professional devs" and them being "glad"... Well... If linaro was interested in helping out at all in this area, they could've tried talking to me oh, i dunno, back in 2011 already? But it's desperately trying to stay on the good side of ARM, and it will definitely be the last to smash some porcelain in areas where some action truly is needed.