Well, thanks a lot for coming. This is a very optimistic talk. Quite tricky to write
it turns out. It's quite tricky to push your testing upstream. You might or may not know
my name. From talking about no, I'm an open QA testing. This isn't the same talk again
talking about no, I'm an open QA testing. Even though I am going to relate it to no because
that's what I'm involved in. And I'm going to relate it to open QA because that's pretty
much the best tool we have. But it's more about kind of a vision of the future than what we have
today. Firstly, we've come a long way, right? The desktop Linux world. In the early 2000s and
in the late 90s we've installed the Linux distribution and we'd go, okay, like the graphical
desktop doesn't work. Let me fix my X config and then move it up. It'd be nice to have
Wi-Fi. One of why that doesn't work. And then if you were lucky you'd have audio as well. So today
things were a lot better. I mean that they used to be a distro as well where you couldn't do
desktop grades. They'd say delete your old installation and install the new one. It'll work better.
And we've come a long way. Now we have much better testing. We test things like upgrades and
I like to use this example as one of the things that's really horrible to test. If you open a
MP4 file in Fedora and you try to play it doesn't work because you don't have the right codec.
So this dialog box looks very simple. It just pops up and says, oh, you need a codec.
But the code underlying this to make it work is hugely complex. You have a G-streamer plug-in,
you have something in the video player to handle messages from the plug-in and you have
known software which is incredibly complex in itself. A testing app is quite tricky, right?
Because you need all of these bits integrated together. So I'm a big fan of end-to-end testing.
Of the kind that OpenQA does, where you test the entire system, you test the entire operating
system and either on hardware or on a virtual machine. So you have your current, you control the kernel,
you control the user space and everything. And another key part of this type of testing is that you
navigate the system like an end user would. So you don't connect to the serial console and type serial
commands. That's not how most people interact with the desktop machine. You inject keyboard events
and mouse events because real users use a keyboard and a mouse. We never go to UI by looking at
what's on the screen, right? So if you want to test the OS like a real user, a real user,
if they're trying to find the console button, they look for a shape that looks like a button and says
cancel on it. Doesn't matter if it's here or if it's here, really, just they're looking for that.
And OpenQA has really good support for looking for something that looks like a button and says
cancel. And if the non-developers decides and the next release it needs to be a pair instead of
on the right, the test might still pass because the user could still find the button.
So who's doing this kind of testing today? I'm going to open the floor to questions.
Shout out to the angels and distros that are doing this.
Fordora? Yeah. One of the earliest to adopt OpenQA. Open Susa, definitely, the original.
Anyone else? I'm a Linux. Anyone else?
Debbie and yes, Debbie and actually has a quite a well-maintained OpenQA instance,
many times by a couple of volunteers. So I have a list. I don't think this is a complete list of
about 10 OpenQA instances out in the wild. If I missed anyone, then let me know after.
Each one's kind of operating independently from the others, so there's not enough collaboration
between these today, which is what I'm going to go into in the talk. But it's very good to see,
right? We've come a long way. Well done, everyone. If you work on downstream tests, have you ever
had anyone to upstream help you write the test or help you maintain the tests?
Has anyone ever said, oh, in the next version of, you know, we're going to replace GED with
non-text editor, so I'll help you update some new tests for non-text editor.
I think we're still lacking in this area, right? The downstream QA pipelines are still kind of
invisible to upstreams, unfortunately. And a lot of people upstream of volunteers, everyone's
overworked, so this is kind of understandable, but can we do anything about it?
So what developers want is fast-food back on our changes, right? If you develop an app,
it's not much fun to get a bug report a year later. You have a contributor, they send a PR,
it has a bug, you merge it, you didn't spot the bug in a year later, you're going to bug
the contributor who sent the PR is probably disappeared and they're doing something else, and
it's all about time, right? What developers upstream want is quick feedback on the changes,
and that's how we kind of have this idealized distro model, which at the moment is experience
and friction has, you know, end users want to talk to developers directly and developers want
to ship binaries to end users directly and the sort of traditional model that we've had for 30 years,
has some friction. So being involved in, no, I put some thoughts into how we could
move testing upstream, and the first thing we did was a few years ago, now back in 2021,
and we set up an open QI instance, which is difficult for a few reasons. Firstly,
known is, and has always been independent from any distro, so we have more than zero paid contributors
from at least four companies, do I miss anyone? I always want to make the point that companies
should be more loud when they pay people to work on upstream projects, because it's very important
that companies do that, but we have more than zero paid contributors from each of these companies,
and most of them are competitors, right? So we don't want necessarily to favor one over the other,
and it's meant to be a neutral space. If we add end to end testing based on Ubuntu,
and only Ubuntu, then it's kind of a slap in the face to red hat, and the same if we add end to
end testing based on only Fedora. Also, we don't have much funding for infrastructure in
home, and always images quite big. So the way we made this work is we don't pick anyone's distro,
in the end we have our own distro called NomOS, which is a very simple, simple distro,
which is intended at this point, just for testing. Everything in NomOS is built from Master,
or Main, so it has all the latest books, breaks regularly, and that's the point, right?
And breaks regularly, and then you go, oh, this broke, why did this break? So,
like I said, I've talked about this before, so I won't go into it in too much detail,
but it was working. We've got five test suites now. It's integrated to the CI for the
NomBuild Meta repo, which is like the integration repo of Nom, so when someone releases Nom48,
that exists as a tag in this repo that then points to all of the modules that make up Nom48.
So that's where we run the tests, and you can go to OpenQA.com.org and see them. Here's a screenshot of
one of the tests that we just test, does every app start. It's maintained by volunteers,
so sometimes it works, sometimes it doesn't. And this is the list of contributors for the
Needles repo, the screenshot's repo. You can see the first one is me. The second one is Jordan,
who's one of the main people volunteering time on NomOS. The third person is me again,
and the fourth person is Dorothy, who's one of the outreach interns. So this list isn't long enough.
The QA initiative is still fairly invisible within Nom, and again, much of maintainers are
most volunteers, then all of the works. So when I chat to them, they say, well, this is great. I really
wanted to do this kind of testing, but I needed to be in my repo. I don't want the test to be
in a separate repo, and the pipeline breaks after I've merged a PR. I want the testing to happen
when somebody opens a merge request in my project, and I want the tests to be maintained as part of
my project, which seems reasonable, right? So can we do it? Well, last year we worked on a prototype,
and this was part funded by the STF, and partly by code think, and we worked on trying to do testing
upstream for NomShell. So the reason we chose NomShell is because it's the pretty much the most
difficult case. I previously did a proof of concept for a flat pack app, and that's fairly straightforward,
but NomShell is the difficult case, right? It has multiple moving parts. It's actually split across
multiple repose. It certainly can't be deployed as a flat pack. So it's hard mode, and if we can
get a prototype of that, we can potentially apply it to every module in known that has a graphical
component. So again, it's still quite difficult. We still don't have enough infrastructure,
and if we try and build a new OS image for every commit to NomShell, then we're going to go crazy,
right? So that's 4 gigabytes times, however many commits there are to NomShell every day.
We also need the tests to be fast if they're going to be in upstream CI pipelines, and
if we do it the sort of naive way, then they are not going to be fast, right? Building an image,
you can't build a Linux desktop image in less than about 45 minutes, like IO just isn't fast enough,
and then you have to download it onto the CI runner, boot, running the install that takes a while,
booting machine takes a while. This is hopefully this will work. I'm going to show you a video
of what it looks like to install NomOS in a VM. So this is what I test to doing.
That's fed up. It actually takes about 3 minutes, but that's what's happening.
And at this point, we could actually test the NomShell, right? But we've already spent almost an hour
getting to that point. With a few tricks, we can make this a bit better, let's go to the next slide.
Thank you. So with a few tricks, we can make this a bit better. Firstly, we can't make image building fast.
So let's instead of building a fresh image, we already have a stock NomOS image, and we just
need to overlay whatever's in the pull request. And systemD has a really cool way of doing this,
called the systemD6, which is fairly new. Of course, it's great because it's
just a neutral, you only have to adopt systemD, which most of us have. The way a system works is
it's a little bit like a package except for none of the things we expect in a package, right?
It's a bunch of binary files. It doesn't track its dependencies.
It's no use for deploying to end users, but very useful for testing, because you get the binary,
you overlay it on your router fast with an overlayFS in one command. Then let's say we're testing
NomShell, so you just restart NomShell, it'll start the new version. And then you can remove it again
when you want to shell it works. And you can also drop them into the file system,
add or during early boot, so you can mount them as disks, use SMS and BIOS magic to have them mounted
at early boot, and then the system just boots into an OS with whatever change you want overlaid.
So this is really good, and it should work for almost anything apart from the kernel.
That saves us most of the time. Downloading images, unfortunately, is an infrastructure problem,
which we're not going to be able to solve without more money for infrastructure. So we still have
a kind of 10-minute delay there. We don't actually have to run the installer, because we can build
a pre-installed OS image when we build NomShell, so that saves a couple of minutes. And we don't actually
have to pre-create a user, we can create a user on boot, we can do that via the SM BIOS.
Or we kind of can. There's still a system D patch that needs completing before we can do that,
the pieces are there. So we got this kind of working to a prototype standard. This is the first
part of building a 6-ext on every commit. That landed in upstream NomShell, the second part, which
is running the tests in NomOS, didn't yet land, and there's more work needed before it can land,
but we made a proof of concept, and it proved the concept, so I'm quite happy. This is a screenshot
of the real upstream NomShell CI pipeline, so this is what NomShell developers are looking at every
time they review a merger request. And we can add in a test-to-sex job, which let me explain this
diagram. I could have done a better job with the diagram, but I was in a hurry. The input to the
build-to-sex job is a container image with the build tools you need to build NomShell. Now system
D6-ex, a binary, so you have to think about ABI compatibility. You're not going to be able to
download the C6-ex and run it on Debian 9 because it's built using a much more, you know, newer tool chain
and NomOS. So what we do need to be careful of is that with the container we used build the binary
is the same ABI as the OS we're going to test it against. But that's not a hard problem to solve,
so we build a binary C6, it's available as an artifact from the GitLab CI, and then the test
C6-ex job downloads the C6, downloads the disk image, overlays the two, and starts an open QA test.
And then we can test the app in, you know, if we solve the download problem a couple of minutes.
And the really nice thing about this is that once it works for NomOS, it can actually work for
any OS, any system D based OS, could do the same, right? If you have a container with the build tools,
you take a postmarker to OS container, you build a C6 that will run against postmarker to OS,
and then you boot it against the postmarker to OS image. Once the system D support lands in postmarker
to OS. So it works pretty well, I'm quite happy with the prototype. It's definitely not ready
for prying time yet, and for a few reasons, partly technical reasons, partly infrastructure reasons,
and partly stuff that requires a bit more thought, such as, I did an open QA workshop at
Waddick, the known conference this year, and it was clear that it was the first time that most of the
people in that room had, if it looked at Open QA. So there's a lot of work to do, both to make
Open QA more accessible to people who aren't using it every day, and for people who want to do
this kind of testing to learn about what tools we have and how they work. So where do we go from
here? This is what I really wanted to talk about today. I hope you all with me so far.
Part of what we need is money to pay people to do work. A lot of this stuff isn't very fun, right?
So it'll be great to volunteer, step up and help, but in general it's not going to fit into
some of the weekend stuff that requires a bit more engineering and probably will have to pay people
to do it. The system de-patch may be quite easy, if anyone's bored and wants to look at
a system de-patch, the link is in my slides. But the next problem I want to tackle is
having a shared base of how to write these tests. Because if you think about it, you go back
when I was looking at all the Open QA instances in the world. We have about 10, maybe 10 different
Open QA test suites, like the suites, the fedora tests, the deviant tests, the Yamalynics tests.
No one has about 200 modules, not all of them make sense to do graphical testing on, but if a
100 of them have Open QA tests, we've now multiplied the number of test routes in the world by 10.
So what works today in that we have quite a lot of copy-paste and duplication between the
repos is not going to work when we have 100 such test routes. So we're going to need to work
on collaborating a bit more and figuring out what stuff can we abstract. I'll go more into that
in a bit. In fact, I'm going to go into all of this in a bit and it slides for each of these topics.
So on the topic of the shared base, when you look at an Open QA test in the abstract,
it's quite nice. The DSL is really straightforward and simple. But behind the scenes,
there is quite a lot of pull with quite a lot of history in it. This is the code that I copied
from the suites and to the numberless tests to wait for the machine to boot, which it does
by checking the serial console to see if Shell Promptors appeared yet. And there's all kinds of,
there's someone obviously spent a long time figuring out that they need to play all of the
content. I don't know. Technical difficulties.
Okay, I hope that still sounds okay. Now that's feeding back a bit. So somebody obviously
spent a lot of time in energy trying to, hopefully they'll stay put. So a lot of blood
sweatin' tears has gone into this. Ideally, we could share that and everyone who's doing Linux
system-d-base testing can share the same code. Now we realize that's difficult because
trying to re-base thousands of existing tests on a new base is not going to be straightforward.
But at least we can start by saying, what should be the shared base that we build on?
The current base test class in OS Auto-Inst, which is the test run or an open QA,
is completely generic. So you could use it to test Windows as much as you could use it to test
Linux. But that means you need all the support for testing. You need all this custom code to
wait for the boot and every Linux system has nine virtual frame buffer consoles, right? So you
have to define each of those that you want to use. We could share all that stuff. And hopefully, we will.
I've been in talks with Santiago Zalati in particular who works at Susa, who's quite interested in
building this common base. A spoiler for the end of the talk, we have a monthly call in which we
try to work through this stuff, which I'm going to invite you all to at the end, but that's my spoiler.
In terms of infrastructure, there's everyone heard about the free desktop.org problem.
So free desktop.org has a fairly major issue, which is one of the infrastructure sponsors,
who are generously donating infrastructure, have to stop doing that. And it's going to cause
some pretty major disruption, because where do you get a whole load of new infrastructure from
with three months notice? This is the reality of doing testing and open source projects, right?
It's quite hard to get an ongoing stream of funding and an ongoing stream of
paid people who maintain infrastructure. Again, volunteers, thismadmins are quite thin on the ground,
quite unreliable because they have lives and people need to be paid to do this work if we're going to
have professional quality development. In no, we also have a very small infantry team. We're very
grateful to Red Hat for sponsoring half of the person to maintain the infrastructure and the
non-foundation pays for the other person. And we have a few hard responses. Most of the
infrastructures currently moving to AWS based on free open source project credits from AWS. So again,
that's very nice. But for QA testing, we're going to have more requirements. And we're going to need
someone to contribute more hardware so that we can run more tests. Also, my dream would be a hardware
test lab, so we can actually test, you know, on real laptops and on devices. This is something
a known cell team really wants. But again, we need to figure out how to do that and where to host the hardware.
With more capacity, we could actually build more stuff as well, so we could conceivably start
providing stuff with different distros. Another thing I'm really interested to talk about is
making open QA more accessible for newcomers. So the DSL that you write testing is excellent, right?
It's very straightforward. Here's an example. If you want to send a key to the virtual
to the machine on the test, how do you do it? So you write send key and then you put the name of the key
and that's everything. You need to put a semicolon after. It's not too difficult, right? It's really
really straightforward to explain this stuff. You need four or five commands to do 90% of the testing.
But when you look at the real tests today, you realize this isn't quite a DSL because you can do anything
you can do in Pearl, and so of course people do do anything that you can do in Pearl, such as
importing 42,000 lines of help a code to do different things that isn't part of the test API.
I can see why things have got to this point, but to make things more accessible,
we really need to start breaking this up into libraries and sharing and documenting the libraries better.
So another fun project. I also personally find that the developer experience for tests is quite
hard. I didn't personally like running a web UI on my laptop to generate tests. I wrote my
own command line tool that connects to ISO to video directly and lets me run tests from the command line.
But that's my tool. I think it does more work before it's available and useful to everyone.
So going even further into the future, right? If we get to a point where the tests are
using a real DSL and they're fully encapsulated and they don't depend on stuff that's specific
to a live directory that's only available for the open-source of tests, we can actually share them.
So imagine the dream, right? The no file manager, Nautilus, provides a test rate that says,
here are tests for all the core functionality, how to connect a network file system,
how to search for a file. Those tests can run in its CI. Those tests, the same tests and run against
normal S. And then downstreams can also use the same tests. Of course there might be some
need to change the needles, right? If you patch, if you change the theme, then you'll need to update
the needles, which is expected, right? And it also gives you a way to test that your theme is working,
but this, this is a distant dream. I don't think we're going to get there this year,
but I think we can, if we think about it now, we can set ourselves off on the right path
to at least not block our way from getting there. And who knows, maybe in 10 years, we've
all be there. Some of this stuff, some of this stuff is fairly simple technical problems, and some
of this is what I call the hardest problem in software development, which is getting people to
agree on things and talk to each other. So my contribution in this space is organizing a monthly
call where it's a vendor neutral space that people who work on this stuff, upstream and downstream
can meet and talk about and to end testing. It's kind of begun with a focus on no,
I'm going to focus on open QA, but every test tool is welcome and every, every desktop is welcome.
We meet once a month, because the world is quite big, there are 24 times on, so we meet
one month it's European morning, Asian evening, and then one month it's US morning, and
European evening, the details are on this, get lab repo, get a hub repo. I think you have to
look in the poll requests at the moment. I'm hoping to merge the poll requests that has the info,
but you're all welcome to join. The next call is this Thursday at 1600 to UTC,
and I hope to see you there. And that's the end. Thank you very much.
So we have five minutes for questions or a bit less, any questions?
All right. Thanks for the talk. Just wondering, how are you integrating with the
git forges? I've seen you are kind of tied to the, like, git lobby, I'm going to end this stuff,
or how we can, for example, move it like to GitHub and other places.
Right, that's something that could be improved in an open QA today. The design of open QA
initially is that you have static workers maintained by an infrastructure team, right? So you
have, Susan has, you know, I don't know how many dedicated machines that run tests.
In no one, we don't have that much infrastructure. We don't have any more capacity to spin up,
you know, more machines. So we wanted it to run on git lab runners, because we really have
git lab runners. The way we do this is a bit of a hack, right? So each git lab runner
boots, well, starts a container image. So we use the open QA worker container image as our container
image in the git lab test. That runs a short cell script, which first registers with the
open QA web UI as a worker. So it says, hi, I'm a worker. Then it submits the job and the job
has a special, I can't remember how we do it, but we have a special parameter so that it goes to that
runner. Runs the test sends the results back to the web API and then destroys itself.
So, it's a bit of a hack, but it does actually work.
Any other questions? Of course, you're welcome to join the monthly call and ask questions there.
Well, I hope I've just tried to put the testing up food. Thanks a lot.