Test Strategies and Spotify – Kristian Karl

In Spotify, we do testing in Production System.

In TESTISTANBUL 2016 we had a nice interview with Kristian Karl from Spotify.

Please describe us the structure of the teams in Spotify?

We’re talking about cross-functional teams. Each team has a given mission, and team members own a part of the product. So, they need to have the competence and the skill sets to accomplish that. Some teams consist of backend developers only, but some teams consist of backend and frontend developers, and testers. Some teams don’t have testers. It all depends on their challenges. If I have to sum it up in one sentence, it is cross-functional.

You are calling it a tribe and a squad

Yes, true. We have ten plus tribes and 100 plus squads. By saying squads, we mean teams. Tribes are informational squads. The squad has a very narrow view of the Spotify world, they know only their mission, whereas the tribe has a broader view of what is going on. So, for instance, I’m working at the media path tribe which has essentially two missions, first is to ingest all the data non-audio and audio, like video, podcast and to give a good experience to our providers like record labels, news providers, BBC, etc. It’s easy for them to provide us the data. The other side is to provide the data to our internal customers, the feature teams that are building for the clients. We provide them the data with an interface, an API. They can populate the views with the metadata. When the time comes to play the music, we provide them with a stream. That’s my tribe’ more infrastructural, for ex. Our aisle tribe provides productivity to the rest of Spotify. Tribes have the broader view, not the full view but the wider view of the product.

You say that testing is like programming.  Would you please explain this expression?  

Programming is a very creative process. As a programmer, I have an idea to accomplish, and I need to think to put the words together, so it makes out a programming language that resembles Java or C++. There’s nobody that can do the programming for you; you need to do the programming. you with your development cycles like IntelliJ or Eclipse, but they won’t do the programming for you. They will assist you on the road.

So, the same goes for testing. When we talk about test automation, it’s not automated testing; we’re talking about automated verification or checking, which is essentially not the same thing. We only tell the programs to look for the obvious stuff. Testing is more a process where you engage the human’s intelligence and experience. When I do my testing, I observe what happens. All of a sudden I realize something is odd; a strange situation occurs where it is not supposed to happen. I can fix it using my experience and intelligence but to try to program this whole process would be very hard. In my opinion, testing is the equivalent of programming and you know how to automate programming, but you can’t automate testing for the same reasons.

The development environment that exists for developers is available also for testing. The unit tests are not testing for me, but it’s a great thing for quality. It helps the developers getting into a development methodology, where they think about the quality from the beginning.

Test automation or automated verifications are tools and methods that help me with my testing. I can make assumptions, and when we have excellent coverage of automation, I’d feel confident. We do not touch the code that much in that area, so I can focus more on exploring the system on another side instead.

How many percent of the work is and will be done by machines then?

I do not know how many percent of the tests can be automated by machines.

I would rather say that you can never replace humans with machines for testing. We can lose 20% of the testers that don’t apply. The number of tests that you can do on this system is infinite. If I can get help with automation, I can focus on further, deeper and broader tests. That’s what I want to achieve and get from automation.

I understood from one of your slides that some part of the tests can be done by the machines because they’re just like clichés and   

Yeah, right. Playlist in Spotify is a collection of tracks that I put together. Let’s say that I want to create, update or delete something in a playlist. Now I can add, remove and change tracks. I run a unit test on a very low level, I can have functional tests in a complete system to verify those cases. If I have that automated, it. I would not entirely trust the automation because it depends on how we interact with the system on the test. What if we do all the operations correctly from a client’s side and also the data point of view, when we remove a track it looks like it is rendered? Horribly, wrong. That’s where the human comes in; humans are so good at detecting anonymous or offbeat things. A user can immediately determine a serious thing which does not look good. The database says the number of tracks is correct, but the list is rendered in a wrong way. These are the things that I don’t entirely trust automation. I make sort of a judgment calling; these verifications are to help proven themselves to be trustworthy so that they may be trustworthy, but I don’t trust them completely. So it’s all about where do I spend my time as a tester?

What are the challenges you face and how do you overcome them? 

Well, the challenges are to have reliable, stable tests and the path to reach that point. If you have a solid system to test, you need to add a lot of testability; I talk to the product owners saying that we need to add this to give you what you want. I know you didn’t order this, and this will take us ten days extra, but this is what we have to do to have a product which we can have high reliability in the future. So it’s a matter of engineering, I would say it’s an engineering problem.

Is it a process engineering?

No, pure engineering. As a team, we sit down and think how we run the tests, verify, and write oracles. They need to be trustworthy. What do you need to reach? That’s where you put your main efforts into. Whatever test you have, you need to rely on them to a certain degree of level. If you don’t, all of sudden they will fail and you will have this attitude against you. Then you have a real problem. That’s the biggest challenge. It needs to be discussed within the team.

In your presentation, you mentioned that you were making tests in the Production System.  

Yes, we are testing in Production System.

It is a kind of risky statement. Can you explain how you do the tests in Production System?

When I say, we do testing in Production System; it means that we are using our production data. We use test accounts. So, we don’t use anybody’s real user accounts. We are only using our accounts, and those test accounts are the ones used in production.

Do you like to create dummy users and then do the tests ?

They are not dummy users; they are real test accounts.

As far as I know, Facebook does this , too but they have different clusters. So if something happens with the production data, something goes wrong then they change the cluster. Do you do something like this?

We always have an open discussion and thoughts on what kind of problems we might get with this methodology. We are not doing any writing. We don’t destroy anything. We are not capable of interacting with banking data or anything like that. That’s off the table. It’s not there. What we do is creating a playlist, we listen tracks, check the amount of traffic. We all live in the same ecosystem. I mean from the production point of view we are just another users. We don’t need to do that yet what Facebook does. I don’t know how they do it.

What will happen in ten years in test sector?

We will have more automated verification.

Will we see more test engineers for example?

Yeah, maybe.

Will machines with artificial intelligence take over more of the tasks?  

I guess so. I can see that. I mean ten years is huge, very far.

Let’s say five?

What I would imagine is that we will have a higher degree of automation in place, and we would start to get more stable and reliable tests in the future. I think we will spend as much time on testing tomorrow as we do today, but the tools and the information that we will have and get from our automation will make our testing job different. As a testing job, we will pretty much do the same. We have started in Spotify to work as test coaches to help teams to understand quality and testing more. They may be more conscious in the future about quality. If we look at a broader scale like banks and finance, I

Would you recommend youngster to go to university and be part of the test sector as test engineer or test automation specialist etc.?

Absolutely. According to my experience, I would say that the best testers are not the ones that are best educated, they may be well educated, no offense. But the key factor here is the experience and how they deal and handle that experience. I don’tIt , typically I would say that any good tester I know has the developing background. They attended the University, have some computer science degree, started development somewhere and then went on the quality side. I think it’s a pretty good route to start with. I think, as a tester you have to know how to program, understand how your system works on the test, both program wise and software wise. You should have a College, University education on computer science, and do some programming and understand how things work program wise and then march on to the quality side.

You mentioned that automation will increase. Will there be still jobs in the testing area?

I would say testing as a service would exist tomorrow. Typically companionable consultants who are selling their expertise will still exist, for sure. 

What would you recommend for startups? Is it a good sector to look into or try to create some solutions; developing software for example?  

Well, if it comes to software like in the old days we had WinRunner, LoadRunnerA speaker here said it’s not about selecting the tool, what you need to do first is to understand what your needs are and then see if you either build it yourself, which is probably a bad idea or provide the solution from outside. There are a lot of open source tools out there that you might want to use. I mean it depends on what your needs are and how much cycles you want to spend on the system. You can pretty much do any testing today with freely available tools. So, Selenium. If you want to do mobile testing, you can use Appium. So these tools are available. If you want to do a startup around tools, I don’t see a future there. It’s more about knowledge and how you use that side. I see more opportunities in testing as a service.

Thank you very much.

My pleasure.